Tool Criticism and the Computational Turn A

As ever more data becomes available to work with, the use of digital tools within the humanities and social sciences is becoming increasingly common. These digital tools are often imported from other institutional contexts and were originally developed for other purposes. They may harbour concepts and techniques that stand in tension with traditions in the humanities and social sciences. Moreover, there are many easy-to-use tools for the collection, processing and analysis of data that require no knowledge of their limitations. Problematically, these tools are often assigned such values as reliability and transparency when in fact they are active mediators caught up in the epistemic process. In this paper, we highlight the need for a critical, reflexive attitude toward the tools we use in digital methods. It is a plea for what we call “tool criticism” and an attempt to think through what this mode of criticism would entail in practice for the academic field. The need for tool criticism is contextualised in view of the emerging ideological and methodological critique toward digital methods. Touching on the so-called science wars we explore knowledge as a construction and consider the importance of accounting for knowledge claims. These considerations open up an assessment of the accountability measures that are being discussed and developed in our field by individuals and institutions alike. In conclusion, we underscore the urgency of this endeavour and its vital role for media and communication scholars.


Introduction
The encounter with 'the computer' has unleashed a moment of "destabilization and deterritorialization" within the humanities and social sciences as their disciplines define themselves in relation to the digital (Rieder and Röhle 2017, 109).As our archives and our social world as a whole became increasingly digitised, researchers unsurprisingly began to develop tools to record, measure, map, and capture data to investigate the emergent digital society (e.g.Rogers 2009;Manovich 2020).With the advent of the internet and the world wide web, scholars modelled their traditional research methods to fit the emerging research object (e.g., Jones 1998;Rogers 2004;Hine 2004).From the early explorations into applying traditional methods to the research of cultural and social phenomena in the nascent digital world, a process of "bootstrapping" (Engelbart 2003(Engelbart [1962]]) unfolded to create tools to apply within these research contexts (e.g.Rogers 2013;Marres 2017;Schäfer and van Es 2017;Manovich 2020).This led to revisiting research methods and questioning the efficacy of novel ones; it has prompted a "methodological moment" (Rieder and Röhle 2017, 210).
While the use of research software is not new, the "computational turn" (Berry 2012) in the humanities and social sciences has stimulated the development and use of a wide range of tools to aid and facilitate the research process.Moreover, easy-to-use tools for collecting, cleaning, analysing and visualising data are widespread and can be employed by students and researchers who lack a "robust understanding of the concepts and techniques the software mobilizes" (Rieder and Röhle 2017, 118).These digital tools are often assigned such values as reliability and transparency (Kitchin 2014, 130) when in fact they are active mediators caught up in the epistemic process (van Es and Schäfer 2017).The situation poses dangers to the critical and interpretive traditions in the humanities and social sciences since the concepts and techniques embedded in these tools are often borrowed from the empirical sciences (Masson 2017, 25;Dobson 2019, 3) or corporate contexts.It demands a specific criticism that deals with the opacities the situation creates and entails the reflexive use of digital tools as research software.These tools are after all an important dimension of the research process as they are involved in preparing, processing, analysing and visualising data.As such, they "require some formalization of the methodology" (Dobson 2019, 8).While this proposition may seem unnecessary and even redundant, such a reflexive attitude is often lacking.
In this paper, we highlight the need for a critical, reflexive attitude towards the tools and their methods that we use in digital methods.Tools throughout this paper refer to a multitude of software applications that are used for research purposes.As scholars trained in interpretive traditions dip their toes into empirical research, they often treat tools as neutral and objective.Tool criticism, we suggest, should be incorporated into

1.
van Es/Schäfer/Wieringa • Tool Criticism and the Computational Turn research and its publications in order to support a shift from digital methods to digital methodologies.
In the first part we point to the heterogeneous development and use contexts of tools and explore digital tools and their relation to method and methodology.This leads us to introduce tool criticism as a term with the potential of uniting scholars around the need to critically reflect on our digital tools and the contexts of their development.Subsequently, the need for tool criticism is contextualised in view of the emerging ideological and methodological critique towards digital methods.Thereafter, we touch on the socalled science wars to explore knowledge as a construction and the role of non-human actors herein, and we go on to discuss, in relation to the drive for replicability and generalisation, the importance of accounting for knowledge claims.These considerations open up an assessment of the accountability measures that are being discussed and developed in the field by individuals and institutions alike.In conclusion, we underscore the urgency of this endeavour and its vital role for media and communication scholars as they find their way through this methodological moment.

Digital tools and their contexts
In academic practice research software cannot be understood as a stable term because in reality we are often using buggy prototypes, tentative solutions, and also commercially developed software.There are many tools at our disposal for data-driven research projects, and they vary in their complexity and sophistication.Easy-to-use tools for collecting, cleaning, analysing and visualising data are widespread, however, tool-building work is not a prerequisite for doing computational research.Importantly, digital methods often deal with tools rather than instruments.The difference, and challenge, lies in the fact that tools are used or appropriated for purposes for which they have not intentionally been constructed (Mey 2002).They are cruder, less tailored, less standardised objects than instruments, and their use does not occur within a web of established professional practices.
Excel, for instance, was designed for tasks in accounting rather than as an instrument for genetic research.Researchers from Baker IDI medical research institute in Melbourne estimated that the automatic conversion of gene names into dates (e.g.SEP2 to September 2) via Microsoft's Excel spreadsheet software plagued about one-fifth of publications containing supplementary gene lists in genomic journals (Ziemann et al. 2016).Excel is hardly the only software application that has been appropriated for research objectives.The contexts of tool development reflect different interests and visions of society.Take social media management software Coosto for instance.It features a sentiment analysis focused on identifying positive, neutral or negative sentiment related to a keyword, for example, a company's brand.The results are visualised in interfaces catering to marketing culture and practices.
Part of the challenge in addressing the role of tools in digital methods in media and communication studies originates from heterogeneous development and use contexts.We can roughly distinguish four contexts in which software or tools used in digital methods and humanities are developed: software and services developed in a corporate context, applications developed by researchers themselves or in a small workshop environment within universities or research institutes, community-driven development of mostly open-source research software, and software developed within traditional research infrastructures (see table 1).Here, we are not even considering the plethora of scripts used solely for the purpose of a single research project, and which often are neither mentioned nor documented.Within these development contexts there is a lot of variety too.They diverge in their division of labour, the involvement of the researcher in formulating their needs, and the commitment of developers to cater to individual research questions.Commercially developed research software often has a longer life cycle than the quickly developed scripts used for collecting data.SPSS, developed in 1976, was the first statistics programme available for personal computers.In comparison ImagePlot developed by the Software Studies Initiative (Manovich et al. 2012) is merely a plug-in for the image-processing software ImageJ (developed by the National Institutes of Health).In contrast to commercial applications, these solutions are not always updated to fit to the changing software environment.

Four contexts of research software development
Other applications depend on access to APIs (application programming interfaces) provided by social media platforms.Netvizz, a tool for scraping Facebook pages (Rieder 2013), was discontinued in 2019 after Facebook blocked the application's access citing privacy regulation.Software applications developed outside stable funding and developing contexts often cannot guarantee the continuity in development and quality control, which is provided by commercially developed research, and also by community-

Between tools, methods and methodology
The current widespread adoption and application of digital tools in the humanities and social sciences needs to be addressed.Because tools do epistemic work, their premises and appropriateness for research purposes need to be critically evaluated.This, for us, is key: to question the innocent, uncritical, and unreflective use of tools (Dobson 2019, 3).We criticise the bias that can emerge if scholars take their digital tools for granted and want to simply apply them to any and all phenomena without considering their suitability for the task at hand.Such "blunt instrumentalism" (Tenen 2016) should be an important topic in critical conversations about the future of each field of research.
Two ways of using software applications in the humanities and social studies prompt the need for tool criticism.First, the digital tools used are often imported from other institutional contexts and were developed for other purposes.They may harbour concepts and techniques that stand in tension with traditions in the humanities and social sciences.But also, in regard to the second, there are many easy-to-use tools -take those of the Digital Methods Initiative in Amsterdam for instance -for the collection, processing and analysis of data that require no knowledge of their limitations.Both these realities create an urgent need to critically assess the research methodology that is bound up with the digital tools.
Dobson points out how "these tools, the components that make up the workflow of the digital researcher, can both expand what we do and raise important questions that proposed alternative methods might not" (2019,11).Researchers thus need to understand what arguments are possible and consider their limitations.In other words, they need to be aware of the built-in assumptions and biases of the tool.It is important to stress that while tools contain methods, they should not be reduced to and understood solely as the method; they also contain other affordances, such as technological qualities, impacting knowledge production.Take Gephi's modularity feature for example (Blondel et al. 2008).When running a cluster analysis in Gephi, the algorithm starts calculating the relations starting from a randomly selected node.This means that when we run the same cluster analysis again, on the same data, the algorithm selects again a random node as a starting point and as a result some nodes with arbitrary relations can end up in a different cluster than before.
To illustrate why there is a need for a reflexive attitude towards tools we touch on two examples from our own research using the Digital Methods Initiative software tool Issue Crawler.In order to analyse the connections of modchip retailers, we mapped the outgoing URLs from a list of websites (Schäfer 2015).The software initially indicated Twitter as the centre of the network (see fig. 1).While such network visualizations might already be of limited analytical value, printed in low resolution and without any additional information about the software and the specific algorithm used for its production, they become superfluous illustrations in scholarly publications.

Misleading placement of Twitter as centre of the modchip retail network
The misleading placement of Twitter at the centre of the modchip retail network was the result of the then popular widgets on many websites linking to the related Twitter account.Twitter as a website had little to do with the activity of distributing or promoting modchips.Issue Crawler has a feature to exclude URLs from the search results, and a more conscientious user might consider which URLs will distort the research results.However, as demonstrated in the following example, reviewers should also be able to flag such issues.
Again using Issue Crawler, a different study looked at the online debate on so-called land grabbing on social media and the blogosphere (Zoomers, Gekker, and Schäfer 2016).An anonymous reviewer pointed out that a relevant player in the debate was overlooked.After several checks, the researchers discovered an organisation playing a role in the debate, that had not appeared in the search results.It turned out that they did not have a website.The embedded assumption in the Issue Crawler as well the research design expected all involved actors to have an online presence.Noortje Marres questions the use of digital methods, "Are we researching society or technology?" (Marres 2017, 116-142).The challenge for the researcher is not to take the results delivered through the computer-aided method for granted, in this case a cross-analysis of outgoing hyperlinks from a set of websites, but to situate it in the social practice of hyperlinking.Media practices, the effects of technological infrastructures and the social interaction facilitated through them become conflated in data captured from the web.This ambiguity is identified as a methodological problem of digital bias.Also referring to examples of digital method research projects, Marres notes that bias occurs on the level of data and content, as well as on the level of the research instrument and the method (Marres 2017, 123).

Towards tool criticism
The various tools used by researchers for research require a new kind of sensitivity, a practice we term tool criticism.Using tool criticism we aim to promote a rigorous practice of inquiry concerned with the epistemic work carried out by digital tools.Developed out of our earlier working definition (van Es, Wieringa and Schäfer 2018, 26), we define tool criticism as: the critical inquiry into digital tools and how they are used for various purposes within the research process.It reviews the qualities of the tool in light of, for instance, research activities, and it reflects on how the tool (e.g., its data source, working mechanisms, anticipated use, interface, and embedded assumptions) affects the user, the research process and output, and its reliance on the user's training.This is especially relevant when those tools are borrowed from other contexts.
In our view, all tools that are leveraged to support the acquisition, cleaning, analysis, and communication of data are considered to be data-driven research tools.In addition, we call for the reflexivity of their entire "technical stack" (Kitchin 2018).The stack is understood as "the interlinked and dependent layers of tools and abstractions that make up any contemporary computational application or procedure" (Dobson, 2019, x).While there are fragmented discussions taking place in media and communication studies on the impact of tools on knowledge production, a term like "tool criticism" can help bring these efforts into some kind of dialogue.
Some confusions might arise when using the term tool criticism; let us be clear from the outset.First, tool -used in the popular vernacular to denote someone who is used and obedient -might potentially connote that its role is simply to replicate data: the opposite of what we are saying, which is that tools are in fact mediators that actively translate data.Second, our use of the term could be mistakenly regarded as placing emphasis on the tool rather than on the relation and interaction between tool and researcher.As expressed in our definition, we recognize tools as relational.
We are not alone in this call that critical attention be paid to tools.In fact, the term "tool criticism" is gaining a bit of traction, but mainly, as yet, within the digital humanities.For instance, a special issue of Digital Humanities Quarterly in preparation by a group of scholars from the Netherlands deals with tools criticism.Our definition of tool criticism, furthermore, shares some similarities with the work on digital tool criticism by Koolen, Van Gorp, and van Ossenbruggen (2019); we too find it necessary to reflect and report on the impact of digital tools on research practice.However, as described elsewhere (van Es et al. 2018, 26), there are four key differences that, here, for the reader's benefit, we will summarise briefly.
-First, for us, tool criticism very much encompasses reflexivity in regard to the interaction between researcher and tool.-Second, tool criticism, being part and parcel of the academic ethos, extends beyond a checklist of questions in a fundamental way.-Third, making a distinction between tool builder and researcher is problematic, because in many instances the two roles have converged.-Fourth, tool criticism is a source not just of reflection but also of action: it will solicit novel and improved tool development.Importantly, and here we hope to play some role, the term "tool criticism" needs to move beyond the boundaries of the digital humanities and begin working for media and communication scholars at large.Again, tool criticism, drawing as it does on established scholarly practice, proposes nothing radically new.However, we are responding to a methodological moment in media and communication studies.We would argue that media scholars are uniquely equipped to develop a critical understanding of how their

4.
tools (which are essentially media) affect their research process and the interpretation of their outcomes (van Es and Schäfer 2017, 17).

The new empiricism
The access to novel data resources and the availability of tools for analysing data has sparked debates within various disciplines about established methods of conducting research.Attempts to branch out and to adapt methods or to develop novel ones are often met with severe criticism.In sociology such a discussion started taking place twenty years ago.Under the title "Reflections on the Future of Sociology" (2000), Andrew Abbott notes how the increasing availability of data and large datasets, the possibility to use an indefinite number of variables and repeated measures for analysis, and the possibilities of storing vast amounts of data, will eventually affect sociology.He expresses doubts that the discipline is fit to respond to the changing research landscape: Our idea of data dredging is having graduate students run thousands of regressions predicting one variable that we happen to have decided to examine.At that rate, it would take us a century to begin to think about any one of these giant new data sets.Nor is it just a matter of ramping up existing methods.We have to rethink data analysis from the ground up.(Abbott 2000, 298) The same situation is now unfolding in the humanities and social sciences.A pragmatic approach to that challenge is left to individual scholars trying to embrace the opportunities of the changing research field and develop new methods for it (see e.g., Marres 2017).What we see taking place now are heated debates about the accuracy, efficiency, and methodological purity of these new approaches.While these debates are useful for revisiting novel practices and questioning their appropriateness for the research agenda and particular research projects, they often carry the dismissive tone of the turf war.
The emerging critique of digital humanities, which surfaced roughly a decade ago, and the use of digital tools in disciplines such as media studies (digital methods, cultural analytics) seemingly address two main issues: ideology and methodology.In terms of ideology, the digital humanities are criticized for an alleged neo-liberal agenda undermining the very essence of humanist scholarship, favouring engagement with technology and the development of managerial skills over the inquiry and interpretation of texts conducted by individual scholars.In regard to methodology, they are called out for being scientifically inaccurate, unnecessary for the research tasks at hand, or as manifestations of a techno-fetishist and empiricist attitude.Both these issues can be seen in the discussion summarised below.
The ideological criticism we identify is found in Stanley Fish's scathing account of the digital humanities in three New York Times op-eds (Fish 2011(Fish , 2012a, 212b), 212b), voiced in Grusin's "Dark Side of the Digital Humanities" (2014) and articulated in Allington, Brouillette, and Golumbia's "Neoliberal Tools and Archives" (2016).The criticism of these scholars also targets the ample funding digital humanities infrastructures and projects have received.There are certainly examples of well-funded projects and initiatives for building digital methods infrastructures.It is also undeniable that there exists an unreflective enthusiasm among university administrators for setting up centres for digital humanities.However, this does not deter from the fact that 'the digital' also offers the development of a skillset that allows humanities scholars to become more than mere commentators at the side-lines of society and engage actively with societal sectors while creating an effective impact (van Es and Schäfer 2017, 16-17).It has the capacity to enhance rather than restrict their capacity for cultural criticism.

5.
van Es/Schäfer/Wieringa • Tool Criticism and the Computational Turn The ideological critique also informs a methodological critique in the accusation that digital methods favour programming and design activities over critical inquiry, are dissociative in establishing a division of labour, and prefer collecting data and building archives rather than engaging in critical inquiry and political commentary.Identifying all these elements as cultural phenomena, Alan Liu argues that there is a lack of cultural criticism within digital humanities which demonstrates insufficient effort and capacity to critically deconstruct these contexts (2012).
Building further upon this sentiment, another criticism comes to light: the inadequacy of both new methods, and the inaptitude of researchers to apply them.From the informative disciplinary dispute between Nan Z. Da and Andrew Piper over the role of quantitative methods in the humanities, we can identify a number of misperceptions about digital methods that relate to these criticisms.Whereas Da (2019) provides cautionary tales and offers a dismissive take on computational literary studies, Piper (2019) fires back and underscores the rich opportunities.
In Da's writings we encounter the expectation that computer-aided methods in computational literary studies lead to an improvement in reproducibility.These lacunae are related to the expectation, also found in the field of media and communication research, that digital methods constitute a more objective process of inquiry, detached from the researchers' specific standpoint.When colleagues note the extent to which researchers are involved in selecting, enriching, or reducing data, and adapting parameters, they remark that these researchers are simply tweaking the results until they get what they desire.The connotation that digital methods are more objective and more accurate than their own qualitative methods is not borne out, however, which leads ultimately to misinformed disappointment in the alleged qualities of digital methods, even as their actual quality is overlooked.It neglects to account for general problems with reproducibility and the field's general objectives and methodological practice.As such, Da's critique of the method is simply a vehicle for the ideological critique that the digital humanities' quantitative methods are inferior to the traditional literary studies' qualitative ones.
We do not follow in this dismissal of digital methods, emphasising, rather, we argue that there is a need to reflect on the use of tools and their relations to the researcher and the epistemological process.We do, however, align with Da in arguing against dataism; a rather positivist understanding of data analysis which goes hand in hand with the "myth of big data" (Couldry 2017).The empiricist vision of data analysis is more palpable in popular commentary, techno-optimist marketing discourse and public policy making.Scholars were quick to point out these "fallacies of empiricism" (Kitchin 2014, boyd andCrawford 2011).But, furthermore, within academia the availability of novel data resources and easy to use analysis tools lead to "a renewed positivist dream" (Dobson 2019, 3).Sociologist González-Bailón connects this empiricism to 19th century emergence of positivist sociology such as Quetelet's 1835 publication "Social Physics", which resonates literally in Pentland's "Social Physics" published in 2014 (González-Bailón 2017, 12).Dobson identifies the half-hearted attempts of digital humanities scholars to counter the positivist approach by claiming that there would be a "human interpretation" but only of the computer-aided output of analysis therein leaving the method free of interpretation (2019, 63).
In his enthusiasm for digital methods and counting words, Andrew Piper -from the above-mentioned debate -can easily be accused of representating the positivist (or dataist) attitude.It can also be found in parts of digital methods research in media and communication studies more broadly, which uncritically places greater emphasis on the outcomes of co-occurrence or vector analysis of words, the cluster analysis of social media accounts, or the simple counting of comments, likes, or views than on a situated analysis of these indicators in media practices and within their social and economic context.Nan Da is right to reject the use of graphs, vector calculations, and word counts that produce vague, often meaningless and rather formalistic outcomes, and at times even erroneous visualisations and flawed results (Leufkens 2020).
As discussed, some scholars continue to dismiss digital methods flat out.Such refusals are unproductive, but we do have to tackle the issue of blunt instrumentalism.In the humanities, there have been calls to reinstate a critical and interpretive tradition in what has been termed critical digital humanities (Berry and Fagerjord 2017;Dobson 2019).In line with the push for a critical digital humanities, we propose tool criticism as a way to implement digital tools in both the humanities and the social sciences.Tools do not stand on their own but are deeply embedded within knowledge communities; they are subject to changes and distinct ways of use and implementation within research processes.What needs to become part of the questioning in digital methods research is the way tools are used.Such reflection should also involve the recognition that tools and their effectiveness are determined by the skills and experience of their users.The more expert users are in using a tool, the more they are aware of the limitations and the appropriateness of its application in various contexts.

Bringing knowledge claims to account
As explained above, the plea for critical reflection on the choice and application of methods is far from new.The digital tools increasingly used in media and communication studies exert their impact on knowledge production.However, this does not mean that knowledge obtained through the application of digital methods research can be reduced to mere constructions.This question around the existence of a reality independent of human observers returns us to the heated debate between realists and postmodernists that has been dubbed the "science wars" (see e.g.Parsons 2003).This quarrel raised questions about the scientific method and its aim of arriving at objectivity and truth through the use of logic and empirical evidence.Thomas Kuhn (1962) made room for the sociology of science when he argued that sociological factors play a role in the acceptance of new paradigms.The idea of science as socially constructed helped give rise to a new field of study, "the Sociology of Scientific Knowledge" (SSK).A variety of SSK known as the Strong Program was represented by a group in Edinburgh that was developed by, amongst others, David Bloor.Another strand in Bath, known as the empirical program of relativism, was associated, amongst others, with Harry Collins.
Influenced by ideas from the Strong Program, but otherwise working independent of it, the sociologists Bruno Latour and Steven Woolgar 1979 published "Laboratory Life: The Construction of Scientific Facts" (1986 [1979]), an anthropological study of a laboratory, where they traced how scientific knowledge is a cultural practice.It was an influential publication for the then emerging field of "Science and Technology Studies" (STS).During his field work, Latour observed that facts were not simply discovered; knowledge was in fact produced through heterogeneous networks.This claim challenges the autonomous and independent status of facts drawn from the work of scientists.Highly relevant to our plea for tool criticism is their insistence "on the importance of the material elements of the laboratory in the production of facts" (Latour andWoolgar 1986 [1979], 238).
In a similar move away from realism, Donna Haraway (1988) lays claim to situated and embodied knowledge.As she puts it, "The alternative to relativism is partial, locatable, critical knowledge sustaining the possibility of webs of connections called solidarity

6.
van Es/Schäfer/Wieringa • Tool Criticism and the Computational Turn in politics and shared conversations in epistemology.Relativism is a way of being nowhere while claiming to be everywhere equally" (584).However, radical relativism has haunted the constructivist project because it was appropriated by outsiders to that project and used to criticize all forms of scientific knowledge, leading to the emergence of a post-truth world.
Both Haraway and Latour, however, debunk strong constructivism, sharing the view that the 'real world' is not simply a screen on which to project, nor simply something to be discovered, but an active agent involved in the production of knowledge.Latour (2003) points out how realism has come to mean the opposite of constructivism and how constructivism is often substituted for social constructivism, leading to the false notion that the construction is made of social stuff.Latour has attempted to repair constructivism as a concept.He argues that the concept reflects not the material but the collective process through which matters of fact are built.In his account, the social and technical are linked entities both involved in the process of construction.Latour (2004) has claimed his intent "was never to get away from facts but closer to them, not fighting empiricism but, on the contrary, renewing empiricism" (231).Calling for a realism dealing with "matters of concern" rather than "matters of fact", he argues that authority can be claimed by laying out the robust network, the practices and institutions involved in the manufacture of science.He urges us not to debunk but to protect and care.By erasing the labour and expertise that are part of research, "the difference between good and bad science, well designed and badly designed experiment, well fabricated and badly fabricated facts has disappeared" (Latour 2003, 87).
Interestingly, Latour has himself formulated a celebratory and somewhat misguided view of how digital methods revolutionise the social sciences.Together with Venturini he argues that 'digital traceability' (Venturini and Latour 2010, 7) erases the old micro/ macro problem in sociology in that digital traces (micro-interaction) provide direct access to the construction of social phenomena (macro-structure).Couldry and Hepp (2017, 163) find their argument flawed: it reduces the complexity of the social world to a "flat plane" and, as such, has limitations in terms of analytical value.Moreover, they treat digital traces as neutral, despite being shaped by the technical procedures of powerful organisations who inscribe their interests and visions of society on the technologies and tools harnessed by the research community (ibid.).
Regardless of such criticism of Latour's take on digital methods, being transparent about and accounting for the work that goes into research provides the means of differentiating good and bad constructions.Similarly, Haraway (1988) writes that responsible knowledge claims are those that can be called to account.Transparency, though necessary to achieve accountability, does not equate to it (Meijer 2003;Meijer 2014).Whereas transparency is a passive openness, accountability is an active relationship, in which one's conduct may be scrutinized and judged by specific fora and under particular circumstances (Bovens 2007).In other words, responsible knowledge claims not only make transparent how they have been constructed, they also provide a justification and account as to why such claims were constructed in the way they have.
Such responsible claims also give rise to questions of reproducibility and replicability.Here, there are important differences to note between empiricist/constructivist epistemologies and interpretive/constructivist epistemic cultures.Holbrook, Penders, and de Rijcke (2019) hold that accountability for research design is important in the humanities; but it needs to be organised differently, as it often relies on interpretation: [...] humanities approaches (including their practices of reporting) allow researchers to deal with the (im)possibility of replication by giving particular accounts of the consequences of methodological decisions and the role of the researcher.(Holbrook, Penders and de Rijcke 2019, np) What the humanities value adds, they argue, is a diversity of arguments.Similarly, Meyer and Schroeder (2015) explain how, in contrast to the physical sciences, the social sciences and the humanities are less aimed towards cumulative knowledge production.Rather, new and competing interpretations may be provided.In other words, they offer partial, locatable, and critical knowledge.
Digital methods are likely to inspire different responses and generate their own scholarly branches within the humanities and social sciences.Such responses will be found among those who are increasingly empirical and positivist and those who take to using digital tools to enhance traditional interpretive abilities and criticise scientization (Meyer and Schroeder 2015, 204).A drive to replication has now also been introduced into these disciplines, necessitating the establishment of procedures and practices for replicability that are supported by the given discipline.This was central to the aforementioned debate between Piper and Da about computational literary research.Piper acknowledges the challenges faced by traditional critical methods for the practice of generalisation, referring specifically to selection bias and the failure to provide a rationale for selections made in the construction of a corpus.It is the observer who, in relation to the tools and techniques employed, constructs a particular perspective, and here we have a responsibility to render these perspectives visible and leave them open to discussion.Of crucial importance, we find, is expressing a rationale concerning the suitability and reliability of the research design and process.

Practical steps towards accountability
We have here made a plea for tool criticism within digital methods research.We have shown methodological disputes to be a recurring theme as fields evolve and, harking back to the science wars, have argued for the importance of accountability in the research process.It is important to offer insights and motivations as to how research results came into being.There remains, however, the more practical question of how to incorporate tool criticism into the research process.Here we are immediately confronted with two spheres in which this plays out: the individual and the institutional.Individual solutions address all attempts to counter the issues raised through using tools at the level of the individual researcher, research group, department or university.The institutional level calls for a broader approach of national and international research associations, university networks, and funding organisations to develop a web of established professional standards.The latter requires more time and negotiation than the former.We will sketch several attempts to incorporate tool criticism into the research process below, as well as identify the drawbacks of these efforts.

Individual solutions
Confronted with the goal of implementing tool criticism at our own department, the Utrecht Data School, we developed, in collaboration with the Utrecht Digital Humanities Lab, a "fieldnote" plugin for Gephi, a network analysis and visualisation software package.This project arose from our own struggle to account for decision-making while working collaboratively on visualisations in larger projects and from our frustration with academic publications that offer very limited insight into, and reflection on, how visu-

7.1
van Es/Schäfer/Wieringa • Tool Criticism and the Computational Turn alisations were made (see Bruns 2013).Elsewhere we offer a fuller discussion of our aim to make network visualisations in Gephi more accountable (see Wieringa et al. 2019).
Here, it will suffice to mention how the plugin logged interactions with the software, allowing details of the working process to be exported and enabling scholarly reflection via the facilitation of greater transparency about the work process.With reference to Haraway, we saw the plugin as a means to facilitate the 'critical positioning' of its users.
There are two drawbacks to this approach.First, transparency does not automatically lead to accountability or critical reflection (but does facilitate it).Second, we run into the problem that this approach does not account for the software's methodological substance.With reference to Gephi's various layout algorithms, Rieder and Röhle (2017) comment on how each one produces specific interpretations by revealing aspects of the structure of the underlying data as a graph in unique ways.They propose that a mastery of graph theory and spatialisation theory are needed to understand the methodological substance of the software; a critical attitude alone is insufficient, and what is actually required is a "deeper involvement with the associated knowledge spaces to make sense of possibilities and limitations" (119).Learning how to code does not solve the problem for users and does not give rise to a complete understanding of the research software being used.Drawing on our own practice, we find (interdisciplinary) teamwork to be a promising avenue.Here individual researchers deepen their own knowledge of and expertise in particular tools, which, when combined with a general understanding of the associated tools, allows them to engage with the relevant logics and principles.As a result they are able to grasp the basic implications entailed by the results' interpretations.
In a similar pursuit of accountability, James E. Dobson (2019) makes a plea to use the free and open-source Jupyter Notebooks to share and publish workflows.A notebook integrates writing and running code (e.g.Python) and adding rich text elements in a single document.
Figure 2 is a screenshot from a Python tutorial from the Utrecht Data School in Jupyter Notebook.Here you can see how comments have been added in the code blocks Screenshot of the Utrecht Data School Python Tutorial by Max Boiten in Jupyter Notebook Dobson (2019, 41) celebrates the fact that the tool allows researchers to add comments.using the symbol # and how the code blocks themselves are annotated in commentary blocks.Dobson says researchers can use these affordances to communicate decisions and signal issues.This accountability 'solution', however, faces the same drawbacks as identified in relation to the Gephi plugin.
In light of our plea it is interesting to note that there are affordances of Jupyter Notebook that complicate documentation of this kind.1 First, all variables that you assign during a session are stored in memory.This feature is useful because it allows you to successively write and execute code: if you make an error in step 20 you don't have to redo steps 1 to 19.But if you have made variables and later adjust their labels or remove the code in which you have defined the variables from your notebook, the variable is still stored in memory (unless you explicitly write the variable's label in Python, but this is not common practice).The consequence is that all code blocks in which the adjusted or removed variable label is still used continue functioning during your session in Jupyter Notebook.However, when other researchers want to reproduce these results on their computers, not only do these code blocks not function, but how these variables came to be it is now untraceable .
Second, Jupyter Notebook, with its code block structure and memorization of variables, enables nonlinear work.You can make a variable in a code block (x) and place a code block above to do something with the variable (x+1) without causing problems during your session.
[11] y = x + 1 [4] x = 1 This will not, however, work in a new session.Such problems are rather easy to prevent before sharing your notebook you can restart your session and run everything again.But running code that takes several hours or days to execute is impractical and requires prior consideration.The alternative of writing code in a simple text file solves these problems because it forces linear code writing and the explication of variables in all cases.However, to do so is to forgo the benefits of Jupyter Notebook or the affordances of other integrated development environments (IDE).
While there are admirable steps that individual researchers and research teams can take toward accountability, both the Gephi plugin and Jupyter Notebook allow the decisions and procedures of the researcher to be documented, but do not consider how the tools operate as mediators in the construction of knowledge.Moreover, as Bruns (2013) has pointed out, there are also spatial and format limitations confronting the publication of data research.In regard to the former, there is not enough room within academic publications for elaborate reflections on research findings and results.This is important as the tools and methods have not matured and are not yet sufficiently documented elsewhere.In regard to the latter, more interactive and dynamic modes of presenting results are needed.Print publications, for instance, restrict how results can be communicated by glossing over the depth and layered nature of the findings being presented.
The recognition that there need to be more accountable methods is also evident from the development of checklists for digital tool criticism in the digital humanities (see Koolen, van Gorp, and van Ossenbruggen 2019) and Digital Methods (see Venturini et al. 2018).Yet, as van Geenen (2018, 34) writes, "'accountability by design' does and should not equal following checklists [sic] or 'ticking boxes'".Accountability requires an active justification and explanation of one's conduct.Checklists might facilitate re-flection on one's tool (as with the Gephi plugin) but do not equate to it.Checklists and plugins may help prompt critical reflection, but ultimately this kind of reflection on tools should be part of the academic ethos; it can never be wholly outsourced.

Institutional solutions
A number of institutionally oriented solutions have been proposed.For instance, the peer review of tools.Joanna Swafford (2016) discusses the example of the software package Syuzhet to illustrate this need: "the package incorporated an overly simplified version of sentiment analysis and a poorly chosen signal processing filter; the latter problem in particular led to distortions that, in extreme cases, could actually invert the results such that the tool reported emotional highs at the novel's emotional lows" (557).She notes how there had been a failure to explore the tools' methodology of code and speculates on how peer review could have contributed to the use of a more reliable tool that would have required collaboration with programmers and experts in the relevant fields.While her suggestion seems reasonable enough, the treatment of tools as objects should be questioned because here the tools' interaction with underlying datasets and the users themselves is ignored.As is clear from debates about the accountability of algorithmic systems, distortions cannot be disclosed without also taking into account the underlying datasets and other interactive elements in the system's environment; a static code review can tell you only so much (Kroll et al. 2017, 647-650).
One serious challenge lies in the perpetual beta state of digital tools.Social media data research, for instance, is currently facing the transition into the post-API age as commercial platforms, after a series of privacy scandals, have taken to further restricting access to their data streams.We see this tightening of access as one of the biggest challenges for those scholars who, like ourselves, study the digital.Because these objects are constantly revised and updated, there is hardly any time to build an increasing affinity with and understanding of the tools we use to analyse them.Here some form of stabilisation and formalisation needs to be reached within national and international research associations.
There are signs of institutional maturity on the horizon -for instance, the network visualisations in Gephi, where the use of the graph layout algorithm ForceAtlas2 is rather common.Significantly, the creators of the software and algorithm alike have provided insight into its functioning in a technical paper which is now commonly cited in publications (see Jacomy et al. 2014).Referencing such a publication is itself a form of accounting.It relieves scholars of the need to provide lengthy explanations of this algorithm and their selection of it for the purpose at hand.The availability of these types of technical papers are crucially important to establish standards and practices of academic conduct.A next step would be for research associations and publishers to define standards of how to reference tools in publications, and for reviewers to pay attention to the way these tools were used.They also could define standards for incorporating high resolution data visualisations into publications while referencing the tools and data used for creating them.
Bolder attempts can be seen within the digital humanities project 'WhatEvery1Says' by 4Humanities.org,that ran between 2017 and 2020.Here, standards of openness, shareability, and reproducibility were created for methodology within the digital humanities.More specifically, the project, in the process of developing tools and guidelines, includes "a manifest schema for data-provenance and processing-steps tracking, an integrated workflow management and virtual environment, and a topic-model interpretation protocol" (see Liu et al. 2017).Such efforts are desirable but will take time and 7.2 require deliberation and debate.Moreover, critical reflection needs to be an ongoing process: formalisation and the implementation of standards cannot be the end of the conversation, nor can we outsource reflection to schemes, plugins, and checklists.These aids merely serve to help support and make explicit the research process itself -enabling the community to assess whether its construction is good or bad.

Conclusion: Doing tool criticism in research practice
The computational turn puts media scholars in an unprecedented situation: they can effectively inform policy making while also adding to current knowledge and educating future workers (see van Es and Schäfer 2017).This development also puts tools at our fingertips that vastly expand the opportunities for conducting research and allow us to engage in the very activities that currently shape our digital society.Combined with our expertise in cultural complexity, mediatization, and media uses, this position makes us distinctly qualified to mount critical interventions.But essential to any such action is a critical grasp of the tools we use, the formative tools for generating knowledge.In possessing such an understanding, we hark back to traditional scholarly practices such as the critical inquiry of sources and methods.We are not claiming to be reinventing the wheel with tool criticism, but we are making an argument for a mature, critical engagement with the tools we use.The emerging criticism directed towards digital methods marks the current methodological moment of our field of research.If it is a crisis, it is one we must not waste, and we should regard it as an opportunity to take a critical look at how we do research and how we make use of tools.
Tool criticism is a way of confronting the criticism that digital methods are used superficially, inaccurately, or in overly formalist or misleading ways.The first step would require us to describe best practices for using tools, develop standards for the use of data visualisations in publications, and train colleagues and students in using tools and engaging in tool criticism.Tool criticism, as a practice reflecting our relation to research tools and how it affects knowledge generation, constitutes an important part of accountable research.
Another important aspect which we could not address in this article but feel the urge to mention are the legal aspects and limitations inherent in using tools for collecting data from web platforms (e.g. via APIs) or from licensed databases (such as NexisLexis).Here, the emerging branches of digital methods, cultural analytics, and digital humanities have not even started to develop guidelines or best practice, let alone a thorough argument and advocacy for the freedom of science.
Tool criticism serves another task that is of increasing societal importance.As expertise and science are under pressure from rising populism and a brave new world of fact-free politics, they are challenged in public debate.Their results are ridiculed, misrepresented, and rejected without factual arguments.Explaining the workings of tools, models, and the methods of scientific research to policymakers and the public more broadly will become an even more important task for scholars in the years ahead.Again, media scholars who have the expertise to critically inquire into how media shape our worldview might well be the very scholars who can fulfil this essential role in informing public debate.With an eye to the increasing politicisation of expertise where widely available data and analysis tools fuel ideologically framed debates, the critical awareness of the epistemic impact of tools and the literacies and skills of their users becomes ever more important.

Table 1 :
van Es/Schäfer/Wieringa • Tool Criticism and the Computational Turn developed software solutions (which are often sufficiently funded as well).The different contexts of development, users, markets, and funding in which research software applications emerge impacts complexity, stability and appropriateness for specific research objectives.