Abstract
The following contribution asks which role standards for research information play in practices of responsible research evaluation. The authors develop the notion of assessment standards against the background of functional standard classifications. The development of semantic and procedural assessment standards in the national research evaluation exercises of the Netherlands, Great Britain, and Italy are investigated using a qualitative case study design. A central finding of the study is that assessment standards incorporate conflicting values. A continuous tradeoff between the transparency of evaluation procedures and provided information as well as the variety of research outputs is being counterbalanced in all countries by compensating a higher level of semantic standardization with lower degrees of procedural standardization.
TopIntroduction
Research organizations, research groups and individual researchers are regularly subject to ex ante and ex post assessments of research quality in multiple contexts, such as the evaluation of grant proposals, scientific publications and in hiring or tenure processes (Butler, 2007). In many countries, institutional funding of research depends on performance-based research funding systems (Hicks, 2012; Lepori, Reale, & Spinello, 2018). Some of these systems make use of comprehensive national evaluation schemes, such as the “Research Excellence Framework” (REF) in the United Kingdom and the “Quality of Research Evaluation” (VQR) in Italy (Rebora & Turri, 2013). In other countries, standardized national evaluation systems are in place that are not tied to resource allocation but used for quality control in a context of organizational learning, as is the case for the Dutch “Standard Evaluation Protocol” (SEP) (van der Meulen, 2007)
Assessment activities comprise the use of qualitative and quantitative methodologies, such as peer review and bibliometric indicators, respectively (Moed & Halevi, 2015). Bibliometrics is “the field of science that deals with the development and application of quantitative measures and indicators for science and technology, based on bibliographic information” (van Leeuwen, 2004, p. 374). A branch of this field, “evaluative bibliometrics” (Narin, 1976), focuses on the evaluation of scientific activities by means of output and impact measurement (van Leeuwen, 2004).
Although peer review is considered to be the most viable method to assessing scientific quality, it has been subject to criticism pertaining to its lack of fairness, reliability and structural conservatism (Hansson, 2010; Reinhart, 2012). In the face of the complexity and scale of national evaluations, bibliometrics are supposed to deliver cost-effective, large-scale and often deemed more objective alternatives to peer review (Butler, 2007; Gläser & Laudel, 2007).
In the wake of the proliferation of quantitative research assessment, prominent initiatives (Cagan, 2013; Hicks, Wouters, Waltman, Rijcke, & Rafols, 2015; Wilsdon et al., 2015) call for an increased focus on practices of responsible research evaluation. These focus on producing research metrics or indicators that adhere to certain principles such as transparency and diversity.
Key Terms in this Chapter
Research Information (RI): Research information describes an institution’s research activities. It comprises information on a research institution’s (scientific) staff and structure, projects, third-party funding, publications, patents, etc.
National Research Evaluation System: A set of procedures and guidelines set up by science policy stakeholders and the scientific community to regularly assess the quality, impact and performance of publicly funded research for the purpose of accountability, funding allocation or improving research quality.
Current Research Information System (CRIS): Current research information systems comprise software solutions that merge research information from different sources and databases in order to describe and report on research (activities) in institutions and their organisational units.
Peer Review: Assessment of the merits of research contributions by esteemed members of the scientific community ('peers'). Peer review serves as the dominant evaluation method in a variety of evaluative contexts (publications, project proposals, applications for promotion and tenure, prizes and awards, etc.)
Responsible Research Evaluation: Responsible research evaluation focuses on producing research metrics that adhere to certain principles, such as data accuracy, transparent data collection and analysis or making use of a diversity of indicators
Governance: Describes the coordination of political, economic and public actors in order to guide and regulate action and achieve common goals. Major coordination modes are hierarchy, the market und networks.
Research Metrics: Research metrics refers to the measurement of research performance or impact through (quantitative) indicators. They form an integral part of research assessment methods with varying implications and inadvertent consequences for science.
Assessment Standard: Assessment standards refer to written procedural guidelines and rules regarding the planning and implementation of national or institutional evaluation processes, established by consensus by recognized stakeholders and approved by the scientific community.