The first in-person meeting of the Association of ERC grantees (AERG) during the celebration of 20th anniversary of IIT’s foundation
What is excellence in research and what is the best way to assess scientific research? These are the main questions that have animated the panel discussion of the first in-person meeting of the Association of ERC grantees (AERG), held on Friday 22 September at IIT during the celebration of 20th anniversary of IIT’s foundation. The discussion was moderated by Agnieszka Wykowska, PI of Social cognition in human-robot interaction lab at IIT and AERG Board member.
The theme of research assessment has become more and more present in the current debate about promoting science and technology in Europe, since the scientific environment has been transforming very fast and funding bodies, like the European Research Council (ERC), are changing the criteria to evaluate project proposals for grants.
“This is a fundamental issue in fundamental research”, commented Axel Cleermans President of the AERG and Research Director with the National Fund for Scientific Research (Belgium) and Professor of cognitive science at the Université Libre de Bruxelles. “People disagrees about what is meant by excellence. Among scientists we recognize outstanding works and that is why we have a perspective like the one of CoARA, that is when it comes to evaluating research it has to be qualitative and involve experts in the field. The challenge is that it takes a lot of time, for example nobody has the time to do reviews anymore, and this opens a question on how to handle it. The easier or the most tempting thing is to use only numbers, such as the number of publications, but it is not the solution”.
CoARA, the Coalition for Advancing Research Assessment, counts about 600 member institutes and universities (among which IIT) in Europe that push for a new approach, where not only standard indicators like h-index and impact factor but also qualitative metrics should be considered, when evaluating individual researchers and research projects. In July 2022 the Coalition released an Agreement on reforming research assessment (https://coara.eu/app/uploads/2022/09/2022_07_19_rra_agreement_final.pdf), which was meant like the “Constitution, the shared commitment of the community”, told us Professor Toma Susi, Researcher at University of Vienna and member of the Steering committee of CoARA. “CoARA is for achieving systemic reform by working together, in a coordinated way, to define the most appropriate methods to assess research” summarized Susi. “Our main goal is to align individual incentives with what is good for research, we want people to be rewarded career wise for doing things that make scientific research better, that help research. Currently this is disconnected”.
The European Research Council (ERC) is among the institutions that signed the CoARA’s Agreement and, as ERC President Professor Maria Leptin underlined during the panel session, the ERC is concretely committed. In December 2022, the Scientific Council of the ERC announced plans for changes to the evaluation forms and processes that are used to decide which research proposals to fund. “For me excellence is a measure on a scale; you can be excellent in publications, or excellent in communication”, commented Leptin. “The discussion is how to assess research, what are the criteria, the dimensions to consider for evaluating a project proposal”. Leptin stressed the importance of the independence of individual researchers and their ability to conduct productive, ground-breaking, creative research.
Among the CoARA core commitments there are the promotion of using qualitative methods to evaluate qualitative aspects like quality and excellence in research. “We do not ban metrics, we ask to apply the correct evaluation”, explained to us Toma Susi. “We are also asking to value more diversity in terms of contributions, for example not only papers, but also open data or software”.
The German Rectors’ Conference, represented in the discussion panel by Professor Jens Peter Gaul, was particularly skeptical about the CoARA movement and how it is communicated. “I think that the definition of excellence should be defined by the scientific community and driven by scientific criteria” he told, but “we have the feeling that CoARA is a European Commission process, an executive instrument, so it makes the German universities very reluctant to join that process”. Moreover, they found that the definition of “excellence” was replaced by a too vague definition of “quality”.
The Nobel Laureate Prof. Martin Chalfie remembered when the initial evaluation of candidates was based on an anonymous presentation, where the focus was only about their work and research project, without knowing where they were from or whom they work for. “They were chosen only because of the quality of their work. No recommendations.”. People are now more aware of their unconscious biases. According to Chalfie and his experience at National Institutes of Health (NIH), the current problem is not excellence but funding. “There are a lot of people putting together excellent projects but there are not enough funds to support them. The big question is how to choose the ones which are more likely to succeed or to have a greater impact.”
The comment of Chalfie remind us that the funding for scientific research in the European Union and in the United States will be influenced on one hand by the European elections in June 2024 and on the other hand by the Presidential elections in November 2024. The debate about excellence in research, then, will continue also thanks to citizens that will support science.
