Peer review at the Research Council of Norway: Quality assurance or border control?

Rolf Andreas Markussen; Ger Wackers

doi:10.4045/tidsskr.17.0849

Perspectives

Peer review at the Research Council of Norway: Quality assurance or border control?

Norwegian

Rolf Andreas Markussen, Ger Wackers

See All Articles

Rolf Andreas Markussen

E-mail: rolf.a.markussen@uit.no

Rolf Andreas Markussen (born 1963), Associate Professor/anthropologist, Department of Health and Care Sciences, Faculty of Health Sciences, UiT The Arctic University of Norway. Research interests include relations between scientific knowledge production and public health policy management.

The author has completed the ICMJE form and declares no conflicts of interest.

See All Articles

Ger Wackers

Ger Wackers (born 1956), Associate Professor in health sciences, Department of Health and Care Sciences, Faculty of Health Sciences, UiT The Arctic University of Norway. Research interests include relations between science, technology and society in different areas in medicine and health and care services, including end-of-life care and public health.

The author has completed the ICMJE form and declares no conflicts of interest.

Article

We will address the question of whether peer review at the Research Council of Norway within the Research Programme on Better Health and Quality of Life inhibits the diversity of perspectives and methods in the scientific production of knowledge on public health and public health policy.

It is said that when people think similarly, they do not think enough. Within the texts and speeches of academia, we keep seeing versions of this most timely reminder in everything from strategy documents to descriptions of learning outcomes, programme announcements and major speeches. Innovation, critique, transparency, interdisciplinarity and diversity of perspectives are fine words that are often heard when academic output is promoted. However, also here one may suspect that there is a gap between ideal and reality.

Based on our own experiences as applicants to the Research Council of Norway's Research Programme on Better Health and Quality of Life (BEDREHELSE), our question in this feature article is whether the conditions for production of scientific knowledge are counterproductive in practice. More concretely, we will shed light on peer review as a compulsory point of transit, and ask: who are the referees?

We are not arguing in favour of removing the peer review process. We argue that this institutionalised convention for quality assessment in academia requires respect, appreciation and knowledge about the diversity of theoretical and scientific methods. Our question encompasses both the matter of how referees are recruited and each referee's ability and willingness to acknowledge their own academic shortcomings.

The absence of such humility entails a risk of methodological tunnel vision, exclusion of scientific disciplines and preclusion of critical research questions – in other words, peer review may lead to people thinking too similarly and too little.

Peer review or academic border control?

The Research Council's public health programme invites scientific production of knowledge to "[...] promote new knowledge about the prevalence and causes of ill health and health, and about the development, implementation and effect of health-promoting measures" (1). The programme announcement thus clearly falls within the framework of epidemiology, defined as "the study of occurrence, cause and control of health disorders and illness" (2).

The language of the programme announcement thus does not attract potential applicants whose research interests lie outside such an epidemiological frame, like critical analysts of public health policy as an all-encompassing state governance project. The same is true of researchers whose approaches are based on qualitative scientific methods (3).

A broader programme announcement would clearly allow more perspectives and illuminate the field of public health more widely. It would not displace epidemiologically-produced knowledge, but epidemiology clearly has epistemic limitations, for example when attempting to understand "health disobedience", in other words why people live their everyday lives ignoring knowledge about risks, causal relationships and the effects of remedies.

It is clear that by inviting the production of knowledge within an epidemiological frame, the programme reduces the diversity of research questions and methodological approaches that reach the expert panels. The recruitment of the expert panels fosters more indirect exclusion. Referees, whose responsibilities include assessing the relevance and scientific quality of projects, are namely also recruited within an epidemiological scientific tradition. This is not particularly unusual, considering the focus of the programme announcement. It does mean, however, that projects with research questions and methods at the periphery of the programme description are peer reviewed by scientists who are referees of other disciplines. Such a gap between projects and referees entails a risk of both unqualified and hostile reading. At the same time, peer review is at risk of becoming a kind of academic border patrol that excludes the perspective of diversity from public health and public health policy.

One research project, two assessments

Together with other Nordic researchers, we submitted an application to the BEDREHELSE programme in the spring of 2016. The project was divided into three work packages. First, we wanted to investigate a selection of epistemological 'mapping machinery' and how it generates images of people's health. One example of such machinery is Ungdata, a national monitoring tool that produces snapshots of young girls' mental health (4). We then wanted to study the "public health snapshots" themselves, the materialised products of science that describe, for example, public health problems, correlations, causal relationships and the effects of remedies. In the project's third work package, we wanted to study how public health interventions are welcomed, i.e. on both the municipal level and among "ordinary people". What happens when epidemiologically-produced knowledge meets other forms of knowledge, people's beliefs and doubts, and the different ways the target groups live and organise their everyday lives? In other words, we wanted to make knowledge production and public health policy our empirical field and study it within a humanist and social science framework.

The members of the project group come from disciplines such as history of ideas, linguistics, anthropology, science and technology studies, political and power analysis, and sociology. The project positioned itself clearly in the periphery of the invitation of the programme description, and funding would require appreciation of the application's arguments regarding the relevance of the diversity of perspective and methods in public health research.

Three weeks after we submitted the application to the programme, we submitted an identical application to the Research Council's FRIHUMSAM programme. According to the Research Council, this is a thematically 'neutral' programme intended to promote, among other things, "boldness in scientific thinking and innovation" in the humanities and social sciences. In other words, topics related to health are neither prioritised nor excluded in FRIHUMSAM. In November 2016, we were informed of the outcome of both of our applications. They were both rejected, but that is where the similarities between the assessments of the two expert panels end.

Both the BEDREHELSE and FRIHUMSAM expert panels assessed the relevance of the project in relation to the programme announcements. It came as no surprise that the referees for BEDREHELSE assessed its relevance as "weak", granting it a grade of 3 on a scale of 1 to 7, where 7 is the highest grade. Such an assessment of the project's relevance made it clear that further consideration was superfluous.

However, the Research Council's expert panels also assess the "scientific quality" of projects they deem to be of little relevance. In our view, it is the difference between the two panels' assessments of the quality of the projects that leads to the more general question: who are the referees?

The assessment of the BEDREHELSE expert panel was that "the project has not been presented adequately and/or has major qualitative deficiencies. It is not likely that any new knowledge will be generated". The FRIHUMSAM panel's assessment was that "the project's objectives, research questions and hypotheses are very clearly presented and are based on an excellently formulated and highly original project concept".

Here it may be particularly interesting to note the discrepancy in the assessment of the language of the application: "not been presented adequately" versus "very clearly presented". One possible explanation is that different scientific disciplines develop their own "jargon", which can appear to be unclear and confusing for the readers of project descriptions who do not belong to the scientific traditions of which the research projects form part.

The differences in the assessment of "scientific quality" were symptomatic of the assessment of the other criteria. "Project manager and project group", "Implementation plan and resource parameters" and "International collaboration" were all assessed as weak/grade 3 by the referees in the BEDREHELSE programme and as very good/grade 6 in the FRIHUMSAM programme. The same was true of the grading in the "Overall assessment". The disharmony reached its apex in the assessment of "Boldness in scientific thinking and innovation". Here the conclusion of the FRIHUMSAM referees was a very good/grade A, and they wrote: "The project has a very high potential for scientific innovation. It is highly likely to result in substantial theoretical advancement, and/or [...] a radical expansion of knowledge. The project is exceptionally creative". Under the "Impact of the project" criterion, the BEDREHELSE referees wrote: "The project offers no significant benefit". Having been assessed as grade 3, our project was filtered out and never made it to the programme board, according to Pål Kraft, chair of the BEDREHELSE programme board (5).

In table 1, we compare the referees' assessments of a number of the criteria for the two programmes.

Table 1

Comparison of the peer reviews

Criteria	FRIHUMSAM	BEDREHELSE
Scientific quality	The project's objectives, research questions and hypotheses are very clearly presented and are based on an excellently formulated and highly original project concept. The project is in the forefront of its field and will contribute to scientific innovation as well as generate important new knowledge.	The project has not been presented adequately and/or has major qualitative deficiencies. It is not likely that any new knowledge will be generated.
International collaboration	There is a satisfactory level of international collaboration, and it is of adequate quality.	The international collaboration activities in the project are weak, and in reality non-existent.
Boldness in scientific thinking and innovation Impact of the project	The project has a very high potential for scientific innovation. It is highly likely to result in substantial theoretical advancement, and/or the development of significantly new methodology and/or a radical expansion of knowledge. The project is exceptionally creative.	The project offers no significant benefit.
Overall assessment	A project at the highest international level and of the utmost interest nationally and internationally. Publications in leading journals are expected. The researchers are leaders in their field.	A project in need of comprehensive qualitative improvements.

Referees or referees of other disciplines?

How does one explain this fundamental difference between two assessments of the same research project? We are fully aware that causal analyses always offer several options, and choose not to look at what has been assessed, but at those who perform the assessments; in other words the referees, both how they are recruited and how they handle their role.

In general, the different programmes at the Research Council have several expert panels. This information is publicly available on the Research Council's website, stating each person's name, nationality and institution. This provides fairly easy access to the referees' academic background, position, research interests and scientific publications.

A search of the members of the BEDREHELSE programme's expert panel who assessed our project gives the overwhelming impression of experienced and highly-lauded researchers from a number of European universities. The same is true of the members of the FRIHUMSAM programme's expert panel. The main difference is that the former, like the programme announcement, clearly belong to an epidemiological framework, while the latter belong to the social sciences and the humanities.

Most people who have worked on scientific commissions, expert panels and other assessment of academic texts written by themselves or others would not call peer review an exact science. This is most clear to us when the manuscripts we have submitted to scientific journals receive such contradictory responses. Even though this happens within formal frames, it is clear that just like many of the other assessments that are made in life, there is a considerable subjective component also in scientific peer review.

Self-authorisation

The inability to have a text read objectively, free of context, is nevertheless a poor argument for rejecting the process or considering all peer review to be equally valid. On the contrary, this is an argument in favour of focusing on the frames of peer review, and how they impact on the scientific production of knowledge.

Our example raises the question of whether the referees of the BEDREHELSE programme, through their epidemiological positioning, help encapsulate the phenomenon public health primarily as a matter for their own discipline. It is our opinion that an expansion of the diversity of perspectives and methods requires referees who are familiar with the perspectives and methods that are presented. There is a 'solution' to this qualification challenge in the "Assessment of grant application submitted to the Research Council of Norway" form, in that the referees of the BEDREHELSE programme authorise themselves when they tick "Yes" in the box for the question: "- I am/We are qualified to conduct this assessment".

Abels tårn [Abel's tower] is one of national broadcaster NRK's excellent programmes on research journalism and dissemination. The element gold was the subject of one of the programmes, and we watched an excellent exemplification of the value of diversity of perspectives in science (6). With gold as the empirical pivot point, we were able to shift between the lenses of physics, geology, history, anthropology, economics and other scientific traditions. Each one illuminates gold in a different way, and together they provide a broader and deeper understanding of the phenomenon. If we swap gold with public health as the empirical point of intersection, we also see the potential offered by a diversity of perspectives and interdisciplinarity.

Today both public health and health in general are empirical fields in many scientific traditions. Nonetheless, it seems as if there is little discourse, reading or research across academic barriers. We believe that bringing different perspectives together and into a dialogue with each other is productive. In order to achieve this, it will be necessary for programme announcements to put greater priority on the diversity of perspectives in science, and for researchers to incorporate different perspectives in their projects. The assessments of projects' relevance and quality also requires peer review to be managed in such a way as to reduce the risk of unqualified and protectionist reading. The alternative is to develop discipline-based, methodological ownership of empirical fields, with the unfortunate result that people think too similarly and too little.

Literature

1.
Forskningsrådet. Program for BEDREHELSE 2016–2025. https://www.forskningsradet.no/prognett-BEDREHELSE/Om_programmet/1254013199397 (7.11.2017).
2.
Susser M, Stein Z. Eras in epidemiology. The evolution of ideas. Oxford: Oxford University Press, 2009.
3.
Buvik K, Hjelseth A, Edland-Gryt M et al. Kan alle tanker måles? Morgenbladet, 13. januar 2017. https://morgenbladet.no/ideer/2017/01/kan-alle-tanker-males (7.11.2017).
4.
Bakken A. Ungdata 2017 – Nasjonale resultater. NOVA-rapport 10/17. http://www.hioa.no/Om-HiOA/Senter-forvelferds-og-arbeidslivsforskning/NOVA/Publikasjonar/Rapporter/2017/Ungdata-2017(7.11.2017).
5.
Time JK. Forskjellsbehandling i Forskningsrådet. Morgenbladet, 17. februar 2017. https://morgenbladet.no/aktuelt/2017/02/ingen-nytteverdi-forskningsprosjektet-anses-ikke-som-realistisk-karakter-3-et (7.11.2017).
6.
NRK. Abels tårn. https://player.fm/series/nrk-ekko-et-aktuelt-samfunnsprogram/abels-trn-xPt8EQk32mQCoFje (7.11.2017).

Comments ( 2 )

Dette kommentarfeltet modereres, men kommentarer blir ikke redaksjonelt behandlet ut over å sikre at de følger retningslinjer for vårt kommentarfelt.

13.02.2018:

Alle som jobber med fagfellevurdering vet at det er vanskelig å vurdere prosjekter og artikler. Vurderingen av vitenskapelig kvalitet vil aldri være objektiv, men derimot basert på fageksperter og fagpanelers beste skjønn. Vårt mål er at graden av tilfeldige utslag blir minst mulig.

Forskerne Rolf Andreas Markussen og Geir Wackers stiller i Tidsskriftet spørsmål knyttet til Forskningsrådets fagfellevurderinger. De spør om programmenes fagfellevurdering hemmer perspektiv- og metodemangfoldet. Analysen er betimelig og setter fingeren på problemstillinger som krever oppmerksomhet. Slik forskerne påpeker, er det spesielt innenfor tverrfaglige områder som for eksempel «folkehelse» at det vitenskapelige perspektivmangfoldet er spesielt betydningsfullt.

Forskerne sendte samme søknad til Forskningsrådets programmer Bedrehelse og Frihumsam, og fikk henholdsvis karakterene 3 og 6 på samme søknad. Bedrehelse-programmet er et handlingsrettet helseforskningsprogram med tematiske og målrettede utlysninger, mens Frihumsam er en åpen arena for forskning innenfor samfunnsvitenskap og humaniora, der alle tema er like aktuelle. Dette vil kunne føre til ulike vurderinger av samme prosjekt og peker på utfordringene med fagfellevurdering i programmer med ulike formål.

Fagpanelene i Forskningsrådet består av internasjonale fageksperter. Panelene oppnevnes på bakgrunn av program og utlysning, og søkerne inviteres til å komme med forslag til fagfeller. Et vesentlig element i fagpanelenes vurdering av søknader er metodikkens hensiktsmessighet for å belyse og gi svar på de forskningsspørsmålene prosjektet stiller. Vurderingen av faglig kvalitet vil alltid vurderes mot en vitenskapelig forskningsfront innen et fagfelt, og vil kunne variere fra ett fagområde til et annet, eller fra én vitenskapstradisjon til en annen. Markussen og Wackers beskriver dette godt med gullet som analogi. Det viktige er å la prosjektet bli vurdert på de faglige premisser og tradisjoner det utgår fra, også om det er tverrfaglig.

Vi ser at prosjekter som ikke svarer på målene i utlysningen, i noen tilfeller også vurderes lavere på vitenskapelig kvalitet av samme panel. Selv om fagfellevurderinger gjøres på grunnlag av gitte kriterier, kan det altså være vanskelig å skille helt mellom de ulike kriteriene som prosjektene vurderes etter. Det er derfor ikke helt uvanlig at ett prosjekt, søkt på en åpen arena og som svar på en utlysning som har helt klare tema og mål, får ulik vurdering også på vitenskapelig kvalitet.

Det er uheldig at søkere sitter igjen med en oppfatning av at vurderingen er tilfeldig, eller faglig innskrenkende. Forskningsrådet er evaluert til å ha gode rutiner for søknadsbehandling, og er i kontinuerlig prosess for å forbedre arbeidet vårt på feltet. Markussen og Wickstrøm peker på viktige utfordringer med fagfellevurderinger i programmer med ulike formål. Synspunktene deres vil være viktige i vårt videre arbeid med å forbedre prosessene for søknadsvurdering. Vi arbeider nå med å samordne de faglige vurderingene i større grad på tvers av våre programmer.

22.05.2018:

Det har gått en debatt i Aftenposten og i Tidsskriftet om Forskningsrådets vurdering av søknader og manglende kvalitetssikring av vedtak (1,2). Svar fra Forskningsrådet har dessverre vært lite klargjørende (3).

To forskere fra Universitetet i Tromsø gjenga i Tidsskriftet hvordan samme søknad som ble sendt inn til to forskjellige fagpanel i Forskningsrådet, endte opp med to diametralt ulike vurderinger (2). De sprikende vurderingene kunne ikke forklares ut fra formelle eller faglige forhold. Min erfaring er i tråd med forfatternes. Det virker som at Forskningsrådet har for dårlig kvalitetssikring av det samlede arbeid som ekspertpanel og fagkomiteer gjør.

Forskningsrådet fordeler over 9 milliarder kroner årlig og er en av de største bidragsytere til forskning i Norge og helt avgjørende for norsk grunnforskning. Rådets samfunnsoppdrag er å sikre kvalitet og relevans på innsendte prosjektsøknader. I 2015 Forskningsrådet 835 ulike prosjekter innen helseforskning (4). Et ekspertpanel vurderer og rangerer søknadene først, før en fagkomite fatter bevilgningsvedtak basert på ekspertpanelets rangering. Sammensetningen av fagkomiteen og panelene offentliggjøres på Forskningsrådets nettsider, men hvordan disse internasjonale ekspertene velges og hvordan deres habilitet sikres, er uklart.

Min erfaring er at forskningsfelt som mottar mange søknader, har sterkere ekspertrepresentasjon i panelet enn felt med færre søknader. Store forskningsfelt som hjerte-kar- og kreftsykdommer er rimelig godt ivaretatt av ekspertfagfeller, mens innen muskel-skjelettfeltet er fagfellenes kompetanse mer tilfeldig. Dette slår spesielt sterkt ut der søknader fra ulike forskningsfelt konkurrerer, for eksempel innen sektoren «Frimedbio – medisin, helse og biologi» som ikke er programbasert, og der vitenskapelig kvalitet er avgjørende.

Det er trolig grunnen til at et sykdomsfelt som forårsaker betydelige samfunnsutgifter pga. sykefravær, og som påfører helse-Norge større sykehusutgifter enn alle hjerte- karsykdommer tilsammen, nesten ikke mottar forskningsmidler. I Forskningsrådets statistikker for 2015 tildeles forskning innen kreft og hjerte-karlidelser 13-76 ganger større bevilgninger enn muskel- skjelettsykdommer innen alle Forskningsrådets kategorier (4).

Hva kan gjøres? Fagfeller i ekspertpanel med vesentlig ulik bedømmelse av samme prosjekt, må harmonisere sine bedømmelser bedre. Ved fortsatt uenighet må en tredje fagfelle inn for å kvalitetssikre karakteren. Forskningsrådets søknadsbehandling trenger et utenforstående, uavhengig kompetent tilsyn for kvalitetssikring, spesielt innenfor fri prosjektstøtte. Forskningsrådets virksomhet bør bli mer transparent, for eksempel hvordan fagfeller til ekspertpanelene rekrutteres, deres faglige bakgrunn og hvordan habiliteten sikres for de søkere/prosjekter som vurderes. Et brukerutvalg bør gå gjennom Forskningsrådets organisering og ressursbruk regelmessig. Det bør også opprettes en kommunikasjonskanal mellom søker og saksbehandler for å ta opp åpenbare faktafeil og misforståelser i bedømmelsen når den engelske prosjektbeskrivelsen er vurdert som klart formulert. Videre, når et prosjekt går over flere år med dokumentert god utvikling fra et år til neste, må dette fanges opp av Forskningsrådet.

Litteratur
1. Gautvik KM. Folkesykdommene som ikke prioriteres. Aftenposten 03.02.2018 https://www.aftenposten.no/meninger/debatt/i/MgL3AK/Folkesykdommene-som-ikke-prioriteres--Kaare-M-Gautvik (22.05.2018)
2. Markussen RA, Wackers G. Forskningsrådets fagfellevurderinger: Kvalitetssikring eller grensekontroll. Tidsskr nor legeforen 2018 doi: 10.4045/tidsskr.17.0849
3. Gautvik KM. Forskningsrådet misforstår. Aftenposten 15.02.2018. https://www.aftenposten.no/meninger/debatt/i/MgA7ER/Forskningsradet-misforstar--Kaare-M-Gautvik (22.05.2018)
4. Helseforskning finansiert av Norges forskningsråd. Porteføljeanalyse med Health Research Classification System (HRCS) 2015. https://www.forskningsradet.no/no/Publikasjon/Helseforskning_finansiert_av_Norges_forskningsrad/1254019270376?lang=no (22.05.2018)

This article was published more than 12 months ago and we have therefore closed it for new comments.

Published: 8 January 2018

Tidsskr Nor Legeforen 8 January 2018 Vol. 138.

doi:

10.4045/tidsskr.17.0849

Received 2.10.2017, first revision submitted 24.10.2017, accepted 7.11.2017.

Published: 8 January 2018

Tidsskr Nor Legeforen 2018 Vol. 138.

doi: 10.4045/tidsskr.17.0849

Received 2.10.2017, first revision submitted 24.10.2017, accepted 7.11.2017.

PDF

Print

Peer review at the Research Council of Norway: Quality assurance or border control?

Peer review or academic border control?

One research project, two assessments

Table 1

Referees or referees of other disciplines?

Self-authorisation

Formål viktig for vurderingen

Norges forskningsråds fordeling av midler

Recent Articles