Report of the ESFRI Working Group on Monitoring of Research Infrastructures Performance (WG Monitoring).

Abstract

Following the invitation by the Competitiveness Council of 29 May 2018, the European Strategy Forum on Research Infrastructures (ESFRI) established a Working Group (WG) to develop a common approach across Research Infrastructures (RIs) to monitor their performance based on Key Performance Indicators (KPIs). The proposed KPIs should provide a comprehensive framework ranging from input to outcome indicators. They will be used in the periodic review of ESFRI Landmarks and moreover, they should be useful and may be adopted by a wider range of RIs, funding authorities and stakeholders. The KPIs were developed to address the most commonly held objectives of pan-European RIs, to ensure that they are likely to be relevant and adopted by the widest range of RIs, and they were tested against the RACER criteria, i.e. they had to be Relevant, Accepted, Credible, Easy to monitor, Robust. The WG is aware that novel methods, such as those based on altmetrics are likely to significantly modify the approach to monitoring in the future. However, currently, they do not meet the RACER criteria and were not considered in the development of the system. Further details of each KPI are given in accompanying reference sheets. The suitability of the KPIs to facilitate the monitoring of performance was tested through surveys and a stakeholder workshop. The results of this consultation indicate, that given the diversity of types and missions of RIs, the KPIs can be implemented effectively if they are adapted to the specific character and context of individual RIs. While the WG proposes that this is achieved through a dialogue between the relevant parties for all of the RIs, it is of particular importance for RIs under construction, which require customized KPIs according to their phase of development. The WG notes that although KPIs are the most often used method to monitor progress towards objectives, they are often poor proxies of progress towards objectives. A move towards enhanced inclusion of narratives, such as theory of change and storytelling has been observed lately, not only in the case of evaluations, but also monitoring. The proposed methodology therefore requests that the RIs accompany the agreed KPIs with a context and develop also their own narratives.