Wie wirkt künstliche Intelligenz im Evaluationssystem? Diskussionsanstöße für die Gestaltung des Evaluationssystems von morgen

Abstract

In this discussion paper, initial considerations are made on possible changes through artificial intelligence (AI) in an evaluation system using the example of research and innovation policy evaluation. Previous discourse on artificial intelligence (AI) has focussed primarily on data protection, ethics and scientific and methodological reliability. This discussion paper aims to introduce an additional, systemic perspective: It examines how the relationships between actors in an evaluation system change when this system is confronted with generative AI. A working group of the Austrian Platform for Research and Technology Policy Evaluation (fteval) addressed this question from July 2023 to March 2024. Based on internal discussions and accompanying literature research, a model was developed that serves as an instrument for reflecting on and jointly discussing the practical actions of the actors in the RTI evaluation system. The starting points are the assumptions that generative AI is entering the existing evaluation system as a disruptive element and that these tools can change the relationships between the actors. Although the focus is on the Austrian evaluation system, it is assumed that the model could also be useful in other evaluation systems or industries with similar structures.