RoRI invites its partners to share evidence, experience and strategic plans for applications of machine learning and AI to support grant management, research assessment and priority setting. The workshop, which will run over consecutive 3-hour sessions on 11, 18 and 25 January 2021, will be co-hosted by the Research Council of Norway (RCN). Contributors and participants will primarily be drawn from partners in the RoRI consortium, though other funders and experts are also welcome to take part by invitation.

Purpose of the workshops:

Across diverse contexts and sectors, many see applications of machine learning and artificial intelligence (AI) as the latest example of ‘general purpose technologies’, with the capacity to boost productivity and change working practices across economies. Within the scientific community, there is growing excitement about how ML/AI may be applied in research – particularly by optimising or accelerating innovative computational methods.

To date, there has been less discussion of applications of ML/AI in the design and management of the research system itself – and to processes of peer review, evaluation, synthesis and assessment – although a handful of funders are starting to experiment with this in various ways. As with all uses of ML/AI, enthusiasm about technological possibilities is tempered with concern about inbuilt biases, blind spots and unintended consequences.

This workshop will bring together a select group of research funders to share insights by:

Creating an arena for funders to share evidence and experiences with ML/AI applications and techniques;
Discussing and disseminating ‘good practice’ in emerging uses of ML/AI among RoRI partners;
Exploring what responsible uses of ML/AI would look like in the context of research management and assessment;
Identifying an agenda for further work through RoRI on these issues, linked to our broader work-stream on randomisation and experimentation.

Agenda for the three sessions:

Workshop 1 - Monday 11 January 14.00 – 17.00 GMT. Setting the scene: the possibilities of ML/AI for research management & evaluation. Join via Zoom: https://forskningsradet-no.zoom.us/j/63826817375

Welcome & introduction by Research Council of Norway, Frode Georgsen & Jon Holm (5 mins)

Scene-setter: ML/AI in research assessment: can we draw lessons from debates over responsible metrics? James Wilsdon, Director, RoRI (15 mins)

Tour de table (20 min) . Experiences, ambitions and options for uses of ML/AI among RoRI partners. Open session.

Keynote 1: The nature and difficulty of building ethics in automated decision making (or AI). Marija Slavkovik, Bergen University (25 mins, plus 10 mins Q&A)

Break (15 min)

Keynote 2: Speakable and unspeakable in AI for research on research: tools, potential and pitfalls. Daniel Hook, CEO, Digital Science (25 mins)

Panel: Emerging possibilities of ML/AI for research management, evaluation and prioritization (60 min)

James Wilsdon, Director, RoRI (chair)

Kuansan Wang, Managing Director, Microsoft Research Outreach Academic Services

Ruth Pickering, Co-founder, Yewno

Daniel Hook, CEO, Digital Science

Conclusion & wrap-up (5 mins)

Workshop 2 - Monday 18 January 14.00 – 17.00 GMTGetting into the act: applications and options under consideration by research funders. Join via Zoom: https://forskningsradet-no.zoom.us/j/68970140016?pwd=V0dUN3ZBYW51NjJ3ZnNwcjAxdGFvQT09

Welcome & introduction by Research Council of Norway Frode Georgsen & Jon Holm (5 mins)

Case presentations by partners (60 mins + Q&A)

- Assigning research proposals to panel members via text mining and optimization. Anne Jorstad (Swiss National Science Foundation)

- Assigning research proposals and expert reviewers to assessment panels.

Marie Haaland (RCN)

- Tagging of awarded projects based on a predefined taxonomy. Marie Haaland (RCN)

- Examples of the use of the neural network model BERT. Alasdair Cowie-Fraser (Wellcome Data Lab)

Break (15 mins) Case presentations continued... (30 mins + Q&A)

- Automated ex post evaluation of journal articles. Steven Hill (Research England, UKRI)

- Use of textual analysis with machine learning to assess thoroughness of peer review. Anna Severin (SNSF)

Common lessons, questions and conclusions

45 min open discussion on areas of application, technology possibilities, ethical and practical issues.

Conclusion & wrap-up (5 mins)

Workshop 3 - Monday 25 January 14.00 – 17.00 GMTThe encore: responsible uses of ML/AI & next steps for RoRI in this area. Join via Zoom: https://forskningsradet-no.zoom.us/j/67023050620

Welcome & introduction by Research Council of Norway Frode Georgsen & Jon Holm (5 mins)

Scene-setter: Trends in the use of ML/AI by research funders. Ludo Waltman, CWTS & RoRI (20 min)

Panel: Ethical and responsible uses of ML/AI in research funding and evaluation (60 min)

Alasdair Cowie-Fraser, director, Wellcome Data Lab

Josh Nicholson, co-founder & CEO, scite.ai

David Pride, Knowledge Media Institute, Open University

Jeroen van den Hoven, professor of ethics & technology, Delft University of Technology

Group discussion (with breakouts) (60 mins)

- Joint interests and common challenges

- Organizational strategies/policies

- Research and collaborative opportunities

- Role of RoRI in facilitating further work

Summing up and looking forward: what more could we do in this area as a consortium? Open session, chaired by James Wilsdon, RoRI (15 min)

Closing remarks & thanks – Research Council of Norway (5 mins)

Good practices in the use of machine learning and AI by research funders