Researchers collaborate across borders and continents. Students go on exchange and go study abroad. Nothing new so far. Yet formalised international strategic partnerships between universities are more recent. These agreements include both research and education and cover a range of departments. And they are expected to contribute to strategic goals and have great impact.

The University of Edinburgh is involved in a range of strategic partnerships, and noticed a lack of consistent, aligned evaluation practices. It proved a challenge to assess whether to enter into an agreement with a potential partner, or whether a strategic partnership indeed delivered its expectations, let alone to discuss this with their strategic partners. They invited five of their strategic partners (Universities of Copenhagen, Helsinki, Leiden, Sydney, and University College Dublin) to join hands, develop an evaluation framework and publish a handbook. The framework had to be based on state-of-the-art literature and had to lead to clear assessments of strategic partnerships.

Enter Leiden University’s CWTS and the University of Edinburgh's Science, Technology and Innovation Studies (STIS). We were asked to contribute to the project, given our expertise in research assessment, research governance and STIS’s knowledge of collaboration and internationalization.

The literature review by STIS is extensive and rich and covers internationalization, mobility, and environmental impacts of international collaboration. Yet it was not straightforward to find literature immediately relevant for international strategic partnerships between universities, nor on the evaluation of such agreements. Therefore, we first unpacked the notion of international strategic partnerships, to understand the different forms and formats it can take. This approach delivered keywords for a broader literature review, which put forward various, separate bodies of literature that are all relevant to international partnerships. The literature review provided our evidence base to develop the framework. And more on (the literature review on) internationalisation in an upcoming blogpost by Rodrigo Liscovsky.

As strategic international partnerships come in different shapes and forms, we had doubts from the start about the feasibility to develop a simple framework, with straightforward guidelines on data and a clear assessment as result. To ensure the framework could be used in a variety of situations, we wanted to take the context into account. Context is relevant both in terms of governance (e.g. how the evaluation is used), as well as in terms of the partnership, including the history of the partnership, its goals and implementation plan.

Some examples to illustrate the diversity the framework needed to cover. According to the definition developed by our partners, strategic partnerships include both research and education. However, it soon became clear that some of their strategic partnerships are research only. In addition, according to their definition, strategic partnerships are between universities, while we also heard about partnerships with a variety of partners, including local authorities. Also, a strategic partnership should be university wide. Yet several of the partnerships are focused on one or two topics only. And regarding use and governance: we noticed that some universities wanted to use evaluation to gain insight into the partnership and develop recommendations for improvement, whereas in other universities the evaluation was going to be used to inform decisions.

We heard a lot about the strategic importance of partnerships, little about its implementation and management, and even less about their evaluation. We realized that whatever framework we would develop, it would differ substantively from the expectations of a comprehensive evaluation tool. Instead, we felt we needed to address simple implementation and evaluation questions, and we wanted to ensure the framework could be used in different governance systems, and at any time in a partnership (before, during, towards the end). We therefore started to think of a framework consisting of a set of questions and were inspired by examples such as the Societal Readiness Thinking Tool and the Toolbox Policy Evaluation (in Dutch).

We realized that in order to develop a useful evaluation framework, we had to collaborate closely with the international officers from the six partner universities and take them along in the development. We proposed to develop the framework on the go. One of us had done so before, when developing a societal impact approach with and for research infrastructures in the ACCELERATE project. We asked each of the six partners to do a case study, i.e., to evaluate an existing strategic partnership and we offered to guide them through this approach. We also suggested to use a “logbook” and keep notes of any changes in the evaluation, new insights, eye-openers, questions, etc.

As a result, our partners contributed evaluations of, in random order, their very first partnership; a research collaboration focused on two topics and two universities; a research collaboration of one university with a variety of partners in a specific region; the portfolio of partners in one continent and university wide collaborations covering research, education as well as professional staff.

We organized four online meetings with each partner and planned to address different topics in every meeting. The topic of the first meeting was the history, rationale and context of the partnership and the reason to evaluate. The result would be a clear evaluation question. The second meeting was dedicated to data collection, the third to analysis of the data and the fourth and final to the interpretation of the evidence collected, and the formulation of the assessment of the partnership.

In practice the flow was iterative, and new topics arose every time. Composing the evaluation question was not easy at all and required many reformulations. The history and rationale of a partnership were addressed in subsequent meetings. The implementation of the partnership and the roles and responsibilities were brought up more than once. And when several partners wanted to reach out to researchers and students, we spent ample time talking about qualitative research methods such as interviews and surveys.

According to their own account, our partners lost confidence when they first started working on their case. They grappled with the complexity of contextual factors and the variety of perspectives of students, researchers, administrators, and the partner university. However, after four online meetings, they regained confidence when we met in person for the very first time. They presented and discussed their evaluations, thought along with each other, and advised on next steps. Several mentioned that it was all about asking the right questions and they asked each other the right questions indeed.

In the end, the framework we developed together is basically a series of questions. About the context of the evaluation (Why evaluate? What is the goal?); about the specific evaluation (What is the evaluation about? What is the central question?); and about the context of the partnership (What is the history? What are the targets?). Plus, it includes an extensive methodological section. When we presented the framework, our partners recognised the questions and accepted it as a useful framework.

The final phase of the project was the development of the handbook. We described the framework, completed the literature review and our partners described their evaluation. Perhaps the most interesting chapter is Lessons Learnt from the EVALUATE project, written by our partners. Remember how they struggled with the diverse perspectives? They now acknowledge the variety of perspectives, suggest taking these into account and use participative methods. They provide recommendations on such diverse topics as data collection, the use of evaluation and the implementation of strategic partnerships. They question unrealistic expectations and bold claims, such as turning students into “global citizens” or using research to address “global challenges”. And they advise thinking about evaluation as an opportunity to build capacity and involve and inspire partners.

We had hoped that talking about evaluation of strategic partnerships would lead to changes in the management and implementation. And indeed, when we met for the second time, we noticed how our partners were changing their practice. Several had discussed expectations and intentions with their partner universities, as well as within their own university. Something they hardly did before. Moreover, they wanted to reach out to partner universities outside of Europe and Australia and use the experience, and the framework, for these partnerships as well.

But we need to be realistic. In general, many see strategic partnerships as, well, of strategic importance. They expect great contributions, however small the partnership. And when it comes to the newest development of university alliances, expectations are even grander. Four universities involved in EVALUATE, including the universities where we are based, are part of Una Europa. The alliance believes a university is created by and for society and as such driven to be of relevance, impact and high quality. It is a laboratory of the here and now, where creativity and experimentation unlock the hidden potentials of tomorrow. Who is against this creed? Not many, we guess. But what does it mean in practice? What is the evidence? How can this be assessed? Let’s talk about our partnership!

EVALUATE is funded under ERASMUS+ Key Action 2.

This blogpost is based on the experience of the authors with the EVALUATE project and on the EVALUATE framework and handbook. Pictures are of EVALUATE meetings and of the EVALUATE framework and handbook: EVALUATE project (2022): The EVALUATE framework and handbook: Harnessing the power of evaluation to build better international strategic partnerships between universities. Edinburgh: The University of Edinburgh.