Launching NOR-CAM – A toolbox for recognition and rewards in academic careers

What is the future of academic career assessment? How can open research practices be included as part of a research evaluation? These were some of the questions we asked ourselves in a working group set up by Universities Norway. Almost two years later, the report is ready. Here I will share some of the ideas behind the suggested Norwegian Career Assessment Matrix (NOR-CAM) and some of the other recommendations coming out of the workgroup.

The Norwegian Career Assessment Matrix (NOR-CAM).

EUA work on research assessment

I have for some years been Norway’s representative in the European University Association’s Expert Group on Open Science/Science 2.0 (on a side note, I have written elsewhere about why I think it should be called Open Research instead). The expert group meets 3-4 times a year, usually in Brussels but nowadays online, to discuss how Open Science principles can be developed and implemented in European universities.

A lot of things have happened in the world of Open Science during the three years that I have been in the expert group. Open access to publications is improving every day. Open access to research data is coming along nicely, although there are still many challenges. Despite the positive developments, there is one key challenge that we always get back to discussing: research assessment. How should researchers get their “points” in the system, who should get the job, and who should get a promotion?

Up until now, publication lists and citation counts have been the most important “currency” for researchers. We have, over the years, seen an unfortunate focus on metrics, like the h-index and the journal impact factor (and others). The challenge is that only asking for publication lists (and publication-related metrics) takes focus away from all the other elements of an open research ecosystem.

Various building blocks in an open research ecosystem.

The need to rethink research assessment led to the EUA Webinar on Academic Career Assessment in the Transition to Open Science last year. As the title of the webinar shows, we decided to broaden the perspective from only thinking about research assessment to considering academic career assessment more generally. This also became the focus of the Universities Norway workgroup and the final report.

Six principles

In the report we list six principles for the future of career assessment:

  1. Measure quality and excellence through a better balance between quantitative and qualitative goals
  2. Recognise several competencies as merits but not in all areas at the same time or by each employee
  3. Assess all results, activities and competencies in the light of Open Science principles
  4. Practice transparency in the assessment and visibility of what should be recognised as merit
  5. Promote gender balance and diversity
  6. Assist in the concrete practice of job vacancy announcements and assessment processes locally

Four recommendations

The work group then went on to suggest four recommendations for different actors (individuals, institutions, research funders, government):

  1. To establish a comprehensive framework for the assessment of academic careers that:
    • balances quantitative and qualitative goals and forms of documentation for academic standards and competencies
    • enables diverse career paths and promotes high standards in the three key areas: education, research and interaction with society
    • recognises the independent and individual competencies of academic staff as well as their achievements in groups and through collaboration
    • values ??Open Science principles (including open assessment systems)
    • values and encourages academic leadership and management
  2. To engage internationally in developing a Norwegian assessment model because:
    • changes in the assessment criteria cannot be made by one country alone
    • a Norwegian model can contribute to related processes internationally
  3. To use NOR-CAM as a practical and flexible tool for assessing academic results, competence and experience for academic personnel. NOR-CAM will highlight six areas of expertise through systematic documentation and reflection
  4. To develop an ‘automagic CV system’ that enables academics to retrieve data that can be used to document competencies and results in their own career, including applications for positions, promotions and external funding.

Follow-up

Today, I presented the Norwegian report for the EUA workgroup. In many ways, the circle is completed. After all, the inspiration for the Norwegian report came directly from the work of EUA. Hopefully, the report can inspire others in Europe (and beyond) to think anew about career assessment.

Even though it took nearly two years, writing a report is only the beginning. Now it is time to work on how NOR-CAM can be implemented. I am looking forward to contributing to making it become a reality.

Read the full report here:

MusicTestLab as a Testbed of Open Research

Many people talk about “opening” the research process these days. Due to initiatives like Plan S, much has happened when it comes to Open Access to research publications. There are also things happening when it comes to sharing data openly (or at least FAIR). Unfortunately, there is currently more talking about Open Research than doing. At RITMO, we are actively exploring different strategies for opening our research. The most extreme case is that of MusicLab. In this blog post, I will reflect on yesterday’s MusicTestLab – Slow TV.

About MusicLab

MusicLab is an innovation project by RITMO and the University Library. The aim is to explore new methods for conducting research, research communication and education. The project is organized around events: a concert in a public venue, which is also the object of study. The events also contain an edutainment element through panel discussions with world-leading researchers and artists, as well as “data jockeying” in the form of live data analysis of recorded data.

We have carried out 5 full MusicLab events so far and a couple of in-between cases. Now we are preparing for a huge event in Copenhagen with the Danish String Quartet. The concert has already been postponed once due to corona, but we hope to make it happen in May next year.

The wildest data collection ever

As part of the preparation for MusicLab Copenhagen, we decided to run a MusicTestLab to see if it is at all possible to carry out the type of data collection that we would like to do. Usually, we work in the fourMs Lab, a custom-built facility with state-of-the-art equipment. This is great for many things, but the goal of MusicLab is to do data collection in the “wild”, which would typically mean a concert venue.

For MusicTestLab, we decided to run the event on the stage in the foyer of the Science Library at UiO, which is a real-world venue that gives us plenty of challenges to work with. We decided to bring a full “package” of equipment, including:

  • infrared motion capture (Optitrack)
  • eye trackers (Pupil Labs)
  • physiological sensors (EMG from Delsys)
  • audio (binaural and ambisonics)
  • video (180° GoPros and 360° Garmin)

We are used to working with all of these systems separately in the lab, but it is more challenging when combining them in an out-of-lab setting, and with time pressure on setting everything up in a fairly short amount of time.

Musicians on stage with many different types of sensors on, with RITMO researchers running the data collection and a team from LINK filming.

Streaming live – Slow TV

In addition to actually doing the data collection in a public venue, where people passing by can see what is going on, we decided to also stream the entire setup online. This may seem strange, but we have found that many people are actually interested in what we are doing. Many people also ask about how we do things, and this was a good opportunity to show people the behind-the-scenes of a very complex data collection process. The recording of the stream is available online:

To make it a little more watcher-friendly, the stream features live commentary by myself and Solveig Sørbø from the library. We talk about what is going on and make interviews with the researchers and musicians. As can be seen from the stream, it was a quite hectic event, which was further complicated by corona restrictions. We were about an hour late for the first performance, but we managed to complete the whole recording session within the allocated time frame.

The performances

The point of the MusicLab events is to study live music, and this was also the focal point of the MusicTestLab, featuring the very nice, young student-led Borealis String Quartet. They performed two movements of Haydn’s Op. 76, no. 4 «Sunrise» quartet. The first performance can be seen here (with a close-up of the motion capture markers):

The first performance of Haydn’s string quartet Op. 76, no. 4 (movements I and II) by the Borealis String Quartet.

Then after the first performance, the musicians took off the sensors and glasses, had a short break, and then put everything back on again. The point of this was for the researchers to get more experience with putting everything on properly. From a data collection point of view, it is also interesting to see how reliable the data are between different recordings. The second performance can be seen here, now with a projection of the gaze from the violist’s eye-tracking glasses:

The second performance of Haydn’s string quartet Op. 76, no. 4 (movements I and II) by the Borealis String Quartet.

A successful learning experience

The most important conclusion of the day was that it is, indeed, possible to carry out such a large and complex data collection in an out-of-lab setting. It took an hour longer than expected to set everything up, but it also took an hour less to take everything down. This is valuable information for later. We also learned a lot about what types of clamps, brackets, cables, etc., that are needed for such events. Also useful is the experience of calibrating all the equipment in a new and uncontrolled environment. All in all, the experience will help us in making better data collections in the future.

Sharing with the world

Why is it interesting to share all of this with the world? RITMO is a Norwegian Centre of Excellence, which means that we get a substantial amount of funding for doing cutting-edge research. We are also in a unique position to have a very interdisciplinary team of researchers, with broad methodological expertise. With the trust we have received from UiO and our many funding agencies, we, therefore, feel an obligation to share as much as possible of our knowledge and expertise with the world. Of course, we present our findings at the major conferences and publish our final results in leading journals. But we also believe that sharing the way we work can help others.

Sharing our internal research process with the world is also a way of improving our own way of working. Having to explain what you do to others help to sharpen your own thinking. I believe that this will again lead to better research. We cannot run MusicTestLabs every day. Today all the researchers will copy all the files that we recorded yesterday and start on the laborious post-processing of all the material. Then we can start on the analysis, which may eventually lead to a publication in a year (or two or three) from now. If we do end up with a publication (or more) based on this material, everyone will be able to see how it was collected and be able to follow the data processing through all its chains. That is our approach to doing research that is verifiable by our peers. And, if it turns out that we messed something up, and that the data cannot be used for anything, we have still learned a lot through the process. In fact, we even have a recording of the whole data collection process so that we can go back and see what happened.

Other researchers need to come up with their approaches to opening their research. MusicLab is our testbed. As can be seen from the video, it is hectic. Most importantly, though, is that it is fun!

RITMO researchers transporting equipment to MusicTestLab in the beautiful October weather.

Why is open research better research?

I am presenting at the Norwegian Forskerutdanningskonferansen on Monday, which is a venue for people involved in research education. I have been challenged to talk about why open research is better research. In the spirit of openness, this blog post is an attempt to shape my argument. It can be read as an open notebook for what I am going to say.

Open Research vs Open Science

My first point in any talk about open research is to explain why I think “open research” is better than “open science”. Please take a look at a previous blog post for details. The short story is that “open research” feels more inclusive for people from the arts and humanities, who may not identify as “scientists”.

Why not?

I find it strange that in 2020 it is necessary to explain why we believe open research is a good idea. Instead, I would rather suggest that others explain why they do not support the principles of open research. Or, put differently, “why is closed research better research”?

One of the main points of doing research is to learn more and expand our shared knowledge about the world. This is not possible if we do not share the very same knowledge. Sharing has also been a core principle of research/science for centuries. After all, publications are a way of sharing.

The problem is that a lot of today’s publishing is a relic from a post-digital era, and does not take into account all the possibilities afforded by new technologies. The idea of “Science 2.0” is to utilize the potentials of web-based tools in research. Furthermore, this does not only relate to the final publications. A complete open research paradigm involves openness at all levels.

What is Open Research?

There are many definitions of open research, and I will not attempt to come up with the ultimate purpose here. Instead, I will point to (some) of the building blocks in an open research paradigm:

One can always argue about the naming of these, and what they include. The most important is to show that all parts of the research process could, in fact, be open.

How does open research help making better research?

To answer the original question, let me try to come up with one statement for each of the blocks mentioned in the figure above:

  • Open Applications: Funding applications are mainly closed today. But why couldn’t all applications be made publicly available? These would lead to better and more transparent processes, and the applications themselves could be seen as something others can build on. For people to avoid stealing ideas, such public applications would, of course, need to have tracking of applicant IDs, version-controlled IDs on the text, and universal time codes. That way, nobody would be able to claim that they came up with the idea first. One example is how DIKU decided to make all applications and assessments for the call for Centres of Excellence in Education open.
  • Open Assessment: If also, the assessment of research applications were open, this would increase the transparency of who gets funding, and why. The feedback from reviewers would also be openly available for everyone to see and learn how to develop better applications in the future.
  • Open Notebooks: Jumping to when the actual research starts, one could also argue for opening up the entire research process itself. This could involve the use of open notebooks explaining how the research develops. It would also be a way of tracking the steps taken to conduct the research, for example, getting ethics permissions. This could be done on web pages, blogs, or with more computational tools like Jupyter Notebook.
  • Open Methods: During review processes of publications, one of the trickiest parts is to understand how the research was conducted. Then it is crucial that the methods are described clearly and openly. Solutions like the Open Science Framework try to make a complete solution for making material available.
  • Open Source: An increasing amount of methods are computer-based. Sharing the source code of developed software is one approach to opening the methods used in research. It is also of great value for other researchers to build on. Some of the most popular platforms are GitHub and Gitlab.
  • Citizen Science: This is a big topic, but here I would say that it could be a way of opening for contributions by non-researchers in the process. This could be anything from participating in research designs to help with collecting data.
  • Open Data: Sharing the data is necessary so that reviewers can do their job in assessing whether the research results are sane. It is quite remarkable that most papers are still accepted without the reviewers having had access to the data and analysis methods that were used to reach the conclusions. Of course, open data are also of value for other researchers that re-analyze or perform new analysis on the data. In my experience, data are under-analyzed in general. There are numerous platforms available, both commercial (Figshare) and non-profit (Zenodo).
  • Open Manuscripts: Many researchers have been sharing manuscripts with colleagues and getting feedback before submission. With today’s tools, it is possible to do this at scale, asking for feedback on the material even at the stage of manuscripts. There are numerous new tools here, including Authorea and PubPub.
  • Open Peer Review: A traditional review process consists of feedback from 2-3 peers. With an open peer review system, many more peers (and others) could comment on the manuscript, thereby also improving the final quality of the paper. One interesting system here is OpenReview.
  • Open Access: Free access to the final publication is what most people have focused on. This is one crucial building block in the ecosystem, and much (positive) has happened in the last few years. However, we are still far from having universal open access. This is a significant bottleneck in the sharing of new knowledge. Fortunately, the political pressure from cOAlition S and others help in making a change.
  • Open Educational Resources: Academic publications are not for everyone to digest. Therefore, it is also imperative that we create material that can be used by students. This is particularly important to support people’s life-long learning. The popularity of MOOCs on platforms such as EdX, FutureLearn, and Coursera, has shown that there is a large market for this. Many of these are closed, however, which prevents full distribution.
  • Open Citations: Whether you work is cited by peers or not is often critical for many people’s careers. It has become a big business to create citation counts and various types of indexes (the h-index being the most common). The chase for citations has several opposing sides, including self-citations, and dubious pushing for citations to reviewers’ material. Therefore, we need to push for more openness, also when it comes to citations and citation counts.
  • Open Scientific Social Networks: The way people connect is vital in the world of research (as elsewhere). Opening the networks is crucial, particularly for minority researchers, to get access. Diversity will generally always lead to better and more balanced results.
  • Open Assessment: The last block takes us back to the first one. This relates to the assessment of research and researchers and is a topic I have written about before. I also helped organize the 2020 EUA Workshop on Academic Career Assessment in the Transition to Open Science, which has a lot of excellent material online.

Conclusion

As my quick run-through of the different parts of the building blocks has shown, it is possible to open the entire research process. Much experimentation is happening these days, and convergence is happening for some of the blocks. For example, the sharing of source code and data has come a long way in some communities. Some journals even refuse manuscripts without complete data sets and source code. Other parts have barely started. Open assessment may have come shortest, but things are moving also here.

My main argument for opening all parts of the process is that is “sharpening” the research process. You cannot be sloppy if you know that it will be exposed. I often hear people argue that it takes a lot of time to make everything openly available. That is also my experience. On the other hand, why should research be so fast. It is better to focus on quality than quantity. Open research fosters quality research.

One of the most common objections to opening the research process is that other people will steal your ideas, data, code, and so on. However, if everything is tagged correctly, time-stamped, and given unique IDs, it is not possible to steal anything. Everything will be traceable. And plagiarism algorithms will quickly sort out any problems.

The biggest challenge we are facing is that it is challenging to balance between the “old” and the “new” way of doing research. That is why policymakers and researchers need to work together with funders to help flip the model as quickly as possible.

NIME Publication Ecosystem Workshop

During the NIME conference this year (which as run entirely online due to the coronavirus crisis), I led a workshop called NIME Publication Ecosystem Workshop. In this post, I will explain the background of the workshop, how it was run in an asynchronous+synchronous mode, and reflect on the results.

If you don’t want to read everything below, here is a short introduction video I made to explain the background (shot at my “summer office” up in the Hardangervidda mountain range in Norway):

Background for the workshop

The idea of the NIME Publication Ecosystem Workshop was to continue community discussions started in the successful NIMEHub workshop in Brisbane in 2016 and the Open NIME workshop in Porto Alegre in 2019. Besides, comes discussions about establishing a NIME journal, as well as better solutions to archive various types of NIME-related activities.

The term “publication” should in this context be understood in a broad sense, meaning different types of output of the community, including but not limited to textual productions. This is particularly important at NIME since this community consists of people designing, building, and performing new musical interfaces.

When I gathered a workshop team and proposed the topic back in January, this was mainly coming out of the increasing focus on Open Research. Please note that I use “open research” here, not “open science”, a significant difference that I have written about in a previous blog post. The focus on more openness in research has recently received a lot of political attention through the Plan S initiative, The Declaration on Research Assessment (DORA), EU’s Horizon Europe, funder’s requirements of FAIR data principles, and so on.

Of course, the recent coronavirus crisis has made it even more necessary to develop open research strategies, as well as finding alternative ways of communicating our research. This also includes rethinking the format of conferences. The need to travel less is not something that will go away after the coronavirus crisis calms down, however. Long-term change is necessary to reduce problems with climate change. While such a move may feel limiting to some of us that could travel to international conferences every year, it also opens possibilities for many others to participate. The topic of this year’s NIME conference was “accessibility,” and it turned out that the virtual conference format was, indeed, one that promoted accessibility in many ways. I will write more on that in another blog post.

When it comes to openness in research, this is something the NIME community has embraced since the beginning. The paper proceedings, for example, have been freely available online all the time. Also, the database of the archive has been made available as a collection of BibTeX files. Some people don’t understand why we do this, but opening up also the metadata for the archive makes it much more flexible to integrate with other data sources. It also makes it much easier to research the community’s output.

Despite these efforts, there are also several things about the NIME conference that we have not been able to make openly available, such as hardware designs, code, empirical data, music performances, installations, and so on. This is not that we don’t want to, but it has proven hard to find long-term solutions that are maintainable by a volunteer-driven community. People in the community have different interests and skills, so it is essential to find solutions that are both innovative and user-friendly at the same time. The longevity of chosen solutions is also important since NIME is central to an increasing number of people’s careers. Hence, we need to balance the exploration of new solutions with the need for preservation and stability. 

In addition to finding solutions for the NIME conference itself, the establishment of a NIME journal has been discussed for several years. This discussion has surfaced again during the testing of a new paper template for the conference. But rather than thinking about the conference proceedings and a journal as two separate projects, one could imagine a broader NIME publication ecosystem that could cover everything from draft manuscripts, complete papers, peer-reviewed proceedings papers, and peer-reviewed journal papers. This could be thought of as a more “Science 2.0”-like system in which the entire research process is open from the beginning.

The aims of the workshop were therefore to:

  1. discuss how a broader publication ecosystem built around (but not limited to) the annual conference could work
  2. brainstorm and sketch concrete (technical) solutions to support such an idea
  3. agree on some concrete steps on how to proceed with the development of such ideas the coming year

Workshop format

We had initially planned to have a physical workshop in Birmingham but ended up with an online event. To make it as accessible as possible, we decided to run it using a combination of asynchronous and synchronous delivery. This included the preparation of various types of introductory material by the organizing committee and some participants. All of this material was gathered into a pre-workshop padlet, which was sent to the participants some days before the online workshop.

The synchronous part of the workshop was split over two-hour-long time slots. We ended up doing it like this to allow people from all time zones to participate in at least one of the workshops. Since most of the organizers were located in Europe, and the conference itself was scheduled around UK time, we ended up with one slot in the morning (9-10 UK time) and one in the afternoon (17-18 UK time). The program for each of the slots was the same so that everyone would feel that they participated equally in the event.

Around 30 people showed up for each time slot, with only a few participating in both. Since preparatory material was distributed beforehand, most of the online workshop time consisted of discussions in breakout rooms with 5-6 people in each group. The groups wrote their feedback into separate padlets and also reported back in a short plenary session at the end of the hour-long session.

A post-workshop padlet was created to gather links after the workshop. The topic was also lively discussed in separate threads on the Slack channel that was used during the conference. After the conference, and as a result of the workshop, we have established a forum on nime.org, with a separate ecosystem thread.

All the pre- and post-workshop material from the workshop has been archived in Zenodo.

Conclusions

It is, of course, impossible to conclude such a vast topic after one workshop. But what is clear is that there is an interest in the community to establish a more extensive ecosystem around the NIME conference. The establishment of a forum to continue discussion is one concrete move ahead. So is the knowledge gained from running a very successful online conference this year. This included pre-recorded talks, written Q&A in Slack channels, plenary sessions, and breakout rooms. A lot of this can also be archived and be part of an extended ecosystem. All in all, things are moving in the right direction, and I am very excited to see where we end up!

Podcast on Open Research

I was in Tromsø to hold a keynote lecture at the Munin conference a month ago, and was asked to contribute to a podcast they are running called Open Science Talk. Now it is out, and I am happy to share:

In this episode, we talk about Music Research, and how it is to practice open research within this field. Our guest is Alexander Jensenius, Associate Professor at the Department of Musicology Centre for Interdisciplinary Studies in Rhythm, Time and Motion (IMV) at the University of Oslo. He is also behind MusicLAb, an event-based project where data is collected, during a musical performance, and analyzed on the fly.

Thanks to Erik Lieungh and the rest of the team at the University Library at UIT The Arctic University of Norway. They are doing a great job in developing Open Science tools and strategies!