Can AI replace humans?

Or, more specifically: can AI replace an artist? That is the question posed in a short documentary that I have contributed to for this year’s Research Days.

We were contacted before summer about trying to create a new song based on the catalogue of the Norwegian artist Ary. The idea was to use machine learning to generate the song. This has turned out to be an exciting project.

I was busy finishing the manuscript for my new book, so I wasn’t much involved in the development part myself. Most of the work has been done by a recent music technology graduate from the University of Oslo, Lars Monstad, gently supervised by my colleague Stefano Fasciani.

The project started with Ary sending us a bunch of her lyrics in text format and dumps from a digital audio workstation. This material was not really machine-readable/listenable, so Lars had to spend a great deal of time manually structuring and annotating it into a symbolic data set that could be used for training. The machine-learning part of the project involved generating lyrics following this approach, using an AI model that can already speak English. The melody was generated using Bachprop, based on a deep recurrent neural network. Then Lars put it all together into a final soundtrack that we played for Ary.

When talking about AI, I always find it important to highlight that humans are important for the final result. Yes, the machine makes something, but not without a lot of human guidance. The song played in the video (and the many other songs we also generated) was “composed” by a computer. However, Lars made many important decisions throughout the project: the initial preparation of the training material, the models used, the methods used, all the settings used, the selection of which lyrics and melodies to choose, and the final putting together of everything.

Last year, I was involved in several discussions about the potential challenges of using AI in music-making. I also wrote a blog post about some of the possibilities. While in theory AI can be used autonomously I believe that the most interesting is the meeting point between humans and machines. Providing artists like Ary with AI-based technologies can lead to exciting new music!

More research should be solid instead of novel

Novelty is often highlighted as the most important criterion for getting research funding. That a manuscript is novel is also a major concern for many conference/journal reviewers. While novelty may be good in some contexts, I find it more important that research is solid.

I started thinking about novelty versus solidity when I read through the (excellent) blog posts about the ISMIR 2021 Reviewing Experience. These blog posts deal with many topics, but the question about novelty caught my attention. Even though the numbers are small, it turned out that the majority of the survey respondents listed novelty as the most important selection criterion for the conference. This is not unique to ISMIR; I think many journals and conferences ask about novelty.

Defining novelty

Given that novelty is a criterion “everyone” considers all the time, few people discuss what it actually means. What does it actually mean that something is novel? Merriam-Webster suggests that it is “something new or unusual.” But what should be new or unusual? The questions? The answers? The methods?

Research is about contributing new knowledge to humankind. After all, it is not really any point in reinventing the wheel. Still, most research is incremental. We all stand on the shoulders of giants. New research questions spring out of the “future work” sections of our colleagues’ articles. Our methods are based on the refinement of disciplinary developments. Even so-called “groundbreaking” projects are incremental in nature if you scrutinize the details. Still, we have an idea that “something unheard of before” is ideal.

Research needs to be solid

My research is creative in both form and content. As such, many people think that my projects are novel in the sense of being new. I also work both multi- and interdisciplinary, which means that I don’t really fit well anywhere. That could also be considered novel in the sense of being unusual. Still, what I am doing is not particularly new or unusual. From my perspective, I am working incrementally—everything I am doing builds on other people’s work. True, I combine theories and methods from different fields. This makes it look novel.

I can illustrate this with a research project I just finished: MICRO. Over the last years, we have studied human music-related micromotion, the smallest actions it is possible to produce and perceive. This is new because no one has studied such motion in a musical context before. It is also unusual because the team comprised researchers from musicology, psychology, human movement science, and computer science.

The MICRO project can be considered novel. However, does that mean that everything we did in the project was novel? Some parts were, I guess. For example, we collected data by running the Norwegian Championship of Standstill annually. This was new and unusual the first time we did it. We even got quite a lot of media interest (it is not so often that music research is featured in the sports news on national TV).

However, collecting data once does not make for outstanding science. Research is about asking questions, finding answers, and verifying those answers. Repeating experiments, making slight modifications to the research design, improving the methods, refining the analyses. That is what solid research is about.

I have researched human music-related micromotion for nearly ten years now. We have some answers, but there are many open questions. Many of these questions are neither new nor unusual any longer. But if we want to understand more about what is actually going on inside our bodies when we experience music, we need to continue researching what is not any longer new and unusual. That is about doing solid research, not novel.

Open Research is better research

I believe that open research is better research. Opening the research process makes researchers think more carefully about what they do and how they document it. This takes (some) more time than working closed. But it also makes it easier for others to understand what has been done. This is important from a peer review perspective. It also facilitates incremental research.

The MICRO project has been an open research flagship project. I began by sharing the funding application openly. Throughout the project, we have continuously described how we have worked. The data has been released in the Oslo Standstill Database, and source code has been shared on GitHub. All of this has taken time “away” from publishing journal articles. However, it is time for researchers to publish fewer articles and focus more on making more data, code, etc., available.

Opening the research process is part of solidifying the research. As researchers, we cannot hide behind a “black box” any longer. Everyone can scrutinize what we have done. In fact, I hope that more people will analyze our data and develop our code. That is part of the incremental nature of science.

Summing up

I am not against novel research. However, I think we have gotten to a point where there is too much focus on novelty. If you are applying for a large research grant, it may make sense to doing something new. But it must be possible to submit a presentation to a conference or a manuscript to a journal based on plain, solid research. That may, in fact, be novel in itself! Hopefully, the transition to open research may actually help to focus more on solidity instead of novelty.

Why universities should care about employee web pages

Earlier this year, I wrote about my 23 tips to improve your web presence. Those tips were meant to encourage academics to care about how their employee web pages look at universities. Such pages look different from university to university. Still, in most places, they contain an image and some standard information on the top, followed by more or less structured information further down. For reference, this is an explanation of how my employee page is built up:

My employee page at UiO contains both standard and customized elements.

Arguments for why universities should care

Academics need to be visible online. If you don’t publish and disseminate your research, it won’t have an impact. So it is in our own interest to have up-to-date personal pages with information about what we do. I would argue that it is also in the interest of universities that their employee’s personal pages are up-to-date and look good. My argument goes like this:

  1. The people are the most important asset of a university
  2. The web is the most important dissemination channel
  3. Hence the employee pages should be the most important part of a university web page

For some reason, this appears to be a radical statement. In my experience, many universities think of the employee pages as a “phonebook.” It is a static page with minimal information about how to get in touch with the employee. There is often a template for the page, including information about educational background. Sometimes there is also information about courses taught and research output. But rarely does it contain much more stuff. There also seems to be little institutional interest in maintaining and improving such web pages.

The employee page is an important research infrastructure

Last year, I wrote in the UiO newspaper about considering the university’s web pages as a research infrastructure. In my experience, the web pages of a university are maintained by a communication department. They have one opinion on the purpose of the web pages: communication with students and the general public. I agree that these are the largest user groups of our web pages. But, in addition, we need to remember that researchers also communicate with other researchers.

Research communication is not the same as researcher communication. The former is dissemination activities targeting the broad public. The latter is based on ongoing intellectual exchange with research colleagues around the world. In my experience, communication departments care mostly about the first category. It is typically the role of university libraries to care about the other. Unfortunately, not many librarians are involved in the making and structuring of web pages. In my opinion, they should be.

Web pages as part of the transition to Open Research

I have previously written about why Open Research is better research and why I prefer Open Research over Open Science. In this context, I would mention that I believe university web pages, particularly employee web pages, are key to making a full transition to Open Research. Yes, we should focus on making publications Open Access and datasets FAIR. That should happen through proper repositories with unique IDs, and so on. However, the ambition of moving towards Science 2.0 goes beyond only opening the research results. We also need to open the various parts of the research process. Then I am thinking about the various building blocks of Open Research, as sketched in this figure:

The building blocks of an Open Research ecosystem.

The various parts in this ecosystem will live in different repositories and be scattered around the web. In my thinking, a person’s employee page is the place to gather all this information. It can serve as the hub of an academic’s activities.

Empowering the academics

It is in the interest of universities to provide their employees with the tools needed to store, share, and link to their research material. Many universities don’t seem to care too much about this. The result is that many employees don’t care either. Those who care will make their own solutions. Many set up private web pages with their own domains. Others use one of the social media sites for academics. These sites have understood how to make it fun to add information. It is a pity that universities don’t do the same.

At UiO, we are fortunate to have the possibility to edit the content of our personal pages. There are still things that could improve our system, and I regularly nag both the IT and communication departments about those issues. Still, I am fortunate to work at a university that empowers its academics with the possibility to update their own information. That is not the case in all universities. Some universities don’t allow employees to modify anything at all. I think that is a bad idea. It is bad for the employees, the university, and the transition to Open Research.

To all university leaders out there: how do you work with your university’s employee pages? To all academics: remember to update your personal page! And if you are not allowed to, ask your leaders to give you the tools and access to do so.

Manuscript in preparation

Ever since I finished my dissertation in 2007, I have thought about writing it up as a book. Parts of the dissertation were translated and extended in the Norwegian-language textbook Musikk og bevegelse (which, by the way, is out of print but freely available as an ebook). That book focused primarily on music-related body motion and was written for the course MUS2006 at the University of Oslo. However, my action-sound theory was only partially mentioned and never properly presented in a book format.

I started on a book manuscript around ten years ago, but it has taken a long time to get it finalized. Family life, a period as Head of Department, and building up RITMO have taken up much of my time over the last decade. Last summer, I managed to complete the first draft of a book manuscript.

I am thrilled to announce that The MIT Press has accepted to publish the book. As an Open Research advocate, I am equally thrilled that the book will be published Open Access. The plan is to submit the final manuscript in August. So over the last month, I have been polishing up the text. What is the book’s content? Well, quite a lot, but here is a short summary:

What is an instrument? How is it used? How do new technologies change the way we perform and perceive music? This is a theoretical music technology book, informed by new research in embodied music cognition. The author argues that there are some fundamental differences between acoustic and electroacoustic instruments. Instruments have traditionally been sound-makers. New electroacoustic instruments are often music-makers. The book explores current and future approaches to music-making by analysing instruments. This is done through four distinctive themesmusicking, embodiment, interaction, and affectionthat all tap into different academic disciplines: music sociology, music psychology, music technology, and music aesthetics. The aim is to combine some influential existing theories from each of these domains with the author’s thinking about the future of musical engagement.

And here is a sneak peek at the table of contents:

Although I think the main structure and content are in place, there will surely be some more changes. The challenge is that as I am reading through and checking citations and references, I come across new exciting things that I want to include. But at some point, I realize that I will have to say that enough is enough…

Some thoughts on non-linear presentation tools

Many people rely on what I will call linear presentation tools when they lecture. This includes software such as LibreOffice Impress, Google Presentation, MS PowerPoint, or Keynote. These tools are great for smooth, timed, linear lectures. I also use them from time to time, but mainly if I know exactly what to say. They are also good when I lecture with others, and we need to develop a presentation together. However, linear presentation tools do not work equally well for general teaching, where spontaneity is required. For example, I often like to take questions during lectures. Answering questions may quickly lead to a different presentation order than what I had originally planned. For that reason, I have explored different non-linear presentation tools.

Document camera as presentation tool

Sometimes, but seldom, I only speak when I teach. I am a person that thinks very visually, so when I want to explain something, I usually prefer to show something as well. I used to be quite happy with using a black- or whiteboard when teaching, but some years ago I invested in a document camera.

Teaching with my document camera in the MCT Portal.

The benefit of teaching with a document camera is that I can show small instruments or electronic parts while teaching. It, of course, also works well to write and draw with pen and paper. In fact, I prefer this to write on a whiteboard.

When we started up the MCT master’s programme, I found that the document camera also worked well for online teaching, and during the pandemic, I have used it for several online presentations. Here is an example of how this looks like from a RITMO presentation about microphones earlier this year.

Such a setup allows me for writing with pen on paper, which leads to a very different delivery than if I am using pre-made slides. It also allows for showing things in front of the camera. The downside to using a document camera is that you need to make all the content on the fly. I usually have a draft of what I want to say, which helps in structuring my thoughts. Sometimes I even pre-make some “slides” that can be shown in front of the camera. But there are also times where I want to pre-make more material. Then I have found that mind-mapping works well.

Mind maps as a presentation tool

I have often found that my drafts for document camera-based lectures were developed as mind maps. That is, multi-dimensional drawings spreading out from a core title or concept. For that reason, I wanted to test whether I could use mind mapping software for presentations.

Over the last couple of years, I have tested various solutions. In the end, I have found Mindomo to fit my needs very well. It is online-based, but they also have a multi-platform app that works well on Ubuntu. It is not the most feature-reach mind mapping software out there, but it has a nice balance of features versus usability. I also like that it has a presentation mode that removes all the editing tools. As such, it works very well for mind map-based presentations.

I have primarily used mind map-based presentations for teaching and internal seminars, but some weeks ago I decided to test it for a research presentation. I was asked to present at the EnTimeMent workshop run by Qualisys, but as I was preparing the presentation I didn’t know exactly who the audience would be and the format of the workshop. Then it is difficult to plan for a linear presentation. Since I had lots of video material to show, this wasn’t an ideal time to use the document camera, either. So I decided to test out a mind map-based presentation.

Below is an embed of the presentation I made:

And here are screenshots showing the fully collapsed and fully open versions of the mind map.

I had planned a structure of how I would run the presentation, moving clockwise through the material. I kept with that plan, more or less. What was nice was that I could adjust how many levels I should dig into the material. After listening to some of the speakers before me, I decided to skip certain parts. This was easy because I could leave out opening some of the sublevels of the presentation.

Here is a recording of the presentation:

I had some issues with the network connection in the beginning (yes, presenting over wifi is not a good idea, but it is sometimes unavoidably), so apologies for the poor audio/video in some parts of the presentation.

I still have to get more familiar with moving around in such presentations, but all in all, I am happy about the flexibility of such a presentation tool. It allows for developing a fairly large pool of material that it is possible to draw on when presenting. Rather than deleting/hiding slides in a linear presentation, a mind map-based presentation can easily be adjusted by not opening various parts.