This semester, I have written a book with AI. I should emphasize the with in the previous sentence, because this has been an experience of co-creation between various large language models (LLMs) and me. This post details my approach to co-writing Sensing Sound and Music and reflects on the process.

The need for a book

The reason for my AI-based writing experiment was the need for a textbook for the course MUS2640 – Sensing Sound and Music at the University of Oslo. This is an introductory course for the bachelor’s students in our musicology program who want to major in music psychology and/or music technology. These are two distinct directions that are usually taught separately. However, at UiO, we have a strong tradition of combining psychological and technological perspectives on and with music (in the fourMs Lab and at RITMO), so I have argued that we need a foundation course showing the connections between the two disciplines.

I developed and taught the first run of the course last year. It was great fun, and I learned a lot about what the students (don’t) know as they enter university. Most students have some foundation in music history and theory in high school. All of them are also required to play an instrument, and most know how to make sound recordings and do basic music production. However, none have been exposed to anything related to music psychology beyond perhaps watching a TV documentary about how music impacts people’s health. They also have no idea that music technology is a research discipline in its own right; most think of music production as “music technology”.

A complicating factor for a joint course is that most students interested in music psychology are not particularly interested in music technology and vice versa. It is thus an interesting (and challenging) pedagogical task teaching them that you can learn a lot about the psychology of music through using new technology, and that a basic understanding of psychology will help you a lot in the future use (and development, for some) of music technologies.

In the end-of-semester evaluation last year, students complained that they wanted a textbook covering the material. The curriculum in the first run was based on a scattered collection of book chapters and journal articles. This material covered the content I wanted to convey to the students, but since it came from multiple sources, it varied in language, content depth, and presentation style. The biggest challenge, however, was the lack of integration of music psychology and music technology, which is a core motivation for the course. This led me to start thinking about writing a textbook for the course on my own.

It is easier to think about writing a book than doing it, and I already had another book-writing project lined up. Writing a book takes time and focus, and for me, the summer months are the only time I have the peace of mind to undertake such a project. I have been working for some time on completing the manuscript for Still Standing, which summarizes my now more than decade-long exploration of human micromotion. This was a long-overdue project that had to be prioritized, so I pushed aside the idea of writing a textbook.

However, as we were approaching the deadline for submitting the course curriculum before summer, I was reading about people using large language models to write entire books. This planted a seed, would I be able to write a textbook on the fly, while teaching, with the help of AI?

Exploring different AI tools

While “AI tools” may be new to some, I have been developing and using both commercial and experimental rule-based and learning-based AI tools since the early 2000s. I always find it interesting to test the latest state-of-the-art systems and see what they can be used for and what their limitations are.

Since systems change quickly, I have been documenting my use on this blog many times over the years. My current use of commercial text-based systems hasn’t changed much since I wrote this blog post half a year ago. I still primarily rely on CoPilot for coding and writing tasks, while using NotebookLM to get an overview of (large) documents. I use the UiO versions of both tools, which ensures that I stay within the GDPR and that my data is not used for training (at least they say so).

I prefer CoPilot over ChatGPT because it runs in Visual Studio Code, my current go-to text editor. I have been a long-time user of Sublime Text; however, the integration of CoPilot in VS Code works very well and saves me a lot of time since I can integrate LLMs directly in the editor. I should mention that all my personal writing is done using Markdown code in plaint text files, which saves both time and (disk) space while giving me powerful tools for searching locally.

Choosing the AI tool is only one thing, though; choosing the model is another. I haven’t kept track of all the models I have used over the semester. In the beginning, I typically worked with the latest versions of GPT-4, but I have also tested various Claude Sonnet versions and, more recently, various GPT-5 flavours. At some point during the semester, VS Code changed to a CoPilot “auto” mode, which chooses a model itself (I have no idea how it selects a model, though, something to explore later). This works fine, but sometimes I get stuck and have to change to a different model manually. It would have been interesting to keep track of that, and Microsoft probably has for me, but it is not very important for my current argument.

Writing in Jupyter Book

I chose to write the book using Jupyter Book 2, which is based on the Myst engine. I have been interested in non-linear, web-based writing and reading solutions for a long time and have been following the Rbook project. However, I have never been an R user; I rely instead on Python and Jupyter Notebooks. When I saw that Jupyter Book 2 was in beta last summer, I decided to give it a try.

The most remarkable thing about Jupyter Book is that it integrates seamlessly with Markdown and Jupyter notebooks. This allows me to embed executable code examples directly alongside explanatory text. It is also developed as an academic tool that supports citations and cross-references. Finally, it also allows for creating both a web page version (through HTML) and a PDF (through LaTeX). Also, building on Myst, you get a nice layout, search functionality, day/night mode, etc., features that students would appreciate.

The downside was that Jupyter Book 2 was still in beta when I started. Fortunately, CoPilot helped a lot in resolving all the dependency issues and setting up GitHub Pages with automatic deployment. This I would never have figured out myself within the allocated time.

Writing with AI

When I have told people that I have written a book “with AI”, they look at me with big eyes. Really??? An academic textbook written for bachelor’s students? Is that allowed?

Some academics warn against using AI for writing and teaching, arguing that we lose control of our thinking and writing. I see that argument, but my counter-argument would be that it is even more problematic to close our eyes to reality. AI tools are here, and our colleagues, students, and everyone else use LLMs daily. As professors, our job is to understand and explain the world (or at least the part of the world we are specialists in) and prepare young people for a future with AI. Then we need to use it ourselves and understand how it works. Co-writing a book on a topic that I know well has helped me understand in more detail how various LLMs work, what they are suitable for, and when they fail.

Developing the book structure

Fortunately, I didn’t have to start the book from scratch. I had already developed a course structure that worked well last fall (2024), so I decided to continue with it, making only minor modifications. I did actually ask CoPilot and ChatGPT about suggestions for how to structure the course based on the course description, but they ended up with a very traditional, psychologically oriented structure that didn’t include embodiment, multimodality, and the various technological components I wanted. So I scrapped those suggestions and stayed with my own plan.

Last year, I taught with mind maps instead of slides. These mind maps don’t contain much information, but they have all the headings and keywords I used in class. This was a good starting point, so I used the text versions of the mind map PDFs as the basis for each week’s chapter.

The aim for the semester was to stay two weeks in front of the students with my writing. In general, most of them only read the material for the coming week, so nobody complained about that. Of course, I told them about my plan, so they knew about how it would roll from the beginning of the semester.

The Co-writing process

Each week, I began the writing process by copying and pasting the keywords from last year’s mind maps into a new Markdown document in Visual Studio Code. Then I asked CoPilot to expand the content for me. This usually worked well; it extrapolated the content with a few lines for each keyword. Sometimes it confused the terms, which I then deleted or modified, but most often it was spot on. I used this content to create the week’s structure, choosing main headings and subheadings. Other times, I asked CoPilot for help finding “holes” in the content, and it did so successfully.

After I had a rough outline of the whole week in place, I would ask CoPilot to write out entire paragraphs for each section. At times, this worked very well. For example, in the etymology section, CoPilot wrote nice descriptions of musicology, psychology, and technology. Other times, it struggled with writing what I wanted to include. For example, it did not manage to write a meaningful section on interdisciplinarity. There, I asked NotebookLM to summarize some of my previous writing on the topic, which I then re-wrote to be more in line with my own argument.

I also had to intervene in the section on embodied music cognition, where it failed to include much recent research. However, when I specified the key researchers and concepts I wanted to include, it wrote several nice paragraphs.

I eventually learned that by explicitly asking it to look up content in Wikipedia, I would get quite good results. Most of the textbook focuses on describing the fundamentals of acoustics, psychoacoustics, and other topics that are not controversial and are well covered on Wikipedia. Fortunately, I know the content well, so it was easy for me to adjust the text as I developed the content.

Adding code

One of the best things about the current approach was that I managed to integrate code snippets within the document. This is particularly prominent in the Acoustics chapter (see, for example, the complex waves section). The ability to include code snippets was the primary reason I chose Jupyter Book as the backend for my book writing. With CoPilot, implementing the various examples was a breeze.

Due to time constraints, I didn’t figure out how to include interactive sound examples this time, which will be one of the core topics for next year’s run of the course.

Lots of editing

As can be seen in the GitHub repository serving the textbook web page, I have made more than 170 commits—big and small—to date. This reflects the co-writing process throughout the semester. Quite often, I have felt like a supervisor instructing a research assistant on what to do.

As a professor, I am used to reading student texts with varying quality. Sometimes, students write excellent texts with a high level of (correct) facts. Other times, they write things they dream up with barely understandable language. In general, CoPilot worked like a good research assistant with excellent writing skills. I knew I had to be skeptical of the content, and I treated it accordingly throughout the process.

Glossy language

One of the benefits of co-writing with an LLM is that it writes well. Sometimes, the result is a bit “glossy”, particularly from a non-native Norwegian point of view. I tended to tone down the language, removing exaggerations and overly confident phrases.

It helped running the text through Grammarly, another AI-based tool I use regularly to improve my writing’s grammar. Grammarly doesn’t invent anything on its own; it only focuses on improving the content as is. It can also be instructed to write academically, which helps in cleaning up the language.

The problem with Grammarly, though, is that it tends to remove all personal twists. Thus, the text becomes easily readable, but without the wit that one sometimes wants to include while writing. So I added back some comments, puns, and jokes that were removed. It helped a little, but the text is quite impersonal. That’s okay, though, for a textbook.

Conclusion

If someone had said that I would be able to write a full textbook in one semester, staying just ahead of the students’ reading assignments each week, and while running RITMO and establishing MishMash, I wouldn’t have thought it possible.

Still, here we are, half a year later, and the book is there. It is far from perfect, but it served my students well, and it is an excellent starting point for further development in the coming years. The nice thing about writing a digital book without a publisher is that it can be updated on the fly.

I fulfilled the most important task with the textbook: integrating some core material on music psychology and music technology in an accessible form. This is something I have been thinking about for years, but would have never been able to do without the help of AI.

I have used various AI tools extensively throughout the whole process. Helping with setting up the backend solution (Jupyter Book 2), solving commit errors (GitHub), sanity checking the structure and content (NotebookLM), co-writing all the text (CoPilot), and grammar checking everything (Grammarly).

The various AI tools have been central in making it happen, but I have been heavily involved myself. Fortunately, I know all the material well, so I have been able to remove errors and steer the project in the right direction. It has truly been a co-writing process. AI didn’t write the book; it helped me write a book. Without AI, I would never have started this project, and without its help, I would never have finished it. However, I spent much more time writing than I initially anticipated. After all, I am the one responsible for the content.

Note on the use of AI: This blog post was written by me, and with grammar checking through Grammarly.