- June 23, 2021: added sub.live and faders.io
- August 12, 2021: sub.live now available for MacOS and Windows
I’ve been getting a lot of questions about collaborative online music in the age of Covid-19. There are lots of performance and DJ streams on Twitch and Youtube. What are the options for collaborating on making music. What is the lowest-latency way to play music together over the internet? Can my choir rehearse over zoom? How can I collaborate on recordings with other musicians while we are self-isolating? What follows is a quick brain dump that I hope will be helpful.
NOTE: The tweet-sized description of my PhD work at the MIT Media lab is: “Understanding the long-term potential of the internet, AI, and streaming technologies for music through experimentation” and I have way more to say about this than almost anyone wants to hear. I’ll try to keep this pragmatic.
The Short Answer
No one wants to hear the short answer, but: Real-time (synchronous) internet-enabled music collaboration will never seriously compete with playing music with other people in the same room. Video conferencing options like Zoom and Google Hangouts have a combination of latency and quality issues, and will be more frustrating than rewarding. When making music together online, I recommend non-real-time (asynchronous or semi-synchronous) approaches.
Below I list a variety of collaborative technologies for creating music on the internet. I’ve written about this in the past, but FWIW, the technologies outlined below are primitive, and are (in their own different ways) attempting to squeeze historical ideas and formats into the online medium. We haven’t yet figured a way to create music collaboratively on the internet where the advantages outweigh the disadvantages.
However because many of us are self isolating right now, playing music in the same room just isn’t possible…. So now is a good time to look at some of the current options for online music collaboration.
Realtime Internet Enabled Collaboration
If you really want to shoehorn the experience of making music together in real-time into the internet, I recommend trying one of the following native applications which are designed to minimize latency.
- JackTrip (Mac, Linux), (Windows). Open source, free software for creating reasonably low-latency audio connections over the internet. It also scales to multiple concurrent users, as bandwidth allows.
- SoundJack I haven’t used this, but it looks slightly more user friendly than JackTrip.
- sub.live prioritizes low latency audio, but also includes video support. As of August 2021 it is available for Mac and Windows.
I would go with one of the options above if everyone in your ensemble knows what an IP address is, and is ready to set up port-forwarding if needed. SoundJack has some nice tutorials and documentation for network configuration. JackTrip requires that one user run a server, which will be easier if that user is familiar with a command line, is familiar with Jack Audio, and has some knowledge of computer networking. JackTrip comes from Stanford’s Center for Computer Research in Music and Acoustics, which published a write-up of one of their early experiments, containing some useful references.
Note that all real-time options will have latency. Your mileage will vary. If your ensemble is small, everyone is in the same geographic region, and everyone has a fast internet connection you might just get usable results.
A quick note about latency: Assume our latency threshold is 20 milliseconds. Anything higher than this and your ability to play together begins to suffer. Light travels approximately 3728 miles in 20 milliseconds (in a vacuum). Absolute best case scenario for round-trip audio is 1864 miles or NYC to Wyoming (but not across the Atlantic). Practical results will rarely be better than half that, and typically be much, much worse, due to network congestion, bandwidth limitations, IP throttling, the fact that light travels slower in the optical cables and internet hardware that it does in a vacuum, and other factors.
Experiments show that musicians can perform together acceptably with latency as high as 50 milliseconds. However, even in the unusual situation when you have a low-ish latency connection to your fellow musicians, there are other difficulties associated with playing music together over the internet in real-time. Think of how difficult it is just to get a Zoom meeting in the office going, with projectors, calls, constantly misbehaving in creative new ways. Now imagine that every participant brings with them the complexity of a small home studio. Suddenly trying to coordinate a performance becomes much harder.
Almost realtime (with optional Zoom video share)
If your use-case can tolerate approximately 100 milliseconds of latency, and you want lossless audio, the subscription service by Audio Movers might be what you need. Their service uses an audio workstation plugin, and can stream to a sharable browser URL. The video below shows how to synchronize a web stream to a Zoom call for Audio+Video.
This might be useful in professional situations as discussed in the video. However it will probably not be suitable for realtime jamming or distributed performance.
Non-Internet Network Audio
If you are lucky enough to be on the same physical network as your musical collaborators then JackTrip might work well. However, depending on your network infrastructure, you can probably get even better results with professional audio-over-ethernet options such as AVB (Supported by modern Motu interfaces like the UltraLightAVB) or Dante (supported by modern Focusrite audio interfaces like the REDNET2XP). These protocols can deliver high channel counts at sub-millisecond latency over a local network. For Dante or AVB, ensure compatibility with your network hardware before buying audio interfaces.
“Google Docs for Audio”
A number of startups are developing tools that add cloud enabled collaboration to the form factor of conventional digital audio workstations. These add real-time remote collaboration to a digital audio workstation, meaning that users record and edit audio at the same time similar to how users can edit text collaboratively in Google Docs. Note that the real-time interaction is for composing, editing, or recording music – these are not helpful if you want to play music together in real-time.
- Ohm Studio
- Amped Studio
- faders.io - This is a brand new option which is as of Summer 2021. In beta
All of these are less mature and less capable than native digital audio workstations. Soundtrap was purchased by Spotify in 2017, and so it has a little more support than some of the others. It is also browser based, which means its sessions are easy to share, but limits its capabilities in other ways. These tend to be aimed at podcast creators, and beat makers, but you can use them for other kinds of music as well.
“Dropbox for Digital Audio Workstations”
If you are accustomed to working with a conventional digital audio workstation like ProTools, Ableton Live, or Reaper, one easy option is to save your session files along with the accompanying audio files in a Dropbox folder, and then pass them off to your band mates so that they can record over them. Unlike the options above, only one user will be able to record or edit at a time.
Dropbox works okay for this purpose, but I would also recommend taking a look at these variants, which are specially designed for working with digital audio sessions. These include extra features like dedicated audio session version control and track preview. These can be helpful for sharing audio sessions between users.
These will try to sell you sample packs and drum loops. If you want to make music with samples or loops, this might even be useful.
These come and go so quickly it is hard to keep up! Here are just a few.
There are a million different mobile apps for creating videos. Want to make a video like the one that recently went viral by the Chino Valley Unified School District in California? Google video collage, and there are numerous apps that will help.
Acapella is one iOS app that cam make video collages, and includes built-in support for remote collaboration. Here’s a self-isolation themed music video made with Acapella.
TikTok is often used with video editor apps (like Acapella), but it also includes features for asynchronous collaboration called “duets” and “reactions” which are pretty self explanatory. If you know of a nice example of a musical TikTok Duet, please send me a link, and I’ll embed it.
Smule’s eponymous mobile app is specifically made to create karaoke style videos collaboratively. It includes a RockBand style vocal pitch display, supports remote collaborations, and can create video collages.
Semi Real-Time Loop Based Jamming
This final category of apps is for jamming with people in almost-real time. Instead of trying to minimize latency, these increase latency to a musical duration in measures. They work kind of like playing with a looper, except that instead of layering over yourself, you are layering over what other musicians played a few bars ago.
- Ninjam is free and open source. It is designed to be used with the commercial, but very affordable DAW, Reaper
- Jammr is a commercial version of the same concept. Might be slightly easier to setup than Ninjam, but for most purposes, I recommend Ninjam
- Endless is a mobile app that supports semi real-time beat making, and can also sync with a MacOS desktop app.
If you are familiar with (or interested in) algorithmic music or code-based composition, there are also tools for “jamming” semi real-time with code. The TidalCycles software package supports several different collaborative paradigms including shared editors for a google-docs like experience capable of generating MIDI messages with code (thanks to Manaswi Mishra for the tip).
Latency Tolerant Music
Music and compositions that do not depend on precise shared rhythm can be tolerant to latent or distributed interaction. Playing this kind of music is one way to circumvent some (but not all) of the obstacles introduced by video conferencing. I participated in Pauline Oliveros’ “World Wide Tuning Meditation” via zoom, and while I wasn’t expecting to enjoy it, I was surprised. A review in the New York Times explains how the piece works.
For technology centric people (such as myself) it’s worth recognizing that what makes this kind of music work is not the technology: it is the community leadership of the organizers.
There’s lots of historical examples, and a million apps I left out. If you think of an important category or example that I missed, please feel free to reach out and I’ll add it to the list. If you don’t know me personally, try @CharlesHolbrow (or email, etc.)
COVID-19 has and will continue to be challenging for musicians. Performing musicians (who don’t typically have extra money laying around) suddenly had their tours cancelled and their income eliminated. Venues are closing. If you are lucky enough to be self-isolating with loved ones or family members, I recommend making music with them. Also please reach out to friends and family who may be more isolated.