part 1 / part 2 / part 3 / part 4 / part 5

Reproducible recordings for the Generative music

When MIDI was popular for expressing music on computers, we could learn a lot from others work on how to create SMF. Considering that we call WWW of HTML, CSS and Javascript as the Generative Internet, MIDI was a Generative music technology. MMLs were the sources. While those MML-based composers don't always publish their MMLs, those published MMLs help others understand how they can do it.

That was possible because their creatives are kind of assured to "reproduce" on others. Of course MIDI does not assure that in theory, but in practice in earlier ages there were (fortunately?) not too many MIDI devices, so those MIDI songs created for a specific device could be still played completely if others had the same device. Roland GS family and Yamaha XG family were popular in Japan in 20th century.

This is not true nowadays anymore. Audio plugin equipments are unique to each composers and it is almost impossible to have the same set. Composers share only the output recordings (.flac, .mp3, .wav) and it is not "Generative" anymore.

That's not ideal. I want to get back the Generative era again. Fortunately, there are not a few open source audio plugins. Take a look at the list of plugins at StudioRack for example. Recently I've been watching a series of "Weekly Vital" YouTuber videos (Japanese, but I observe some non-Japanese comments there) and see how people make quality sound using Vital. Or Unfa live where he uses solely free software stack (I only watched the one with sfizz but others would be similar).

Fortunately my MML authoring environment is backed by Tracktion Engine, so it should be totally reproducible in theory. Well, actually there are some roadblocks. In this post I will unblock those wherever possible. Let me reiterate the first post of this series, I have reproducible builds on GitHub Actions. Anyone else achieved similar workload for full audio plugins setup?

augene-ng-production CI builds

Ideally it would be nice if there is an "orchestral Docker image" for this too (that is what Paul at the sfizz team suggested when I talked about the idea at their community), it wouldn't be difficult based on my Actions setup and traditional Makefile script in the repo.

This also kind of explains why I don't extend supported formats to other DAWs' yet (one reason here, another reason for portability over Android). It is impossible without open source music renderer (for example, bitwig/dawproject).

setting up audio plugins and instruments

Producing recordings on the CI servers may not sound like a crazy idea at first if those software components are all open source. All things you have to do is to install them and run Tracktion Edit renderer headlessly, right? Well, actually...

  • You have to scan all the plugins you installed within your renderer application. Plugins are looked up not directly from the local files system but from the scanned plugin list. Retrieving plugin metadata takes long time (as it will first have to LoadLibrary/dlopen the plugin for every plugin), and if it directly loads the plugin without any checks, it is accidentally possible that app loads plugins that are in the "dead man's pedal" denylist.
  • You have to make sure that all the plugins you use do not either depend on external files, or ensure that they are on the identical file paths. For my example case, .sfz files loaded by sfizz had me for a while.
  • Your app has to run headlessly. Actually, plugin scanning in practice requires juce::PluginListComponent and it does not run headlessly. We have some fixes.

This time I only needed sfizz and SimpleReverb as the plugins, but SFZ samples also have to be appropriately set up. I used those sfz samples this time:

They are either organized as git submodules, or downloaded on the build environment.

Relocating external resource directories

I use sfizz extensively nowadays, and plugins like sfizz bring in a problem. They need external resources (for sfizz, .sfz and the linked contents) which plugin state binaries would point to the authors' filesystem paths. I put my .sfz files under my home directory (like /home/atsushi, which of course does not exist on whoever do not name atsushi.

While (re)naming yourself as atsushi is certainly a solution, we would prefer other ways in general. And the latest development version of sfizz provides such a functionality. You can set the sfz directory which are "shared" among all sfizz instances. The path is stored as ~/.config/SFZTools/sfizz/settings.xml on Linux (and likewise on other platforms respectively). If you have the same .sfz files under the identical relative directory location, sfizz would still load it from your path. It makes the plugin relocatable.

When you are using LV2 plugins, things it would be smoother if the plugin supports LV2 state mapPath. Unfortunately I'm using VST3 this time (until JUCE and Tracktion Waveform supports LV2, which is not unrealistic anymore), I have to resort to the sfizz feature. When I tried, it was not in the version that StudioRack supports yet, so I install sfizz from their OBS packages.

JUCE apps on CI, including PluginListComponent (patched) this time

When JUCE6 was announced, it came up with support for headless builds on Linux. Sadly, while it is true to some basic components, it is not for juce_audio_processors module. There is no border between audio processors and the relevant GUI components in this module. Therefore your Tracktion Engine application still needs to run with head. Fortunately it is doable on GitHub Actions if you set up required X11 components and [run app with xvfb-run](https://github.com/atsushieno/augene-ng-production/blob/ 7ed1f87/.github/workflows/actions.yml#L54).

That is still a non-issue. A real issue shows up when you need to "scan all xxx plugins" headlessly.

The functionality comes with complex UI feature requirements that the JUCE team did not strictly adhere to UI/logic separation principles. It needs to show scanning progress, opening some modal UI when some plugins request (they seem to). At this state it seems there is no viable solution to everyone (in particular see this). But for my case, what is really blocking is that it does not proceed to scanning unless I pass certain irrelevant arguments to the component. I don't care if it does not complete headlessly if there are some blocking plugins. I don't use such ones. So I apply a patch to JUCE locally.

Once it's rendered to .wav, it's super easy to convert to mp3 using ffmpeg tools.

Remaining issues

As the sshot of the top of this entry shows, I could successfully complete the workflow on GitHub Actions. Still, it took like 70+ minutes just to render the edit to .wav. It is not desirable and I would like to shorten it, especially now that those runs are failing due to resources runout. Probably I should set up my own build server somewhere...

But overall, it seems possible to produce those music files on a CI server and prove portability and reproduction on ther machines. Some efforts toward "cross-platform" builds would be still required, but it is a good observation on the technical possibility.