music to your ears

UAPMD v0.4 x AAP v0.10

Tue, 28 Apr 2026 14:45:00 +0900

Almost 4 months ago I wrote about my latest development, UAPMD, which WAS about realizing virtual MIDI 2.0 devices on desktop using arbitrary audio plugins in VST3/AU/LV2/CLAP formats. Many changes happened after that, until now. Yesterday I released the latest UAPMD, which is version 0.4. This is the latest screenshot.

Is it looking as complicated as v0.1?

But note that the screenshot was mostly filled with various plugins. They were just to show off the variety of supported formats. The v0.4 sshot above only contains one plugin, and it is taken on Android. What's going on there?

I will explain the progress I had been making these months step by step.

UAPMD v0.2: from audio plugin host to multi-track sequencer

The initial rationale behind UAPMD - well, it still is kind of - is to achieve MIDI 2.0 sequencer i.e. standardized ways to play music in the modern and flexible set of instruments, not just traditional hardware MIDI module lunch boxes.

But that means, we need way more complicated software. When I released v0.1 milestone and wrote the blog post last time, I was like:

Support for decent multi-track engine is one of the post-0.1 milestones, and that would come up with simple-ish imaginary MIDI 2.0 Container file player (imaginary as the spec. document is still not published yet) which would look more like a DAW.

Therefore, UAPMD went that path. It became more like a multi-track sequencer, as this v0.2 sshot tells:

At this state, it started looking much more like a DAW. It was still MIDI oriented sequencer, but the clips can be MIDI2 or audio. It featured multi-track MIDI 2.0 clip player. The MIDI 2.0 events are actually translated to plugin notes, parameter changes, and direct MIDI messages, mostly down-converted to MIDI 1.0. Each SMF2 Clip is saved and loaded as a file, and then bundled within a zip project archive (.uapmdz file).

In UAPMD, an audio track is like an "instrument" - we can apply audio plugins and they can be controlled by those MIDI2 clips. There is actually no concept of "track type" - both audio clips and MIDI clips are just timed events.

The project archive is a set of project files, that looked (well, it still is for now) like what SMF2 Container Format is supposed to be (the link may die sooner or later, The MIDI Association has no interest in preserving their blog post URLs). As of now, SMF2 Container is not going to be our project file format anymore though.

There was of course a lot more of bugfixes and quality improvements. As of v0.1, the plugins I could retrieve MIDI-CI properties was quite limited. They tend to stall in the middle. There is no such a problem, nowaday. AUv2 and AUv3 have their hsoting implementation respectively to get full features to not miss whatever AUv2 bridge lacks support. Wayland detection and support was awkward in v0.1 (it may still not work, as I haven't set up Wayland-only environment). UMP Function Blocks have more solid mappings.

There were handful of additional features. We started supporting QuickJS scripting.

In the meantime, UAPMD v0.2 had been polishing the UMP device mapping support (which still continued until 0.3). If you remembered the previous blog post, it also mentioned "midicci" MIDI 2.0 keyboard app. Now it looks like this:

UAPMD v0.3: ubiquitous sequencer (Android, WebAssembly, iOS, and Windows)

UAPMD v0.2 achieved another important milestone; I got a nice music project file format for a DAW-alike sequencer engine, all based on open technology. That is what I kind of wanted to have for my Android audio plugin ecosystem. Thus, support for Android slipped into my post-v0.2 milestones.

UAPMD on Android

As Audio Plugins For Android is one of my primary projects, Android should be a first citizen target.

Supporting Android as a target was not hard - API wise. UAPMD API is designed to be common abstraction with format-specific extensions and platform-specific extensions. The biggest hurdle is ImGui-based UI - I made it work, based on SDL3 on Android, which I think is fairly straightforward. SDL3 on Android however does not work like other native platform API - especially its "main" thread is NOT the platform main thread. It is the same kind of problem like Flutter < 3.29 had before. But so far, it is working. When I deal with AAP, it is still crashy compared to the desktop version - I need to figure out what is causing them.

The greatest thing about this is that, now we have most solid AAP hosting application ever. Formerly, we had AAP hosts as:

AAPHostSample: it is a PoC host app written in Compose. It does not involve complicated process flow like a valid DAW has e.g. parameter refresh, state loading and saving, bus connectors, etc.
aap-juce-plugin-host(-cmake): JUCE audio plugin host. Since JUCE itself does not support plugins on Android, everything is not quite stable.
aap-juce-simple-host: it was created to isolate various issues I experienced at aap-juce-plugin-host. It has both problems that JUCE has and AAPHostSample has (not a real-world use).
aap-juce-helio: it was the only JUCE-based DAW that I could bring to Android. It was already great I coudl actually do that. But I need in-depth debugging to resolve problems as I'm not sure if they are caused by juce_audio_plugin_client or the DAW itself.

So UAPMD is going to be a savior for me. Having said that, the JUCE API is still going to be useful as it provides the common way to deal with plugin APIs, and its behavior should be still informative. However, it is also crashy so that we have to patch around various issues. One big reason I dumped the idea of investigating it was that every time I close the plugin UI the container window caused crashes, and it was not limited to the plugin UI window. Making changes to all those plugin UI code is beyond one human work.

UAPMD on iOS

After I dealt with Android, I thought that it would also be possible to run it on iOS, although I don't have any iOS device. But since we already have first-class AUv3 support, it should be possible in a bit - yes, it was:

UAPMD on Windows

As time went by, it was becoming more likely that Microsoft was finally going to release the stable version of Windows MIDI Services. Supportting Windows was going to be a big win to the MIDI 2.0 ecosystem, which was kind of my interest as of that time, so enhancing Windows support became an objective in the next step. The MIDI 2.0 support is still stuck in the middle, but I made some effort to stabilize plugin hosting side and I reached the status that there is no known VST3 hosting issue. (We have some CLAP hosting issues, but that's not limited to Windows.)

UAPMD on the Web

After I have successfully got those platforms covered, I became optimistic like "why not support WebCLAP as well?" because I was sure that ImGui works well on the browser (built as WebAssembly), and I don't have hard native platform dependency. I don't see a lot of information on WAM2 and there was no development activity in these months, so my interest is only on WebCLAP so far.

I still don't have a lot of experience on Web Audio, not just WebCLAP, so I ended up letting Codex inspect the existing WebCLAP headers and samples to get them working within UAPMD, with some help by signalsmith (the WebCLAP developer).

There was no synth plugin ported to WebCLAP at the time of building it (and as of when I am writing now), so only effect plugins are tested.

You can try the latest wasm build from here: https://atsushieno.github.io/uapmd/playground/latest/

Support for WebCLAP was not actually straightforward. It requires such an architecture that splits audio processing and anything else into respective process es, not just separate threads. It seemed like audio engine separation that I wanted to implement at some stage for isolating the entire plugin hosting process, but that wasn't. Separate audio engine only requires data sharing across processes, but separating the entire plugin host requires a lot more IPC work (still not implemented).

I also had to make a change in our State API. CLAP requires the state API functions invoked on the main thread, but UAPMD cannot block the wasm main thread. To reconcile the situation, I had to make changes to our plugin client API (not the plugin API abstraction layer itself) to become asynchronous. The WebCLAP official sample host does not provide state features yet, so it is all my own experimental stuff, but it works for now.

Another related improvements is that the audio graph API is now part of the realtime-safe part of the entire system. Until this version, UAPMD only provided simple linear connection of nodes so there was not much pressure on RT-safety requirement. But what happens when someone started implementing complicated audio graph - can we really require that? This question led me to the conclusion - I have to prove that anyway. That became part of the next release.

UAPMD v0.4: realistic AAP GUI integration

Up until now, UAPMD is getting more features such as piano roll editor, audio graph, and audio warp editor. However, UAPMD is primarily a plugin host so far, and while I keep bringing in DAW-like features, they exist for providing some way to implement common plugin format features such as latency reporting (and thus compensation).

In the meantime, I had to work on a complicated audio graph. I had to provide some PoC implementation anyway, to really provide latency reporting API and tail samples API in the plugin hosting API abstraction (remidy). Fortunately the theory here is very traditional. We need DAG, and editing the tree has to be RT-safe by the new requirement we just got as mentioned earlier.

Adding support for DAG-based audio graph was done with some API redesigning. What I aim to provide with UAPMD is a flexible API that brings in developers choices. I don't want to force specific graph design and implementation. I have been designing the entire sequencer engine as split in a few chunks and making them independent of each other. This means, the overall audio graph is still independent of multi-track sequencer / editor. as well as plugin instancing API abstraction. To achieve that, I ended up defining the abstraction layer for minimum requirement (like "add this plugin instance to this graph - how to actually connect nodes is implementation dependent though"). It was especially not easy when I was dealing with UMP Function Blocks vs. tracks, but it kind of makes sense by now. If I didn't work on this kind of adjustment, it could quickly bring in leaky abstraction and made API changes harder.

Another notable feature is support for external API endpoints such as HTTP server and Android Provider for embedded scripting and MCP endpoints. The MCP endpoint helps development, not just trackmaking, when telling coding AI agents perform some investigating and verification based on my human language instructions.

All those new features were already enough to tag v0.4, but unlike the existing versions, v0.4 was not tagged for achieving general milestone. It was tagged in along with the new release of AAP, because UAPMD can become the first real-world use case for AAP. While UAPMD it still primarily a plugin host, it can load existing music projects that were created on desktops, then assign AAPs as audio plugins, and play it on the device. And there was a big achievement in AAP land: I could finally get JUCE plugin UI working. This is a copy-pasted capture of the latest androidaudioplugin.org landing page:

I had to make various fixes to get the latest plugin UIs working with UAPMD, but now it's almost good. That's how AAP got released as v0.10.0, and UAPMD v0.4.0.

It's still NOT usable yet though; we still have some issues to get AAP working fine with project saving in UAPMD.

Those plugin UIs are not designed for mobiles, so there are many pieces that are not working great on Android. Having said that, it's still better to have the plugin UIs so that we can pick some presets from their UI. An inconvenient truth about JUCE plugins is that most of those plugins do not expose their presets through the plugin format API (via JUCE API). On such a plugin, we can only pick their presets on the UI.

On many plugins, presets are not designed for platforms like Android where people usually do not have home directory. I sometimes added a first-time-run content extractor on those plugins that enumerates presets from home or user data directories.

Another Android-specific issue (or maybe mobile-specific issue, depending on how AUv3 works) is that many plugins use addtoDesktop() directly or indirectly, which does not work on remote plugin UI controller (Android SurfaceControlViewHost). This includes JUCE PopupMenu kind of API. As soon as the plugin UI attempts to invoke that, the UI goes unresponsive because it renders the component to the global desktop, which does not exist on the audio plugin Service contexts. There are many plugin app patches I added there. These crazy amount of works are not very human friendly. They are labor, not fun part of coding. This is certainly what coding AI agents take place.

But all those efforts are worth. With usable presets, dealing with AAP is a lot more of fun, even though we still cannot save the manipulation results.

There are still some design thoughts that I am not fully decided yet: those resizable plugin UIs don't work well on Android yet. I rather made the host window resizable and made "plugin UI as the content" scrollable, so that those big plugin UI contents can be still shown on the mobiles (I assume mostly tablets as of now). If we simply make resizable window resizable and have content scale with the window, the content will become too small to manipulate. Even worse, we are based on touch inputs on mobiles. That might not fit the actual resizable UI that the plugin developers intend, but I think the current state is the most realistic option.

In the next development cycle, I will have to bring in some ABI incompatibility in AAP and that's going to be a big deal. But once I get all those features working on UAPMD, hopefully I will be able to move forward to the first 1.0 release cycle. There are many parts that I want to "modernize" from the API to the samples, but things now look realistic than ever.

In about a month I will be talking about these efforts in my session at ADC Japan 2026, and I wich I can make these missing bits done by then. Though I will also be busy visiting KotlinConf 2026, so I'm not very optimistic. I believe all these above are already fruitful.

UAPMD: Building MIDI 2.0 Ecosystems on Desktop

Thu, 01 Jan 2026 16:20:00 +0900

Last year (okay, I'm a bit late; read it as 2024, and other years likewise), I wrote a post titled "Building MIDI 2.0 Ecosystems on Android". There I explained that we can build MIDI 2.0 ecosystems on Android on every aspect. There was nothing much happened this year on that platform - I had been focusing on MIDI 2.0 on desktop this year. I named it UAPMD (ubiquitous audio plugin MIDI device). I kinda started that effort last year, but I was busy on many months this year to accomplish something "releasable". But at last, I tagged version 0.1 of the effort within this year.

Here is the screenshot for v0.1 status quo. It covers a lot of features it supports.

You can find the sources and its essential information here: https://github.com/atsushieno/uapmd

What UAPMD offers are:

uapmd-app: a plugin host application that instantiates any audio plugins and expose them as virtual MIDI 2.0 devices
remidy: a plugin hosting foundation for VST3, AudioUnit, LV2, and CLAP (without API stability though)
uapmd: a virtual MIDI 2.0 device bridge layer to the plugin host

The entire codebase is MIT licensed. Those modules I used are under the permissive licenses like MIT, ISC, or zlib. It involves a lot of Claude Code and Codex works in the last few months, from providing packaging setup to implementing a handful of plugin features by feeding the spec. documentations and FLOSS sources to find bug defects and problematic implementations.

I wrote almost everything we need on the repository (README, docs, wiki). What I'm going to write here is the rest that don't fit there.

Rationale

UAPMD achieves whatever anyone else didn't yet.

Last year I wrote:

It works fairly well with MIDI 1.0 protocol. But what we want now is MIDI 2.0 music player. While it is able to send UMPs to MIDI output devices, we don't really have MIDI 2.0 devices. It is probably not a big deal as UMP outputs from those MIDI 2.0 apps will be translated to MIDI 1.0 messages if the connected device is MIDI 1.0. But then it does not make sense to use MIDI 2.0 there. What I want instead is to be able to play "MIDI 2.0 music" over a "MIDI 2.0 device" which has more than 16 channels. Now I'm exploring that kind of fields, but I need more time to achieve something useful.

So, UAPMD filling the missing piece.

Actually I had a handful of virtual MIDI 2.0 devices on Android (blogged last year, explained later), but that happens only on Android, which is not a "creative" platform to me - I need PC. Also, those virtual devices are not what I find applicable here (explained later). What I needed is virtual MIDI 2.0 devices on the desktop.

In the very early stage, I figured that I'm going to need my own plugin hosting foundation. There were only JUCE and JUCE-based projects (even including Carla, which contains old fork of JUCE v4 named as "water"). I need UMP processing at common hosting part and JUCE is only dealing the input events in MIDI 1.0 manner. (They announced their plan to support UMP this year, but it still won't not happen until v9.0 and I don't plan to wait for them.)

I'll explain more details, but UAPMD itself ended up being an extra stack to expose MIDI 2.0 control points for those plugin instances over the entire plugin hosting foundation. So, the actual development effort is spent more on the plugin hosting foundation (apparently).

UAPMD v0.1 is not a solid foundation for multi-track sequencers yet, but it should be at some stage, because hosting one plugin is not enough for my ultimate plan for "MIDI 2.0 music player". It is also why AAP virtual MIDI 2.0 devices are not enough. But when someone implements a decent multi-track sequencer over (lib)uapmd sequencer engine, anyone can build a DAW that features many audio plugin features.

Recap: AAP as MidiUmpDeviceService

When I had been working primarily on Android plugin stuff, I have added support for virtual MIDI 2.0 devices last year. It is to expose arbitrary audio plugins as UMP devices, so I could use plugins including (but not limited to) MDA, Sfizz, Dexed, OPNplug(-AE), Odin2, Vital OSS (when it was open source), airwindows... a lot.

The way how I mapped audio plugin features to MIDI 2.0 was twofolds:

parameter changes are mapped to Assignable Controllers (NRPNs), preset changes are mapped to Program Changes, and other UMPs are sent to the MIDI event port.
for more complicated features such as saving states and controlling GUI, we use our own Universal SysEx8 to transmit plugin API requests and responses in async manner. (More details are blogged here).

This was kind of interoperable for UMP level, but we still had no public way to expose parameter list and program list. They were specified as MIDI-CI Property Exchange especially as AllCtrlList, ProgramList, and even Device States, but no one had realized them (except for ProgramList on KORG KeyStage and their siblings).

midicci: MIDI-CI implementation

Creating virtual MIDI 2.0 devices became possible when ALSA supported MIDI 2.0 on Linux kernel 6.5 (and CoreMIDI has supported that earlier). And thanks to libremidi, creating those virtual devices is made quite easy. Libremidi is so far the only option I know of if we want to achieve that.

Now that we can expose virtual UMP devices, the next step is to have MIDI-CI integrated. Last year I have built fairly advanced MIDI-CI implementation (detailed in two blog posts), but in Kotlin. Kotlin works in general and ktmidi-ci builds as a Kotlin/Native module, but when it comes to providing fully compilable sources with C++ without binary dependencies, it is awkward.

So I decided to port it to C++, now that we could achieve somewhat higher level porting easily using coding AI agents like Claude Code and Codex. It is how midicci was born.

It is an experimental app called midicci-keyboard. It does not only act as UMP keyboard, it also implements controller list and program selector by name. This was not doable with MIDI 1.0, only doable with MIDI 2.0 with proper MIDI-CI support. Note that this MIDI 2.0 client application does not resort to any audio plugin features. It should work with any MIDI 2.0 devices that support AllCtrlList and ProgramList. As I mentioned earlier, there is no MIDI 2.0 devices that support AllCtrlList as far as I know of. You can still try KORG MIDI 2.0 products for ProgramList.

remidy: audio plugin host

The primary audio plugin host component "remidy" is a common publish API designed to be cross-platform, multi-format manner. It started more like a research project to focus on which plugin features can be commonized, which are format specific (e.g. CPU multicore threading is specific to CLAP), etc. The primary content of this uapmd repository was docs directory at that time.

I never thought I was going to implement such a "fully" featured plugin host. The initial development idea was optimistic. If I provide minimum plugin API implementation things would just work for simple MIDI features. That was all wrong. If I do not provide appropriate audio bus buffers or event bus buffers plugins often crash. If I do not implement some interface on VST3 IHostApplication they crash. If I leave some interfaces not implemented on IEditController they don't sound. I only had some experience on LV2 while I was working on Android plugins. I knew almost nothing about VST3 and AudioUnit APIs.

I was also using DPF which contained VST3 compatibility layer called Travesty, and while it worked perfectly, it was C API with manual vtable tricks, which I was sometimes stuck at my misimplementation. Fortunately VST 3.8.0 became available under the MIT license very recently, I could switch to the Steinberg API and directly understand other developers use of the API. Translation from Travesty to VST3SDK was quite straightforward thanks to Claude Code or Codex (I don't remember which). Of course they understood Travesty API fairly well, especially I told them that it is a C compatibility API.

The plugin hosting implementation was actually GUI-less for the first year (I had been working on it only about 3 months though), which was actually almost no working IEditController - technically we could implement this interface and component without GUI, but it'd be easier to have it working. It is almost impossible for most plugins to get the right parameter sets without GUI. You might think that if we choose a preset by some index or id, but that is hardly possible - most plugins do not expose presets in each plugin API's presets API.

It is most likely because there is not good enough ways to provide both factory presets and user presets in consistent manner. For example, how do you deal with preset IDs? Of course, each plugin format offers their way to appropriately to handle that, but having all into a decent common API in plugin SDKs (such as JUCE) would not be easy - especially if they had to cover old APIs such as VST2.

I had a proof-of-concept standalone plugin host called "remidy-plugin-host" (which became part of uapmd-app) using Web UI, and thought that everything could be implemented through URL handler. I quickly gave up the idea as its feature requirements became complicated. I switched to ImGui later which reduced a lot of boilerplate binding work.

uapmd: UMP mapping and MIDI-CI bridge

My initial plan for the virtual MIDI device management layer was, one simple UMP device to target one single instrument plugin, like how Timidity, Fluidsynth, Microsoft GM Synth, etc. worked for MIDI 1.0. This layer was not supposed to be very complicated - once a plugin instance got working, then I could simply set up a virtual MIDI device (as I mentioned earlier, it is quite easy with libremidi) and map the incoming UMPs to note on/off, parameter changes, etc. while detecting MIDI-CI universal SysEx to route to the MIDI-CI device I set up for the plugin.

But when it comes to "MIDI 2.0 music player", it becomes a lot more and it becomes more like a DAW sequencer engine that is supposed to process and merge multiple audio tracks simultaneously. A potential solution is Tracktion Engine here, except that JUCE projects are in general not suited for reusable libraries. Their graph library might be still useful if people don't mind their (A)GPL-ed library. It doesn't perform well on Android which I target either.

Support for decent multi-track engine is one of the post-0.1 milestones, and that would come up with simple-ish imaginary MIDI 2.0 Container file player (imaginary as the spec. document is still not published yet) which would look more like a DAW.

One thing that would still look annoying is that uapmd exposes ALL the instantiated plugins, including effect plugins connected to the instrument plugin, as UMP devices. So there will be hundreds of virtual MIDI devices when they are to construct a song. uapmd-app can disable those UMP devices, but it would be nicer if one track exposes only one UMP device. My initial idea was to behave like that, but I was then mixing all those plugin parameters from those multiple plugins which of course doesn't work. The ideal solution would be to have separate Function Blocks for each plugin, but I need some design changes in midicci and ktmidi-ci (its origin). It is another post-0.1 task.

The UMP transport layer was (or it still is) cause of a lot of problems. Things get unstable on macOS when I fix some issues on Linux, and vice versa. AllCtrlList is one of the largest UMP blobs that I had to deal with, and now State binaries as well (if I remember correctly, KORG MIDI-CI developers were talking about that too when I talked to them).

misc. notes

AI coding agents: it is my first experience to introduce generated code by AI coding agents. The beginning of midicci was an automated translation from Kotlin to C++ by Devin. Then I switched my primary engine to Claude Code (also Codex as well). I kept it AI-only project for a while. While it needed a lot more work, the initial conversion from Compose Multiplatform to Qt was impressive - as you would know, their architectures are totally different, but the conversion was just one pass (apparently, it was actually a lot of stubs on the converted code though).

Then I used the same approach to convert the plugin host app UI from WebUI to ImGui to simplify use of various plugin API features (i.e. in the same C++ code), and that was quite successful. While I know how typical GUI frameworks work (like when I implemented XIM support in mono Windows Forms around 2008), I had almost no experience in Qt or ImGui, and I don't bother to dig in depth.

Apart from GUI, uapmd module was left untouched by the AI engines, but starting migration from Travesty to VST3SDK, I figured that they should perform fairly good work to implement well documented specifications like plugin format APIs. So I tried to create a common plugin UI API based on CLAP, and let them to implement it for each plugin format as well (it was kind of intentional to see how they could figure out the mental model differences as UI extension design is quite different between CLAP, VST3, AU, and LV2). Then expanded the uses of them almost everywhere. It's good that remidy was API first, so that I don't have to worry much about awkward copypasting from somewhere else.

Testing: uapmd lacks comprehensive tests. What I really miss in the audio world is solid testing foundation. It is one of the recurring topics at ADC (audiodevcon) these years. They run in realtime manner and not very reproducible. Even for non-realtime part, audio plugins have to be installed first, then listed and instantiated on the CI servers. I had some experience from a few years ago, but that's very specific to the song project.

I had some plan to get plugins easily installed on GitHub Actions using StudioRack, namely studiorack-cli, but I have to wait for some fixes to get it really install and process audio (currently it does not work).

vNext

There are couple of things I want to improve for v0.2 milestones:

sequencer for multiple tracks and clips (as mentioned above)
non-desktop targets: migration to AUv3 API, Android, maybe Web (WebCLAP) ?
hopefully more integrated Function Blocks

I'm not sure how I prioritize them, but hopefully I have time to work on them.

On 'Android Audio: Beyond Winning on It' at DroidKaigi 2025

Tue, 07 Oct 2025 21:00:00 +0900

It is a compound translation from my Japanese blog post, Google Translate then a bunch of edits by myself. Some contents that do not make sense in English are altered.

At DroidKaigi 2025, I gave a talk "Android Audio: Beyond Winning On It" (in English), where I discussed the latest audio features missing on the platform at the Android audio application development frontline. The session recordings were already published the very next day :

https://2025.droidkaigi.jp/en/timetable/941476/

The session slides are available at speakerdeck:

https://speakerdeck.com/atsushieno/android-audio-beyond-winning-on-it

My discussions there were that, how Android has much less music apps compared to iOS, and how we should address our problems. The title implies that "you can achieve things fairly well on Android" is not enough. However, I wrote too much in the talk proposal. I intended to mention everything I wrote, but I regret that I ended up being too tied down by it. There were also some things I had intended to talk about at the time that I didn't mention at all (e.g. how MIDI2 can take place). There were things I couldn't possibly cover in 40 minutes, so I'd like to discuss them in writing, adding various supplementary information.

What are music apps?

Since DroidKaigi is not an event where audio app developers gather (while such occasions are rare, they do exist e.g. audiodevcon), I gave some bits of explanation on how audio features are provided on operating systems and platforms such as Android.

Except for small apps like command-line tools that run on the desktop, most applications use some kind of sound-related functionality. The most obvious way to trigger sound is through notifications. Platform notifications, by default, will play some kind of sound unless the user has muted it. For example, a warning sound might be emitted when the user sends an invalid command in a terminal, or an error dialog might be displayed if the application notifies you of an error, or if the application itself encounters an error. App developers rarely call audio functionality directly for this purpose; at most, they might play a single audio file, like System.Media.SystemSounds.Beep.Play() on Windows.

What about audio applications in terms of music? Typical users listen to music and watch videos, so in this context it is still generally tied to features that typical users use. While this falls within the scope of what's known as a media API, playing media like audio and video with a proper user experience requires some skills, and if the platform's media API itself is lame, then mere playback feature could become awkward.

Of course, Android doesn't any concern at such level, and as of 2025, these features can be mostly achieved by using the Jetpack Media3 (formerly ExoPlayer) API. The Google Play Store offers a wide range of media players, including YouTube, TikTok, Spotify, and even Apple Music. Since Android isn't far behind Apple in this area, there's little point in discussing app development in this area. There were handful of sponsor companies that offer media streaming services with their booths at DroidKaigi, and some of them told me that they are using modified versions of Media3.

What Android lacks is not applications for playback, but applications for the creative class. Music production apps such as DAWs used in trackmaking (in Japangrish we call this kind of work "DTM", desktop music - regardless of that this term is appropriate in a mobile environment) require more strict real-time audio processing (we'll explain why later).

Android Platform Audio state of union (2025)

Audio developers are always reluctant to develop for mobile, and it's worse on Android development compared to iOS development, for various reasons. Here we discuss how so.

"Android has a lot of audio latency"

A common complaint about Android audio in the 2010s was that Android lacked or had insufficient real-time audio capabilities. Although it is only historical nowaday, there used to be some detailed analysis of Android audio latency published by Superpowered around Android 5.x:

https://superpowered.com/androidaudiopathlatency

It used to be true, but their points are outdated in the 2020s, especially in 2025. On Android, starting with Android 2.3 (API Level 7), audio apps could implement audio processing in C/C++ instead of Java, and Android 8.1 (API Level 27) made it possible to use AAudio, which supports MMAP-ed streams (strictly speaking, AAudio itself was usable in Android 8.0, but since it did not support MMAP-ed streams, it was only about their new API without low latency support).

Another thing to keep in mind is the Audio Latency (5.6) and Professional Audio (5.10 ) sections of the CDD (Compatibility Definition Document), which Google requires device vendors to comply with in order to call their devices Android. These sections define audio latency standards; for example, a device must have a round-trip latency of 20 ms or less to claim that they "support Pro Audio". Since the Android platform will remain open at the source code level, as represented by AOSP, until 2025 and in theory any low-quality device can be built, arguments like "Android audio latency is..." does not make sense. When people discuss hard low latency on Android, it should be primarily about Pro Audio compliant devices.

The time-consuming processes that cause delays are found in ALSA and HAL (android.hardware.audio.service process or bus and DAC/ADC), or in the AudioFlinger implementation, including AAudio. Google's implementation of AudioFlinger does sufficient job for low-latency audio, and since Google has clarified quality standards, there is probably nothing that Google could work on anymore. (BTW you might sometimes see some misleading technical explanations such as "Audio Flinger was the audio foundation before AAudio". But if you carefully trace the source code, you would see that AAudio also processes audio via AudioFlinger.)

Since DroidKaigi is not an conference for audio developers, so I took the time to explain what problems arise when audio delays occur, what real-time audio processing is, etc. I didn't mention this in the session, if you're interested, I recommend reading Ross Bencina's Real-Time audio programming 101: time waits for nothing or watching the ADC19 Real-Time 101 session video .

Also, although not mentioned in the session, there were some statistics on actual latency per device.

https://superpowered.com/latency : As mentioned above, this was around Android 5.x. The information is outdated, only for historical material.
https://juce.com/maq : Statistics collected by JUCE around 2018. The page has gone.
https://ntrack.com/android-latency/devices : These figures are published by n-track studio, a company that makes DAWs. But I don't find them reliable - even if we only look at Pixel devices, the figures show that the Pixel 4a has the lowest latency and the Pixel 10 has the slowest.

…And so, the only documents I was able to provide were these two:

https://android-developers.googleblog.com/2021/03/an-update-on-androids-audio-latency.html : A blog post written by Don Turner, who was an audio developer advocate on the Android audio team at that moment. However, this blog post focuses on the latency of the average Android device as of 2021, so it's not for those who are interested in low-latency audio. Even numerically, an average latency of 30 ms is still quite poor for that.
https://chromium.googlesource.com/external/github.com/google/oboe/+/devicelist/docs/Devices.md : This document summarizes the audio latency of Android Pro Audio compliant devices, showing that latency of less than 20ms is possible. This document mentions not only Pixel but even Nexus, and mainly contains information on devices that are even older than the above blog.

These are old, but while "audio delay is too large" is no longer useful as it is old information, "audio delay has been improved enough" is in general useful even if it is outdated. (Technically there can be some regressions, such as "MIDI latency has increased" in iOS 14, so I do say "in general".)

"Android is Java, so..."

When people say "Android is Java," they're really talking about two separate things:

(1) Java cannot be used because it is not a language suitable for real-time audio processing. Most languages, including C#, JavaScript, Python, and Ruby, are not RT-safe. Therefore, the argument goes, Android is not suitable for audio applications. It is simply misunderstanding. No proper audio app developer would say this. Audio processing is required on a real-time audio thread, and Android's audio processing is of course done in C/C++. Developers up to Android 2.2 could not use OpenSL ES, so this argument was correct until then, but in 2025, almost no one uses Gingerbread.

(2) Having to code everything except the audio processing in Java is a hassle. This is true to some extent, but there are two points to note:

(a) Multi-language app development isn't only about Android. If you're developing an audio application for Windows using WPF, you'll likely handle audio processing in C++ and GUI in C#. Even on Apple platforms, where UI and audio code can run on the same native code runtime, building Swift for SwiftUI and C++ for audio together requires some annoyance to interoperate, at least not as easily as with Objective-C++.
(b) Audio app developers tend to favor cross-platform development, including support for multiple plugin formats, more than app developers in other fields. Therefore, their GUI part is often implemented using cross-platform C++ APIs. APIs like JUCE , OpenGL/Vulkan, and Qt are also available for Android, so if Android were to be excluded in this sense, it would only be that the library doesn't support Android. While DPF and iPlug2 don't support Android, the DPF plug-in seems to work on Android except for the GUI (some have been ported as aap-lv2 applications), and iPlug2 is rarely used in OSS (I'm not aware of non-OSS implementations), so this doesn't matter much.

Also, I didn't have time to mention this in the session, but you should design a separate GUI for the mobile platform. I've ported several JUCE plugins to Android, and even though JUCE supports Android, I've never found the GUIs of individual plugins made for desktop to be usable.

"iOS has great audio features, but Android doesn't."

It's generally true that Apple is a company that's committed to audio features, and Google is not, but the Android platform does indeed have various audio features, including:

MIDI 2.0 support: Android 13 added support for the USB-MIDI 2.0 protocol, and Android 15 added support for virtual MIDI 2.0 devices. Since Apple has not developed BLE-MIDI 2.0, MIDI 2.0 support in Android is complete. (In recent years, real-time performance has become a strict requirement for MIDI device input and output, and BLE cannot achieve real-time performance and is not RT-safe in principle.) USB-MIDI 2.0 support was implemented before ALSA, so unlike MIDI 1.0, it was implemented in Java without ALSA rawmidi, so it was ahead of MIDI 1.0 to the extent that it was implemented in Java. Windows still does not support MIDI 2.0.
- By the way, the MIDI 2.0 specification had revamped in 2023, and support for MIDI-CI, as implemented by Apple, is not required in the platform API (Apple also states that the MIDI-CI API has been deprecated). The specification process was kind of in haste before quality.
The term "Android XR" arose last year, but Android 12 added support for Spatializer, which allows acceleration by hardware (mainly head trackers).
BLE Audio codec implementation. Since Pixel Buds are in the market, Google is actively committed to this field at the platform level.
Android 14 adds support for car audio plugins, which allow you to adjust DSP functions required for Android Auto.

Should Google still support any further audio features on the platform? Well, actually yes... there is audio plugin support. Android doesn't have an audio plugin feature equivalent to iOS AudioUnit V3, DAW features as well as plugin products in general are built on the premise of audio plugin formats, and therefore they don't exist on Android. This needs to be fixed somehow.

"Mobile platforms are cumbersome"

Last but not least, but for developers who primarily work on desktops, mobile platforms are simply a pain. They have small memory and storage capacities, weak CPUs, and can only display one application per screen. Even if you could feature the fully functional DAW on a mobile device, it's too small to make tracks. Both iOS and Android have strict boundaries for app installation and execution, making it hard for apps to communicate with each other.

These problems cannot be solved by app developers alone, so the most reasonable approach is to release a product tailored to the mobile platform. It would be great to be able to carry out complementary tasks such as editing synth presets on the mobile device while lying down, and then transfer the resulting sound to a PC via the cloud.

Of course, mobile is troublesome not only for Android but also for iOS, so in the context of what makes it inferior to iOS, there is no further points digging in this topic.

What's the point of an audio plugin?

So far, the only remaining issue Android has to overcome, especially compared to iOS, is the lack of audio plugins. Is this an issue that can't be resolved?

Since the audience at DroidKaigi is not audio developers, I had to explain what an audio plugin is in the first place. If you are similar, take a look at the slides. There were hacky explanations like "DAW is like an IDE," "audio plugin formats are like programming language runtimes," and "plugin formats are like LSPs", because the attendees were mostly programmers or software engineers.

When we release a DAW product on the platforms like Android where audio plugin formats don't exist, the DAW must provide all of the instruments that can be used within that DAW. However, just like there are various piano brands like Steinway, Yamaha, and Kawai, and various guitar brands like Gibson and Ibanez, software synth developers also have their own unique sounds. It is not realistic to expect that a DAW vendor (rather than instrument vendors) provides all the available plugins. Even if a DAW vendor were able to design an online marketplace, limiting the catalog of instruments available there harms users' freedom on trackmaking.

In this slide, I gave an example of a Cubasis 3 review, where a user left kind of negative feedback like "we can't use plugins like in the iOS version", and the developer responded like "because there's no plugin system on the platform" (I can't link to the individual reviews, so please search for "VST".), but there are many more similar reviews for FL Studio Mobile . The lack of a plugin format affects reviews as users are dissatisfied with it.

Effects can also be used in situations other than music software. For example, if you want to process audio in a live streaming app like OBS Studio, you can apply some VST (although whether you can apply a VST "3" plugin or not is another matter ).

Barriers to Audio Plugin Mechanisms on Android

If there isn't a plugin format that can be used on Android, why not create one? For platforms other than Apple, it was Steinberg, not Microsoft, that developed VST2 and VST3 for Windows (Microsoft even created an audio plugin system called DirectX Plugin in the past, and also has a system called APO, though it's not generally used here). LV2, which is primarily used on Linux but can be used on both Windows and Mac, was started by a member of the Linux audio community. Until AudioUnit V3 was released, people used AudioBus on iOS, created by app developers in the Apple ecosystem. Should Google create a plugin format for Android by their own?

Based on all those kind of thoughts, I manage a project called AAP: Audio Plugins For Android (I only mentioned the link briefly at the end of the slides, didn't even mention the name of the project in the session at all). Based on my experience, I would like to explain some of the features of plugin formats that are hard to achieve on platforms like Android.

(I wrote a similar article about AAP at shibuya.apk (occasional Android dev meetups) two years ago (in Japanese), with a more about AAP internals. The content is a bit outdated: https://speakerdeck.com/atsushieno/building-audio-plugin-ecosystem-on-android (written in Japanese)

DAWs are unable to dlopen() plugins

On the desktop, a DAW can load a plug-in executable program dlopen()(or some NSBundle API on macOS, LoadLibrary()on Windows), and run it in the same process as the DAW, but Android runs isolated apps i.e. a separate Unix user is created for each application, and the DAW and plugin run in separate processes, so communication between the DAW and plugin must be achieved using IPC (inter-process communication).

To go a bit further, while it is technically possible for an Android app to dynamically load native libraries passed to it by other apps, such apps are prohibited by Google Play Store policy. Such apps would have to be distributed outside the Google Play Store. I only mentioned during the session talk though, at the Linux Audio Conference 2025 , which I attended in June, there was a session on a project called LDSP , which aims to freely access ALSA and use audio functions on rooted Android devices .

At DroidKaigi we only talked about this level of granularity, but here is a more detailed discussion:

On desktops, DAW audio engines and audio plugins generally run in a single process. In contrast, the plugin mechanisms of typical web browsers are equipped with a sandbox mechanism, which maintains safety by preventing the browser runtime crashes even if a plugin crashes. IDEs are not as clear as that. VSCode has an Extension Host Process mechanism, but in IntelliJ IDE, plugins run in the IDE process. Generally, a process-separated model increases robustness, but the inability to share memory space between the host and plugins increases communication costs and complicates plugin system development.

One of the fundamental principles of realtime programming is "no system calls." System call implementations are system-dependent, and depending on the implementation, they may contain code that breaks realtime safety, so they are generally not marked as safe. In reality, some syscalls, such as memcpy()are commonly used and not considered as problematic (though this is not considered good for effective memory caching). Syscalls used by IPC features are usually non-RT-safe.

I described "DAW audio engines" (not like "the DAW app" entirely) because I'm having such a design in mind that the audio processing part and the other part (including the DAW GUI) reside in respective process spaces. It's better to load plugins in the audio engine process, and even if they crash the process, it would not affect the main part. Loading individual plugins in their own processes reduces the scope of the impact, but communication across process boundaries cost a lot, and the disadvantage of going beyond process boundaries between the audio engine and all plug-ins within the strict constraints of real-time processing likely outweighs the cost of more frequent audio engine crashes. Bitwig Studio has an option for such a mode, but should be considered as "for debugging plugins only".

Binder IPC is not RT-safe

Since its early days, Android has had a mechanism called Binder that enabled low-latency IPC on the Linux kernel. In Android 8.0, this mechanism has been further improved to support realtime priority inheritance . Priority inheritance allows two threads communicating via IPC to raise the priority of the thread when handing over control to the other application. Without this mechanism, for example, if an AAudio thread running at real-time priority passes control to another application's process, that thread would run at normal priority and have to wait for the other thread to finish processing before being handed over control. This results in the original AAudio thread having to wait endlessly for this. This is known as priority inversion .

Binder's priority inheritance has a fundamental defect: it doesn't work in the framework domain. It can only be used by developers of the Android platform itself or by vendor device drivers. Binder, which ordinary users use in their own services, runs in the framework domain, so priority inheritance is not possible. If we, as Android app developers, design an audio plugin format and use Binder internally, audio processing will not be performed in real time.

This is one of the limitations that makes it impossible to create an audio plugin mechanism in Android at the moment.

Plug-in GUI and the principle of one app, one screen

Android applications generally run one app per screen. If the DAW and plug-in are separate applications, their GUIs cannot be displayed at the same time. Therefore, some kind of workaround is required.

GUI is not mandatory for audio plugins, and it is possible to retrieve and manipulate lists of parameters and presets. In fact, AAP uses Jetpack Compose to create such a default GUI, but it is generally cumbersome for trackmakers to use. It is similar to the idea that "you can do anything by objectifying complex business logic and manipulating it via a PropertyGrid". It is better if we can simply support GUI instead.

When I talked about AAP at shibuya.apk, I introduced two approaches to implementing it:

Web UI: If it is not possible to load the plugin's execution code in the host process, then the approach is to display a WebView on the host and load the Web UI resources provided by the plugin as the plugin's GUI within it. Since plugin parameter operations can be performed via IPC, the maximum that can be performed from the GUI is "all operations that can be performed from the host via IPC" (this is only the maximum, and it is not necessary to support everything).
SystemAlertWindow: By obtaining special permission from the user, you can display an overlay on top of the host UI in response to a special request from the host, while keeping the GUI in the plugin process.

However, I later realized the following approach is possible, and now I believe this is the most optimal solution:

SurfaceControlViewHost: Android 11 added a feature that allows the host SurfaceView to reflect any android.view.View on another (plugin) app side, transferred via IPC , so this approach uses this. UI events are handled by the plugin's View implementation. The host can use surface control APIs as well as control SurfaceViewitself e.g. show and hide it. It is closer to audio plugins in desktop DAWs. Since the host itself displays the UI of other applications by the host's own will, there is no security concern like UI hijacking (because of "one app per screen" system).

The problem with the Web UI approach is that the DSP and GUI of a plugin cannot share memory space. This model is well suited to plugins developed for the LV2 format, where the DSP and GUI are separate dynamic libraries and do not share memory space. However, only some DPF plugins use the Web UI in LV2, and it is unclear whether they can be universally reused for Android porting.

The SurfaceControlViewHost approach can handle any view and share memory space in the plugin process, so if your desktop GUI code works on Android, it's in theory possible to port it directly. Having said that, depending on the GUI framework, transferring it to a SurfaceView may not work as expected. It's actually quite uncertain. For example, while I was able to display the GUI of JUCE (described below), I have never been able to get its inputs to work properly in the plugin process. JUCE's Android support is not well-developed with such a use case in mind.

I explained how to use SurfaceControlViewHost in detail in TechBooster's INSIDE [Technology Secrets] two years ago (but only in Japanese...). In English, the only information I have is a blog post by CommonsWare published around the time of the release of Android 11. There are some Chinese blog posts, but they are not very insightful either (I can read some Chinese). I also briefly talked about this API at a Mobile Dev Japan meetup this year (just the basics).

Regarding GUI, I think it's safe to say that the technical hurdle of mastering SurfaceControlViewHost is "difficult" but no longer "impossible."

Appendix: Plugin format based on Binder IPC and maintaining forward and backward compatibility

This is something I always skip due to lack of time, whether at DroidKaigi or shibuya.apk, but audio plugin formats require APIs designed with forward and backward compatibility in mind. If a situation were to arise where a plugin with a new API couldn't be loaded on an older DAW, or vice versa, the plugin ecosystem shrinks. Maintaining compatibility is a key requirement. Both LV2 and CLAP avoid changing the core by defining extensions as individual APIs, and VST3 ensures this with a COM-like query interface called VST3-MA. AudioUnit takes a similar approach to VST2, but explaining it would be tedious, so I'll skip it this time (it is not important this time).

Existing desktop audio plugin APIs are all built on the assumption that dynamic libraries can be loaded, so they cannot be applied as is to mobile platforms, and simply communicating through a proxy client/server cannot solve this problem of forward and backward compatibility.

If a plugin format is built using IPC, forward and backward compatibility can only be achieved by adding new methods. If the data indexes assigned once could be fixed, as in the case of Protocol Buffers, forward and backward compatibility might be possible, but deleting a method in the AIDL used by Binder would simply change the index and result in an incompatible interface. This would require supporting multiple versions of the protocol to bridge the gap, necessitating a cumbersome mechanism for filling the version gaps, like how Android Audio HAL works. (The difference between HIDL and AIDL in Android's lower audio layers makes things even more complicated, but I'll skip it for now.)

For this reason, I believe it is best to structure the API of a plug-in format so that it does not depend on AIDL as much as possible. In AAP, all operations based on the plug-in API's extended functions are realized using MIDI 2.0 SysEx to avoid hard binary dependencies on the API. Plugins or hosts that do not respond to the extended function's SysEx will ignore those commands.

How Apple Overcame Technical Challenges with iOS and AudioUnit V3

Apple originally had its own desktop plugin format called AudioUnit V2, and simply built an API to make it compatible with the iOS architecture. That being said, it is relatively easier than designing a plugin format from scratch, but designing a plugin format to fit iOS's architecture, especially its separated process architecture, is not easy (I hope you understand why I've been writing so much up to this point).

How has Apple overcome this technological challenge?

First, a mechanism called App Extension was built for the iOS platform. This allows the plug-in mechanism to be used in any application running on the iOS or macOS platform, and AudioUnit V3 is designed to work as an App Extension for DAWs. Communication between the App Extension host and the plugin is carried out via IPC, which is similar to what is done with Android Services.

The GUI display will be developed as an AudioUnit GUI Extension. Like SurfaceControlViewHost, it is explicitly loaded by the DAW as an extension, so there is no concern about UI hijacking.

While App Extension IPC itself did not have realtime IPC support like Binder, iOS 12 introduced a realtime IPC priority inheritance mechanism called Turnstiles to the XNU kernel. The design idea for Turnstiles is not originated by Apple; it seems to have been implemented by Sun Microsystems (then) in Solaris in the last century, 199x. A SunWorld article from 1999 provides more details ( via hacker news ). This feature is now available in AudioUnit V3.

XNU turnstiles allows AUv3 DSP processing at realtime priority when the DSP is running on a single thread, which already made it possible to achieve RT-safety ahead of Android. However, iOS 14 further added a feature called Audio Workgroups. This "groups" not only the audio thread run by the AUv3 host and the DSP thread running the audio processing function in the AUv3 plugin, but also other threads running at real-time priority, maintaining the real-time nature of cooperative operation. It is not generally necessary unless such real-time threads are created in the plugin. See the WWDC 2020 session video for more details.

Looking at it this way, some readers might get the impression that Apple is ahead of Android in audio features, but in fact, iOS 12, which implemented XNU turnstiles, was released in 2018, which is quite recent. Five years have passed since then, until Apple releases Logic Pro for iPad in 2023.

As I explained in a blog post (note: in Japanese), I wrote following the release of Logic Pro for iPad , AudioUnit V3 recently added a feature called Compact View. This allows you to create a UI apart from the main plugin UI. It displays just a few knobs for key parameters, allowing you to see and adjust multiple plugins installed on each track on a relatively small screen like an iPad. It's these kinds of UX optimizations that make the UX of DAWs that support AUv3 possible. Of course, this is merely a matter of plug-in format conventions, so it should be easy to implement this for plug-in formats that work on Android.

How should the Android platform respond?

Challenges for the Android platform

The only feature missing from the Android platform seems to be audio plugins, but the platform is still evolving in terms of optimizing its internal design. One interesting feature added in Android 16 is in-process media codecs. This means that hardware vendors can implement media codecs in a "safe language" that can be called and executed within the application process. While it says "safe language," this is currently believed to refer to Rust. Conventional media codecs that do not use IPC will continue to run in a sandbox, so IPC is required to communicate with the application (although it should be at the level of exchanging control commands rather than passing data via IPC). For more details, see this AndroidAuthority article .

(PS: AOSP has a loader implementation here.)

This is an interesting move by the Android audio team, as one possible solution would be to allow safe plugin DSP code to be loaded into the host process.

However, the problem is not that simple. DSP code running in the audio thread needs to cooperate with other code (not necessarily GUI). For example, a sampler plugin would have to implement the code that loads and prepares a file stream and passes it to the audio thread in an RT-safe way in a non-RT-safe thread. If this thread were to run in a separate process space from the audio processing thread, then IPC would be required within the plugin implementation. This would be a significant difference from the existing plugin development paradigm, and thus would go against code sharing.

How about this, Google allow DSPs implemented in "safe" code to inherit realtime priory in Binder? This seems possible. (Dangerous) operations such as file I/O are usually performed in non-DSP code, so we can say they should not be called in DSP threads. However, it is unclear whether safety verification can be performed on a function call graphs, and the details of the safety verification are currently too unclear. Relevant sources may be in AOSP, but I did not dig in depth.

Another challenge is the lack of official support for Rust in the Android NDK. This issue has been logged in GitHub issues since 2021 , but progress has been slow. User space is currently seeking code in a language that isn't officially supported. While audio developers' continued use of C++ is a drawback, plug-in developers, like those working with real instruments, have a wide variety of brands, so developers likely won't be as enthusiastic about switching to Rust. Furthermore, plugins will likely continue to be developed using cross-platform technology, so migration to Rust just for Android is unlikely to happen. In this sense, I assume that an approach like safe C++ or alt-C++, where developers reuse and build C++ code, may still be viable. In that sense, Google's Carbon , which from the outside seems like it's unclear what it's doing, could be one solution (although it's not particularly popular in the alt-C++ community).

Challenges for the Audio Developer Community

If we wait for the platform to have all the necessary features, the audio application ecosystem will fall behind even worse, so we had better act on whatever we can as user developer community.

It would be tough to get audio developers to directly use frameworks and plugin SDKs that can be used on Android alone. The same is true for the AudioUnitSDK and VST3SDK. When searching for open source plugin projects on web sites like GitHub, the vast majority of them are JUCE projects. JUCE is a cross-platform, multi-plugin format framework for audio app development, and is widely used in this field. Projects that directly use the VST3SDK are rare. To put it in terms of mobile platforms, it's like everyone is writing apps in Flutter or React Native instead of Kotlin (ignoring KMP here).

JUCE is designed to make it relatively easy to extend the platform and plugin format, so for my AAP, I can use my JUCE module called aap-juce (this is also a unique world, but it's not the topic for today so I'll skip it) to create and host AAP. If you're starting your own plugin format project like AAP, it's a good idea to create a JUCE module.

A pain point here is that JUCE is (effectively) a commercial product, dual-licensed under both a commercial license and AGPLv3, and is a closed project that ignores contributions that are unrelated to the company's current direction (though at least they're not completely refusing contributions, which is much better than before). Android support isn't as powerful as other platforms, but compared to most other plugin SDKs that don't support Android in the first place, it's still relatively robust. So, for now, it seems like the most effective approach is for JUCE to utilize what it has and support Android support for other plugin SDKs like DPF.

Furthermore, since MIDI 2.0 has already standardized the functionalities that used to be fragmented among various audio plugin formats, I believe that actively adopting some MIDI2-oriented approach, where "whatever can be done with MIDI 2.0 should be done with MIDI 2.0, and whatever can be done only with audio plugins should be done with audio plugins", will lead to a breakthrough. It is a matter of development on the host side rather than the plugin side, but if plugins can accept MIDI 2.0 input, they will no longer need to process plugin-format-specific events, which will increase portability. JUCE is expected to support MIDI 2.0 input in its next major version update , so the future looks bright in this area.

Additionally, in terms of plugin GUI support, active use of Web UIs may make it easier to support GUI-equipped plugins on Android. However, how a plugin development framework supports Web UIs is still a unique solution for each framework, so some research would be required to determine what is needed to enable seamless use of them on mobile platforms. Another issue is how to enable the exchange of audio buffers and complex data types when hosting a GUI in a WebView (currently, audio streams are linked from the WebView via resource resolution using the URL resolver).

summary

The Android platform is comparable to other platforms when it comes to developing music applications such as DAWs, except for the lack of a plugin mechanism, but without a plugin format the musical instrument ecosystem cannot develop, so something should be done. However, to do this, the complicated issue of real-time priority inheritance of IPC needs to be somehow resolved at the platform level, and for the time being, the app developer community should focus on expanding Android support.

Recap 2024

Mon, 30 Dec 2024 13:45:00 +0900

It is around the end of the year 2024, so it's time to summarize up what I have done this year, just like I had been doing in the past. I'm primarily writing this for myself to record history.

2024 was a year that I got various work requests and I became much busier than last couple of years. On the other hand, I completely stopped managing local audio dev. community and interact with much less people in the worldwide audio domain. Nowadays what I have been working on is more interesting than them, to me.

Books

There was much less public development activity this year. What I had been spending time instead was, writing tech books in Japanese. Two are already published at my store at techbookfest.org. If you are curious about techbookfest, I have written a blog post about that many years ago. Now they have online store for ebooks. It's not like commercial publishing; take it more like Comic Market for programming books.

MIDI-CI Guidebook was literally about MIDI-CI. There was no book in the world that explains what MIDI-CI is about (maybe not even what MIDI 2.0 is; I wrote one about UMP back in 2020, and last year for June 2023 revamps). It is mostly based on my ktmidi development (I have written a detailed blog post about that in January: https://atsushieno.github.io/2024/01/26/ktmidi-ci-tool-released.html and https://atsushieno.github.io/2024/01/26/midi-ci-tools.html ). The relevant MMA/AMEI specifications (MIDI-CI core, Common Rules for PC, and Common Rules for PE) are of a few PDF documents that sum up to about 200 pages, and more if we count individual specifications such as GM profiles. This book compresses them into about 100 pages (in Japanese). The book will become more useful after a few years later, maybe not now.

Some folks told me that I should publish them in English, but haven't spent time on that.

"The Ideal Audio Plugin Format" is more like a collection of thoughts on audio plugin technology through my attempt to develop a unified plugin hosting foundation (WIP). The idea was born while I was writing another book that includes audio plugin host development where I will mostly just explain the plugin APIs. This book is totally different and I thought I should write what aspect of specific plugin formats are good and/or bad, where their technical choices have advantages and/or defects. For example...

VST3 and CLAP chose to require host to invoke their functions on the GUI (main) thread while AU does not. I describe it like VST3 and CLAP are single-threaded COM while AU is multithreaded COM. Technically AU is superior, but it'd be easy for plugin developers to just write single-threaded COM just like VisualBasic developers did (and that's kind of rational).
Fast plugin scanning; LV2 and AU, and VST3 after 3.7.6 are good as they can only parse metadata, while CLAP and VST3 prior to 3.7.5 still require loading of the DLL for just scanning (it was intentionally done at CLAP in favor of single source of truth).
On plugin instancing lifecycle, I mentioned LV2 instancing is annoying because its sampleRate must be given at instancing; unlike other plugin formats, changing sample rate after connecting requires re-creating plugin instances.

AAP and ktmidi for MIDI 2.0 updates

Unlike last year when I have achieved a bunch of new features, there was surprisingly not a lot of AAP development this year. There are handful of reasons for that, but the biggest reason was that I spent a lot of time on some part time work on Android audio stuff, apart from my primary jobless. It is still ongoing, and another book writing is still ongoing, and I can only spend the rest of my time for coding.

There was still a big new feature addition in the latest AAP v0.9.0 release: AAP plugins can now act as software MIDI 2.0 devices!

It is actually dependent on the new Android 15 feature: android.midi.MidiUmpDeviceService provides the overall foundation for UMP-based software MIDI devices, and I just adopted our "existing" MIDI 2.0 device service support (I have done that years ago; AAP MidiDeviceService sent and received UMPs with some protocol negotiation hack).

MIDI 2.0 device access

But. I had to question to myself: what can we do with this? What we clearly need is some playground for MIDI 2.0 devices. My Resident MIDI Keyboard already supported MIDI 2.0 UMPs (after the blog post I have fixed some issues to make sure that things really work well), but that's not enough. At least I want to "play some MIDI 2.0 based song files".

But there are too many missing bits for that. First of all, we don't even have MIDI 2.0 song files. I had been working on my MML (music macro language) compiler that supports MIDI 2.0 song file (it's like SMF but for UMP, in my own format, as MMA hadn't published any specification for SMF2 Container File format). And we don't have a decent MIDI 2.0 song file player.

That led me to rather work on ktmidi software stack than AAP stack. As an early adopter like me, 2024 was fairly a good year to start working on MIDI 2.0 device access support. Ubuntu 24.04 LTS was released with Linux kernel post-6.5 version that came with ALSA MIDI 2.0 support, and as I mentioned, Android 15 started to support UMP software devices. ktmidi had the most advanced software stack for building UMP and MIDI-CI support, but as its MIDI device access had been stuck around MIDI 1.0 APIs, its usage was limited.

And managing platform API access is quite tough area in Kotlin Multiplatform libraries - we have to provide platform bindings for JVM, Android (note that it is different from JVM), native (for each platform!), JS, and now WebAssembly. The most wanted areas are UMP support on JVM and Android. Supporting Android is easy, but JVM is quite annoying. If it were about native, cinterop support for CoreMIDI would not be hard. But on desktop, Compose Multiplatform apps run on JVM and there is no cinterop tool. ALSA support was another source of headache, as it was kind of special that we have to bundle libasound.so that may not blend with locally installed ALSA (especially at config sections), and bundling native libraries do not always work (like, Panama and jextract tool do not support that kind of packaging). I ended up buidling libremidi JavaCPP wrapper for MIDI 2.0 I/O support. It does not seem to always work yet, but on some environment I could finally send and receive UMPs.

KMDSP

Once I got MIDI 2.0 I/O working, I wanted to implement "MIDI 2.0 player" kind of stuff. One last missing piece I didn't bring from C# managed-midi project was a visual MIDI player called xmdsp. Since Kotlin 2.0 stable and the first stable version of Kotlin Multiplatform was released this year, it felt like a good time to implement it in Kotlin. It was done shortly:

https://github.com/atsushieno/kmdsp

... and even better, now it runs on the web browser (using Kotlin/Wasm):

https://atsushieno.github.io/kmdsp/

It works fairly well with MIDI 1.0 protocol. But what we want now is MIDI 2.0 music player. While it is able to send UMPs to MIDI output devices, we don't really have MIDI 2.0 devices. It is probably not a big deal as UMP outputs from those MIDI 2.0 apps will be translated to MIDI 1.0 messages if the connected device is MIDI 1.0. But then it does not make sense to use MIDI 2.0 there. What I want instead is to be able to play "MIDI 2.0 music" over a "MIDI 2.0 device" which has more than 16 channels. Now I'm exploring that kind of fields, but I need more time to achieve something useful.

AAP GUI frontline

It seems really disastrous that there was no accomplishments on AAP 2024 roadmap items. But as I described above, MIDI 2.0 UMP device work had disrupted and took place there. I needed too improve the overall ecosystem in other parts.

And on AAP itself, I was stuck at the first item I tackled. As it became possible to get in-plugin-process UI working last year already, I continued to get more plugin UI stuff working, namely aap-juce plugin UIs. However, apart from the fact that JUCE is not quite ready for non-standalone apps, there are still some blockers that fails to show remote UI via SurfaceControlViewHost. I got some plugin UI showing up, but its UI interaction does not work. I don't spend much time on things that are not fun to investigate. JUCE desktop plugin UI does not contribute much to AAP either. Their UI should respect mobile in the first place.

When juce_gui_basics die and they switch to newer UI technology like Web UI, things may become easier for porting and I'm looking forward to it. I myself also have been exploring WebView UI integration on desktop audio apps in my latest experiments. When I worked on that kind of stuff a few years ago, I was so minority, but it seems different nowadays.

On iOS side (AUv3) things are more like, desktop and mobile should not share the UI bits, and the same kind of trend would work for Android. I'm not hoping against code sharing, but redesigning UI code for diversity will cause rethinking the UIs, dumping obsolete technology for new ones.

Building MIDI 2.0 Ecosystems on Android

Fri, 12 Apr 2024 15:30:00 +0900

Building MIDI 2.0 Ecosystems on Android

These days I have been exploring the new MIDI 2.0 opportunity on Android. There has been a lot of potential spaces to make things happen, and I would break various aspects of audio and MIDI 2.0 application features down to small pieces.

New in Android 15: MidiUmpDeviceService

For Android music app developers like me, Android 15 is a big update! It brought in MIDI 2.0 to "everyone" as MidiUmpDeviceService, i.e. support for non-USB MIDI 2.0 devices.

Back in Android 13, we were able to use MIDI 2.0 USB devices on Android, and MidiManager class connected our MIDI client apps to USB MIDI 2.0 devices. USB MIDI 2.0 is a tailored USB class that supports "UMP" which is the new MIDI 2.0 transport which is not the traditional MIDI 1.0 bytestream transport. Android 13 came only with USB MIDI 2.0 support, and it left non-USB MIDI devices behind MIDI 1.0 legacy.

Around that time, I have created an issue at Google issue tracker, asking for some way to create software MIDI 2.0 devices. In the end we needed a new platform feature to achieve that.

There has been confused years on how MIDI 2.0 devices would be supported at platform layer. Only macOS was a platform player that supported MIDI 2.0 devices natively. The game has changed in June, 2023 though, when MMA/AMEI published the totally revamped MIDI 2.0 specification documents. Protocol Negotiation is gone and platforms are getting clearer that we would need dedicated ports for MIDI 2.0 UMP ports alongside traditional MIDI 1.0 ports.

MidiUmpDeviceService is the only class for the latest MIDI 2.0 additions in Android 15, and it is basically MidiDeviceService for UMP. There are surprisingly small difference between MidiDeviceService and MidiUmpDeviceService - both accepts ByteArray (in Kotlin) as the input and output messages. Their input receivers and output senders are created almost the same way. The differences are trivial:

UMP ports are bidirectional so that you only describe <port> in MIDI device descriptor XML
for <service> element in AndroidManifest.xml, we need <property> element instead of <meta-data> element.

API wise, UMP ports being bidirectional does not affect how we deal with MIDI input and output streams in terms of MidiSender and MidiReceiver.

I do not create any example project to demonstrate the feature - instead I have various use cases on my existing projects that already supported MidiDeviceService. In other words, they are already real world usages:

Audio Plugins for Android already integrates MidiUmpDeviceService so that any instrument plugin could behave as a UMP MIDI device, just like it used to for MIDI 1.0 device. aap-lv2-mda, aap-lv2-sfizz, and aap-juce-hera already have these changes applied.
Resident MIDI Keyboard works as a virtual MIDI keyboard that can send MIDI 1.0 events to other MIDI apps, and now it could also work as a MIDI 2.0 keyboard that is recognized by Android platform and usable with any UMP device.
ktmidi-ci-tool in ktmidi handles MIDI-CI messages, and provides virtual MIDI I/O ports so that it could work as a virtual MIDI-CI service too. It now works as a virtual MIDI 2.0 device and can handle groups too.

Note that they are available only as sources. Android 15 is still at Developer Preview state whose API is not stable yet. We cannot publish any app that uses the preview API to Google Play Store yet. They are available via AAP APK Installer instead (you need GitHub token to get those non-release GitHub Actions builds) - if it works (unconfirmed).

Dealing with UMP

MidiUmpDeviceService is just an aspect of MIDI 2.0 apps. Like MidiDeviceService for MIDI 1.0 bytestream, it only provides byte stream of UMPs. It was alright for MIDI 1.0 as it is popular enough and everyone could handle the bytestream inputs and outputs. It is probably not the case for UMP - people have little idea about UMP and would not know how to deal with them. There is nothing to be afraid though - they are still just arrays of integers that you can always parse and/or generate. We just need to understand the new format.

MIDI 2.0 UMP (Universal MIDI Packet) is a totally different representation of MIDI events that is extended to 32-bit, 64-bit, and 128-bit integers (compared to 7-bit MIDI 1.0 bytes). For example, MIDI 1.0 Note On message looked like this (in hexadecimal):

| 90 | 30 | 78 |

MIDI 2.0 UMP Note On message looks like:

| 40 | 90 | 30 | 00 | F8 | 00 | 00 | 00 |

If you want to write code in Kotlin to deal with UMPs, ktmidi has been always there to support MIDI 2.0 UMP development. I also have C library called cmidi2 in case you prefer coding in C. If you write C++ code, there have been more options these days:

libremidi is a cross-platform MIDI 1.0/2.0 API that can handle platform MIDI accesses.
JUCE has been providing its preliminary UMP support API in juce_audio_basics module.
ni-midi2 offers some API to handle UMP and MIDI-CI messages.

Dealing with "SMF2"

As of writing this in April 2024, there is no corresponding format to SMF for MIDI 2.0 UMPs - there is a MIDI2 specification for "MIDI Clip File" which is basically a single track of a UMP sequence. The thing is that, the concept of a song file has changed through the decades and we have some intermediate concepts between a MIDI event and a "song" - SMF1 consists of a sequence of tracks, and a track consists of a sequence of MIDI events. But most modern DAW tracks have "clips" i.e. a group of MIDI events, and multiple clips could be reused, and/or overlapped each other. So, MMA has been taking longer time to finalize the "song" file format which is going to be a bundled collection of those clips and miscellaneous resources.

So far, SMF2 "Clip File" is already a finalized specification so that we could expect to be usable. SMF2 is basically a UMP sequence with a bit of non-UMP header chunk, followed by UMP-based header chunk (i.e. you can use standard UMP parser to parse the rest), with a prefixed "delta clockstamp" (like SMF1 did with 7-bit variant length), which is also a UMP packet. ktmidi supports it too. ktmidi has been supporting a feature parity of SMF (song file) for UMP in its own format too, so that you can store them for playback.

Dealing with MIDI-CI and UMP Endpoints

MIDI-CI is loudly advertised as part of MIDI 2.0. To support MIDI 2.0, however, it is not required at all. You can leverage full power of MIDI 2.0 UMP without needing any MIDI-CI stuff at all, and it's totally fine. MIDI-CI brings in slight benefits to your MIDI apps, and most of other MIDI apps do not really support MIDI-CI features.

So, what is MIDI-CI? MIDI-CI makes use of bidirectional messaging channels between two MIDI devices so that they can "understand each other" and then provide advanced features based on the established session. With MIDI 1.0, everything was uni-directional like UDP. MIDI 2.0 transport made it like HTTP/3, by combining two uni-directional message channels together as one. Note that bidirectional messaging itself is NOT what MIDI-CI achieves. MIDI-CI is built on top of such messengers.

If you are still interested in MIDI-CI, then you can use ktmidi-ci if you prefer writing code in Kotlin, or juce_midi_ci module in JUCE 7.0.9 (or later), or ni-midi2 to some extent, if you prefer C++ and NDK. I wrote about ktmidi-ci here a few months ago. The API has greatly improved in the latest v0.8.0 release and became much nicer to deal with as a library.

MIDI-CI brings in three notable features: Profile Configuration, Property Exchange, and Process Inquiry. There used to be another component called Protocol Negotiation, which used to be essential to establish MIDI 2.0 connection and protocol promotion, but it's totally gone in June 2023 updates. MIDI-CI itself is on top of MIDI 1.0-compatible SysEx messages and no need to use other MIDI 2.0 bits.

Profile Configuration matters if (and only if) your application provides additional features to certain MIDI devices, based on "profiles". A profile indicates that a supporting MIDI device provides whatever feature it expects by its profile specification. For example, there is "GM2 single channel profile" that is to declare that the MIDI device channel supports the controllers along with what GM2 specifies. Profiles are not strongly typed and less bound to programmable APIs yet. Neither ktmidi nor juce_midi_ci offers Profile related features other than on-off switches.

Property Exchange matters if your application wants to control MIDI devices that provides something that goes beyond standard MIDI controllers such as CCs or NRPNs (Assignable Controllers in MIDI 2.0 wording), and still not too specific as device-specific SysEx. Properties are typically represented as JSON, so, it is typically used for something that works in non-realtime way. Since there is no MIDI 2.0 apps that offer MIDI-CI Properties on Android yet, it would be either a controller app to control USB MIDI 2.0 devices (if any), or you build a MidiUmpDeviceService which you intercept its incoming MIDI messages and handle MIDI-CI property messages. ktmidi-ci-tool is a reference implementation that does intercept those MIDI-CI messages.

There are slightly better ideas for proper use of Property Exchange, documented as MMA/AMEI defined properties, but I would skip it for now.

Process Inquiry matters if you want to have your virtual MIDI device expose current state of controllers, or you want to get your connected MIDI devices dump all of its status (note status/controllers/pitch bend etc.), IF it supports Process Inquiry (MIDI Message Report).

They are all supported in ktmidi. If you ignore Process Inquiry (you most likely at this state), they are available with juce_midi_ci too.

Lastly, in case you want to build somewhat complex UMP device and have to deal with UMP endpoint configuration and messaging, the latest ktmidi v0.8.0 added a new UmpEndpoint class that can manage Function Blocks and can handle incoming UMP stream message requests and send replies. It can also be used as a UMP stream client so that you can easily handle the connected UMP endpoint details (endpoint info, device identity, function blocks, etc.).

Bridging MIDI 2.0 devices over MIDI 1.0 ecosystem

Now we have our MIDI devices ready for MIDI 2.0 clients using MidiUmpDeviceService, and we can connect to MIDI 2.0 services using MidiManager since Android 13. UMP ports have their own ecosystem along with MIDI 1.0 ports.

But most of those DAWs are still based on MIDI 1.0. Can we use our UMP devices with those MIDI 1.0 DAWs? The simplest answer is "no", but if you provide some way e.g. MidiDeviceService that are supposed to receive MIDI 1.0 inputs and then your device code translates its 1.0 inputs to UMP inputs and forward to another UMP based device, as well as receive its UMP outputs and down-translate to MIDI1.

MIDI 2.0 "protocol" is designed to be easily translatable to MIDI 1.0 "protocol", and we could use this UMP translator trick to generate MIDI 2.0 inputs from MIDI 1.0 inputs, and output MIDI 1.0 bytes back when the UMP device is done with UMP outputs. ktmidi offers UmpTranslator class that implements normative conversion UMP-to-and-from-MIDI1 streams.

Access to MIDI 2.0 devices in Kotlin Multiplatform

If you build Kotlin Multiplatform apps or planning to migrate your Android app to KMP, and want to have MIDI 2.0 features, the latest ktmidi v0.8.0 added support for UMP ports in MidiAccess. The latest MidiUmpDeviceService was the great motivation for me to work on the stack. On Android it just uses the not-that-new API added in Android 13.

ktmidi v0.8.0 also added CoreMidiAccess (which is actually either TraditionalCoreMidiAccess or UmpCoreMidiAccess) that you can use in your Kotlin Native apps. So it would be realistic to use KMP for cross-platform MIDI 2.0 apps. It still lacks ALSA support (and Windows is out of course until Windows MIDI Services became ready for general consumption around the end of year), but hopefully soon.

AAP 2023 Year in Review

Sun, 28 Jan 2024 01:00:00 +0900

AAP 2023 Year in Review

Usually I write this "AAP Year in Review" blog post around the end of the year. I have been privately busy around the end of last year, and also been concentrating on ktmidi-ci-tool I just blogged a few days ago. Now that ktmidi-ci-tool is "released" to some shape, it's time to come back to AAP. This is the first thing to do in AAP world.

Project management work

I began 2023 with rebranding the project: it is renamed to Audio Plugin For Android, renamed from Android Audio Plugin Framework. It was to avoid possible trademark confusion.

I have published FAQ and Participating wiki pages. The former one worked whenever I want to forward any question to.

Now we have AAP APK Installer application that lets you install the AAP hosts and plugins that are registered in the app, so that those plugin project APKs can be easily installed by this installer. The behavior is still unstable (especially access to GitHub Actions artifacts without PAT seems quite lame), but our overall installation experience became better than nothing.

Remote plugin UI support

The biggest challenge in AAP would be support for in-process plugin UI. Before 2023, we planned to primarily use Web UI that can be instantiated by host "remotely". It was then supposed to communicate with the plugin itself through Binder. In 2023, we began with experimental in-process GUI using System Alert Window, and it worked to some extent. During the development process, I noticed that Android 31 has added a new remote UI foundation called SurfaceControlViewHost and found it useful enough to achieve in-process plugin UI. Now AAP's in-process UI can be instantiated "remotely on a DAW" (I will mention aap-juce-helio later) with some glitch (like it cannot be moved; UI adjustment needed on app code in juce_gui_basic):

This default plugin UI is based on Jetpack Compose, which is apparently not part of this JUCE-based DAW. We still haven't figured out how we can integrate JUCE plugin UI yet, but it won't be hard (at least not harder than managing it as a cross-process plugin UI)

New MIDI extension and MIDI message mappings

The GUI feature above is backed by a handful of improvements in the AAP framework foundation. We used to have aap-midi2 extension that only managed MIDI protocols. What I figured this last year was that there is not a lot of JUCE audio plugins that treats MIDI Program Change message as a preset changer, while we could easily achieve. This led me to implement default mappings for program changes and parameter changes (which was already sent over MIDI2 channel in 2022) to CCs, Assignable Controllers (NRPNs), and per-note controllers, in the name of (simply) midi extension. After this change, an arbitrary MIDI client can connect to AAP MidiDeviceService and send a program change message, and that results in the preset change.

New plugin manager app, resident-midi-keyboard, and aap-juce-simple-host

One of the annoyance in AAP until last year was that the default main launcher activity was not great and it demonstrated the audio effector feature only by pre-defined wav file in fixed sample rate, without appropriate WAV loader. Also the MIDI phrase it previewed was fixed. It needed complete overhaul.

The first simplification I could achieve was removal of any MidiDeviceService testing feature on the UI. They could be done totally outside the activity. While exploring the GUI support, I figured that System Alert Window can be useful for a "remote MIDI keyboard" app, and the idea came to realize as a standalone MIDI keyboard app: resident-midi-keyboard. It ended up to become the first Android app on Google Play Store "by the AAP Project". I blogged more details about the app last uear.

The next one was the biggest work: On the plugin default UI (PluginDetails Composable screen) I rewrote all the backend part (was written in Kotlin), in C++. Now we make use of Tracktion/choc to process sampled audio files, as well as designed and implemented super-simple AudioGraph processor (after evaluating some existing solution...) in brand-new androidaudioplugin-manager module. We can play MIDI notes, send parameter changes in more compact UI, and choose presets now. Web UI and Native UI can also send those events as long as the controllers are implemented and on the UI.

On another aspect, I have been using aap-juce-plugin-host, which is an Android port of JUCE AudioPluginHost. Since it involves a lot of extra code and is designed for desktop, it is always cumbersome and not appropriate for mobile uses. I ended up creating another simplified plugin hosting app, aap-juce-simple-host, which I can use with much less hassle.

Serious realtime processing for plugin extensions

2023 was a year that I started working seriously on realtime processing at various levels. One of the technical field that needs realtime processing is extension (AAPXS) messaging. The work, Realtime AAPXS, is also detailed in another blog post, but the key thing is that extensions cannot be messaging while audio processing is active, because the audio thread is the only messaging channel. Thus, every extension requests have to be translated to MIDI 2.0 SysEx8 messages. And we made it to work.

This was a big change and involved breaking changes, so I wanted to make a lot more changes at this chance (I avoid breaking changes unless it is really needed). Thus it took longer time, but then now we have "URID" extension (similar to what LV2 URID does), host extension foundation, and so on.

Similarly, we made another breaking API changes around audio processing buffers: aap_buffer_t. It looks more like CLAP audio processing model. There we could hide some implementation details and eliminate unnecessary API exposure.

Build simplification

The AAP plugins have become a lot over the years, and their build scripts had been diverse, which was just hard to maintain, especially at GitHub Actions which we cannot really run the tasks locally. Now we have simplified and commonized GitHub Actions build settings so that we do not have to worry about incompatibility. What we do mess instead is the Make-based build system, which is another annoyance but way better than dealing with GitHub Actions *.yml.

Another big restructuring in AAP is manifest simplification. that we do not need to generate static aap_metadata.xml in most case anymore. We can dynamically populate ports and parameters from the plugin code and extensions now, which makes it possible to have aap_metadata.xml almost manually editable.

Building and running AAP metadata importer tool has been always problematic, especially on aap-juce plugins, because we have to make changes to the desktop builds to add AAP classes resolved. But we kind of had to do it because it was not practical to manually write aap_metadata.xml without tooling aid. But we do not have to do it anymore. This opened the door to AAP world for some plugin projects. Namely we have aap-juce-surge project as an outcome.

Quality improvements

At a bit of background, I had been running online audio dev. meetup in Japan, and for one of the meetup we were discussing realtime audio processing (like, talking about the famous Real-time 101 ADC19 session). Around that time, aap-juce plugin ports were quite in low quality and needed vast improvements. I have spent many hours to identify what caused the problems from finding any audio glitch causes to realtime safety defects. Namely -

applied strict thread switching using ALooper API to dispatch any non-RT task to non-RT thread in MidiDeviceService implementation
disabled and prohibited auto MIDI device opening in aap-juce ports. JUCE on Android, by default, tries to open MIDI port that can be a software synthesizer which can perform realtime audio processing, and caused unnecessary audio glitches imposed on our code for no reason. It is an evil behavior by JUCE and strictly prohibited.
tracing support. We can examine how much time an AAP plugin process() takes, on both host and plugin sides.
Big aap-juce hosting performance improvements by identifying a JUCE Oboe issue: JUCE until 7.0.5 was allocating memory on Oboe realtime audio thread.

These changes have vastly improved aap-juce-helio, an AAP port of Helio Workstation DAW. It is known to run on Android, so I could just add some build tweaks to get it working. But the quality was not great until I fixed the issues I mentioned above.

New AAP plugin ports

I did not spend a lot of time to create new audio plugin ports (I cannot manage too many ports anyway), but there are still some. Here is the list of newly -or- emerged plugin ports that started working last year:

aap-juce-os251: OS251 port means react-juce works
aap-juce-byod: BYOD port means RTNeural works
aap-juce-peak-eater
aap-juce-seele
aap-juce-adlplug-ae: OPL/OPM/OPN emulator as JUCE CMake-based porting project. (I cannot move forward without OPN...)
aap-juce-ddsp: DDSP-VST port means tensorflow works (not TFLite though)
aap-juce-surge
aap-lv2-aida-x

Review

There had been a lot of achievements in 2023! I spent a lot of hours on the underlying work in 2022, and 2023 was of a bunch of leaps. We have GUI integration. We have new AudioGraph based plugin manager. We have realtime AAPXS foundation. We also have an installer foundation. We have nicer build automation. And we have a kind-of-working DAW that can apply AAP. It is finally getting the initial blueprint of the entire project.

There are, however, still a lot of incomplete works and TODOs. I will come up with the updated "Roadmap for 2024" to replace the 2023 version soon, but already seeing a lot of milestones.

Understanding MIDI-CI tools

Fri, 26 Jan 2024 20:00:00 +0900

Understanding MIDI-CI tools

These months I had been working on my latest project, ktmidi-ci-tool. I have written another blog post on the development itself, but since MIDI-CI itself is quite unknown at this state, I thought we need some explanation on what MIDI-CI tools are for, what we can achieve with MIDI-CI tools, and how to use them. Hence, I will be explaining a lot of the specification and the tools with lots of words this time.

evaluating MIDI-CI implementation interoperability

MIDI-CI is about interoperability, and there are some MIDI-CI implementations. So, why not try to connect each other? We should make it clear what and how they implement though. Here are the list of implementations I know of (listing from most featureful ones):

JUCE CapabilityInquiryDemo: based on juce_midi_ci, it exposes almost all MIDI-CI features it supports. It can define its own profiles and properties to test its receiver functionality. No Process Inquiry.
ktmidi-ci-tool: it is build upon similar mindset to JUCE CapabilityInquiryDemo provides, with additional features like in-place value text editor, Process Inquiry support, runs on mobiles and web browsers.
MIDI 2.0 Workbench: has various "checklist" items to verify if the connected device correctly implements those MIDI-CI features.
Apple CoreMIDI provides MIDI-CI features. No Property Exchange, no Process Inquiry.

MIDI 2.0 Workbench and ktmidi-ci-tool provide virtual MIDI ports so that if the other device (such as CapabilityInquiryDemo) does not come up with any MIDI ports it can still connect to those tools. (All those tools should not be tied to the actual MIDI connections, but everything - including ktmidi-ci-tool - so far expects MIDI connections.)

Among those implementations, ktmidi-ci-tool and JUCE CapabilityInquiryDemo are fully featured MIDI-CI tools that is enough to check almost all MIDI-CI implementation behaviors on both Initiator and Responder sides of a MIDI-CI device. Since JUCE CapabilityInquiryDemo is "just" a demo app, it does not come up with any users manual, sort of. Since MIDI-CI is not something every musicians or even music app developers understand well, the UI would look totally alien. Let's try to understand what they do so far.

These tools by themselves would be primarily demonstrating what you can achieve when you integrate those MIDI-CI features into your real-or-virtual MIDI devices, like synthesizers, effectors, controllers, etc. Such an actual product would not expose the same kind of UI as these tools. Instead, those tools would be useful to connect to your MIDI devices (or anyone else's).

Connect on JUCE CapabilityInquiryDemo

Here is the main UI. I shrunk it to small size as it's almost blank.

MIDI-CI assumes there is a pair of MIDI input and MIDI output.

Conceptually MIDI-CI is merely based on a binary I/O agents i.e. an Initiator and a Responder. They do not necessarily have to be MIDI devices on the platform, but CapabilityInquiryDemo expects them so far. It is similar to client and server, but since MIDI-CI is bidirectional, sometimes a device that acted as an Initiator might also work as a Responder when the other party sends a request to it. These labels are used per request (Inquiry and Reply in typical MIDI-CI names) basis.

When you use this tool, it is mostly for manipulating another MIDI-CI device (either really or virtually, like ktmidi-ci-tool), so you would need to choose the right pair of the MIDI in and out ports of the same device. Some MIDI devices would provide multiple in and out ports, then you need to pick up the right I/O ports. In the future this tool might present different form of ports selector for MIDI 2.0 support, but let's skip that so far.

Some other tools, such as MIDI 2.0 Workbench and ktmidi-ci-tool, also offer "virtual MIDI ports" to let other MIDI-CI devices connect from it without hassle, making connection tests easy. You can do it with JUCE, but CapabilityInquiryDemo doesn't, so far. (It is doable only on modern platforms such as Linux, MacOS, and iOS though. Windows developers will have to wait for Windows MIDI Services.).

The sshot below is where ktmidi-ci-tool configures MIDI port connections, but you don't have to choose anything if you are connecting it to CapabilityInquiryDemo and that app chooses "KtMidi-CI-Tool Virtual In/Out ports":

Save and Load the state on ktmidi-ci-tool

JUCE CapabilityInquiryDemo saves the state when it quits the application (or it might be doing that at any state change). Our ktmidi-ci-tool does not always do that - you need to explicitly perform "Save configuration" to save any local configuration. At this state there is no way to load arbitrary configuration file yet (cross-platform support messes it).

Discover our device(s) on JUCE CapabilityInquiryDemo

The next step would be to discover your device from CapabilityInquiryDemo. Go to "Discovery" and click "Discover Devices".

This tab shows information about and controllers for the "remote" "connected" devices. Clicking "Discover Devices" button publishes "Discovery Message" MIDI-CI message to the connection.

It would usually result in only one "Reply to Discovery Message" message, but there can be many replies from multiple Endpoints. I will explain what an Endpoint is later, but for now take it as an operation target MIDI-CI entity. This tab shows the details of only one Endpoint, so if you need to control another Endpoint, choose the device from top-right combo box that shows current device's MUID and device name (if available).

It should also be noted that there can be multiple "MIDI-CI devices" that replies to the Discovery Message. For example, your connection target may be a MIDI-CI "hub" kind of device that connects to multiple MIDI-CI device, or a virtual MIDI port that shoots Discovery Inquiry on different transport protocol such as Bonjour.

At the first sight the "Discovery" tab looks like it shows only "Basics" and "Profiles", but it also supports "Properties" if you scroll down.

The initial messaging interactions

The "Discover Devices" click event itself only triggers MIDI-CI initiator messaging of "Discovery Message". But when the client received a "Reply to Discovery Message", a lot of subsequent messages are sent to the responder:

Endpoint Message: it asks for the details about the Endpoint. We will discuss it soon.
Profile Inquiry: if the Responder supports Profile Configuration, it asks for currently available profiles on the MIDI-CI device, either enabled or disabled. We will discuss Profile Configuration later.
** Get Property Exchange Capabilities Inquiry**: if the Responder supports Property Exchange, it will first start asking about its Property Exchange Capabilities, namely the max number of concurrent property requests. We will discuss Property Exchange later.

And if the responder is CapabilityInquiryDemo or probably any juce_midi_ci based MIDI-CI device, this Initiator will receive the same set of these messages above from the Responder as well(!). It is because the Responder kind of need to know what this connecting MIDI-CI device can do. If you go to the "Logging" tab, you can see how it is interacting with the other device:

Let's focus on the replies to each of those further requests. The replies are unordered.

(1) It receives a Reply to Endpoint Message. It is supposed to contain various information about the endpoint, but as of current MIDI-CI v1.2 specification, it only contains the "product instance ID" of the MIDI-CI device.

This ID is usually a serial ID of the device, if applicable. Sometimes more than one devices of the same product are connected, and we often need to identify those different instances of them. Their MUIDs are reset every time the device restarts, so they cannot be used as their identifiers when we need to identify them across the MIDI-CI session.

(2) It receives Reply to Profile Inquiry messages, per address and group. There can be many MIDI-CI Profiles defined on the MIDI-CI device, and they all have to be returned. It is due to a glitch between how MIDI-CI profiles would be defined and how one Reply to Profile Inquiry message contains multiple profiles:

When it is being defined, a profile ID would be first defined and the target channels are specified as its details
When they are represented in the messages, they are grouped, each reply message is bound to at most one "address", by common MIDI-CI message definition

The profiles replies are visible on the other side. ktmidi-ci-tool logging screen looks more verbose on the log record height - it is because the tool also targets mobile screens (Android and iOS) where I want to avoid expanding horizontally:

(3) It receives a Reply to Property Exchange Capabilities message. It contains (a) the number of the number of simultaneous Property Exchange requests supported, and (b) the Property Exchange version (major and minor). (b) needs to be clarified to determine if the Initiator can actually send further PE requests that could be understood by the Responder. (a) also needs to be clarified to limit the number of parallel Get Property Data Inquiries.

Profile Configuration

Profile Configuration is a feature that a MIDI-CI device declares that it supports a set of features that can be enabled or disabled by the connected Initiator.

Profiles are defined either by MMA/AMEI, as a Standard Defined Profile, or by each MIDI device manufacturers. A MIDI-CI Profile ID is a 5-bytes array, and is used to identify them.

According to Common Rules for MIDI-CI Profiles (M2-102-U_v1-1) specification section 2.2, if the first byte is 7Eh then it is a Standard Defined Prpfile. There is only one Standard Defined Profile as of 2023 though: Default Control Change Mapping Profile specification (M2-113-UM_1-0). For MIDI-CI tools, it is not very important specification (take it just an example profile). It should also be noted that the last byte is used to identify the "Profile Level" which has the fixed set of values, according to the specification, section 3.2.

Defining a Profile and defining the set of Profiles on an Endpoint are different work, but defining a Profile in terms of Profile Configuration is not very different from defining the Profile set. You specify a Profile ID and its target (available) channels. That's all. You cannot even name a profile. Here is how you define the set of Profiles on JUCE CapabilityInquiryDemo, on "Local Configuration" tab:

A Profile is defined and applied per channel basis, often on all channels in a Group, or even all channels in all Groups in a Function Block. The term Group is a MIDI 2.0 concepts that expands total number of channels from 16 to 256 - there can be 16 groups on a MIDI 2.0 device, and each one groups 16 channels. A Function Block consists of one or more Groups that acta as an Endpoint.

This explains about 50% of the Channel/Group diagram in the screenshot above - each Profile needs to be enabled on particular channels. Each row represents a Group, and the right-end one represents the Group. Those "active" boxes, including the large "Block" bar at the bottom, indicate where they are registered. If you want to register a Profile target channel but want to have it disabled by default, switch to "Show Active Channels" mode and click "Toggle Member Channels". This corresponds to "Set Profile Off" operation from the Initiator, in case you are familiar with messaging.

Every MIDI-CI message comes with the common message header part (juce_midi_ci calls it "message header", and ktmidi-ci calls it "common" to avoid confusion with Property header) and it contains an address field, which is to indicate the channel being addressed. For Profile related messages, the channel part I explained above is set to this "address" field. The address field is one-byte, and unlike channel part in a MIDI channel voice message, 00h-0Fh indicates the respective channel, or 7Eh indicates the entire Group, or 7Fh indicates the whole Function Block.

(Personally I have been playing around it and have some extraneous Profile entries on non-zero Groups, but I do not recommend that over MIDI 1.0 connections, as MIDI 1.0 cannot handle Groups and JUCE CapabilityInquiryDemo sends messages related to those Groups without specifing it anyway, causing various weird-looking problems.)

ktmidi-ci-tool has quite different UI for defining the Profile set:

It has the list of Profiles on the left side, with add/edit controls, but it does not show that nicely drawn set of channel/Group boxes that JUCE has. Instead, it has boring per-line Profile addressing definitions. The pseudo "Ch.Grp." column has a dropdown list that can choose 0-15 channel, or "Group" or "Function Block". It's easy to enable and disable the active Profiles there. There is no "delete" button on the Profiles list - if all those profile address lines are gone then it vanishes too. It follows the way how Profile Configuration messaging works.

(The "+" button the Profile addressing list is disabled. Clicking the button adds a new entry whose target address defaults to "Function Block". If there is already a definition for Function Block like on this screen shot, then it avoid duplicates.)

When you connect to a MIDI-CI device that supports Prrofile Configuration on JUCE CapabilityInquiryDemo, the client side looks like this:

On the client side, the channel/Group diagram is looking simpler, as there is no need to toggle "definitions". It only shows the defined Profiles that are sent back from the Responder as a set of Reply To Profile Inquiry messages, rearranged to fit with the diagram. From there, you can toggle "enablement" (this term is also used in juce_midi_ci module code) on each address (channel/Group).

ktmidi-ci-tool UI is similar to how we define local Profiles:

It has an additional feature "Details Inquiry", which you can send a Profile Details Inquiry over the specified channel, with the "target byte". A Profile Details Inquiry is a message to the Endpoint to give back more details about the Profile, identified by a byte key (0-127). The reply is not shown on the app UI itself, but the Reply To Profile Details Inquiry message will be on the Logging screen.

The Profile Details target is defined as either a Registered Target (00h-3Fh) or Profile Specific Target (40h-7Fh). The Registered Targets are to be defined by MMA/AMEI specifications. Having it aside "Profile Specific" means, they are common to any Profile. There is already one definition in Common Rules for MIDI-CI Profile specification - 00h is defined as "Number of MIDI Channels". It is kind of related to the concept of "multi-channel Profiles" defined in Common Rules for MIDI-CI Profiles specification, but this concept is not very well tied to the messages that MIDI-CI tools deal with for now, so I'm skipping it. If you are curious, read that specification, namely section 2.3.

Property Exchange

Property Exchange is a feature that a MIDI-CI device declares a set of "resources" (take it as "properties") in MIDI-CI manner, and let clients get and set the resource data, from primitive bytes up to JSON based object manipulation, in either the "full" content, or "partial" portions of it.

Property Exchange in MIDI-CI specification v1.2 itself is type-system agnostic and does not mention JSON, nor even metadata i.e. it does not indicate how properties are defined. The specification states that every Property Exchange messages consists of a header and body, and properties can be "get", "set", "subscribed", and notified.

Common Rules for PE

But apparently, that's not enough. For a property and/or type system that makes sense, metadata system is an essential part. Apart from MIDI-CI specification itself, there is another specification that defines it: Common Rules for MIDI-CI Property Exchange (I will mention it as Common Rules for PE). According to the Common Rules, properties can be retrieved in its own JSON format, and optionally be formalized using JSON Schema (not mandatory). If you request the Responder for its ResourceList resource (property) using Get Property Data message, the Responder will return a Reply To Get Property Data message that contains a JSON array of those defined resources (properties).

Why is there a separate specification? There must be some reasons, but it is worth mentioning that unlike MIDI-CI itself Common Rules for PE is not something everyone would agree. MIDI and MIDI-CI themselves are realtime ready (as MIDI-CI is just on top of SysEx), but JSON based metadata system isn't. JSON is string based data format, unlike binary based message packets such as Protocol Buffer which can statically resolve all types and members. You can still build your own formats apart from the Common Rules, but since it is not "common", other parties would not understand your format.

It should be also noted that Property Exchange itself is actually type-system agnostic, so MIDI-CI app developers can still Get and Set property data without knowing which properties are being manipulated if the some predefined byte chunks are provided as the property headers. No Common Rules are applied in that case, and it's still interoperable.

Having separate specification has another advantage: MIDI-CI specification itself has updated in June 2023, but this Common Rules for PE has unchanged since December 2020. There could be future specifications that could replace this Common Rules i.e. that works with MIDI-CI Property Exchange but provides better and potentially realtime safe metadata system(!).

Defining Properties

To try Property Exchange messaging, you would first need to define a set of properties on the Responder. If the responder is JUCE CapabilityInquiryDemo, then switch to "Local Configuration" tab and scroll down.

There is this "Properties" section down next to the Profiles. Enable it by the checkbox, and go to the bottom of the tab page to find "+" and "-" buttons. Then click "+" to add a new property.

It is at an invalid state. There is a blank-looking line on the left-bottom box, but it actually works as the property list. It looks empty because the "Name" field is left blank. Select the textbox and give a name.

At this state, I would mention that Common Rules for PE states that an end-user-defined resource name should begin with "X-" as non-X- resources are reserved by MMA/AMEI. That may sound quite arrogant and you might want to ignore, but some tools such as MIDI 2.0 Workbench reports an error if such a non-X- resource is "settable" (canSet != none).

In any case, once you give a property name, it becomes valid state. You can continue to customize the rest of the properties, but if you have a look at the property details, you would not understand what they are:

Is it understandable on ktmidi-ci-tool? Not really either:

Looks like everything is just counter-intuitive. It is. So far, Skip these details and only make these changes:

On "canSet", select "partial" (there are "none", "full" and "partial")
Check "canSubscribe"
If you are using ktmidi-ci-tool, click "Update Metadata" to reflect the changes

Your Initiators can then make changes to the property value (canSet) and observe the property value changes (canSubscribe).

The next thing is to set the property "value" locally. On JUCE CapabilityInquiryDemo, there is a "Value" tab on the property details:

You can set the property value here, but not on the text edit field directly(!). You can update the value only by uploading a file. Click "Set Full...". It shows an open file dialog, and if you pick a file, then its content is set as the property value. At this state you want to use only JSON file. Maybe in the real world there would be no chance to edit the text content directly and this UI makes sense, but you might find it absurd especially when you are working on your own product and want intuitive manipulation experience...

Property value editor on ktmidi-ci-tool looks different. If you click the "edit" checkbox, it shows some additional buttons etc.

There is "Set value by file (choose)" button which just works like JUCE CapabilityInquiryDemo. In addition, it supports in-place text editor IF the property media type is application/json. Ideally there should be some JSON validator there...

Manipulating Remote Properties

OK, now that we have defined local properties, it's time to try connecting from another tool. This is what happens when ktmidi-ci-tool connects to CapabilityInquiryDemo and sends Discovery:

These tools turn your local property definitions into the "ResourceList" meta property that MIDI-CI Property Exchange implementations (based on Common Rules) will query in the first place. In this case, after the first series of messaging that began with Discovery is done, the Initiator ktmidi-ci-tool sends Get Property Data message to the Responder CapabilityInquiryDemo, for "ResourceList" resource.

You can check how things work on the "Logging" tab on CapabilityInquiryDemo (from Responder's point of view), or "Logs" tab on ktmidi-ci-tool (from Initiator's point of view).

On the other hand, this is what happens if CapabilityInquiryDemo acts as the Initiator that connects to ktmidi-ci-tool.

From this CapabilityInquiry Discovery tab, we can update the property value as a client. The UI looks similar to what we did as a local property.

We have different set of controls though. First, there are "Get" and "Subscribe" buttons. "Get" can be used to retrieve the value, and "Subscribe" registers this client as a subscriber. When we register a client to a Responder as a Subscriber, it will receive property change notification messages until the subscription ends. You can experience it by following these steps below:

On CapabilityInquiryDemo, click "Subscribe" button
On ktmidi-ci-tool, select the property, click "edit" checkbox, enter some number or any string that works as a valid JSON (e.g. "a quoted string", or {}), and click "Commit changes"
Go back to CapabilityInquiryDemo and confirm that the property value text box now has the same value you entered above.

Partial property updates

The "Set Full..." button and "Set Partial..." buttons do the same job as the ones at "Local Configuration" tab. I haven't explained "Partial" updates yet. In MIDI-CI Property Exchange there are "full" and "partial" updates. On full updates, Initiator sends the whole property value to the Responder.

On partial updates, Initiator sends a set of diffs to the value what Responder currently holds. The diffs consist of a JSON object key-value pairs, whose key represents a JSON Pointer and the value is the replacement value. The target property value should be, therefore, a JSON object (otherwise those JSON Pointers won't simply match).

Here is an example by screenshot of partial updates, from ktmidi-ci-tool to CapabilityInquiryDemo:

If ktmidi-ci-tool detects that the remote property supports "partial" updates by its canSet metadata field, it shows an extra textbox for RFC 6901 Pointer and value in a JSON object. When we perform "Commit changes", the Initiator sends only the text in that textbox. And the Responder performs "patching" to the existing JSON value. The resulting JSON looks like this:

The value looks totally different because all the JSON formatting is gone, but we can see the value specified by the JSON Pointer is now replaced with 1.

Partial updates are useful when the resource is huge (like, a huge set of parameters we have as an AudioProcessorValueTreeState, though PE is not realtime safe) and we don't want to send the entire data all the time.

When it comes to JUCE CapabilityInquiryDemo as the Initiator, there is "Set Partial..." button, and it works just like "Set Full..." button, except that the file content is supposed to be the JSON object that contains the diffs based on JSON Pointer. It is not a great UX, in case we care this minority...

Miscellaneous Property Exchange features

Apart from those features I explained above, there are still couple of unmentioned features:

(1) Content encoding: the message body parts can be "encoded" other than ASCII string. A mutualEncoding header field can be used to indicate which encoding the body uses:

ASCII string. It is the default. Note that unlike typical environment, it is NOT UTF-8. UTF-8 strings cannot be simply put in MIDI 1.0 compatible SysEx 7-bit buffer, so any non-ASCII characters and control characters must be converted to \uXXXX form.
Mcoded7: for non-string binaries, they have to be converted from 8-bit BLOBs to 7-bit BLOBs by applying Mcoded7 encoding. The concept of Mcoded7 had existed since MIDI 1.0 age and it is documented in the Common Rules for PE specification so I do not spend my words here.
zlib+Mcoded7: after Mcoded7 conversion, it tends to become bloated, so we can optionally apply zlib Deflate compression.

The last zlib compression is something we (I) could not verify interoperability. ktmidi-ci-tool runs into somewhat problematic situation here. There is no built-in zlib / Deflate implementation in Kotlin common API until ktor-io 3.0 lands. As a workaround, we have expect functions that are implemented only on Kotlin/JVM and Kotlin/Android (using java.util.zip API). It is known that JUCE CapabilityInquiryDemo does not work this API though. And MIDI 2.0 Workbench has a blocking issue that prevents Property Exchange testing so far (neither of JUCE CapabilityInquiryDemo nor ktmidi-ci-tool can be examined).

Acceptable encodings can be defined for each property metadata and the definition is a string array. ktmidi-ci-tool UI is designed based on that. However Current Common Rules for PE specification only accepts these 3 encodings. On JUCE CapabilityInquiryDemo, there are 3 checkboxes to choose whether each of those encodings can be used or not. It probably makes good sense (at least it is easier to choose them just by clicks).

When sending and receiving Property Exchange messages with body encoded with one of the above, we use an optional mutualEncoding header field to specify it. Then the message recipient should be able to decode it accordingly. On ktmidi-ci-tool, there is an extra dropdown menu that you can indicate which mutualEncoding to use on the Inquiry explicitly. CapabilityInquiryDemo seems to pick one of those supported encodings by its own. I prefer my choice of making it explicit (it is optional), as what we want is a development testing helper that lets us flexibly diagnose any problem.

(2) Pagination: if a resource only returns a JSON array, its items can be "paginated" i.e. queried partially. A Get Property Data Inquiry message may have offset (from which item) and limit (max number of the returned items) fields in the header, and the Reply comes with totalCount field in the header. ktmidi-ci-tool could tell these fields on the UI:

To examine pagination feature, there are some extra tasks: you have to ensure that the property is set canPaginate metadata value as true. If your Responder is JUCE CapabilityInquiryDemo, you can enable it only if the property is named ending with List (therefore, this example screenshot says the property is X-bazList). Then you can check canPaginate option there.

ktmidi-ci-tool shows these text input fields on the "Paginate?" row if and only if the property has canPaginate value as true. The query result is shown on the UI as if the entire result were the value on the text field. (We could show offset, limit and totalCount somewhere, but it is not implemented yet...)

Process Inquiry

Process Inquiry is a feature that a MIDI-CI device can provide current state of the device. In MIDI-CI v1.2 specification, the only feature it covers is MIDI Message Report. It can be used to provide the state of MIDI device, such as note on state, pitchbend state, controller state, program/bank state, etc., like MIDI bulk dump on a device.

If I were the specification author, I would name it "Device State Inquiry". Why "Process" Inquiry? MIDI 2.0 used to feature "three Ps" which were Protocol Negotiation, Profile Configuration, and Property Exchange. As of version 1.2, MMA had to discard one P: Protocol Negotiation. So they would like another P to complement the lost one. That's all my guess.

In any case, juce_midi_ci does not support Process Inquiry yet. So if we want to test MIDI Message Report interoperability, there are only two I am aware of: ktmidi-ci-tool and MIDI 2.0 Workbench. It does not work as a Responder, but it is not a big deal because when we implement this feature, we mostly care about the receiver part.

An interesting defect by design on MIDI Message Report from my point of view is that it seems impossible for a MIDI-CI Initiator to concurrently wait for MIDI Message Reports from more than one MIDI-CI Responders. It is because this bulky operation comes with no "session" MUIDs except for the first Reply To MIDI Message Report message and the last End Of MIDI Message Report message. Everything inbetween is mere MIDI message such as Note On, Control Change, Program Change, etc. are just MIDI channel voice messages or system common messages that does not convey MIDI-CI MUID. If MIDI Message Report packets from Endpoint #1 and Endpoint #2 are mixed, then we cannot identify which Endpoint it sends the channel voice messages inbetween. Thus, a MIDI-CI Initiator needs to block incoming other MIDI channel voice messages and system common messages until the Responder finishes its MIDI Message Report session.

In any case, we can see how MIDI Message Report works, by using MIDI 2.0 Workbench as the Initiator. Connect to ktmidi-ci-tool and open up the device window (due to some issue, I needed to open, close and open the window again to get "Process Inquiry" part correctly shown - probably the tabs don't get refreshed when the app received Reply To Process Inquiry Capabilities message), and go to "MIDI Message Report" submenu. Then click "Get MIDI Report" button:

The ktmidi-ci-tool logs would show you what kind of messages were received and sent through the session.

Those "[sent MIDI Message Report..." log lines look too simple, but if you scroll down each of those text fields you can see the actual MIDI bytes in the message. Since they can be thousands of log lines if I do not assemble them into a few chunks, those MIDI Message Report bytes are collected per a channel, per request block. It was only for Channel 1 so there were only 3 chunks, but if I choose "Port" at "Channel" on MIDI 2.0 Workbench, it can be a lot more.

ktmidi-ci-tool implements this feature by leveraging ktmidi Midi1Machine class (and it will enhance it with Midi2Machine when we add UMP support to ktmidi-ci) - this class existed to preserve any internal state of a virtual "MIDI device" that holds these information items MIDI Message Report are supposed to contain. I implemented this class when I was creating a visual MIDI player which existed only on C# version of the library... The class was ported to Kotlin too, but there was no actual use until this time.

Although it is not a perfect match - MIDI Message Report feature offers an option to indicate whether it should send literally all the state or only those what were set to non-default. Midi1Machine does not really preserve the state whether it has changed or not, so the internal structure will have to be changed.

Summary

We have run through all the features currently available MIDI-CI tools offer. They would be helpful when you are building your MIDI 2.0 synths, effectors, or whatever your code that integrates MIDI-CI features.

ktmidi-ci-tool: MIDI-CI controller tool for desktop, Android, and Web

Fri, 26 Jan 2024 20:00:00 +0900

ktmidi-ci-tool: MIDI-CI controller tool for desktop, Android, and Web

Today I'm announcing a new pet project ktmidi-ci-tool, a new experimental MIDI-CI controller tool. It is part of my ktmidi project which just released the latest version 0.7.0, written in Kotlin, and runs on desktop (Linux / Mac / Windows), Android, and Web browsers (thanks to the latest Compose Multiplatform on Kotlin/Wasm), also potentially on iOS (no MIDI device access yet).

For Android, I published an open testing version on Google Play: https://play.google.com/store/apps/details?id=dev.atsushieno.ktmidi.citool

For Web, you can find the actual running version here: https://androidaudioplugin.web.app/misc/ktmidi-ci-tool-wasm-first-preview/

(I can build and run the app on iOS Simulator too, but currently we have no working Kotlin/Native MIDI device access API implementation on iOS yet.)

Since MIDI-CI feature is not very well known and the entire UI is not very intuitive per se, I have written another dedicated article on what the tools feature, interoperating with similar MIDI-CI tool in JUCE: CapabilityInquiryDemo: https://atsushieno.github.io/2024/01/26/midi-ci-tools.html

MIDI 2.0 in 2023, observed

MIDI 2.0 was released back in 2020, but its adoption was, well, Largo. It seemed like chicken and egg situation: no DAWs would support MIDI 2.0 inputs if there is no MIDI2-ready keyboards, and now hardware vendors would have built ones if there is no DAW that supports MIDI2 inputs. A typical interoperability situation.

The situation has somewhat changed in 2023 when MMA released updated specifications (here we call it June 2023 Updates), and there are some development activities emerged afterwards:

Roland finally supported MIDI 2.0 on their A-88MkII (which had long stated that "it will support MIDI 2.0" since the specification release in 2020). By supporting MIDI2, it receives and sends finer velocity than 128 levels, natively.
KORG Keystage supports MIDI 2.0 with Property Exchange, along with some plugin that supports those messages
ALSA project supports MIDI 2.0 UMP ports since Linux Kernel 6.5 and alsa-lib 1.2.10
libremidi, a modern C++ cross-platform MIDI library, started to support MIDI 2.0 since v4.0
JUCE since 7.0.9 (released in November 2023) supports MIDI-CI in the new juce_midi_ci module

And in prior to those, I should mention that, Apple CoreMIDI and AudioUnit supported MIDI2, Google supported USB MIDI 2.0 on Android 13, CLAP supported MIDI2 as one of event types since 1.0 (although it discourages use of MIDI2 in favor of its own event type...), etc.

ktmidi, AAP, and MIDI 2.0 interoperability until 2023

My take on MIDI 2.0 was different, in the first place. MIDI 2.0 UMPs should be quite useful as better interoperable music sequence structure. There was no SMF 2.0 (there still isn't as an SMF replacement), but the basic data model is there. We made sure that there are no Control Changes beyond 127 even in MIDI 2.0 (compared to, for example, SFZ format expanded them). I support the idea behind UMP that respects backward compatible translators as much as possible.

My ktmidi library started as a new cross-platform MIDI library for Kotlin (as a successor of C# version of it, managed-midi), which covered only MIDI 1.0 but soon expanded its potential to MIDI 2.0 UMPs. It has been useful for me to generate some music information abstractions (namely I had been authoring my own Music Macro Languages translated to modern Tracktion Engine sequencer).

Also since I wanted to incorporate MIDI 2.0 support in Android audio plugins (AAP), I have created another C library called cmidi2 back in 2020 and added MIDI 2.0 extension feature in AAP. It was done apart from ktmidi (as audio plugins need realtime safety).

Determine whether MIDI-CI is useful and makes sense or not

As MIDI 2.0 related development became active among many manufacturers and developers, connectivity is getting a real chance as well as a concern. As an audio plugin format developer, I see it as an opportunity that we may be able to have MIDI 2.0 devices connected and controlling our plugins directly through the MIDI protocols. To achieve that, UMP support is not enough. We will have to support MIDI-CI.

(Note that we do NOT really need MIDI-CI if we just want to transmit UMPs without letting the sender or the receiver understand what your MIDI device is like. There is no protocol negotiation anymore. Your user simply connects your devices just as UMP transport based MIDI devices.)

I don't know what is the driving motivation, but JUCE has added juce_midi_ci module a few months ago. It would similarly give more enhancements to JUCE plugins. Audio plugins that are made with JUCE would expose some configuration points via Profile Configuration and Parameters via Parameter Exchange, in the standard MIDI-CI manner. Apple CoreMIDI supports MIDI-CI Device Discovery and Profile Configuration, so AudioUnit might benefit more from the platform integration later and JUCE plugins, when building AUv3 plugins, might fit there well.

Note that "parameter" in MIDI-CI Parameter Exchange does not really map to audio plugin parameters. MIDI-CI is NOT suitable for realtime processing, as it involves a lot of JSON parsing and generation, at least under the Common Rules for Property Exchange specification. You want to make use of something else like MIDI 2.0 channel voice messages like Assignable Controllers (or Per-Note version of them) in UMP, with some MIDI mappings. AAP supports parameters based on that approach, with some mapping configuration options (to avoid some kind of pitfalls).

What MIDI-CI offers

If its "parameters" are not audio plugin parameters, is that even useful? MIDI-CI, as of v1.2 specification, gives opportunity for your MIDI devices to offer control points over:

Profiles: you can either define your profiles (or third-party profiles), or declare that your device conforms to "MMA/AMEI defined profiles"
- Each of them is an unnamed feature set that your MIDI controller can assume your device conforms to a common behavior e.g. "its channel x, y, and z are usable", "the channel accepts CC number x, y, and z", etc.
- they can be enabled and disabled, per channel, per group, or all (the entire "function block")
Property Exchange: you can define what kind of "parameters" your MIDI device provides
- The parameter getter and setter comes with a header and a body parts
- There is "Common Rules for Parameter Exchange" specification apart from MIDI-CI specification itself, and MIDI-CI implementations would mostly conform to that, to achieve basic interoperability.
- The parameter body part is BLOBs i.e. untyped, unless you give some JSON Schema along with the Common Rules.
- In the Common Rules, identifying the target parameter is done by name string, which means, it is not suitable for realtime processing (avoid string comparison; or in other words, why LV2 URID matters).
- value can be partially updated (i.e. parts of the entire JSON value tree)
Process Inquiry and namely MIDI Message Report: offers the standard way for "bulk dump", kind of.
- The interchanged messages involve non-SysEx MIDI messages
- The dump is most likely bulky so it is very unlikely useful for realtime processing
- It is one way; no setters

Property Exchange at this state is most likely useful to offer named configuration points which can be formed in JSON tree structure.

Does that make sense to your MIDI product? If no, MIDI-CI is not for you and you can skip that.

Supplemental

MIDI protocols and MIDI transport protocols

The MIDI 2.0 support development around ktmidi was, therefore, without caring interoperability with others much. It was mostly because it was unclear for me how MIDI 2.0 things should work, up to the underlying platforms. MIDI-CI specification used to have Protocol Negotiation as one of the three core concepts (they call it three-Ps along with Profile configration and Propety exchange) and it looked like we could switch MIDI 1.0 and 2.0 protocols, different transports (MIDI 1.0 bytestream and UMPs), over the same underlying (platform-level) transport.

This was not always doable. For example, Web MIDI API implementation inspects the bytestream to see if it contains System Exclusives and rejects them if the web session is not approved the SysEx messaging permission. UMPs are totally alien to them. Google Chrome (as of v120) reports a failure if it starts with <80h byte (treating it as running status, and reports that it is unexpected), regardless of whether we permit SysEx on the webpage.

The modern consensus on MIDI transports after June 2023 Updates seems that we define a transport with a fixed transport protocol (either MIDI 1.0 bytestream or UMP) in prior, and it is not assumed to be promotable. (Note that the MIDI protocol within UMP transport protocol can be still promoted, by the new UMP Stream Configuration Request message.)

I have some more words on that matter, but it was mostly described on another post.

Upgrading MIDI-CI implementation for June 2023 Updates

Late last November, I was writing a blog post about juce_midi_ci (among Japanese "advent calendar" culture), and to investigate what it really is, I ended up rewriting my existing MIDI-CI implementation in ktmidi for experiment. Surprisingly(?), I had already implemented basic agent model as MidiCIInitiator and MidiCIResponder that gets connected each other, back in 2022 (in addition to MIDI-CI SysEx packet factory class written in 2021). But it was written when there was no June 2023 Updates.

It was a total rewrite. MIDI-CI specification has changed a lot at v1.2, the June 2023 Updates. The core of the specification was "three Ps" a.k.a Protocol Negotiation, Profile Configuration, and Property Exchange, but the entire Protocol Negotiation got deprecated(!). My implementation was only about connections (because its protocol negotiation was my only interest), so only the MIDI-CI SysEx factory classes remain useful.

If you have ever learned anything about MIDI-CI protocol establishment, forget about them all. MIDI-CI is based on MIDI 1.0-compatible SysEx, and (based on the modern consensus I mentioned above) there is no chance for MIDI 1.0 devices to get Protocol Negotiation working. The application-level MIDI protocol promotions can occur only in UMP land, and thus MMA moved the protocol negotiation part to the UMP specification (UMP Stream Configuration message).

Achieving realtime safety on inter-process audio plugin extension foundation

Sun, 24 Dec 2023 23:45:00 +0900

Achieving realtime safety on inter-process audio plugin extension foundation

I have been exploring audio plugins opportunity and possibilities on Android which ends up my AAP project. My primary goal these months is realtime safety in extension foundation, which was finalized in the upcoming aap-core v0.8.0 release.

There should be some explanation on what that means.

Process isolation and realtime safety

AAP (Android Audio Plugin) is unique in that audio plugins cannot be loaded as an in-process dynamic library within the host DAW. When an AAP DAW "loads" a plugin, the AAP lives in its own process, so as the DAW in its own process. Your app cannot load native executable code that was provided by another app (it seems technically you can, but Google Play Store will kick you out).

DAW and plugin needs to process audio and MIDI-like event stream together i.e. the DAW passes the plugin audio buffers and event buffers. As AAP plugins and hosts reside in their respective process spaces, we use IPC (inter-process communication) between the processes, with shared memory buffers (instead of embedding audio data buffers within the message, as they are not small and processed very frequently). It is usually a big disadvantage for audio processing; audio needs to be processed in realtime manner, but IPC foundation usually involves mutex locks that violate realtime safety.

AAP uses Android Binder IPC framework in C++ (NdkBinder) which is minSdk = 29 since the beginning in 2019. Binder is designed to be realtime ready, implemented in the underlying Linux kernel (in its fork in the beginning of Android OS, and merged to the mainline kernel later). One thing to note is that while Binder itself is realtime ready, the realtime binder capability is not openly available to us Android app developers. But that is another story, we ignore that so far. What is important here is it is technically capable.

Android Binder is usually used with some interface descriptions called AIDL, and we do have one for AAP. It defines its interface like this (excerpt):

interface AudioPluginInterface {
	int beginCreate(String pluginId, int sampleRate);
	void endCreate(int instanceID);
	void beginPrepare(int instanceID);
	void prepareMemory(int instanceID, int shmFDIndex, in ParcelFileDescriptor sharedMemoryFD);
	void endPrepare(int instanceID, int frameCount);
	void activate(int instanceID);
	void process(int instanceID, int frameCount, int timeoutInNanoseconds);
	void deactivate(int instanceID);
	void destroy(int instanceID);
}

If you are familiar with any of audio plugin APIs (VST, AU, LV2, CLAP, etc.), you would find it quite similar to existing ones.

Plugin extension foundation and extensibility service

AAP is designed to be extensible. An extension in an audio plugin format usually means an optional feature that plugins and hosts can use or ignore, so that the core plugin API itself does not have to introduce breaking changes. Audio plugin features are often designed to be extensible such as VST-MA (module architecture), LV2 extension APIs. AAP follows the same pattern. AAP has parameters, state, presets, MIDI, and GUI extensions, to name some.

In general audio plugin formats that support extensibility, plugin extensions usually come up with functions, and they are supposed to be invoked directly in the DAW code. However, as I explained earlier, Android DAW process and plugin process are separate, so we use Binder IPC here too. AudioPluginInterface interface above actually has some additional methods:

	void addExtension(int instanceID, String uri, in ParcelFileDescriptor sharedMemoryFD, int size);
  void extension(int instanceID, String uri, int opcode);

Unlike typical plugin format APIs, AAP extensions cannot be simply achieved by defining their own AIDL. Technically we can ask DAW and plugin developers to instantiate their own Binder objects for every AIDL for respective extension API, but dealing with such multiple Binder connections is quite cumbersome and also wasteful on device resources (memory, threads, shared memory handles, etc.). Also, it is not very encouraging for DAW and plugin developers to write their code if their plugins or DAWs have to deal with this IPC foundation directly.

To sort out the situation, AAP has the extension foundation in different way: each extension API designer / developer also offers "extensibility service" implementation that handles binary serialization and deserialization of their IPC requests and replies, and DAW developers and plugin developers just deal with the extension "proxies" that AAP framework (or hosting reference implementation I would say) offers. It's similar to GRPC on top of Protocol Buffers, but runs on top of Binder. It is how extension() function works. To make every extension invocation typeless, their arguments and return values are serialized in the shared memory.

This way, DAW and plugin developers have to deal only with the plugin API, without needing to deal with Binder IPC by themselves. The extension API developer needs to deal with the IPC parts, but AAP v0.8.0 provides some public API facade that they can gain access to implement their requests and replies. We call it AAPXS (AAP eXtensibility Service).

Plugin extension foundation and realtime safety

Until AAP v0.7.7, it was the only way how we invoke extension functions over the IPC. However, there was a problem.

Many of those extension functions do not have to be processed in realtime manner, but some extension functions do. For example, if your plugin format handles parameter changes, you would most likely need the changes applied within certain audio processing cycle rather than any random point of time.

Binder itself is realtime safe in theory, but (even if we have access to realtime Binder) the Binder needs to be occupied only by RT-safe audio loop. If we use the one and only Binder connection for audio processing purpose i.e. process() function in the AIDL above, we should not try to use the same Binder for potentially non-RT safe uses.

If we only support extension functions via Binder, they could be always invoked via extension() and those non-RT-safe functions will block. There will be only one invocation on one Binder connection, so simultaneous process() call won't happen. And considering that there should not be multiple RT-ready Binder threads running IF it were possible, no Binder method invocation other than process() should happen anyways.

But is it possible to pass extension invocations at process() ? Certainly - it is how AAP v0.7.8 introduced extension controller MIDI 2.0 SysEx8 messages. As a quick background, AAP makes full use of MIDI 2.0 UMPs as its event messages. Those note-on, note-off, control changes, program changes, etc. are directly passed to the plugin. The extension requests and replies are serialized to mere binary chunks, so it is fairly straightforward to reuse the binary data into UMP packets.

Since AAP v0.7.8, we don't invoke extension() method when the plugin is in "active" == realtime mode, and send AAPXS SysEx8 messages on the MIDI2 port instead.

Async-ify plugin API or not

With AAPXS SysEx8, it becames possible to invoke extension functions within realtime processing pipeline. It is good for RT-safe extension functions. But how about non-RT-safe functions? We also need to ensure that those non-RT-safe functions do not block audio processing.

Here we need some asynchronous dispatcher that forwards the non-RT request invocation to non-RT thread. This also means, the extension function does not return within the audio processing in the process() cycle and thus the extension reply needs to be sent back to the host in another audio process() function. Every AAPXS request is assigned a request ID and the reply to that message has the corresponding request ID.

Data passing wise, the MIDI2 messaging works perfectly. AAP plugin reads through the MIDI2 input messages, detects and interprets AAPXS SysEx8 messages, and dispatches the extension request to the AAPXS "request handler" implementation that the AAPXS developer provides.

Function signature wise, it is not easy to determine what to do. In typical audio plugin format APIs, extension functions are designed to be synchronous e.g. "getState()", "setState()", "getPresetName()", "setParameter()" etc. If those extension functions should work asynchronously to work along with the AAPXS, they should become different, like a pair of "beginGetState()" and "endGetState()", or those functions with "callback" argument.

(To preserve ABI stability, we have to define extension API in C, not C++. Therefore there won't be types like std::function or std::future in the API.)

Alternatively, we could differentiate plugin extension API and AAPXS API, but that means we give up simple proxying of extension API to AAPXS. We could do that, but it is not what we have so far.

Resident MIDI Keyboard and compose-audio-controls

Mon, 07 Aug 2023 00:00:00 +0900

Resident MIDI Keyboard and compose-audio-controls

I have been quiet on this blog, but I have been making some progress in my AAP (Audio Plugins For Android) land. It's not going to be the primary topic today but it's the driving engine behind it. Today I'm introducing Resident MIDI Keyboard and compose-audio-controls.

For Resident MIDI Keyboard, there is a few more links to some relevant pages that I had worked on these days (which I usually don't...):

Designing audio plugin dogfooding GUI

These days I have been exploring ways to implement new audio plugin GUI stuff for AAP, for both host side and plugin side.

Current AAP plugins come with some inflexible dogfooding page on its main Activity, which is actually a host - it was okay for the first step but it's super hacky and quite far from useful nowadays (it shows knobs and sliders together unnecessarily, but skip that so far, it's all experimental):

I'm not going into its details this time (it's dying anyways). For better dogfooding plugins, I also ported JUCE AudioPluginHost to which many of audio app developers would point, in very early stage of AAP development, but it is not designed to be useful on Android (no wonder, Android has no audio plugin basis). I ended up creating another simple mobile-ready version of audio plugin host in JUCE, which looks like this:

Since it uses JUCE it must be a GPLv3 application, so we cannot really bundle it into all audio plugins even optional, by default. The GUI should be offered in Kotlin anyways so that it can be easily used by non-native host app developers. And plugin developers should not have to worry about the default UI.

At the same time, our plugin format started to support native GUI, which means, GUI that can be controlled on the host (DAW) side but still managed by the plugin service. This is the current state of "plugin native UI (Jetpack Compose) shown on a host in C++ code" - in this case, Helio Workstation (JUCE based DAW) on Android:

The UI above contains the new stuff I built in these months aka. compose-audio-controls.

compose-audio-controls: audio controls for music apps

The new audio plugin UI will be running on mobiles. Therefore, those plugin UIs should be designed to be very compact. Apple AUv3 supports compact UI, and it seems widely used in their new Logic Pro for iPad (released in May, 2023), on Plug-in Tiles UI. We should achieve something similar.

ImageStripKnob

One thing I wanted to avoid when designing our default native UI is Jetpack Compose sliders - they are designed for mobiles, but it takes too much space. We have a lot of controllers, if not to show everything (in AUv3 you can pick up which parameters you put on the UI as controllers). I concluded that a knob would work better - it does not need horizontal or vertical space, and we can customize the degree of precision.

There are people who had implemented knobs in Jetpack Compose, but I had something different in mind - there is already a bunch of knob resources called Knob Gallery that you can apply to your music apps as well as instruments such as Kontakt instruments or SFZ ARIA extensions, just by reusing those image strips. There is no complicated mechanisms required in code. We present the exact section of the looooooong image for the value on the knob. I have implemented ImageStripKnob based on this image strip format.

There is already webaudio-controls that applied that knob image format (also from the developer of that Knob Gallery), but since it assumes desktop PC keyboard e.g. shift + mouse drag for fine-tuned controlling, I needed different logic. Compose ImageStripKnob switches to "fine mode" after holding the finger (pointer) for 1000 milliseconds.

We are serious about how to control the knob. There are some principles that we should keep in mind:

We are not with mouse. Our touch region should be 48x48 at minimum (as per Android Accessibility Help states.
Our finger hides the knob, so we have to offer some way to let user see the value.
We need simple way to control. No realism in operation is wanted.

For example, when we implement the knob rotation, we could implement it by some detailed touch direction checks, like "How to Make a Draggable Music Knob in Jetpack Compose - Android Studio Tutorial" instructs - while it is educational, I find its "real" motion is unnecessarily complicated. Our fingers are too big to have control space halved (left and right move differently). Another complicated example is pinching knobs - we do not want to have two fingers occupied to control just one knob. We don't need such "realism". Lots of modern knob controllers on PC simply expect drag up and down (or right and left).

It should be also noted that dragging is a sensitive operation. Basically we should not expect users dragging in both horizontal and vertical directions precisely. Therefore, ImageStripKnob does not assign any event on horizontal motion. What we expect for horizontal motion by users instead is, to see current value position of the knob. ImageStripKnob shows value tooltip when it is being dragged, but it may be still hidden by the user's finger. So we want to offer some way for users to "safely" move their finger.

I also thought about the disadvantages of my knob over slider. One immediately found issue was "height" - sliders do not need much height as the circle knob needs. So if we need to compact the overall height then sliders might do better job. For my case, we could simply put two or more knobs in a row, so knobs give more spaces. I also thought about overlaying "fine tuning" small slider over the base slider when the pointer is held for 1sec. before I started working on this knob control, but then I find long-press on a knob would be better, less confusing.

DiatonicKeyboard

Once I was one with the knob for audio plugin parameters, I figured that I need a keyboard control, just like how I added the keyboard to my simple JUCE host app (but in Jetpack Compose). Drawing keyboard is easy, but it needs to be mobile ready. JUCE MidiKeyboardComponent is apparently not (too small). We have to follow the Android Accessibility principle - at least we should try to. Thus, our keyboard is bigger than typical ones.

Then is our key width is 48.dp ? Well, actually no - it is 30.dp. And the black keys are of 24.dp. That sounds weird, but these keys still preserves general controllability for 48x48 square space. When we tap on the keyboard, it actually measures the distance between the tapped point and the "center" of each key around it - both white and black key. And the closest key is picked. This is different from detecting "precise" touch target. Let me rephrase: we are not with mouse. Our finger (pointer) is not very precise. It oes not matter if the touch location is actually over a white key, when there is another closer-looking black key. The way how I picked up the target key may not be the "best", but it is better than merely picking up one by precise pointer location.

Now you would find that the black key likely has 48.dp x 48.dp space. How about white keys? There are only 12.dp extra spaces... Then they key height fills the gap. The white key height is 60.dp, and the black key height is 35.dp. Since the key is chosen based on the distance, tapping near the bottom of the keyboard most likely gives you the expected touch result.

These tight requirement over the control points also brings a design decision - we do not assign any velocity differences over the touch location, like JUCE MidiKeyboardComponent does. It simply violates the Accessibility Help principle again. It is clever on desktop, but not so much on mobile. If you need some dedicated velocity control, that should be given by another slider or knob or whatever.

Dragging over the keyboard brings in another design decision. I implemented it like, both horizontal and vertical changes trigger some value change handler, ranged between -1.0f and 1.0f, and it's up to the app to determine what to assign (if it is not for pitchbend). Jetpack Compose will trigger only horizontal or vertical changes, unless we instruct to report pointer location change (that means both directions), so it worked well. But I figured that people would mostly expect note changes when we drag over multiple keys, instead of control changes, like what JUCE MidiKeyboardComponent or webaudio-keyboard control from webaudio-controls do. Therefore, note-changing behavior is the default, and we optionally support "expression" mode.

Controller change detects horizontal drag, vertical drag, and pressure. I'm not sure how practical the pressure detection is, but some people might appreciate that. Vertical controllers could also bring special use cases - unlike horizontal moves, vertical moves could be used for per-note controllers because those keys run vertically. In DiatonicLiveMidiKeyboard control that I explain later I use the vertical value changes for MIDI 2.0 per-note pitchbend.

All those concepts are implemented as DiatonicKeyboard. Since it is not necessarily tied to MIDI, it could be used for anything - having in mind that it assumes "diatonic" and there are 128 keys at most though (not too flexible for e.g. microtonal notes without diatonic key mappings). And of course, I designed and created it for better MIDI integration support.

Now, there are some extended concepts beyond those basics. One apparently missing thing from above is key ranges - we would apparently need octave settings, so DiatonicKeyboard accepts the octave setting. We need a slider or knob that user can change the octave. We want to have a checkbox to enable or disable the "expression" mode. Therefore, there are some additional controls beyond the keyboard itself. compose-audio-controls library comes up with them, as DiatonicKeyboardWithControllers composable function. It takes 20 parameters that I'm not going to explain here. But I spent a lot of effort to document the API so if you are curious have a look at it.

MIDI Keyboard as MIDI 2.0 controllers in need

One of the cool (IMO) features I implemented in AAP is that any instrument plugins can be easily transformed into virtual MidiDeviceServices i.e. MIDI OUT devices. Since we already have various LV2 and JUCE plugins ported to AAP, we have a bunch of virtual MIDI devices.

Since we do not have a lot of DAWs that could be extended to support AAP (it must be open source, unless I have access to the sources), our uses of AAP is quite limited so far. Exposing AAP via Android MIDI API is an extensible way to achieve that.

As of current version (0.7.7), AAP tightly integrates MIDI 2.0 UMPs as the way to control plugin parameters, since (1) it provides 32-bit data for control changes, per-note assignable controllers, etc. and (2) MIDI message stream could be processed in real-time manner (might not be sample accurate depending on the software stack, but that's relatively a minor concern). Also, (3) UMPs are also being used as ways to transfer AAP extension messages between the host and the plugin because there should be only one audio call (IPC) and return for one plugin processing, and audio processing is the only chance for realtime extension messaging.

Now, going back to the default plugin details Activity - currently there is no dedicated controller for MIDI - it has "MIDI" button (disable on the screenshot above though, as it is an effect plugin) that sends some fixed MIDI 1.0 note ons and note offs after some delay. It is solely done in the client part of Android MIDI API i.e. via MidiManager. The button is obviously uncool - there should be some MIDI keyboard.

Thus, the new plugin details Activity, which I'm calling "plugin manager" now, should contain some MIDI keyboard for dogfooding.

But, AAP is NOT only about MidiDeviceService. Basically what we should test is AAP functionality that can be both parameter changes and MIDI note events, of MIDI 2.0. So I added another keyboard to the new "plugin manager" Activity. And it looked like:

Apparently, there are rooms for improvements...

ResidentMIDIKeyboard

I figured that the generic MidiDeviceService testing part could be "outsourced" from the plugin manager Activity UI. I had been actually using my Android MIDI keyboard app, kmmk, for testing. I moved those MIDI testing functionality into the default plugin Activity only because switching to another application Activity is annoying. But what if I could put some MIDI keyboard as an overlay window and use it? Then I don't have to keep that MidiDeviceService dogfooding part on the UI anymore(!)

That was the beginning of the new MIDI keyboard app called "Resident MIDI Keyboard". It makes use of the "dangerous" System Alert Window that can float around anywhere on the screen, with special user permission on Settings. The app screenshot on Google Play tells that you can use even on the application list on your home screen.

System Alert Window is kind of classic technique to pop up your app window in a way that violates security policy that no app should show its UI over other apps as it could be used for phishing etc. ResidentMIDIKeyboard actually offers a safer option - it provides a service callled MidiKeyboardViewService based on SurfaceControlViewHost. that other apps can use it without referencing any AAR.

You would probably have never heard of SurfaceControlViewHost. It is very unknown feature new in Android 11. This blog post from Mark Murphy is the only material I found so far. I use this feature extensibly in my AAP native UI support (that's how it is possible to embed the plugin UI in Jetpack Compose on juce_gui_basics UI).

I do not recommend to use RMK via SurfaceControlViewHost much though, as there is no type safety and feature versioning, so there may be any sort of protocol mismatches between my latest ResidentMIDIKeyboard version and the version you used when implementing. I could declare an unassured promise that "I won't break the interoperability", but things could happen and I don't trust myself.

RMK as a MIDI input device

Device connection wise, my initial idea was that ResidentMIDIKeyboard (RMK) acts just as a MIDI client so that I can connect to AAP MidiDeviceService while I'm running the same AAP plugin manager Activity. But when I was explaining RMK like "you can run it anywhere e.g. over your favorite DAW", I noticed that this RMK did not really offer what DAW users want - they want a working MIDI keyboard that sends MIDI inputs to the DAW.

Is that possible? It is, if RMK acts as a MidiDeviceService: it can be an input device, while usually MidiDeviceService would be for virtual output devices (I'm speaking in normal audio development term here; in Android MIDI API or javax.sound.midi, they use "sender" and "receiver", and "input" and "output" are kind of flipped). RMK offers its own MIDI input, so you can connect to it from your DAW IF it properly supports Android MIDI API.

There are some DAWs that I could not find any MIDI input support:

Audio Evolution seems to support only USB
Caustic3 seems to support some MIDI inputs but not sure if/where I can choose the device to use

But in general I could confirm that RMK works. To name a few - n-Track DAW:

FL Studio Mobile:

Roland ZenBeats (JUCE does not properly support MIDI outputs, but MIDI inputs work)

Cubasis3 LE - I couldn't find anything that lets me choose the input but it is automatically detected (I could only run trial):

Those screenshots tell that wherever I held on RMK is also at note-on state on the context channel on each DAW.

more details on MIDI 2.0 support

Since the MIDI feature is based on my ktmidi library, it supports MIDI 2.0 UMPs with no surprise. So RMK is also useful to test MIDI 2.0 features too. However, it should be noted that the way how it sends the MIDI device to "switch" the MIDI protocol is not obvious, especially considering that Android MIDI API does not currently support MIDI 2.0 protocol except for USB MIDI 2.0 protocol.

Basically, we send UMPs without any protocol change attempts - it sends UMP Stream Configuration Request message which is newly defined in the latest MIDI 2.0 June 2023 updates. It offers an input virtual port (output in Android MIDI API wording) too, but any input to it is ignored (a standard-conforming recipient would send Stream Configuration Notification message to the port). Also we do not reply any Stream Configuration Notification by the client DAWs. So MIDI2 feature may not work for you, but if the client expects only UMPs, then it would just work. My AAP MidiDeviceServices are implemented to work as such.

Before June 2023 updates, there used to be Protocol Negotiation, which let us send MIDI-CI message for switching to the new protocol. Since it is based on MIDI-CI and the messages are thus simply universal SysEx messages, it is technically possible to send them over MIDI 1.0 transport without hesitate. Though the way how the same transport channel switches the protocol was not obvious - I simply kept sending UMPs after sending it, but with the latest MIDI 2.0 specification it seems that the platform should provide different MIDI I/O connections for 2.0 and 1.0 respectively. Since there is no such mechanism in current Android MIDI API, I keep reusing the same connection, but now that there is no chance for MIDI-CI messaging anymore, the message exchange became brutal compared to before.

Fortunately, UMP Stream Configuration Request begins with F0h which only conflicts with existing SysEx message for some very early-stage vendors (see this list for such ones), there is almost no practical concern (I would start worring about it if Moog is going to support Android).

CC, RPN/NRPN, Per-Note Controllers, Program changes, etc.

RMK comes with a set of controllers below the keyboard. They are useful to test the MIDI features for the recipient (either a DAW or a MIDI out device). The initial selected target is CC, but it can be various MIDI status targets such as CAF, PAF, RPN/NRPN (compound message in MIDI 1.0, single message in MIDI 2.0), Per-Note Controllers, Program changes with bank select (again compound messages in MIDI1, single message in MIDI2), etc.

The idea is simple, but the implementation is complicated, especially in that -

Discrete or not: it depends on the selected item whether sending value changes as user controls the knobs is appropriate or not. For example, sending value changes continuously for like some CCs like pan, expression, volume and pitch bend is good. Sending program change for dialing the knob is not.
Send changes immediately or not: We want to send CC messages only when we change the value, not index. Same goes for RPN/NRPN MSB and LSB - the entire changes should be sent only when value (i.e. DTE in MIDI 1.0) is the target.
We want to know what the actual control for an index is. Therefore, for CC index, show "Expression" instead of "7".
- Maybe same could go for Program Changes, but I'm not very persuaded that a MIDI out is usually a GM synthesizer. They are rather not usually.
We remember what user was controlling on each selected target: RMK remembers last CC target, last RPN/NRPN target, last Program Change, etc. and they are restored when the user is back to the target.

To make it easily possible, RMK makes use of ktmidi Midi1Machine and Midi2Machine API, for each MIDI mode. Those classes work like a virtual MIDI device machines that remembers their virtual register state.

All those MIDI support features on top of DiatonicKeyboardWithControllers and these additional knobs etc. are collected into one DiatonicLiveMidiKeyboard composable function, and it is packaged as compose-audio-controls-midi AAR, which now depends on ktmidi library (let me rephrase: DiatonicKeyboard is not necessarily tied to MIDI; DiatonicLiveMidiKeyboard is, for sure).

Summary

Resident MIDI Keyboard helps dogfooding your MIDI output devices, or works as a MIDI input device and get connected from your DAWs. It supports both MIDI 1.0 and 2.0 so it would be also useful for your next MIDI 2.0 development.

compose-audio-controls library is the core background of the app. I have explained various design principles behind those ImageStripKnob and DiatonicKeyboard components. DiatonicLiveMidiKeyboard enhances those features with extensive use of ktmidi library.

And as I have explained a lot of background on why I needed them in the beginning, it would help further AAP development.