Contents

• A review of last year’s goals
• A concrete project to focus the attention
• A generic pipeline for microtonal music
• What is a musical tuning?
• Adding microtonal support to Verovio
• The multiple accidentals controversy
• More tuning toys
• What is music-i18n, really?
• Goals for the coming year

A review of last year’s goals

Around the same time last year, I presented a progress report of where I was with my music work. It ended with the following goals for this year:

• Embed playable music sheets into actual CMS systems, starting with my own Arabic Real Book sheets. I did not work on this goal at all :cry: BUT! I did get involved in an exciting music publishing project that I will share below.

• Reach a milestone with musicxml-mscx to convert full music scores from MusicXML to MuseScore format. I did reach a good milestone with this work. I focused on rendering style beyond what MusicXML affords, by accepting user-defined MuseScore stylesheets (.mss, which can be created from within MuseScore’s style settings) in the conversion process. For example, this module can now produce stylized MuseScore lead sheets like the following:

• Explore the feasibility of using pre-rendered scores in musicxml-player to replace resource-intensive JavaScript notation engines. I dug deep to understand the various assets produced by MuseScore and Verovio cli tools. The outcome of this work was to create new musicxml-player classes MuseScoreConverter, MuseScoreRenderer, VerovioStaticConverter, VerovioStaticRenderer which accept asset files from these respective engravers, and require no additional JavaScript modules to render and interact with the scores. In brief, those engravers produce the following assets: SVG files for the rendered score, MIDI files for playback, and JSON metadata that describe how the various music objects (notes, measures, etc.) are laid out in time and space. The converter and renderer classes parse those various files to integrate them into the main player. Here’s an example of a simple score rendered to SVG / MIDI by Verovio and displayed by the MusicXML Player demo:

• Replace my simplistic MIDI soft-synth in musicxml-player with a more complete one such as SpessaSynth. I spent a few weeks integrating the awesome SpessaSynth, and participating with its author to test their brand-new TypeScript version. This browser-based MIDI synth has an impressive list of features, and for me the deciding factor to adopt it was its support for MIDI Tuning Standard (MTS), which is an indispensable component of music-i18n (more on this below). Other than the synthesizer, SpessaSynth also features a mature sequencer that allowed me to significantly simplify my code and simultaneously gain exciting new features. The above screenshot of the latest version of the MusicXML Player shows how SpessaSynth is being used: As a local synth, as a sequencer, and as an external synth exposing a Web MIDI port. A major win on many fronts! :tada:

• Explore multiplayer playback in musicxml-player. Honestly, I don’t know what got into me to mention this as a goal :roll_eyes: :shrug: I didn’t spend a single cycle thinking about this. Next!

• Support microtonality in MusicXML to MIDI conversion. This goal, on the other hand, is where I spent the great majority of the year. It was all triggered by a fellow coder reaching out for my help with their Arabic sheet music website Maqamatna, to enhance the sheet page with an online music player (exactly like musicxml-player that I’ve been working on!) The big issue with Arabic music is that it includes notes (pitches) that are tuned differently than Western music, sort of like blues bends that are stuck halfway between a fret and the next on the guitar. This necessitates a rethink of the full music production pipeline, from music engraving to MIDI file production to online playback. Since this is the central use-case for my music-i18n vision, I gladly accepted this opportunity to jump into a concrete project that would challenge and solidify my vision. And how challenged I was! I will spend the rest of this post describing the details of the work that ensued.

• Expand the groove conversion algorithm in musicxml-grooves to handle full MIDI files. Given the above, I could not devote much attention to this fascinating problem. My research led me to some very interesting academic experiments such as qparselib (which has unfortunately gone closed-source recently). I hope to return to this problem in the future.

A concrete project to focus the attention

There’s nothing like a concrete project to crystallize a vision and challenge one’s fanciful assumptions against the cold, hard reality. When the Maqamatna author contacted me to apply my (claimed) expertise to their website, I knew I had found this opportunity and this challenge. I was a little nervous, because up to this point many details of my music-i18n vision had been left as exercises to the proverbial reader. Now, I needed to deliver on my claims. After a few sessions of elicitation and brainstorming, we put together a plan of action. In order to enhance the song page with playback, both the backend and the frontend need modifications:

We quickly agreed that the pipeline I had been working on applies well for this use-case:

This high-level plan had many, many details to fill in. Most importantly, we had to select the technologies to generate the static assets on the backend and to render them visually and aurally on the frontend. The requirements are:

• Generator can be invoked as a headless console tool
• Generator and renderer support Arabic music, including text, accidentals, tunings
• Renderer can play back the music with custom tunings - solved by integrating SpessaSynth as discussed earlier

Out of the box, both MuseScore and Verovio support the first requirement, but not the second. In fact, microtonal / xenharmonic / world music support is severely lacking in most music software, especially in the open source domain. Even MusicXML, the W3C standard for music notation, does not address the question of microtonal music except in the most simplistic way of specifying the pitch alteration for each note separately. It has no concept of a “tuning”: How each degree in a scale is tuned, how it is represented in musical notation and how it maps to a MIDI key.

This was the main puzzle that I spent the following several months solving.

A generic pipeline for microtonal music

I wanted to design a pipeline for microtonal music that would require the least modifications to existing softwares, and preferably none to existing standards. For each step in the pipeline above, we need to ensure that the important music features for this project are correctly preserved:

The pipeline starts with a MuseScore score that includes non-standard accidentals and possibly non-standard key signatures. MuseScore exports MusicXML files which are reasonably faithful to the original, including the correct accidentals and key signatures. In terms of accidentals, MusicXML supports SMuFL symbols, which is another W3C standard that functions as an add-on to Unicode specialized in music symbols. Although MuseScore also supports SMuFL accidentals, it exports them to MusicXML incorrectly and thus needs to be corrected (either by patching or by post-processing the output MusicXML document).

The next step is to convert the MusicXML to visual and audio assets that are passed to the frontend. I selected the excellent engraver Verovio as the main conversion engine, instead of either MuseScore or my own musicxml-midi module. Verovio is attractive because it is a C++ console tool that specializes in converting scores to such assets, as opposed to MuseScore which is weighed down by a full GUI application. Verovio’s full functionality is also cross-compiled to WASM, making it doubly-attractive because any fixes to the backend would automatically reflect on the frontend. After evaluating Verovio’s almost-comprehensive MIDI output, I came to the conclusion that spending the effort understanding its codebase and patching it to fill the gaps would be much more efficient that recreating a full-featured and robust MIDI converter in my own musicxml-midi module. I decided to rethink the conversion pipeline of musicxml-midi to delegate the MIDI conversion to Verovio, and focus instead on generating automatic accompaniment using MMA in this module. Since accompaniment generation is not a feature of the Maqamatna project, I left this part for later and kept going.

Now Verovio comes with its own set of challenges and idiosyncrasies. For one, its internal document representation is not based on MusicXML, but on MEI (Music Encoding Initiative), another widely used music notation format. This means that whatever work I do in Verovio will also have to be compatible with MEI :sob: In my calculations, this was an unavoidable cost I would have to absorb. Fortunately, MEI is a mature format that is largely compatible with MusicXML so I was reasonably confident that the balance still tilted in the positive.

What is missing from Verovio to implement the microtonal features needed by Maqamatna?

• Specifying a tuning that Verovio should apply to the MIDI export
• Converting that tuning to MTS (MIDI Tuning Standard) during MIDI export
• Carrying over non-standard accidentals in the score from one note to the next, including non-standard accidentals in the key signature
• Mapping notes/accidentals to their expected entries in the tuning, and thus exporting them correctly to MIDI

What is a musical tuning?

Whenever questions of musical tunings arise, the first place to check is the incredible Scala application, which is considered to be the reference implementation for all things microtonal. Scala has defined the de-facto standard to describe tunings.

A tuning defines how each note should be tuned (i.e. its pitch) relative to some base note. Because the tuning is relative, we’re not using actual frequencies in Hertz but rather frequency ratios (for example, the octave ratio is 2/1 of the base note), or a logarithmic scale in units called “cents” where 1200 cents correspond to one octave, and 1200/12 = 100 cents correspond to one semitone. Here’s a Scala SCL file for the most common Arabic tuning, which in addition to the common 12 Western musical tones, includes 12 other tones tuned halfway between pairs of semitones (called quarter-tones):

! 24-edo.scl
!
! This is a comment.
!
! Tuning description:
24 equal divisions of an octave. 24 proportionally equal and equal sounding semitone intervals per octave.
!
! Count of tones:
24
!
! List of tones:
50.0
100.0
150.0
200.0
250.0
300.0
350.0
400.0
450.0
500.0
550.0
600.0
650.0
700.0
750.0
800.0
850.0
900.0
950.0
1000.0
1050.0
1100.0
1150.0
2/1

This information alone is not enough to produce a MIDI file from a musical score. There are questions left unanswered:

• What is the actual frequency of each tone in the different octaves?
• How will notes in the score map to the tones above?
• How will the tones map to MIDI notes in the output MIDI file?

To answer these questions, we need to supply more information. After some research, I found that Ableton, the makers of the famous Live DAW (Digital Audio Workstation), have created an extension to the Scala SCL format that answers these very questions:

! 24-edo.ascl
!
! This is a comment.
!
! Tuning description:
24 equal divisions of an octave. 24 proportionally equal and equal sounding semitone intervals per octave.
!
! Count of tones:
24
!
! List of tones:
50.0
100.0
150.0
200.0
250.0
300.0
350.0
400.0
450.0
500.0
550.0
600.0
650.0
700.0
750.0
800.0
850.0
900.0
950.0
1000.0
1050.0
1100.0
1150.0
2/1
!
!!! ASCL EXTENSIONS START HERE !!!
!
! @ABL NOTE_NAMES "Bs/C" "C1qs" "Cs/Df" "Dbf" "D" "D1qs" "Ds/Ef" "Ebf" "E/Ff" "E1qs/Fbf" "Es/F"  ↩
"F1qs" "Fs/Gf" "Gbf" "G" "G1qs" "Gs/Af" "Abf" "A" "A1qs" "As/Bf" "Bbf" "B/Cf" "B1qs/Cbf"
! @ABL REFERENCE_PITCH 4 0 261.6256

Here, the ASCL file augments the SCL file with @ABL entries that specify the note names as they are expected to be found in the score, and a reference tone from which all other tones can be computed and which maps to the MIDI key 60 (midpoint-ish between 0-127). I gladly adopted the Ableton ASCL format to supply tuning information to Verovio, instead of inventing yet another format that would pollute the collective cognitive field :exploding_head:

Adding microtonal support to Verovio

Now I needed some C++ code to integrate ASCL parsing and processing inside Verovio. As I am loath to create unnecessary new code, I found a header-only C++ library that parses Scala SCL files, maintained by the same team that maintains the excellent open-source synth Surge XT. Since ASCL provides relatively few extensions to SCL, I decided to take a chance and attempt a PR to add ASCL support to this library. In about a month, and thanks to the willingness and help of the library maintainers, I was able to get my changes integrated into tuning-library :tada:

With this PR done, I dove into the Verovio codebase. I’ve been working on a fork of the repo for 3 months now, and I’ve reached a point where microtonal support is fully functional in both the console tool and the WASM module. The “theory of operation” of microtonal support goes as follows:

First, Verovio needs to accept Ableton ASCL tunings, whether in MusicXML files via the existing element play/other-play[@type='tuning-ableton'], or passed to the console tool via an option verovio --tuning /path/to/tuning-file.ascl, or in the JavaScript bindings via a new attribute VerovioOptions.tuning which takes an ASCL definition.

Here’s how the tuning looks when it’s embedded in a MusicXML file:

  [..]
  <part id="P1">
    <measure number="1" width="224.66">
      <sound>
        <play>
          <other-play type="tuning-ableton">
            <![CDATA[
! 24-edo.ascl
!
24 equal divisions of an octave. 24 proportionally equal and equal sounding semitone intervals per octave.
!
! default tuning: degree 18 (900.0 cents) 440 Hz, or degree 0 = 261.625565 Hz
!
24
!
50.
100.
150.
200.
250.
300.
350.
400.
450.
500.
550.
600.
650.
700.
750.
800.
850.
900.
950.
1000.
1050.
1100.
1150.
2/1
!
! Note names are formatted per MEI accidentals.
!
! @ABL NOTE_NAMES Bsharp/C Cquarter-sharp Csharp/Dflat Dslash-flat D Dquarter-sharp Dsharp/Eflat Eslash-flat  ↩
E/Fflat Equarter-sharp/Fslash-flat Esharp/F Fquarter-sharp Fsharp/Gflat Gslash-flat G Gquarter-sharp Gsharp/Aflat  ↩
Aslash-flat A Aquarter-sharp Asharp/Bflat Bslash-flat B/Cflat Bquarter-sharp/Cslash-flat
! @ABL REFERENCE_PITCH 4 0 261.6256
! @ABL NOTE_RANGE_BY_INDEX 0 21 6 4
! @ABL LINK https://www.ableton.com/learn-more/tuning-systems/24-edo
            ]]>
            </other-play>
          </play>
        </sound>
      <attributes>
  [..]

and in JavaScript:

import createVerovioModule from 'verovio/wasm';
import { VerovioToolkit } from 'verovio/esm';

createVerovioModule().then(async VerovioModule => {
  const verovioToolkit = new VerovioToolkit(VerovioModule);
  const score = await (await fetch('/path/to/score.musicxml')).text();
  verovioToolkit.loadData(score);
  verovioToolkit.setOptions({
    font: 'Bravura',
    tuning: `
! 24-edo.ascl
!
24 equal divisions of an octave. 24 proportionally equal and equal sounding semitone intervals per octave.
!
! default tuning: degree 18 (900.0 cents) 440 Hz, or degree 0 = 261.625565 Hz
!
24
!
50.
100.
150.
200.
250.
300.
350.
400.
450.
500.
550.
600.
650.
700.
750.
800.
850.
900.
950.
1000.
1050.
1100.
1150.
2/1
!
! Note names are formatted per MEI accidentals.
!
! @ABL NOTE_NAMES Bs/C C1qs Cs/Df Dbf D D1qs Ds/Ef Ebf E/Ff E1qs/Fbf Es/F F1qs Fs/Gf Gbf G G1qs Gs/Af  ↩
Abf A A1qs As/Bf Bbf B/Cf B1qs/Cbf
! @ABL REFERENCE_PITCH 4 0 261.6256
! @ABL NOTE_RANGE_BY_INDEX 0 21 6 4
! @ABL LINK https://www.ableton.com/learn-more/tuning-systems/24-edo
    `.trim()
  })
});

The proverbial astute reader will have noticed that the @ABL NOTE_NAMES differ in the MusicXML case from the Verovio case. In the former, the accidentals are formulated as MusicXML accidentals, whereas in the latter, they are formulated as MEI accidentals. It’s worth mentioning that neither MusicXML nor MEI encompass the full set of SMuFL accidentals (488 at the latest count), so in both cases SMuFL accidentals are also recognized as valid names.

Once stored in Verovio’s internal data store, the tuning is ready to be used during MIDI export. This happens in two steps:

1. Export an MTS (MIDI Tuning Standard) message based on the tuning information above. This is a MIDI SysEx message to which we pass the retuned frequencies of all 128 MIDI keys. The new frequencies are computed by tuning-library.

2. With the MIDI tuning now in place, we need to find the MIDI key that corresponds to each note in the score. For this, we maintain a map between the incoming @ABL NOTE_NAMES and the tuning entries. Again, tuning-library knows how to map tuning entries to MIDI keys, so we output this result to the MIDI file, instead of the default MIDI key that would have been sent in the absence of a tuning.

To clarify what this means, let’s look at the 24-edo.ascl tuning file just above in the context of an actual score. For lovers of Arabic music, you can follow Sami Abu Shumays’ excellent maqam tutorials, of which this is an excerpt:

Examining this score, we find two half-flat notes: B½♭ and E½♭, notated in their typical Arabic / Turkish notation. Expressed in MEI syntax, this gives us Bbf and Ebf, which we can find in the @ABL NOTE_NAMES above, corresponding to 350¢ and 1050¢ respectively.

We now map the 24 tuning tones (per octave) to MIDI, starting at the @ABL REFERENCE_PITCH (C4) which we map to MIDI key 60. Here’s the output of tuning-library’s utility showmapping with the tuning above, spanning all 128 MIDI keys:

$ showmapping 24-edo.ascl
Note,        Freq (Hz),        ScaledFrq,        logScaled,  Pos, Name
   0,    46.2493028390,     5.6568542495,     2.5000000000,   12, Fs/Gf
   1,    47.6045108553,     5.8226127314,     2.5416666667,   13, Gbf
[..]
  50,   195.9977179909,    23.9729132300,     4.5833333333,   14, G
  51,   201.7408895050,    24.6753732065,     4.6250000000,   15, G1qs
  52,   207.6523487900,    25.3984168315,     4.6666666667,   16, Gs/Af
  53,   213.7370270538,    26.1426472519,     4.7083333333,   17, Abf
  54,   220.0000000000,    26.9086852881,     4.7500000000,   18, A
  55,   226.4464920616,    27.6971699522,     4.7916666667,   19, A1qs
  56,   233.0818807590,    28.5087589805,     4.8333333333,   20, As/Bf
  57,   239.9117011864,    29.3441293825,     4.8750000000,   21, Bbf
  58,   246.9416506281,    30.2039780058,     4.9166666667,   22, B/Cf
  59,   254.1775933119,    31.0890221169,     4.9583333333,   23, B1qs/Cbf
  60,   261.6255653006,    32.0000000000,     5.0000000000,    0, Bs/C
  61,   269.2917795270,    32.9376715726,     5.0416666667,    1, C1qs
  62,   277.1826309769,    33.9028190195,     5.0833333333,    2, Cs/Df
  63,   285.3047020232,    34.8962474453,     5.1250000000,    3, Dbf
  64,   293.6647679174,    35.9187855459,     5.1666666667,    4, D
  65,   302.2698024408,    36.9712862999,     5.2083333333,    5, D1qs
  66,   311.1269837221,    38.0546276801,     5.2500000000,    6, Ds/Ef
  67,   320.2437002253,    39.1697133857,     5.2916666667,    7, Ebf
  68,   329.6275569129,    40.3174735966,     5.3333333333,    8, E/Ff
  69,   339.2863815897,    41.4988657488,     5.3750000000,    9, E1qs/Fbf
  70,   349.2282314330,    42.7148753334,     5.4166666667,   10, Es/F
  71,   359.4613997130,    43.9665167187,     5.4583333333,   11, F1qs
  72,   369.9944227116,    45.2548339959,     5.5000000000,   12, Fs/Gf
  73,   380.8360868427,    46.5809018509,     5.5416666667,   13, Gbf
  74,   391.9954359817,    47.9458264601,     5.5833333333,   14, G
  75,   403.4817790101,    49.3507464131,     5.6250000000,   15, G1qs
  76,   415.3046975799,    50.7968336630,     5.6666666667,   16, Gs/Af
  77,   427.4740541076,    52.2852945037,     5.7083333333,   17, Abf
  78,   440.0000000000,    53.8173705762,     5.7500000000,   18, A
  79,   452.8929841231,    55.3943399044,     5.7916666667,   19, A1qs
  80,   466.1637615181,    57.0175179610,     5.8333333333,   20, As/Bf
  81,   479.8234023727,    58.6882587651,     5.8750000000,   21, Bbf
  82,   493.8833012561,    60.4079560116,     5.9166666667,   22, B/Cf
  83,   508.3551866238,    62.1780442338,     5.9583333333,   23, B1qs/Cbf
  84,   523.2511306012,    64.0000000000,     6.0000000000,    0, Bs/C
  85,   538.5835590540,    65.8753431452,     6.0416666667,    1, C1qs
  86,   554.3652619537,    67.8056380390,     6.0833333333,    2, Cs/Df
  87,   570.6094040464,    69.7924948906,     6.1250000000,    3, Dbf
  88,   587.3295358348,    71.8375710918,     6.1666666667,    4, D
  89,   604.5396048816,    73.9425725998,     6.2083333333,    5, D1qs
  90,   622.2539674442,    76.1092553602,     6.2500000000,    6, Ds/Ef
  91,   640.4874004506,    78.3394267715,     6.2916666667,    7, Ebf
  92,   659.2551138257,    80.6349471933,     6.3333333333,    8, E/Ff
  93,   678.5727631795,    82.9977314977,     6.3750000000,    9, E1qs/Fbf
  94,   698.4564628660,    85.4297506669,     6.4166666667,   10, Es/F
  95,   718.9227994261,    87.9330334373,     6.4583333333,   11, F1qs
  96,   739.9888454233,    90.5096679919,     6.5000000000,   12, Fs/Gf
  97,   761.6721736854,    93.1618037019,     6.5416666667,   13, Gbf
  98,   783.9908719635,    95.8916529201,     6.5833333333,   14, G
  99,   806.9635580201,    98.7014928261,     6.6250000000,   15, G1qs
[..]
 127,  1811.5719364925,   221.5773596176,     7.7916666667,   19, A1qs

From this table, Verovio is now able to output the correct notes to the MIDI file! Here’s what this particular transcription sounds like.

The multiple accidentals controversy

Are multiple accidentals per note accepted in music theory?

The answer seems to vary depending on who you ask. But for music-i18n, the answer is a resounding YES! Witness this excerpt from the Extended Helmholtz-Ellis JI Pitch Notation booklet:

Or this microtonal tune written in the encyclopedic Sagittal Notation:

Software support for multiple accidentals is also uneven:

• MusicXML flatly does not support multiple accidentals per note, because the note/accidental element is only accepted once. I sadly resolved to fork and patch the MusicXML schema to support multiple accidentals, until I prepare a good case to support this feature in the official version :sweat_smile:

• Whereas MEI does technically support the notion of multiple accidentals per note, it seems the Verovio maintainers are unsure about it. Which means it’s up to me to ensure this support fully works for MusicXML import, MIDI export and even regular visual engraving :sweat_smile: :sweat_smile:

More tuning toys

In order to test the patched Verovio, I created some utilities to create tuning files. In the Unix spirit, those are small separate tools that are combined to build up the final result. It’s best to be familiar with the Linux environment to run these comfortably:

• A Python script to convert the Sagittal reference spreadsheet to JSON:

{
    "accidentalNatural": {
        "range": "Conventional Sagittal-compatible accidentals",
        "unicode": {
            "character": "",
            "code_point": "U+E261"
        },
        "sagitype": {
            "long": {
                "revo_pure": "|//|",
                "evo_mixed": {
                    "comma": null,
                    "sharp_flat": "h or |//|"
                }
            },
            "short": {
                "evo_mixed": {
                    "comma": null,
                    "sharp_flat": "h"
                }
            }
        },
        "pitch": {
            "description": {
                "sharp_flat": "natural",
                "commatic_alteration": null,
                "direction": null
            },
            "cents": 0.0,
            "ratio": {
                "numerator": 1,
                "denominator": 1
            },
            "prime_count_vector": {
                "2": null,
                "3": null,
                "5": null,
                "7": null,
                "11": null,
                "13": null,
                "17": null,
                "19": null,
                "23": null,
                "29": null,
                "31": null,
                "37": null
            }
        },
        "ji_pitches": {
            "3^-2": null,
            "3^-1": null,
            "3^0": null,
            "3^1": null,
            "3^2": null
        },
        "notation_membership": {
            "prime_factor": null,
            "12_relative_fractions": "natural",
            "12_relative_cents": 0.0,
            "edo_degrees": {
                "17": 0,
                "19": 0,
                "22": 0,
                "27": 0,
                "29": 0,
                "31": 0,
                "34": 0,
                "39": 0,
                "41": 0,
                "43": 0,
                "46": 0,
                "50": 0,
                "53": 0,
                "60": 0,
                "72": 0,
                "96": 0
            }
        },
        "sagispeak": {
            "simple": {
                "spelling": null,
                "ipa_1": null,
                "ipa_2": null,
                "ipa_3": null,
                "ipa_4": null
            },
            "alternative": {
                "spelling": null,
                "ipa_1": null,
                "ipa_2": null,
                "ipa_3": null,
                "ipa_4": null
            },
            "sharp_flat": "natural"
        },
        "symbol": {
            "symbol_or_accent": "symbol",
            "shaft_count": null
        },
        "smufl": {
            "glyph_name": "accidentalNatural",
            "description": "Natural"
        },
        "glyph_description": {
            "graphical": "natural",
            "heraldic": "Hera's throne"
        }
    },
[..]
    "accSagittal11LargeDiesisUp": {
        "range": "Spartan Sagittal single-shaft accidentals (U+E300–U+E30F)",
        "unicode": {
            "character": "",
            "code_point": "U+E30C"
        },
        "sagitype": {
            "long": {
                "revo_pure": "(|)",
                "evo_mixed": {
                    "comma": "(|)",
                    "sharp_flat": null
                }
            },
            "short": {
                "evo_mixed": {
                    "comma": "m",
                    "sharp_flat": null
                }
            }
        },
        "pitch": {
            "description": {
                "sharp_flat": null,
                "commatic_alteration": "11-L-diesis",
                "direction": "up"
            },
            "cents": 60.412,
            "ratio": {
                "numerator": 729,
                "denominator": 704
            },
            "prime_count_vector": {
                "2": -6,
                "3": 6,
                "5": 0,
                "7": 0,
                "11": -1,
                "13": null,
                "17": null,
                "19": null,
                "23": null,
                "29": null,
                "31": null,
                "37": null
            }
        },
        "ji_pitches": {
            "3^-2": "128/99",
            "3^-1": "64/33",
            "3^0": "16/11",
            "3^1": "12/11",
            "3^2": "18/11"
        },
        "notation_membership": {
            "prime_factor": null,
            "12_relative_fractions": null,
            "12_relative_cents": null,
            "edo_degrees": {
                "17": null,
                "19": null,
                "22": null,
                "27": null,
                "29": null,
                "31": null,
                "34": null,
                "39": 3,
                "41": null,
                "43": null,
                "46": 3,
                "50": null,
                "53": null,
                "60": null,
                "72": null,
                "96": null
            }
        },
        "sagispeak": {
            "simple": {
                "spelling": "jatai",
                "ipa_1": "/dʒɐ ˈtaɪ/",
                "ipa_2": "/ʒɐ ˈtaɪ/",
                "ipa_3": "/jɐ ˈtaɪ/",
                "ipa_4": "/hɐ ˈtaɪ/"
            },
            "alternative": {
                "spelling": "wai",
                "ipa_1": "/waɪ/",
                "ipa_2": null,
                "ipa_3": null,
                "ipa_4": null
            },
            "sharp_flat": null
        },
        "symbol": {
            "symbol_or_accent": "symbol",
            "shaft_count": 1
        },
        "smufl": {
            "glyph_name": "accSagittal11LargeDiesisUp",
            "description": "11 large diesis up, (11L), (sharp less 11M), 3° up [46-EDO]"
        },
        "glyph_description": {
            "graphical": "double arc up",
            "heraldic": "Dionysus' wine cup"
        }
    },
[..]
}

• A jq script to convert the Sagittal JSON above + SMuFL accidentals into a JSON map of accidental => pitch alteration. The resulting file needs to be edited manually for the missing accidentals - see my current version. This script was mostly vibe-coded and it took multiple iterations before :robot: got it right!!

{
  "accidentalDoubleFlatArabic": -200,
  "accidentalThreeQuarterTonesFlatArabic": -150,
  "accidentalFlatArabic": -100,
  "accidentalQuarterToneFlatArabic": -50,
  "accidentalNaturalArabic": 0,
  "accidentalQuarterToneSharpArabic": 50,
  "accidentalSharpArabic": 100,
  "accidentalThreeQuarterTonesSharpArabic": 150,
  "accidentalDoubleSharpArabic": 200,
  "accidentalBuyukMucennebFlat": -181.1,
  "accidentalKucukMucennebFlat": -113.2,
  "accidentalBakiyeFlat": -90.6,
  "accidentalKomaFlat": -22.6,
  "accidentalKomaSharp": 22.6,
  "accidentalBakiyeSharp": 90.6,
  "accidentalKucukMucennebSharp": 113.2,
  "accidentalBuyukMucennebSharp": 181.1,
  "accSagittal7v11KleismaUp": 9.688,
  "accSagittal7v11KleismaDown": -9.688,
  "accSagittal17CommaUp": 14.73,
  "accSagittal17CommaDown": -14.73,
  "accSagittal55CommaUp": 31.767,
  "accSagittal55CommaDown": -31.767,
  "accSagittal7v11CommaUp": 33.148,
  "accSagittal7v11CommaDown": -33.148,
  "accSagittal5v11SmallDiesisUp": 38.906,
  "accSagittal5v11SmallDiesisDown": -38.906,
  "accSagittalSharp5v11SDown": 74.779,
  "accSagittalFlat5v11SUp": -74.779,
  "accSagittalSharp7v11CDown": 80.537,
  "accSagittalFlat7v11CUp": -80.537,
  "accSagittalSharp55CDown": 81.918,
  "accSagittalFlat55CUp": -81.918,
  "accSagittalSharp17CDown": 98.955,
  "accSagittalFlat17CUp": -98.955,
  "accSagittalSharp7v11kDown": 103.997,
  "accSagittalFlat7v11kUp": -103.997,
  "accSagittalSharp7v11kUp": 123.373,
  "accSagittalFlat7v11kDown": -123.373,
  "accSagittalSharp17CUp": 128.415,
  "accSagittalFlat17CDown": -128.415,
  "accSagittalSharp55CUp": 145.452,
  "accSagittalFlat55CDown": -145.452,
  "accSagittalSharp7v11CUp": 146.833,
  "accSagittalFlat7v11CDown": -146.833,
  "accSagittalSharp5v11SUp": 152.591,
  "accSagittalFlat5v11SDown": -152.591,
  "accSagittalDoubleSharp5v11SDown": 188.464,
  "accSagittalDoubleFlat5v11SUp": -188.464,
  "accSagittalDoubleSharp7v11CDown": 194.222,
  "accSagittalDoubleFlat7v11CUp": -194.222,
  "accSagittalDoubleSharp55CDown": 195.603,
  "accSagittalDoubleFlat55CUp": -195.603,
  "accSagittalDoubleSharp17CDown": 212.64,
  "accSagittalDoubleFlat17CUp": -212.64,
  "accSagittalDoubleSharp7v11kDown": 217.682,
  "accSagittalDoubleFlat7v11kUp": -217.682,
  "accidentalDoubleFlatOneArrowDown": -248.88,
[..]
}

• Finally, an XSL script to parse a MusicXML score into an ASCL tuning. The script gathers note+accidental combinations in the MusicXML score, and emits a correctly-formatted tuning file for those combinations. The SMuFL accidentals JSON file above is consulted to obtain the right pitch alteration in case it’s missing in the score. The reusable XSL library lib-musicxml.xsl does the heavy lifting of computing carried-over and implicit accidentals throughout the score :raised_hands:

What is music-i18n, really?

I keep mentioning music-i18n in this post, without defining it. What do I mean by this term?

Just like human languages, I consider music to be a language with many dialects. Pardon the imprecise analogy: My aim is not to dive into linguistics or ethnomusicology. At the moment, and for the most part, existing music software are speaking one specific dialect of music: The Western mainstream dialect, based on 12 tones tuned to specific frequencies and from which a huge edifice of theory and assumptions has been built. Music software followed suit and integrated the theory and assumptions of mainstream Western music down to its deepest layers. It takes significant re-design and re-engineering effort to go back and refactor the many software layers to accommodate other assumptions, or to provide generalizations that fit multiple music dialects, without disrupting the whole software edifice.

Thankfully, music standards today support global music systems to a large extent. To cite some examples that I mentioned above:

• The MIDI Tuning Standard (MTS) allows to reprogram the 128 MIDI keys to any tuning, outside the mainstream 12 tones
• SMuFL is a wonderful W3C standard that extends Unicode with an ever-increasing set of musical symbols from different music cultures, and includes a reference font
• MusicXML supports a wide range of accidentals beyond the mainstream ones, including all SMuFL symbols, both for individual notes and for key signatures

I view music-i18n as an umbrella term for any activity that goes towards retrofitting existing music software (and creating more) with the ability to handle non-mainstream music systems, reusing to the fullest the capabilities of open standards, and working on extending those standards when they fall short. Of course, musicians and programmers all over the world routinely engage in music-i18n as they attempt to express non-Western musical cultures in software. I find that thinking about this activity holistically helps organize the individual efforts and comprehend its full scope.

Goals for the coming year

Wow, this was a mouthful! Thank you for reading all the way down. Where am I going from here?

First and foremost, I’ll be focusing on completing the Maqamatna work to see it through release. This will validate the work done over the year and provide a valuable new resource to the Arabic music community.

I also want to submit my work on Verovio for inclusion in the mainline. I expect that my current fork will need to be divided into smaller patches for submission, and with fairly significant changes before they are accepted. I hope the core maintainers will support my work in this PR :crossed_fingers:

Similarly for my MusicXML modifications to support multiple accidentals and other music-i18n elements, I will work on submitting them for inclusion :pray:

When the microtonal dust settles, I hope I can devote some attention to world rhythms. This has always been a personal interest in my music practice, and I hope to express it in software soon :drum:

As always, I hope to hear from interested readers and I welcome all opportunities for collaboration! :saxophone: :handshake:

drush cr; drush updb -y; drush cim -y; drush deploy:hook -y; drush cr;

This little sequence allows us to reliably update Drupal core, the site configuration and our own custom modules without running into dependency loops. Drupal’s Configuration API is a very well-designed system that has greatly simplified this process since Drupal 8, especially with its plugin-based architecture that allows contrib modules to fine-tune the process.

For us, the Config Ignore contrib module is invaluable because business users typically require control over some aspects the site’s configuration, typically when it comes to end-user-facing settings like labels and titles. By using Config Ignore’s excellent support for wildcards, individual subkeys and exclusion operator, we have a powerful toolset to give business users what they need.

Overriding config_ignore.settings in settings.php

During development, it’s common to want to override the official configuration with different settings. The usual approach is to use the settings.local.php file with a hard-coded $config entry - in our case $config['config_ignore.settings']. However, I quickly discovered that these overridden settings don’t get picked up by Config Ignore! Here we go, a new debugging dive 🤿… It turns out that the default Drupal config factory is only consulted if the config_ignore.settings entry is NOT present in the sync folder. I am pretty sure this is the opposite of the usual expectation, and I may submit an issue to discuss that. In the meantime, here’s a small workaround that will pick up your overridden settings:

// my_custom_module.module
use Drupal\config_ignore\ConfigIgnoreConfig;

/**
 * Implements hook_config_ignore_ignored_alter().
 */
function my_custom_module_config_ignore_ignored_alter(&$ignoreConfig) {
  $override = \Drupal::config('config_ignore.settings');
  if (!empty($override)) {
    try {
      $ignoreConfig = ConfigIgnoreConfig::fromConfig($override);
    }
    catch (\Throwable $e) {
      \Drupal::logger('my_custom_module')->error('Invalid value for config_ignore.settings override. Ignoring.');
    }
  }
}

⚠️ BUT! How to remove config entries, instead of adding them? Consider the following config_ignore.settings.yml file in your config sync:

# config_ignore.settings.yml
mode: simple
ignored_config_entities:
  - mimemail.settings
  - openid_connect.settings
  - 'openid_connect.settings.*'
  - system.maintenance
  - system.performance
  - system.site
  - update.settings
  - 'webform.webform.abilities_quiz:third_party_settings.my_custom_module.*'
  - 'webform.webform.interests_quiz:third_party_settings.my_custom_module.*'
  - 'webform.webform.learning_styles_quiz:third_party_settings.my_custom_module.*'
  - 'webform.webform.multiple_intelligences_quiz:third_party_settings.my_custom_module.*'
  - 'webform.webform.work_preferences_quiz:third_party_settings.my_custom_module.*'
  - 'webform.webform.work_values_quiz:third_party_settings.my_custom_module.*'

What happens if you declare the following override in your settings.local.php:

// settings.local.php
// INCORRECT VERSION!
$config['config_ignore.settings'] = [
  'mode' => 'simple',
  'ignored_config_entities' => [
    'mimemail.settings',
    'openid_connect.settings',
    'openid_connect.settings.*',
    'system.maintenance',
    'system.performance',
    // 'system.site',
    'update.settings',
    // 'webform.webform.abilities_quiz:third_party_settings.my_custom_module.*',
    // 'webform.webform.interests_quiz:third_party_settings.my_custom_module.*',
    // 'webform.webform.learning_styles_quiz:third_party_settings.my_custom_module.*',
    // 'webform.webform.multiple_intelligences_quiz:third_party_settings.my_custom_module.*',
    // 'webform.webform.work_preferences_quiz:third_party_settings.my_custom_module.*',
    // 'webform.webform.work_values_quiz:third_party_settings.my_custom_module.*'
  ]
];

Will system.site and the webform.webform.* be now kept out of the ignore list? ❌ NO!! As per the module code linked above, the $config array is merged with the original, resulting in the original bottom keys being kept. In order to truly override the settings, you would write the $config array to contain at least as many entries as the original:

// settings.local.php
$config['config_ignore.settings'] = [
  'mode' => 'simple',
  'ignored_config_entities' => [
    'mimemail.settings',
    'openid_connect.settings',
    'openid_connect.settings.*',
    'system.maintenance',
    'system.performance',
    '', // 'system.site',
    'update.settings',
    '', // 'webform.webform.abilities_quiz:third_party_settings.my_custom_module.*',
    '', // 'webform.webform.interests_quiz:third_party_settings.my_custom_module.*',
    '', // 'webform.webform.learning_styles_quiz:third_party_settings.my_custom_module.*',
    '', // 'webform.webform.multiple_intelligences_quiz:third_party_settings.my_custom_module.*',
    '', // 'webform.webform.work_preferences_quiz:third_party_settings.my_custom_module.*',
    '' // 'webform.webform.work_values_quiz:third_party_settings.my_custom_module.*'
  ]
];

Now we are correctly overriding Config Ignore settings :tada:

Ignoring Webform element titles

With this out of the way, let’s go back to the initial business requirement: Allowing admin users to modify webform element titles without these changes getting reverted during the next config sync.

Here’s what a typical webform config looks like:

# webform.webform.interests_quiz.yml
langcode: en
status: open
dependencies:
  module:
    - webformautosave
third_party_settings:
  webformautosave:
    auto_save: true
    auto_save_time: 5000
    optimistic_locking: false
weight: 0
open: null
close: null
uid: 1
template: false
archive: false
id: interests_quiz
title: 'Interests Quiz'
description: ''
categories: {  }
elements: |-
  page_1:
    '#type': webform_wizard_page
    '#title': 'Page 1'
    '#prev_button_label': Back
    '#next_button_label': Next
    i_would_like_to_building_kitchen_cabinets:
      '#type': radios
      '#title': 'I like building kitchen cabinets.'
      '#options': options_interests
      '#category': Realistic
      '#required': true
    i_would_enjoy_laying_brick_or_tile:
      '#type': radios
      '#title': 'I would enjoy laying brick or tile.'
      '#options': options_interests
      '#category': Realistic
      '#required': true
    i_would_like_to_develop_a_new_medicine:
      '#type': radios
      '#title': 'I would like to develop a new medicine.'
      '#options': options_interests
      '#category': Investigative
      '#required': true
[...]

As you can see, there’s no individual YAML key for each element title - instead, all elements are stored together in the elements key, with each element title specified in a #title subentry. How to ignore these #title entries while keeping the rest of the elements under config sync?

I don’t know about you, but the thought of hacking Drupal Configuration API + Config Ignore to handle synchronization of array sub-entries does not sound like a productive approach to me. Instead, I decided to reuse Webform’s Third Party Settings mechanism to store entries for each element label individually, and apply those labels instead of the originals during rendering. Here’s how the webform config would then look:

# webform.webform.interests_quiz.yml
langcode: en
status: open
dependencies:
  module:
    - webformautosave
    - my_custom_module # THIS IS NEW
third_party_settings:
  webformautosave:
    auto_save: true
    auto_save_time: 5000
    optimistic_locking: false
  my_custom_module: # THIS IS NEW
    i_would_like_to_building_kitchen_cabinets: 'I REALLY 💙 building kitchen cabinets.'
    i_would_enjoy_laying_brick_or_tile:
    i_would_like_to_develop_a_new_medicine: 'I would like to develop a new medicine and make 💰💰💰.'
weight: 0
open: null
close: null
uid: 1
template: false
archive: false
id: interests_quiz
title: 'Interests Quiz'
description: ''
categories: {  }
elements: |-
  page_1:
    '#type': webform_wizard_page
    '#title': 'Page 1'
    '#prev_button_label': Back
    '#next_button_label': Next
    i_would_like_to_building_kitchen_cabinets:
      '#type': radios
      '#title': 'I like building kitchen cabinets.'
      '#options': options_interests
      '#category': Realistic
      '#required': true
    i_would_enjoy_laying_brick_or_tile:
      '#type': radios
      '#title': 'I would enjoy laying brick or tile.'
      '#options': options_interests
      '#category': Realistic
      '#required': true
    i_would_like_to_develop_a_new_medicine:
      '#type': radios
      '#title': 'I would like to develop a new medicine.'
      '#options': options_interests
      '#category': Investigative
      '#required': true

With this in place, it’s now trivial to add the third party settings to config_ignore.settings, as we’ve seen above:

# config_ignore.settings.yml
mode: simple
ignored_config_entities:
  [..]
  - 'webform.webform.interests_quiz:third_party_settings.my_custom_module.*'
  [..]

Here’s the code needed to create the new title settings:

// my_custom_module.module

/**
 * Implements hook_webform_third_party_settings_form_alter().
 */
function my_custom_module_webform_third_party_settings_form_alter(array &$form, FormStateInterface $form_state) {
  /** @var \Drupal\webform\WebformInterface $webform */
  $webform = $form_state->getFormObject()->getEntity();

  // Add an entry for the title of each element.
  $questions = array_filter($webform->getElementsInitializedAndFlattened(), some_condition_function);
  foreach ($questions as $key => $question) {
    $form['third_party_settings']['my_custom_module'][$key] = [
      '#type' => 'textfield',
      '#title' => t('Override: @question', ['@question' => $question['#title']]),
      '#required' => false,
      '#default_value' => $webform->getThirdPartySetting('my_custom_module', $key, '')
    ];
  }
}

And here’s a rudimentary way to display them:

// my_custom_module.module

/**
 * Implements template_preprocess_fieldset().
 */
function my_custom_module_preprocess_fieldset(&$variables) {
  if (isset($variables['element']['#webform'])) {
    /** @var \Drupal\webform\WebformInterface $webform */
    $webform = \Drupal::entityTypeManager()->getStorage('webform')->load($variables['element']['#webform']);

    // Override the element title with the corresponding third party setting.
    $variables['element']['#title'] = $webform->getThirdPartySetting('my_custom_module', $variables['element']['#webform_key'], $variables['element']['#title']);
  }
}

Et voilà ! Happy site builders and happy business users 👷‍♀️🤝🤵‍♀️

A recent interaction on the Drupal community’s Slack prompted me to describe the work I’ve been doing to create a fully local, open source setup for Drupal AI tools. My use case is to provide relevant search results based on natural language (English) queries. There are deployment scenarios, such as government projects, where the full system needs to be deployed in the home country and to avoid communicating with API services located elsewhere - this is the scenario that interests me here. Since I received positive feedback on my system description, I thought I’d clean it up and share it here. Hope it helps someone!

Theory of operation

The general idea of using Search API with natural language queries is to create vector embeddings of the relevant content, which are then matched against the embedding of the incoming user query. Vector embeddings are computed by an LLM (Large Language Model, in case you just landed on our planet) that is served by a local instance of Ollama running on my CPU-only laptop. At this time, I am using the LLM mxbai-embed-large to generate the embedding vectors. These vectors are stored in the same database as Drupal - I always use PostgreSQL and its pgvector extension turns it into a perfectly acceptable vector database. The pretty amazing Drupal AI ecosystem supports these tools out of the box, so there’s almost no coding involved in this setup. Drupal AI even provides a Search API connector that is able to perform vector indexing within the familiar Drupal search infrastructure.

I’ll be illustrating this setup with content from WorkBC.ca, a large Drupal site for the Ministry of Post-Secondary Education and Future Skills, British Columbia, that my team and I have been building and maintaining for the past 2+ years. The content describes the 500+ official careers that are identified by the Federal Government of Canada as representing the Canadian workforce.

Docker setup

I always start with Docker Compose. I use the excellent Wodby Drupal stack as a starting point - it includes all needed services and has intelligent defaults. In my case, I want to add ollama to the services, as well as inject the pgvector extension into Postgres - here are the relevant bits:

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:${OLLAMA_TAG}
    ports:
    - "11434:11434"
    volumes:
    - /usr/share/ollama/.ollama:/root/.ollama # To store models on my local host
  postgres:
    build:
      context: .
      dockerfile: pgvector.Dockerfile
      args:
        POSTGRES_TAG: ${POSTGRES_TAG}
        PGVECTOR_TAG: ${PGVECTOR_TAG}

# pgvector.Dockerfile
ARG POSTGRES_TAG
FROM wodby/postgres:${POSTGRES_TAG} AS pgvector-builder
ARG PGVECTOR_TAG
RUN apk add git
RUN apk add build-base
RUN apk add clang
RUN apk add llvm-dev
WORKDIR /home
RUN git clone --branch v${PGVECTOR_TAG} https://github.com/pgvector/pgvector.git
WORKDIR /home/pgvector
RUN make
RUN make install

FROM wodby/postgres:${POSTGRES_TAG}
COPY --from=pgvector-builder /usr/local/lib/postgresql/bitcode/vector.index.bc /usr/local/lib/postgresql/bitcode/vector.index.bc
COPY --from=pgvector-builder /usr/local/lib/postgresql/vector.so /usr/local/lib/postgresql/vector.so
COPY --from=pgvector-builder /usr/local/share/postgresql/extension /usr/local/share/postgresql/extension

We are now ready to download the embedding model:

docker-compose run ollama
docker-compose exec ollama ollama pull mxbai-embed-large:latest # in a different console

Drupal modules setup

Here is the relevant configuration in my composer.json file:

{
    "require": {
        "drupal/ai": "^1.2@alpha",
        "drupal/ai_provider_ollama": "^1.1@beta",
        "drupal/ai_vdb_provider_postgres": "^1.0@alpha",
    }
}

and the enabled modules:

wodby@php.container:/var/www/html $ drush pml | grep AI
  AI                                            AI Core (ai)                                                                     Enabled    1.2.0-alpha1
  AI Providers                                  DropAI Provider (dropai_provider)                                                Disabled   1.2.0-alpha1
  AI                                            AI API Explorer (ai_api_explorer)                                                Enabled    1.2.0-alpha1
  AI Tools                                      AI Assistant API (ai_assistant_api)                                              Disabled   1.2.0-alpha1
  AI                                            AI Automators (ai_automators)                                                    Disabled   1.2.0-alpha1
  AI Tools                                      AI Chatbot (ai_chatbot)                                                          Disabled   1.2.0-alpha1
  AI                                            AI CKEditor integration (ai_ckeditor)                                            Disabled   1.2.0-alpha1
  AI                                            AI Content Suggestions (ai_content_suggestions)                                  Disabled   1.2.0-alpha1
  AI                                            AI ECA integration (ai_eca)                                                      Disabled   1.2.0-alpha1
  AI                                            AI External Moderation (Deprecated) (ai_external_moderation)                     Disabled   1.2.0-alpha1
  AI                                            AI Logging (ai_logging)                                                          Disabled   1.2.0-alpha1
  AI (Experimental)                             AI Search (ai_search)                                                            Enabled    1.2.0-alpha1
  AI                                            AI Translate (ai_translate)                                                      Disabled   1.2.0-alpha1
  AI                                            AI Validations (ai_validations)                                                  Disabled   1.2.0-alpha1
  AI Providers                                  Ollama Provider (ai_provider_ollama)                                             Enabled    1.1.0-beta2
  AI Vector Database Providers (Experimental)   Postgres VDB Provider (ai_vdb_provider_postgres)                                 Enabled    1.0.0-alpha2

Drupal AI setup

With the infrastructure ready, it’s now time to configure the Drupal AI components and wire them together. I’ll be using the Admin UI with URIs and screenshots to better illustrate the setup.

/admin/config/ai/providers/ollama

/admin/config/ai/settings

/admin/config/ai/vdb_providers/postgres

/admin/config/search/search-api

/admin/config/search/search-api/server/rag/edit

Note the Vector Database Configuration > Collection setting search_api_rag which is the name of a database table created to hold the vector embeddings.

/admin/config/search/search-api/index/career_profiles_rag/edit

/admin/config/search/search-api/index/career_profiles_rag/fields

The tricky bit here is understanding the Indexing option setting - namely, the difference between Contextual content and Main content for indexing purposes. During indexing, each content item is divided into several chunks whose size is chosen to generate vector embeddings that accurately reflect its semantic meaning without overwhelming the LLM’s context window limit. For example, the open source vector database Milvus mentions a chunk size of 128-512 tokens - a token roughly corresponds to subwords of ~4 characters. But by subdividing the item, some chunks may miss the information that reflects the nature of the overall content item. The solution offered here is to repeat some fields in all the chunks - such fields are labeled as Contextual, whereas the Main content is the one being subdivided.

:warning: A word of caution: This is probably the part that requires the most tweaking to get good search results!!

Results

Once we’ve indexed the content in the index above, we’re ready to test the search:

/admin/config/ai/explorers/vector_db_generator

In case you’re curious, here’s the database table that the Search API server creates to store vector embeddings. You can notice that each node is broken up in several chunks, each starting with the node title which was selected as one of the Contextual fields above:

This concludes part 1 of my Drupal AI setup. Next time, I’ll look at more specialized Search API use case before getting into the treacherous waters of generated responses. Happy vibing :robot:

This one was pretty confusing. I needed to dynamically update a drop-down every time a “parent” drop-down changed (think 2-level taxonomy vocabulary), which is a well-documented feature in the Forms API. In a nutshell, the parent element gets an #ajax callback that is called upon user interaction, and that returns the updated child element from the $form structure. The Drupal AJAX frontend code takes care of replacing the child element in the HTML form. Neat and simple. In my case, though, I needed this behaviour in a Views exposed form, and that’s when the trouble started. When changing the parent element, the callback was not being called, and instead, an unrelated error was displayed, saying An unrecoverable error occurred. The uploaded file likely exceeded the maximum file size (XXX) that this server supports.

Fortunately, I was able to find an existing issue (submitted in Jan 2016 :sob:) which was useful to confirm I was not vastly misunderstanding the problem. The workarounds mentioned in this ticket did not work for me, though, so I had to keep digging on my own. Here’s the result of my analysis:

Why this error?

The displayed error has absolutely nothing to do with the situation at hand: There’s no uploaded file, and there’s not even a POST‘ed form, since the AJAX request uses the GET method. My approach to find the source of an error is to start by locating the text of the error in the codebase and work backwards up the call stack - in this case, it is thrown by FormAjaxSubscriber::onException which is itself triggered by FormBuilder::buildForm under an unexpected condition:

    // In case the post request exceeds the configured allowed size
    // (post_max_size), the post request is potentially broken. Add some
    // protection against that and at the same time have a nice error message.
    if ($ajax_form_request && !$request->get('form_id')) {
        throw new BrokenPostRequestException($this->getFileUploadMaxSize());
    }

I don’t know about you, but to me the condition of a missing form_id seems unrelated to a file limit issue. By examining the AJAX GET request in the browser, I was able to verify that no form_id query argument is actually sent - which means that further down this function, the form builder will be unable to find the form object that should be built. Looks like a legitimate error and the AJAX frontend seems to be at fault.

The workaround needed a workaround

At this point, I had the choice of debugging and fixing the Drupal AJAX frontend code, or find a workaround that would allow me to keep working on my business feature. Although I am a firm believer that we should allocate some of our professional time to contribute to the open source software that we use, this seemed a deeper dive than I could afford at that point. Instead, I opted for the most generic workaround that I could reuse in similar future scenarios. Here’s what I came up with:

The general idea is to simply send the missing form_id in the AJAX request. The Form API #ajax properties helpfully include a customizable url entry, so I decided to augment the current URL with the form_id query argument. Something like that, maybe?

function my_module_form_views_exposed_form_alter(&$form, FormStateInterface $form_state, $form_id) {
    // DANGER: THIS WILL NOT WORK!
    $uri = \Drupal\Component\Utility\UrlHelper::parse(\Drupal::request()->getRequestUri());
    $uri['query']['form_id'] = $form['#id'];
    $uri['query']['ajax_form'] = 1;
    $form['my_parent_element']['#ajax'] = [
      'callback' => 'my_parent_element_callback',
      'wrapper' => 'my-parent-element-container',
      'url' => Url::fromUri('internal:' . $uri['path'], ['query' => $uri['query'], 'fragment' => $uri['fragment']]),
    ];
}

If only things were that simple! This did not work - the AJAX request kept missing ALL query arguments after this change. How on earth could URL options get ignored?? More hours, more digging revealed this code deep inside RenderElementBase::preRenderAjaxForm - someone decided to overwrite the incoming URL options with those from another key THAT IS NOT EVEN DOCUMENTED :angry: - I’m sure it seemed like a good idea at the time and I’ve edited the documentation to reflect this quirk :angel:

So the final code looks like:

function my_module_form_views_exposed_form_alter(&$form, FormStateInterface $form_state, $form_id) {
    // Override the AJAX request to include `form_id` and `ajax_form`.
    $uri = \Drupal\Component\Utility\UrlHelper::parse(\Drupal::request()->getRequestUri());
    $uri['query']['form_id'] = $form['#id'];
    $uri['query']['ajax_form'] = 1;
    $form['my_parent_element']['#ajax'] = [
      'callback' => 'my_parent_element_callback',
      'wrapper' => 'my-parent-element-container',
      'url' => Url::fromUri('internal:' . $uri['path']),
      'options' => ['query' => $uri['query'], 'fragment' => $uri['fragment']]
    ];
}

And this, my friends, is how I fixed the file size limit error that occurs on AJAXified Views exposed form elements :tada:

Sober concluding thoughts

In a codebase as large as Drupal’s, it is normal to expect inconsistencies and edge cases. Since this is the second issue that involves Views exposed forms (the first one being an unwanted interaction with Big Pipe), I am now expecting more bugs to emanate from this area - namely, the intersection between Views exposed forms and advanced Drupal features. I wonder if anyone’s done an analysis of open Drupal issues to find clusters of bugs based on Drupal core components or recurring keywords.

In this particular case, the error that Drupal reports is not only useless, it is actively misleading. This is not particularly unusual either, as error handling is notoriously one of the harder aspects of programming, and much virtual ink has been spilled to try to make sense of it. What should be reported to the user? What should be logged? What should be handled silently? As a software architect who interacts a lot with business users, I can tell you that core Drupal has its own share of confusing and unhelpful error messages. The most egregious one for me is the infamous message An illegal choice has been detected. Please contact the site administrator. which only serves to confuse users but offers them no help. In our own software process, I make sure to review the errors thrown by the developers and ask myself the following questions in each case:

• UX (User eXperience): Should end users see an error, a warning, or should the UI keep functioning silently? What information will best help end users to accomplish their task at hand?
• DX (Developer eXperience): Should site builders see an error, a warning, or should the application keep functioning silently? What information will best help site builders to develop the application?
• DevOps: Should system engineers see an error, a warning, or should the system keep functioning silently? What information will best help system engineers to manage the site’s operation?

The detail about the URL options being overridden by an undocumented ['#ajax']['options'] key not only illustrates the difficulty of keeping documentation in sync with the code, but also the importance of thinking about DX when designing APIs, to minimize surprises and inconsistencies which directly translate to bugs or wasted effort.

In the spirit of contributing back, I documented my workaround in the original issue and updated the AJAX Forms documentation accordingly - hoping it will prevent further unnecessary hair pulling!

I wrote a manifesto of sorts about it and naively set off to code my way out of this situation. I was driven by my own musical interests: Rediscovering and arranging songs from the popular Arabic repertoire into modern idioms. Although I achieved a modest milestone towards that particular goal, it opened up a universe of questions and possibilities about how music is computed, notated, played back. I have not stopped learning and coding in this space since then.

Here is a snapshot of where I am in this journey, and where (I think) I am headed.

An open source, standards-based, Unix-inspired, global-music ecosystem

At the core of my vision is an ecosystem of tools for publishing interactive musical ideas, ultimately delivered through the Web. The target audience includes music practitioners and institutions that are looking to publish interactive music material on the Web, like music teachers, university departments, cultural heritage institutions. Of course, Web-based music publishing already exists: It is possible to embed music scores via various platforms. But these platforms are proprietary, commercial, unextensible, and Western-music-centric. To me, this feels too restrictive for something as important as music. I am aiming for something more inclusive.

To produce an open music publishing system for the Web, I need to build on open standards:

• MusicXML is the W3C format for music sheet exchange - based on XML that I’ve come to appreciate for its maturity and incredible ecosystem of tools.
• MIDI is the 40+ years old (and evolving) standard protocol for communicating musical devices - in our context, it’s used to both encode the music content that is converted from MusicXML (via MIDI files) and to produce the actual sound (via MIDI synths).
• Web Audio API is the W3C API for producing audio within Web applications.
• Web MIDI API is the W3C API for integrating Web applications with MIDI devices.
• SMuFL is a Unicode extension to represent musical symbols, also part of the W3C Music Notation Community Group with maintains MusicXML.

By careful adherence to these standards, music applications can be built to transcend the limiting assumptions of Western mainstream practice. In some cases, it is necessary to get involved in tweaking the standards, and in many cases, open source implementations of these standards are incomplete or incorrect and need fixing.

The general workflow I have in mind for music publishing involves the following steps:

• Express musical ideas into MusicXML (by explicit score writing or by generation from other sources)
• Convert MusicXML to MIDI for playback (with optional augmentation such as auto-generated accompaniments)
• Load MusicXML in a Web player for display using existing components such as OSMD, Verovio, etc.
• Play MIDI in the Web player via Web MIDI or Web Audio using existing components such as Tone.js, WebAudioFont, etc.

What follows is a quick overview of the tools that I am maintaining to support this workflow.

Lead sheets to MusicXML

For pop/rock/jazz players, lead sheets are essential to convey the crux of a tune. To support the use case of playing lead sheets, I’ve created ireal-musicxml, a library to convert iReal Pro lead sheets to MusicXML. For example, this iReal Pro tune:

MusicXML to MIDI with accompaniment

To support conversion of MusicXML to MIDI, I’ve created musicxml-midi, a library and API server that supports the addition of auto-generated accompaniments via another open source tool that I’ve adopted and enhanced. Here’s how the same tune above is played back with accompaniment:

Groove to MusicXML

This library also includes musicxml-grooves, a tool to convert raw “grooves” (i.e. accompaniment patterns) into MusicXML sheets. This is how the following MIDI groove is interpreted by MuseScore and by musicxml-grooves:

MusicXML to MuseScore

MuseScore is one of the few serious open source music writing software, but it suffers from incomplete MusicXML import/export. I’ve recently started work on musicxml-mscx, a new library to perform more robust MusicXML conversion to and from the native MuseScore format.

Putting it all together: A Web-based audio player

Once the music assets are produced, they are ready to be loaded in a Web application. For this purpose, I’ve created musicxml-player, a Web component that loads MusicXML and MIDI files, in order to synchronize the audio playback with the animation of the music sheet. It’s an ambitious component that packages several 3rd-party modules into a flexible foundation to build Web-based music applications. Here are 2 video captures from the demo app in the component’s repo:

The challenges and rewards of this project

Writing this post, I realize I came a long way since that original manifesto in 2018… and that the road ahead is arbitrarily long. To remain motivated, I keep challenging myself to mini-projects that tickle my immediate fancy, and I do my best to fit them within the general framework of this ecosystem. I am constantly faced with new questions, from deeply philosophical ones to immediate programming problems:

• What are musical tunings, from a mathematical point of view? Why are the frequencies of the notes the way they are?
• What are the commonalities and differences between Western scales, Arabic maqams, Indian ragas, and musical modes from other world cultures?
• How to represent the musics of different cultures in a single programming system, without completely diluting the former and keeping the latter manageable?
• How to extract from MIDI events a sequence of notes that is understandable and playable by humans?
• How to reliably schedule notes from a MIDI file to be played in real-time on a Web page?
• How to link the information in a MusicXML score to the audio events in a MIDI file?
• What the heck is XSL and how can I write data transformations with it??

Answering these questions is the reason why I am still motivated to go on! Over the course of the years, I’ve encountered some truly inspiring projects that have expanded my musicological mind. Here are a few:

• Gareth Loy’s “Musimathics: A Guided Tour of the Mathematics of Music”

• Manuel Op de Coul’s Scala, a mind-bogglingly comprehensive tool for exploring musical tunings.

• Chris Wilson’s A tale of two clocks, the seminal article about robust Web audio sequencing.

I’ve been fortunate to work with others who are interested in this domain: A year ago, I was sponsored to add a “horizontal scrolling” mode to the player, as well as a method to synchronize the playback with a YouTube video (hint: it uses the Timing Object W3C draft specification) - both of which went back into musicxml-player. Today, I am exploring adding multiplayer capability to the player, also using Web standards. I’m also fortunate to interact with like-minded developers, like Christoph Guttandin who maintains a dizzying array of well-crafted audio modules - we collaborate on his excellent midi-player component which is a cornerstone of musicxml-player. Since the early days, I’ve been in touch with Bob van del Poel, a fellow British Columbian who wrote the ridiculously great Musical MIDI Accompaniment (MMA) system which is a cornerstone of musicxml-midi.

Looking ahead one year

Here’s what I hope to work on within the next year:

• Embed playable music sheets into actual CMS systems, starting with my own Arabic Real Book sheets - GitHub issue here.

• Reach a milestone with musicxml-mscx to convert full music scores from MusicXML to MuseScore format - focusing on correctly handling the bugs in MuseScore’s own MusicXML import.

• Explore the feasibility of using pre-rendered scores in musicxml-player to replace resource-intensive JavaScript notation engines - GitHub issue here.

• Replace my simplistic MIDI soft-synth in musicxml-player with a more complete one such as SpessaSynth - GitHub issue here.

• Explore multiplayer playback in musicxml-player - GitHub issue here.

• Support microtonality in MusicXML to MIDI conversion - GitHub issue here.

• Expand the groove conversion algorithm in musicxml-grooves to handle full MIDI files - GitHub issue here.

I hope to continue working on this project for a long time to come, and I welcome any and all contributions! :handshake:

Now that my rant’s over, let’s get into the technical details of this story.

Detecting the bug

The tell-tale sign that you hit this bug is when you enable the Reset button on a view’s exposed form, and instead of resetting the view filters, you get a blank page. The log says something like:

Drupal\Core\Form\EnforcedResponseException: in Drupal\Core\Form\FormBuilder->buildForm() (line 357 of /var/www/html/web/core/lib/Drupal/Core/Form/FormBuilder.php)
#0 /var/www/html/web/core/modules/views/src/Plugin/views/exposed_form/ExposedFormPluginBase.php(134): Drupal\Core\Form\FormBuilder->buildForm()
#1 /var/www/html/web/core/modules/views/src/ViewExecutable.php(1243): Drupal\views\Plugin\views\exposed_form\ExposedFormPluginBase->renderExposedForm()

Solution 1: Applying the patch

The relevant bug report has a patch that worked for me. I had to apply the patch manually to Drupal 9.x (please, don’t shoot me because I’m not in charge of our Drupal update schedule!!) but the code changes are exactly the same.

When you apply this patch, the Reset button works again. But clumsily: First, you see the URL changing to your current filters followed by &op=Reset, then the browser redirects to the page’s bare URL, thereby resetting the filters. This is of course a consequence of using Big Pipe, which optimizes page rendering by returning all cached blocks first, and deferring uncacheable blocks to be requested by the front-end. A marvel of engineering by Wim Leers! Still, the flickering leaves to be desired.

In my case, this particular view is the principal component of the page, so I feel OK disabling Big Pipe for just this page if at all possible. But how?

Solution 2a: Disable Big Pipe for a specific route

The standard approach to disabling Big Pipe is to inject the setting _no_big_pipe: TRUE in the options of the relevant route. If your page’s route is unique, then all you need is to follow the official guide on altering existing routes. Specifically, for a view page, the route is of the form view.view_id.page_id. So you would have something like the following:

  protected function alterRoutes(RouteCollection $collection) {

    // Disable Big Pipe for my view.
    if ($route = $collection->get('view.view_id.page_id')) {
      $route->setOption('_no_big_pipe', TRUE);
    }

  }

But in my case, the view is a block that’s embedded in a node page. I cannot simply alter the route entity.node.canonical, because this would disable it on 99% of the site!!

Solution 2b: Disable Big Pipe for a specific URL

I turned to good old Stack Overflow (technically, Drupal Answers) to query the hive-mind. Thanks to the ever-helpful and super-knowledgeable 4uk4 for his suggestion! Although I ended up taking a different approach, I will remember that I can override parameterized routes with specific ones because this will surely come in handy in the future.

The approach I ended up following is based on Wim Leer’s Big Pipe Strategy demo, where he catches every request in real-time and decides whether to return the Big Pipe placeholders or to ignore them. In my case, instead of examining the request’s query arguments for a specific “disable” signal, I compare the URI itself with the target page’s URL:

  public function processPlaceholders(array $placeholders) {

    // Ignore Big Pipe for my page URL.
    $current_uri = \Drupal::request()->getRequestUri();
    if (str_starts_with($current_uri, '/path/to/page-to-ignore')) {
      return [];
    }

    return $this->bigPipeStrategy->processPlaceholders($placeholders);
  }

Et voilà ! Another bug bites the dust.

To help debug this, I needed a way to associate user cookies with entries from the sessions table. I wrote a drush script to do exactly that: Given the value of the SESSXXX cookie in your browser, the script will find the corresponding sessions entry and dump its information, decoding the session metadata in the process:

$ drush scr export_sessions.php -- --cookie=5XvW3NGG8q1PcCrEXn676THvQBitaUwDiPw8XzAgXtihV43u

[
    {
        "uid": "1",
        "sid": "-Xcm0ar3mWcMhIhhBANA3K-jUx3JNOsu190LPEUzIN8",
        "hostname": "172.24.0.1",
        "timestamp": "1724179990",
        "session": "_sf2_attributes|a:1:{s:3:\"uid\";s:1:\"1\";}_sf2_meta|a:4:{s:1:\"u\";i:1724179990;s:1:\"c\";i:1723574737;s:1:\"l\";i:2000000;s:1:\"s\";s:43:\"OCpNT7IvSsWNfPeYXam7E7XFPTKqb-8qWPUTMe8MFlQ\";}",
        "sf2": [
            {
                "name": "attributes",
                "value": {
                    "uid": "1"
                }
            },
            {
                "name": "meta",
                "value": {
                    "u": 1724179990,
                    "c": 1723574737,
                    "l": 2000000,
                    "s": "OCpNT7IvSsWNfPeYXam7E7XFPTKqb-8qWPUTMe8MFlQ"
                }
            }
        ]
    }
]

The same script will dump ALL sessions if you don’t pass in a cookie value. Here’s the source code of export_sessions.php:

<?php

/**
 * Retrieve session entry for given cookie.
 * Based on https://drupal.stackexchange.com/a/231726/767
 */

use Drupal\Component\Utility\Crypt;

if (!empty($extra)) {
  if (!str_starts_with($extra[0], '--cookie=')) {
    die("Usage: drush scr export_sessions.php [-- --cookie=<value of SESSxxxx cookie>]\n");
  }
  else {
    $cookie = trim(str_replace('--cookie=', '', $extra[0]));
    $cookie = urldecode($cookie);
    $sid = Crypt::hashBase64($cookie);
  }
}
$connection = \Drupal::database();
if (isset($sid)) {
  $query = $connection->query('SELECT * FROM {sessions} WHERE sid = :sid', [':sid' => $sid]);
}
else {
  $query = $connection->query('SELECT * FROM {sessions}');
}

echo json_encode(array_map(function($session) {
  preg_match_all('/_sf2_(\w+)\|/', $session->session, $matches, PREG_OFFSET_CAPTURE | PREG_SET_ORDER);
  $session->sf2 = array_map(function($match, $index) use ($session, $matches) {
    $offset = $match[0][1] + strlen($match[0][0]);
    $length = $index + 1 < count($matches) ?
      $matches[$index + 1][0][1] - $offset:
      strlen($session->session) - 1 - $match[0][1];
    return [
      'name' => $match[1][0],
      'value' => unserialize(substr($session->session, $offset, $length)),
    ];
  }, $matches, array_keys($matches));
  return $session;
}, $query->fetchAll()), JSON_PRETTY_PRINT) . "\n";

That’s it. Short and sweet! :candy:

The problem with this issue is that the Cache API is called thousands of times per request - for pretty much every theming function participating in a page render. Further, the caching logic is complex as it involves combinations of cache tags, max-age settings, and various other mechanisms that affect the decisions of which caching tables to use and which caching headers to return in the HTTP response.

Trying to pinpoint the particular condition that caused the cache invalidation in this case using print statements would have been an inefficient and tedious process, and the client wouldn’t have liked to pay for that inefficiency. Kind of like the game of 20 questions, but with incomplete information and many, many decision branches. So I decided to bite the bullet and set up my Xdebug environment to catch the bug red-handed, so to speak. With its pants down, so to speak. To catch it in the act, so to speak.

Here’s a high level diagram of the various components at play here. I slightly modified it from the original at this other tutorial on the same topic.

My development environment is made up of the excellent Docker-based Drupal stack by Wodby. I can’t say enough good things about this framework, which has allowed me to start new Drupal projects, and even adopt legacy ones, on a solid footing without breaking a sweat. The architecture is simple, documentation is clear, customization is easy. I’ve been able to share development environments with team members using macOS and Windows systems with minimal changes.

The wodby/drupal-php image comes loaded with the Xdebug extension, and it’s “only” necessary to configure the right environment variables to activate it. I say “only” because many of the settings are non-obvious and required some experimentation before I could get them running, in addition to a VS Code configuration to match.

Here’s my current setup, in the main docker-compose.yml file running the full Drupal stack:

services:
  php:
    image: wodby/drupal-php:$PHP_TAG
    environment:
      PHP_EXTENSIONS_DISABLE: '' # or any value that does NOT include xdebug
      PHP_XDEBUG: 1
      PHP_XDEBUG_MODE: debug
      PHP_XDEBUG_START_WITH_REQUEST: yes
      PHP_XDEBUG_CLIENT_HOST: host.docker.internal
      PHP_XDEBUG_LOG: /tmp/php-xdebug.log
    extra_hosts:
      - "host.docker.internal:host-gateway"

Here’s what the non-obvious settings mean:

• PHP_EXTENSIONS_DISABLE: '' prevents the PHP container from disabling the xdebug extension - which for some reason is the default in wodby/php.
• PHP_XDEBUG_MODE: debug enables Xdebug step debugging, which is our purpose here.
• PHP_XDEBUG_START_WITH_REQUEST: yes means that Xdebug is activated at every request, automatically.
• PHP_XDEBUG_CLIENT_HOST: host.docker.internal is the all-important address of the machine running the debugging client - in my case, VS Code on my local machine. According to documentation, the name host.docker.internal is automatically available in Docker 18.03+ Mac/Win, but not on Linux. For Linux, we add the stanza extra_hosts: "host.docker.internal:host-gateway" which maps that domain name to Docker’s gateway IP, which is the Docker host, which is my laptop OS running VS Code :sweat_smile:

But that’s only half of the story. The other half is convincing VS Code to act as a debugging client to Xdebug. To do that, we use the PHP Debug VS Code extension and we customize the Launch configurations to add the Xdebug endpoint. Basically, we create a .vscode/launch.json file in the project root with the following content:

{
  "version": "0.2.0",
  "configurations": [
   {
      "name": "Listen for Xdebug",
      "type": "php",
      "request": "launch",
      "port": 9003,
      "pathMappings": {
        "/var/www/html/": "${workspaceFolder}/src"
      }
    }
  ]
}

Here’s what the non-obvious settings mean:

• "port": 9003 is the default port that Xdebug hits on the client, and that’s where VS Code should be listening for debug events.
• "pathMappings": { "/var/www/html/": "${workspaceFolder}/src" } maps the Docker filesystem path /var/www/html where the app resides to the actual host path "${workspaceFolder}/src" where ${workspaceFolder} is a VS Code variable.

With these in place, it should be now possible to place a breakpoint in, say, src/web/index.php (the Drupal main entrypoint) and catch every request! Select Run > Start Debugging or or click the Listen for Xdebug configuration in the bottom status bar. We are finally ready to start debugging the Drupal Cache API :ghost:

Troubleshooting

Of course, this setup didn’t come by without many failures and much head-scratching, perhaps even some teeth-clenching. If your 100% guaranteed breakpoint (like one in src/web/index.php) is not being hit, then it’s time to put on your sleuthing hat :detective:

Check that the Xdebug log is active and connected. Running docker-compose exec php tail -f /tmp/php-xdebug.log should show messages like:

[112] Log opened at 2023-10-25 06:42:12.885888
[112] [Step Debug] INFO: Connecting to configured address/port: host.docker.internal:9003.
[112] [Step Debug] INFO: Connected to debugging client: host.docker.internal:9003 (through xdebug.client_host/xdebug.client_port). :-)

Yes, that final smiley is part of the log :-)

If instead, you see a message like:

tail: can't open '/tmp/php-xdebug.log': No such file or directory
tail: no files

Then the Xdebug extension is not active, which could mean PHP_EXTENSIONS_DISABLE is still set to include xdebug.

If you see a sad smiley message like:

[112] [Step Debug] ERR: Could not connect to debugging client. Tried: host.docker.internal:9003 (through xdebug.client_host/xdebug.client_port) :-(

Then check that a connection can be established between Xdebug and VS Code. Running docker-compose exec php nc -zv host.docker.internal 9003 should return a successful response like:

host.docker.internal (172.17.0.1:9003) open

Anything else is a sign that the Docker container is unable to connect to the host on port 9003. Check your host.docker.internal name resolution, check the launch.json port setting, turn it off and on again, talk to your rubber duck - you know the drill!

Appendix: Annoying drush warnings

With Xdebug activated, you may be bombarded with multiple lines of warnings when running drush commands, especially when you are not debugging on the IDE side:

[20-Aug-2024 19:10:28 UTC] Xdebug: [Log Files] File '/tmp/php-xdebug.log' could not be opened.
[20-Aug-2024 19:10:28 UTC] Xdebug: [Step Debug] Could not connect to debugging client. Tried: host.docker.internal:9003 (through xdebug.client_host/xdebug.client_port).

In this case, you can run export XDEBUG_MODE=off in the bash session where you’re running drush, thereby deactivating Xdebug in the session, and saving a few bits from your eyes :sob:

The typical usage scenario is the following:

$ drush bamb default_db private_files
// => {
//    "status": "success",
//    "message": "Backup complete."
//}
$ drush bamls --files=private_files
// => {
//    "sources": [
//        {
//            "id": "default_db",
//            "label": "Default Drupal Database",
//            "type": "DefaultDB"
//        },
//        {
//            "id": "entire_site",
//            "label": "Entire Site (do not use)",
//            "type": "EntireSite"
//        },
//        {
//            "id": "private_files",
//            "label": "Private Files Directory",
//            "type": "DrupalFiles"
//        },
//        {
//            "id": "public_files",
//            "label": "Public Files Directory",
//            "type": "DrupalFiles"
//        },
//        {
//            "id": "ssot_database",
//            "label": "SSoT Database",
//            "type": "PostgreSQL"
//        }
//    ],
//    "destinations": [
//        {
//            "id": "private_files",
//            "label": "Private Files Directory",
//            "type": "Directory"
//        },
//        {
//            "id": "s3_bucket",
//            "label": "S3 Bucket",
//            "type": "awss3"
//        }
//    ],
//    "files": {
//        "private_files": [
//            {
//                "id": "backup-2023-01-27T15-44-19.sql.gz",
//                "filename": "prod-2023-01-27T15-44-19.sql.gz",
//                "filesize": 19499222,
//                "datestamp": 1674869134
//            }
//        ]
//    }
//}
$ drush bamr default_db private_files backup-2023-01-27T15-44-19.sql.gz
// => {
//    "status": "success",
//    "message": "Restore complete."
//}

And here’s the source for the command:

<?php

namespace Drush\Commands;

use Drush\Drush;
use Drush\Commands\DrushCommands;
use Drush\Boot\DrupalBootLevels;
use Drupal\backup_migrate\Core\Destination\ListableDestinationInterface;
use Symfony\Component\Console\Input\InputOption;

class BackupMigrateCommands extends DrushCommands
{
    /**
     * List sources and destinations.
     *
     * @command backup_migrate:list
     * @aliases bamls
     *
     * @option sources Flag to list sources (default: yes, use --no-sources to hide)
     * @option destinations Flag to list destinations (default: yes, use --no-destinations to hide)
     * @option files Flag to list files for a comma-separated list of destination identifiers (default: none)
     *
     * @param options
     *
     * @return string JSON listing of sources, destinations, files
     *
     */
    public function list(array $options = [
        'sources' => true,
        'destinations' => true,
        'files' => InputOption::VALUE_REQUIRED,
    ]): string {
        Drush::bootstrapManager()->doBootstrap(DrupalBootLevels::FULL);
        $bam = \backup_migrate_get_service_object();
        $output = [];
        if ($options['sources']) {
            $output['sources'] = array_reduce(array_keys($bam->sources()->getAll()), function($sources, $source_id) {
                $source = \Drupal::entityTypeManager()->getStorage('backup_migrate_source')->load($source_id);
                if ($source) {
                    $sources[] = [
                        'id' => $source_id,
                        'label' => $source->get('label'),
                        'type' => $source->get('type'),
                    ];
                }
                return $sources;
            }, []);
        }
        if ($options['destinations']) {
            $output['destinations'] = array_reduce(array_keys($bam->destinations()->getAll()), function($destinations, $destination_id) {
                $destination = \Drupal::entityTypeManager()->getStorage('backup_migrate_destination')->load($destination_id);
                if ($destination) {
                    $destinations[] = [
                        'id' => $destination_id,
                        'label' => $destination->get('label'),
                        'type' => $destination->get('type'),
                    ];
                }
                return $destinations;
            }, []);
        }
        if ($options['files']) {
            foreach(array_map('trim', explode(',', $options['files'])) as $destination_id) {
                $destination = $bam->destinations()->get($destination_id);
                if (!$destination) {
                    $this->logger()->warning(dt('The destination !id does not exist.', ['!id' => $destination_id]));
                    continue;
                }
                if (!$destination instanceof ListableDestinationInterface) {
                    $this->logger()->warning(dt('The destination !id is not listable.', ['!id' => $destination_id]));
                    continue;
                }
                try {
                    $files = $destination->listFiles();
                    $output['files'][$destination_id] = array_reduce(array_keys($files), function($files_info, $file_id) use($files) {
                        $files_info[] = array_merge([
                            'id' => $file_id,
                            'filename' => $files[$file_id]->getFullName(),
                        ], $files[$file_id]->getMetaAll());
                        return $files_info;
                    }, []);
                    usort($output['files'][$destination_id], function($file1, $file2) {
                        // TODO What if datestamp is not available?
                        $a = $file1['datestamp'];
                        $b = $file2['datestamp'];
                        if ($a == $b) return 0;
                        return ($a < $b) ? -1 : 1;
                    });
                }
                catch (\Exception $e) {
                    $this->logger()->error(dt('The destination !id caused an error: !error', [
                        '!id' => $destination_id,
                        '!error' => $e->getMessage()
                    ]));
                }
            }
        }
        return json_encode($output, JSON_PRETTY_PRINT);
    }

    /**
     * Backup.
     *
     * @command backup_migrate:backup
     * @aliases bamb
     *
     * @param source_id Identifier of the Backup Source.
     * @param destination_id Identifier of the Backup Destination.
     *
     * @return string Backup completion status
     *
     * @throws \Drupal\backup_migrate\Core\Exception\BackupMigrateException
     *
     */
    public function backup(
        $source_id,
        $destination_id
    ): string
    {
        Drush::bootstrapManager()->doBootstrap(DrupalBootLevels::FULL);
        $bam = \backup_migrate_get_service_object();
        $bam->backup($source_id, $destination_id);
        return json_encode([
            'status' => 'success',
            'message' => dt('Backup complete.')
        ], JSON_PRETTY_PRINT);
    }

    /**
     * Restore.
     *
     * @command backup_migrate:restore
     * @aliases bamr
     *
     * @param source_id Identifier of the Backup Source.
     * @param destination_id Identifier of the Backup Destination.
     * @param file_id optional Identifier of the Destination file.
     *
     * @return string Restore completion status
     *
     * @throws \Drupal\backup_migrate\Core\Exception\BackupMigrateException
     *
     */
    public function restore(
        $source_id,
        $destination_id,
        $file_id = null,
    ): string
    {
        Drush::bootstrapManager()->doBootstrap(DrupalBootLevels::FULL);
        $bam = \backup_migrate_get_service_object();
        $bam->restore($source_id, $destination_id, $file_id);
        return json_encode([
            'status' => 'success',
            'message' => dt('Restore complete.')
        ], JSON_PRETTY_PRINT);
    }
}

Once diagnosed, the fix was obvious: Detect that a tab is selected to refresh the charts contained therein. The following JavaScript file can be added to your theme as is and should handle the standard Bootstrap tabs (it also fixes the window resize event handling). It does depend on a small patch made to the charts_google module, to avoid leaking memory when the graph is redrawn:

(function ($, Drupal, once) {
  ("use strict");

  function redrawGoogleChart(element) {
    const contents = new Drupal.Charts.Contents();
    const chartId = element.id;
    if (Drupal.googleCharts.charts.hasOwnProperty(chartId)) {
      Drupal.googleCharts.charts[chartId].clearChart();
    }
    const dataAttributes = contents.getData(chartId);
    Drupal.googleCharts.drawChart(chartId, dataAttributes['visualization'], dataAttributes['data'], dataAttributes['options'])();
  }

  Drupal.behaviors.redrawGoogleCharts = {
    attach: function (context, settings) {
      $('.nav-link', context).on('shown.bs.tab', function (e) {
        if (Drupal.Charts && Drupal.googleCharts) {
          $('.charts-google', $(e.target).attr('data-bs-target')).each(function () {
            if (this.dataset.hasOwnProperty('chart')) {
              redrawGoogleChart(this);
            }
          });
        }
      });

      window.addEventListener('resize', function () {
        if (Drupal.Charts && Drupal.googleCharts) {
          Drupal.googleCharts.waitForFinalEvent(function () {
            $('.charts-google').each(function () {
              if (this.dataset.hasOwnProperty('chart')) {
                redrawGoogleChart(this);
              }
            });
          }, 200, 'google-charts-redraw');
        }
      });
    },
  };

})(jQuery, Drupal, once);

diff --git a/modules/charts_google/js/charts_google.js b/modules/charts_google/js/charts_google.js
index f7abe81..76143bc 100755
--- a/modules/charts_google/js/charts_google.js
+++ b/modules/charts_google/js/charts_google.js
@@ -6,7 +6,7 @@

   'use strict';

-  Drupal.googleCharts = Drupal.googleCharts || {charts: []};
+  Drupal.googleCharts = Drupal.googleCharts || {charts: {}};

   /**
    * Behavior to initialize Google Charts.
@@ -122,6 +122,7 @@
         options['colorAxis'] = {colors: colors};
       }
       chart.draw(data, options);
+      Drupal.googleCharts.charts[chartId] = chart;
     };
   };