Audio and MIDI #

Budo gives you several audio tools, each suited to a different job. Pick the simplest one that does what you need:

• Oscillators play tones and simple synthesizer voices, with an optional envelope. Use them for beeps, UI feedback, and instruments.
• Buffers hold short sounds in memory: samples you generate, or WAV, MP3, Ogg Vorbis, and FLAC files loaded with sys.audio.loadBuffer. Use them for sound effects and short loops.
• Streams call your function for every chunk of audio, so your code writes the samples (a synthesizer, a sequencer, a mixer) or receives them from the microphone.
• Decoders read long audio files a block at a time, usually into an output stream, so music of any length plays without being loaded whole.
• Analysis computes spectra and detects pitch natively, for meters, tuners, and visualizers.
• MIDI talks to MIDI keyboards and controllers, and to other apps over the network with RTP-MIDI.

Oscillators, buffers, and MIDI are available in JavaScript and Lua. Streams, decoders, and analysis are available in JavaScript only.

Starting audio #

The audio context starts stopped. Calls that play sound, such as startOscillator, noteOn, playBuffer, and openOutput, start it for you, so most apps never call sys.audio.start() themselves.

// Start the audio context explicitly (optional: playback calls do it for you).
sys.audio.start();
// Set the master volume, from 0.0 to 1.0 (the default is 0.5).
sys.audio.setMasterGain(0.8);

The master gain applies to oscillators and buffers, and is clamped to the range 0.0 to 1.0. Browsers only let a page play sound after the user has interacted with it, so on the web, start sound from a tap, a click, or a key press.

Oscillators #

An oscillator is a tone generator identified by an integer ID. Create it once, set its waveform, frequency, and gain, then start and stop it as often as you need.

// Create an oscillator and keep its ID.
const osc = sys.audio.createOscillator();
// Choose the waveform.
sys.audio.setOscillatorType(osc, 'sine');
// Tune it to A3 (220 Hz).
sys.audio.setOscillatorFrequency(osc, 220);
// Set its volume, from 0.0 to 1.0.
sys.audio.setOscillatorGain(osc, 0.25);
// Start playing; it plays until stopOscillator.
sys.audio.startOscillator(osc);

The waveform is one of 'sine', 'square', 'sawtooth', 'triangle', or 'noise', or the matching constant (sys.audio.SINE, SQUARE, SAWTOOTH, TRIANGLE, NOISE). setOscillatorDetune(osc, cents) shifts the pitch by hundredths of a semitone. Every setter applies immediately, including while the oscillator plays, so you can glide a pitch from your frame function. Release an oscillator you no longer need with destroyOscillator.

Envelopes #

An ADSR envelope shapes each note for you: it rises over the attack, falls over the decay to the sustain level, holds while the note is on, and fades over the release. noteOn starts a note and noteOff releases it.

// Attack 10 ms, decay 120 ms, sustain at 70%, release 250 ms.
sys.audio.setOscillatorEnvelope(osc, 0.01, 0.12, 0.7, 0.25);
// Start the note: attack, then decay to the sustain level.
sys.audio.noteOn(osc);
// Release the note 300 milliseconds later.
sys.timer.once(300, () => sys.audio.noteOff(osc));

Times are in seconds and the sustain level goes from 0.0 to 1.0. sys.audio.midiToFreq(note) converts a MIDI note number to a frequency: midiToFreq(69) is 440, the A above middle C.

Buffers #

A buffer holds sample data in memory. Play it with playBuffer(buffer, loop, gain), which returns a playback ID; stop that playback with stopBuffer, and free the buffer with destroyBuffer when you are done with it. The same buffer can play several times at once.

Generated sounds #

To synthesize a sound ahead of time, create a buffer and fill it with samples between -1.0 and 1.0.

// The buffer's sample rate, in samples per second.
const sampleRate = 44100;
// One second of mono samples, in a plain array.
const samples = new Array(sampleRate);

// Fill it with a quiet 110 Hz sine wave.
for (let i = 0; i < samples.length; i++) {
  // The sample index, turned into a phase angle.
  samples[i] = Math.sin((i / sampleRate) * Math.PI * 2 * 110) * 0.25;
}

// Create a mono buffer that holds the samples.
const buffer = sys.audio.createBuffer(sampleRate, 1, samples.length);
// Copy the samples in, starting at sample 0.
sys.audio.setBufferData(buffer, samples, 0);
// Play it once (no loop) at full gain.
const playback = sys.audio.playBuffer(buffer, false, 1.0);

createBuffer(sampleRate, channels, frames) takes 1 or 2 channels; a stereo buffer holds interleaved samples (left, right, left, …). setBufferData takes a plain array of numbers, not a Float32Array.

Audio files #

For sound effects, music cues, and voice clips, load a WAV, MP3, Ogg Vorbis, or FLAC file with sys.audio.loadBuffer(path). Budo decodes the whole file into a buffer and returns its ID, which plays, loops, and stops like any other buffer.

// Decode a file from the project folder into a buffer.
const hit = sys.audio.loadBuffer('sounds/hit.ogg');

if (hit < 0) {
  // The file is missing, unreadable, or not a supported format.
  console.log('Cannot load hit.ogg: ' + sys.audio.getError());
} else {
  // Play it once at 80% gain.
  sys.audio.playBuffer(hit, false, 0.8);
}

The path is relative to your project folder, such as sounds/hit.ogg or music/theme.flac. Absolute paths, empty segments, and . or .. segments are rejected, which keeps an app inside its own files. When loading fails, loadBuffer returns -1 and sys.audio.getError() says why.

To decode bytes you already have, from a file in files/, a network response, or generated data, pass an ArrayBuffer to sys.audio.loadBufferFromBuffer.

// Read a downloaded file, then decode its bytes.
const bytes = sys.files.readBinary('files/downloads/voice.mp3');
const voice = sys.audio.loadBufferFromBuffer(bytes);
// Report a decoding failure.
if (voice < 0) console.log(sys.audio.getError());

To loop a sound, pass true as the second argument of playBuffer, and stop it with the playback ID.

// Load a background loop.
const ambience = sys.audio.loadBuffer('audio/ambience.mp3');
// Play it in a loop at a low gain.
const ambiencePlayback = sys.audio.playBuffer(ambience, true, 0.35);

// When the scene changes: stop the loop and free the buffer.
sys.audio.stopBuffer(ambiencePlayback);
sys.audio.destroyBuffer(ambience);

loadBuffer decodes the whole file before it returns, so load sounds while entering a scene or opening a tool, not in the middle of an animation. For long music, use a decoder instead.

Streams #

A stream calls a function you give it for each chunk of audio. For an output, Budo calls it whenever the stream needs more sound, and your function writes the next samples. For an input, Budo calls it with each chunk the microphone captured.

// The oscillator's phase, kept between chunks.
let phase = 0;

// Open a stereo output; the callback fills each chunk.
const tone = sys.audio.openOutput({ channels: 2 }, (buffer, info) => {
  // The phase step of a 440 Hz tone at the device's rate.
  const step = 2 * Math.PI * 440 / info.sampleRate;
  for (let i = 0; i < info.frames; i++) {
    const value = Math.sin(phase) * 0.2;
    buffer[i * 2] = value;     // left
    buffer[i * 2 + 1] = value; // right
    phase += step;
  }
});

// Later: stop the stream.
sys.audio.closeOutput(tone);

openOutput(options, callback) and openInput(options, callback) return a stream handle, or -1 on failure (see sys.audio.getError()). The options are optional; channels is 1 or 2, and defaults to 2 for outputs and 1 for inputs.

Each call handles one chunk of 256 frames. buffer is a Float32Array of info.frames * info.channels interleaved samples (left, right, left, …). An output's buffer starts silent, so whatever you leave untouched plays as silence. info also gives:

• sampleRate: the device's rate, the same as sys.audio.getSampleRate();
• time: the stream time of the chunk's first frame, in seconds;
• latency: the current size of the stream's queue, in seconds;
• underruns (outputs): how many times the queue ran dry because the app was late; a dropout counts once, however long it lasts;
• overruns (inputs): how many times captured audio was dropped because the app fell behind.

An output plays exactly the samples you write: the master gain does not apply. It is mixed with oscillators and buffers.

Latency #

Between your function and the device sits a queue of audio. When you leave out latencyMs, an output's queue is adaptive: it starts small (20 ms, 40 ms on Android, 30 ms on the web), grows after an underrun by the frames that were missing, and shrinks back when playback has been steady. Read info.latency to see its current size.

Pass latencyMs (1 to 2000) for a fixed queue instead. A smaller queue responds faster; a larger one survives slower frames.

// A fixed 100 ms queue, for a sequencer that must never drop out.
const sequencer = sys.audio.openOutput({ channels: 2, latencyMs: 100 }, fillSequencer);

When callbacks run #

Stream callbacks run on the app's thread, before and after each frame's drawing, as many times as needed to keep the queue filled. Keep them quick, and avoid long work in your frame function, because the stream is only refilled between frames. If a callback throws, Budo stops calling it and sys.audio.getError() reports the error.

// The callback reads `note`, which the frame function updates.
stream = sys.audio.openOutput((buffer, info) => {
  // Advance the phase by the current note's step.
  const step = 2 * Math.PI * note / info.sampleRate;
  for (let i = 0; i < info.frames; i++) {
    const value = Math.sin(phase) * 0.15;
    buffer[i * 2] = value;     // left
    buffer[i * 2 + 1] = value; // right
    phase += step;
  }
});

Recording #

An input stream works the same way: samples holds the chunk the microphone captured. It is valid only during the call, so copy what you want to keep.

// The microphone's level, smoothed, from 0 to 1.
let level = 0;

// Open a mono input; the callback receives each captured chunk.
const mic = sys.audio.openInput({ channels: 1 }, (samples, info) => {
  // Find the chunk's peak.
  let peak = 0;
  for (let i = 0; i < samples.length; i++) peak = Math.max(peak, Math.abs(samples[i]));
  // Jump up to a new peak, fall back slowly.
  level = Math.max(peak, level * 0.9);
});

// -1 when the microphone is unavailable or permission is pending.
if (mic < 0) console.log(sys.audio.getError());

Recording needs the user's permission. On Android, list RECORD_AUDIO in the permissions of app.json. The first openInput asks the user and returns -1, so call it again once permission is granted, for example when the app regains focus. On the web, the page must be served over https (or from localhost), and the browser asks the user. Close the microphone with closeInput when you are done.

examples/20_microphone records, plays back, and monitors the microphone with input and output streams.

Decoders #

A decoder turns a WAV, MP3, Ogg Vorbis, or FLAC file into samples a block at a time. Use one to play music of any length, or to process a file too large to hold in memory. Ask for the device's rate and channel count, and decode straight into an output stream's buffer.

// Open a song for block reads, decoded at the device's rate, in stereo.
const song = sys.audio.openDecoder('assets/music/mix.mp3', {
  sampleRate: sys.audio.getSampleRate(),
  channels: 2,
});
if (song < 0) console.log(sys.audio.getError());

// Its length in seconds, among other details.
const { duration } = sys.audio.getDecoderInfo(song);

// Each chunk decodes the next frames; at the end, the rest stays silent.
const player = sys.audio.openOutput({ channels: 2 }, (buffer) => {
  sys.audio.decode(song, buffer);
});

// Jump to 1 minute 30 seconds.
sys.audio.seekDecoder(song, 90);

openDecoder(source, options) takes a path in the virtual file system (assets/... for files bundled with the app, files/... for the app's storage, as with sys.files), or bytes already in memory as an ArrayBuffer or typed array. It returns a decoder handle, or -1 on failure. A file decoder reads the file in blocks, so it can seek and knows its duration. The options are sampleRate (0, the default, keeps the file's own rate) and channels (1 or 2, default 2).

decode(decoder, out) fills the Float32Array out with interleaved frames and returns how many frames it decoded: 0 at the end, and -1 on error. getDecoderInfo(decoder) returns sampleRate, channels, sourceSampleRate, sourceChannels, duration and position in seconds, and the flags ready, ended, needsData, and seekable. seekDecoder(decoder, seconds) moves a file or memory decoder.

Push decoders #

A push decoder decodes bytes you give it as they arrive: the blocks of sys.files.read, a network response, or generated data. Create it with createDecoder(options) and pass bytes with feedDecoder(decoder, bytes, end), setting end with the last bytes. It reads straight through, so it cannot seek and reports no duration (-1).

// Decode a recording a block at a time, without holding the file in memory.
const file = sys.files.openRead('files/recording.flac');
const decoder = sys.audio.createDecoder({ sampleRate: sys.audio.getSampleRate() });
// Room for 4096 stereo frames.
const block = new Float32Array(4096 * 2);

for (;;) {
  // Read the next 64 KiB of the file; an empty block means the end.
  const bytes = sys.files.read(file, 65536);
  if (bytes === null) break; // read error: see sys.files.getError()
  const end = bytes.byteLength === 0;
  // Give the bytes to the decoder; the last call says "end".
  sys.audio.feedDecoder(decoder, bytes, end);
  // Take every frame the decoder can produce from them.
  let frames;
  while ((frames = sys.audio.decode(decoder, block)) > 0) {
    // Use block.subarray(0, frames * 2) here.
  }
  if (end) break;
}

// Release the file and the decoder.
sys.files.closeRead(file);
sys.audio.closeDecoder(decoder);

For a push decoder, decode also returns 0 when it needs more bytes; getDecoderInfo(decoder).needsData tells that case from the end. Close every decoder with closeDecoder when you are done: a file decoder keeps its file open until then.

examples/07_file_explorer plays audio files of any size with a decoder and an output stream, with a seek bar and looping.

Analysis #

sys.audio includes native signal analysis, much faster than the same maths in JavaScript. The functions take Float32Arrays of mono samples, such as the chunks of an input stream.

// Analysis size: the last 2048 samples.
const size = 2048;
const history = new Float32Array(size);
// One level per frequency bin, in dB.
const levels = new Float32Array(size / 2);
let pitch = 0;

sys.audio.openInput({ channels: 1 }, (samples) => {
  // Shift the history and append the new chunk.
  history.copyWithin(0, samples.length);
  history.set(samples, size - samples.length);
});

function frame() {
  // The pitch in Hz, or 0 when there is none.
  pitch = sys.audio.detectPitch(history).frequency;
  // The spectrum, in dB per bin.
  sys.audio.getSpectrum(history, levels);
  draw();
  sys.animation.requestFrame(frame);
}
sys.animation.requestFrame(frame);

getSpectrum(samples, outDb) writes the Hann-windowed spectrum of samples, in dB, to outDb and returns it. The length of samples is a power of two (4 to 2^20) and outDb holds half as many values; bin k is the frequency k * sampleRate / samples.length. A full-scale sine reads about 0 dB, and silence -180 dB.

detectPitch(samples, sampleRate, options) finds the pitch of a voice or an instrument with the McLeod pitch method, which finds the fundamental even when a harmonic is louder. sampleRate defaults to the device's rate. It returns { frequency, clarity, level }: frequency in Hz, or 0 when there is no pitch (noise, silence, or out of range); clarity, from 0 to 1, how periodic the sound is (above 0.9 for a clear note); and level, the RMS level in dB. The options are:

• minFrequency and maxFrequency: the range of pitches reported, 50 to 4000 Hz by default;
• minClarity: the clarity below which there is no pitch, 0.7 by default;
• minLevel: the RMS level, in dB, below which there is no pitch, -60 by default.

samples holds 16 or more values. The lowest pitch it can find has a period of half the samples: 2048 samples at 48 kHz reach down to 47 Hz.

fft(re, im, inverse) transforms two Float32Arrays, the real and imaginary parts, in place, for your own analysis. Both have the same power-of-two length (2 to 2^20). The inverse is scaled by 1 / length, so a forward transform followed by an inverse one gives back the input.

examples/20_microphone uses these functions for a spectrogram, a pitch readout, and a tuner.

MIDI devices #

sys.midi reads from and sends to MIDI devices: keyboards, controllers, synthesizers, and other apps. Start by scanning for devices.

// Scan for devices before showing a picker.
sys.midi.refreshDevices();

// List the inputs: each has a position and a name.
for (const device of sys.midi.getInputDevices()) {
  console.log(device.id + ': ' + device.name);
}

A device's id is its position in the list, which is what openInput and openOutput take. Positions shift when a device is plugged in or out, so to keep using the same device, remember its name and look it up again when the devices change.

Device changes #

Register a listener with onDevicesChanged to hear when devices are connected or removed. It is not called at registration, only on later changes. A new registration replaces the previous listener; pass null to unsubscribe.

// React to devices being connected or removed.
sys.midi.onDevicesChanged((event) => {
  // Fresh lists of inputs and outputs.
  console.log(event.inputs.length + ' inputs, ' + event.outputs.length + ' outputs');
  // Reopen the devices you use here.
});

The event carries fresh inputs and outputs lists and a generation number that increases with each change. It is called on the app's thread, between frames, and also reports a device replaced by another, even when the number of devices stays the same. In Lua, the same listener is sys.midi.onDevicesChanged(function(event) ... end), and nil unsubscribes.

Receiving messages #

openInput(position, callback) opens an input and calls your function with each message. It returns an input handle, or -1 on failure.

// Open the first input.
const input = sys.midi.openInput(0, (message) => {
  // A Note On with a velocity above 0 is a played note.
  if (message.type === sys.midi.NOTE_ON && message.data2 > 0) {
    console.log('Note ' + message.data1 + ', velocity ' + message.data2);
  }
});

A message has status, data1, and data2 bytes, its type (sys.midi.NOTE_ON, NOTE_OFF, CONTROL_CHANGE, PROGRAM_CHANGE, PITCH_BEND, CHANNEL_PRESSURE, POLY_PRESSURE, or SYSEX), and its channel (0 to 15, or -1 for system messages). message.timestamp is the receive time in microseconds from a monotonic clock, or 0 when the platform gives none.

Messages arrive in the order the device sent them, System Exclusive included, between frames. A SysEx message (type === sys.midi.SYSEX) carries its complete bytes, F0 through F7, in message.data, even when the device split it across many packets. SysEx messages up to 65535 bytes are delivered; longer ones are dropped. Running status is expanded, and real-time messages (clock, active sensing) are not reported. Up to 1024 messages, 32 of them SysEx, are kept between two frames.

Sending messages #

openOutput(position) opens an output and returns its handle, or -1 on failure. Channels are 0 to 15, and each send returns true on success.

// Open the first output.
const output = sys.midi.openOutput(0);
// Play middle C (note 60) on channel 0 at velocity 100.
sys.midi.noteOn(output, 0, 60, 100);
// Release it.
sys.midi.noteOff(output, 0, 60, 0);
// Set the modulation wheel to half.
sys.midi.controlChange(output, 0, sys.midi.CC_MOD_WHEEL, 64);
// Bend the pitch up halfway (-8192 to 8191).
sys.midi.pitchBend(output, 0, 4096);
// Send a raw System Exclusive message (up to 65535 bytes).
sys.midi.sendRaw(output, [0xf0, 0x7e, 0x7f, 0x06, 0x01, 0xf7]);

The output also has programChange, channelPressure, polyPressure, and sendMessage(output, status, data1, data2) for any three-byte message. sys.midi.getType(status) and getChannel(status) split a status byte. Close handles with closeInput and closeOutput when you are done.

examples/11_midi shows incoming messages, sends notes, and handles SysEx; examples/21_midi_tempo follows the devices as they change.

RTP-MIDI #

RTP-MIDI exchanges MIDI messages with other computers and apps over the network. Create a session, connect it to a peer, and send and receive through it.

// Create a session named "Jam"; port 0 lets Budo choose.
const session = sys.midi.createSession('Jam', 0);

// Receive the session's messages.
sys.midi.onSessionMessage(session, (message) => {
  console.log('Network MIDI: status ' + message.status);
});

// Connect to a peer.
sys.midi.connectSession(session, '192.168.1.20', 5004);
// Send a note through the session.
sys.midi.sessionNoteOn(session, 0, 64, 100);

Messages from a session have the same shape as device messages, with network set to true. A session also has sessionNoteOff, sessionControlChange, sessionSend, and sessionSendRaw. getSessions() lists the open sessions with their name, port, state, and peerCount, and destroySession closes one.

RTP-MIDI uses UDP, so it is not available on the web, where browsers do not expose UDP sockets.

Platform notes #

MIDI depends on the platform's MIDI support: check sys.capabilities.midi.available and show a clear state when it is missing. On the web, MIDI uses the browser's Web MIDI API: the browser may ask the user for permission, and the device lists stay empty until it is granted, so listen with onDevicesChanged.

Streams run at the device's rate (sys.audio.getSampleRate()): usually 44.1 or 48 kHz on desktop, the phone's own rate on Android, and the browser's on the web, where an AudioWorklet plays them (a ScriptProcessor where worklets are unavailable). Their callbacks run between frames, so a browser tab in the background, which gets no frames, lets them run dry.