AI Inference #

Budo runs models on the device, with no server involved. sys.neural runs ONNX models through ONNX Runtime: classifiers, detectors, small regressors, anything exported to .onnx. sys.llamacpp runs GGUF language models through llama.cpp and streams their replies as a chat.

Both engines are optional parts of a Budo build, so every app should check that the one it needs is present and show a clear message when it is not.

Neural inference with ONNX #

Enable it #

Opt in from app.json. Without this flag, sys.neural is not defined.

app.json
{
  "neural": true
}

Then check that this build of Budo includes ONNX Runtime. It is available in desktop and Android builds made with neural support, and never on web.

main.js

// Check both the build capability and the app.json opt-in.
if (!sys.capabilities.neural.available || typeof sys.neural === 'undefined') {
  // Show a fallback instead of failing later.
  console.log('No neural runtime in this build');
}

Load a model #

sys.neural.loadModel(path) loads a .onnx file from the project directory. The path is relative; absolute paths and .. are rejected. To load a model you downloaded or generated, pass its bytes to sys.neural.loadModelFromBuffer(buffer), which takes an ArrayBuffer or typed array. Both return a model id and throw if the model cannot be loaded.

getModelInfo(model) describes the model: its inputs and outputs, each with a name, a shape (where -1 marks a dynamic dimension), and a dtype of float32, int32, int64, or uint8.

main.js
// Load the model bundled with the app.
const model = sys.neural.loadModel('models/classifier.onnx');
// Read the declared inputs and outputs.
const info = sys.neural.getModelInfo(model);
// Use the first input and output names.
const inputName = info.inputs[0].name;
const outputName = info.outputs[0].name;

Run inference #

Pass the inputs to run(model, inputs) as an object that maps each input name to a typed array: Float32Array, Int32Array, BigInt64Array, or Uint8Array, matching the input's dtype. It returns an object that maps each output name to a typed array.

main.js
// Allocate a 1x3x224x224 image tensor.
const data = new Float32Array(1 * 3 * 224 * 224);
// Run the model with one named input.
const result = sys.neural.run(model, { [inputName]: data });
// Read the output tensor by name.
const output = result[outputName];

You can also stage inputs one at a time, then run and read outputs separately. This is the same flow Lua and WebAssembly use.

main.js
// Stage the input tensor.
sys.neural.setInput(model, inputName, data);
// Run the model on the staged inputs.
sys.neural.run(model);
// Read the output tensor by name.
const output = sys.neural.getOutput(model, outputName);

In Lua, tensors are tables with data and an optional shape, and outputs come back with a dtype as well; see Lua and WebAssembly. WebAssembly modules use the staged flow with float32 tensors.

Release a model #

Unload a model when the app no longer needs it, for example when the user switches models. Models take real memory, and phones have much less of it than desktops.

main.js
// Free the model and its session.
sys.neural.unloadModel(model);

Local chat with llama.cpp #

sys.llamacpp loads a GGUF language model and holds chat conversations with it, streaming each reply as it is generated. It is available from JavaScript, TypeScript, and Lua in desktop builds made with llama.cpp support and in Android builds. It is not available on web or from WebAssembly. Inference always works on the CPU; macOS builds also use the GPU through Metal.

main.js
// Check that this build includes llama.cpp.
if (!sys.capabilities.llamacpp.available || typeof sys.llamacpp === 'undefined') {
  console.log('This build has no llama.cpp support');
}

Load a model #

Model files are large, so loadModel(path, options, callback) loads them in the background and calls you back with the model, or with null and an Error. The path must start with assets/ or files/, like any sys.files path. A model the app downloads belongs in files/.

main.js
// Load a GGUF model from the app's storage.
sys.llamacpp.loadModel('files/models/model.gguf', { device: 'auto' }, (model, error) => {
  // model is null when loading failed.
  if (!model) {
    console.log(error.message);
    return;
  }
  // getInfo() describes the model and where it runs.
  const info = model.getInfo();
  console.log(info.name + ' on ' + info.backend);
});

The options object is optional. Its fields are:

device: "auto" (the default) picks the best device and falls back to the CPU; "cpu" forces the CPU; or a device id from sys.llamacpp.getDevices().
contextSize: the context length in tokens, 2048 by default.
gpuLayers, threads, batchSize: a number, or "auto".
useMmap: map the file instead of reading it into memory, true by default.
allowFallback: fall back to another device when the requested one fails.

sys.llamacpp.getDevices() lists the devices llama.cpp can use, each with an id, backend, name, available, memoryBytes, and a reason when it is unavailable.

Chat #

model.createChat({ systemPrompt, contextSize }) starts a conversation. chat.send(text, options, callbacks) adds the user's message and streams the reply through three callbacks: onText for each new piece of text, onComplete when the reply ends, and onError if generation fails. The chat keeps the history, so the next send continues the conversation.

main.js
// Start a conversation with a system prompt.
const chat = model.createChat({ systemPrompt: 'Answer concisely.' });

// Send a message and stream the reply.
const generation = chat.send('Name three primary colors.', { maxTokens: 128 }, {
  // Called with each new piece of the reply.
  onText(text) { console.log(text); },
  // Called once with the full reply and statistics.
  onComplete(result) { console.log(result.finishReason, result.generatedTokensPerSecond); },
  // Called if generation fails.
  onError(error) { console.log(error.message); }
});

The generation options are maxTokens (256 by default), minTokens, temperature, topK, topP, minP, repetitionPenalty, seed, and stop, an array of strings that end the reply. onComplete receives the full text, the token counts and speeds, and a finishReason of "stop", "length", or "cancelled".

send returns a generation whose running property turns false when it ends; generation.cancel() stops it early. chat.getMessages() returns the history as { role, content } objects, chat.clear() starts over, and chat.destroy() and model.destroy() release memory when you are done.

On Android, generation is cancelled when the app is paused.

The 18_llamacpp_chat example builds a complete chat app: it downloads a small model into files/models/ in chunks, then loads it and streams replies into a scrolling transcript.

Designing for availability #

Whether inference is available depends on how Budo was built and on the platform: neither engine runs on web, and ONNX also needs the app.json opt-in. Check the capability when the app starts, and give the user a clear state ("This device cannot run the model") rather than an error later.