zai-bot
b08490e80e
Sustained near-digital quiet on the overall level (< 0.006 for 300 ms, sticky until a decisive 0.015) now places an immediate HOLD on the beat tracker: nothing fires (real or predicted -- previously the predictor kept faking beats for up to ~2 s into any silence), the tempo estimate and confidence freeze exactly where they are, and beatphase freezes for scripts instead of saturating at 'beat imminent'. Decisive audio releases the hold instantly and the grid resumes from the phase the audio went quiet (captured before the debounce window, so a pause/resume keeps perfect time); after a hard cut the comb + first real beats re-align within a couple of periods. 'Set Beat' taps release the hold outright. Waveform/band data is never gated -- it still flows (it is simply ~0, because the audio is gone). - BeatFrame.silent carries the state; scripts get the silent variable (1 while held); the beat indicator shows an amber HOLD badge + amber phase ring while held. - rust-agent/src/beat.rs + engine.rs: same gate ported for parity (its bass level doubles as the silence signal; NOT compile-verified -- no cargo in this sandbox). - Verified: headless 26/26 assertions on the real BeatTracker (music -> silence -> in-phase resume -> onsetless pad -> tap; the pre-fix fake-beat window 12.8-15.5 s is empty) and browser E2E on the live app with a generated 120 BPM + silence looping WAV (HOLD badge tracks the silent spans exactly across two loop cycles; console clean).
216 lines
12 KiB
TypeScript
Executable file
216 lines
12 KiB
TypeScript
Executable file
// AVA — Rust / Vulkan native-export blueprint
|
||
//
|
||
// Documents the architecture for the native port of this engine (the web
|
||
// edition you are looking at is the reference implementation). The scaffold
|
||
// lives in /rust-agent at the repository root and can be downloaded as a zip
|
||
// from the button above. The UI renders these sections in the "Native" tab
|
||
// of the control panel.
|
||
|
||
export interface BlueprintCodeBlock {
|
||
label: string
|
||
language: string
|
||
content: string
|
||
}
|
||
|
||
export interface BlueprintSection {
|
||
title: string
|
||
body: string[]
|
||
code?: BlueprintCodeBlock[]
|
||
bullets?: string[]
|
||
}
|
||
|
||
export interface Blueprint {
|
||
title: string
|
||
intro: string
|
||
downloadNote: string
|
||
sections: BlueprintSection[]
|
||
}
|
||
|
||
export const BLUEPRINT: Blueprint = {
|
||
title: 'Rust / Vulkan Native Export',
|
||
intro:
|
||
'This web edition is the reference implementation of the AVA engine. The native edition (rust-agent/) shares the architecture and speaks the exact same .ava script language -- every preset you write here runs unmodified on the Rust/Vulkan build, and the engine math (band smoothing, beat prediction, expression semantics) was verified value-for-value against this implementation.',
|
||
downloadNote:
|
||
'Download the complete cargo project (scaffold + all six presets) as a zip and build it on Linux, Windows or macOS. Button above, or GET /api/rust-scaffold.',
|
||
sections: [
|
||
{
|
||
title: 'Architecture at a glance',
|
||
body: [
|
||
'One cargo crate, one binary: ava-agent. Every module of the web engine has a native counterpart with the same responsibilities and the same constants:',
|
||
],
|
||
bullets: [
|
||
'src/lib/ava/audio.ts (WebAudio AnalyserNode) ↔ rust src/audio.rs + src/spectrum.rs -- cpal capture, ring buffer, rustfft 8192-point FFT, band averaging',
|
||
'src/lib/ava/beat.ts ↔ rust src/beat.rs -- statement-for-statement port: same history/interval/tap constants, same EMA tempo lock, same phase-locked predictor',
|
||
'src/lib/ava/expression.ts + script.ts ↔ rust src/script/{lexer,parser,env,mod}.rs -- the same AVS-eel flavored grammar and the full 39-function set',
|
||
'src/lib/ava/engine.ts (rAF loop + Canvas2D) ↔ rust src/engine.rs + src/render/* -- the same band → beat → [frame] → [beat] → [point] × n pipeline on a wgpu frame loop',
|
||
'src/lib/ava/presets.ts ↔ rust presets/*.ava -- the five scene classics plus the starter template, byte-identical files on both engines',
|
||
],
|
||
},
|
||
{
|
||
title: 'Why wgpu (and the Vulkan story)',
|
||
body: [
|
||
'The target graphics API is Vulkan. wgpu is the Rust GPU abstraction that speaks native Vulkan on Linux and Windows -- and keeps one codebase running everywhere:',
|
||
'On Linux and Windows the agent requests a Vulkan adapter first; if the machine has no Vulkan loader it falls back to the other primary backends with a warning. On macOS -- which has no native Vulkan -- wgpu translates to Metal automatically, so the default build just works. For strict Vulkan on macOS, install the Vulkan SDK and build with --features vulkan-force: the agent then requests a genuine Vulkan adapter through MoltenVK instead of Metal.',
|
||
'One codebase, three platforms, Vulkan-first. The backend is chosen at startup in render/mod.rs (backend_attempts), and the chosen adapter is printed to the console.',
|
||
],
|
||
},
|
||
{
|
||
title: 'Audio pipeline',
|
||
body: [
|
||
'cpal captures the default input device (48 kHz mono preferred, any channel count averaged to mono) on its own callback thread into a lock-guarded ring buffer. Once per rendered frame, AudioProcessor::poll() windows the last 8192 samples, applies a Hann window, runs rustfft, and averages the magnitudes over the user-tunable band ranges -- bass 20–100 Hz, mid 700–2000 Hz, high 7000–16000 Hz -- with the same fast-attack (tau 35 ms) / slow-release (tau 220 ms) smoothing that produces the *_att variables and feeds the beat detector. Magnitudes are unit-normalized by a fixed calibration constant so band gains and thresholds transfer 1:1 from the web engine.',
|
||
],
|
||
code: [
|
||
{
|
||
label: 'rust-agent/src/spectrum.rs -- Hz ranges → band levels',
|
||
language: 'rust',
|
||
content: `/// Calibration: a full-scale sinusoid through the Hann window lands at a
|
||
/// magnitude of ≈ FFT_SIZE/4 (≈ 2048). Dividing by this constant maps it to
|
||
/// 1.0 — the same 0..1 range the web engine gets from the AnalyserNode's
|
||
/// byte spectrum (\`value / 255\`), so band gains, attack/release smoothing and
|
||
/// beat thresholds transfer unchanged between the two engines.
|
||
const MAG_TO_UNIT: f32 = 1.0 / (FFT_SIZE as f32 / 4.0);
|
||
|
||
/// Hz per FFT bin (web: \`sampleRate / 2 / frequencyBinCount\` — identical).
|
||
pub fn bin_hz(sample_rate: f32, fft_size: usize) -> f32 {
|
||
sample_rate / fft_size as f32
|
||
}
|
||
|
||
/// Average unit-magnitude over an inclusive Hz range — the \`levelOf()\`
|
||
/// closure inside \`getBands()\` in audio.ts.
|
||
pub fn band_level(mags: &[f32], lo_hz: f32, hi_hz: f32, sample_rate: f32) -> f32 {
|
||
let bh = bin_hz(sample_rate, mags.len() * 2);
|
||
let lo = ((lo_hz / bh) as usize).max(1);
|
||
let hi = ((hi_hz / bh).ceil() as usize).min(mags.len() - 1);
|
||
if hi <= lo {
|
||
return 0.0;
|
||
}
|
||
mags[lo..=hi].iter().sum::<f32>() / (hi - lo + 1) as f32
|
||
}`,
|
||
},
|
||
],
|
||
},
|
||
{
|
||
title: 'Beat detection & prediction',
|
||
body: [
|
||
'Identical algorithm to the web edition, ported statement for statement into beat.rs. Detection: the smoothed bass must exceed threshold × the rolling 60-frame average, gated by a minimum interval and an absolute floor. Learning: consecutive beat intervals (200–2500 ms) build a median + spread estimate; when the spread is under 0.3 the period locks with an EMA (0.6/0.4 blend) and confidence rises. Prediction: a phase accumulator advances by dt/period every frame -- when the bass drops out for longer than period × 1.4, the predictor keeps firing beats on the learned grid (confidence-gated, slowly spending confidence at 0.02 per predicted beat and decaying at 0.06/s) so the visuals never lose the beat. Silence hold (v7.15.1): sustained near-digital quiet (level under 0.006 for 300 ms, sticky until a decisive 0.015) freezes detection, prediction, learning and the phase accumulator outright -- no beats fire during silence -- and decisive audio releases the hold, resuming the grid from the frozen phase. Tap tempo: the Space bar taps into an 8-tap median, re-phasing the predictor and teaching the interval learner.',
|
||
],
|
||
code: [
|
||
{
|
||
label: 'rust-agent/src/beat.rs -- the phase-locked predictor',
|
||
language: 'rust',
|
||
content: ` // --- predictor (phase-locked grid) -------------------------------------
|
||
if self.period_ms > 50.0 {
|
||
self.phase += (dt_sec * 1000.0) / self.period_ms;
|
||
}
|
||
if self.phase >= 1.0 {
|
||
self.phase -= self.phase.floor();
|
||
let bass_gone = now_ms - self.last_real_beat_at > self.period_ms * 1.4;
|
||
if !beat
|
||
&& self.cfg.prediction
|
||
&& self.confidence > 0.35
|
||
&& bass_gone
|
||
&& now_ms - self.last_beat_at >= self.cfg.min_interval
|
||
{
|
||
beat = true;
|
||
source = BeatSource::Predicted;
|
||
self.last_beat_at = now_ms;
|
||
self.beat_count += 1;
|
||
// predicted beats slowly spend confidence so a stale lock fades out
|
||
self.confidence = (self.confidence - 0.02).max(0.0);
|
||
}
|
||
}`,
|
||
},
|
||
],
|
||
},
|
||
{
|
||
title: 'The script engine',
|
||
body: [
|
||
'The .ava format and the expression language are the portability contract -- parse once per section, run per frame. Grammar summary (identical to expression.ts):',
|
||
],
|
||
bullets: [
|
||
'statements separated by ";" or newlines (newlines inside parens are not separators); comments with # or //',
|
||
'operators + - * / % ^ (power, right-associative, 2^-3 works), == != < <= > >=, && || ! (short-circuit), ternary',
|
||
'assignment is an expression and is valid inside call args and ternary branches: if(cond, x = 1, x = 2)',
|
||
'if(c, a, b) and the ternary are lazy -- only the taken branch evaluates (AVS semantics)',
|
||
'constants pi, tau, e; the full function set: sin cos tan asin acos atan atan2 abs sqrt sqr pow exp log log10 floor ceil round int frac sign min max rand lerp clamp inrange above below equal band bor bnot sigmoid if',
|
||
'rand(seed) is the same deterministic hash on both engines -- sin(x*127.1 + 311.7) * 43758.5453, fractional part -- so per-point randomness is stable',
|
||
'sections: [init] once, [frame] per frame, [beat] on beat frames, [point] n times per frame; headers name/author/description/mode/blend/persist/linewidth; errors report section + line',
|
||
],
|
||
},
|
||
{
|
||
title: 'Rendering',
|
||
body: [
|
||
'The Canvas2D renderer maps onto three wgpu pipelines sharing one uniform block (time, dt, bass/mid/treb + _att, vol + _att, beat flash, bpm, beatphase, aspect, resolution, persist): a fullscreen fade pass (persistence trails toward black, alpha = 1 − persist), an audio-reactive background pass (bass → warm amber center glow, treble → neon-green edge shimmer, beat → decaying flash), and the content pass -- a line-list or point-list point cloud with per-vertex color and additive blending, the equivalent of the web engine\'s \'lighter\' composite mode. Persistence v0 uses the fade pass; an offscreen feedback texture for true motion trails is the marked next step (TODO(avb) in render/mod.rs).',
|
||
],
|
||
code: [
|
||
{
|
||
label: 'rust-agent/src/render/shader.wgsl -- audio-reactive background',
|
||
language: 'wgsl',
|
||
content: `@fragment
|
||
fn fs_bg(@builtin(position) frag: vec4<f32>) -> @location(0) vec4<f32> {
|
||
let uv = frag.xy / u.resolution; // 0..1
|
||
let p = (uv - 0.5) * vec2<f32>(u.aspect, 1.0); // aspect-corrected
|
||
let r = length(p);
|
||
|
||
// slow breathing shimmer for the edge glow
|
||
let pulse = 0.5 + 0.5 * cos(u.time * 1.7 + r * 6.0);
|
||
|
||
// bass -> warm amber glow at the center
|
||
let warm = vec3<f32>(1.0, 0.62, 0.16);
|
||
let glow = u.bass_att * exp(-r * 3.2) * 0.9;
|
||
|
||
// treble -> thin neon-green shimmer toward the edges
|
||
let neon = vec3<f32>(0.25, 1.0, 0.45);
|
||
let shimmer = u.treb_att * smoothstep(0.25, 0.9, r) * (0.35 + 0.3 * pulse) * 0.55;
|
||
|
||
// beat -> brief bright flash (the envelope decays on the CPU side)
|
||
let flash = u.beat * exp(-r * 2.0) * 0.35;
|
||
|
||
let color = warm * glow + neon * shimmer + warm * flash;
|
||
return vec4<f32>(color, 0.85);
|
||
}`,
|
||
},
|
||
],
|
||
},
|
||
{
|
||
title: 'Building for each platform',
|
||
body: [
|
||
'The scaffold is a single cargo crate with no build scripts -- the only platform dependencies are the system audio (ALSA) and Vulkan headers on Linux.',
|
||
],
|
||
code: [
|
||
{
|
||
label: 'Linux (Debian / Ubuntu) -- native Vulkan',
|
||
language: 'bash',
|
||
content: `sudo apt install libasound2-dev libvulkan-dev
|
||
cargo run --release -- --preset presets/gap-the-mind.ava`,
|
||
},
|
||
{
|
||
label: 'Windows (VS Build Tools + Rust MSVC) -- Vulkan, DX12 fallback',
|
||
language: 'powershell',
|
||
content: `cargo run --release -- --preset presets/gap-the-mind.ava`,
|
||
},
|
||
{
|
||
label: 'macOS -- Metal by default, strict Vulkan via MoltenVK',
|
||
language: 'bash',
|
||
content: `# default: wgpu auto-selects Metal
|
||
cargo run --release -- --preset presets/gap-the-mind.ava
|
||
|
||
# strict Vulkan: install the Vulkan SDK (MoltenVK) first, then
|
||
cargo run --release --features vulkan-force -- --preset presets/gap-the-mind.ava`,
|
||
},
|
||
],
|
||
},
|
||
{
|
||
title: 'Roadmap',
|
||
body: ['What the scaffold grows into, roughly in order:'],
|
||
bullets: [
|
||
'multi-scope composition -- AVS-style effect lists with per-effect blend/persist (the [point] scope becomes one effect type among several)',
|
||
'sprite/Texer-style particles -- instanced quad rendering with textures and per-point size (the scaffold\'s point-list pass is the placeholder)',
|
||
'offscreen feedback trails -- render-to-texture ping-pong so persistence becomes a real feedback shader instead of a fade pass',
|
||
'output to NDI / Syphon -- feed club rigs and media servers directly from the native engine',
|
||
'MIDI / HID controllers -- map knobs to band gains, threshold, persist and script parameters',
|
||
'preset hot-swap on beat -- load the next .ava exactly on a beat boundary for seamless transitions',
|
||
],
|
||
},
|
||
],
|
||
}
|