gitzcreations/openvisengine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
openvisengine/worklog.md
zai-bot b08490e80e v7.15.1 — silence hold: beat detection & prediction freeze during silence 
Sustained near-digital quiet on the overall level (< 0.006 for 300 ms,
sticky until a decisive 0.015) now places an immediate HOLD on the beat
tracker: nothing fires (real or predicted -- previously the predictor
kept faking beats for up to ~2 s into any silence), the tempo estimate
and confidence freeze exactly where they are, and beatphase freezes for
scripts instead of saturating at 'beat imminent'. Decisive audio
releases the hold instantly and the grid resumes from the phase the
audio went quiet (captured before the debounce window, so a
pause/resume keeps perfect time); after a hard cut the comb + first
real beats re-align within a couple of periods. 'Set Beat' taps release
the hold outright. Waveform/band data is never gated -- it still flows
(it is simply ~0, because the audio is gone).

- BeatFrame.silent carries the state; scripts get the silent variable
  (1 while held); the beat indicator shows an amber HOLD badge + amber
  phase ring while held.
- rust-agent/src/beat.rs + engine.rs: same gate ported for parity (its
  bass level doubles as the silence signal; NOT compile-verified -- no
  cargo in this sandbox).
- Verified: headless 26/26 assertions on the real BeatTracker (music ->
  silence -> in-phase resume -> onsetless pad -> tap; the pre-fix
  fake-beat window 12.8-15.5 s is empty) and browser E2E on the live
  app with a generated 120 BPM + silence looping WAV (HOLD badge tracks
  the silent spans exactly across two loop cycles; console clean).
2026-11-06 06:49:46 +00:00

236 KiB
Executable file
Raw Blame History

Worklog — AVA (Advanced Visualization Agent)

Project: Audio-reactive visualization agent (AVS-style scripting, 3-band spectrum detection, predictive beat tracking). Built as a Next.js 16 web app (reference implementation) plus a Rust/Vulkan scaffold for the future native cross-platform export.

User requirements:

  • 3-band detection: Bass 20–100 Hz, Mid 700 Hz–2 kHz, High 7 kHz–16 kHz (user-adjustable)
  • Beat detection from bass + beat PREDICTION (keeps beat when bass drops out)
  • Green dot beat indicator + "Set Beat" tap-tempo button
  • AVS-style custom visualization script editor: edit, save as default, save to file, load from file
  • Help section with variables/functions reference
  • 5 preset scripts (user provided real Winamp AVS presets on Google Drive)
  • Rust + Vulkan target for Linux/Windows/macOS export (scaffold + blueprint delivered here)

Task ID: 1 Agent: main (Z.ai Code) Task: Project inspection + Google Drive preset retrieval

Work Log:

  • Verified dev server running on port 3000 (Next.js 16.1.3, dev.log clean)
  • Fetched Google Drive folder (public) — extracted 5 file IDs and downloaded all files:
    • "Synth-C - Gap The Mind.avs" (12.5 KB)
    • "zamuz - wonder 'bout.avs" (34 KB)
    • "Synth-C - Quantization.avs" (14.8 KB)
    • "zamuz - keep tryin'.avs" (19 KB)
    • "zamuz - vorticity (brain activity remix).avs" (17 KB)
  • Parsed binary AVS format (null-terminated strings): extracted Superscope code, Texer II configs, movement expressions. These are scene classics from visbot.net.
  • Decided engine design: AVS-eel-flavored expression language (if/above/below/equal/rand, getspec/getosc-equivalents sv/v, reg-free flat variables) so the presets port naturally.

Stage Summary:

  • All 5 original presets secured and understood; will be adapted to the AVA script dialect (originals preserved as downloadable files in public/avs-originals/).
  • Core engine architecture defined: expression compiler, script parser, BeatTracker (PLL-style predictor + tap tempo), AudioEngine (WebAudio FFT, mic/file/demo synth), canvas renderer.

Task ID: 2 Agent: main (Z.ai Code) Task: Project setup — deps, Prisma schema, DB, original preset files

Work Log:

  • Installed jszip (for the Rust scaffold zip download API)
  • Prisma schema rewritten: Preset (name/author/description/content/isBuiltIn, unique name) and Settings (defaultScript + bandConfig + beatConfig JSON) — ran bun run db:push OK
  • Copied the 5 original Winamp .avs binaries to public/avs-originals/ (downloadable from UI)

Stage Summary:

  • DB ready; jszip available; originals served at /avs-originals/*.avs

Task ID: 3 Agent: main (Z.ai Code) Task: AVA core engine libraries (src/lib/ava/*)

Work Log:

  • types.ts — BandConfig / BeatConfig / BandLevels / BeatFrame + defaults (bass 20–100 Hz, mid 700–2000 Hz, high 7000–16000 Hz as requested)
  • expression.ts — full expression compiler: tokenizer → recursive-descent parser → closure tree. AVS-eel flavored: if() is LAZY, && / || short-circuit, ternary, ^ power (right-assoc), assignment valid inside call args + ternary branches (if(cond, x=1, x=2) works), newline & ';' statement separators (newlines inside parens ignored), '#'/'//' comments, deterministic hash rand(seed). 30+ functions incl. above/below/equal/band/bor/bnot/frac.
  • script.ts — text format parser: header keys (name/author/description/mode/blend/persist/ linewidth) + [init]/[frame]/[beat]/[point] sections, error objects with section+line, validateScript() never throws.
  • beat.ts — BeatTracker: rolling-average energy detection w/ threshold+minInterval+floor, interval learning (median + EMA period lock, spread gating), PLL-style phase predictor that keeps firing beats when bass drops (confidence-gated, slowly decays), tap() tempo learning.
  • audio.ts — AudioEngine: WebAudio AnalyserNode FFT 8192 (~5.4 Hz bins), byte-spectrum band averaging with attack 35ms / release 220ms smoothing; sources: demo (synthesized techno: kick 160→43Hz sweep, snare bandpass 1.8k, hats HP 8.2k, saw bassline — audible + analyzable), mic (never routed to speakers), file (loops, audible). getWave/getSpectrum for scripts.
  • presets.ts — 5 built-in presets ADAPTED from the user's real AVS files: Gap The Mind (radial warp rings), wonder 'bout (bass-spawned expanding rings), Quantization (grid-snapped dot rings), keep tryin' (bass burst dots + swirl), vorticity (twisting spiral arms) + STARTER_TEMPLATE.
  • engine.ts — AvaEngine: rAF loop (bands → beat → env → frame/beat progs → point prog ×N → draw), canvas 2d renderer with additive blending, per-point color, 2-pass glow, persistence trails, lines/dots/solid modes, DPR-aware resize via ResizeObserver, throttled stats callback for the UI. Singleton via getEngine().
  • SMOKE TESTED via bun script: all 5 presets parse & execute 120 frames; lazy-if/ternary/pow OK; beat sim locks 124.1 BPM from 124 BPM kick train; predictor sustains beats through 10s of silence; tap tempo locks 120 BPM from 500ms taps. (2 parser bugs found & fixed: assignment inside call args; newline statement separators with paren-awareness.)

Stage Summary:

  • Core engine complete and verified. Env variables exposed to scripts: n, t, time, dt, frame, bass/mid/treb (+ _att), vol (+ _att), beat, bpm, beatphase, beatconf, beats, w, h, aspect, persist, i, v, sv, x, y, red, green, blue, skip, size, alpha.

Task ID: 4 Agent: main (Z.ai Code) Task: API routes

Work Log:

  • GET/POST /api/presets — GET seeds built-ins on first run and lists all; POST upserts user presets by name (built-ins protected, 409 on name clash with built-in)
  • PUT/DELETE /api/presets/[id] — user presets only
  • GET/PUT /api/settings — defaultScript + bandConfig + beatConfig (upsert, JSON strings in DB)
  • Created src/lib/ava/blueprint.ts STUB (interface Blueprint + placeholder) for Task 3-b

Stage Summary:

  • Backend complete. Frontend contract: GET /api/presets → { presets: [...] }; POST /api/presets {name, content, author?, description?} → {preset}; GET /api/settings → { defaultScript, bandConfig, beatConfig }; PUT /api/settings { defaultScript? , bandConfig?, beatConfig? }.

Task ID: 3-a Agent: full-stack-developer (UI) Task: Complete UI for AVA — single-page app (dark club/VJ aesthetic) with header (audio source controls, BPM readout, green beat dot + Set Beat pairing), visual stage (canvas + engine loop), 5-tab control panel (Presets / Script / Detect / Beat / Native) and sticky footer status bar.

Work Log:

Stage Summary:

  • Files created: src/lib/store.ts; src/components/ava/{shared.ts, audio-source.ts, beat-dot.tsx, header.tsx, visual-stage.tsx, presets-tab.tsx, script-tab.tsx, help-panel.tsx, detect-tab.tsx, beat-tab.tsx, native-tab.tsx, footer-bar.tsx, control-panel.tsx}. Modified: src/app/page.tsx (replaced), src/app/layout.tsx, src/hooks/use-toast.ts (toast duration/limit only).
  • KEY DECISION — beat flash sampling: engine stats are throttled (~8 Hz) but beat=true lasts one frame, so most beats are invisible in stats. The watcher in page.tsx ALSO fires on beat.phase running backwards (real beats reset phase to 0; the predictor grid wraps at 1). Classification: stats.beat.beat sampled → exact source; phase drop + prediction on + confidence > 0.35 + bassAtt < 0.06 → 'predicted'; phase drop + bassAtt > 0.05 → 'audio'; silent no-lock → no flash (idle dot doesn't pulse). Tap flashes suppressed for 300 ms to avoid double pulses. Verified working in-browser.
  • KEY DECISION — desktop uses lg:h-screen + internal panel scroll so the footer stays at the viewport bottom and the canvas fills the height; mobile uses natural flow (min-h-screen + mt-auto footer) so the footer is pushed down.
  • Store is the only place configs touch the engine (applyBandConfig/ setBandValue etc.), keeping live-update + persistence consistent everywhere.
  • NOTE for Task 5 (integration): /api/rust-scaffold is linked from the Native tab but the route does not exist yet (Task 3-b owns it — expect 404 until then). BLUEPRINT is still the stub; the Native tab renders whatever 3-b puts in it, no UI changes needed. Settings row 'main' currently holds defaultScript="Gap The Mind" + default bandConfig from testing (both equal defaults — safe to leave or clear). Pre-existing tsc errors in src/lib/ava/{audio,script}.ts and examples/ are NOT from this task (ignoreBuildErrors is set).

Task ID: 3-b Agent: general-purpose (Rust scaffold + blueprint) — [timed out at the very end; deliverables verified complete by the main agent, which appends this entry]

Task: Rust/Vulkan native-export scaffold + architecture blueprint + zip download API

Work Log:

  • Built rust-agent/ cargo crate (ava-agent, edition 2021): winit 0.30 (ApplicationHandler API), wgpu 22 (Vulkan-first; vulkan-force feature for MoltenVK on macOS), cpal 0.15, rustfft 6.2, pollster, anyhow
  • src/audio.rs — cpal default-input capture, ring buffer, AudioProcessor::poll() per frame
  • src/spectrum.rs — Hann-window 8192 FFT, band levels w/ same attack(35ms)/release(220ms) smoothing as the web engine
  • src/beat.rs — 1:1 port of beat.ts (same constants, median+EMA lock, PLL predictor, taps)
  • src/script/{lexer,parser,env,mod}.rs — expression engine + .ava format parser with the same lazy-if / assignment-in-args / deterministic-rand semantics
  • src/engine.rs — script runtime host (init/frame/beat/point stepping, shared state)
  • src/render/{mod.rs,shader.wgsl} — wgpu pipelines: audio-reactive fullscreen background + additive point cloud, #[repr(C)] uniforms, persistence via TODO'd feedback texture
  • presets/*.ava — all six scripts (5 classics + starter) portable across engines
  • Rewrote src/lib/ava/blueprint.ts with 8 real sections (architecture map, wgpu/Vulkan strategy, audio pipeline, beat engine, script engine, rendering, per-platform builds, roadmap)
  • Built src/app/api/rust-scaffold/route.ts (jszip walk, skip target/.git/node_modules)

Stage Summary:

  • Scaffold ~1,700 lines, all files present; zip route verified 200 in dev.log; lint clean.
  • NOT compiled (no cargo in sandbox) — scaffold quality, clearly TODO-marked where partial.

Task ID: 5 Agent: main (Z.ai Code) Task: Integration, lint, agent-browser E2E verification & fixes

Work Log:

  • bun run lint → zero errors across the whole project
  • dev.log clean: / 200, /api/presets 200, /api/settings 200, /api/rust-scaffold 200, no runtime errors, no hydration warnings
  • Agent-browser E2E golden path, all VERIFIED:
    • initial load: idle overlay, 5 preset cards + .avs original links, tabs render
    • Demo Track → canvas lit (30.7% pixels @60fps, 480 pts), footer SRC/FPS/PTS/BPM live
    • footer shows detected 120.0 BPM from the demo kick pattern
    • Set Beat ×4 taps → "Locked: 99.8 BPM" (matches real ~600ms tap intervals), confidence 65%
    • Beat tab: AUDIO badge, beat count, sensitivity ×1.40, min interval 250ms, prediction switch
    • Stop audio → badge switches to PRED, beat count 20→35 (predictor keeps the beat), confidence decays 65%→35% — core requirement verified
    • Script editor: broken script → "✘ [point] line 8: Unknown function 'badfunc'"; fixed script → "✔ Compiled OK — mode: lines, n=300"
    • Help panel: variables table, functions, operators, examples
    • Save as preset → "Smoke Test Ring" listed + toast; two-step delete verified
    • Save as Default → PUT /api/settings verified via curl; reload auto-loads the default
    • Detect tab: Bass 20–100 Hz / Mid 700–2000 Hz / High 7k–16 kHz cards, gains, live meters reading real audio (0.78/0.60/0.32)
    • Native tab: blueprint sections render + zip download (116 KB, 25 files, valid zip)
    • Mobile 390×844: responsive, footer pushed naturally, sticks to viewport bottom when scrolled
  • Cleanup: deleted the test preset, cleared test defaultScript (DB + localStorage) so a fresh visit loads "Gap The Mind"

Stage Summary:

  • ALL acceptance criteria verified in-browser. Zero console errors. The application is fully interactive: 3-band detection, beat detection + prediction + tap learning, AVS-style scripting with editor/help/save/load, 5 authentic presets (adapted), Rust scaffold export.

Task ID: 6 Agent: general-purpose Task: "Stream a link" backend — yt-dlp link resolution + same-origin audio byte proxy (POST /api/resolve, GET /api/audio/stream)

Work Log:

  • Read worklog (Tasks 1–5) and existing routes; confirmed dev server already running on :3000 (Node, next-server v16.1.3) — never restarted it; no build, no new deps, no git
  • Installed yt-dlp standalone binary: bin/yt-dlp (GitHub latest yt-dlp_linux), chmod +x, --version prints 2026.10.21
  • src/lib/server/ssrf.ts — assertPublicHttpUrl(raw): http(s) only, explicit port must be 80/443, localhost/.localhost/.local/.internal banned, IPv4 privates (0/8, 10/8, 100.64/10, 127/8, 169.254/16, 172.16/12, 192.168/16), IPv6 ::/::1/fc00::/7/fe80::/10 plus IPv4-mapped ::ffff:a.b.c.d (embedded v4 checked; hand-written IPv6 expander handles v4 tails + zone ids), then dns.promises.lookup(all:true) with EVERY result re-checked by the same rules (kills nip.io / DNS-rebind tricks). Throws user-safe messages; returns the parsed URL.
  • src/lib/server/stream-store.ts — Map on globalThis.__avaStreams via a typed interface with an optional property + ??= (no any): token = randomBytes(16).hex (32 chars), entries { remoteUrl, title, source, expires }, 30-min sliding TTL, prune-on-every-access; exports createStreamEntry / getStreamEntry / refreshStreamEntry.
  • src/app/api/resolve/route.ts — JSON body { url } (≤2048, trim) → 400s; SSRF gate → 400; 10-min success cache on globalThis.__avaResolveCache (repeat links replay instantly); direct-audio bypass for .mp3 .m4a .aac .ogg .oga .opus .wav .flac .mp4 .webm (source "direct", title = decoded last path segment, fallback hostname); otherwise yt-dlp -J --no-warnings --no-playlist --no-progress --socket-timeout 15 -f "bestaudio[protocol^=http]/bestaudio/best" with a 30 s kill → 504; non-zero exit → ANSI/control-stripped stderr tail (300 chars) → 422 ("Unsupported URL" gets its own message); stream URL pick = data.url ?? requested_formats[0].url ?? last http-protocol format; .m3u8/.mpd path or m3u8/mpd/dash protocol → 422 adaptive message; SSRF re-check on the RESOLVED stream URL (public CDNs like googlevideo pass) → 422 if not; title control-stripped + clamped 140 (fallback "Stream"), source = extractor_key|extractor|hostname lowercase, durationSec = numeric duration or null.
  • src/app/api/audio/stream/route.ts — token → entry (prune first) else 404; SSRF re-check of the stored remoteUrl; forwards the client Range header, Chrome UA, redirect follow, no-store; !ok → 502 Upstream failed (HTTP n).; content-type must be audio/, video/, application/octet-stream or application/ogg else 502 "Upstream did not return audio."; refreshes sliding TTL, then passes through the upstream status with content-type/-length/ -range/accept-ranges (only when upstream provided them) + Cache-Control: public, max-age=1800.
  • FIX found while testing: the upstream host intermittently RST/ETIMEDOUTs full (non-ranged) fetches — verified against raw node fetch (1/6 ETIMEDOUT) — added ONE immediate retry on network-level failure only (HTTP error statuses are never retried); after that: 2× full GET 200 audio/mpeg 8,945,229 bytes.
  • tsc --noEmit: zero errors in the new files (the project's 8 pre-existing errors are all in src/lib/ava, examples/, skills/, rust-scaffold — untouched).

Stage Summary:

  • yt-dlp: /home/z/my-project/bin/yt-dlp, version 2026.10.21 (runtime order: $AVA_YTDLP → /bin/yt-dlp if existsSync → "yt-dlp" on PATH)
  • Files created (nothing else touched, no frontend files): bin/yt-dlp, src/lib/server/ssrf.ts, src/lib/server/stream-store.ts, src/app/api/resolve/route.ts, src/app/api/audio/stream/route.ts
  • Test 1 MUST-PASS direct MP3 (soundhelix Song-1): resolve → ok:true, source:"direct", title:"SoundHelix-Song-1.mp3"; full stream GET → 200 audio/mpeg 8,945,229 B (>1 MB; takes ~30 s because the origin throttles at ~300 KB/s — proxy adds ~0); Range 0-1023 → 206 with content-range "bytes 0-1023/8945229", accept-ranges, cache-control all verified
  • Test 2 MUST-PASS SSRF: http://127.0.0.1/x → 400, http://169.254.169.254/latest/meta-data → 400, ftp://example.com/a.mp3 → 400. Extras: localhost:8080 → 400 (port), foo.bar.local → 400, [::1] → 400, [::ffff:10.0.0.5] → 400 (v4-mapped), 127.0.0.1.nip.io → 400 (DNS-result check), "not a url" / {} / {"url":5} / bad JSON → 400 "Expected { url } JSON body."
  • Test 3 YouTube jNQXAC9IVRw: bot-blocked from this datacenter IP ("Sign in to confirm you're not a bot"). Tried the 2 extractor-args variants (youtube:player_client=web_embedded, then =tv) within the 5-minute budget — still blocked. Surfaces as a clean 422 with the sanitized tail: "Could not extract audio from that link: …Sign in to confirm you're not a bot…". Honest outcome: blocked here, ACCEPTABLE per spec (would work from residential IPs / with cookies).
  • Test 4 SoundCloud forss/flickermood: yt-dlp extracts fine (title "Flickermood", 213.9 s) but this serve only exposes m3u8_native formats (hls_aac_96k/160k — confirmed in raw -J JSON) → clean 422 adaptive(HLS/DASH) message, exactly the manifest-only contract case.
  • Test 5 https://example.com/ → clean 422 "That site is not supported — try YouTube, SoundCloud, or a direct MP3 link."; example.com/stream.m3u8 → 422 (does NOT hit the direct bypass).
  • Extractor SUCCESS path proven end-to-end with archive.org: resolve → ok:true, title:"mp3 test file", source:"archiveorg", durationSec:12.42; stream → 200 audio/mpeg 198,658 B + 206 ranged; cache replay 16 ms.
  • Stream negatives: unknown/missing token → 404 "Unknown or expired stream token"; upstream 500 (missing file) → 502 "Upstream failed (HTTP 500)."
  • lint: my 4 files pass eslint clean; bun run lint overall reports exactly 1 remaining error in src/components/ava/link-dialog.tsx:72 (set-state-in-effect) — that file is owned by the parallel frontend agent and was NOT touched by me per the ownership rules.
  • dev.log: no exceptions/stack traces from the new routes — only expected 200/206/400/404/422/502 request lines.
  • Deviations from spec (all deliberate): (1) one network-error retry in the proxy (upstream RSTs, never retries HTTP statuses); (2) third store helper refreshStreamEntry() — the two required helpers plus this one implements the specified "refresh sliding expiry AFTER upstream success"; (3) proxy fetch uses cache:'no-store'; (4) 500 wording "Audio extractor unavailable (yt-dlp not found or failed to start)."; (5) success cache keyed by the normalized URL href (target.href).

Task ID: 7 Agent: main (Z.ai Code) Task: Frontend integration of the "Link" audio source (YouTube / SoundCloud / direct MP3)

Work Log:

  • Read worklog + all relevant files (audio.ts, store.ts, audio-source.ts, header.tsx, visual-stage.tsx, footer-bar.tsx, types.ts) before editing.
  • src/lib/ava/types.ts — AudioSourceKind += 'link'
  • src/lib/ava/audio.ts — new startUrl(streamUrl, title): same-origin proxied element → MediaElementAudioSourceNode → analyser (NOT cross-origin-tainted, so FFT works) → speakers; loops by default; public sourceLabel field (file name / stream title) maintained by all start* methods + stopSources
  • src/lib/store.ts — AudioSourceState += 'link'; new sourceTitle + linkDialogOpen state
  • src/components/ava/audio-source.ts — new resolveAndPlayLink(url): POST /api/resolve → engine.audio.startUrl → store sync; never throws (returns {ok,title,source,error}); on playback failure stops the engine and resets store; syncSource now also syncs sourceTitle
  • src/components/ava/link-dialog.tsx — NEW: dialog with URL input + Load button (spinner), inline error box with YouTube bot-wall hint, RECENT LINKS chips (localStorage, max 5, one-click reload); body REMOUNTS on every open (fresh state — no setState-in-effect, fixes the lint error the Task 6 agent saw mid-build); results arriving after the user closed the dialog surface as toasts
  • src/components/ava/header.tsx — "Link" segment button between File and Stop (opens dialog); mounted in the header
  • src/components/ava/visual-stage.tsx — idle overlay gains a 4th button "Stream a Link" + updated copy
  • src/components/ava/footer-bar.tsx — SRC now shows "LINK ·