gitzcreations/openvisengine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
openvisengine/worklog.md

209 KiB
Executable file
Raw Blame History

Worklog — AVA (Advanced Visualization Agent)

Project: Audio-reactive visualization agent (AVS-style scripting, 3-band spectrum detection, predictive beat tracking). Built as a Next.js 16 web app (reference implementation) plus a Rust/Vulkan scaffold for the future native cross-platform export.

User requirements:

  • 3-band detection: Bass 20–100 Hz, Mid 700 Hz–2 kHz, High 7 kHz–16 kHz (user-adjustable)
  • Beat detection from bass + beat PREDICTION (keeps beat when bass drops out)
  • Green dot beat indicator + "Set Beat" tap-tempo button
  • AVS-style custom visualization script editor: edit, save as default, save to file, load from file
  • Help section with variables/functions reference
  • 5 preset scripts (user provided real Winamp AVS presets on Google Drive)
  • Rust + Vulkan target for Linux/Windows/macOS export (scaffold + blueprint delivered here)

Task ID: 1 Agent: main (Z.ai Code) Task: Project inspection + Google Drive preset retrieval

Work Log:

  • Verified dev server running on port 3000 (Next.js 16.1.3, dev.log clean)
  • Fetched Google Drive folder (public) — extracted 5 file IDs and downloaded all files:
    • "Synth-C - Gap The Mind.avs" (12.5 KB)
    • "zamuz - wonder 'bout.avs" (34 KB)
    • "Synth-C - Quantization.avs" (14.8 KB)
    • "zamuz - keep tryin'.avs" (19 KB)
    • "zamuz - vorticity (brain activity remix).avs" (17 KB)
  • Parsed binary AVS format (null-terminated strings): extracted Superscope code, Texer II configs, movement expressions. These are scene classics from visbot.net.
  • Decided engine design: AVS-eel-flavored expression language (if/above/below/equal/rand, getspec/getosc-equivalents sv/v, reg-free flat variables) so the presets port naturally.

Stage Summary:

  • All 5 original presets secured and understood; will be adapted to the AVA script dialect (originals preserved as downloadable files in public/avs-originals/).
  • Core engine architecture defined: expression compiler, script parser, BeatTracker (PLL-style predictor + tap tempo), AudioEngine (WebAudio FFT, mic/file/demo synth), canvas renderer.

Task ID: 2 Agent: main (Z.ai Code) Task: Project setup — deps, Prisma schema, DB, original preset files

Work Log:

  • Installed jszip (for the Rust scaffold zip download API)
  • Prisma schema rewritten: Preset (name/author/description/content/isBuiltIn, unique name) and Settings (defaultScript + bandConfig + beatConfig JSON) — ran bun run db:push OK
  • Copied the 5 original Winamp .avs binaries to public/avs-originals/ (downloadable from UI)

Stage Summary:

  • DB ready; jszip available; originals served at /avs-originals/*.avs

Task ID: 3 Agent: main (Z.ai Code) Task: AVA core engine libraries (src/lib/ava/*)

Work Log:

  • types.ts — BandConfig / BeatConfig / BandLevels / BeatFrame + defaults (bass 20–100 Hz, mid 700–2000 Hz, high 7000–16000 Hz as requested)
  • expression.ts — full expression compiler: tokenizer → recursive-descent parser → closure tree. AVS-eel flavored: if() is LAZY, && / || short-circuit, ternary, ^ power (right-assoc), assignment valid inside call args + ternary branches (if(cond, x=1, x=2) works), newline & ';' statement separators (newlines inside parens ignored), '#'/'//' comments, deterministic hash rand(seed). 30+ functions incl. above/below/equal/band/bor/bnot/frac.
  • script.ts — text format parser: header keys (name/author/description/mode/blend/persist/ linewidth) + [init]/[frame]/[beat]/[point] sections, error objects with section+line, validateScript() never throws.
  • beat.ts — BeatTracker: rolling-average energy detection w/ threshold+minInterval+floor, interval learning (median + EMA period lock, spread gating), PLL-style phase predictor that keeps firing beats when bass drops (confidence-gated, slowly decays), tap() tempo learning.
  • audio.ts — AudioEngine: WebAudio AnalyserNode FFT 8192 (~5.4 Hz bins), byte-spectrum band averaging with attack 35ms / release 220ms smoothing; sources: demo (synthesized techno: kick 160→43Hz sweep, snare bandpass 1.8k, hats HP 8.2k, saw bassline — audible + analyzable), mic (never routed to speakers), file (loops, audible). getWave/getSpectrum for scripts.
  • presets.ts — 5 built-in presets ADAPTED from the user's real AVS files: Gap The Mind (radial warp rings), wonder 'bout (bass-spawned expanding rings), Quantization (grid-snapped dot rings), keep tryin' (bass burst dots + swirl), vorticity (twisting spiral arms) + STARTER_TEMPLATE.
  • engine.ts — AvaEngine: rAF loop (bands → beat → env → frame/beat progs → point prog ×N → draw), canvas 2d renderer with additive blending, per-point color, 2-pass glow, persistence trails, lines/dots/solid modes, DPR-aware resize via ResizeObserver, throttled stats callback for the UI. Singleton via getEngine().
  • SMOKE TESTED via bun script: all 5 presets parse & execute 120 frames; lazy-if/ternary/pow OK; beat sim locks 124.1 BPM from 124 BPM kick train; predictor sustains beats through 10s of silence; tap tempo locks 120 BPM from 500ms taps. (2 parser bugs found & fixed: assignment inside call args; newline statement separators with paren-awareness.)

Stage Summary:

  • Core engine complete and verified. Env variables exposed to scripts: n, t, time, dt, frame, bass/mid/treb (+ _att), vol (+ _att), beat, bpm, beatphase, beatconf, beats, w, h, aspect, persist, i, v, sv, x, y, red, green, blue, skip, size, alpha.

Task ID: 4 Agent: main (Z.ai Code) Task: API routes

Work Log:

  • GET/POST /api/presets — GET seeds built-ins on first run and lists all; POST upserts user presets by name (built-ins protected, 409 on name clash with built-in)
  • PUT/DELETE /api/presets/[id] — user presets only
  • GET/PUT /api/settings — defaultScript + bandConfig + beatConfig (upsert, JSON strings in DB)
  • Created src/lib/ava/blueprint.ts STUB (interface Blueprint + placeholder) for Task 3-b

Stage Summary:

  • Backend complete. Frontend contract: GET /api/presets → { presets: [...] }; POST /api/presets {name, content, author?, description?} → {preset}; GET /api/settings → { defaultScript, bandConfig, beatConfig }; PUT /api/settings { defaultScript? , bandConfig?, beatConfig? }.

Task ID: 3-a Agent: full-stack-developer (UI) Task: Complete UI for AVA — single-page app (dark club/VJ aesthetic) with header (audio source controls, BPM readout, green beat dot + Set Beat pairing), visual stage (canvas + engine loop), 5-tab control panel (Presets / Script / Detect / Beat / Native) and sticky footer status bar.

Work Log:

Stage Summary:

  • Files created: src/lib/store.ts; src/components/ava/{shared.ts, audio-source.ts, beat-dot.tsx, header.tsx, visual-stage.tsx, presets-tab.tsx, script-tab.tsx, help-panel.tsx, detect-tab.tsx, beat-tab.tsx, native-tab.tsx, footer-bar.tsx, control-panel.tsx}. Modified: src/app/page.tsx (replaced), src/app/layout.tsx, src/hooks/use-toast.ts (toast duration/limit only).
  • KEY DECISION — beat flash sampling: engine stats are throttled (~8 Hz) but beat=true lasts one frame, so most beats are invisible in stats. The watcher in page.tsx ALSO fires on beat.phase running backwards (real beats reset phase to 0; the predictor grid wraps at 1). Classification: stats.beat.beat sampled → exact source; phase drop + prediction on + confidence > 0.35 + bassAtt < 0.06 → 'predicted'; phase drop + bassAtt > 0.05 → 'audio'; silent no-lock → no flash (idle dot doesn't pulse). Tap flashes suppressed for 300 ms to avoid double pulses. Verified working in-browser.
  • KEY DECISION — desktop uses lg:h-screen + internal panel scroll so the footer stays at the viewport bottom and the canvas fills the height; mobile uses natural flow (min-h-screen + mt-auto footer) so the footer is pushed down.
  • Store is the only place configs touch the engine (applyBandConfig/ setBandValue etc.), keeping live-update + persistence consistent everywhere.
  • NOTE for Task 5 (integration): /api/rust-scaffold is linked from the Native tab but the route does not exist yet (Task 3-b owns it — expect 404 until then). BLUEPRINT is still the stub; the Native tab renders whatever 3-b puts in it, no UI changes needed. Settings row 'main' currently holds defaultScript="Gap The Mind" + default bandConfig from testing (both equal defaults — safe to leave or clear). Pre-existing tsc errors in src/lib/ava/{audio,script}.ts and examples/ are NOT from this task (ignoreBuildErrors is set).

Task ID: 3-b Agent: general-purpose (Rust scaffold + blueprint) — [timed out at the very end; deliverables verified complete by the main agent, which appends this entry]

Task: Rust/Vulkan native-export scaffold + architecture blueprint + zip download API

Work Log:

  • Built rust-agent/ cargo crate (ava-agent, edition 2021): winit 0.30 (ApplicationHandler API), wgpu 22 (Vulkan-first; vulkan-force feature for MoltenVK on macOS), cpal 0.15, rustfft 6.2, pollster, anyhow
  • src/audio.rs — cpal default-input capture, ring buffer, AudioProcessor::poll() per frame
  • src/spectrum.rs — Hann-window 8192 FFT, band levels w/ same attack(35ms)/release(220ms) smoothing as the web engine
  • src/beat.rs — 1:1 port of beat.ts (same constants, median+EMA lock, PLL predictor, taps)
  • src/script/{lexer,parser,env,mod}.rs — expression engine + .ava format parser with the same lazy-if / assignment-in-args / deterministic-rand semantics
  • src/engine.rs — script runtime host (init/frame/beat/point stepping, shared state)
  • src/render/{mod.rs,shader.wgsl} — wgpu pipelines: audio-reactive fullscreen background + additive point cloud, #[repr(C)] uniforms, persistence via TODO'd feedback texture
  • presets/*.ava — all six scripts (5 classics + starter) portable across engines
  • Rewrote src/lib/ava/blueprint.ts with 8 real sections (architecture map, wgpu/Vulkan strategy, audio pipeline, beat engine, script engine, rendering, per-platform builds, roadmap)
  • Built src/app/api/rust-scaffold/route.ts (jszip walk, skip target/.git/node_modules)

Stage Summary:

  • Scaffold ~1,700 lines, all files present; zip route verified 200 in dev.log; lint clean.
  • NOT compiled (no cargo in sandbox) — scaffold quality, clearly TODO-marked where partial.

Task ID: 5 Agent: main (Z.ai Code) Task: Integration, lint, agent-browser E2E verification & fixes

Work Log:

  • bun run lint → zero errors across the whole project
  • dev.log clean: / 200, /api/presets 200, /api/settings 200, /api/rust-scaffold 200, no runtime errors, no hydration warnings
  • Agent-browser E2E golden path, all VERIFIED:
    • initial load: idle overlay, 5 preset cards + .avs original links, tabs render
    • Demo Track → canvas lit (30.7% pixels @60fps, 480 pts), footer SRC/FPS/PTS/BPM live
    • footer shows detected 120.0 BPM from the demo kick pattern
    • Set Beat ×4 taps → "Locked: 99.8 BPM" (matches real ~600ms tap intervals), confidence 65%
    • Beat tab: AUDIO badge, beat count, sensitivity ×1.40, min interval 250ms, prediction switch
    • Stop audio → badge switches to PRED, beat count 20→35 (predictor keeps the beat), confidence decays 65%→35% — core requirement verified
    • Script editor: broken script → "✘ [point] line 8: Unknown function 'badfunc'"; fixed script → "✔ Compiled OK — mode: lines, n=300"
    • Help panel: variables table, functions, operators, examples
    • Save as preset → "Smoke Test Ring" listed + toast; two-step delete verified
    • Save as Default → PUT /api/settings verified via curl; reload auto-loads the default
    • Detect tab: Bass 20–100 Hz / Mid 700–2000 Hz / High 7k–16 kHz cards, gains, live meters reading real audio (0.78/0.60/0.32)
    • Native tab: blueprint sections render + zip download (116 KB, 25 files, valid zip)
    • Mobile 390×844: responsive, footer pushed naturally, sticks to viewport bottom when scrolled
  • Cleanup: deleted the test preset, cleared test defaultScript (DB + localStorage) so a fresh visit loads "Gap The Mind"

Stage Summary:

  • ALL acceptance criteria verified in-browser. Zero console errors. The application is fully interactive: 3-band detection, beat detection + prediction + tap learning, AVS-style scripting with editor/help/save/load, 5 authentic presets (adapted), Rust scaffold export.

Task ID: 6 Agent: general-purpose Task: "Stream a link" backend — yt-dlp link resolution + same-origin audio byte proxy (POST /api/resolve, GET /api/audio/stream)

Work Log:

  • Read worklog (Tasks 1–5) and existing routes; confirmed dev server already running on :3000 (Node, next-server v16.1.3) — never restarted it; no build, no new deps, no git
  • Installed yt-dlp standalone binary: bin/yt-dlp (GitHub latest yt-dlp_linux), chmod +x, --version prints 2026.10.21
  • src/lib/server/ssrf.ts — assertPublicHttpUrl(raw): http(s) only, explicit port must be 80/443, localhost/.localhost/.local/.internal banned, IPv4 privates (0/8, 10/8, 100.64/10, 127/8, 169.254/16, 172.16/12, 192.168/16), IPv6 ::/::1/fc00::/7/fe80::/10 plus IPv4-mapped ::ffff:a.b.c.d (embedded v4 checked; hand-written IPv6 expander handles v4 tails + zone ids), then dns.promises.lookup(all:true) with EVERY result re-checked by the same rules (kills nip.io / DNS-rebind tricks). Throws user-safe messages; returns the parsed URL.
  • src/lib/server/stream-store.ts — Map on globalThis.__avaStreams via a typed interface with an optional property + ??= (no any): token = randomBytes(16).hex (32 chars), entries { remoteUrl, title, source, expires }, 30-min sliding TTL, prune-on-every-access; exports createStreamEntry / getStreamEntry / refreshStreamEntry.
  • src/app/api/resolve/route.ts — JSON body { url } (≤2048, trim) → 400s; SSRF gate → 400; 10-min success cache on globalThis.__avaResolveCache (repeat links replay instantly); direct-audio bypass for .mp3 .m4a .aac .ogg .oga .opus .wav .flac .mp4 .webm (source "direct", title = decoded last path segment, fallback hostname); otherwise yt-dlp -J --no-warnings --no-playlist --no-progress --socket-timeout 15 -f "bestaudio[protocol^=http]/bestaudio/best" with a 30 s kill → 504; non-zero exit → ANSI/control-stripped stderr tail (300 chars) → 422 ("Unsupported URL" gets its own message); stream URL pick = data.url ?? requested_formats[0].url ?? last http-protocol format; .m3u8/.mpd path or m3u8/mpd/dash protocol → 422 adaptive message; SSRF re-check on the RESOLVED stream URL (public CDNs like googlevideo pass) → 422 if not; title control-stripped + clamped 140 (fallback "Stream"), source = extractor_key|extractor|hostname lowercase, durationSec = numeric duration or null.
  • src/app/api/audio/stream/route.ts — token → entry (prune first) else 404; SSRF re-check of the stored remoteUrl; forwards the client Range header, Chrome UA, redirect follow, no-store; !ok → 502 Upstream failed (HTTP n).; content-type must be audio/, video/, application/octet-stream or application/ogg else 502 "Upstream did not return audio."; refreshes sliding TTL, then passes through the upstream status with content-type/-length/ -range/accept-ranges (only when upstream provided them) + Cache-Control: public, max-age=1800.
  • FIX found while testing: the upstream host intermittently RST/ETIMEDOUTs full (non-ranged) fetches — verified against raw node fetch (1/6 ETIMEDOUT) — added ONE immediate retry on network-level failure only (HTTP error statuses are never retried); after that: 2× full GET 200 audio/mpeg 8,945,229 bytes.
  • tsc --noEmit: zero errors in the new files (the project's 8 pre-existing errors are all in src/lib/ava, examples/, skills/, rust-scaffold — untouched).

Stage Summary:

  • yt-dlp: /home/z/my-project/bin/yt-dlp, version 2026.10.21 (runtime order: $AVA_YTDLP → /bin/yt-dlp if existsSync → "yt-dlp" on PATH)
  • Files created (nothing else touched, no frontend files): bin/yt-dlp, src/lib/server/ssrf.ts, src/lib/server/stream-store.ts, src/app/api/resolve/route.ts, src/app/api/audio/stream/route.ts
  • Test 1 MUST-PASS direct MP3 (soundhelix Song-1): resolve → ok:true, source:"direct", title:"SoundHelix-Song-1.mp3"; full stream GET → 200 audio/mpeg 8,945,229 B (>1 MB; takes ~30 s because the origin throttles at ~300 KB/s — proxy adds ~0); Range 0-1023 → 206 with content-range "bytes 0-1023/8945229", accept-ranges, cache-control all verified
  • Test 2 MUST-PASS SSRF: http://127.0.0.1/x → 400, http://169.254.169.254/latest/meta-data → 400, ftp://example.com/a.mp3 → 400. Extras: localhost:8080 → 400 (port), foo.bar.local → 400, [::1] → 400, [::ffff:10.0.0.5] → 400 (v4-mapped), 127.0.0.1.nip.io → 400 (DNS-result check), "not a url" / {} / {"url":5} / bad JSON → 400 "Expected { url } JSON body."
  • Test 3 YouTube jNQXAC9IVRw: bot-blocked from this datacenter IP ("Sign in to confirm you're not a bot"). Tried the 2 extractor-args variants (youtube:player_client=web_embedded, then =tv) within the 5-minute budget — still blocked. Surfaces as a clean 422 with the sanitized tail: "Could not extract audio from that link: …Sign in to confirm you're not a bot…". Honest outcome: blocked here, ACCEPTABLE per spec (would work from residential IPs / with cookies).
  • Test 4 SoundCloud forss/flickermood: yt-dlp extracts fine (title "Flickermood", 213.9 s) but this serve only exposes m3u8_native formats (hls_aac_96k/160k — confirmed in raw -J JSON) → clean 422 adaptive(HLS/DASH) message, exactly the manifest-only contract case.
  • Test 5 https://example.com/ → clean 422 "That site is not supported — try YouTube, SoundCloud, or a direct MP3 link."; example.com/stream.m3u8 → 422 (does NOT hit the direct bypass).
  • Extractor SUCCESS path proven end-to-end with archive.org: resolve → ok:true, title:"mp3 test file", source:"archiveorg", durationSec:12.42; stream → 200 audio/mpeg 198,658 B + 206 ranged; cache replay 16 ms.
  • Stream negatives: unknown/missing token → 404 "Unknown or expired stream token"; upstream 500 (missing file) → 502 "Upstream failed (HTTP 500)."
  • lint: my 4 files pass eslint clean; bun run lint overall reports exactly 1 remaining error in src/components/ava/link-dialog.tsx:72 (set-state-in-effect) — that file is owned by the parallel frontend agent and was NOT touched by me per the ownership rules.
  • dev.log: no exceptions/stack traces from the new routes — only expected 200/206/400/404/422/502 request lines.
  • Deviations from spec (all deliberate): (1) one network-error retry in the proxy (upstream RSTs, never retries HTTP statuses); (2) third store helper refreshStreamEntry() — the two required helpers plus this one implements the specified "refresh sliding expiry AFTER upstream success"; (3) proxy fetch uses cache:'no-store'; (4) 500 wording "Audio extractor unavailable (yt-dlp not found or failed to start)."; (5) success cache keyed by the normalized URL href (target.href).

Task ID: 7 Agent: main (Z.ai Code) Task: Frontend integration of the "Link" audio source (YouTube / SoundCloud / direct MP3)

Work Log:

  • Read worklog + all relevant files (audio.ts, store.ts, audio-source.ts, header.tsx, visual-stage.tsx, footer-bar.tsx, types.ts) before editing.
  • src/lib/ava/types.ts — AudioSourceKind += 'link'
  • src/lib/ava/audio.ts — new startUrl(streamUrl, title): same-origin proxied element → MediaElementAudioSourceNode → analyser (NOT cross-origin-tainted, so FFT works) → speakers; loops by default; public sourceLabel field (file name / stream title) maintained by all start* methods + stopSources
  • src/lib/store.ts — AudioSourceState += 'link'; new sourceTitle + linkDialogOpen state
  • src/components/ava/audio-source.ts — new resolveAndPlayLink(url): POST /api/resolve → engine.audio.startUrl → store sync; never throws (returns {ok,title,source,error}); on playback failure stops the engine and resets store; syncSource now also syncs sourceTitle
  • src/components/ava/link-dialog.tsx — NEW: dialog with URL input + Load button (spinner), inline error box with YouTube bot-wall hint, RECENT LINKS chips (localStorage, max 5, one-click reload); body REMOUNTS on every open (fresh state — no setState-in-effect, fixes the lint error the Task 6 agent saw mid-build); results arriving after the user closed the dialog surface as toasts
  • src/components/ava/header.tsx — "Link" segment button between File and Stop (opens dialog); mounted in the header
  • src/components/ava/visual-stage.tsx — idle overlay gains a 4th button "Stream a Link" + updated copy
  • src/components/ava/footer-bar.tsx — SRC now shows "LINK ·