Vision Zero Agent

Hours of raw footage from a conflict zone. Five seconds that matter. Real-time explosion alerts for the news desk, and the same agent watching streets and warehouse floors.

VAST Builders Challenge · team-4

Archival film: 51 of 56 explosions found

Explosions: 91% recall · 85% precision

222 hand-checked 5 s clips from five films: B-17 raids, the RAF attack on the Tirpitz, RAF night bombing, USAAF "Special Delivery" (1946) and "Disney Bomb" (1945).

Known limit: bomb release

10% precision. Parachutes and supply drops read as bombs; a stricter prompt only reached 12%. We show it, not hide it.

Wikimedia Commons, public domain. Sent to Cosmos directly for evaluation; not ingested into the VSS archive.

2

Live trigger: every blast alerted, zero false alarms

  • Incoming film is cut into 5 s segments as it arrives; Cosmos Reason checks each one.
  • On an explosion: the clip is saved, a toast, chime and desktop notification fire, and an email or webhook goes out.
  • Three films it had never seen (1957 A-bomb newsreel, 1946 Bikini underwater test, 1952 Ivy Mike): 3 of 3 blasts alerted, 0 false alarms in 36 segments, 13 of 21 explosion segments flagged.
  • Misses: the first flash before the cloud forms; colour fireball close-ups read as a sunset.

Truth: human frame check of every segment. Wikimedia Commons, public domain; sent to Cosmos only, not ingested into VSS.

3

Same agent, live camera archives on VAST

3,537 clips · 13 cameras

Indexed on VAST: Toronto dashcam, a New York cyclist's helmet camera, NYC and San Francisco street cameras, a highway, a neighborhood camera.

2 warehouse floors

Forklifts, AGVs and humanoid robots sharing space with workers.

The question

Where did a person come dangerously close to a moving vehicle or machine?

The answer

A ranked, human-verified incident feed with bounding boxes, plain-English alerts and written reports.

4

How it works

1 · VAST VSS

Ingests and segments video into 5 s clips; YOLO boxes, captions and embeddings in VastDB.

→

2 · Cosmos Reason

Watches each clip (on CoreWeave GPUs) with a street or warehouse safety prompt: risk 0–5, event, machine, people nearby.

→

3 · Checks

Consistency rule (no human near the machine means no alert); human confirm/dismiss with notes.

→

4 · W&B

gpt-oss-120b writes reports and compiles watch rules; Weave traces every call; eval runs measure accuracy.

Web app deployed on the team Kubernetes namespace behind the /app Ingress. Open it from workshop.thecosmoslabs.com → App.

5

One view of every camera

  • Scoreboard ranks cameras by average risk.
  • 357 clips scored on the video by Cosmos.
  • 29 high-risk events; 10 confirmed and 5 dismissed by a reviewer.
  • Report, watch-rule and Q&A panels on the right.
6

Warehouse floor: workers vs. machines

  • Cosmos: RISK 3 near miss, forklift, 1 worker near.
  • Red boxes: people in a risky clip, from the VSS YOLO detections, synced to playback.
  • Reviewer note outranks the model: "beside it as it turns; well under 1 m".
  • 240 warehouse clips scored; 20 at RISK 3.
7

City streets: Vision Zero

  • Cyclist helmet camera, New York.
  • Red people, amber vehicles; dashed line to the closest vehicle.
  • RISK 4 crosswalk conflict, confirmed: rider squeezed against an SUV as pedestrians step into the bike lane.
8

Watch rules in plain English

  • "Alert me when a forklift comes near a worker on the warehouse floor."
  • gpt-oss-120b on W&B inference compiles it to a JSON filter: machine = forklift, warehouse cameras.
  • Alerts list instantly, confirmed clips first.
9

One-click incident report

  • Executive summary, worst incidents, patterns across cameras, recommended actions with evidence.
  • Reviewer notes override the model; unreviewed clips are labeled unverified.
10

Measured, not guessed: W&B Weave + evaluation

Check (human frame review = ground truth)Result
Street clips: Cosmos risk vs. reviewer8 / 10 agree
Warehouse clips (after the zero-humans consistency rule)6 / 6 agree
Archival war film (5 films, 222 clips): explosions85% precision · 91% recall
Live trigger, 3 unseen films (36 segments): explosions3 / 3 blasts alerted · 0 false alarms
Archival war film: bomb release10% precision

Every Cosmos Reason and LLM call is traced in Weave (project jayakumaronline/vision-zero-agent). Results are logged as the W&B run "cosmos-reason-frame-checked-eval". Reviewed street and warehouse clips are spot checks, not a random sample.

11

Built in one day

VAST

Video storage, semantic search, Q&A over the archive, YOLO detections, streaming.

NVIDIA Cosmos on CoreWeave GPUs

Cosmos Reason watches every clip and scores the risk.

Weights & Biases

Serverless inference for reports and watch rules; Weave tracing; evaluation runs.

Cursor

Built, deployed and tested with Cursor agents, subagents and skills.

12

Vision Zero Agent

Any camera, any archive. Every finding checked on the video, and every number measured.

Thank you · team-4