Independent world progression
A server monotonic clock starts at first browser ready. Inference, waiting, and refreshing do not reset it. Background events continue independently.
A BENCHMARK FOR ACTING IN TIME
The world keeps moving.
Can your agent keep up?
Computer use under shared state, asynchronous change, and real deadlines. Build agents that notice what changed, preserve what matters, and act in time.
52catalogue tasks
20OSWorld candidates
32implemented prototypes
THE CORE CHALLENGE
A correct ending is only part of the job.
Requirements arrive while the agent is working. Every round has a saved-state deadline. Missing an earlier checkpoint cannot be repaired by writing the right answer at the end.
REPORT TASK · SERVER CLOCK100 s total
08 s → 32 sRead. Update. Save.Round 1 · 24-second window
36 s → 60 sRequirements change.Round 2 · a new saved version
64 s → 88 sKeep up. Preserve edits.Round 3 · final checkpoint
Partner finishes at 90 sDeliver before 100 s ↗
Release scope
32 implemented local task packages and 20 OSWorld candidates. Six new English video-learning tasks run in native applications. Three interview/live-sports prototypes require locally supplied source media; the other packages include their task assets. Native applications and host dependencies are installed separately. Model runs and regraded results remain versioned; native OSWorld VM evaluation is not claimed.
New tasks and model replays
Explore the native rocket task, inspect actual model trajectories, and review each track’s maturity.
Native video learning
Blender, Calc, Impress, Inkscape, KiCad and Godot: complete tasks, tutorials and 14 versioned model videos.
ONE CATALOGUE, CLEAR STATUS
Task catalogue
Explore the scenarios, inspect complete prototype instructions, and download a self-contained task package.
MEASURING ACTING IN TIME
A clock the agent cannot pause.
Screenshot and GUI control
The agent sees the live interface and past or current messages. It acts with ordinary mouse and keyboard. Future events and grading targets stay on the trusted side.
Replay what actually happened
The evaluator replays ordered server events and saved versions. It checks deadlines, preserved content, conflicts, and actual delivery. Self-reported completion is insufficient.
Event visible→Screenshot captured→Inference returned→Action dispatched→Effective state change
For future unified latency reporting, record visibility and effective action times using one authoritative clock or a measured clock mapping. Report misses and timeouts alongside successful responses. The catalogue-wide protocol is still under development.
VERSIONED EVIDENCE
Historical model results
These results were reported for the upstream Chinese tasks in local screenshot-browser evaluations. They are included as provenance, not as results for this English release. Each cell is successful episodes / 5.
| Original task | Version | GPT | Qwen | Experiment |
|---|---|---|---|---|
| Quarterly report | 0.2.0 | 0 / 5 | 0 / 5 | 2538 |
| Procurement plan | 0.2.0 | 1 / 5 | 0 / 5 | 2538 |
| Release plan | 0.2.0 | 0 / 5 | 0 / 5 | 2538 |
| Story co-writing | 0.1.1 | 0 / 5 | 0 / 5 | 2552 |
| Changing arithmetic problem | 0.1.2 | 2 / 5 | 0 / 5 | 2552 |
| Ticket release · 1 s window | 0.4.0 | 0 / 5 | 0 / 5 | 2528 |
Experiments 2538 and 2528 use GPT-6.1 Sol (low) and original Qwen3.5-4B (thinking=true). Experiment 2552 arm labels are GPT and Qwen. Partial checkpoint performance is available in the aggregate JSON. Story grading needs trusted semantic review; pending review is not a numeric success score.
FROM TASK TO REPRODUCIBLE EPISODE
Run it. Reset it. Verify it.
Start a local prototype
git clone https://github.com/Rongjiehuang/RealtimeCUABench.git
cd RealtimeCUABench/tasks/rcua_doc_001
python3 initial_setup.py \
--run-dir /tmp/report-trial01 --port 0Open the printed URL. Use a fresh run directory for every episode. Browser tasks require Python 3.10+ and a modern browser; some bundled servers also require Node.js 18+. Native tasks require Linux and the application versions listed in HOST_SETUP.md. Three livestream tasks require local source video.
What each task includes
- Versioned task instructions and scenario configuration
- Application source and an independent event clock
- Clean setup and task-owned process cleanup
- Trusted journal-replay evaluator
- OSWorld-style entries for supported browser packages; Linux setup for native tasks
- Source provenance and SHA-256 checksums
OSWorld adapters require a separately installed framework and VM. Existing upstream candidates may require gated assets and version-matched services; they are catalogue references here. English local tests do not establish OSWorld VM compatibility or English model performance.