Build one data sample and one environment sample to this standard and nothing more. Everything else waits until a buyer scopes a paid delivery and we agree acceptance criteria with them.
08 menu, one Environment Card or Data Sheet with real numbers, 2–3 baseline figures.
Echo-S1 colour-mixing environment (20–50 target RGBs) and the PP-01 interpretation set (~100 items).
Everything in the grey "later" rows, against acceptance criteria agreed in writing.
Target: PP-01 interpretation set, about 100 double-reviewed items across colour mixing, titration and absorbance.
| Item | Done when | Owner |
|---|---|---|
| Items | 50–150 records in one JSONL file; each has id, prompt, input file references and a difficulty tag | — |
| Raw evidence | every item links to its spectra, images or instrument files, with units | — |
| Answer / rubric | every item has a reference answer or weighted binary criteria | — |
| Schema | one schema file; a script validates every record without errors | — |
| Double review | two reviewers per item, disagreements adjudicated, agreement figure reported | — |
| Frontier baselines | 3 frontier models scored on the full set; no model above roughly 60% | — |
| Short Data Sheet | one page: contents, counts, how made, who made it, known issues | — |
| Rights statement | confirms own production and signed contributor consent for AI-training use | — |
pp01-interpretation-sample/ ├── items.jsonl ├── raw/ # evidence files ├── schema.json ├── baselines.json # 3 models ├── review_agreement.md ├── DATASHEET.md # one page └── RIGHTS.md # one paragraph
Target: Echo-S1 colour mixing. Each task gives a target RGB; the agent proposes volumes of dyed stock solutions and gets back the RGB predicted by a model fitted to our real camera and spectrometer runs. 20–50 target colours. Every competent agent can reach the colour, so the score comes from the path: the final ΔE is a gate, then safety and efficiency decide the score. Many different paths score full marks.
| Item | Done when | Owner |
|---|---|---|
| Task folders | 20–50 folders, each with instruction.md, task.json, environment/, verifier/, witnesses/ | — |
| task.json | one template shared by all tasks: id, family, tier, version, difficulty, agent and verifier timeouts, CPU and memory, allow_internet false | — |
| Shared base image | one image holds the colour model, propose_recipe and query_history; each task Dockerfile is a one-line FROM | — |
| Rubric | one rubric.json: ΔE ≤ 3 as the gate, then weighted safety, attempts and material use; scorer reads every step, never compares recipes to a witness | — |
| Witness check | 2–3 witnesses per task taking clearly different paths all score full under the current rubric; check.json regenerates on every rubric change | — |
| Baselines + headroom | 3 models, several runs each; pass@k within the attempt budget and ΔE distribution; share of runs reaching full score roughly 10–60% (tighten the rubric if higher), and compared with random search and Bayesian optimisation at the same budget | — |
| Anti-hacking note | tried: probing the twin, interpolating from query history, out-of-range or negative volumes, blind guessing on the first attempt, doing nothing to avoid violations, format tricks; each scored low | — |
| One-command run | a colleague outside the team runs it on a clean machine with network off; run.sh also exports the Harbor layout; rough GPU-hours recorded | — |
| Environment Card | one page: goal, tools (propose_recipe, query_history), ΔE scoring, colour range the twin covers, held-out error against real runs | — |
| Rights statement | confirms the S1 runs behind the twin are ours to license | — |
echo-s1-sample/ ├── tasks/ │ └── task-0007/ │ ├── instruction.md # target RGB, budget, rules │ ├── task.json # timeouts, resources, network off │ ├── environment/ │ │ └── Dockerfile # FROM echo-s1-base │ ├── verifier/ │ │ ├── rubric.json # the only scoring standard │ │ ├── score.py # reads every step │ │ └── test.sh # writes the reward │ └── witnesses/ │ ├── expert_01.jsonl │ ├── expert_02.jsonl │ ├── bo_01.jsonl │ └── check.json # auto: scores under rubric ├── run.sh # one command; Harbor export ├── baselines.json ├── ENVIRONMENT_CARD.md ├── ANTI_HACKING.md # NDA copy └── RIGHTS.md
| Ours | Harbor | Why ours differs |
|---|---|---|
| instruction.md | instruction.md | same |
| task.json | task.toml | JSON instead of TOML, same fields |
| environment/ | environment/ | same |
| verifier/ | tests/ | says what it does |
| witnesses/ | solution/ | full score is a space, not one answer; these prove it is reachable |
run.sh converts to the Harbor layout for buyers who run Harbor directly, including task.json to task.toml. Live tier later reuses instruction.md, task.json and verifier/ unchanged; only the world moves from the colour model to real instruments.