July 2026
Open Data Eval for Robotics
Auditing 109 egocentric datasets and turning “is this data good?” into a machine-readable score
Models are only as good as the data they train on, yet dataset quality still gets judged by vibes — paper reputation, download count, whether a lab you trust released it. Code gets audited systematically, quantitatively, transparently. Data doesn't. Open Data Eval is my attempt to fix that for egocentric and manipulation datasets: audit them the way we audit code, and publish the results as an interactive scorecard.
The catalog
The base layer is a catalog of 109 egocentric video datasets, each audited across 33 fields — 135,500+ video hours in total. The headline finding is not about resolution or frame rate. It's about paperwork:
| Datasets audited | 109 |
| Total video hours | 135,500+ |
| Fully accessible | 96 (89%) |
| Broken downloads | 5 (5%) |
| Dead links | 2 (2%) |
| No license specified | 67 of 109 (61%) |
Almost two-thirds of these datasets ship with no license at all. People train on them anyway. “Open” and “usable” turn out to be very different claims once you actually try to check them.
From catalog to quality profiles
Catalog rows tell you what exists. To say how good a dataset is, 29 of them get a machine-readable Quality Profile (QP) — a structured record of what the dataset is and how it scores. Each QP is:
- Croissant-compatible JSON-LD — it slots into MLCommons metadata infrastructure instead of inventing a new format.
- ISO/IEC 5259-2 aligned — 9 of the standard's 23 data-quality characteristics are mapped at the metadata level.
- Progressively enriched — metadata → file → frame → content, each phase adding depth to the same profile.
A profile carries 13 numeric scores across five dimensions, plus classifications (device, lens, video format, annotation format) and a metadata-completeness confidence score. Ego4D, for example, scores fps 1.0 (30fps), resolution 1.0 (1080p), license_clarity 0.5 (custom), accessibility 0.80, and annotation_coverage 1.0 — though that last one hides a catch I'll come back to.
The five dimensions
Every metadata-level score rolls up into five readable dimensions:
- Technical — frame rate, resolution.
- Scale — hours, environment diversity, participant count.
- Annotation — how much of the footage is actually labeled.
- Accessibility — license clarity, access level, URL status, dataloader, docs.
- Reliability — camera-calibration tier, modality richness per use-case.
Accessibility is the one fully-computed dimension, scored 0–10 from six weighted components: access level (3), URL status (2), license clarity (2), documentation (1.5), dataloader (1), and commercial clarity (0.5). Nothing here is a vibe — every point traces back to a checkable fact.
On top of that sits Downstream Fit: how well a dataset serves a specific job. The scorecard scores three — Action Recognition, Hand-Object Interaction, and Navigation — against tiers of what each use-case treats as Critical, Important, and Bonus. A dataset can be excellent overall and still a bad fit for your task.
Navigating the scorecard
The live scorecard is where the profiles become legible. Two ways to use it:
- Profile view — pick a dataset and read its card, broken into six sections: Technical, Accessibility & Docs, Reliability, Scale & Diversity, Downstream Fit, and Hardware & Format. Reliability even spells out which calibration parameters exist — intrinsics, distortion, extrinsics, whether any of it was validated.
- Compare view — put two datasets side-by-side on license, access level, URL status, dataloader, docs, and the three downstream-fit scores. This is the “which one do I actually pull?” view.
A few things the scores surface
Accessibility separates cleanly. The datasets that are easiest to actually get:
| Dataset | Score | License |
|---|---|---|
| EPIC-KITCHENS-100 | 9.8 | CC-BY-NC-4.0 |
| HO-Cap | 9.5 | CC-BY-4.0 |
| EPIC-KITCHENS-55 | 9.2 | CC-BY-NC-4.0 |
| DexYCB | 9.2 | CC-BY-NC-4.0 |
| EgoDex | 9.2 | CC-BY-NC-ND-4.0 |
Calibration is mostly missing. Ego4D provides no camera calibration across 7 device types and 74 collection sites — fine for action recognition, a real problem for anything geometric. Reliability scores make that gap visible instead of leaving it buried in a paper appendix.
“Annotated” is a headline, not a fact. Ego4D's annotation coverage reads 100% — because narrations cover all 3,670 hours. Gaze covers 2.2%, 3D covers 17%. The profile records the caveat so the top-line number can't mislead you on its own.
Where this is going
Metadata is only the first layer. The full framework has five:
- Technical — resolution, fps, codec, sharpness, frame drops.
- Content — hand visibility, action density, scene diversity.
- Collective — diversity indices, demographic balance, coverage.
- Derivative — downstream model lift: which data actually makes models better.
- Accessibility — license, download, docs, dataloader.
Phase 1a — what's live today — covers Accessibility fully and the first three layers partially, all from metadata. File Eval (an ffprobe pipeline) and Frame Eval (ML models on sampled frames for blur, occlusion, hand visibility) come next, then a robot dataset catalog. The endgame is Layer 4, Derivative Quality: the only score that answers the question underneath all the others — which data actually produces better models. Everything above it is a proxy until that exists.