University of Catania's ENIGMA-360 Exposes How Industrial AI Fails Across Camera Views
ENIGMA-360 pairs 180 headset videos with 180 synced fixed-camera videos of real repair work. Top video models break down when they have to switch between the two views.
Takeaways
- ENIGMA-360 offers 180 headset videos and 180 time-synced fixed-camera videos of repair work in a real industrial lab.
- Labels cover 14,556 steps across 68 step types, plus 275K object boxes, 56K hands, masks, DINOv2 features and 3D scans.
- Segmentation models score well within one view, but F1 falls to single digits when switching camera views.
- Keystep recognition is much stronger on headset video (GLEVR 0.82 F1) than fixed-camera video (best 0.63 F1).
- On a new cross-view matching task, Qwen3-VL-8B and InternVL-3.5-8B fall to near-chance 25% on hard cases.
- Data and downloader are on GitHub. Footage comes from one lab, two procedures and 33 participants.
Industrial procedure models are often trained on staged assembly benchmarks featuring toy components or visually simple parts. Working facilities are harder to record because employee privacy, safety controls, and proprietary processes restrict access. Researchers at the University of Catania and its spin-off Next Vision have released ENIGMA-360, a benchmark built from 360 videos captured in an operating industrial laboratory. Dense annotations support four tasks on the same footage: temporal action segmentation, keystep recognition, hand-object interaction detection, and cross-view video question answering.
Benchmark results expose a severe camera-transfer problem. Leading models perform well when training and evaluation use the same viewpoint, but some scores fall to low single digits when models move between a worker’s headset and a fixed wall camera. On the hardest clip-matching task, vision-language models score close to the 25% random baseline. Developers building multi-camera assistants therefore face unresolved problems in view alignment, procedural tracking, and fine-grained temporal reasoning.
A Soldering Job From Two Angles
Egocentric, or ego, video records the wearer’s point of view. ENIGMA-360 captures it with a Microsoft HoloLens 2, showing hands, tools, and close contact in detail. Exocentric, or exo, video comes from a fixed ZED camera observing the whole scene. That view preserves posture and workspace context, while small hand movements may appear tiny or become occluded. Pairing both perspectives lets researchers test whether a model can recognize the same event across sharply different visual conditions.
The collection contains 180 ego videos and 180 exo videos arranged as synchronized pairs. Its 33 participants ranged from 20 to 70 years old and from novices to experts. They completed two electronics repair procedures designed with industrial specialists, one involving a high-voltage board and the other a low-voltage board. The laboratory contains 25 types of equipment, including an oscilloscope, soldering station, power supply, electric screwdriver, and electrical panels with buttons.
The synchronization method uses a lamp on the workbench. Each participant switches it on at the start of a session, producing a visible flash that aligns the headset and fixed-camera streams.
During recording, a HoloLens application delivered spoken instructions and mixed-reality reference images. Participants moved through the procedure using commands such as “Forward” and “Repeat.” Each step change generated a timestamp, giving annotators an initial temporal boundary to review. The average video lasts 18.59 minutes, yielding about 111.5 hours across both views.