Skip to main content
Back

OpenSLA Turns Raw Sensor Streams Into Language-Guided Medical Decisions

A new multimodal framework turns raw sensor time series into language-grounded action predictions across clinical care, operating rooms, and metabolic health.

  • New Sensor-Language-Action framework unifies sensor time series, language, and open-ended action prediction in one paper.
  • OpenSLA uses LoRA adaptation of a pretrained LLM, beating Flamingo and LLaVA style fusion.
  • Benchmark covers 116,000+ individuals, 79 sensor modalities, 60 action groups across three healthcare domains.
  • Zero-shot generalization to unseen actions like held-out antibiotics via text embedding similarity.
  • CGM-derived representations predict future glucose statistics, hinting at implicit world-modeling behavior.
  • Code, weights, and demo available on GitHub with B and H variants.

Researchers have released Sensor-Language-Action modeling, a framework that turns multichannel sensor histories and language prompts into structured action predictions. Its open implementation, OpenSLA, also generates a caption describing the sensor evidence behind each prediction.

The framework extends sensor modeling from state recognition into action selection. Existing systems often classify conditions or forecast outcomes through task-specific output heads, so each new drug, intervention, or workflow label can require another classifier. OpenSLA represents those actions in language, giving different action vocabularies a shared semantic interface.

From sensor readings to decisions

OpenSLA receives a time-series history, relevant user or patient context, and a prompt describing the task. It returns two connected outputs:

  • Action prediction: A structured recommendation or label, such as an antibiotic, insulin dose, or surgical phase.
  • Sensor-state caption: A natural-language account of the signal patterns associated with that prediction.

Language connects the sensor evidence to open-ended action names. A single model can therefore work across label spaces that would otherwise need separate output heads, provided the actions have meaningful text descriptions and the model has enough grounding data.

LoRA handles the sensor-language fusion

OpenSLA converts sensor histories into tokens and inserts them into a pretrained language model’s context. The paper compares three alignment strategies: direct adaptation with low-rank adapters, Flamingo-style cross-attention, and LLaVA-style projection. In the reported experiments, direct LoRA adaptation performed best.