Goal
The aim of my thesis was a system that estimates a person's cognitive load in real time from wearable sensors while they walk on a treadmill, and responds with adaptive spoken feedback, entirely on-device, with no cloud services. I designed and built the system end to end, from the sensor drivers to the language model running on an edge computer, and validated it in a study with 12 participants.
What I built
- Pupil Labs Neon driver (
pupil_labs_ros): streams the eye tracker's real-time API into ROS 2 topics, with eye-tracking message types. Submitted upstream to the open-source ROS4Healthcare framework as a pull request. - Polar H10 driver (
polar_ros2): streams ECG and R-R intervals from the chest strap over Bluetooth Low Energy into ROS 2. - Synchronized recording: a time-synchronized multimodal stream, logged with rosbag2 for offline analysis.
- Cognitive-load estimator (
cognitive_load): heart-rate variability (RMSSD), z-scored against each participant's baseline, combined with pupil features into a live cognitive-load index. - On-device voice feedback: I brought an NVIDIA Jetson AGX Orin up from a bare JetPack install to run Qwen 2.5 3B Instruct through Ollama, with Piper for text-to-speech. The participant's physiological state is turned into a prompt, and the model's reply is spoken through a Bluetooth headset.
Study and results
In a 12-participant treadmill study, participants walked with and without an added mental task. The cognitive-load index followed the mental task in every participant and agreed with their NASA-TLX workload ratings.
- 12participants in the treadmill study
- 12 / 12participants in whom the index tracked the mental task
- r = 0.80correlation with NASA-TLX workload ratings