Our lab's work is very inter-disciplinary and often cuts across domains. Thus, it is often hard to put our work into discrete buckets. Still, in case you are interested in learning about our work in a particular area, we have tried to categorize our work into four broad categories: Health Sensing, Activity Recognition, Privacy-aware Sensing, New Interfaces, and Situational Impairments. Many publications are in overlapping areas and you will find those publications in multiple categories.
2024
PrISM-Q&A: Step-Aware Question Answering with Large Language Models Enabled by Multimodal Procedure Tracking using a Smartwatch.
PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch
PDF Video
Perception of Indoor Lighting: A Comparative Study in Physical and Virtual Environments
Bring Privacy To The Table: Interactive Negotiation for Privacy Settings of Shared Sensing Devices
EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose Estimation
PDF Video
Kirigami: Lightweight Speech Filtering for Privacy-Preserving Activity Recognition using Audio
JoulesEye: Energy Expenditure Estimation and Respiration Sensing From Thermal Imagery While Exercising
PDF Video
2023
VAX: Using Existing Video and Audio-based Activity Recognition Models to Bootstrap Privacy-Sensitive Sensors
MI-Poser: Human Body Pose Tracking using Magnetic and Inertial Sensor Fusion with Metal Interference Mitigation
PDF Video
LemurDx: Using Unconstrained Passive Sensing for an Objective Measurement of Hyperactivity in Children with no Parent Input
uKnit: A Position-Aware Reconfigurable Machine-Knitted Wearable for Gestural Interaction and Passive Sensing using Electrical Impedance Tomography
PDF Video
IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds.
PDF Video Code
PrISM-Tracker: A Framework for Multimodal Procedure Tracking Using Wearable Sensors and State Transition Information with User-Driven Handling of Errors and Uncertainty
2022
A semantic-based approach to digital content placement for immersive environments
RGBDGaze: Gaze Tracking on Smartphones with RGB and Depth Data
PDF Video
SAMoSA: Sensing Activities with Motion and Subsampled Audio
PDF Video
Objective Measurement of Hyperactivity Using Mobile Sensing and Machine Learning: A Pilot Study
FitNibble: A Field Study to Evaluate the Utility and Usability of Automatic Diet Monitoring in Food Journaling Using an Eyeglasses-based Wearable.
PDF Video Code
2021
IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data
PDF Video
TouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images
PDF Video
EyeMU Interactions: Gaze + IMU Gestures on Mobile Devices
PDF Video
Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition
PDF Video
Pose-on-the-Go: Approximating User Pose with Smartphone Sensor Fusion and Inverse Kinematics
PDF Video
Classroom Digital Twins with Instrumentation-Free Gaze Tracking
PDF Video
PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition
CoolMoves: User Motion Accentuation in Virtual Reality
2020
Detecting Depression and Predicting its Onset Using Longitudinal Symptoms Captured by Passive Sensing: A Machine Learning Approach With Robust Feature Selection.
BodySLAM: Opportunistic User Digitization in Multi-User AR/VR Experiences
PDF Video
Direction-of-Voice (DoV) Estimation for Intuitive Speech Interaction with Smart Devices Ecosystems
PDF Video
Understanding Client Support Strategies to Improve Clinical Outcomes in an Online Mental Health Intervention
PDF Video
Eyes on the Road: Detecting Phone Usage by Drivers Using On-Device Cameras
PDF Video
Beyond the Prototype: Understanding the Challenge of Scaling Hardware Device Production
Digital Ventriloquism: Giving Voice to Everyday Objects
PDF Video
FitByte: Automatic Diet Monitoring in Unconstrained Situations Using Multimodal Sensing on Eyeglasses.
PDF Video Dataset
2019
MeCap: Whole-Body Digitization for Low-Cost VR/AR Headsets
PDF Video
LightAnchors: Appropriating Point Lights for Spatially-Anchored Augmented Reality Interfaces
PDF Video
Detachable Smartwatch: More Than A Wearable
PDF Video
Interferi: Gesture Sensing using On-Body Acoustic Interferometry
PDF Video
BeamBand: Hand Gesture Sensing with Ultrasonic Beamforming
PDF Video
GymCam: Detecting, recognizing, and tracking simultaneous exercises in unconstrained scenes
PDF Video
Challenges in Realizing Smartphone-based Health Sensing
2018
RainCheck: Overcoming Capacitive Interference Caused by Rainwater on Smartphones
Ubicoustics: Plug-and-Play Acoustic Activity Recognition
PDF Video Code
In only 3 minutes: perceived exertion limits of smartwatch use
EyeSpyVR: Interactive Eye Sensing Using Off-the-Shelf, Smartphone-Based VR Headsets
PDF Video
2017
Carpacio: Repurposing Capacitive Sensors to Distinguish Driver and Passenger Touches on In-Vehicle Screens
PDF Video
EarBit: using wearable sensors to detect eating episodes in unconstrained environments
DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-mounted Displays
PDF Video
2016
Design and Learnability of Vortex Whistles for Managing Chronic Lung Function via Smartphones
SpiroCall: Measuring Lung Function over a Phone Call
PDF Video
2015
MagnifiSense: Inferring Device Interaction Using Wrist-worn Passive Magneto-inductive Sensors
HyperCam: Hyperspectral Imaging for Ubiquitous Computing Applications
PDF Video
Tongue-in-Cheek: Using Wireless Signals to Enable Non-Intrusive and Flexible Facial Gestures Detection
PDF Video
SwitchBack: Using Focus and Saccade Tracking to Guide Users’ Attention for Mobile Task Resumption
Wibreathe: Estimating respiration rate using wireless signals in natural settings in the home
2014
BiliCam: Using Mobile Phones to Monitor Newborn Jaundice
PDF Video
AirLink: Sharing Files Between Multiple Devices Using In-air Gestures
PDF Video
SurfaceLink: Using Inertial and Acoustic Sensing to Enable Multi-device Interaction on a Surface
PDF Video
2013
DopLink: Using the Doppler Effect for Multi-device Interaction
PDF Video
ContextType: Using Hand Posture Information to Improve Mobile Touch Screen Text Entry
2012
GripSense: Using Built-in Sensors to Detect Hand Posture and Pressure on Commodity Mobile Phones
PDF Video
SpiroSmart: Using a Microphone to Measure Lung Function on a Mobile Phone
WalkType: using accelerometer data to accomodate situational impairments in mobile touch screen text entry
Open data kit sensors: a sensor integration framework for android at the application-level
Open Data Kit Sensors: Mobile Data Collection with Wired and Wireless Sensors
Health Sensing Publications
JoulesEye: Energy Expenditure Estimation and Respiration Sensing From Thermal Imagery While Exercising
PDF Video
LemurDx: Using Unconstrained Passive Sensing for an Objective Measurement of Hyperactivity in Children with no Parent Input
uKnit: A Position-Aware Reconfigurable Machine-Knitted Wearable for Gestural Interaction and Passive Sensing using Electrical Impedance Tomography
PDF Video
PrISM-Tracker: A Framework for Multimodal Procedure Tracking Using Wearable Sensors and State Transition Information with User-Driven Handling of Errors and Uncertainty
Objective Measurement of Hyperactivity Using Mobile Sensing and Machine Learning: A Pilot Study
FitNibble: A Field Study to Evaluate the Utility and Usability of Automatic Diet Monitoring in Food Journaling Using an Eyeglasses-based Wearable.
PDF Video Code
Detecting Depression and Predicting its Onset Using Longitudinal Symptoms Captured by Passive Sensing: A Machine Learning Approach With Robust Feature Selection.
Understanding Client Support Strategies to Improve Clinical Outcomes in an Online Mental Health Intervention
PDF Video
FitByte: Automatic Diet Monitoring in Unconstrained Situations Using Multimodal Sensing on Eyeglasses.
PDF Video Dataset
GymCam: Detecting, recognizing, and tracking simultaneous exercises in unconstrained scenes
PDF Video
Challenges in Realizing Smartphone-based Health Sensing
Design and Learnability of Vortex Whistles for Managing Chronic Lung Function via Smartphones
SpiroCall: Measuring Lung Function over a Phone Call
PDF Video
HyperCam: Hyperspectral Imaging for Ubiquitous Computing Applications
PDF Video
Wibreathe: Estimating respiration rate using wireless signals in natural settings in the home
BiliCam: Using Mobile Phones to Monitor Newborn Jaundice
PDF Video
SpiroSmart: Using a Microphone to Measure Lung Function on a Mobile Phone
Open data kit sensors: a sensor integration framework for android at the application-level
Open Data Kit Sensors: Mobile Data Collection with Wired and Wireless Sensors
Privacy-Aware Sensing Publications
Bring Privacy To The Table: Interactive Negotiation for Privacy Settings of Shared Sensing Devices
Kirigami: Lightweight Speech Filtering for Privacy-Preserving Activity Recognition using Audio
VAX: Using Existing Video and Audio-based Activity Recognition Models to Bootstrap Privacy-Sensitive Sensors
SAMoSA: Sensing Activities with Motion and Subsampled Audio
PDF Video
IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data
PDF Video
Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition
PDF Video
PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition
GymCam: Detecting, recognizing, and tracking simultaneous exercises in unconstrained scenes
PDF Video
Activity Recognition Publications
PrISM-Q&A: Step-Aware Question Answering with Large Language Models Enabled by Multimodal Procedure Tracking using a Smartwatch.
PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch
PDF Video
VAX: Using Existing Video and Audio-based Activity Recognition Models to Bootstrap Privacy-Sensitive Sensors
IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds.
PDF Video Code
PrISM-Tracker: A Framework for Multimodal Procedure Tracking Using Wearable Sensors and State Transition Information with User-Driven Handling of Errors and Uncertainty
SAMoSA: Sensing Activities with Motion and Subsampled Audio
PDF Video
FitNibble: A Field Study to Evaluate the Utility and Usability of Automatic Diet Monitoring in Food Journaling Using an Eyeglasses-based Wearable.
PDF Video Code
IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data
PDF Video
Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition
PDF Video
PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition
FitByte: Automatic Diet Monitoring in Unconstrained Situations Using Multimodal Sensing on Eyeglasses.
PDF Video Dataset
GymCam: Detecting, recognizing, and tracking simultaneous exercises in unconstrained scenes
PDF Video
Ubicoustics: Plug-and-Play Acoustic Activity Recognition
PDF Video Code
EarBit: using wearable sensors to detect eating episodes in unconstrained environments
MagnifiSense: Inferring Device Interaction Using Wrist-worn Passive Magneto-inductive Sensors
New Interfaces Publications
PrISM-Q&A: Step-Aware Question Answering with Large Language Models Enabled by Multimodal Procedure Tracking using a Smartwatch.
PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch
PDF Video
EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose Estimation
PDF Video
MI-Poser: Human Body Pose Tracking using Magnetic and Inertial Sensor Fusion with Metal Interference Mitigation
PDF Video
uKnit: A Position-Aware Reconfigurable Machine-Knitted Wearable for Gestural Interaction and Passive Sensing using Electrical Impedance Tomography
PDF Video
IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds.
PDF Video Code
A semantic-based approach to digital content placement for immersive environments
RGBDGaze: Gaze Tracking on Smartphones with RGB and Depth Data
PDF Video
TouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images
PDF Video
EyeMU Interactions: Gaze + IMU Gestures on Mobile Devices
PDF Video
Pose-on-the-Go: Approximating User Pose with Smartphone Sensor Fusion and Inverse Kinematics
PDF Video
Classroom Digital Twins with Instrumentation-Free Gaze Tracking
PDF Video
CoolMoves: User Motion Accentuation in Virtual Reality
BodySLAM: Opportunistic User Digitization in Multi-User AR/VR Experiences
PDF Video
Direction-of-Voice (DoV) Estimation for Intuitive Speech Interaction with Smart Devices Ecosystems
PDF Video
Eyes on the Road: Detecting Phone Usage by Drivers Using On-Device Cameras
PDF Video
Beyond the Prototype: Understanding the Challenge of Scaling Hardware Device Production
Digital Ventriloquism: Giving Voice to Everyday Objects
PDF Video
MeCap: Whole-Body Digitization for Low-Cost VR/AR Headsets
PDF Video
LightAnchors: Appropriating Point Lights for Spatially-Anchored Augmented Reality Interfaces
PDF Video
Detachable Smartwatch: More Than A Wearable
PDF Video
Interferi: Gesture Sensing using On-Body Acoustic Interferometry
PDF Video
BeamBand: Hand Gesture Sensing with Ultrasonic Beamforming
PDF Video
RainCheck: Overcoming Capacitive Interference Caused by Rainwater on Smartphones
In only 3 minutes: perceived exertion limits of smartwatch use
EyeSpyVR: Interactive Eye Sensing Using Off-the-Shelf, Smartphone-Based VR Headsets
PDF Video
DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-mounted Displays
PDF Video
HyperCam: Hyperspectral Imaging for Ubiquitous Computing Applications
PDF Video
Tongue-in-Cheek: Using Wireless Signals to Enable Non-Intrusive and Flexible Facial Gestures Detection
PDF Video
SwitchBack: Using Focus and Saccade Tracking to Guide Users’ Attention for Mobile Task Resumption
AirLink: Sharing Files Between Multiple Devices Using In-air Gestures
PDF Video
SurfaceLink: Using Inertial and Acoustic Sensing to Enable Multi-device Interaction on a Surface
PDF Video
DopLink: Using the Doppler Effect for Multi-device Interaction
PDF Video
GripSense: Using Built-in Sensors to Detect Hand Posture and Pressure on Commodity Mobile Phones
PDF Video
Situational Impairments Publications
Eyes on the Road: Detecting Phone Usage by Drivers Using On-Device Cameras
PDF Video
Detachable Smartwatch: More Than A Wearable
PDF Video
RainCheck: Overcoming Capacitive Interference Caused by Rainwater on Smartphones
Carpacio: Repurposing Capacitive Sensors to Distinguish Driver and Passenger Touches on In-Vehicle Screens
PDF Video
Tongue-in-Cheek: Using Wireless Signals to Enable Non-Intrusive and Flexible Facial Gestures Detection
PDF Video
SwitchBack: Using Focus and Saccade Tracking to Guide Users’ Attention for Mobile Task Resumption
ContextType: Using Hand Posture Information to Improve Mobile Touch Screen Text Entry
GripSense: Using Built-in Sensors to Detect Hand Posture and Pressure on Commodity Mobile Phones
PDF Video
WalkType: using accelerometer data to accomodate situational impairments in mobile touch screen text entry