Intelligent Motion Laboratory at UIUC

The Tele-Robotic Intelligent Nursing Assistant (TRINA)

Yifan Zhu, Joao Marcos Correia Marques, Patrick Naughton, Jing-chen Peng, James Seungbum Nam, Jane Li, Peter Moran, Carrina Dong, Ryan Shaw, Kris Hauser

Summary

During outbreaks of contagious diseases, healthcare workers are at high risk for infection due to routine interaction with patients, handling of contaminated materials, and challenges associated with safely removing protective gear. This project is developing the Tele-Robotic Intelligent Nursing Assistant (TRINA), a remote-controlled robot to address these challenges. Such robots could perform common nursing duties inside hazardous clinical areas, which could reduce infection risk to healthcare workers by minimizing exposure to contagions and other biohazards.

TRINA consists of a mobile manipulator robot, a human operator's console, and software for connecting the operator's input to the robot. Ideally, the robot should act as a "surrogate body" (or avatar) so that the operator can perform any task that he/she could normally perform. However, robot actuation and sensing hardware is far from the capabilities of the human body. There are a number of ongoing research challenges including hardware that provides human-level strength, dexterity, perceptual sensitivity, and situational awareness. Moreover, TRINA (version 1.0) has 28 independently controllable degrees-of-freedom, and coordinating these movements can be quite challenging. Tests in 2017 showed the robot successfully completing approximately 60% of nursing tasks at an average of 20x slower than human performance. Enhancing the physical capability of tele-nursing robots and their ease of use for the operator has been a major effort and poses engineering, scientific and clinical application challenges.

UIUC's Intelligent Motion Lab has been investigating several dimensions of the tele-robotic nursing problem, including intuitive input devices, improved contextual awareness, and operator assistance algorithms that automate or partially-automate tedious and error-prone tasks. We also use TRINA as a platform to study different aspects of developing immersive robot avatars for telepresence, developing two novel TRINA robots over the course of the ANA Avatar XPRIZE , a global competition aimed at fostering the development of this technology. Our entry, AVATRINA, placed 4^th in this competition and was one of only 4 teams to complete all the tasks in the competition's course. Notably, our team demonstrated that using commodity control devices and VR systems, untrained operators could control the robot to perform complex tasks, such as blind remote texture sensing.

The TRINA effort has inspired several related efforts, including the Human-Inspired Robotics Lab at WPI that has been investigating tele-manipulation for many years now. The TRINA platforms continue to be a resource for robotics and AR/VR research, education, and outreach, and are often featured in public demonstrations at UIUC. Demos involving TRINA have twice won Best Demo Award at UIUC's Coordinated Sciences Lab (CSL) Student Conference.

P. Naughton, J. S. Nam, A. Stratton and K. Hauser. Integrating Open-World Shared Control in Immersive Avatars. IEEE International Conference on Robotics and Automation, 2024. Also in Arxiv, January 5, 2024. Supplemental video
K. Hauser, E. Watson, J. Bae, J. Bankston, S. Behnke, B. Borgia, M. Catalano, S. Dafarra, J.B.F. van Erp, T. Ferris, J. Fishel, G. Hoffman, S. Ivaldi, F. Kanehiro, A. Kheddar, G. Lannuzel, J.F. Morie, P. Naughton, S. NGuyen, P. Oh, T. Padir, J. Pippine, J. Park, D. Pucci, J. Vaz, P. Whitney, P. Wu, and D Locke. Analysis and Perspectives on the ANA Avatar XPRIZE Competition. International Journal on Social Robotics, January 30, 2024. doi:10.1007/s12369-023-01095-w
N. Zhong and K. Hauser. Attentiveness Map Estimation for Haptic Teleoperation of Mobile Robot Obstacle Avoidance and Approach. IEEE Robotics and Automation Letters, January 16, 2024. Print ISSN: 2377-3766, Online ISSN: 2377-3766, doi:10.1109/LRA.2024.3354613
J. M. C. Marques, P. Naughton, J.-C. Peng, Y. Zhu, J. S. Nam, Q. Kong, X. Zhang, A. Penmetcha; R. Ji, N. Fu, V. Ravibaskar, R. Yan, N. Malhotra, and K. Hauser. Immersive Commodity Telepresence with the TRINA Robot Avatar. International Journal of Social Robotics, January 14, 2024. doi:10.1007/s12369-023-01090-1.
J. M. C. Marques, J.-C. Peng, P. Naughton, Y. Zhu, J. S. Nam, and K. Hauser. Commodity Telepresence with Team AVATRINA's Nursebot in the ANA Avatar XPRIZE Finals. ICRA 2023 2nd Workshop on Toward Robot Avatars, June 2, 2023.
P. Naughton, J. S. Nam, J. M. C. Marques, J.-C. Peng, Y. Zhu, Q. Kong, and K. Hauser. Pan-Tilt-Roll Televisualization With Adjustable Baseline Stereo. ICRA 2023 2nd Workshop on Toward Robot Avatars, June 2, 2023.
J. M. C. Marques, P. Naughton, Y. Zhu, N. Malhotra, and K. Hauser. Commodity Telepresence with the AvaTRINA Nursebot in the ANA Avatar XPRIZE Semifinals. RSS 2022 Workshop on Toward Robot Avatars: Perspectives on the ANA Avatar XPRIZE Competition. July 1, 2022.
Y. Zhu, A. Smith, and K. Hauser. Automated Heart and Lung Auscultation in Robotic Physical Examinations. IEEE Robotics and Automation Letters, 7(2):4204-4211, April 2022. doi: 10.1109/LRA.2022.3149576
P. Naughton and K. Hauser. Structured Action Prediction for Teleoperation in Open Worlds. IEEE Robotics and Automation Letters, 7(2):3099-3105, April 2022. doi: 10.1109/LRA.2022.3145953. Supplemental video
Y. Zhu, A. Smith, and K. Hauser. Informative Path Planning for Automatic Robotic Auscultation. ICRA 2021 Workshop on Impact of COVID-19 on Medical Robotics and Wearables Research, June 4, 2021. Summary video
K. Hauser and R. Shaw. How Medical Robots Will Help Treat Patients in Future Outbreaks. IEEE Spectrum, May 4, 2020.
T. Lu, H. Bader, and K. Hauser. The Design and Doffing of Personal Protective Equipment for Healthcare Robots. Military Health Systems Research Symposium (MHSRS), August 2018.
Z. Li, P. Moran, C. Dong, R. Shaw, and K. Hauser. Development of a Tele-Nursing Mobile Manipulator for Remote Care-giving in Quarantine Areas. IEEE Int'l. Conf. on Robotics and Automation (ICRA), May 2017.
J. Li, Z. Li and K. Hauser. A Study of Bidirectionally Telepresent Tele-action During Robot-Mediated Handover. IEEE Int'l. Conf. on Robotics and Automation (ICRA), May 2017.

Video explaining the integration of shared control in immersive avatar technologies using VR

Video showing a novice operator using TRINA 3.0 to complete a challenge course during the ANA Avatar XPRIZE Finals

Video showing team AVATRINA (using TRINA 2.0) at our ANA Avatar XPRIZE semifinals verification demo.

Video showing TRINA 2.0 performing a demonstration of autonomous heart and lung auscultation

Video showing the capabilities of TRINA v1.0, as tested in a nursing simulation lab at Duke University School of Nursing.

Video showing TRINA's personal protective equipment (PPE) and self-doffing capabilities. (link to TRINA self-doffing project)

TRINA 3.0 and AVATRINA Team at the ANA Avatar XPRIZE competition

TRINA v1.0 as shown in the nursing simulation lab at Duke University School of Nursing

TRINA v1.0 operator's console, showing dual haptic input devices, game pad, several camera streams, and 3D visualization of robot.

Several tasks on which TRINA has been evaluated

TRINA has been evaluated on 26 common nursing tasks and is able to complete many of them. We show a human nurse in the frames corresponding to tasks that TRINA is not currently able to complete.