• Virtual reality interactions v...

Treffer: Virtual reality interactions via a user-generic ultrasound human-machine interface for wrist and hand tracking.

Title:
Virtual reality interactions via a user-generic ultrasound human-machine interface for wrist and hand tracking.
Authors:
Grandi Sgambato B; Department of Bioengineering, Imperial College London, London, UK., Hodossy BK; Department of Bioengineering, Imperial College London, London, UK., Barsakcioglu DY; Department of Bioengineering, Imperial College London, London, UK., Yang X; Department of Bioengineering, Imperial College London, London, UK.; School of Automation, Southeast University, Nanjing, China., Jakob A; Ultrasound Department, Fraunhofer Institut für Biomedizinische Technik, Sulzbach, Germany., Fournelle M; Ultrasound Department, Fraunhofer Institut für Biomedizinische Technik, Sulzbach, Germany., Tang MX; Department of Bioengineering, Imperial College London, London, UK., Farina D; Department of Bioengineering, Imperial College London, London, UK. d.farina@imperial.ac.uk.
Source:
Nature communications [Nat Commun] 2025 Dec 11; Vol. 16 (1), pp. 11062. Date of Electronic Publication: 2025 Dec 11.
Publication Type:
Journal Article
Language:
English
Journal Info:
Publisher: Nature Pub. Group Country of Publication: England NLM ID: 101528555 Publication Model: Electronic Cited Medium: Internet ISSN: 2041-1723 (Electronic) Linking ISSN: 20411723 NLM ISO Abbreviation: Nat Commun Subsets: MEDLINE
Imprint Name(s):
Original Publication: [London] : Nature Pub. Group
MeSH Terms:
Virtual Reality* , Wrist*/physiology , Wrist*/diagnostic imaging , Hand*/physiology , Hand*/diagnostic imaging , User-Computer Interface* , Man-Machine Systems*, Humans ; Ultrasonography/methods ; Ultrasonography/instrumentation ; Male ; Female ; Adult ; Biomechanical Phenomena ; Young Adult ; Deep Learning
References:
J Neuroeng Rehabil. 2024 Apr 16;21(1):57. (PMID: 38627772)
Sensors (Basel). 2019 Sep 04;19(18):. (PMID: 31487894)
IEEE Trans Neural Syst Rehabil Eng. 2022;30:2623-2629. (PMID: 36074871)
Sensors (Basel). 2024 Jun 06;24(11):. (PMID: 38894476)
J Rehabil Res Dev. 2008;45(1):187-95. (PMID: 18566937)
IEEE J Biomed Health Inform. 2022 Apr;26(4):1718-1725. (PMID: 34699373)
IEEE Trans Neural Syst Rehabil Eng. 2014 May;22(3):549-58. (PMID: 24235278)
IEEE Trans Biomed Eng. 2024 Feb;71(2):484-493. (PMID: 37610892)
IEEE Trans Neural Syst Rehabil Eng. 2012 Nov;20(6):788-97. (PMID: 22855234)
J Rehabil Res Dev. 2011;48(6):643-59. (PMID: 21938652)
IEEE Trans Ultrason Ferroelectr Freq Control. 2024 Jul;71(7):730-744. (PMID: 38090856)
IEEE Trans Biomed Eng. 2023 Feb;70(2):459-469. (PMID: 35881594)
Sci Data. 2014 Dec 23;1:140053. (PMID: 25977804)
J Biomech. 2020 Jan 2;98:109512. (PMID: 31767287)
J Biomech. 2005 May;38(5):981-992. (PMID: 15844264)
Med Eng Phys. 2006 Jun;28(5):405-15. (PMID: 16115790)
Annu Int Conf IEEE Eng Med Biol Soc. 2014;2014:4627-30. (PMID: 25571023)
IEEE Trans Neural Syst Rehabil Eng. 2013 Jul;21(4):616-23. (PMID: 23193252)
Nat Biomed Eng. 2018 Sep;2(9):687-695. (PMID: 30906648)
J Biomech. 2016 Jun 14;49(9):1742-1750. (PMID: 27102160)
IEEE Trans Biomed Eng. 2016 Aug;63(8):1687-98. (PMID: 26560865)
J Electromyogr Kinesiol. 2012 Dec;22(6):908-13. (PMID: 22613823)
Sensors (Basel). 2021 Mar 03;21(5):. (PMID: 33802495)
Science. 2022 Jul 29;377(6605):517-523. (PMID: 35901155)
IEEE Trans Ultrason Ferroelectr Freq Control. 2024 Jul;71(7):831-841. (PMID: 38787674)
Nature. 2025 Sep;645(8081):702-711. (PMID: 40702190)
IEEE Trans Ultrason Ferroelectr Freq Control. 2020 Nov;67(11):2345-2353. (PMID: 32746188)
Sensors (Basel). 2021 Mar 10;21(6):. (PMID: 33802231)
Sensors (Basel). 2024 Aug 04;24(15):. (PMID: 39124090)
IEEE J Transl Eng Health Med. 2022 Jan 06;10:2100311. (PMID: 35070521)
Annu Int Conf IEEE Eng Med Biol Soc. 2019 Jul;2019:982-988. (PMID: 31946058)
IEEE J Biomed Health Inform. 2019 Jul;23(4):1639-1646. (PMID: 30176612)
IEEE Trans Hum Mach Syst. 2024 Jun;54(3):317-324. (PMID: 38974222)
Sensors (Basel). 2021 Sep 28;21(19):. (PMID: 34640807)
J Neuroeng Rehabil. 2011 Apr 20;8:19. (PMID: 21507238)
IEEE Trans Biomed Circuits Syst. 2025 Jun;19(3):536-548. (PMID: 39312417)
Nature. 2023 Jan;613(7945):667-675. (PMID: 36697864)
Grant Information:
899822 EC | Horizon 2020 Framework Programme (EU Framework Programme for Research and Innovation H2020)
Entry Date(s):
Date Created: 20251211 Date Completed: 20251211 Latest Revision: 20251214
Update Code:
20251214
PubMed Central ID:
PMC12699032
DOI:
10.1038/s41467-025-66001-6
PMID:
41381418
Database:
MEDLINE

Weitere Informationen

As computers move from desktop screens into our glasses, traditional controllers such as keyboards and mice have proven impractical. A control interface for immersive experiences needs to seamlessly transport intention from the real to the virtual world while remaining portable, accurate, and robust. Here, we present an easy-to-wear, dry-contact and portable ultrasound armband that can decode morphological information and act as a virtual reality controller by predicting hand and wrist kinematics. Using our armband, we collected a large dataset of paired ultrasound and hand kinematics and used it to train supervised deep-learning models capable of predicting hand kinematics from ultrasound. We explored how diverse intra-session, cross-session, and cross-participant data shifts affect model performance. Further, we proposed methods for data conditioning, augmentation, and a referencing strategy to mitigate the influence of confounding factors and to achieve accurate prediction of hand kinematics on unseen users without fine-tuning. Finally, we demonstrated the feasibility of our interface in a real-time virtual reality control framework. Using the developed ultrasound interface, participants completed challenging interaction tasks with simulated contact physics. This work demonstrates the potential of ultrasound-based technologies as a virtual reality interface, showcasing strong performance, robustness, and generalization potential.
(© 2025. The Author(s).)

Competing interests: The authors declare no competing interests.