Educator Developer Blog

8 MIN READ

UCL MotionInput v3.2: Equitable Computing for Public Health, Ageing and Physical Conditions

Copper Contributor

Jun 15, 2023

Introduction

UCL MotionInput, supported by Intel, Microsoft and IBM, was developed with the latest open source of Artificial Intelligence (AI), Speech Recognition and Computer Vision technologies to provide a solution for touchless computing as inputs to existing software during the Covid-19 pandemic [1]. Innovative features include in-air multitouch, facial navigation, hit triggers in the air, speech hotkeys to override keyboard shortcuts and more.

A report published by WHO[9] states that approximately one-sixth of people worldwide have a significant disability. Physical movement limitations that people face make it challenging to use technology which has now become a part of our daily lives. As physical abilities decline with old age or physical illness, some technology interactions become out of reach. Users adopt methods for interactivity making the most of technology with the capabilities they have. Smart devices have reduced access complexities in some cases, such as finding news and communicating, however, more can be done in examining the extent of user capabilities and relaying those as daily human control instructions. We present a work-in-progress engineering and feature update (v3.2) for several conditions, prototyped as input combinations to existing user interfaces. UCL MotionInput 3.2 has over 150 students and academic developers that are modelling use case requirements for patients, physical rehabilitation and accessibility needs.

UCL MotionInput accommodates various physical limitations, and its suitability can be tailored to a person’s physical capabilities. Version 3.2 now presents significant improvements in features for specific disability requirements. Meanwhile, the software engine has been re-developed and tested with partners at Intel, for robustness, performance, and compatibility with a wider range of computer hardware, making it especially applicable also in the fields of home and remote-based physiotherapy, exercise, self-enabled care, and gaming entertainment.

Figure 1 Images of MotionInput users performing different activities. User 1 (Left) is using In-Air Multitouch functionality to navigate a PC with Windows Touchpoints registered in the air, User 2 (Middle) User is playing a FIFA game by using her feet only, User 3 (Right) is playing a boxing game by using punching gestures as input with forcefields of varying depths registered as well as closed hands detection.

Technology Discussion

UCL MotionInput captures visual input via a device’s webcam and combines several different sources such as hands, body, face and speech to form human inputs. These could be in the form of mouse cursor events as well as different keyboard actions and customised combinations of actions, for instance, joypad inputs. These can be modified by a user as needed for their software and games that they already own. We can generally divide all types of UCL MotionInput Combinatorial Modules described in Table 1, which can form inputs alone or mixed together.

Table 1 Explanation of the MotionInput modules

Motioninput 3.2 use cases

The previous sections provided an overview of UCL MotionInput’s range of technical capabilities. This section explores high-impact fields from given user requirements collected in 2022 from UK and International charities, including members from the International Alliance of ALS MND Associations, Team Gleason, various Physiotherapy trusts in NHS England & Wales, GOSH DRIVE and VR Therapies UK.

MND/ALS

MND patients often have difficulties with full body movement, including limitations of the neck range of motion and speech clarity [2].UCL MotionInput’s Facial Navigation enables users to independently control computer actions through user-chosen combinations of facial gestures and expressions, such as smiling, opening the mouth, raising eyebrows, and tilting the head, to tailor the range of movement to the user’s comfort levels.

Cerebral Palsy

Users suffering from Cerebral Palsy have difficulties with movement, posture and coordination [3]. A specific symptom of this condition is contractures resulting in a curved hand. Standard hand tracking will also not work in this case and movements can be irregular and repetitive. Wrist tracking will allow the user to still interact with mouse movement and perform page up/page down commands. 2-level forcefield interactions will enable a user to move between a close-to-body position posture, to a limited reached extent posture.

Mild Stroke

Post-stroke patients can often experience significant motion control difficulties [4] in the left or right side of the body and cerebellar tremors (shaky hands). The current work in progress aims to (a) with hand tracking, smoothen/isolate/dismiss the unintended user input that these patients may provide with a single hand (left or right) gesture control and (b) with In-range facial navigation, examine the extents of the user’s movements and correlates that to the full dimensions of the screen space for navigation.

Autism

Users with varying levels of Autism often have difficulties with verbal communication and visual acknowledgement [5]. UCL MotionInput’s engine expand on requirements using visualisation and music to generate visual and sound effects with customisable gesture shapes such as swimming, wall projection triggers and audible feedback to drive reassurance and acknowledgement of inputs.

Early Dementia and Alzheimer’s

UCL MotionInput’s customisable speech commands and speaker identification feature are being designed to assist people with cognitive impairment disorders such as Dementia and Alzheimer’s in operating devices and remembering information [6]. Custom speech commands can also simplify actions like contacting loved ones or playing movies.

Wheelchair-bound users

Customised hand gestures to trigger events and customised arm movements for hit triggers (wheelchair hand/arm movements to go forward in a game/walk) and facial navigation for further restricted use cases.

Amputees and loss of arms

Many 3D modern games software are unplayable for a user with a loss of one or both arms, where the left hand on a joypad may correlate to a game character’s movement, and the right-hand correlates to game character actions and aiming. UCL MotionInput features a hybrid mode for one arm to aim, speech, facial switches and in-air multitouch for actions, and feet movement or facial navigation for character movement.

Motor limitations in elderly/children

Adapting UI event triggers to a person’s height and speed of movement by configuring combinations of hands, face, speech, and body gestures comfortably to everyone, this could include selecting a multiplayer option. UCL MotionInput provides a calibration process for the range of motion that a user is comfortable with once the software is run, which can also include custom hand or facial gestures, or speech commands.

ADHD

One of the barriers with ADHD is the lack of motivation with repetitive activities [7]. This is especially challenging for users trying to learn. Interaction with movements focuses on visual and audible triggers, which can be useful in future software design [8]. New APIs in UCL MotionInput 3.2 will explore these triggers with the charity VR Therapies UK and the International Alliance of ALS/MND Associations.

UCL MotionInput’s software offers combinations of user capabilities to be mapped to the user’s needs. The next stages of this work from these reported software development feature updates will involve usability testing in these reported domains, in alignment with the supporting charities and organisations.

Demostration

The MotionInput software can be used for any application that receives keyboard/mouse input. During a live demonstration of the tool, the MotionInput teams tested different combinations of modules across a variety of games. Figure 2 shows images from the practical demonstration of UCL MotionInput. Table 2 then lists other exemplary uses of MotionInput in relation to various software tasks.

Figure 2 Info-graphic displaying examples of UCL MotionInput software used in practice

Table 2 Example uses of UCL MotionInput v3.2 in various categories of software tasks

Limitations

UCL MotionInput 3.2 does not solve input for all combinations of physical abilities. However, as a starting point and in collaboration with various charities and accessibility needs groups, we wish to start user testing domains to improve this software. Noise reduction and gesture accuracy are continually being improved, as is the software engine for performance on further devices. The first of the software packages has already been published to the Microsoft Store (search “UCL” in the UK, USA and Canada).

Speech commands may also not be as effective or accurate in a noisy environment and this is something to be improved upon in the near future with noise band filters and further voice identification methods.

Summary

This technology expands upon a well-defined and highly optimised software engine that encompasses recognition models for speech recognition and synthesis to facial switches, body landmarks, sound triggers and head movements for equitable access to existing software and especially games.

The latest version 3.2 has achieved significant improvement in feature ranges thanks to working with various charities and significant efficiency and systems design improvements. It shows promise for future applications and usefulness for computer vision based accessibility, population health and wellbeing, and motion-based gaming.

References

[1] Sheena Visram, Dean Mohamedally, Graham Roberts, Ali Hassan, Ashild Kummen, Chenuka Ratwatte, Robert Shaw, Stefano Giuliani, An- drew Taylor, Joseph Connor, Atia Rafiq, Neil Sebire, and Yvonne Rogers. 2022. UCL MotionInput: Touchless computing interactions in clinical training, radiology and operating theatres. Future Healthcare Journal 9, 3 (2022), 343–345. https://doi.org/10.7861/fhj.2022-0018 arXiv: https://www.rcpjournals.org/content/9/3/343.full.pdf

[2] Ben Niu, Zhenxing Gao, and Bingbing Guo. 2021. Facial Expression Recognition with LBP and ORB Features. Computational intelligence and neuroscience 2021 (01 2021), 8828245. https://doi.org/10.1155/2021/8828245

[3] Fatimahwati Hamzah and Saiful Hasley Ramli. 2022. A Systematic Review of Assistive Technology Devices to Promote Independent Living in Children with Cerebral Palsy. EAI. https://doi.org/10.4108/eai.24-8-2021.2315270

[4] Yu Chen, Kingsley Travis Abel, John T. Janecek, Yunan Chen, Kai Zheng, and Steven C. Cramer. 2019. Home-based technologies for stroke rehabilitation: A systematic review. International Journal of Medical Informatics 123 (2019), 11–22. https://doi.org/10.1016/j.ijmedinf.2018.12.001

[5] Allison L. Wainer and Brooke R. Ingersoll. 2011. The use of innovative computer technology for teaching social communication to individuals with autism spectrum disorders. Research in Autism Spectrum Disorders 5, 1 (2011), 96–107. https://doi.org/10.1016/j.rasd.2010.08.002

[6] M. de Sant’Anna, C. Vallet, R. Kadouche, D. Stefanucci, A. Tomascakova, B. Morat, and A.-S. Rigaud. 2010. Computer accessibility for individuals suffering from mild to moderate Alzheimer’s disease. European Geriatric Medicine 1, 3 (2010), 186–192. https://doi.org/10.1016/j.eurger.2010.04.003

[7] Miriam Götte, Sabine Kesting, Corinna Winter, Dieter Rosenbaum, and Joachim Boos. 2014. Experience of barriers and motivations for physical activities and exercise during treatment of pediatric patients with cancer. Pediatric Blood & Cancer 61, 9 (2014), 1632–1637. https://doi.org/10.1002/ pbc.25071 arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1002/pbc.25071

[8] B. Bonnechère, B. Jansen, L. Omelina, and S. Van Sint Jan. 2016. The use of commercial video games in rehabilitation: a systematic review. International Journal of Rehabilitation Research. Internationale Zeitschrift fur Rehabilitationsforschung. Revue internationale de recherches de readaptation 39, 4 (2016), 277–290. https://doi.org/10.1097/MRR.0000000000000190

[9] https://www.who.int/publications/i/item/978924156418

Microsoft Store links: Search for "UCL" on the Microsoft Store (UK/USA/Canada), instruction guide and download links on https://www.facenav.org, project website on https://www.touchlesscomputing.org.

Video and article prepared by and in collaboration with:

ANELIA GAYDARDZHIEVA, University College London, UK

FILIP TRHLÍK, University College London, UK

IMAAD ZAFFAR, University College London, UK

CHRIS ZHANG, University College London, UK

MOHSEEN HUSSAIN, University College London, UK

LIV URWIN, University College London, UK

TINA HOU, University College London, UK

DONGYEON PARK, University College London, UK

POOJA CHHAYA, University College London, UK

JASON KIM, University College London, UK

ABID ALI, University College London, UK