Replicating a Sense of Touch in the Metaverse with Robotic Hands
A way to touch and feel in virtual worlds, powered by novel soft robotics
We could all use more reasons to be optimistic, so I’m starting a side project called precisely that.
Going forward, I’ll be sharing deep dives on the technologies I believe everyone should pay attention to, but perhaps aren’t. Think low-level enough to interest engineers, but high-level enough for everyone to grasp why it matters. Read on below for the first deep dive.
VR today is an impressive experience. Wearing a Meta Quest 2—with no awkward cables or setup—can feel like you’ve been teleported to a virtual world. What’s more, the potential of VR is massive, with the promise to unlock new opportunities in billion or trillion-dollar spaces like labor, social, gaming, entertainment, healthcare, and fitness. However, as impressive as VR is and as far-reaching its potential is, the experience today often feels uneven and like it’s missing something. In other words, VR does not yet enable true immersion (otherwise known as the metaverse).
Why is this? Well, VR headsets today are isolated to input primarily for your eyes and ears, meaning your other senses (touch, taste, smell) are left to remain in the physical world. This disjointed experience means you have a foot in both the physical and virtual worlds, resulting in fragmented immersion.
What if VR could connect more of your senses—beyond just sight and sound—to a virtual world? What might that leap enable? This is precisely what Meta Reality Labs (MRL) is working on, beginning with touch. Imagine if you could touch objects in VR. Imagine if you could hold an object and feel the shape, ridges, and every nook and cranny. What important VR applications might that core sense enable?
This future is becoming a reality, thanks to advancements in technology at the intersection of haptics, soft robotics, and microfluidics, enabling people to touch, feel, and manipulate virtual objects. The team at MRL is bringing this to life, by building soft, lightweight, and robotic gloves.
Generating Artificial Touch and Feel
For a glove to produce artificial sensations requires placing lots of tiny actuators along the surface. These actuators then work in harmony to create different pressures in parts of your hand, replicating what it might feel like when you pick up or hold objects.
Integrating hundreds of actuators in a glove that both feels natural and lightweight is a tremendous challenge, requiring a fresh approach to robotics. Traditional robotic, mechanical actuators are uncomfortable, produce a lot of heat and are too bulky, rigid, expensive, and power-hungry to render realistic haptic sensations.
For these reasons, MRL is turning to soft robotics to construct a robotic glove that can make you touch and feel in virtual environments.
Soft robots are a kind of robot constructed with soft, deformable, and elastic materials. This is in stark contrast to traditional robots, which are heavy, inflexible, and industrial. A fundamental aspect of soft robotics is the creation of intricate, many-segmented units that can move non-linearly and in a more versatile way. For example, instead of a hard metal surface, a soft robot surface is made up of tiny soft materials that can move almost like human skin.
The MRL team is leveraging these characteristics of soft robotics to build a robotic glove that is dramatically more lightweight, precise in its control, and power-efficient than traditional, mechanical robots.
The team is not just leveraging off-the-shelf soft robotics, but bringing many of their own advancements to the space. They have developed a PneuSleeve, which is a forearm sleeve with a broad range of haptic stimuli that includes compression, skin stretch, and vibration. These stimuli are created by pneumatic pressure inside the stretchable tubes, with two compression actuators and four uniformly distributed linear actuators. In effect, they have integrated soft haptic devices, sensors, and actuators directly into garments and improved the usability and wearability for daily computing interactions.
How Actuators in Soft Robotics Work
The most critical part of the robotic glove is the soft actuators. These actuators are utilized to create sensations and apply forces that mimic the feeling of lifting or touching a real object. They are then often combined with rigid robot bodies, or embedded in soft robots to control them. Soft actuators are mainly categorized based on their actuation methods as:
Pneumatic Actuators (SPAs): SPAs convert the energy of compressed air or gas into a mechanical motion. MRL is primarily using these actuators in their robotic glove.
Cable-Driven Actuators: cables are connected to the actuators which are outside of the robot and transmit the tensions through the cable.
Electroactive Polymers (EAPs): EAPs are flexible actuators that are composed of ion exchange polymer films, with electrodes that have relatively low voltages on both sides and are capable of sensing and actuating under harsh conditions. MRL has also worked with these, as they can change shape or size in the presence of an electrical field.
Shape Memory Alloys (SMAs): SMAs generate relatively large displacements and high force/weight ratios.
Although soft actuators have clear advantages over robots made of rigid materials, they are not without issue. Compared to rigid robots, soft actuators have challenges in modelling, calibration, and control in that the innate characteristics of the soft materials can cause complex behaviours. This is due to non-linearity and hysteresis, which means that there is often a lag between input and output of a system once direction changes. Put another way, controlling soft actuators is a difficult task today. However, controlling the actuators is crucial to precisely release sensations on the hand of the user when they are interacting with an object.
To control these new soft actuators, MRL has been focused on building the world’s first high-speed microfluidic processor. This is a (very) tiny microfluidic chip on the glove that controls the airflow that moves the actuators, by telling the valves when and how far to open and close. Because of this, MRL has delivered a response time with air in the order of milliseconds. This is compared to existing processors, which process in the order of seconds.
Another emerging field that promotes a way to overcome limitations in soft actuator control is using model predictive control (MPC) to perceive the position of the devices and control them to the desired positions. MPC allows us to control a process using neural networks and first-principles modelling while satisfying any set of constraints in the system.
If you’re interested to dive deeper into this approach, this is a paper that utilized and tested the results for two different linearized models (first-principles and DNN) for predictive control.
Soft Robotics Will Power Many Amazing Applications
Soft robotics not only presents the possibility of breakthrough consumer devices, but also promises to push forward lab-on-chip diagnostics, micro-fluidic biochemistry, materials science, and computational science. Soft robotics is a world-changing technology, and MRL is not alone in pushing it forward:
The Harvard Biodesign Lab leverages soft robotics to build a portable soft robotics glove to facilitate home-based neuromuscular rehabilitation for stroke survivors with hand impairment through repetitive stretching exercises. The soft robot can mimic the natural movement of humans which can help the patient restore motor functions.
Neuroprosthetics today are extremely expensive and are built using metal and electrical parts which are heavy and rigid. Researchers at MIT are developing a soft, lightweight, and low-cost prosthetic hand using soft robotics.
Soft robotics is often used in medical applications such as surgery, due to their ability to conform and have safe interactions with the human body. For example, researchers have used soft robotics for endoscopic surgery.
There are other companies building gloves and the entire systems necessary for accurate motion tracking and strong force feedback. For example, HaptX and SenseGlove.
Prototype Today. Consumer Product Tomorrow
MRL is early in its journey to enable touch and feel in virtual worlds. I am certain that the reliability, cost, and accuracy of its robotic glove are not yet where it needs to be to transform into an everyday consumer product. But, like the early iPhone prototypes before it, it is obvious to see its world-changing potential.
What’s more, as MRL pushes forward this project, they also push forward the fields of microfluidics, hand tracking, haptic rendering, perceptual science, and much more. It’s impossible to predict exactly what those many advancements will enable for the world, but it will be mind-blowing!