Extended reality (XR) is an umbrella term covering technologies that merge the physical ‘real world’ with the virtual ‘digital world’. A popular topic in science fiction, XR has inspired the Holodeck in Star Trek, the head-up display of Terminator, and more recently the OASIS in the novel and film - Ready Player One.
Despite its association with futuristic storylines, the first developments of XR predates the moon landings, when Morton Heilig prototyped his machine – the Sensorama, in 1962. Consisting of a booth that displayed five short films, the Sensorama simultaneously engaged with multiple human senses, including sight, sound, smell and touch. The result was that a person could take a virtual ride through the streets of New York, while feeling the wind on their face, and smelling the fumes in the air. In other words, the person could perceive being physically present in a non-physical world, a concept known as immersion.
In the years that followed, the desire for deeper and more believable immersive experiences led to developments of wearable technology, such as head mounted displays (HMDs), full body suits and handheld controllers with haptic feedback. These complementary systems provided the user with freedom to move and realistically engage with digital objects found within the virtual environment.
Within modern-day uses, XR has become a mainstream technology, which has been adopted by a range of industries, from military simulations to medical training. Furthermore, reports have indicated that patent applications filed at the USPTO relating to XR technology have doubled from 2015-2018, and forecasts are predicting further growth in XR over the next 5 years (even after accounting for the disruption from COVID-19).
Before exploring the technical reasons why XR technology is booming at the moment, this blog (the first in a series of XR related content) will provide a brief introduction to the technology behind XR, and the main industries that are currently adopting it to better society.
What does XR encompass?
Extended reality (XR) encompasses three technologies that sit along the spectrum of the physical and digital environment: virtual reality (VR), augmented reality (AR) and mixed reality (MR).
Virtual reality (VR) technology immerses the user into a fully digital environment. This digital environment comprises of a variety of digital objects that the user can interact with through either moving or gesturing in the real world. Current VR systems mostly use a headset, consisting of a display placed in front of the eyes to fill the peripheral vision and provide a sense of depth. Other VR systems include multiple projectors that shine onto a cylindrical or spherical screen to construct an expansive digital image.
Augmented reality (AR) technology overlays digital objects or information onto a visible real world environment. Early examples include head-up displays (HUDs) in military aircraft, which project digital information onto a transparent screen in front of the pilot. However, nowadays AR systems are able to detect physical objects, use AI algorithms to interpret and map these objects, and then display appropriate information or animations on the screen.
Mixed reality (MR) technology can be seen as a hybrid between VR and AR technologies. Similarly to AR, MR technology superimposes digital objects onto a physical environment, however MR technologies also enables for the user to interact with these digital objects. For example, by tracking hand-movements, MR technology allows for digital objects to be moved or manipulated using gestures. This technology therefore adds an extra layer of immersion compared to AR, whilst maintaining the visual perspective of the real world (unlike VR).
Where is XR currently in use?
There is no denying that XR has huge potential. It can teleport a person into a completely simulated environment or materialise virtual objects directly in front of the eyes, all without endangering the person’s health. Below is a list of the industries that have invested the most into XR technologies and some innovative examples of XR currently in use.
Entertainment
A significant factor for the growth in XR technologies is the investment from the entertainment industry - particularly the gaming industry. Virtual reality headsets, such as the Oculus Rift S, Playstation VR and HTC Vive Cosmos are very popular and enable consumers to play dedicated VR games, simulate a trip to the cinema or offer a front-row seat at a music concert. On the other hand, mobile based applications are increasingly including augmented reality features. For instance, camera apps now include filters that enhance or overlay objects onto facial features; navigation apps include textual information and directional prompts on top of the surrounding landscape; and mobiles games, such as the enormously popular Pokémon Go, encourage players to explore the world by combining with the view from the mobile camera and overlaying digital characters on screen to collect.
Healthcare
XR technology has been exploited for a variety of pain relief and therapeutic applications. A study at the University of Washington has shown that distractions are an excellent way to reduce patient stress levels, with VR experiences proving to be a highly effective means of distraction. Other studies have discovered correlations between using VR technologies, and improvements in mental health and a reduction in memory loss. VR is also in use for speeding up the recovery time in physical therapy, by training the patient’s cognitive response through training simulators and exercise games.
Another area within the healthcare industry that is utilising XR is medical training. XR technology is now able to transport a medical student inside the human body, allowing access to areas that were not previously possible without a cadaver. As well as reducing cost of storing a cadaver, the digital alternative provides a more representative experience of a living patient, and allows for students to practice operating with no risk to a patient’s health. More recently, studies have focussed on synchronising VR technology with robotic surgery equipment, enabling medical specialists to remotely operate on their patients.
Education and training
XR has been extensively used in education and training, because it offers an engaging and safe environment in which students can interact. For example, some schools are investing into AR teaching resources comprising t-shirts, which show digital representations of body parts when observed through a tablet or phone. Other schools are incorporating VR headsets into their teaching schedules to help their students understanding of the conditions and environments associated with the topic being studied, e.g. visiting the pyramids in Egypt without leaving the classroom.
Outside of schools, VR simulators are commonly used to train skilled professionals, such as pilots, soldiers, and athletes. These simulators allow for multiple people to be synchronised together, enabling teams to practice formations and communication in a controlled and safe environment.
In the business world, XR is being used to train relevant soft skills by showing a variety of role play scenarios and monitoring how employees respond. This approach is believed to be more effective than traditional training methods, e.g. multiple choice tests or instructive videos, because of the increased trainee engagement and the ability to track progress in much more depth.
Product design and engineering
XR technologies provide tools for presenting and visualising products in their intended environments without the need of building a prototype. This can be valuable when seeking opinions from stakeholders about the aesthetics of a design, or when checking the manufacturing procedures needed to assemble a product. Nowadays, computer aided design (CAD) software include VR compatible extensions for viewing product renders using a virtual reality headset, and allow for designs to be altered in real-time as they are being viewed.
Recently, augmented reality has been used to replace physical manuals commonly used by technicians during product assembly. In the building of the spacecraft Artemis II, NASA engineers have being using AR to improvement build speed and efficiency. In this case, components were labelled at specific areas, such that relevant manufacturing instructions would overlay on the respective component when the engineer is wearing a headset. This technology therefore allows engineers to focus on the job at hand, rather than be distracted by a printed manual or computer screen.
Conclusion
Extended reality (XR) encompasses all technologies that merge the real world and the digital world, e.g. virtual reality (VR), augmented reality (AR) and mixed reality (MR). Over the past decade, XR has become a mainstream technology, used in the entertainment, healthcare, education and engineering industries. The rapid increase in patent applications and promising financial forecasts indicate that XR will be used more and more during the 2020s.
In our next blog posts, we will explore the technical features that have accelerated the adoption of XR within society in the last decade, and the challenges associated with the field, which may cause further adoption of XR technology to stagnate.
