Imagine Martha, an octogenarian who lives independently and uses a wheelchair. All objects in her home are digitally catalogued; all sensors and the devices that control objects have been Internet-enabled; and a digital map of her home has been merged with the object map. As Martha moves from her bedroom to the kitchen, the lights switch on, and the ambient temperature adjusts. The chair will slow if her cat crosses her path. When she reaches the kitchen, the table moves to improve her access to the refrigerator and stove, then moves back when she is ready to eat. Later, if she begins to fall when getting into bed, her furniture shifts to protect her, and an alert goes to her son and the local monitoring station.

The “spatial computing” at the heart of this scene is the next step in the ongoing convergence of the physical and digital worlds. It does everything virtual-reality and augmented-reality apps do: digitize objects that connect via the cloud; allow sensors and motors to react to one another; and digitally represent the real world. Then it combines these capabilities with high-fidelity spatial mapping to enable a computer “coordinator” to track and control the movements and interactions of objects as a person navigates through the digital or physical world. Spatial computing will soon bring human-machine and machine-machine interactions to new levels of efficiency in many walks of life, among them industry, health care, transportation and the home. Major companies, including Microsoft and Amazon, are heavily invested in the technology.

As is true of virtual and augmented reality, spatial computing builds on the “digital twin” concept familiar from computer-aided design (CAD). In CAD, engineers create a digital representation of an object. This twin can be used variously to 3-D-print the object, design new versions of it, provide virtual training on it or join it with other digital objects to create virtual worlds. Spatial computing makes digital twins not just of objects but of people and locations—using GPS, lidar (light detection and ranging), video and other geolocation technologies to create a digital map of a room, a building or a city. Software algorithms integrate this digital map with sensor data and digital representations of objects and people to create a digital world that can be observed, quantified and manipulated and that can also manipulate the real world.

In the medical realm, consider this futuristic scenario: A paramedic team is dispatched to an apartment in a city to handle a patient who might need emergency surgery. As the system sends the patient's medical records and real-time updates to the technicians' mobile devices and to the emergency department, it also determines the fastest driving route to reach the person. Red lights hold crossing traffic, and as the ambulance pulls up, the building's entry doors open, revealing an elevator already in position. Objects move out of the way as the medics hurry in with their stretcher. As the system guides them to the ER via the quickest route, a surgical team uses spatial computing and augmented reality to map out the choreography of the entire operating room or plan a surgical path through this patient's body.

Industry has already embraced the integration of dedicated sensors, digital twins and the Internet of Things to optimize productivity and will likely be an early adopter of spatial computing. The technology can add location-based tracking to a piece of equipment or an entire factory. By donning augmented-reality headsets or viewing a projected holographic image that displays not only repair instructions but also a spatial map of the machine components, workers can be guided through and around the machine to fix it as efficiently as possible—shrinking down time and its costs. Or if a technician were engaging with a virtual-reality version of a true remote site to direct several robots as they built a factory, spatial-computing algorithms could help optimize the safety, efficiency and quality of the work by improving, for example, the coordination of the robots and the selection of tasks assigned to them. In a more common scenario, fast-food and retail companies could combine spatial computing with standard industrial engineering techniques (such as time-motion analyses) to enhance the efficient flow of work.