Conference Presentation, Keynote, Panel
Matthew Turk
- The speaker defines Augmented Reality (AR) not by hardware, but by an experiential integration of digital information (visual, audio, haptic) into real-world experiences in real time.
- Key examples of existing AR include the "yellow first down" marker in football broadcasts and digitally inserted campus scenes in UC Santa Barbara labs.
- The technology aims to provide "superpowers" such as X-ray vision, seeing the invisible, changing scale, and teleportation.
- The speaker proposes "Being There," a system designed for mobile tele-collaboration in unprepared physical environments.
- The goal is to enable "teleportation" where a remote expert can virtually join a local novice to perform physical tasks, such as equipment repair.
- The system prioritizes two core capabilities: independent exploration of the scene by multiple users and communication within a shared 3D reference frame.
- Unlike Skype or FaceTime, the system allows remote users to detach from a live video stream to explore a generated 3D model independently, zoom, freeze viewpoints, and navigate freely.
- The "Being There" prototype utilizes a smartphone to build a 3D model of the physical scene, which is transmitted to the remote user for independent navigation.
- Annotations made by the remote user on the model are sent back to the local user's device, where they are spatially registered to the real-world view.
- Real-world validation included experiments with a student fixing a computer chassis and a scenario involving car engine troubleshooting in a remote location.
- The research addresses three primary technical challenges to achieve effective remote collaboration:
- Environment Modeling: Creating usable 3D models from noisy, imperfect data captured by mobile devices in unprepared environments.
- The solution merges Simultaneous Localization and Mapping (SLAM) for 3D structure with panoramic tracking for 2D rotational data, dynamically selecting the best model based on camera motion (translational vs. rotational).
- Image-Based Navigation: Enabling remote users to navigate the scene using the imperfect hybrid 3D/2D model.
- Annotation Rendering: Ensuring 2D gestures made from one perspective are correctly rendered from another user's perspective.
- A study with 88 Amazon Mechanical Turk users evaluated four rendering models: spray paint, minimum depth plane, median depth plane, and dominant surface plane.
- The "gesture-enhanced by surface normal" model was identified as the most effective, successfully inferring user intent to render 2D circles and arrows correctly across different viewpoints.
- Environment Modeling: Creating usable 3D models from noisy, imperfect data captured by mobile devices in unprepared environments.
- Cognate, a startup founded by the speaker and a former PhD student, is commercializing this technology for the field service sector.
- The initial target market includes 5.5 million field service technicians in the U.S.
- The system addresses a demographic shift where there are fewer senior experts and more inexperienced young technicians.
- Key value propositions include reduced expert travel costs, decreased equipment downtime, faster problem resolution, and higher customer satisfaction.
- Future development focuses on solving the challenge of dynamic scenes and non-rigid objects, which remain unsolved in the current framework.
- The long-term business strategy envisions an evolution from Business-to-Business (B2B) field service applications to B2C and eventually C2C remote collaboration.