3D Holograms in AR
The AT&T 5G Pilot Product Launch
JAUNT XR CAST PROMISES
AN EASY-TO-USE, HIGH FIDELITY, TRANSPORTABLE SOFTWARE & HARDWARE SOLUTION FOR ONSITE PERFORMANCE CAPTURE AND LIVE/STREAMING DELIVERY TO AR APPS.
JAUNT XR CAST TECHNOLOGY
Jaunt XRCast is a volumetric capture technology that records human performance in 3D for playback or live streaming in AR applications. It’s comprised of a multi-sensor array or “capture stage” that is portable and features a simple calibration and capture application for on-site events. The capture technology processes the depth (voxel) and rgb information into a streamable format that includes meshes and UV texture maps. Quality and density of the texture and mesh is optimized for the platform, device and network conditions in real-time.
How do we elevate Jaunt’s latest product XR CAST and show it’s creative potential?
PILOT PROJECT
Demonstrating viable use cases for XRCast as a product brought Jaunt into a partnership with AT&T. The upcoming 5G network from AT&T promises consumers low-latency and incredibly large data streams. As lead designer, I was tasked with demonstrating to AT&T how XRCast can provide consumers with a look at what is possible with 5G.
““How do we demonstrate
the kinds of experiences
possible through 5G?”
“How can we make
low-latency and high
data throughput
meaningful to the
consumer?” -AT&T”
JAUNT XR CAST
TAILGATE LAUNCH EVENT
The partnership with AT&T, Nokia and Jaunt would mark XRCast’s first product release and would take the form of an event activation meant to highlight the upcoming features of AT&T’s 5G network. The event would take place at the Atlanta Falcons vs Dallas Cowboys NFL Game Day Tailgate village. The event would provide an XR Cast demonstration that would utilize a 5G network to deliver an AR experience for fans to discover.
The pitch - “There’s no better way to show off your team spirit than with your very own touchdown dance - in augmented reality.”
The event would feature two volumetric stages set in the tailgating fan area. Fans would be able to create a 3D hologram of their end zone dance that is both a memorable and shareable experience.
JAUNT XR CAST
DESIGN GOALS
The overarching message from the partnership with AT&T was to increase awareness of what types of experiences are possible with 5G. To amplify this message it was clear that providing a social sharing aspect was a key goal. At it’s core, XR Cast is designed to capture human performance in a unique way and would also be an important focus in the design. Lastly, delivering an AR experience today requires solving many technical gaps that still exist between the technology and consumers.
JAUNT XR CAST
CORE FEATURES
To make this product launch event successful for everyone it was absolutely necessary to create an end-to-end system that was focused on reliability, simplicity and performance in a unique network condition. A core feature of the XR Cast product design is to be simple, flexible and reliable at scale. A key metric for the success of this demonstration was to provide social sharing by the consumer.
JAUNT XR CAST
SYSTEM ARCHITECTURE
The system architecture provides several modular components that can be customized and managed for multiple use cases from the XR Platform. Event managers can configure an event or distribution channel and the system will essentially create a closed loop that integrates all the individual components.These components are managed by a data architecture that manages 5 main aspects: User ID, Capture ID, Marker ID, Event ID (distribution channel) and Share ID (social).
JAUNT XR CAST
USER JOURNEY
This user journey maps out the entire experience from end-to-end. Storyboarding helps communicate and illustrate who each of our users are and how their goals would need to be met. The experience itself will be further defined through user personas and journey maps and initial user flows for each of the components in the design: user sign-up, capture operator app, AR kiosk demo experience and the sharing application (to deliver users’ captures to their own devices for social sharing after the event).
JAUNT XR CAST
USER PERSONAS
Throughout the user journey several user types are identified such as Brand Ambassadors, Capture Stage Operator and the Participants who attends the event. Each user type is assigned a specific role with specific goals they will want to achieve.
JAUNT XR CAST
JOURNEY MAPS
Journey maps can provide a deeper understanding of the user experience, training or comfortability with accomplishing their goals. Pain and touch points are identified that should be taken into consideration in the design.
JAUNT XR CAST
WHY AN AR MARKER?
Why an AR marker? Delivering AR experiences to consumer devices presents a few problems that need to be overcome. Many user devices do not support AR and their computer vision frameworks yet. Delivering a specific AR experience to a dedicated app requires additional steps the user must take to locate (on a store), download and install.
For volumetric captures of humans, each marker is unique and is tied to a specific user ID and their capture ID.
Many users find that moving, sliding and rotating the physical marker to be intuitive and provides a unique ability to interact with their capture in their own physical space. All early indications during testing and demo events showed users playfully interacting in fun ways such as putting multiple captures together to dance with each other or holding the markers up to faces in playful and completely intuitive ways.
JAUNT XR CAST
AR MARKER SYSTEM
The AR marker, or to be specific, the WebAR marker system can be supported by most current user devices and works with the widest browsers and platforms. Delivering AR experiences (to a first-time AR user) in a browser provides many advantages over dedicated apps constrained to only the latest consumer devices (currently). It is dead simple- click on URL, place the marker in camera view, view in AR. No flags, no downloads, few blacklisted devices. This solution helps overcome many technology gaps in a way that is quite simple for the user.
JAUNT XR CAST
AR DEMO KIOSK
The WebAR marker system can support the widest range of most current user devices, browsers and platforms. Delivering the AR experience at the demo kiosk area utilizes a unique AR table surface design that provides a micro-texture for surface tracking stability as well as a number of detectable “glyphs” to keep the entire AR scene calibrated. When AR markers are placed on the surface and recognized by the camera the AR Demo Kiosk App spawns and tracks a specific capture to each unique marker recognized.
JAUNT XR CAST
MULTI CAPTURE PLAYBACK
The provided tablet devices utilize the Demo Kiosk App and allow the user to view animated scene objects such as a floating 3D blimp and miniature capture stages at each “Live” end zone. Simultaneous playback occurs of participant captures for each of the unique AR Markers in view. Pre-placed captures are also viewable and are calibrated to the scene utilizing “glyph” detection. These captures feature the Atlanta Falcon cheerleaders, mascot and football stars. For the user they simply need to pick up a device and begin viewing the entire AR scene and interacting with the captures.
JAUNT XR CAST
DAY OF EVENT OVERVIEW
Watch our users’ journey through the entire live event- from sign-up, capture and then moments later seeing their own 3D Holograms for the first time!
JAUNT XR CAST
SOCIAL SHARING APP
Post event, the ability to deliver each participant’s own 3D captured performance to experience on their own devices was the cornerstone to creating a shareable experience. The social sharing app design was focused on reducing as many user pain points as possible including eliminating the need for an app download and considerations for device support and first time user expectations. By creating an app experience within the browser a user simply receives a notification in text/email to their user ID provided at the day of the event. The notification provides a simple clickable URL unique to their performance capture.
The web app design provides for viewing the capture in a WebGL and WebAR environment to further increase support for older user devices and browsers. Additionally the user has the ability to change themes for the 3D environment their capture appears in such as an Atlanta Falcon theme or a Dallas Cowboy theme. The three main goals of the browser app is to view, customize, create a shareable asset from their experience and then share on social channels. Although AR viewing mode is secondary to maximizing the number of users who can access and view a 3D capture. AR viewing mode would also require a marker to allow for tracking the capture in world space. A lost marker flow to allow the user to save and print a marker image was an additional consideration in the design.
VIEW FACEBOOK DEEP LINKING TEST PROTOTYPE
JAUNT XR CAST
SHARING APP FLOWS
Early wireframes and flows for the “sharing” app begin to define and shape the features that would allow users to reach specific goals. Here I have begun to define features such as viewing the user’s captures in 2 modes -WebGL (supported by most devices and browsers) and WebAR. Additional features: customizable themes and beginning to define the social sharing flow.
“PLAY” CLICK THROUGH WIREFRAME PROTOTYPE BELOW
JAUNT XR CAST
MAXIMIZE SCREEN AREA
Through wireframe prototyping and testing, design decisions were made that focus on simplifying the user interactions and creating the shareable asset (snapshot) as well as maximizing the usable screen real estate for viewing and interacting in WebGL and WebAR. Snapshots are created for a specific frame of the capture playback and are generated on the user’s device based upon the supported format by social channel.
JAUNT XR CAST
SHARING APP IN BROWSER
The sharing app allows for a simple experience in browser for both mobile and desktop interactions (such as rotation and scale) and include generous tap/click targets. The accordion UI “shelf” is optimized to always maximize the usable screen area not already utilized by lower navigation menus or header bars.
VIEW SOCIAL SHARING APP IN BROWSER
TEAM CONTRIBUTORS
Additional design contributions by Alejandro Delgado.
Unity & technical prototyping by Gregory Lutter.
Front end development & testing by Neil Sotirakopoulos, Sarah Parks.