Best Practices in Mixed Reality Design: An In-Depth Guide
Mixed reality is a new technology medium that unlocks a new generation of interactive applications. In this post, we'll explore the unique challenges of designing software for mixed reality and review our best practices in agile design methodology.
In our recent post: How to Build a Mixed Reality Prototype: Agile Software Development, we explored the process of creating a fully functional mixed reality prototype starting from the idea stage.
We will now evolve this prototype into an Alpha version of what could potentially become a final product in a later phase. When we refer to Alpha we're referring to a feature complete application with final design and functionality.
Let’s dive deep into the design process for creating a world-class mixed reality software application.
🧗 Challenges Designing for Mixed Reality
Mixed reality offers a new frontier in technology, unlocking a new medium for human-computer interaction. This medium enables the ability to blend the physical world with digital content unlocking a new range of use-cases and experiences previously only available in science fiction.
With new opportunities comes new challenges. Here are some of the main challenges we encountered in the design and implementation process of Newton’s Room Alpha build.
The main challenges are:
- 🌅 Designing for a New Medium: Navigating the challenges that come with pioneering in a new technological medium.
- 🤏 New Paradigm for User Interfaces: Designing user interfaces tailored for first-time users of this new hardware category.
- 🔍 Designing for Passthrough: Creating experiences that cohesively merge users' physical environments with digital content.
- ❓ Working in the Unknown: Managing the unpredictability and lack of control when designing an experience in an undefined physical environment.
We'll go over these main challenges and our recommended best practices to overcome the unique challenges of the industry.
1. 🌅 Designing for a New Medium
Mixed reality is a new medium that is being defined by the earlier industry developers. There are currently very few existing mixed reality digital products in contrast to mobile or web development that have decades of development and millions of apps built and iterated upon.
Our previous experience developing enterprise applications with see-through mixed reality glass technology has been key in the design process of Newton’s Room. However, mixed reality devices like Magic Leap 2 and HoloLens 2 leverage more simple user inputs like controllers or point & click interfaces and don’t have the capacity for fully immersive mixed reality experiences available in technology like Meta Quest.
Developing on Meta’s Presence Platform unlocked a new suite of mixed reality capabilities like hand interactions, scene recognition and immersive passthrough experiences. Meta has provided the developer community key knowledge resources for developing immersive mixed reality, including great documentation and a library of open-source demo projects that explore the platform’s key features.
We are still in the early stages of the industry and much research and development is needed to continue defining the best practices when it comes to designing mixed reality experiences.
We are committed to contribute to the collective knowledge of the industry through research and an open development approach to projects like Newton’s Room.
Here’s the process of our approach to the agile design phase of the project:
Explore → Prototype → Reflect → Repeat
When it comes to a new medium like mixed reality no one really knows exactly what works and what doesn’t. So we took the following agile approach.
i. 🔭 Explore
Instead of starting with defining a final design, we started with the exploration of multiple design ideas. A design exploration phase enabled the team to explore ideas, open the conversation and evaluate what direction feels best.
For example for all components of the app, we designed multiple ideas before committing to implement the prototypes of the best design ideas.
Launcher ideas
Control panel ideas
Projectiles with different mass ideas
Mirrors with different mass ideas
Exploring design ideas gives the team the raw material to discuss the ideas and make informed design decisions before committing to implementation. In this lean approach, failing fast in the exploration stage provides a higher chance of success for the designs that go into prototyping phase.
ii. 🛹 Prototype
The designs that were approved by the team in the design exploration phase moved onto the prototyping phase. Prior to doing a full implementation we implemented multiple variants or created sandbox tools that enabled the team to test how different components worked and felt.
For example we implemented multiple variants of the launcher audio source to evaluate which option worked best.
In some cases like in the environments design, we developed sandbox tools that enabled the team to play around all the possible variables to experiment what works best.
iii. 🤔 Reflect
After completing a prototype the team would reflect on the results and decide which design option we should more forward with.
In interactive software and more so in a new medium like mixed reality, being able test, feel and reflect is key for informed decision making.
iv. 🔁 Repeat
Once a component is integrated we move onto the next component.
2. 🤏 New Paradigm for User Interfaces
Historically, we've interacted with technology through 2D screens. Spatial computing, however, unveils an entirely new medium for interaction, moving beyond the established boundaries of flat displays.
While 2D interfaces have matured with universally recognized user interactions (like tapping to select or pinch-to-zoom), these principles aren't directly translatable to spatial computing.
In mixed reality, designing interfaces requires a paradigm shift. Interactions resemble our natural engagements with real-world objects. It’s fascinating how we are now actually interacting with software applications in the same ways we interact with the physical world.
When designing interfaces for Newtons Room, our guiding principles were:
- Diegetic UIs: Every digital element should be intrinsic to the environment.
- Familiar physical-world Interactions: Designing actions users innately understand.
- Real-world analogies: Leveraging pre-existing known concepts.
- Expected communication channels: Methods of communication that align with user anticipations.
With these principles in mind, We designed the following interfaces.
🔘 The Button
The inspiration: That big circular button that you are craving to press.
A big circular button is part of popular culture. Everyone knows what to do with a big circular button. You just want to press it right? So we decided to center the main interface of Newton’s Room around this concept.
This circular button is the main call-to-action in the application and enables the user to engage with the primary functions of: start, launch, next and retry.
⚪ Interactive Levers
Secondary interactions in Newton’s Room are enabled through levers controlled by white spheres. Drawing inspiration from objects like ⚾ baseballs, ⚪ golf balls or 🏐 volleyballs, we designed a grabbable white ball as the universal interface lever—invoking an instinctual urge to grasp.
✊ The Natural Grab Mechanism
In 2D, interacting with objects involves a click-hold-drag-drop action, a principle drawn from our physical-world behaviors. In spatial computing, we return back to the roots. We grab, hold, move, and release objects with our hands as we would in physical reality.
🗣️ Screenless Communication
In a world without screens, how do computers communicate with humans? Let’s take inspiration from pop culture science fiction.
We’ve centralized the communication to the user through the central motif of the application: the main button.
3. 🔍Designing for Mixed Reality Passthrough
While 3D development tools, such as Unity, offer advanced rendering capabilities for crafting interactive 3D applications, a key feature is post-processing. However, in the Presence Platform Passthrough, post-processing isn't available. The reason? Protecting user privacy. Granting developers access to the camera feed poses significant privacy risks which could easily be avoided by disabling app level development features that rely on camera feed like post processing.
Despite these limitations, the Presence Platform provides an array of tools to modify the camera feed. Coupled with particle effects, sound adjustments, and other visual effects, we managed to create highly realistic mixed reality environments.
4. ❓Working With The Unknown
A unique characteristic of mixed reality is that the user’s physical environment is the canvas of the application. This is a monumental leap forward from working with a 1920x1080 2D screen. However, it presents challenges, notably, the unpredictable nature of each user's unique environment.
Our guiding principles for designing a mixed reality experience without knowing the users environment are:
- Prioritizing mixed reality: Integrating the user's environment as an integral part of the experience and interaction.
- Ensuring universal compatibility.
- Tapping into the full spectrum of mixed reality.
Based on these guidelines, our priorities were:
- Design for the common denominator space: The physics puzzle design must fit in a minimal room space — specifically, the area between standard furniture and the ceiling. Targeting a 2x3 area ensures a balance between compactness and an engaging environment.
- Interaction with the environment: Using the Presence Platform's Scene Understanding, we designed puzzles that allowed users to leverage their surroundings as part of the solutions, providing a unique experience for each user. This additional tools provide more width and depth in problem solving as there is never only one solution to any problem.
- Providing fallback options for limit case scenarios: For users that are in an environment that does not covert the common denominator designed in the levels the app provides fallback options so the user can still engage with the experience. These include room scan being optional and repositioning feature for the mixed reality content.
⏭️ Next Steps
Newton’s Room is now an Alpha of the final product we envision publishing. Our next milestone would be Beta, which is a validated build by final user testers and much more polishing. This next milestone will focus on:
- User testing: Validate and iterate the user interface designs and user experience quality focusing on first time mixed reality users.
- Focus on education: We will be collaborating with physics educators to help craft the educational theory module and content of the app.
- App Polishing: Building upon our Alpha to expand quality of content and overall user experience to reach a public Beta build.
Stay tuned for more updates.
Until next time!
