How to Test Spatial Computing Apps

One of the primary challenges when developing Spatial Computing applications is navigating a new paradigm of user interactions and design patterns. In contrast to traditional technology mediums like mobile or web that have a very mature universal user interaction and design patterns, spatial computing offers a new canvas for user interaction with technology.

This makes testing and especially user testing such an important activity to be done during the development of applications. The following guide dives into the details of one specific type of testing: User Testing.

🧪 Different Types of Testing

There are three primary types of testing in spatial computing application development. These are:

1. Functional Testing: Functional testing is about validating there are no software defects, also known as bugs, in the software.
2. Design Review: Design review is the type of testing to ensure that the quality of the front-end implementation is aligned with what aesthetically feels good. This is a challenging type of testing as there is no universal correct answer for each front-end implementation. These items can range from, the distance interfaces are from the user, transition animation speeds, or hover feedback for interactable objects.
3. User Testing: Finally, we have user testing. This is where we understand if the users “get it”, do they understand the purpose of the app, do they understand the core mechanics, is the app giving them value, do we need to better communicate certain components, what are the main points of friction for the user.

The interesting aspect of user testing is that it does not focus on testing software itself, or even the implementation of the software. This testing is one level of abstraction higher, it’s the review of the product itself.

Here’s a step-by-step guide on how to conduct user testing. We'll use the process we did for Newton’s Room as a case study:

1. Finding Users
2. Pre-Testing Survey
3. Recording Testing Session
4. Post-Session Survey
5. Analyze, Digest, and Action.

1. Finding Users

The first step is to get other humans interested in being user testers. It’s important that the users you get are target users of the application you're building. Since were are building a consumer mixed reality education app, our target audience is lifelong learners and technology early adopters with an interest in STEM.

Asking someone to test your app is asking them to give you their valuable time and attention. What can you offer them in return? In our case, we wanted to give back more access to mixed reality content and offered testers a $50 Meta Quest Gift Card that they could use on any game or app they wanted.

By setting up an incentive structure, it’s easier to find testers in the watering holes of the internet, as the intent is an exchange of value.

2. Pre Testing Survey

To better understand the user profile and starting point, we conducted a pre-testing survey with questions designed to gauge the tester's background, experience with XR, and expectations from a mixed reality learning experience. Some of the questions we asked included:

• How many years have you been using XR?
• How would you rate your understanding of Newton's laws of motion?
• How interested are you in learning about physics?
• What are your expectations from a mixed reality learning experience?

3. Recording Testing session

The key to understanding a user test is to see the experience through their eyes. We asked users to screen record their first session with the microphone enabled and to speak their thoughts aloud while using the app.

4. Post Session Survey

After the testing session, we collected additional quantitative data to better understand the impact on the app's objectives, including changes in understanding Newton's laws of motion and engagement with the mixed reality experience. Some of the questions we asked included:

• Explain how your understanding of Newton's laws of motion has changed after using the app.
• How would you rate your understanding of Newton's laws of motion?
• How much has the app influenced your interest in learning more about physics or using mixed reality for education?
• How does this learning experience compare with traditional methods you've used in the past?

Some key insights we gathered for Newton’s Room included:

5. Analyze, Digest, Action, Repeat

With all the valuable information gathered during the user testing session, the team will review it in detail and come to their own conclusions about the feedback. It’s good practice not to use users' suggestions as direct actionable items, as the development team has much more context and a higher-level view. Focus more on identifying the pain points and areas of friction.

User testing should be done in small batches, as it allows for iterations through each batch of users. After addressing the items reviewed in following user testing sessions, check if the user stumbles upon the same challenges.

It takes around 4-5 users to cover 90% of the primary issues of a build. Use this as a benchmark to iterate as many sessions as needed to cover and address the pain points of the build.

Wrapping up

User testing is often skipped in Spatial Computing application development but is key to successfully publishing digital products in this medium. We hope this post serves as a guideline for conducting your own projects in the future.

If you're working on something interesting, feel free to reach out [link to contact].

Want to stay up to date with the latest in Spatial Computing development? Subscribe now.

Until next time,

Horacio