Spoken Shortcuts
A work sample for rethinking how 3D creation in VR can be streamlined. Spoken Shortcuts addresses the issue of quickly and efficiently accessing tools when keyboard shortcuts are not a thing. This project includes a working prototype that can be downloaded and used on any Quest device.
ROLE
Designer, Developer
Project
Self Initiated
Tools
Unity, Figma, Photoshop
Date
Summer 2024
Creating in XR
Designing in 3D for 3D is a pivotal shift in how creators bring ideas to life, enabling a more direct and immersive connection between the designer and the end product. Unlike traditional 2D workflows, 3D design aligns with the spatial and dimensional context of the final output, making it especially impactful for industries ranging from architecture to entertainment.
interaction design
Beyond 2D
"Our job is to figure out what they want before they do” -Steve Jobs
I thought about this quote a lot as I worked on this project because VR is still a new medium to create for and while it's useful to look at what worked for 2D, it's important to step out of that comfort zone and try new things :)
We know how to interact with a computer. The screen gives us visual information, to move around we use a screen and for more detail input, a keyboard. However, how does this work when you're not on a computer but a VR headset?
Persona
Pain Points
Cleo is a Outdoor Gear Product Designer developing a new type of tent. She knows her preferred work style in 3D design programs but she comes across some pain points:
Cleo’s design flow is interrupted by switching tools and navigating menus, there is no easy way to switch.
Expanding tool windows saves time but clutters her workspace.
To Cleo, constantly stopping to switch tools feels like hitting a speed bump, she loses her momentum.
Challenge
The Problem
How might we integrate a new form of interaction to facilitate the creation process in VR and not break creators out of their creative flow state?
The solution
Spoken Shortcuts
Similar to how Photoshop and Microsoft Word have keyboard shortcuts for quick access of tools; for the 3D programs I present: Spoken Shortcuts.
Cleo can now use her voice to switch tools, adjust settings, or apply actions.
“Switch color to green”
“Turn reference layer off”
“Switch to sketch pen"
Spoken Shortcuts enhances workflow without interrupting the creative process or cluttering the user’s space
Implementation
Activation
Although there are various ways to activate voice commands (e.g. "Hey Siri"), the best approach for this use case is a push-to-talk button.
The button, along with audio and haptic feedback, provides clear and intentional feedback, ensuring users know their voice command will be heard without ambiguity
Voice Commands
Direct Commands
One of the most important aspects of spoken shortcuts ar the voice commands and how they work with the existing app functionality. Spoken Shortcuts isn't meant to replace any tool but enable the user to more easily access them.
Dynamic Adjustments
In addition to enabling users to more easily access the tools, Spoken Shortcuts can also expand their functionality by dynamically adjusting the tool being used.
Instead of going to the color setting UI (seen above), you can simply say, "Make color 20% darker", to make the adjustments desired.
Built-in Constants
You know when you get a good idea and you need to note it down before you forget it. You don't care so much about the specific shades of a color but that it just is that color. Many programming languages have named colors that the user can call without knowing the specific hexacode.
"Change color to purple"
A similar approach can be taken with Spoken Shortcuts to facilitate quick sketching without worrying about predefining the colors you want beforehand.
limitations
Different Accents and Languages
While using voice to control tools is an incredibly powerful tool, there are some limitations to consider. Such as people who speak different languages or have accents.
For the prototype I created I used Wit.ai to handle the voice commands through using Natural Language Processing (NLP). Therefore, the solution to these limitations is mapping phrases in different languages or with various accents to the same intent.
In this way, the we further train the NLP model to understand the user's way of speaking and create a more seamless experience.
benefits
New Tech
Spoken Shortcuts establish a more intuitive way of creating by streamlining the design process, allowing more time for creativity and less time spent searching for the right tool. It reduces the number of screens needed to adjust tools and opens the door to more AI usage (see Road Map).
Additionally, with the rise of non-controller-based tech like the Apple Vision Pro or Logitech MX Inc, alternative inputs beyond limited hand gestures are needed to help designer streamline their design process.
Future outlook
Road Map
The intent behind Spoken Shortcuts is not to just add voice commands to existing tools in VR design apps but to allow creators to create in a more natural and intuitive manner, using their voice.
It starts with voice commands through Spoken Shortcuts but it could grow to know the user and become a AI Co-creator that provides 3D models based on what you describe.
Hover over the milestones
Spoken Shortcuts
Crawl
Voice commands for existing tools
AI Assistant
Walk
Expand voice commands and learn user's mannerisms
Co-Creator
Run
Co-create models with AI based on user’s descriptions
Conclusion
Takeaways
As we move forward and no longer do our work on 2D screens, it's important to think about how our interaction design should change as well. Being in VR already feels so real and physical that it makes sense you could use your voice as well.
There is still hesitance towards using voice commands in public but many apps are already integrating it into their functionality, eventually it will become a normal interaction.
It was a lot of fun to work on this project because I was able to design and implement my ideas to prototype completion. I learned a lot about NLPs and how best practices for Voice Commands. If I had more time, I'd start exploring how to have the NLPs learn individual user's ways of speaking and adjust to their style.
If you want to try out Spoken Shortcuts, download the APK for any quest device below!