Meta
Visual & interaction design for Horizon OS
From Feb 2024 to May 2026 I had the pleasure to work on the Input & Interaction team at Meta, a stellar team of multi-disciplinary designers, engineers and researchers focused on path finding the next generation of human-computer interaction for spatial computing and immersive technologies.
During this I helped design, prototype and ship features for Meta Horizon OS, ISDK developer tools and future unannounced devices.
The challenge
Designing a VR operating system
The core challenge behind most of my work was evolving the Quest VR OS from a single app, gaming OS like experience to a more open-ended general purpose OS, where multiple interactable panels can be brought up and arranged in your environment.
Adding this capability exponentially increases the design complexity. For example, once panels become free-floating, you need to think about how to let users manipulate them in space (move, rotate, size), if/how they interact with each other (occlusion, collision..etc), keeping them usable (if it goes too far away and becomes too small to see / interact with)…etc.
Most of my work contributed to addressing these design challenges in the context of using hands, gaze and voice as input devices.
My role
Most of my work contributed to helping design, prototype and ship many different interactions and UI across Meta’s Horizon OS for their VR headsets as well as future devices.
I specialized in developing complex shaders and visual feedback systems for the interactions we worked on, lending my eye, graphics programming knowledge and extensive experience creating real-time visual effects.
- prototyped interactions and visual feedback optimized for gaze, hands, voice + AI that turned into shipping features for the OS
- prototyped systems related to more intuitively managing multiple interactable panels in VR
- part of strikes teams responsible for adding crucial visual polish and fidelity to shipping features of the OS
- created a visual design language for UI in AR smart glasses
- developed Unity packages to enable more efficient prototyping and shader development
Input types
On the Input & Interaction team, we specialized in designing comfortable and intuitive interactions all spatial computing input types including
- hands
- eyes
- controllers
- voice
- peripherals like physical keyboards or gamepads.
circle on the left is eye direction, grid on the right is position from a ray intersecting a plane
Gaze input
Eye tracking captures eye movements and translates them into a virtual ray cast from the user’s eyes to target objects with.
Gaze is a very tricky input to design for. Your eyes move very fast and constantly make tiny, involuntary movements called tremors, drifts, and microsaccades. Not to mention our optical system has evolved to react fast to any noticeable visual cues in our environment, such as areas of high contrast or sudden and fast movement. There is a lot to design around, and filtering the noisey eye movement signal is a must.
Hand input
With hands you have a hand ray but you also have a concept of direct touch, i.e. a position where your fingers are making contact with some object. Both require different approaches to designing thoughtful interactions and visual feedback. Hand input signal is more stable than eyes, but still requires filtering to feel smooth.
Projects
As of today most of what I worked on is still unreleased, so I’ll give here a general overview of the methods and aesthetics I approached my work with and an overview of a feature that is public, Magnetism
