Magnetism
Snapping & alignment for spatial UI
Magnetism is a feature released on v2.1 of Meta Horizon OS that enables apps, windows and panels to automatically snap and align to nearby walls in your environment (VR or in real life)
I worked on a team of three for this feature. My contributions included:
- Handling all the shader development and design
- Visual & motion design iteration
- Early explorations into surface detection using Meta Scene API
- Creating the C# classes for abstracting away visual state transitions, animation & control logic
- Overseeing shader port into production code for shipping
The reasoning
Why make this?
A key feature of the OS was to allow users to persist panels anywhere in their environment, making them feel like permanent objects and allowing for more use cases to use the headset outside of just a gaming device.
Attaching objects to your walls and tables is a strong mental model for peristance in a space, allows users to better organize their content and can create moments of delight, where you see how the virtual world is aware of your physical environment.
The feature is composed of two core elements, the interactable content, i.e. the thing you are moving and anchoring to your wall, and the visual feedback of the interaction.
When peeling a panel off a wall, there’s a certain feeling of surface tension, where the panel is being unstuck from the wall, and you feel it in the interaction and see it in the deformation of the shader
Procedural UI
All visual elements were rendered procedurally using SDFs, both in the prototype in Unity and in the final GLSL shader code.
All code related to visual parameters was kept in its own class, and code related to controlling those parameters via state machines or continuious mapping of input data was designated to its own controller class
World locked grid
The coordinate space for deriving the dot grid SDF was anchored to the world in position, but stayed local to the quad mesh it rendered on when rotated. This way the dots appeared stuck on a wall or table, but the grid could be rotated in any way and the grid would still stay flush on that plane
Depth API
The Depth API was used for surface detection. Originally I had implemented the Meta Scene API, however that requires the user to setup their room first. The Depth API is not as accurate in its detection as we estimate planar surfaces based on an array of normals from shooting rays out into the depth map, but it requires no setup at all.
Masking
Using position derived from the depth map reverse projection, I set up an occlusion effect in the shader, discarding fragments whose distance from the depth position at that pixel exceeded a certain threshold, enabling the effect to not draw over surfaces like tables or walls with doorways
Challenges
- Must create visuals that can realistically be implemented in the OS within compute budget
- Surface detection is tricky
- Achieve seamless & delightful interaction with low friction, e.g., avoiding a button to enter “placement mode”
- How to best map our input data (hands, gaze, controllers) to actions like pulling a panel off a wall, or placing it?