Microsoft is again trying to get developers focused on augmented reality, Virtual Reality, and Mixed Reality applications. This time, the focus is on Microsoft Mesh.
There are challenges to building Mixed Reality applications that allow for simultaneous, multi-user interaction and collaboration. As Mixed Reality is used collaboratively across multiple devices and platforms, it gets harder. Actions and expressions need to be synchronized across geographically distributed sessions, and holograms have to be kept stable across devices and over time. Representing people in Mixed Reality in a way that is realistic and not goofy requires a lot of resources. Additionally, Mixed Reality tends to use high fidelity 3D models. These are not the easiest things for the average developer to create, let alone bring into a Mixed Reality environment.
The pandemic has challenged organizations to find ways to effectively work and collaborate both inside and outside of an organization. For Microsoft this has resulted in growth for the market of their Teams software. With Microsoft Mesh, Teams, and Azure they are working on the tools and platform to make collaborating in real-time in Mixed Reality possible.
Like being ‘in the same room’
Microsoft Mesh is Microsoft’s newest platform for Mixed Reality. It focuses on creating Mixed Reality solutions that bring together digital and physical worlds while adding tools and capabilities to make it easier for developers to get to working solutions. The intent is to make it easier for end users to be able to meet and collaborate using Mixed Reality – with the results at a level to where you feel like you are in the same room.
Mixed Reality is already being used in a variety of ways. This includes doing immersive meetups, training, concerts, and presentations. It also is being used to bring remote expertise onsite where an expert can see what the local attendees are seeing and offer support as if they were present onsite. Collaboration efforts, whether working on a whiteboard or doing modular designs, are also happening. Mixed Reality provides in-situ opportunities without being on site.
Building on Azure services, Microsoft Mesh allows for this sharing of augmented and Mixed Reality. More importantly, it allows for programmers to combine Mixed Reality worlds to be combined with traditional experiences. While hundreds of millions of people have already experienced Mixed Reality, with the Microsoft Mesh Platform, those people using a standard flat screen interface can see and interact with those in a pure Virtual Reality world.
The Microsoft Mesh Developer Platform
Microsoft Mesh resides on top of Microsoft Azure and taps into Microsoft Teams collaborative features and more. The Microsoft Mesh Developer Platform provides a cross-platform SDK that can target AR, VR, PCs, phones, and more. More specifically, the Microsoft Mesh Platform provides:
- Toolkits including Mesh SDKs and user experience constructs that can be used in your applications. This includes tools such as an avatar rig and customization studio. It also provides for AI controlled motion models that can connect a user’s avatar to the user’s motions.
- Native cross-platform support for Hololens, Quest, phones, PC, and the Mac.
- Capabilities for doing immersive presence, spatial mapping, holographic rendering, and multi-user synchronization.
- A platform with support for users, sessions, consumer and commercial graphics, billing, audio & video, identity management, graphics, and application infrastructure.
In the area of capabilities, the immersive presence includes the ability to create real-time, interactive solutions. This includes the use of standard avatars as well as photorealistic “holoportation” where your avatar is a photorealistic 3D representation of you. Whether avatar or photorealistic, real-time interactions with spatial placement of items is supported.
In the area of spatial placement, this can be holograms locked in place within the real world as well as locked relative to a user or other position. Sharing and persistence of holograms is also supported. This includes hologram support of real-time rendering pose updates and spatialized audio.
There is support for a range of formats and file types, including the ability to drag and drop files into your Mixed Reality development. Using cloud services, the platform is able to provide a high level of fidelity even on lower-powered machines such as phones.
Multi-user synchronization is also supported. The ability for people to interact and move virtual items is important to realism. Keeping items synchronized in virtual space for all participants requires positioning to be understood so that virtual items can be handed from one person to the next and align with each user correctly. This synchronized motion and expression of the people in the world is what delivers a sense of true engagement.
The status of Microsoft Mesh
Microsoft Mesh is currently in limited preview. For those with a Hololens, you can download a Mesh App on HoloLens today from the Microsoft Store. Mesh can also be previewed using Mesh-enabled AltspaceVR. For developers, the focus continues to be with the Mixed Reality Developer Program. For more on Microsoft Mesh, you can go to the Microsoft Mesh site.
