Passthrough camera access is causing quite a stir in the XR world these days. While we’ve got a pretty good grasp of where Meta, Apple, and Pico stand, the big question mark has been Google’s plan with Android XR. After a direct conversation with them, I can reveal that they’ll likely opt for a workaround similar to what’s on phones. Stick with me to learn more!
### The Camera Access Dilemma
Okay, let me break it down for those a bit lost here. Recent standalone VR headsets are doubling as MR devices, giving us a live RGB passthrough view of our surroundings via front-facing cameras. This is what makes those fantastic mixed reality apps like Cubism, Starship Home, and Pencil possible.
Developers, myself included, are eager to get our hands on these camera feeds. The reason? We want to deepen the immersive experience using AI and computer vision to understand and enhance the surrounding world. Camera access is key to unlocking next-level mixed reality experiences because it allows our applications to be fully aware of the user’s environment. As I’ve shared in past articles, I’ve managed to prototype an AI+MR app using crafty camera access techniques on Quest. This app assists people with home interior design, which clearly wouldn’t be feasible without access to the camera.
It sounds exciting, but there’s a catch: privacy concerns. If camera access falls into the wrong hands, there’s a risk of misuse. Malicious app developers could covertly capture users’ surroundings, possibly picking up sensitive information like IDs or bank cards left on a table, or even more personal images.
Balancing users’ privacy and the potential of mixed reality is a tricky yet crucial issue.
### How XR Companies Are Responding
Initially, XR devices allowed full camera access without much fuss. Some of you might remember my early experiments with camera textures on the Vive Focus back in 2019 – we tried everything from diminished reality to Aruco marker tracking.
But as mixed reality gained traction, companies got cold feet. Privacy concerns prompted Meta, Pico, HTC, and then Apple to restrict camera frame access.
However, as the XR community recognized the importance of this feature, developers started campaigning for these companies to reopen camera access. Influential voices, including Cix Liv, Michael Gschwandtner, and myself, pushed for a solution that’s transparent to users. We argued that with user consent, sufficient processing could be done on the XR headsets to maybe run object recognition algorithms safely. Why all this fuss over XR gadgetry when mobile devices, which practically everyone owns, grant camera access with a simple permission request?
This movement bore fruit with Meta’s pledge for a “Passthrough API” earlier this year. But where does Google and Android XR fit into this landscape?
### Treating Android XR Like Your Smartphone
Android dominates the smartphone OS market. Currently, if you’re making an app for it, you simply ask the user for camera access permission. If granted, you specify the camera ID (0 for the rear, for instance) and utilize the frame data as necessary.
Google seems to aim for similar compatibility with Android XR and plans to implement it in the same manner. Lots of rumors were swirling, so I directly communicated with a Google rep to clarify this:
“For an Android XR app, similar to any Android app, developers can access existing camera frames after obtaining user consent…”
The rep’s message highlights that you can access the world-facing stream by calling camera_id=0 and the selfie stream with camera_id=1, akin to how mobile cameras work. The standard Camera APIs, Camera2 and CameraX, will manage these processes.
This approach is fantastic news! Android XR headsets should support the same functions through the familiar Android framework, enabling media storage or machine learning analysis on device.
Yet, there’s a catch: while the front camera shows an avatar generated by internal cameras tracking head, hand, and face movements, the rear is a mirror of Apple’s tech with Vision Pro, giving the AI’s best guess of the user’s face.
Google is aiming for seamless function between Android apps and Android XR devices, which cleverly maintains consistency. I personally appreciate this approach of aligning permissions on phones with those on headsets and glasses.
At this stage, you’re likely curious about the access to raw camera data. Here’s the less favorable update:
Currently, there isn’t a provision for apps to tap into non-standard camera feeds, like the reconstructed inward-facing sensors. We might see this change in future versions, particularly for enterprise use.
For Unity developers, who often rely on native Android classes like CameraX for camera functions, the potential for leveraging these capabilities through Unity’s WebcamTexture class is promising. In trickier cases, developers might resort to JNI methods to develop a native wrapper, providing a bridge to these features within the Unity environment.
### A Cautionary Note on Android XR
Bear in mind, Android XR is still in its preview phase; no devices are officially on the market yet, so there could be changes before the final version is out. I doubt this, but it’s worth noting.
### Camera Access: A Game-Changer?
With both Google and Meta making camera access more accessible, it’s likely others will follow, marking 2025 as a transformative year for mixed reality. I’m eager to see the creative solutions developers will craft!
(Header image credits go to Samsung.)
Disclaimer: This blog hosts ads and affiliate links for financial support. Clicking an affiliate link brings me joy as I earn a small commission on purchases. For more info, check my boring disclosure here.
Share this with other innovators!
Related: [Links to related articles]
