Spreadsheet "breaks" Apple Vision Pro eye-tracking

Introduction

Today’s article is just some early findings on the Apple Vision Pro (AVP). I’m working on many things related to the AVP, and it will take me a while to prepare all of them for publishing. Among the things I am doing, I am trying to capture “through the optics” pictures of the AVP, and it is unveiling both interesting information on how the AVP works and the second test pattern I tried “broke” the foveated rending of the AVP.

Having done several searches, I have not seen any “through-the-optics” pictures of the AVP yet. They may be out there, but I haven’t found them. So, I thought I would put up a couple of my test pictures to be (hopefully) the first to publish a picture through the AVP’s optics.

Eye Tracking Display Based Rendering, Mabye “Too smart for its own good”

The AVP is sometimes “too smart for its own good,” resulting in bad visual artifacts. In addition to being used for selection, the AVP’s eye-tracking varies the resolution and corrects color issues (aberrations) in the optics by pre-processing the image. This makes it tricky to photograph because the camera lens looks different to the human eye.

Today, I threw together some spreadsheets to check my ability to take pictures through the AVP optics. I started with two large Excel spreadsheets displayed using the AVP’s native Excel App. One spreadsheet used black text on a white background, which looked like the AVP was making the text and lines look “bolder/thicker” than they should look, but it didn’t act that crazy; the AVP seems to be “enhancing” (not always what you want) the spreadsheet’s readability.

But then I tried inverting everything with white text and lines on a black background, and the display started scintillating in a square box that followed the eye tracking. Fortunately, the AVP’s recording captured the effect in the video below.

I want to emphasize that it is not just the camera or the AVP’s video capture that shows the problem with the foveated rendering; I see it with my own eyes. I have provided the spreadsheets below so anyone with an AVP can verify my findings. I have only tested this with the Excel running on the AVP. The effect is most easily seen if you go into “View” in Excel” and make the view smaller with “-” magnifying glass 3 or 4 times to make the text and boxes smaller.

My First Through-the-Optics Picture Experiments

With its eye-tracking-based rendering, the AVP will be tricky to capture through the optics. The tracking behaves differently with different cameras and lenses. When setting up the camera, I can see the AVP changing colors, sometimes resulting in pictures that are colored differently than what my eye sees.

It seems pretty clear that the AVP is using “foveated,” variable resolution rendering even on still subjects like a spreadsheet. This re-rendering is based on the eyes and due to the change in the 3-D space locking (aka, SLAM) that caused the artifacts seen in the White text and lines on the BLACK spreadsheet.

Furthermore, the resolution of the displays is definitely lower than the eye’s resolution, as you can easily see the anti-aliasing “twisted rope” rippling effect if you look at the white-on-black spreadsheet. The highest rendered resolution (“foveated”) part of the image that scintillates. I discussed this issue in Apple Vision Pro (Part 5A) – Why Monitor Replacement is Ridiculous, Apple Vision Pro (Part 5B) – More on Monitor Replacement is Ridiculous, and Apple Vision Pro (Part 5C) – More on Monitor Replacement is Ridiculous.

I should point out that if not for the foveation, the whole image would scintillate. Still, the foveated rendering worsens because it creates a visible square at the boundary between the foveated area and the lower-resolution region. The “foveated rendering” makes it worse by changing the text and lines’ resolution and thickness. I would argue that a more graceful degradation would be to have the whole image rendered the same way (it is not a processing limitation to render a spreadsheet), with the whole image scintillating rather than having boundary lines where it does and does not and with the boldness changing at the boundaries as well. The key point is that the AVP’s display, while much better than almost all other VR/MR headsets, is not, as Apple puts it, “retinal resolution” (or beyond what the eye can see).

Anyway, for the record, below are a couple of through-the-optics test pictures. The first was taken with an R5 camera with a 28mm lens and “pixel shifting” to give a 400-megapixel image. Click on the crop of a very small portion of the center of that picture below to see it in full resolution.

Crop of a very small portion of the original image to show the full detail

The second image below was taken with an Olympus D mark III (Micro Four-Thirds camera) with a 17mm lens. It does not have the resolution of the R5, but the AVP’s eye tracking behaves better with this lens. This camera has a 24mp sensor, and then I used its pixel-shifting feature to capture the image at about 80 megapixels. The whole image (click to see at full resolution) is included below.

If you scroll around the full-resolution image, you can make out the pixel grid through most of the image, yet the text becomes blurrier much more quickly. Preliminarily, this seems to suggest foveated rendering. I have not had time to check yet, but I suspect the resolution falloff coincides with the squares in the white-on-black spreadsheet.

Very Small crop from the image above to show the detail

Conclusion

Designers have to be careful when it comes to applying technology. Sometimes, the same smarts that make one thing work will make others behave poorly.

The biggest part of the problem could be a bug in the AVP software or the Excel port. I’m not saying it is the end of the world, even if it is not improved. There is probably a way to “tone down” the foveated rending to reduce this problem, but I don’t think there is any way to eliminate it, given the display resolution. At the same time, the second test I tried caused it to “break/escape.” Since it happens so readily, this problem will likely show up elsewhere. Fundamentally, it comes down to the display not having a resolution as good as human vision.