Apple Vision Pro: A Glorified Display Device?
I just bought my Apple Vision Pro from the local Apple Store in Singapore. Let’s’ just call it AVP in short. I have previously written some articles about my thoughts about what a special harbinger it is, and here I like to share my impressions of the device thus far.
Now, AVP is an instance of a new device class. It can be a display monitor, a portable computer, an augmented reality headset, a media entertainment device, and a spatial computer. A new device class typically replaces some other classes of devices and adds new functionality. So, let’s see how AVP does for me.
For the benefit of those who have not experienced the use of AVP, let me provide a brief description of the experience of using AVP. I am a frequent swimmer of 50-meter laps. So the AVP feels like a swimming goggle to me, where you can view the exterior through the goggles. It’s also like cupping both your hands as if you are holding a pair of binoculars, and looking through the space cupped by your fingers. When you wear the AVP, your face will be covered by the device, and you’ll look through the little lens placed just in front of both your eyes. But your experience would be like looking through the swimming goggles. I think I would call the AVP a pair of smart goggles, rather than its original name. Probably the goggles word is too close to Google and so Apple prefers something else.
Now, you might think that with AVP, you can see everything sharply. Nope, you can only view objects at a distance of about 1— 3 meters from your face (eyes) sharply. Objects that are further are blurred. This feels a little like the depth perception feature in camera lenses; varying apertures of the lens offer different ranges of distances where your subjects are in focus. A pinhole camera lens lets you see “infinite” depth of field but is heavily dependent on abundant light to be available for all your subjects. I believe Apple has chosen a suitable aperture with sufficient light in the typical use cases to reduce the unnecessary processing of the AVP graphics system and still offer a meaningful depth-of-field viewing experience within the AVP. But if you are using the AVP Environment (immersive background) feature, you can view the distant background similar to reality (which under normal circumstances, you cannot really focus sharply on them).
One of the ways to judge the value of AVP is by the combined value of the devices it replaces or by the value of new capabilities it enables. So, let’s talk about the most obvious device type it replaces: display screen.
First impression of AVP is that it is a very nice display device. Big and beautiful, even better than my 32-inch Qd-OLED display from Alienware. For comparison, let’s discuss how it compares with the displays of the devices I have: mobile phone (iPhone Pro/Max size), tablet (iPad Pro 12.9 inches), laptop (MacBook Pro or Dell laptop 15 and 13 inches), OLED monitor (32 inches), and OLED TV (55 inches). Note that we need to compare the display based on the relative size seen by your eyes and the distance of the display from the eyes. Conventionally, a display that is further away needs more brightness to offer the same look in your eyes.
Compared AVP to a mobile phone, the screen of a mobile phone is small but it can be brought very close to your face (one foot or less). It might be uncomfortable for some, but with that screen close to your face, it looks big within the space your eyes can see and appears as large as a 55-inch TV placed about 3 meters in front. With AVP, I can place a “window” screen at about 3 meters in front and make it as large as 4 sets of 55-inch TVs (assembled as a landscape rectangle similar in size to a 110-inch TV). Now if you compare it with an iPad of 12.9 inches at about 1 foot from your face, it then appears as big as that AVP “window”. So what would you choose? A 55-inch TV at 3-meter distance, or an iPhone Pro at 1-foot distance, or an iPad Pro at 1-foot distance, or in AVP a 110-inch Screen at a virtual 3-meter distance? For me, I’ll pick the AVP. I have watched YouTube videos, Disney Plus, Apple TV+, and tennis matches in AVP, and the big screen is simply amazing. Here is a sample image from Apple’s immersive video. Imagine looking at this in a full-size display in your eyes.
Compared the AVP with the screen of a laptop, whether it’s 15 or 13 inches, the laptop display at a distance of 2 feet is fairly small. When connected to an external screen (32 inches at a distance of 1 meter), it will be comparable to a 55-inch TV at a distance of 3 meters. When using the AVP external Mac display support, you can have a really large virtual screen in AVP. I have compared viewing my Mac Studio via AVP external display and the 32 inch Qd-OLED Alienware Display. Although the Alienware Diplay has more brilliant colors, the AVP external display for my Mac Studio looks just as comfortable (and can be sized even bigger). And in the next release of Vision OS, the screen can be made even wider. Connecting a PC or laptop to an external display also involves connecting a keyboard and mouse, speakers and headphones, web cam and RGB lights (if you are young). With the AVP external Mac display, as long as I am sitting at the same place where my external display is, it would be a very comfortable working desktop solution. If I start to use AVP external Mac display with a Mac laptop at a different location, then the keyboard and mouse would need to be connected to the laptop (via Bluetooth).
Now, how about watching videos? We can watch videos from Youtube, Netflix, Disney Plus, Apple TV+ and etc., in casual window style (video runs in a window up to a quarter of the size of your display) or in fullscreen mode (video runs completely in the entire screen). With AVP, as the screen size can be set to the maximum (equivalent to a 110-inch TV at a distance of 3 meters), watching video with AVP has no competition. Furthermore, AVP provides an “Environments” feature to cover the entire display field with a 3D immersive background image to provide a more cinematic experience (like in a movie theatre ). The viewing experience is similar to viewing in a physical theatre with 60–65 feet screens. You may also get some movies in wide-screen display formats that bring a lot of nostalgia to those who have been to such cinemas in the past (many of these are unfortunately disappearing due to the slowdown in the cinema business).
Now, let’s compare the AVP use case as a portable computer. There are two device classes that are considered portable computers: touch devices like iPad and laptop computers (Windows and Mac). In terms of pure computing performance, AVP would be comparable to a middle-class laptop or a high-end iPad. Currently, many iPad applications can simply run unchanged in AVP. Though they are not optimized to use some of the native features of AVP, the 2D types of applications can be readily supported. AVP also has bluetooth support, so game controllers, keyboards, mice and headphones are supported. In terms of portability, AVP is fairly bulky compared with the laptop and iPad, but they are comparable in weight. In any indoor environment, AVP would be a great portable computer for casual use (no high CPU/GPU load, no high-quality imaging or video processing, no high storage processing, etc.).
Next, let’s look at AVP as an Augmented Reality Headset. Before the launch of AVP, many VR headsets also claimed augmented reality as one of the new use cases. Augmented Reality (AR) is often brought together with Internet of Things (IoT), where all the objects in a physical setting can be connected to some kind of network (such as UWB, bluetooth, Wifi, 4G/5G, etc.) so that information can be shared. The AR Headset is usually a special pair of glasses that is capable of layering information onto the IoT objects that are recognized through the glasses (which have cameras). As it is now, AVP cannot let you look at an object in your vicinity, and somehow fetch information about it. AVP requires the use of a custom application that is then programmed to use the onboard cameras to scan for physical objects that it can recognize. Then it can show the user interface for any information associated with the physical object and even let the AVP user interact with the user interface.
Another common use case is the viewing of digital twins. A physical object is digitally encoded and then a VR headset or display device can then be used to view the digital model possibly with live information that is updated to the digital model in real time. Now, this typically requires extensive 3D data models with surface texture and other graphical capabilities. The interesting thing is the format used for these models is often USD (universal scene description language), which is the underlying graphical standard used in the AVP graphics system. That is probably why many enterprises are starting to look at using AVP for many industrial use cases as the ease of building applications with USD support is readily available in AVP (through the use of ARKit programming with USD models and Reality Composer Pro for manipulating USD models, and the integration with other Apple API frameworks). Incidentally, USD was created by Pixar, a company also owned by Steve Jobs. USD is built with 3D realistic animation in mind but has now been adopted by major companies like Nvidia and Adobe for more professional and industrial applications.
What about Spatial Computing? Apple defines it as a technology defined by computers blending data from the world around them in a natural way. Today with our current computing, we are essentially interacting with a 2D screen with audio, video and tactile sensing, all in 1D or 2D. With Spatial Computing, we need to see in 3D, hear in 3D and feel in 3D. AVP is probably the only computer that is capable of supporting these 3D features. The AVP ability to enable us to see in 3D is enabled by the tiny micro-OLED display and the use of graphics capability like the USD. AVP can allow us to hear in 3D by the use of Spatial Audio and Dolby Atmos sound processing, and AVP can let us interact in 3D via the use of our fingers, eye focusing and tracking, and the geospatial tracking of the AVP itself by the multiple cameras and sensors in AVP. No other device has this level of support for 3D capabilities. Our spatial computing needs are often confused with what we are getting from modern gaming and VR headsets. They are all using 2D interaction with some form of simulation for 3D effect (eg. the use of controllers to move in a virtual 3D environment). Even the surround sound effect in 3D gaming is a simulated effect of rear projection sounds rather than the capture of a true directional sound. Modern sound recording capabilities can truly capture the stereoscopic positions of the various instruments used in playing the music or song.
Now, let’s look at how AVP stacks up based on the type of use cases I have listed at the beginning of this article. As a display monitor, AVP offers the largest screen and the most comfortable viewing experience (let’s not compare the weight of the headset versus the portability of other display devices; you cannot bring the 32-inch monitor or the 55-inch OLED TV anywhere you go). As a portal computer, it is almost as versatile as an iPad or laptop; it only lacks a portable keyboard which can be easily solved or eventually solved by Apple Intelligence (why type when you can speak). As an augmented reality headset, its support for USD format and its rich API libraries make it the best platform to port AR and Industrial 3D applications to run on it. As a media entertainment device, its immersive viewing experience is second to none. As a spatial computer, there is no competitor. For me, AVP is clearly the best overall device to aggregate all the use cases.
For most people, AVP would seem to be obscenely expensive to acquire. That is because they have not considered it as a device to replace all their other existing devices. With AVP, I don't need an external monitor, I don't need an Apple TV, I don't need a 55-inch TV, I don't need external speaker systems (eg. stereo pair of home pods), I don't need a laptop computer, I don't need a special computer table and chair (I still need a comfortable seat somewhere to use AVP, but that can be anywhere). If you add up all the cost of these devices and estimate the other value of AVP for AR and Spatial Computing, you would probably conclude that AVP is a huge cost consolidator.
Now, there are other further benefits to the use of AVP. Conventionally, if you are using a PC or laptop, you tend to sit with your back angled forward a little. This could cause neck fatigue or back injury, which may affect your health and productivity. With AVP, you do not need a table and you do not need a room big enough for a large TV, so you will save some real estate money. For senior folks with the need for reading glass, AVP can allow you to read any doc at any distance at any size. So, you do not need a reading glass.
Of course, there are some downsides to AVP. You need stronger cheek muscles to support its weight but its physical form factor will improve with future generations. If you need the optical inserts for AVP, once you remove AVP, you need your glasses to see. That may be an inconvenience if you cannot carry the glasses with you all the time. And if you try to eat something with the AVP on you, you are likely to miss placing food into your mouth accurately. AVP is also currently not meant for outdoor activities, so you cannot wear it anytime anywhere. As a consumer or generic user device, the AVP is still rather new and there will probably be new capabilities added by Apple in the future. Something like a 3D geo-spatial bar code like what we currently have in 2D bar codes would be great (for example, if you see a supported geo-spatial bar code, you may be able to double-blink your eye and activate it, which then launches an application to “process” it with its location information). Maybe Apple may offer a new Mac Mobile (aka Mac Mini with battery), a battery-powered Mac sized like a brick without a screen and keyboard, but uses the AVP for its display (and a Pro version with more powerful CPUs/GPUs for Pro users). I think the future with AVP is wonderful. Hope you enjoy this article.
Here are links to my previous articles on AVP if you like to read them:
https://medium.com/@jangthye/apple-vision-pro-and-computational-reality-cd631a4c9b14
