07 Aug A route towards Highly-Realistic Displays
The key to realistic 3D display technology is answering one, seemingly simple, question:
What makes the real world so real?
This question has been haunting me for a few years now.
A stepping stone towards answering it can be found in the recent SIGGRAPH publication that I supervised and contributed to. This work, “Enabling Reflective and Refractive Depth Representation in Computer-generated Holography”, shows how holography can implement advanced effects like reflection and refraction. ACM SIGGRAPH appreciated our research by awarding our first author Aaron Demolder first place in the Graduate Student Research Competition.
In this post, I will give you an overview and a broader picture of the significance of this work.
Let’s start with an experiment and let me show you what I can see in front of me right now (in case you’ve been wondering, yes, I did clean my desk just to take this picture 😀 )
There are some monitors, a glass of water, camera, and a tablet.
How many of these objects are either reflecting the light or transmitting a portion of what’s behind them?
Reflective objects I have in front of me are:
- Desk
- Glass of water
- Tablet
And ones that refract light are:
- Water
- Camera lens
Have a go yourself and do this simple exercise:
How many reflective/refractive surfaces do you have in your surrounding?
If you wanted to describe this as a scene in Computer Graphics, you could specify your scene and pass it through a rasterizer or a raytracer or (such as Unity or Redshift). What this would produce is the 2D flat image that can be displayed on conventional monitors. Some effects like depth of field are possible to produce, but nonetheless, the output of the renderer is always a flat image. In some cases, you could extract the depth buffer (corresponding to the distance from the object to the camera).
3D Nature of Holography
Holography, unlike any other mainstream display technology, naturally offers a three-dimensional viewing of the images.
The problem
To then “add” the third dimension to our holographic image source, we often use depth data of the first surface encountered by our ray.
I hope you begin to see the problem already: we have no means of easily dealing with objects which focus their image in places other than their surface. The list includes anything reflective – like a mirror or a shiny plastic/metal or refractive – like a lens or a glass of water that you look through.
Solution: Holographic Compositing
As it happens, there is one very neat trick that allows us to work around this: use multiple render passess of our raytracer and separately handle effects like reflection and refraction. Modern renderers support this functionality out-the-box via the process called compositing. In the paper, we propose “Holographic compositing” which only requires colour and depth data of each render pass and combines the hologram in the Fourier Domain to achieve truly impressive effects.
Well, don’t take my word for it …
… see it for yourself!
In Summary
1. Next-gen displays must be to the real world what monitors with CGI content are to photo prints today
Display media (TVs, Monitors, Projectors) combined with Computer Generated Imagery already achieve a sufficient level of photorealism i.e we are unable to distinguish whether what we see is a photograph, or something that an artist designed and rendered digitally.
Whatever the next-generation display technology is going to be, it ought to reproduce all the effects present in the real world to be viable!
2. It’s hard to imagine a world without reflective and refractive materials
It really doesn’t matter where you are. If you look around your environment, chances are that you will find a few reflective/refractive surfaces.
These are not only perfectly polished mirrors whose only job is to reflect light, but materials which are far more common and reflection is not their primary function. For instance plastic enclosures which are not quite matte, jewellery, any electronic screens. Actually, that’s where the show Black Mirror got its name from:
When a screen is off it looks like a black mirror. Any TV, any LCD, any iPhone, any iPad, if you just stare at it, it looks like a black mirror.
Source: Creator Charlie Brooker Explains… | Black Mirror: White Christmas
Refractive materials are equally common – water, transparent plastic, windows. Some even have both effects at the same time, like a glass of water, a lens of a camera, or a magnifying glass.
Whether we want it or not, displays will never be life-like, unless we, tech folks, inventors, successfully learn to reproduce every single tiny detail, a composition of which makes our world so very real!
Our SIGGRAPH paper is a stepping stone on that journey, and I’m sure there will be plenty more works exploring this topic.
Finally, if this post leaves you hungry for more, a two-page poster abstract can be found here (free download for non-commercial use).