The Challenge to Maintain and Translate Creative Visual Ideas to Everyone’s Viewing Devices
Many articles have already been posted drawing conclusions on what went wrong with the visual quality of a very popular TV show that was recently aired. In this post written by Johan Skaneby and the team at Eyevinn, we will not jump to any immediate conclusions and instead highlight and discuss the steps in the production and distribution chain where the image quality can be affected.
Recently a debate around the globe suddenly blew up as a consequence of an aired episode of a very popular TV show. The Director of Photography and the Director had made an artistic choice to shoot large parts of the episode in natural light using candles and fires in the set. This produced a claustrophobic and dramatic effect for the viewers with high performance TV sets in perfect environments, whereas the rest of the viewers complained of severe artifacts and a basically no visual experience at all due to the same artistic choice — very dark images. This highlighted the discussion of TV performance and the new standard HDR as a way to preserve the artistic intentions from glass to glass.
The era of file-based workflows has introduced a chain of digital production points where media needs to be converted and properly assigned with the correct metadata in order to present an accurate viewing experience. In this blog we are not jumping to any immediate conclusions on what went wrong with the episode in the very popular TV series. Instead we will take this opportunity to highlight and discuss a selected number of production and distribution points where the image quality can be affected.
Adding to this we can also conclude that an artistic decision that resides in the technical border line of what can possible be achieved with today’s TV technology will of course result in difficulties in many living rooms.
Grading — working with and preparing the final master.
It is a challenge to translate creative visual ideas to conform properly to all targets in a distribution chain, be it iOS, Androids, Chromecasts, Apple TVs, web browsers and IPTV boxes. They all have their different limitations based on decoders, bandwidth, screens and more.
Adding to the already complicated mix we see different emerging technologies representing the new BT 2100 HDR standard where we aim not only to widen up the color description to mimic more closely the human eye, but also the dynamic range in how we perceive our environment. The car light in the night is not only represented as a white round area on your TV screen, you should be able to feel it, like in real life.
But does it all matter? Doesn’t it all work well as it is?
Well, this might still be a common view in order to avoid making new technical decisions and with associated financial commitments. However, it becomes clear that not only artistically but also general technical exposure problems for a soccer field actually requires a better understanding of video technology and the future.
In order to make use of the much wider color and dynamic specification defined in the BT2100 specification a production need to be shot with this specification in mind already on location. The Director of Photography needs to decide how low and high exposure elements in the picture are balanced for the final viewing target. In essence, if the feature film is going to only be viewed on 8-bit SDR this will affect decisions on lighting already on the set to optimize for the SDR traditional rec.709 viewers.
Now, if we shoot for both HDR and SDR viewing it is very much up to the post production company to work according to the artistical intentions.
- The master can be graded for both HDR and SDR — this is from a quality and artistically point of view likely the best way to go, but it will of course require more grading resources
- The master can be graded for HDR — and then technically down converted by machine to SDR. This a much more efficient way, but it is not a creative process as above
- The master is graded as SDR — and then upconverted to HDR. Yes, it is true — technically an HDR master can be produced this way also, but you can’t add information if it is not there from the beginning, right?
So next time you order an HDR 10 version of your master, make sure you are aware of the procedures.
The result of this grading job will most like be one or several master delivery formats that we are all familiar with.
- ProRes
- DCP
- IMF JPEG2000
However, in a grading environment I would also like to point out that the grading artist has a set of different monitors to accurately verify how the graded image presents itself. Of course, there are professional grading monitors for both SDR and HDR work, but also consumer TV-sets can be used for both SDR and HDR to view the final consumer look as well.
In a standard scenario this works all well, but is environment of the consumer taken into consideration here? Well, the consumer TV-set will most likely be properly set up in a dark grading room, not with a faulty setup with all the factory settings in a light living room. The latter will of course make a huge difference when making an artistic choice as the episode discussed.
Conclusion
In all alternatives we will have an HDR and SDR master by technical metadata, but it is vitally important to understand the differences.
Here is also our first point of failure — the master can be created in the correct way, but also in the not so correct way given the circumstances. Expect the unexpected — make sure you make the correct decisions to ensure the highest quality here.
NOTE: I do want to emphasize that this point is where we find the people with in-depth image expertise, and because of that I would expect the least technical errors to occur here.
Preparing for delivery
The master itself now needs to be distributed for delivery. Most likely the next stop is a distribution house where transcoding and metadata management is the core service.
A few different routes for this master will most likely happen here. It is quite common that the incoming master file needs to be prepared for different locations, adding languages and subtitles, and because of this many transcoding houses creates a new high mezzanine for inhouse preparing. And with this new transcode and a new digital generation has also been created. The output from the transcoding house could be a new media file for digital distribution and / or a set of ABR files.
In above process there are a number of conversions that needs attention. The color space information of incoming master needs to be correctly translated, and of course also on the output side. RGB data can be translated to Rec 709 and back to RGB — such a case can of course severely affect the image and with this the viewing experience.
Also, in general a generic transcoding profile is most likely tuned by default to process image data based on common exposure. You might actually need to tweak settings on the profile to ensure that bitrate and image data is addressed to action scenes in the dark, that can otherwise be discharged as standard shadows of no interest. Some encoders like MainConcept and x264 offers settings to target these areas.
In the x264 we have some flags that can be of interest in the case of transcoding very dark content in this example. There are some examples of settings in the x264 that might make a difference.
The AQ — adaptive quantizier is a setting that more intelligently stores and distributes data more efficiently over temporal compressions.
A note from the https://en.wikibooks.org/wiki/MeGUI/x264_Settings#aq-mode
aq-modeAdaptive Quantization ModeDefault: 1Without AQ, x264 tends to under allocate bits to lower details sections. AQ is used to better distribute the available bits between all macroblocks in the video. This setting changes what scope AQ re-arranges bits in:
- 0: Do not use AQ at all.
- 1: Allow AQ to redistribute bits within each frame.
- 2: Allow AQ to redistribute bits across the whole video.
- 3: Auto-variance AQ with bias to dark scenes.
As someone correctly mentioned, the dark scenes of the discussed episodes required most likely less bitrate data, so adjusting settings to actually assign more data to these areas could have helped.
Other settings that might impact the result of difficult content is of course CRF, tune modes (film), psyco visual /trellis modes (and similar modes on commercial alternatives).
These are not final conclusions, just examples on how to discuss and approach the subject. But in general, some kind of analysis and detection that notifies the responsible technical staff involved is most likely very advisable here.
Transcoding of ABR files can be executed in-house at the streaming service house as well, where of course above technical observations also applies.
The end
Now, when the master has successfully passed all above production points, you might think the work is done? Unfortunately, not.
For ABR distribution, especially for AppleTV, Chromecast and similar the viewer has often no way of knowing what quality of the ABR ladder that actually enters the living room. Is it the high quality 6 mbit 1080P or is it a 2,5 mbit 1080P or maybe a 1,2 mbit 720P we are watching? The network conditions at this particular time could actually cause some different profiles to load.
Also, one of the most serious reasons for bad viewing experience is the default settings of all new delivered TV sets. Motion plus setting that interpolates 24 new frames on a 24 fps feature film, generic digital noise reductions and more. I estimate that over 90 % of all TV sets has these settings on, and the results for the final viewing experience can be critical.
Other TV settings like gamma, contrast, viewing profiles can also awaken the banding and noise monsters that will slumber in the low light areas on a correctly tuned TV set.
Summary
Unfortunately, you have spent almost 10 minutes reading a blog post without an answer on what and where something went wrong in the episode. The new HDR / BT2100 technologies will of course, when matured and properly standardized and applied, dramatically extend the viewing experience on our TV sets and devices — but it will be important, no matter if you are a Director of Photography, grading, DIT technician, post-producer, CEO / CTO to consider these expected and unexpected results in the production and distribution pipelines in order to preserve and extend the creative and artistic possibilities.
Eyevinn Technology is the leading independent consultant firm specializing in video technology and media distribution, and proud organizer of the yearly nordic conference Streaming Tech Sweden.
