Video Quality Assessment

Andreas Rossholm, Ph.D and consultant at Eyevinn Technology

Subjective Quality Assessment

The legitimate judges of visual quality are humans as end users, and the opinions of which can be obtained by subjective experiments. Subjective experiments involve a panel of participants which are usually non-experts, also referred to as test subjects, to assess the perceptual quality of given test material such as a sequence of videos. Subjective experiments are typically conducted in a controlled laboratory environment, even if crowdsourcing based quality assessment bears promising values of correlation with laboratory based testing.

Objective Quality Assessment

Due to the time-consuming nature of executing subjective experiments, large efforts have been made to develop objective quality metrics, alternatively called objective quality methods. The purpose of such objective quality methods is to automatically predict MOS with high accuracy. Objective quality methods may be classified into psychophysical and engineering approaches [4].

  • FR methods: With this approach, the entire original video is available as a reference. Accordingly, FR methods are based on comparing a distorted video with the original video.
  • RR methods: In this case, it is not required to give access to the original video but only to provide representative features of the characteristics of the original video. The comparison of the reduced information from the original video with the corresponding information from the distorted video provides the input for RR methods.
  • NR methods: This class of objective quality methods does not require access to the original video but searches for artifacts with respect to the pixel domain of a video, utilizes information embedded in the bitstream of the related video format, or performs quality assessment as a hybrid of pixel-based and bitstream-based approaches.
Figure 1: Objective assessment, FR, RR, NR

Pixelbased metrics

The most used engineering metric is the pixelbased metrics where Peak Signal to Noise Ratio (PSNR) is the most widely used measure. PSNR is full reference method, easily calculated and gives a measure of the fidelity on a logarithmic scale between an original and degraded frame. And even if PSNRs limitation is well documented when it comes to reflect the human quality perception adequately in specific situation it has its benefits. In many situations it can also be an advantage to look at PSNR-Y since the chroma components is typically subsamples in 4:2:0 color formats. Further, to adapt against human perception a modification of PSNR where properties of human perceptions was added was proposed in PSNR-HVS [5] and PSNR-HVS-M [6]. For PSNR-HVS this is achieved by determining the PSNR of a DCT version of the frame weighted by a contrast sensitivity function and for PSNR-HVS-M this was extended this by including an additional masking model.

Other objective quality metrics

One psychophysical metrics that has been used both in industry and in academy is Opticom’s Perceptual Evaluation of Video Quality (PEVQ), standardized in ITU-T J.247 Annex B [9]. PEVQ is a full reference metric that after temporal and spatial alignment uses distortion classification of measures of the perceptual differences in the luminance and chrominance domains between corresponding frame. Together with temporal information this is then aggregated, forming the final result. An updated version for HD, PEVQ-HD, has also been presented and was evaluated by the VQEG HD project in 2010. In 2011 was Swissqual’s VQuad-HD standardized in ITU-T J.341 [10]. VQuad-HD is also a FR psychophysical metric targeting HD content. The model is designed based on aligning the reference and degraded signals after pre-processing including noise removal and sub-sampling, after this are the spatial and temporal perceptual degradation evaluated and a score predicted.

Content Independent Metrics

As was described in the previous blog QoS can be seen as a contributor to QoE. This can be used as a piori information for using content independent metrics to extract information about the quality of a streaming session, also known as Parametric Models. This is especially useful for service providers in a pint-to-multipoint access network where information of ever frame for every user is required becomes impactable. By knowing the codec and the designed behavior content independent metrics, for example latency, packet loss, frame size, and frame type, can reveal QoE related information.

References

[1] Methodology for the Subjective Assessment of the Quality for Television Pictures. Standard ITU-R BT.500, revision 13. ITU-R, January 2012.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Eyevinn Technology

Eyevinn Technology

We are consultants sharing the passion for the technology for a media consumer of the future.