Can Video Quality Testing be Scripted?
Abstract to Video Quality Analysis
As high-tech, video equipment becomes more complex and release cycles shorten, quality assurance (QA) & quality control (QC) testers have less time to test more features. Testers scramble to perform basic feature testing. So what happens to video quality testing? How do you know if your system is introducing artifacts and distortions?
Video Quality Testing is Important
With the advent of Digital Television (DTV), video processing has changed significantly. Almost all videos are compressed, scaled, broadcasted or IP delivered, and decompressed. This has created an opportunity for video technology manufacturers.
Moreover, the average time to market from concept to delivery for high-tech equipment has shrunk from five or six years in the ’90s to one or two years today. The first manufacturer to introduce a high quality product gains a sustainable competitive edge. As companies race to introduce more features in less time, the controlling factor for product introduction has shifted from the speed of development to the speed of validating that the system works.
To answer this demand, some testing organizations have adopted scripting in hopes of saving time. While running test scripts is faster than manual testing, how do you automate video quality analysis? Unfortunately, video quality testing has not kept up. Most companies employ testers with “golden eyes” to verify that the quality is acceptable. In addition, many talented testers prefer manual testing so that they can spend their time devising creative test cases rather than becoming proficient with complex scripting languages.
Video Clarity created a breakthrough approach to video quality testing that empowers equipment manufacturers to increase coverage and spend more time on creative testing. This technology, termed ClearView , is a scriptable, automated video quality testing system. ClearView combines an uncompressed video server, a video recorder, a video player, and a quantitative video quality scorer. By providing all of these in one unit, ClearView becomes a complete, testing system for advanced video quality scoring and comparisons.
What is ClearView ?
Video Clarity’s ClearView system offers a set of video quality analysis tools for software developers, hardware designers, QA/QC engineers, video researchers, and production/distribution facilities. ClearView plays, records, displays, and analyzes video sequences. The device can capture video content from virtually any source-file or digital or analog source such as SDI, HD-SDI, DVI, VGA, HDMI, component, composite, or S-video. Regardless of the input, ClearView can ingest and convert it to fully uncompressed 4:2:2 Y’CbCr, 4:4:4 RGB, ARGB, or RGBA. This allows CODECs to be compared and scored relative to each other.
ClearView provides many mathematical models that approximate results of subjective quality assessment. Objective mathematical methods are classified based on the availability of the original video signal, which is considered to be of high quality:
- Full Reference Methods (FR),
- Reduced Reference Methods (RR) and
- No-Reference Methods (NR).
Have you ever wanted to compare your H.265, H.264/VC-1 or MPEG-2 algorithms relative to each other? Now you can. You can even measure video delay and audio/video lip-sync.
Figure 1: ClearView GUI with Pass/Fail Results
Digital Compression Basic Issues
Video compression, like data compression, is a tradeoff between disk space, video quality and the cost of hardware required to decompress the video in a reasonable time. However, if the video is overly compressed visible artifacts can appear. It is video quality test & measurements mission to accurately model the video quality.
Before we can do any video quality measurement, we must align the video data. A few alignment issues may occur:
- The processed video is not temporally aligned with the original
- The processed video moves horizontally
- The processed video moves vertically
- The processed video loses color fidelity
- The processed video has lost Audio/Video synchronization
The objective metrics are measuring the video quality on a frame-by-frame basis, so the video sequences must be aligned in time. Since color bars/calibration stripes are not normally present, ClearView starts with a video frame and locates the frame in the other video sequence with the smallest sum-of-differences. ClearView then trims to temporally align the video sequences.
Spatial Alignment Horizontally
Interlaced video was created in 1932 to reduce flicker caused by CRTs not being able to paint an entire screen size in the refresh time. Interlaced video is still used in digital TV specifications as it reduces the signal bandwidth by a factor of two, for a given line count and refresh rate. However, only CRTs can display interlaced video. Thus, deinterlacing must occur to display an interlaced video on a DLP, LCD or Plasma TV. Depending on the deinterlacing algorithm, the processing unit may cause a horizontal shift of a few pixels. ClearView includes an algorithm to locate the frame shift and adjusts the metric window size appropriately.
Spatial Alignment Vertically
DCT-based compression algorithms – MPEG, JPEG, and VC-1 – require that the video must be divisible by 16 pixels. Due to this block size limitation; NTSC video is shrunk from 486 lines to 480 and HD 1080 video is actually compressed as 1088 lines. The mismatch between the compressed video size and the displayed video window may cause a vertical shift. ClearView includes an algorithm to locate the frame shift and adjusts the metric window size appropriately.
Loss of Color Fidelity
If the video is output via an analog source (component, composite, S-Video), then the colors may be shifted slightly due to the conversion from analog-to-digital. ClearView applies a linear normalization offset to compensate for this effect.
In the television field, audio video sync problems are commonly caused when significant amounts of processing are performed on the video part of the television program. This is usually caused by video compression encoders and decoders. ClearView measures the temporal shift between the source and processed A/V sequences so that an offset can be presented to the compression decoder to compensate for the processing delay.
Scoring Video Quality
Today, video equipment manufacturers verify the video quality subjectively. Subjective testing allows creativity but does not scale well. How can you test thousands of units? How do you provide for regression testing? How do you calibrate the test equipment and add features based on subjective feedback?
Results from our previous case study stated that subjective video analysis is the only accurate assessment of video quality (aka, showing the video sequence to a group of experts). However, subjective video analysis is only applicable for development and evaluation purposes; it does not lend itself to operational monitoring, production line testing, trouble shooting, or equipment specific repeatable measurements. The demand for objective video quality testing arose from the need for quantitative, repeatable video analysis.
ClearView objectively scores the video quality using the Human Visual System (HVS) tool known as Just Noticeable Differences (JND) or via MS-SSIM on the DMOS scale. JND or DMOS/MS-SSIM algorithmically predicts the video quality based on modeling hundreds of expert viewers, and ranking their score based on viewing distance, perceived artifacts, viewing angle, and TV display size.
To start, two video sequences are recorded:
- The original source
- The processed video
The Videos are temporally and spatially aligned and analog noise is normalized before starting the JND or other metric. Our JND implementation generates a Mean Opinion Score (MOS) on a scale from 0-100 where less than 5 is excellent video quality. For more information on JND, read Sarnoff Laboratories’ White Paper.
The main goal of calculating a JND is to automatically estimate the average viewer’s opinion score. The simplest use of ClearView is to compare two video sequences on the same display under identical conditions, and decide subjectively which one looks better. The videos can be displayed in split-screen, seamless-split, split-mirror (butterfly), or A-B (source minus result) modes where the split is either horizontal or vertical. Playback supports zoom, jog, shuttle, and pause for in-depth analysis.
After the video is scored using JND or your own “golden eyes”, this score can be compared for regression testing against any number of units. This video will be termed “Gold” hereto after. In this case, ClearView aligns the two video sequences and compares their video quality. Any video sequence, which scores below a user specified threshold, will generate a failure.
ClearView controls the test setup as follows:
- ClearView serves the original video sequence to the Device Under Test (DUT)
- The DUT processes the video data
- ClearView records the output of the DUT
- ClearView aligns and normalizes the “DUT” with respect to the “Gold” sequence
- ClearView calculates A/V delay and drift (drift = lip-sync errors)
- ClearView compares the “DUT sequence” to the “Gold sequence”
- ClearView records pass/fail per frame and generates an overall pass/fail score
- ClearView creates a log file with the results.
All of the metric data is archived into small log files. These log files can be replayed on any ClearView machine with access to the original video sequences, which ensures a repeatable test environment when anomalies are resolved. Further, these log files can be added to the regression testing to ensure that the anomalies stay fixed. For additional convenience, our Metric Log Graphing tool kit imports these log files into a set of instantly viewable pdf graphs with each type of metric score in the log overlaid for comparison purposes. CSV files are also generated by the Metric Log Graphing tool.
Figure 2: Encoder/Decoder (DUT) Pass/Fail Video Testing Example
GUI Testing versus Automated Scripting
ClearView can be controlled via its GUI or from its command line interface. The simplest way to use ClearView is through its interactive GUI. The results are logged & graphed; while the videos are previewed and outputted to an external display. This works well when ClearView , the Device Under Test (DUT), and the engineer/tester are in the same location.
When the development and system verification teams work in different countries and speak different languages, problems arise in detailing anomalies. ClearView includes a scriptable, command line interface, which offers the same features as the GUI with two added benefits:
- Commands can be entered from anywhere that has network access to the ClearView system,
- Commands can be batched with return codes dated and logged for reusability.
Using the command line interface, a tester in the United States can find an anomaly and send a report to the development team in India or China. Another team member can access the ClearView logs; thus replicating the issue, and fix it. Language issues are minimized and system verification can work around the clock.
Figure 3: Basic Principle behind Scriptable Pass/Fail Scoring
Video quality testing is a hard problem. Performance of a digital video processing system can vary significantly and depends on dynamic characteristics of input video signal (e.g. amount of motion or spatial details). Combining a video server, video recorder, video player, and video quality analyzer into one system with both an interactive GUI and an automated, scriptable interface, Video Clarity (http://www.videoclarity.com) has made objective video quality analysis and regression testing a reality.