Using the Quality Dashboard
Full-Time Monitoring for Broadcast Signals
Modern broadcasters use a variety of techniques to monitor the quality of their outbound signals to make sure that signals delivered to viewers contain the desired programming and advertisements. Typically, this process uses human viewers to watch one or more signals displayed in real-time on a monitor wall. This method catches many of the major errors as they occur, but is of little value for troubleshooting subtle or intermittent problems.
With continuous, automatic signal quality measurement, a wider variety of flaws can be detected and captured for problem diagnosis and resolution. Full-reference monitoring technology (where an output is compared to “known good” signal) allows errors to be captured, including subtle changes in audio/video synchronization. Through the use of the quality dashboard feature of Video Clarity’s RTM Manager™, system operators and engineers can quickly check the status of a multi-channel playout facility, and easily identify the quantity and severity of errors that have occurred over a given time period. This allows troubleshooting to be prioritized for those channels and equipment that have the greatest need, and provides a quick summary of overall system health.
Delivering a high-quality signal from a modern broadcast facility requires the complex interplay of a wide variety of video, audio and data processing equipment. Taken individually, each of these components is, for the most part, highly reliable and stable. However, when they are combined into a complete operational system, the overall performance can be adversely affected by any one of the constituent parts. As new technologies, services, and regulations are introduced (such as the CALM act and new captioning rules), broadcast equipment needs to be upgraded or replaced on a continual basis. This churn, whether due to a new firmware download or a simple reconfiguration, can cause subtle changes in the outbound broadcast signal. Monitoring these output signals is a full-time responsibility for broadcast facilities.
Figure 1: The Video Clarity RTM Manager
Most local and national broadcasters employ a crew of tal-ented, experienced operators who continuously monitor the outbound broadcast signals that are delivered through their facility. Using multi-screen viewers, all of the programming is constantly observed by professional staff, both before delivery to satellite teleports or transmitter and also after transmission via local satellite or over-the-air return receivers. While this methodology works well, and catches most of the significant service-affecting problems when they occur, it is not well-suited for catching intermittent errors or subtle degradations. Catching these types of problems requires continuous, machine-based monitoring of the outbound signals to provide more repeatable and reliable results. This philosophy forms the basis for installation and use of a signal quality reporting dashboard.
The basic concept of a quality dashboard is to provide an easy-to-read, visual summary of the performance of a complex video delivery system. Key quality parameters are displayed alongside a simple red/yellow/green indicator, allowing the user to quickly determine if any measurements have exceeded pre-set tolerance limits. Minor excursions are indicated with yellow, major issues are shown in red, and fully conformant measurements use a green indicator. Each parameter is also viewable in graph form that shows how the measured values have varied over time. When installed as recommended, the quality dashboard is used to monitor both audio and video signals in real time. Video signals are monitored for loss of signal, quality degradation, color distortion, compression artifacts, and other defects that affect picture quality. Audio signals are monitored for noise and other quality degradations. In addition, the dashboard system provides continuous measurement and reports changes in audio/video (lip) synchronization. Using these three measurement categories, the vast majority of problems that impact viewer satisfaction can be detected.
Figure 2: RTM and RTM Manager in a Broadcast Facility
Full Reference Monitoring
Making accurate measurements is fundamental to the useful-ness of the dashboard. False negatives (where actual problems are not reported) make dashboard users think that all systems are operating properly when they really are not. False positives (reports of errors that did not actually happen) are even more detrimental, because they can cause broadcaster staff to re-search and attempt to troubleshoot errors that did not actually occur. To help prevent both false negatives and positives, and to support a wide variety of tests, the quality dashboard uses full reference monitoring.
All testing involves measuring an unknown entity using a known scale or standard. In the case of full reference monitoring, the equipment performing the test is supplied with two copies of the content: a source (or reference) version of the video content, and a version that has been processed through some type of network or equipment. Using these two signals, measurements can be made in-service, so that normal video delivery continues uninterrupted while tests are being performed. For a national network or cable broadcaster, the reference video is the uncompressed, live-to-air video signal being created by the active playout server. This signal is compared in real time (after adjusting for system delay) to the output of a decoder, which receives the compressed signal either before or after it is delivered to the satellite uplink. Figure 2 shows a typical setup for a network originator. For a local broadcaster with network-supplied and locally-generated content, the reference video could come from a satellite receiver’s baseband output, or from the local studio feed or playout server. This signal is compared in real time to the outbound broadcast signal, via return signal decoders. Any problems that occur in signal processing, compression, encryption, commercial insertion, or localization can be detected, flagged and logged.
Using the Dashboard
Summary results are displayed on the quality dashboard dis-play screen shown in Figure 3 at any time for review by broad-cast staff. Video, audio and lip sync are continuously tracked and any deviations outside their defined range generate an alert. Whenever this happens, the monitoring system logs the occurrence, and saves a sample of both the reference and the tested streams, both before and after the occurrence. The recording period of the two streams lasts for the entire duration of the fault, plus additional video and audio both before and after the fault. The amount of extra video and audio recorded with each fault is user-defined, with typical values between 300 and 600 frames (10 to 20 seconds) both before and after the faulty sequence. These captured video/audio segments allow later analysis of both streams to be performed, so that potential problem sources can be identified and corrected. After looking at the dashboard results, the system operator typically resets the threshold alarms, so that any new errors will be indicated.
Video analysis is performed using the PSNR metric, which stands for Peak Signal-to-Noise Ratio. This commonly-used technique measures the amount of video noise (compression errors, distortions, erroneous pixels) in a signal as an approximation of how the video would appear to a viewer. High PSNR values (typically 35-40 dB) indicate good image quality. Low PSNR values (below 20 dB) indicate an image that would appear degraded to most viewers. A PSNR threshold of 20 dB is commonly used, so that error reports are triggered and video/audio is captured whenever the PSNR dips below that value.
Audio signals are analyzed by breaking each of the two signals (reference and tested) into frequency components and com-paring them. This permits two measurements to be made: delay and distortion. In order for the tested signal to be compared to the reference signal, any delay that has occurred must first be removed. Some delay is normal, because both the video encoder and decoder devices introduce significant delay. Once the two signals are time-aligned, any audio distortions that have been introduced can be measured and reported. A threshold of 50 is commonly used on a scale of 0 to 100 to flag excessive changes to the audio signal; this measurement value permits any audio losses or significant distortions to be recorded for later analysis.
Audio Loudness is measured using the ITU standard for LKFS measurement. With this capability, any video segments where the loudness threshold is exceeded can be captured for later analysis, and violations are reported to the video dashboard. This capability allows stations to monitor compliance with the CALM act and other regulations.
The amount of audio delay is also compared to the video de-lay that is measured between the two input signals. If the audio delay is identical to the video delay, then lip synchronization has been maintained through the system under test. Whenever audio delay differs from the video delay, lip sync errors are present. A limit of +/- 10 milliseconds is often used as a thresh-old for errors in lip sync and any excursions beyond this limit are logged and video/audio is recorded for subsequent analysis.
One major benefit of full-reference monitoring is the system’s ability to capture both the defective video as well as the original source video. This provides an excellent reference for communicating with equipment suppliers, because they can be shown exactly what the errors look like, and the exact point in the source video sequence where the error occurred. This information allows fault analysis to fairly easily determine if the fault was repeatable due to a compression algorithm error, of if the fault was caused by a spurious signal transient (also known as a glitch). Whenever a repeatable error is found, equipment suppliers can be informed and provided with two recorded audio/video clips: one that contains the source sequence and an-other with the captured fault. Figure 4 shows an example of a side-by-side comparison being made between one frame of the captured reference signal (on the left) and the corresponding
frame in the captured defective video (on the right). Both of these 10-second clips were captured automatically by the RTM. Continuous monitoring also allows broadcasters to know whether or not errors that are detected downstream have been caused by their equipment; this info is valuable for discussions with distribution partners and advertisers.
Figure 3: The RTM Manager Status Window
Figure 4: The RTM Manager Control Window
Broadcast and cable network originators can use quality dashboards to support multi-channel operations. These systems can be used to ensure that compression encoders, multiplexers, encryption devices and other signal processing equipment are producing valid outputs. End-of-line testing can also be per-formed by connecting the full-reference monitoring system to consumer video services that are available in their office. This configuration allows observation of what happens to the video and audio signals after they have been transcoded and manipulated on their way to a consumer-grade set top box.
IPTV, CATV and DTH Satellite providers can also use quality dashboards to monitor the performance of their signal processing equipment. The RTM system can use an incoming network work feed as a reference, which is then compared to the output of a reference decoder (such as a consumer set-top). This setup is particularly useful for measuring changes in audio/video synchronization and for spotting problems that arise from transcoding signals from one compression standard into an-other. To obtain the most useful results, the reference video should contain all of the content that is delivered to consumers, including any locally-inserted commercials.
Figure 4: Comparison of video clips captured by RTM
Quality dashboards using the Video Clarity RTM Manager™ have been deployed at several major broadcasters in the recent past. These systems have helped ensure delivery of a more consistent, higher quality output signal. The full-reference test method enabled these broadcasters to record, analyze, and correct problems in their playout facilities, particularly for trouble-shooting transient compression artifacts. These systems saved hours of time and effort that would otherwise have been spent to diagnose intermittent problems. Plus, their compression and multiplexing vendors have been able to more accurately repro-duce problems and more quickly resolve them, resulting in better video transmissions.
For the future, downstream distribution partners and other network broadcasters may want to consider adopting the quality dashboard, at least to provide the ability to do full-reference monitoring of operations within their own processing facility. This will help any organization get a better understanding of how their systems are performing, and give them solid data to feed back to their equipment suppliers. Automated, full-reference monitoring benefits every step in the distribution chain, and ultimately provides a better experience for viewers.