Synopsis
The Real-Time Video Quality Monitoring system and application – RTM – is a multi-channel full reference test solution for monitoring and logging video quality, audio quality, audio/video offset (lip-sync), and audio program loudness. RTM also checks for VANC line data integrity from any uncompressed video inputs.
RTM tests these signal parts, reports the results graphically, and applies test score data in text logs for any duration. It will record incoming audio/video (A/V) segments as uncompressed video when the quality drops below a user-configured threshold applied to each test’s scale of measurement.
The recorded uncompressed video sequences from source and test input are then available for playback as a synchronized side-by-side comparison from the included RTM Player or ClearView to its system’s installed video output or desktop window for event review. Custom scripted routines via a command line or REST API may be easily applied for product QA regression testing.
Wireless Devices and Network Encoding Transmission Test with ClearView and RTM
for Medical, Military, Enterprise, or Encoder Manufacturer
Video delivery solutions over wireless networks are proliferating rapidly in several applications that need to be tested for quality. There are several variables in these applications that create a significant set of test requirements such as the video content characteristics, lighting variability, encoding set up including bit rate, wireless IP or 5G and the coming 6G transmission conditions.
To start this process, the RTM application for real-time long form testing or the ClearView application using short recordings of video sourced from the actual camera or other device is used to gain a benchmark for the quality.
-
- Applying several bit-rates or possibly varying camera, video content, and/or codec combinations in real-world examples through the encode, wireless transmission, and decoding path is recommended.
If the actual source device is not available, then several of the uncompressed sample video library clips that come with the ClearView application may be used to test initial source samples.
-
- As shown in the following diagram, the ClearView application can play these source clips to its system outputs via HDMI or SDI depending on the input required of the source side of the encoder/transmitter in the wireless transmitter/receiver pair.
At the processed end of this workflow is the receiver device which generally will output HDMI video and will need to be split between a display device and the Video Clarity system running ClearView to record using input/output mode.
-
- Video Clarity can recommend fully tested HDMI splitter devices for this purpose.
- Additionally, either the ClearView or RTM used with Video Clarity systems housing only SDI inputs may need an external HDMI to SDI converters added to this generalized diagram to record the decoder’s HDMI output.
Once inputs are provided with either live sources for long form testing with RTM or short captures with ClearView, each of these testing applications can automatically align the source video input to the encoded and transmitted input and then accurately run various quality metrics either via GUI, RTM Manager interface, or command line script.
Metrics and their application to quality analysis in wireless device testing:
-
- PSNR can be used to gauge quality levels for codecs that are intended to be at or above human perception of quality differences to the original. PSNR will give a very good indication if quality levels are at or higher than perceptual quality in high bit rate encoding and for lower bit rate lossy codec pass/fail threshold settings in either RTM or ClearView.
- Perceptual metrics such as MS-SSIM help to find the best level of human perceptual scoring for lossy types of encoding that can be achieved by the encoding technology in combination with the signal transmission requirements of applications that may have transmission bandwidth limitations.
Either of these metrics may be run in real-time to allow equal scoring methods between the RTM and ClearView quality testing applications
-
- After initial benchmark tests to understand the average quality level through transmission, several tests are then run using each metric with variables applied such as differing picture content, camera settings, lighting variations, a range of encoded bit rates and enhancement settings, or even different network conditions.
- Throughout these tests using either RTM for long form test runs and ClearView for short-form recordings, the benchmark phase allows the user to set a threshold for initially understanding below threshold events throughout the testing process.
The final decisions for acceptable quality levels are generally made through visual comparisons of tested processed content versus source originals.
-
- Each test may be recalled by drag-n-drop of its saved log file which recalls the test pair for viewing and inspection via the various ClearView comparison view modes.
Initially it will be easy to target and visually analyze below threshold segments generated from RTM real-time testing or from the threshold line placed in the ClearView GUI inside a tested video segment. However, finding and observing the effects of encoding and transmission artifacts may not be perceptually evident or identified via test thresholding.
-
- For this, ClearView’s enhanced viewing functions are highly effective by providing picture zoom/scroll and pixel difference highlighting such as A minus B with color addback .
- These enhanced viewing functions allow the user, who may not be an expert at unassisted visual analysis, to find and judge whether artifacts are objectionable based on the criteria of the application such as critical medical surgeries and surveillance video where highly accurate encoded and transmitted versions of the original video are required.
Full Transmission Path Testing for OTT, Cable, DBS, IPTV, and Broadcast
Many titles now originate in UHD for SVOD services and continue to be delivered in HD for broadcast origination. Testing delivered quality accurately for varying content characteristics creates a significant and on-going project to also test the different bit rates and delivery resolutions to the home or inter-facility networks. To simplify video quality testing of differing delivery services and receiver types, Video Clarity’s integrated HDMI input option for the RTM or ClearView Analyzer systems allow ease of use with highest accuracy full-reference quality metrics for video, audio, and lip sync.
In these examples the RTM 4K system is input with live source content from either SDI, ST 2100, MPEG over IP network as UHD or multiple HD videos with the resulting delivery samples being input from set top boxes or other service path devices. This way the type of network and delivery path is included in the test for a complete end-to-end quality assessment.
-
- Create quality average data over short or long duration tests
- Set thresholds for quality to find and record low quality events
- Measure PSNR for high bitrate, low loss codec transmissions
- Use DMOS/MS-SSIM metric for human perceptual accuracy to test home service quality
- Apply up to 16 channel audio test per program with millisecond accurate lip-sync test
- Compare different home service delivery paths with real-time video quality monitoring
Multi-Channel Testing for Statistically Multiplexed Channel Quality and Comparison
For broadcasters and other home entertainment service providers there is a requirement for comparison of quality testing scores for multiple channels being statistically multiplexed using the highly accurate full reference method. This allows channel groups coming from different ownership entities to be compared in real-time showing color coded video quality graphs. This way each entity may view a real-time comparison of score data in a browser with a time stamped log of full reference quality test scores that can be used to verify quality of video, audio, lip sync, and loudness after processing.
In this example one RTM system is used to test four different HD channel services with each tested channel being sourced from the incoming network feed and then each tested channel is sent to the RTM system via IP network directly from the processing device.
Additionally, RTM is shown applying multiple network sources versus processed inputs from the transmission to the studio channel group. This provides quality measurements including the transmission path so that any transmission degradation may be monitored and included in the full reference testing.
- Four tests are shown in a browser page which updates in real-time with each channel having its own distinct color on the graph with user set labels.
- If one or more of the testing channels dips below a user set threshold for quality a recording is automatically made of the source and downstream video pair.
- Additionally, one or more channel’s quality event recordings may be reviewed after the test run via the included RTM Player.
Full Reference Testing In A Media Delivery Network
- Source A/V signal is fed using professional decoder HDSDI or IP to the input #1 on RTM from points A or B as in the diagram below
- The processed A/V feed is then also fed via HDSDI, IP, or converted to HDSDI from HDMI on the back channel delivery device in the headend to input #2 on RTM from points B, C or D
- RTM tests for preset quality levels and also reports quality measurements for all parameters continuously on the desktop, log file and data unicast for as long as required
- RTM can be set to record A/V segments for the duration of the threshold errors with a preset heads and tails length for each recording
- RTM Player application is provided to recall recorded errors and play as synchronized side-by-side uncompressed video on desktop or RTM system video output for review and diagnosis
IP Delivery Service Monitoring – With A Scheduled Quality Test Using RTM
- RTM allows selection of source and processed stream from IP networks
- Source can be from baseband HDSDI or IP – Point A in the above diagram
- Processed feed can be from MPTS or SPTS in IP from point B, C or D
- A program test can be scheduled for single or multiple test instances using RTM Scheduler
- Additional tests of any duration can be preset for demux/decode, time alignment and restart of a monitoring session
- Tests include support for multi-resolution services via selective source scaling
Long Duration Product Audio/Video QA
Testing Functions Applied
In Real-Time
- Measures and reports audio and video quality
- Measures and reports audio offset versus video (lip-sync)
- Meters and reports both inputs for audio loudness
- Measures the VANC lines for complete data versus reference from SDI inputs
- Records both incoming video with audio during below test threshold quality drops
- Also with VANC lines when data is missing or loudness or lip-sync is outside set thresholds
- RTM Player provides side by side playback for detailed error analysis