Case Study – How TelevisaUnivision Ensures Real-Time Visibility of All-IP Broadcast Network with AlvaLinks Observability Suite
TelevisaUnivision, the premier global Spanish-language media company, is known for its vast repository of Spanish content and leading original content production across various genres. In a strategic move towards digital transformation, the company embarked on transitioning its international broadcast operations from traditional satellite networks to a more versatile all-IP workflow.
This initiative focused at enhancing broadcast quality, and integrating a cutting-edge ecosystem of services and solutions. Acknowledging the unpredictable nature and the complexities associated with IP networks and live video streaming, TelevisaUnivision’s engineering team proactively adopted a comprehensive toolkit designed for real-time network monitoring, ensuring the seamless delivery of high-quality video content.
Enhanced Video Distribution Architecture
To ensure utmost reliability and align with recognized industry standards, TelevisaUnivision implemented a highly resilient dual x dual redundant IP architecture for its video distribution network. The primary site features a dual encoder system, transmitting content via SRT to two distinct cloud availability zones, complemented by a secondary on-premise site for geographical diversity, mirroring the primary site’s functionality..
Each network path is designed with robust switching and redundancy capabilities to seamlessly handle potential router failures and maintenance requirements. Within the cloud infrastructure, each availability zone houses a transcoder to process incoming feeds, with the output then distributed to four strategic cloud entry points for content preparation and delivery across the U.S. The choice of SRT protocol underscores TelevisaUnivision’s commitment to reliable, high-quality video transport over IP.
Unexpected Quality Issues During IP Network Testing
TelevisaUnivision’s engineering team, fully aware of the inherent challenges and intricacies in IP-based video delivery, embarked on thorough network testing to preemptively identify and resolve any potential disruptions. During these tests, it became evident that video stability was compromised, and the existing redundancy strategies did not meet their standards and expectations. Despite diligent efforts to monitor workflow alarms triggered by ETR290 events and conduct standard IPERF/Traceroute tests, the root causes of instability remained elusive, highlighting the complexities of maintaining consistent video quality over IP networks.
Effective troubleshooting and optimization of IP-based broadcast operations
necessitates a profound understanding of network behaviors.
In line with best practices, TelevisaUnivision’s remediation efforts began with several IT fine-tuning activities such as MTU expansion, setting ACL rules, DNS update, and setting QoS for each stream.
The team then performed an in-depth network evaluation, which revealted several network problems:
- The cloud-based SRT transcoder experienced intermittent stoppages lasting several minutes.
- Ping and IPERF round-trip time (RTT) tests yielded inconsistent results.
- Network paths to the cloud demonstrated a packet loss rate of 2-4%.
- Frame errors were detected without corresponding network errors in traditional monitoring tools.
- The operations team periodically utilized IPERF while logging ETR 290 alarms, in an effort to correlate them with detected network events.
The evaluation led TelevisaUnivision to the realization that gaining deeper insights into the fluctuating behaviors of their network was crucial for identifying the root cause of their broadcasting challenges
TelevisaUnivision partnered with AlvaLinks to perform deep analysis and understand their network’s behavior, aiming for an in-depth insight into vital network parameters like latency, jitter, packet loss, and RTT within their MPLS and fiber circuits. This collaboration equiped the engineering team with the necessary tools and information to customize systems and align equipment with the unique requirements of IP network management.
Alvalinks Cloudrider™ Observability Suite Solution
Cloudrider is a patent-pending technology based on proactive network discovery, path detection, network probing and packet-by-packet testing. It helps broadcasters detect any network issue that may impair video delivery. Cloudrider effectively maps the network from source to destination, provides insight into network behavior, and allows engineers to pinpoint networking challenges and transient events that impact video streams.
- Quickly identifies root causes, reducing troubleshooting time from days to minutes.
- Decreases operational expenses by lowering MTTR.
- Resolves issues in real-time, safeguarding the viewer experience.
- Designed for the specific challenges of managing video traffic.
Helping TelevisaUnivision and Its Partners Pinpoint Root Cause
AlvaLinks software agents were installed on each on-prem encoder and the cloud transcoder node. Within an hour of activating CloudRider, it identified RTT and latency irregularities that matched with packet loss incidents. CloudRider’s buffer emulation revealed that the minimum buffer latency exceeded the SRT settings during loss episodes, highlighting a misconfiguration that hindered the handling of network transients, often undetectable by standard IP monitoring tools.
Additionally, CloudRider pinpointed simultaneous routing and error events across all links within seconds of each other, without any external traffic interference, suggesting a synchronization issue. This was particularly surprising since the network links were intended to be fully diverse for redundancy.
CloudRider also revealed the network’s capacity to reach 500mbits, albeit with a consistent packet loss of 4%, indicating underlying performance constraints.
Summary and Next Steps
Through AlvaLinks’ technology, TelevisaUnivision gained unprecedented insights into its network’s performance, uncovering issues invisible to traditional monitoring tools. This deep dive into network analytics enabled precise adjustments in their operational strategy, significantly enhancing service stability and efficiency.
- Detected packet loss rates between 15-22%, not identified by IPERF.
- Revealed network capacity up to 500mbits with a baseline packet loss of 4%.
- Identified synchronized events affecting ETR 290 alarms and uncovered incorrect bandwidth limits, highlighting the intricacy of network challenges faced.
This information enabled reconfiguration and streamlining of the video equipment, network links and cloud contribution, optimizing traffic flows, and finally eliminating the recurring service interruptions and ensure smooth service operations.
Building on the success of the initial AlvaLinks deployment, TelevisaUnivision has expanded its use of CloudRider for continuous network observability. This includes monitoring contribution feeds from Latin America over private MPLS and fiber circuits, ensuring potential issues are addressed proactively to maintain impeccable video quality. Looking forward, CloudRider’s network observability capabilities will also support TelevisaUnivision’s dynamic video production for sports, venue, and music events, exemplifying a forward-thinking approach to broadcast excellence