EBU’s Euroradio Live Network transformed into Radio over IP
Streamlined by 2wcoms IP-4c Audio over IP codec
Author: Anke Schneider
Photo © by EBU
The European Broadcasting Union (EBU) is the world’s leading alliance of public service media (PSM). The Euroradio Live Network of the EBU has been used by 47 EBU radio organizations so far to broadcast concerts and events via the channels Haydn and Liszt.
In 2017, it was decided by engineers of the EBU radio organizations and Eurovision Services (ES SA) that it was time to renew it. The previous contribution and distribution via satellite had to be replaced by Audio over IP, without any loss of quality. At that time, the existing system comprised three channels (satellite and web). Most of the solutions in use had almost reached the state of end of life, the audio codecs were obsolete, and multiple ways to contribute/distribute did exist. The challenge was to replace the existing equipment before the current network becomes unreliable and to find a solution that fits the following requirements:
- To simplify the exchange of live radio concerts among the EBU radio organizations.
- To achieve levels of reliability and quality at least as good as the Satellite solution.
- As a bonus, the possibility to take advantage of the new infrastructure for extra services (like bi-lateral exchanges and applications beyond concerts).
- The overall latency shall not exceed a threshold of 10 seconds.
- To lower the Total Cost of Ownership (TCO) in comparison to the existing network.
For the system architecture, the Euroradio Contact Engineers, in cooperation with ES SA, opted for the EBU Flex platform, which was already in use for the television area of Eurovision. The solution was developed by the engineers of ES SA. As an independently operating digital transmission tool, it enables the distribution of live content over several networks. The basis for the provision and distribution of content is the Secure Reliable Transport protocol (SRT), which ensures IT security and stable operation via its integrated features, such as:
- Security through AES up to 256-bit encryption.
- An advanced technique to recover lost IP packets: Only in the event of a packet being lost will the exact missing packet be requested from the sender. Compared to Pro MPEG FEC or SMPTE 2022-7 (dual streaming), the latency and the required bandwidth are lower.
- At the receivers, a timestamp recovery synchronizes the audio and video streams.
- Each endpoint is configurable as a caller, listener or rendezvous mode.
- The monitoring tracks the network performance parameters between endpoints, such as IP packet loss, latency, and jitter.
To connect the 47 EBU radio organizations involved with live transmissions to the EBU Flex System, the ROIP project team searched for an Audio over IP codec that ingests live concerts into the orchestrating EBU Flex system or receives them. Therefore, a shortlist of minimum requirements was defined. The equipment should provide at least:
- At least two channels in one unit to reduce the total number of devices.
- AES3/XLR interfaces for receiving live concerts.
- Audio codecs such as PCM and Opus for audio encoding.
- SRT and RTP/RTPC for IP streaming.
- The ability to control the equipment remotely via HTTPS, SSH, or SNMP.
- PTPv2 for time synchronization.
- TS over SRT.
The vision for the future ecosystem has been set. From this point on, a very intensive research process started to find an audio-over-IP codec that met the above requirements and technical specifications. In addition, the new hardware should allow the use of shared resources (devices, networks, etc.) and preferably support distribution via the Public Internet or CDN. Part of the project scope was a proof of concept to validate solutions from three manufacturers. The POC started in December 2018. Among other things, ES SA engineers tested the following in the lab:
- PCM streaming over six hours.
- Cutting the internet line with a buffer higher than one minute.
- Switching off and on an audio-over-IP codec.
- Encoding and playing a short one-minute clip 60 times in a row.
- Set the buffer to ten seconds.
- Mixing Raw PCM and Opus in/out.
- Shut everything down for 30 seconds and restart the equipment.
- Internet line with at least 500 Kbits/s (Opus).
An international team consisting of engineers from ES SA in Switzerland and four EBU Contact Engineers in England, Germany, the Czech Republic and Bulgaria worked on the proof of concept. One of the aims was to be able to test the new system under the most varied Internet line conditions.
An anecdote on the side: The 2wcom project team delivered the IP-4c codec personally on the 11th of December 2018. On this day, one of the ES SA’s engineers decided not to participate in the joint lunch break. Instead, he took the opportunity to test if the configuration of the web interface is possible intuitively without support or manual. He was impressed.
As an extra surprise, the 2wcom team also had the virtualized version of the IP-4c in their luggage called MoIN (Multimedia-over-IP-Network-Software). Both solutions were tested in the EBU laboratory and validated regarding the digital transformation process.
At the end of the POC in January 2020, four key points tipped the scales in favor of 2wcom's Audio over IP codec.
- The IP-4c codec scores with the best technical test values, intuitive configuration of all settings and the monitoring function in the web interface.
- The codec was the only hardware device that supported SRT as required.
- The IP-4c offers up to 4 decoders and 8 encoders in one unit.
- Outstanding responsiveness regarding support requests. The manufacturer's team of engineers always took care of the new requirements accordingly. For example, audio stream distribution using the SRT protocol via DVB-S/S2 was implemented while the evaluation was still in progress.
After decision-making, the delivery and installation process began in summer 2020. Logistically, the project teams had to organize the delivery of the IP-4c codec to each EBU radio organization. A total of 65 people had to be contacted by 2wcom to coordinate the delivery dates and to inform the technical contact partners about the further procedure.
To efficiently provide all relevant information on the IP-4c codec, the EBU Project Manager organized a webinar. 2wcom presented the functions and configuration options relevant for the project. An overview of the additional possibilities was also given, as only the first two of the four channels are intended for EURORADIO live concerts (defined configuration). The third and the fourth channel can be used for flexible usage - for example for EURORADIO news or similar purposes. After the webinar, the ROIP project team offered further training dates and technical checklists for the system configuration.
The proof of concept in the field for all members started mid of August 2020. The basis was and is an architecture containing the EBU Flex platform and 2wcom Audio over IP codecs. The transmission and distribution of concerts work as follows: At the contribution site, the IP-4c encoder packages the uncompressed PCM audio into the SRT protocol and establishes redundant point-to-point connections via SRT to the EBU FLEX headends in Geneva and Leuk. Within the EBU FLEX headends, the incoming audio streams via SRT are converted into IP multicast by the codecs placed there and routed again as unicast via the outgoing SRT streams to the IP-4C codecs of the EBU radio organizations members. This concept makes it possible to realize point-to-multipoint connections via SRT.
The first level of transmission stability is provided by the redundant set-up of the orchestrating EBU Flex server farms in Geneva and Leuk. The second instance for redundancy is the previous satellite system until the quitting of the transponders for Euroradio purpose. Thus the provision of live concerts was possible via IP and satellite in parallel. If the IP connection is down, the delivery of the contributions is ensured due to the satellite backup.
At that time, abandoning the parallel mode was scheduled for the end of December, depending on the Radio Over IP solution was working fine without any satellite backup. It was a time full of challenges, but any stumbling blocks that arose were overcome in cooperation with the EBU radio organizations and all partners involved. Let us have a look at the story of learnings, successes, and struggles:
- In autumn 2020 some EBU radio organizations noticed a bug on the IP-4c that occurred only every three weeks. The long interval between the failure occurs made it complicated to replicate it and to identify the problem. In cooperation with the EBU Contact Engineers monitoring the bugs behavior, the engineers of 2wcom managed to fix it in time before the real launch in January 2021.
- For the upcoming Christmas Music Day in December 2020, where 15 concerts were broadcast live in a row, the project team decided to use the old satellite network as a backup for the new Radio over IP network. As expected, with the option of having a backup on the EBU Flex system, there were no failures. It went perfectly. It was the first time in 4 years that not a single request was received for a replacement sound file due to transmission quality issues or dropouts.
- At that time, everyone involved was looking forward confidently to the satellite system being removed from the operational workflow and the live set-up being Radio over IP only. It was too good to be true. During the New Years Concert, the satellite backup was still operational. Which turned out to be a good decision because an older version of the SRT protocols' synchronization method, the Timestamp-Based Packet Delivery (TSBPD), was sensitive to discontinuous jumps in the system time. This led to incorrect clocking and failures. With the latest SRT versions, this problem no longer exists.
- After that some IP connection caused problems on the Flex Farms.
Based on the learnings of the POC in the field, the final architecture now includes several IP backups for encoding and decoding. Besides the redundant setup of the Flex farms A and B, one more instance was installed at the ARD-Sternpunkt in Frankfurt am Main (Germany). The service is called Technical Coordination Service Provider (TSCP) and comprises two IP-4c. The first IP-4c receives a concert contribution when the connections to Flex A and Flex B fail as a Direct-SRT stream and via a SIP connection. The second IP-4c is dedicated to forward the received contribution to the orchestrating Flex eco-system. In addition, two separate Icecast servers are served. In case the SRT stream to the receiver sites fails, members can fall back on the Icecast streams.
To complete the redundancy concept, the IP-4c decodes at the receiving sites provide three backup sources that can be switched to: SRT stream from the redundant Flex Farm, Icecast stream from Flex Farm A and Icecast stream from Flex Farm B.
Conclusion: After some months of usage of EBU Radio Over IP solution, it appears that all the technical choices made were good. The system is very robust and reliable. There had been a few issues, but the significant ones were always due to external factors, like a problem with an Ethernet link (provider change).
During a live concert broadcast, members particularly appreciate the possibility of real-time monitoring via the IP-4c's web interface. Added to this, is 2wcom's very professional support. Always very reactive to investigate and resolve any difficulties in using the IP-4c equipment as quickly as possible.
Redundancy was an important aspect when designing the solution. When a connection went down, one of the farms was not operational. Thanks to the alternate farm, there was no interruption in service. Very transparent to Members receiving live concerts. In the worst case, if the two FLEX farms were down, a contribution can be established by direct SRT connection and, distribution is still available via the Icecast servers. The quality would be slightly lower, but probably not noticeable at all. The enhanced setup for redundancy finally allowed to abandon the use of two satellite transponders. As a result, the operational costs of this service offered to EBU radio organizations are lower.
The new system appears to be more open, and it is now even possible to include new participants, which were previously not under the satellite coverage and therefore not accessible. For instance, it is planned to include a contributor from the US into the network by the end of the year.
In short, the EBU and its radio organizations are excited about this new way of exchanging live concerts among them and over the public Internet. All of them are amazed at how robust the SRT protocol is, allowing the streaming over the Internet of high quality-broadcast media. This protocol combined with performant hardware, a good design, and a dedicated project team, were the keys to delivering this successful broadcast solution.