20.11.2017
Going Deeper : Cable, Standards and ITIL
by Tony Orme.
At the start of 2013, BCE at RTL City was a hole in Luxembourg’s ground and in less than four years they were on air broadcasting 35 different channels across Europe and Singapore. Costas Colombus is BCE’s Special Projects Manager and gave a unique insight into how they made this mammoth installation work, including describing the issues and how they overcame them along the way.
Transitioning to IP is a major challenge for any broadcaster, especially when moving an existing facility. Greenfield sites have it difficult too, and learning how they successfully made their installations work will help us all.
BCE’s wish was to be format agnostic and ready for future technologies such as 1080P, 4K and HDR. Preliminary requirements dictated the system should be flexible, scalable, redundant and reliable. Cable installation was identified as a major area for cost savings so fibre was used extensively. To de-risk reliance on a single supplier, multiple vendors and COTS equipment was used wherever possible.
Waiting for ST2110
During the design phase ST2110 had not been ratified so Costas and his team needed to decide on the specification of the system that was available at the time, but would allow for future upgrade to ST2110. SMPTE 2022-6/-7 was chosen for SDI video and AES67 for audio over IP, and DANTE was used for RTL Radio.
To reduce the amount of copper cable required, fibre was used wherever possible, to the extent where all connections between buildings, floors and rooms were fibre, and the only cables carrying video that exist are intra connections within rooms and racks.
Fibre Challenges
And this is where the challenges began.
Fibre is an established technology and has been in active use since the 1970’s. However, the combination of interfacing it with SDI, and the variable quality of fibre available, caused Costas no end of issues. It soon became apparent that fibre and the associated connectors, SFP and QSFP’s vary wildly in quality and more attention had to be paid to the procurement stage. Systems could fail in a non-predictable manner.
During investigations into the issues Costas stumbled across another challenge, the differing attitudes and working practices of broadcast and IT engineers. Broadcast engineers are used to working in new, dynamic, fast paced environments where equipment often goes wrong and creative, out-of-the-box solutions must be found.
Banking Influence
IT engineers have their training and influence in the well-ordered world of the banking industry. Many of the working practices IT engineers work to are well established in the finance market, and their ITIL processes exist for good reason as decimal points can make an enormous difference in the finance sector.
Information Technology Infrastructure Library (ITIL) is a set of detailed procedures used in the IT industry and are designed to bring order to a system to stop engineers jumping into systems and fixing them without proper authorization. This method of working is usually alien to most broadcast engineers, but in the IT world there must be an audit trail for every change or modification. As networks typically affect so many different services, one change on one part of the network could have unintended consequences on another. ITIL aims to stop these actions but the system only really works in an established working environment.
IP television is at the cutting edge of technology and driving IT to its limits. Engineers are finding obstacles nobody had predicted or expected. Consequently, when something goes wrong it must be fixed immediately, there is no time to raise a help desk ticket or have five meetings about the proposed fix.
Forward With ITIL
Broadcast television will eventually move to the ITIL model and we have already witnessed this in SDI playout facilities. The last thing anybody wants is an unscheduled software upgrade causing outage on fifty pay-per-view channels. But we must have working systems before these processes can be instigated.
ITIL is used to maintain, install and modify systems that are understood, well-ordered and defined. Broadcast IP is not there yet. But it’s fair to say that if we had stuck with SDI playout systems we would probably be using ITIL extensively now. However, to facilitate the new business models of flexibility, scalability and COTS, we must move to IP and we will continue to find solutions to complications we do not yet know we have.
Improve Equipment Monitoring
One of the areas Costas highlighted for equipment design improvement is monitoring the SDI-IP interface where television turns from copper coax to fibre. He also identified varying quality in SFP/QSFP light sources causing packet loss. Using light power meters from the telecom industry, BCE’s engineers could prove that fibre channels were failing, but in a further twist this always wasn’t due to light transmitters.
Procurement Specifications
Terminating fibre proved to be fraught with the same cost issues affecting fibre. Inferior quality connections don’t tend to just fail, they start by creating unpredictable erratic packet loss. Channels would work one minute but not the next, again the origin of the failure proved difficult to find. Was packet loss due to issues in the SDI domain or IP? A failing SDI connection can cause packet loss on an IP connection due to the amount of erratic data being created.
BCE solved the dilemma by assigning a small team of wiremen (fibre-men) to running and terminating or connecting fibre exclusively, a task requiring great skill, patience and experience, and giving highly specified fibre specifications to his procurement department ensuring the use of high quality fibre.
Dirt is the fibre’s greatest enemy as even small amounts of debris can cause major headaches for video-IP installations running at the limits of their capabilities.