DR/BR lesson from 9/11: have backup on backup on backup – and a plan.
“Inaction is not an option. It’s only a disaster if you don’t plan for it”.
During the last few weeks we’ve been reminded about the events of 11 September 2001, now known universally as “9/11”, when the World Trade Centre twin towers collapsed.
Over many trips to the US I had personally visited the towers a few times, doing all the obligatory tourist things. That included heading to the roof on my first trip. As a broadcaster, I marveled at the antenna system on the top of 1 WTC, the North Tower, extending a further 110m from the roof of a building already 420m above ground.
From 1978, most of New York’s broadcasters had FM and TV transmitters at 1 WTC. Being the tallest spot in the city (indeed: tallest spot in the US and, for a while, the whole world) made it an ideal site for STLs and remote links; not just for TV but also for the many AM broadcasters who needed to bounce their program feed from a mid-town studio to mast sites along the Jersey shore and elsewhere.
All broadcasters operating at the site had several layers of hardware backup in the transmitter room. Main and standby transmitters, parallel link paths, duplicated processors, signal and control path switching, supported by backup power generation and multiple antenna arrays. As you would. No chance of going off air. Sure: despite all the equipment duplication, there was only one mast on the building, but that mast was comfortably engineered to take all the services.
It was such an obviously appropriate location that almost every broadcaster had cancelled their legacy leases at the Empire State Building (and the WTC had paid for many years of those leases in a sweetheart deal to get the broadcasters to relocate from Empire to WTC). So Empire State was mostly fallow, a 1930s building just capable of, but not required to do, a 2000s job.
Surprisingly, backup-on-backup didn’t include site diversity. Nobody seemed to have considered whether the building itself needed a backup; building failure was not on anyone’s radar. There was just no plan to deal with the bizarre consequence: what would happen if the North Tower suddenly was not there.
“Suddenly” turned out to be 102 minutes, the interval between when the plane hit the north tower, and when the building came down. It was all over in seconds.
In that moment, all New York free-to-air broadcast transmission and link services stopped, silencing every service that was on, or went via, 1 WTC. Hardwire backup links and other services – ISDN, landlines, phone lines, still worked until, at the end of an already unthinkably bad day, the Verizon Central Office (telephone exchange) went offline with the collapse of 7 WTC.
The extent of all these cause-and-effect relationships was extreme, and totally implausible. And yet, it happened.
The 9/11 incident contains huge lessons for everyone in our industry. So here’s a real gem: at the Audio Engineering Society 115th convention held in New York on 10 October 2003, a group of broadcast engineers staged a panel titled “REBUILDING OF NEW YORK BROADCASTING” in which they took the audience through the events of the day, what broadcasters did to recover, and many of the lessons learned.
All the people on the panel, even the Public Broadcasters, are part of the big end of town: consider that their audience reach required them to be on the world’s tallest building. But here’s the thing: what they say is incredibly relevant to small community broadcasters – even small stations running 10 watts – because the learnings from any recovery exercise are universal truths.
This is one of those talks that you really need to give a listen. The full session ran 2.5 hours, and most of it was captured in this recording.
Hint: the file isn’t very big (20MB) and is easier to navigate if you download it – although you can listen to it just fine from the link above. I have not yet been able to find the slides that accompanied the talks but I’ll reach out to AES over the coming week. Meanwhile, there are plenty of generic pictures on the web.
To help put things into perspective, check out Wikipedia articles on
and you can look at maps of the area.
Also, here’s a fascinating 1967 article about the Empire State Building antennas, written before the WTC was commissioned and fully 11 years before WTC North Tower came online.
Notes on the recording
Many times the location “Kearny” is mentioned; that’s a suburb of New Jersey, just west of lower Manhattan, home to many AM transmitters and backup FM. You can see the towers around 282 Polito Ave Lyndhurst NJ (Westinghouse site, four mast directional), and another two sites literally a few minutes walk away at 1427 Valley Brook Avenue (look for towers on both sides of the road). There’s another directional tower set about 200m south of Valley Brook.
At least two of the AM tower clusters have an FM array at the top of one mast. Wouldn’t that work? Well… it looks really high and effective; a mid-band 1/4 wave AM mast is probably around 80m high. That’s equivalent to putting an FM antenna on the top of a 25-storey building with nothing around to obstruct it. Perfect, until you consider the wall of buildings that is Manhattan, and compare an 80m mast to the WTC pole, which was 110m high and had a 420m building underneath it. Six times the height and right in the middle of the greater NYC/New Jersey area.
Listen carefully, and you will also hear occasional references to another tower in Alpine NJ, north west of New York City. That’s the same location from which Edwin H Armstrong did the very first practical proof tests of FM broadcasting.
The Verizon Central Office next door to 7 WTC, and carrying significant amounts of broadcast traffic, was impacted when 7 WTC also came down. Although much of the exchange appears to have been OK, cables and interconnects were affected.
7 WTC’s collapse was not from direct physical damage caused by the collapse of 1 WTC adjacent, but from fires started on multiple floors. Some sources have attributed fires and the subsequent collapse to burning of diesel fuel, part of emergency power generation plant located quite low down. The 7 WTC investigation suggested otherwise, and it’s very difficult to get diesel fuel to burn, but lack of water, bad sprinkler system design, and flying av-gas and flaming debris from the towers, created a perfect storm. 7 WTC is regarded as the only steel skyscraper ever to have collapsed due to fire.
What can you learn from the session?
Everyone on the panel was violently in agreement about some really valuable conclusions:
- you can’t be too prepared;
- you need a plan;
- you need to test the plan;
- you need to consider what’s the impact of being not on the air; and
- your plan needs to cover eventualities that can’t possibly happen, because they can happen.
Enjoy, and comment in the Technorama Q&A.
Panel participants were:
- David K. Bialik – Systems Engineering Consultant
- Howard Price – ABC (the American corporation)
- Joe Giardina – DSI
- John Lyons – Durst Organization
- Kevin Plumb – WABC, WPLJ
- Steve Shultis – WNYC Radio
- Thomas Silliman – ERI Inc.
There are many programs and stories which have been told about the events of 9 September 2001. None that I have seen comes anywhere near delivering the clarity and impact of “9/11: Life under attack”. It is beautifully crafted, and tells the story through the eyes of people who had the presence of mind to hold a camera and keep it running.
At the time of writing, the program is available on ABC iView and also on YouTube. It’s worth looking at both: they are two quite different edits all the way through to the end (tech note: you can sync the closing credits, and the overlay is identically timed, but the backgrounds are different).
Final word on backup
So did you notice the photo at the top of this article? Who is this person who is standing at the top of a pole, over half a kilometre off the ground, leaning over into space while holding on with one hand, and with only a shirt and workboots as protection? Wow. You would not want to sneeze. Even more bizarre? The photo was shot from above. Someone even more crazy was higher up, hanging in mid-air to take the pic.