Witness Post: Metcalf Echoes Hinsdale

Metcalf Substation, San Jose, CA

 

Witness Post: Metcalf Echoes Hinsdale

 

The Wall Street Journal published an article on February 4, 2014, highlighting the serious concerns about the vulnerability of our country’s power grid.  The surgically executed assault took place on April 16, 2013 on the Metcalf sub-station in San Jose, California should have set off alarm bells, but few seemed to be listening. The power at the sub-station was cut just before the snipers opened fire. The attack was carried out in less than 20 minutes. More than 100 bullet shells from AK-47’s were found outside the fence in key locations, destroying 17 giant transformers.  Those transformers carry all of their electricity to power Silicon Valley. We have heard these warnings before. Are we not listening? [1]

 

The story echoes a saga from 16 year ago which exposed the soft underbelly of the US system of wires, fiber optics, and cables that connect our electrical power and communications grids.  In 1988 a smoldering fire broke out in a telecom substations in Hinsdale, Illinois, and although the flames were extinguished, the wide ranging devastation of the fire only became clear over the next few days. Hinsdale and other hub stations proved extraordinarily vulnerable; they were weak spots in our grid. When our country specifically designs power and communication systems that funnel energy through “strategic gateways,” those nexus stations are prime targets for terrorists. Failing to protect them threatens our nation’s security.  Are we afraid of bad people or bad practices?  Food for thought: we live in a world where both thrive.

 

It took Pacific General Electric 27 full days to repair the Metcalf substation, which is about as long as it took Illinois Bell to repair telephone and fiber optic cables in Hinsdale.  The main difference between these two incidents is that Hinsdale was accidental and Metcalf was sabotage.  The former Chairman of the Federal Energy Regulatory Commission (FERC), Jon Wellinghoff, called the carefully planned and executed attack on the Metcalf facility “the most significant incident of domestic terrorism” to the US power grid that had ever occurred. [2]  Was it that bad?  Or were his comments full of hyperbole?

 

An Illinois Bell Substation

 

The Hinsdale Switch

 

In May of 1988, a “wet behind the ears” newbie at TESSCO Technologies, I had a thing or two to learn about telephony.  We sold state-of-the-art wireless components to techies around the country. TESSCO was based in Maryland and had an office in Chicago at the time. All TESSCO team members were on high alert after the Hinsdale fire.  The reports talked about outages at a telephone switching station in a suburb to the city.  Not realizing that that fire was pivotal to our customers, it was time for us to take notes, listen, and learn.  Apparently the fire, which had broken out at Illinois Bell’s substation in Hinsdale had melted key fiber optic cables.  In turn the melting cables fried some computers and telephone lines, which knocked out communications to customers all over the mid-west.  “How bad could it be?”  It was very serious.  As the problems unfolded, the issues of a strategic gateway came into focus.

 

First problem: 35,000 customers had lost their analog telephone lines and all phones were out of service.  The second problem was the collateral damage – data for 50 other switching centers flowed on high-capacity optical fibers through Hinsdale connecting to AT&T’s long-distance services. With over 3.5 million calls a day routing through this greater Chicago gateway, service was disrupted to a vast swath of suburbs south and west of the US’s third largest city, impacting millions of telephone customers. As Jerry Hoppe, chief engineer of Illinois Commerce Commission commented at the time, “Ten years ago there were hubs, but there were dozens of them because you couldn’t put everything you needed to handle all of Chicago in one or two buildings.  There were certain advantages to scale, but there obviously is a disadvantage when you put all your eggs in one basket.”  [3]

 

To illustrate the problem on the ground, Holiday Inn was one of the largest chains of hotels in the world.  They had their reservation system disabled, which impacted 35,000 calls a day into their Midwest call center.  Some calls were rerouted to their Southern call center in North Carolina, but the director of Holiday Inn’s Chicago center commented, “I have been in this business for 18 years and I have never seen a failure this disastrous.”  And the group director of communications for FTD Florists was blooming-mad when he asked, “Why has so much traffic been concentrated in one switch? And why didn’t they have proper fire protection?” The FTD network’s computer in Hinsdale was cut off, which made it impossible for 12,500 florists around the nation to send flowers by wire or to process their credit card transactions. [4]

 

Businesses, from the small and insignificant to the large and substantial, were all equally impacted.  The Hinsdale fire cut a critical communications link between the FAA’s air traffic control computers at O’Hare Airport and other control centers in the region.  The cut line caused delays and cancellations of flights in Chicago and then an expanding ripple of flight problems across the country.

 

Six months before the fire, in August, 1987 the Defense Department was concerned enough about potential vulnerability of switching station to do a survey of AT&T and the regional Bell operating companies.  The Chairman of Defense Department panel warned readers in the report: “The results are scary…the network is getting thin enough in certain areas that if a guy like Qaddifi knew where they were, he could wreak havoc with the phone system.” In a public statement AT&T claimed that although it has the nation’s largest telephone network, all 14 of its centers were heavily protected and back up each other in case one were disabled.  [5]

 

The spokesperson for AT&T at the time was an executive named Joseph Nacchio, who came to the bold defense of the company – more on Joe Nacchio later.

 

TESSCO Technologies

 

Q: What does the Hinsdale fire have to do with a wireless products provider like TESSCO?

A: It can help build a redundant wireless system!

 

Calls came into our Chicago office for tower parts, cable, antennas, connectors, and wireless solutions to the Hinsdale Switch fire.  The entire technical team was on alert to lend a hand with its expertise and inventory.  The good news is that with Motorola and Andrew Corporation in the region, the transmitters/receivers and long cable runs were ready for deployment.  The tools to quickly put technicians to work, however, were in short supply, which is where TESSCO shone.

 

TESSCO is like the supply house to the prospectors of the Gold Rush.  Having all of the picks, shovels, wheelbarrows, and tools necessary to help with the mining projects, the stocking supply house is critical to a miner’s success.  So it was in the wireless business: TESSCO had all of the tools, antennas, cable, test equipment, and delivery services to ship what a wireless technician needed to install a wireless system.  With Motorola providing the backbone and other companies providing the towers, TESSCO became a key partner with the companies who were building a redundant wireless system to back stop the fiber-optic and landline systems.

 

Over the years the need for redundancy would be crucial to our national security.   Joe Nacchio had a high security clearance, so he must be trustworthy, right?  He must have earned the right to be believable to the rest of the telecom world.

 

Joe Nacchio at Qwest in it’s hayday

 

Joseph Nacchio

 

One of the threats to the communication system focused on the data centers.  Beyond backhoe’s cutting cables and mud slides causing havoc, the network hubs seemed particularly vulnerable. Joe Nacchio, the director of network engineering and operations at AT&T in Illinois, had to argue against accusations that fiber optics made the networks less reliable: “I think that perception is wrong! We have more route miles and more alternate path capabilities than we had before.”  It made sense that the expansion of fiber was progress, but did the proliferation of fiber cables lead people and processes to become too lax and casual.  Even if they were redundant, were they still “secure”?  Think MCI, Global Crossing, et al. [6] Who was watching over the network integrity?

 

Sometimes the danger is in the power networks. Other times it is with the people in power. Bernie Ebbers comes to mind.  But let’s look at Joe Nacchio, for another example.

 

Nacchio was an acclaimed expert in communications, serving on two distinguished Federal advisory panels: the Network Reliability & Interoperability Council, and the National Security Telecommunications Advisory Committee, which he chaired.  As a result of these positions Nacchio was given a top secret security clearance by the NSA and DOD in the 1990’s.  

 

In 1997 Nacchio moved from AT&T to Qwest, a relatively small telecom operator based in Denver.  Two years later Qwest planned to buy US West, which was a struggling Regional Bell Operating Company in the Rocky Mountain States. Nacchio showed his stripes, selling the SEC on his capabilities of creating a successful merger.  The Board of Qwest, elected him as their Chairman.  The regulators, who has seen the wave of consolidation before, approved the merger with little fanfare.  The name of the combined enterprises was soon changed for US West to Qwest.

 

As the company was talking to the Regulators, Nacchio believed that with his position on the Federal advisory councils, plus top secret clearance, Qwest had excellent chances of landing lucrative national security-related work from the Federal Government.  He told the investment world what a great deal the merger would be. Simultaneously, he was extolling to the Street that Qwest would hit his aggressive revenue targets. Meanwhile, with its share prices flying, Joe Nacchio was secretly selling his stock. [7] 

A few years later, however, word got out that there were some “irregularities” with the Qwest marriage to US West.  Although this was two years after the merger, the revelations of MCI and WorldCom were just emerging and people at the SEC were on high alert. On the mere hint of troubles the stock market would duck and cover.  And that is what they did when Qwest made some surprise disclosures: shares fell from a high of $41/share to a low of $2/share. In 2002 after long written and verbal denials to the contrary, Qwest finally admitted to falsifying it’s accounting documents during the merger. 

 

In 2005, Nacchio and six other former Qwest executives were sued by the SEC, which accused them of a massive $3B financial fraud between 1999 and 2002, when the company benefitted from an inflated stock price.  Apparently Chairman Nacchio sold nearly $39 million in stock between April and May 2001, when Qwest stock was trading from $41.12 and $38.31.  Nacchio was convicted of 19 counts of insider trading and sentenced to six years in federal prison.  As retribution, Nacchio was ordered to pay a $19 million fine and to forfeit $52 million in ill-gotten gains from his sales of Qwest stock. He served his sentence eight years after the crime, spending his time at a federal prison camp in Schuylkill, Pennsylvania from April, 2009 to September, 2013. [8]

 

Joe Nacchio, 2013

 

Metcalf Substation

 

Was the Metcalf sabotage a case of some bad systems or bad people?  It may be the result of both.

 

“This wasn’t an incident where Billy-Bob and Joe decided, after a few brewskis, to come in and shoot up a substation,” said Mark Johnson, retired VP of PG&E.  “This was an event that was well thought out, well planned and they targeted certain components.”  The gunmen struck in the early hours of the morning, they were prepared and they stayed hidden.  To this day no one has been charged in the case.

 

Repairs to the substation cost $15.4 million to fix. The transformers alone, which were the main targets of the gunmen, are custom-made and weigh up to 500,000 pounds. The assailants riddled their targets with bullet holes.  The transformers leaked 52,000 gallons of oil, causing them to overheat and melt down.  There are only seven companies who can manufacture these transformers, so depending on the complexity of the unit, lead times can range from a few months to years and cost $500,000 and up! [9]

 

 

Great Dangers

 

While our country is right to be concerned about cyber-attacks on our utility and communication grids, we also need to be cognizant of the imminent dangers of terrorist attacks on those same systems.  According to the Wall Street Journal, US companies have reported 13 cyber-attacks, due to hackers trying to penetrate the grid in the past three years.  There have been no reports of major outages linked to these events, although companies have generally declined to provide any details of the damage out of concern for imitators.We should be grateful we are in a sovereign country.  Terrorist organizations overseas have been linked to over 2,500 attacks on transmission lines and at least 500 substations in the decade from 1996 to 2006. [10]

 

There are always threats to public places with attractive assets.  When the price of copper went through the roof a few years ago, hundreds of cable lines were cut, stripped of their outer sheathing, and sold for the copper. When burglars go underground, they are hard to catch, but it often takes them longer to collect their booty. The Metcalf shooters, however, under the cover of darkness were not identifiable in the surveillance cameras.  To prevent a future problem PG&E now has 1,200 security cameras in place, which will trigger lights and coverage of the blind spots in the substation’s precious security configuration. [11]

 

What To Do?

 

Not to be a doomsday artist or an alarmist, but we are in trouble.  We have some serious work to do and too little time to do it.  The answer is that we need to treat our communication and power systems as worthy of protection.  That includes substations.  Their safety is important to national security. 

 

We need to expand our thinking beyond the substation, as we did after 9/11.  We need to ask the right questions: what should our most heavily protected resources be?  That means we must pull more people into the conversation.  Why not ask Joe Nacchio if he would serve on a panel to help us redress our telephony and fiber technology problems? He has paid his dues and he is a technically savvy person.  We need smart people from all walks of life to help us develop the best plans of protection. 

 

From Hinsdale to Metcalf, we need the security conversation to include all telephone lines, cable hubs, fiber optic loops, wireless transmission stations, satellites, hydroelectric, wind power, transmission hubs, internet nodes, etc.  We need to audit them for the most trustworthy people and the most robust processes we can imagine.  We need to continue to make them as hack-proof as possible.  Our country depends on it.