The Feds Wake Up and Smell the Coffee: Contingency Planning, for Technology and Terrorism

Small things often trip up large organizations. That's what happened at the Los Angeles International Airport last weekend.

A common piece of computer hardware -- a network interface card -- at a U.S. customs work station malfunctioned, taking down the agency's network at the airport. The system failure, which lasted nearly eight hours, delayed the arrivals of at least 17,000 international passengers and left many travelers stranded for hours in airplanes.

A failure at a U.S. customs work station at the Los Angeles airport last weekend delayed at least 17,000 arrivals.

Network cards allow computers to communicate with one another, and most home computer users know them as Ethernet cards. Most of the time, when network interface cards feel like having a nervous breakdown, they go all the way and fail completely. This usually means that one computer goes down, but other work stations continue to function.

Customs and Border Protection, part of the Department of Homeland Security, wasn't so lucky. The network card at LAX took down the local area network when the card repeatedly began seeking attention and assistance in performing its functions, setting off a "data storm," overloading the network's efforts to manage itself.

"It was a fluke," said Ken Ritchhart, acting assistant commissioner for the CBP's office of information and technology.

Officials in the aviation industry have criticized the CBP for taking too long to fix the computer problem. Los Angeles airport and city officials have expressed frustration, and airplane passengers have complained that the agency needed a better, faster back-up plan for when computers go out.

But balancing "customer service" against national security is not easy for Customs and Border Protection. About 46,000 people move through customs lanes every hour, on average. Their names and passports are matched by customs computers against terrorism watch lists and FBI databases.

When the computers are humming, it typically takes five seconds to determine a passenger's status and a minute or so to clear him for entry. When work stations go down, customs officers get out laptops to connect into databases.

Ritchhart said work-station computers at LAX were about four years old and were scheduled for replacement next year. The cables that link work stations are about 20 years old.

Before the L.A. outage, the CBP had plans to upgrade work stations, cables and electronic components at its major sites, including New York and Miami. About $15 million has been set aside for replacing cables, switches and satellite links, and $10 million more will be spent to upgrade work stations, Ritchhart said.

For the short term, the CBP scrambled a tiger team of techies to review the agency's procedures for handling major computer outages. Officials also will review the contract with their telecommunications vendor to see whether the required response time of four hours should be cut in half.

http://www.washingtonpost.com/wp-dyn/content/article/2007/08/15
AR2007081502282.html?wpisrc=newsletter&wpisrc=newsletter