Could last week's glitch-aided Wall Street collapse happen again? You had to ask?

Think the whole glitch enchildada is so last week? Eyeball this, Dude.

A series of computer trading snags last week provided a not-so-subtle reminder to Wall Street and individual investors that technology can be a fussy mistress, even for systems designed for critical uptime.

A heavy day of stock trading on Feb. 27 resulted in clogged networks, exacerbated by a computing error that made it appear that the Dow Jones industrial average had dropped 178 points in one minute. Systems slowed to a crawl at the New York Stock Exchange and many brokerage Web sites, especially as additional trading kicked in. Web performance monitoring company Keynote Systems noted slowdowns and unsuccessful transactions on trading sites beginning at 1:30 p.m. EST. Systems that usually process an order in seconds were taking up to a minute and sometimes longer. At the height of the problem, Keynote saw a 25% drop in the number of successful online trades. The sites of Charles Schwab, Fidelity First, Muriel Siebert, ShareBuilder, TD Ameritrade, and Wells Fargo were affected.

Soon after Keynote noticed the slowdown, the Dow Jones system that calculates the industrial average stopped receiving the underlying stock price information in a timely manner, which led to a pricing backlog that took 70 minutes to be recognized and switched over to a redundant system. The new system calculated all the queued data at once, shocking brokers, who watched as the Dow appeared to drop 178 points in one minute and 240 points in three minutes.

"There's always been this question: Is this market ready to handle spikes of trading volume as electronic trading increases?" says Sang Lee, managing partner of Aite Group, a financial consulting firm. Dow Jones didn't have the appropriate trigger mechanisms to figure out things were going down, Lee says, nor the appropriate notification mechanisms.

But that wasn't the end of it. Large swings in the market trigger computerized algorithmic trading from any number of fund managers, spitting out reams of orders at once. The NYSE's systems are designed to handle three to four times the normal message traffic. However, traffic was already up before the Dow trigger (thanks largely to market activity in China and comments by former Fed Chairman Alan Greenspan), and the stocks hit immediately by algorithmic trading were isolated rather than spread out across the market.

That created a backlog of requests at the NYSE's Designated Order Turnaround system, which routes orders to trading floor specialists, causing further chaos by keeping orders in a queue or routing them to different exchanges. Late in the day, the NYSE instituted trading curbs and even suspended electronic trading altogether. For the day, the Dow was down 416 points, or 3.3%. "The floor was a disaster," says Steve Swanson, CEO of trading technology and brokerage firm Automated Trading Desk. "I thought the Street was in pretty good shape, but I was very disappointed by what I saw." Nasdaq, Wall Street's largest fully electronic stock market, experienced only sporadic slowdowns.

Last week's problems resonate even louder as this week brings a crucial deadline in the adoption of Reg NMS, a set of new Securities and Exchange Commission rules. Those rules, when fully implemented in October, will force investment houses to analyze and execute orders for the best price across all exchanges. This week, the system for communicating those best prices goes live.

The NYSE moved to expansive e-trading only within the last year through its Hybrid System, a combination of electronic and floor-based trading. Swanson has participated in tests of the Reg NMS systems, but says the NYSE wasn't part of them, though it has done its own tests. Unlike preparations for the Y2K bug, the SEC hasn't mandated testing for Reg NMS. "I'm shocked that people think Monday's going to go well," he says, "and God forbid something like that happens on Monday, there's going to be a massive failure."

http://www.informationweek.com/software/showArticle.jhtml;jsessionid=GRNVJIVJTX4LGQSNDLPSKHSCJUNN2JVN?articleID=197700918