“On September 11, we at the Federal Reserve never had to shift our Fedwire operations. On that day the opera- tions were being managed by staff at our backup facility, a procedure we undertake regularly. Fedwire was up and running at all times.”
Jamie B. Stewart, Jr.
First Vice President and Chief Operating Officer of the Federal Reserve Bank of New York remarks before the Bankers’ Association for Finance and Trade - April 25, 2002
If the Navy had this relationship with Cantor-Fitzgerald, and had made economic attacks a part of their war gam- ing drills during two prior occasions, it seems appropriate to assume that on September 11, an employee(s) of Cantor-Fitzgerald might have been tasked with running computer simulations of an economic attack on the US securities market, not knowing a real attack was underway. In the simulation, one would have a computer program that simulates an “attacking” server and a “response” server. One server simulates buys, and the other simulates sells. It does not take a lot of effort to switch simulations from targeting test database servers to real-life produc- tion database servers. However, if the war game servers are simulating banks, none of the originating bank trade data ever gets copied to a disaster recovery server (test servers are not copied to disaster recovery sites), nor does it get found on an originating bank server. Hence, under a scenario like this, one half of the clearing and settle- ment data would be missing – which is exactly what caused and led to the US securities settlement problems in the weeks that followed the attack. At least two of the traders who knew how to run those programs were present and died in the attack.
Who made that scheduling decision?
One of the many strange observations from the technology reviews of the events September 11 is that the government security trading systems never went down and trade data was never lost! Much is made on Convar’s observation that there was lost data of illegal trades, but subsequent reports suggest the data was never really lost!
“All the data and software was mirrored to Rochelle Park,” said Noviello, adding that the facility was established as a full-fledged data center, not just a recovery site, in February. Many Rochelle Park sys- tems are primary, as well as backup. A triangular architecture, with points at the WTC, Rochelle Park and London connected by DS 3 pipes, worked exactly as planned. “When one corner went down, we never stopped trading,” Kiewel said. “The system kept functioning in Europe and Asia.” ...said Novi- ello, adding, “we did not lose any data.”
Cantor Fitzgerald - Forty- Seven Hours
Edward Cone, CIO Insight and Sean Gallagher
October 29, 2001 - Copyright ©2004 Ziff Davis Media Inc. All Rights Reserved Originally appearing in eWeek
“With many key facilities for the fixed income markets located in or near the World Trade Center, the loss of communication services prevented market participants from interacting with one another and with clearing and settlement providers, such as the GSCC and the Bank of New York (BONY). Clear- ing and settlement of fixed income transactions in general was a major problem in the days and weeks after the attacks, especially since the bulk of government securities cash and repo trading occurs early in the day – meaning that significant amounts of data from the IDBs on almost a full day’s trading in this area was lost and had to be meticulously reconstructed. BONY, which was the clearing bank for many major market participants and maintained some of GSCC’s settlement accounts, had to evacuate four facilities including its primary telecommunications data center and over 8,300 staff located near the WTC. BONY conducted processing activities as part of clearing and settlement of government securities transactions at several of these facilities. The critical payment systems, such as Fedwire and CHIPS, continued to operate with minimal disruption on September 11, 2001. While both Fedwire and CHIPS (as well as some of their customers/users) were affected by communications disruptions in the aftermath of the terrorist strikes, transactions could continue being processed because the actual pro- cessing facilities are not located in lower Manhattan. Retail payment systems, including check clearing and ACH transactions, also generally fared well following the attacks and continued to operate without significant disruptions. Some check clearing functions were delayed, however, due to the grounding of air transportation. The U.S. banking system as a whole continued to function relatively smoothly on and after September 11, 2001.”
Survey and Analysis of Security Issues in the U.S. Banking and Finance Sector
What a detailed review of the technology experts review of activities that day suggests is that the primary trading systems did not go down and data was not lost because of system redundancy, mirroring, and extensive back-up centers. What did fail was the telecommunications infrastructure which prevented buyers and sellers of securities from making “subsequent” trades. Hence, the settlement and clearing data should have been “whole,” and a securities buy record and sell records should have been able to be matched in the settlement process.
There is one last element of the settlement process that was
disrupted by the attack on the WTC and it is the single most significant aspect of that disruption.