Some speculation and a vision of how to make finance safer.
What I know
- Something horrible happened
- Adequate testing did not occur
- Either there was no fire alarm for the event “hemorrhaging money” or the fire brigade was asleep
Regarding the last point, the New York Times reports — essentially — that there was no fire alarm, the fire brigade did notice, but there were no wheels on the firetruck.
What I don’t know
- I know nothing about the particular bug or the computing system
- I know nothing about Knight
If you assume that you know more facts about the situation than I do, you will almost surely be right.
What I imagine
bug size
I’ve seen the bug described as “a big bug”. Bugs are hard to measure, but my guess is that it was a tiny bug. I suspect it was the sort of bug that pretty much any programmer might create any day. It just had big, big consequences.
solutions
There is talk of reeling in high frequency trading. Certainly in terms of market-making stuffing that genie back into the bottle is more than Hercules will be able to handle.
But before instituting a fix, it is best to have isolated the bug.
causes
I hypothesize that the problem is that there is a pony express rider steering a rocketship, that it is easier to thread a camel through a needle than to promote a technical person to a position of real authority.
We humans are too stupid to be able to fully learn from the mistakes of others. One of the interview questions to get a job overseeing computer trading should be: “What is your most embarrassing bug?” If the answer is none or something minor, the candidate should be disqualified.
Someone who has had their knuckle hair singed would have:
- a seriously rigorous test suite for all code going into production
- a contingency plan if the code went berserk
A real solution
To my knowledge employment at large financial firms is all about politics. It is the Peter Principle on steroids. If they thought about it, I don’t think that the top financial people would want the airplanes that they ride around in to be built by a company that had the same promotion system.
My understanding of Boeing’s structure is that there is a parallel ladder: one side for management, one side for technical. The managers take care of the political and bureaucratic elements, the technical people are given the respect and authority to take care of the technical aspects. The managers are not bosses of their technical people, they are advocates and protectors.
Finance companies didn’t used to be technology companies. Now they are, but their structures haven’t changed to match. I’m doubtful that there’s a structure that can be successful for large companies that doesn’t have a separate technical track. If politics and technical are combined, politics is always going to win out.
Fully testing every possibility is impossible. Bugs in trading software will always occur when you are reading a market d
My hunch is that it was a minor bug that percolated down the chain. Example, they modify their real time pricing and it starts publishing some nan’s to an unchanged trading model and it goes absolutely berserk.
The insanity is that they had no automated volume, position,pnl, cutoffs.
Michael,
I agree whole-heartedly that testing is never fool-proof. I’m guessing that in this case — with zero evidence — even moderately good testing probably would have found the bug.
I have experienced, in a trading context: “We don’t need to test, let’s put it in.”
I have experienced both untested code going into production and generally hard to foresee errors due to some combination of issues. I would be very surprised if it was just one bug. I have seen too many times bugs triggering other bugs.
I think you missed the point completely.
Let us begin with the assumption you make as accurate; they should be irrelevant. Knight Capital’s losses from this should remain a more than effective self regulatory process. The question is more one of loopholes and vulnerabilities of the NYSE in not catching these transactions before $400 million in losses occurred. Next time need not occur by error; there are clear vulnerabilities under certain circumstances to financial 9/11s. The fact that commentators continue to discuss the matter within the context of error only indicates a lack of realism similar to national air defense prior to 9/11.
G,
Great comment. You’re correct that I missed the responsibility of the exchanges in this. I don’t have any direct experience with stock exchanges, do they tend to be as dismissive of technical people as brokers?
I’m not as sanguine as you about firms having big losses. I don’t mind them killing themselves, but not when they do it with an AK-57 in a crowded room.
An interesting theory is at http://www.nanex.net/aqck2/3525.html