Small Is Not Separate
Knight Capital received 97 error emails before the market opened, but none sent anyone to check the servers.

On August 1, 2012, Knight Capital's order router took in 212 small retail orders. In about 45 minutes, it sent millions of orders into the market. That led to over 4 million executions in 154 stocks, with more than 397 million shares traded.
The orders reached Power Peg, old code for handling orders that Knight had stopped using but never removed. A new release reused a flag, a switch that had once turned that old code on.
The new release was meant to replace Power Peg across the router's eight servers. But it was never copied to one server, so the reused switch still turned on Power Peg there.
No second technician reviewed the deployment, so nobody noticed that the server still held the old code instead of the new release.
Before the market opened, an internal system sent 97 automated emails reporting the error "Power Peg disabled". The first went out at about 8:01 a.m. Nobody acted on them before the 9:30 a.m. open.
In 2005, Knight moved the function in Power Peg that tracked the running total of shares. It did not retest whether Power Peg would still work if called.
Each gap on its own
One of eight servers missed the new release: you can see why that looks small without excusing what it became.
Knight stopped using Power Peg in 2003. The new software began rolling out in stages on July 27, 2012. One server did not receive it. The SEC's order records no second-technician review and no written procedure requiring one. Other groups at Knight had written procedures.
The emails were not designed as system alerts. Personnel generally did not review them when they arrived. A message existed, but the process did not treat it as a warning requiring action.
A second pair of eyes could have caught the missed server. That is not wishful thinking after the loss. The SEC says a deployment double-check could have averted the incident. It also says integrating those messages into system monitoring could have prevented it.
Either safeguard could have been enough. Neither was in place.
By the time Knight stopped sending orders, it was net long about $3.5 billion in 80 stocks. It was also net short about $3.15 billion in 74 stocks.
Knight estimated its realized pre-tax loss at $440 million on August 2, 2012. The SEC's later figure was more than $460 million.
The cost crossed the boundary
My read is that each gap imposed its cost beyond the team that held it. The old code became a deployment problem. The deployment problem became a monitoring problem. With neither check in place, the cost reached the whole firm.
From inside one department, those can look like different kinds of work. Removing unused code differs from confirming a deployment. Reviewing messages differs from managing trading exposure.
I use the extra chair, the telephone, the lag and the structural zombie to trace costs that can look harmless where they begin. The extra chair is a new person, such as an extra reviewer, who brings a possible conversation with everyone already there. The telephone is a change that never becomes shared, checked knowledge, as with the server that missed the release. The lag is a warning that arrives in time and changes no decision, as with the 97 emails. The structural zombie is old work that outlived its use and can still act, as with Power Peg.
Power Peg was a structural zombie: unused work that still had the power to act. The missing deployment confirmation left a telephone problem. What had changed on the servers had not become shared, checked knowledge.
Power Peg was a structural zombie: unused work that still had the power to act.

Click a numbered dot to see what it stands for. Start with 1.
The lag was in the emails: a warning that arrived in time and changed no decision. The emails arrived before the open, while there was still time to act. Information flow depends on what a message changes, not just whether it arrives. Here, 97 emails arrived, and none of them sent anyone to check the servers.
I see the gaps as hiding one another until they met. The SEC's order says a deployment double-check could have found the missed server. It also says monitoring the automated emails could have prevented the events. To me, each missing check left the next gap harder to see.
Hiring into the same gaps
Knight was a trading firm, not a Series B startup. Its incident does not tell us how often funded companies fail this way.
The connection I see is structural. Series B success brings pressure to grow, and growth means hiring. The habits that helped you get funded can remain in place while the number of people depending on them changes. One such habit is an engineer shipping to production alone, because with five people everyone knows what changed.
The possible connections grow faster than the headcount. With 50 people, there are 1,225 possible pairs. With 150, there are 11,175. Three times the people means about nine times the pairs.
Three times the people means about nine times the pairs.

Click a numbered dot to see what it stands for. Start with 1.
That does not mean everyone must speak to everyone else. With more people, more handoffs must carry what each person knows to someone else.
An extra reviewer could catch a missed server. But that reviewer also needs current information about the code and the deployment. Adding the role does not settle how the information reaches it.
That is the pattern I worry about when hiring outruns structure. Each team adds people to address its own workload. The connections between teams still depend on habits formed when fewer people were involved.
Old work remains reachable. Updates need confirmation across more people. Warnings have farther to travel before they reach someone able to act.
A team can defend each addition by pointing to real work. Across the company, each addition can increase the work of keeping everyone aligned. A new reviewer of a deployment needs current word from the person who deployed it on what changed. Hiring faster than you remove obsolete work leaves more people depending on it.
Knight announced $400 million in equity financing on August 6, 2012, five days after the incident. Its merger with GETCO was announced on December 19, 2012, and completed on July 1, 2013.
The trading lasted 45 minutes. The consequences lasted far longer.
In your company, the old work may have another name. So may the missed update, the absent check and the warning that arrives without changing a decision. Each can remain small within the department that holds it.
Small is not the same as separate.
I'm writing down where these chains start as I find them.