Working In Uncertainty

What Jurassic Park tells us about uncertainty suppression

You probably know that the story of Jurassic Park is about a theme park with genetically reconstructed dinosaurs that goes horribly wrong. Although the film brought Jurassic Park to the attention of millions of people, the original book by Michael Crichton contains many fascinating details and powerful messages. What drew me into the book was not the page turning danger and suspense, but the realistic way things go wrong.

The park starts as the vision of one man, John Hammond (played by Richard Attenborough in the film). He imagines the park as something that will have the world's children looking on in wonder. To be specific, he is interested in the world's rich children, because the other part of his vision is to make billions of dollars a year.

It starts with salesmanship

With the help of lawyer and financier, Donald Gennaro, he sets about raising the money he needs. Hammond sees nothing wrong in holding back awkward facts. No need to mention, for example, that the geneticist whose work he is relying on has terminal cancer. Not helpful to point out that the miniature elephant he carries in a bird cage as a prop for his fund raising speeches is not, in fact, genetically modified, is prone to illness, and has an unnaturally vicious temperament. It's not lying, after all. It's just good presentation.

Although the venture is highly speculative Hammond has a knack for presenting it as a logical next step, a sure thing. To play down the risk involved he stresses that everything proposed has been done before. It's not the first zoo in the world, not the first theme park, not the first time animals have been cloned, and so on.

This reminded me of the Eurotunnel prospectus of 1987, which told investors that 10% should be allowed for unforeseen costs of constructing a tunnel under the English Channel. The justification was that "Whilst the undertaking of a tunnelling project of this nature necessarily involves certain construction risks, the techniques to be used are well proven." In fact the tunnel cost twice as much as estimated and a massive financial rescue was needed to finish the job.

Next comes pressure

Hammond recruits some key people for his team. Henry Wu is the most promising student of the dying geneticist so Hammond persuades him to join in, promising huge funding, minimal bureaucracy, and the chance to accomplish something extraordinary. Hammond also recruits a leading theme park engineer, John Arnold.

Although both these men share Hammond's vision and believe that, fundamentally, the park is safe and will succeed, they each raise some worries with their boss, John Hammond. His reaction is to use any tactic to stop them considering outcomes other than the success he envisions.

When Arnold lists problems with the park's systems and, in particular, with the behaviour of the animals, Hammond uses a series of tactics to keep him tied down. First he tries to paint Arnold as a negative thinker. He says "You're just a worrier." then "It's not as bad as all that." When the alarming behaviour of the velociraptors is mentioned Hammond does his best to shut the conversation down. He says "Let's not start on the velociraptors. I'm sick of hearing about the velociraptors." Finally, when Arnold suggests that the park is inherently hazardous (which it obviously is) Hammond starts to bully, and questions Arnold's loyalty saying "Oh balls. Whose side are you on, anyway?"

Every day, in thousands of business meetings around the world, these same tactics are used in less exotic projects to achieve the same ends.

Then technical over-confidence

Although Wu and Arnold raise technical challenges with Hammond, they each believe that the park is safe. They believe this because they are over-confident of their own inventions, as we all tend to be.

Henry Wu believes the animals cannot leave the park. First of all, it's on an island 100 miles off Costa Rica. But as an extra precaution he has engineered them to be unable to manufacture the lysine they need. To stay alive the dinosaurs need to eat the special lysine-rich food given to them in Jurassic Park.

Wu believes the dinosaurs cannot breed. Their gonads are irradiated to sterilise them. And then of course there's the fact that they are all female. Breeding is not possible.

But Wu is wrong. Dinosaurs do escape to the mainland where they live for a long period and even migrate. It seems that lysine is found in some natural foods and the dinosaurs soon learn what makes them feel better. They cross the sea by stowing away on the supply ship that visits every two weeks.

The dinosaurs also breed. It turns out that irradiation is not reliable and that some species have the ability to change sex during their life time. This ability is not a bizarre dinosaur trait but something found in certain frogs. Wu reconstructed the dinosaur's DNA, which involves a great deal of inference, carried out automatically by supercomputers. In effect the reconstructed DNA is a series of educated guesses combined with fact. Some of the educated guesses were based on the fact that living things more complex than a bacterium have a lot of DNA that is the same. The computers fill in some gaps using fragments of frog DNA, and it is this DNA that carries the sex-changing ability.

This is another example of faulty reasoning about risk. Although each inference made by the computers is a very strong one and the individual chances of error are small, the combined effect of so many inferences is a high likelihood of errors. Indeed, Henry is aware of "bugs" in his DNA, which he thinks he can iron out in future versions, but it doesn't occur to him that a bug might affect his safety precautions.

By the time the breeding is discovered (by visiting experts) there are some seven nesting sites and over 50 extra dinosaurs on the island alone.

None of this would matter if John Arnold's security systems were as effective as he imagines them to be. Obviously all the pens have high electrified fences. In theory the dinosaurs should learn to avoid the fences, so even if the power goes down for a while none should escape. The risk of a short circuit is reduced because trees that might be blown onto them are held steady by insulated guy ropes.

More impressive still is the computerised surveillance system. Motion sensors and cameras cover 92% of the island and every 15 minutes the computer searches for the dinosaurs of each species, recognising them by their appearance, and shows the number expected and the number found. Arnold knows this works because very occasionally there have been differences between the two counts, such as when a dinosaur has died.

The flaw is that the computer searches for the number of dinosaurs it has been told to expect. When it finds that many it stops counting. That, combined with a computer bug that prevents them from measuring exactly how much food the animals are eating, is why breeding goes unnoticed.

Also, the 8% of the island that is not covered by cameras and motion sensors is largely connected, creating secret corridors along which the dinosaurs can roam. Also the 8% does not include the maintenance tunnels, which some small animals discover by burrowing. There are velociraptor families playing on the beach!

Michael Crichton introduces all these details beautifully and my first thought on reading about the security precautions was that they did indeed sound impressive and reliable. But, as the flaws were revealed I found them totally realistic. Years as an auditor have taught me to be suspicious, even when I'm impressed.

Forgetting the possibility of deliberate attacks

Designing controls against deliberate attack is much more difficult than just guarding against accident. When attacks are deliberate you can expect them to be against your weakest point and to be much more complex than simple mistakes.

In Jurassic Park the possibility of deliberate attempts to break security is not recognised. First, the intelligence of the dinosaurs is underestimated. We often think of dinosaurs as being stupid. The 9 metre long Stegosaurus, with a brain the size of a walnut is the classic example. However, dinosaurs were as varied then as mammals are today. It is quite possible that some would have been intelligent pack hunters. Then there's the fact that they are motivated to escape from their pens. Some just want to get to the pens where their prey are living. Others want to migrate. Instead of shying away from the fence, they continue to try their luck, so when the electricity is switched off they soon push through.

And why is the electricity switched off? That's caused by another deliberate act, this time by a human being. Dennis Nedry, the sleazy computer programmer (Nedry is an anagram of Nerdy), switches off all the security systems so that he can steal some frozen embryos for a rival biotech company.

Nedry is yet another person who thinks his plan cannot go wrong. He has every detail of his plan worked out - except that in a tropical storm he misses a turn off and ends up as prey for a dilophosaur.

Blindness and denial

Along the way there are plenty of clues that should have shown the Jurassic Park team that things were getting out of control. They either don't notice them or do not act. This is typical human behaviour. Once we're committed and have set our minds in a particular direction it is easy to slide into denial.

When three construction workers are killed by raptors Hammond and the others decide it is best to cover things up and say they were killed in an accident involving earth moving equipment. They ignore evidence that small, venomous dinosaurs have reached the mainland and begun killing babies and old people.

When they look at the height profile of procompognathus dinosaurs they see what they expect to see, a bell curve. But, hang on, the dinosaurs were hatched in three batches so surely there should be three peaks in the curve, not one? In my work as an auditor I have often seen clear numerical evidence ignored in this way.

When Henry Wu suggests to John Hammond that the outside world might try to limit their operations, Hammond replies "I can't see how." This is a good example of the dangers of the "availability heuristic". We tend to judge the likelihood of something by how easy it is to call to mind. If we can't imagine how something might happen we think it is unlikely. Hammond can't see how awkward things might happen because he rarely allows himself to think about them, and because he does not want to think about them.

Lack of imagination is no guide to the future.

As the storm rages outside, the security systems are down, the phones have stopped working, and contact has been lost with his grandchildren, stranded in the park, John Hammond sits eating ice cream, and worrying that at the age of 77 he may not live to see the park fully open. He says he is sure the children will return safely and refuses to take any action.

The moral of the story

Michael Crichton's message is that genetic manipulation is dangerous and certainly not worth the risk for mere entertainment. Life is not so easily controlled, especially when it doesn't want to be.

For me it is also a warning about the dangers of an exciting vision, and all the other social and cognitive behaviours that go with uncertainty suppression. It shows how these can stop us from seeing other things that might happen, and even from seeing what is happening now. Gradually, people lose touch, lose flexibility, and run out of options.

Made in England