One of the most valuable things about you has been your willingness to ask the most naive possible questions in a highly penetrating way.

avoiding 'group think'

Had the banks identified the potential for 'sub prime' borrowers to default on their mortgages?  Of course.  Had most banks correctly assessed and acted on this risk (and/or had their regulators spotted that they had failed to do so)?  Clearly not.

The reason lies in the terrible power of 'Group Think' - intelligent people who develop a collective mind-set that things are other than as they are.  This is by no means unique to financial circles - anyone who has worked in the scientific world is familiar with the casual conjecture made by a conference speaker being cited by a couple of articles, which in turn get cited by a few more, until that unsupported conjecture has become a generally acknowledged truth.

Group Think is a big problem when dealing with risk and uncertainty, for a number of reasons including

Absence of Disproof

This one is fairly obvious - when dealing with risk and uncertainty, 'Group Think' can be stubbornly persistent because hard evidence to disprove the cherished (but wrong) assessment of risk is not available - or, as in the sub-prime mortgages example, becomes apparent too late to prevent a major loss.

pressure on experts

One of the first responses of a decision maker faced with an important uncertainty is to call in an expert for advice.  A common expectation is that the expert will resolve the uncertainty.  If one expert fails to do so then perhaps we'll ask a few more to form a committee and come up with a shared opinion.  This can work very well BUT can also place great pressure on those involved to understate uncertainty in the interests of helping the decision maker make an easier choice. 

We came across one study where, to calibrate 'expert judgments', scientists at a US national laboratory were asked to estimate the distance from the canteen to the site gate (something very familiar to all of them), in terms of

Half of all the bands of CERTAINTY did not include the actual correct answer!  In this case that didn't particularly matter - but this illustrates a very important point, that people may feel uncomfortable stating that uncertainty is large.  This is especially true when the issue is something on which the person is considered expert, and when "highly uncertain" is not what the listener wants to hear.

communication

What one person hears may not be the same as what the other one said.  In one case in our experience, the experts had agreed that the most likely outcome of a future event (which was certain to happen at some point) was at the worst end of the spectrum of possible consequences.  This was then reported to a senior civil servant, and thence to Ministers.  The responsible Minister was briefed that because it was a worst case it could be discounted as very unlikely - an interesting case of 'Chinese whispers' leading to a pronounced misconception.

expressing ignorance as probabilities

This can be quite subtle.  Many risk assessments aim to calculate a probability or likelihood of a given event occurring.  One way of dealing with uncertainties along the way is to carry out multiple calculations using different values of key input parameters.  This provides a valuable indication of the sensitivity of answers to the inputs.  But we can go a step further and associated a subjective probability (in effect a degree of belief) with each of the inputs, and then propagate this through to estimate the corresponding probability of getting different answers.

This can be very helpful in exploring what range of answers is associated with what range or strength of assumptions about inputs.  But it can also have the effect of translating total ignorance into something that looks like knowledge.  "We don't know whether A or B is correct" can be turned into "Here's the risk, based on a 50% chance of it being A and a 50% chance of it being B".  When several such assumptions get buried in a complex calculation it can be very difficult to disentangle what is actually known and what is the true extent of uncertainty - and it is very easy to combine such subjective probabilities in a way that understates uncertainty in the ultimate answer.

Avoiding the pitfalls

The solutions to these problems are straightforward in principle but can be quite hard to implement in practice:

Perhaps most importantly of all, seek advice from people who are willing and able to challenge the established 'givens' within your walk of life.  This is an area where we feel particularly well qualified to help - we pride ourselves in our ability to spot a flawed risk assessment at 1000 paces, rather than in our knowledge of any particular industry or business domain.