Security Implications for the Financial Sector of Developments in Information & Communication Technologies

A CSFI/RAND Europe round-table held at Wax Chandlers' Hall, Gresham Street, London EC2, 14 February 2002

Summary:
This talk aims to provoke debate on the trends in the financial sector so that industry, and government, will be able to design policies and deploy technologies to ensure a trustworthy future environment. I first examine some failures of complex socio-technical systems, examine how ICT systems may evolve and identify some issues that will be faced by the finance sector in the next ten years.

What can we learn from history?

It is 3pm in the afternoon of Sunday October 28th 1962. In a few hours the Beatles are due to appear at Liverpool's Empire Theatre in Lime Street playing third on a bill headed by Little Richard.Three thousand miles away it is 9.00am in Moorestown radar station; a 15 story, 140 foot wide, snow white radar dome "golf ball" located on Centerton Road, easily visible from the New Jersey Turnpike. It is the middle of the early shift and the last tense day of the Cuban missile crisis. The radar detects a missile launch from Cuba and alerts the headquarters of the North American Air Defence Command (NORAD) at Colorado.

I'm sure you can imagine the urgency and confusion at the command posts across the USA as Strategic Air Command prepares to strike. The missile impact is predicted 18 miles west of Tampa, Florida in two minutes time. However no nuclear explosion is detected by the remote sensors (deployed on telegraph poles) so Tampa is telephoned to see if it still exists. It does.

What had happened? At the radar station a test tape had been inserted into the system. However there was redundancy in the system — to prevent single points of failure — so the operators checked with the radar data and this confirmed a missile. This false confirmation was due to a satellite coming into orbit over Cuba at the same time. Adding redundancy had increased our confidence in the warning and confused the operators [1].

Had Khrushchev not announced withdrawal of those missiles, later on that Sunday Kennedy's military and security advisers would have recommended, on Monday, launching the invasion that had been planned. At the time, the CIA was reporting there were no nuclear warheads on the soil of Cuba. 30 years later we learnt that there were in the order of 162, including 90 tactical warheads that would have been used against an invasion force [2].

Had the technology been more automated, as it might well be today or soon in the future, we would not be here today.

There was another NORAD training tape incident in 1979; indeed the whole history of the safety of nuclear weapons is a rich one for those, like yourselves, who own and operate complex real-time systems.

Thirty-six years later on 18th Nov 1998 a junior trader cost his employers an estimated 10 million pounds after a training exercise went disastrously wrong and he ended up taking part in an 11.5-billion-pound transaction. The trader, who worked for a German financial institution, apparently pressed the wrong buttons on his computer and caused panic on dealing floors in the City. German bond futures are predominantly traded on Eurex, a German-based electronic exchange, on which traders say it is relatively easy to enter the dealing programme instead of the training simulation programme [3].

And there are many other stories, but we don't have time for them [4].

What does this tell us? Well-designed complex system involving computers, sensors and people can fail and, more often, get into uncomfortable near-miss states. These systems have complex unpredicted dependencies and failure modes. Their complex redundancy can lead to confusion and less reliability if common mode or coincident failures occur.

It also tells us of the difficulty organisations have in learning from experience. Nearly 40 years on we still see complex systems in the finance sector defeated by confusion of modes. It also shows, perhaps from our laughter at their stupidity, hindsight bias where we underestimate the skill and expertise of those making decision when we have the benefit of knowing what the outcome was. A well known but pernicious psychological tendency when we are trying to learn from the past to cope with the future [5].

We also know from incident investigation that many failures and losses could have been prevented by the application of modest good practices and basic good management. So we must not be entirely focused on looking at new solutions. Many of the old ones will continue to be applicable and continue to be ignored; the issue is to understand how and why they are not used; to understand organisational competence and corporate memory.

Relevance to finance sector

What is the relevance to the finance sector? The systems you are responsible for are also complex, tightly coupled and interdependent and their failure could also have catastrophic economic and social consequences for the world (or at least your shareholders). The term "Normal Accidents" has been coined to describe what some see as the inevitability of their failure [6]. Even if your organisation is somehow immune to ICT system problems your supply chain and the people you insure, invest in and trade with will not be.

Future systems

If the systems are complex today then what might we expect of the future? One common concept that seems to pervade policy discussion is that the world is changing rapidly driven by political, environmental and technology developments. But you might argue that we have always faced a rapidly changing world. Isn't the fact of rapid change the one stable thing we have to plan for? Why do we think that it is changing faster than in Victorian times, that the turbulence is greater than the first half of the last century? What is new? Well what might be new is the scale of connectivity and, the tumbling cost and increasing strength of computational power. This leads to systems with tight coupling and the timescales on which events unfold become shorter.

Connectivity

People talk of the convergence of technology of communication and computation and words like pervasive and ubiquitous computing are used to describe the future. We will become dependent on applications and technologies that we didn't know we needed but that insinuate and impose themselves into our lives. People talk of the intelligent coffee mug, the electronic home, of high tech wallets of medical records that record in graphic detail our lifetime's interaction with the health services and of war based on the digitisation and automation of the battle space. Implanted devices will blur the distinctions between human and machine and we will interact effectively through speech, iris recognition and other biometrics. This interaction with the environment will be mediated by devices that somehow know and adapt to our preferences (These factors taken together are sometimes referred to as ambient intelligence). How these translate into changes in the finance sector, its shape and indeed the meaning of money itself are of course hard to predict. I would not like to join the many who had tried to make specific predictions about the future and failed. I am reminded of the (in)famous quote that in 1943, Thomas Watson, chairman of IBM, said: "I think there is a world market for maybe five computers", and in 1981 Bill Gates stated: "640 K ought to be enough for anybody" [7].

Computing Power

Moore's Law — that computing power of chips doubles every 18 months — shows every sign of continuing and costs per teraflop or terabyte continue to fall and bandwidth continues to increase. The delivery of this computing power may also radically change: this is the GRID[8]. By analogy with the electricity supply this is a vision where we just plug into a socket in the wall and negotiate how many megaflops we want, when and with what level of dependability. This would then be delivered by all the idle machines sitting on the www. It could be seen as a natural evolution of the present trend of outsourcing IT that itself can lead to unsatisfactory concentrations of risk

This may not be something that you see as remotely sensible for business critical applications but it may be that the price is compelling or that there is some killer application that can be accomplished with such an enormous, cheaper source of computing power, or you may have no choice. Although there will be risks of engagement there will equally be risks of not taking up the technology. So we need to understand these technology risks, we need to shape the GRID so that it delivers levels of trustworthiness that we understand and can factor into the business decision making. (The GRID may also lead to new markets where people buy futures in tera-flops and trade this as with any commodity.)

This illustrates an important message: security is just one attribute of interest: availability, reliability — and all the other 'ililites — as well as robustness, performance may be as or more important. So security has to be traded-off against other factors. There is no such thing as absolute security and if there was you couldn't afford it or your shareholders would not welcome it. The most secure bank, like the safest nuclear reactor, may be one that is shut down.

Furthermore the techniques deployed to address the attributes and the solutions they produce are interdependent and we need a system perspective to trade off and understand what we are doing. A useful term for these over-arching concerns is dependability; and so I would recommend that you consider the dependability of your ICT systems, not just their security.

Yet I don't think this is broad enough. It is not just, or even, a technology issue but one we must understand and appraise in the wider context, we must not forget the operator and user. You are probably familiar with the trend of not blaming the pilot in aviation safety and you may have been following the House of Lords inquiry into the Chinook accident, which reported last week [9]. Similarly in your institutions we see that a view that blamed the new trainee or the rogue dealer as the cause of the incident is too simplistic. We need to consider not just the individual interaction but the behaviour of the group and the culture of the organisation, institute and industry within which it is situated.

So changes in technologies mean we have to tackle system level issues; tackle principles that were before tacit or not adequately addressed. It reminds me of the late 80's early 90's when I was involved with developing new and radical standards for the MoD for safety critical software. It was soon apparent that there were not sufficient equipment standards, or system standards and that policy and institutional matters needed addressing as well.

Consequences of failure ? risks, threats and vulnerabilities

In parallel and facilitated by greater connectivity and interdependence the threats (or at least perceived threats) to the state, to institutions and to our own personal liberties will be increasing in intensity and in sophistication. The capacity for these systems to interact and surprise us is surely increasing.

This view of the future is not fanciful speculation but a future that is being planned for us now — some of my best friends are working on it — and it is arriving faster than we think.

The future, particularly the wired-up on-line future, is often presented to us in terms of success, of the wealth creation and lifestyles that will accompany it of what it might be like if it all works. But there is another view, Murphy's view [10] ? that if something can go wrong it will.

If it goes wrong, will it be a fun place to get to? A bleak and fearful future full of gadgets we don't want, that don't work. Populated by glum geeks whose latest gizmo is "off-line" and terrified of those about to attack using stunning and unanticipated methods. A surveillance society where every move, word and deed is recorded and monitored somewhere either for entertainment or control. Where reality is defined and adjusted on-line. 1984 meets 2001. Big brother meets big brother.

Viewed incrementally we make small steps into the future, each step can be justified on cost/benefit or moral grounds, but the total is more than the sum of the parts. After a while we arrive at a completely different landscape and don't know how to get back. We get emergent behaviour of systems; complex non-linear behaviours. We cannot just retrace out steps, we may have to follow the system trajectory; the breaking of the wave in catastrophe theory.

If it doesn't work who will get the blame? Banks as the Railtrack of the future? We need to understand how trust is won and lost and take notice of the work on what is called the social amplification of risk [13]. In judging risk we need to be aware of the biases; the impact of social distance ? the so-called uncertainty trough, the fact that our choices of what to worry about are socially constructed [11],[12].

So what is my list of solutions, what are the slogans and sound bites? Well this talk is meant to galvanise discussion not provide answers. I have a shopping list of ideas that I could share with you later. But there is one overarching principle ? there are no easy answers. Fred Brookes, a distinguished IBM scientist, popularised the phrase "there is no silver bullet" when discussing the problems of software projects over 20 yrs ago [14]. There will be no one-size-fits all solution but we be able to extract enduring principles.

One thing is that we will need more understanding, more thought. So corporate intellectual capital in this area will be vital and I think we need to find a way of valuing this that is not so off the balance sheet.

Conclusion

What can we conclude, what is my message?

The future will see a continuing increase in connectivity, bandwidth and computational power. The security of ICT systems is extremely important and too important to be addressed in isolation. To deal with it we need to address the dependability of ICT systems and that this is not just, or even, a technical issue. We need to consider the individual, group and cultures involved and take a proper systems view. In taking this more holistic approach there is much work that we can draw on. We need a cross-sector, inter-disciplinary approach. A first step in tackling the future is to learn from the past, but that, as we all know, is far from easy.

References

[1] Scott D Sagan, The Limits of Safety: organisations, accidents and nuclear weapons, Princeton, 1993

[2] A transcript of MacNamara's talk http://www.bbc.co.uk/worldservice/people/features/mycentury/transcript/wk39d4.shtml

[3] As reported in the Electronic Telegraph http://www.telegraph.co.uk:90/

[4] For example "Creek Web Site Snafu Causes Alarm in P.A." by Elaine Goodman, Palo Alto Daily News, 12 Feb 1999, in the Risks Digest, http://catless.ncl.ac.uk/Risks.

[5] A Tversky and D Kahneman, "Judgement under uncertainty: Heuristics and biases". In Judgement and Decision Making (an interdisciplinary reader), edited by Hal R Arkes and Kenneth R Hammond, CUP, 1986

[6] C Perrow, Normal accidents: living with high-risk technologies, New York, Basic Books, 1984.

[7] ERCIM, The Challenge of the Next Century, see http://www.ercim.org/publication/Ercim_News/enw39/fp39.html

[8] I. Foster, C. Kesselman, and S. Tuecke. The anatomy of the grid: Enabling scalable virtual organizations. Intl. J. Supercomputer Applications, 2001. Available from http://www.globus.org/research/papers/anatomy.pdf.

[9] Chinook report is at http://www.publications.parliament.uk/pa/ld200102/ldselect/ldchin/25/2501.htm

[10] See R Needham, Newcastle Dependability Workshop 2002.

[11] D. MacKenzie, Inventing Accuracy: A Historical Sociology of Nuclear Missile Guidance, MIT Press, 1990.

[12] Mary Douglas and Aaron Wildavsky, Risk and Culture: An Essay on the Selection of Technological and Environmental Dangers, University of California Press, 1982

[13] Kasperson, R, O. Renn, P. Slovic, H. Brown, J. Emel, R. Gobie, J. Kasperson and S. Ratick, 1988. The social amplification of risk: A conceptual framework. Risk Analysis, vol 8, no. 2, pp177-187 see also J Petts, T Horlick-Jones,G Murdock, Social amplification of risk: the media and the public, HSE Research Report CR 329/2001, ISBN 0 7176 1983 4, HMSO 2001

[14] J Brooks, No Silver Bullet, Essence and Accidents of Software Engineering, Computer Vol20, No 4, April 1987

Robin Bloomfield is a founder of the specialist consultancy Adelard and Professor of System and Software Dependability at the City University, London. He has worked for a wide range of clients within government and the privates sector addressing both policy and technical issues associated with the achievement and evaluation of the dependability of complex systems. He is Liaison Director for the Dependability Interdisciplinary Research Collaboration (DIRC).

©Adelard and CSR City University 2002