Are the Data Representative?

Are the Data Representative?

How representative are these data? Pfleeger et al. have evaluated the available data, which collectively paint a mixed picture of the security landscape.

Classification of Attack Types

Understandably, the surveys measure different things. One would hope to be able to extract similar data items from several surveys, but unfortunately that is not often the case.

For example, the Australian Computer Crime and Security Survey reported a decrease in attacks of all types, but 43 percent of CSI member organizations reported increases from 2003 to 2004. The Deloitte survey found the rate of breaches to have been the same for several years. The variation may derive from the differences in the populations surveyed: different countries, sectors, and degrees of sophistication about security matters.

Types of Respondents

Most of these surveys are convenience surveys, meaning that the respondents are self-selected and do not form a representative sample of a larger population. For convenience surveys, it is usually difficult or impossible to determine which population the results represent, making it difficult to generalize the findings. For example, how can we tell if the CSI/FBI survey respondents represent the more general population of security practitioners or users? Similarly, if, in a given survey, 500 respondents reported having experienced attacks, what does that tell us? If the 500 respondents represent 73 percent of all those who completed the survey, does the result mean that 73 percent of companies can expect to be attacked in the future? Or, since completing the questionnaire is voluntary, can we conclude only that respondents in the attacked 500 sites were more likely to respond than the thousands of others who might not have been attacked? When done properly, good surveys sample from the population so that not only can results be generalized to the larger group but also the results can be compared from year to year (because the sample represents the same population).

Comparability of Categories

There are no standards in defining, tracking, and reporting security incidents and attacks. For example, information is solicited about

"electronic attacks" (Australian Computer Crime and Security Survey)

"total number of electronic crimes or network, system, or data intrusions" and "unauthorized use of computer systems" (CSI/FBI)

"security incidents," "accidental security incidents," "malicious security incidents," and "serious security incidents" (Information Security Breaches Survey)

"any form of security breach" (Deloitte Global Security Survey)

"incidents that resulted in an unexpected or unscheduled outage of critical business systems" (Ernst and Young Global Information Security Survey)

Indeed, it is difficult to find two surveys whose results are strictly comparable. Not only are the data characterized differently, but the answers to many questions are based on opinion, interpretation, or perception, not on consistent capture and analysis of solid data.