Previous Table of Contents Next


Select Areas of Emphasis. An evaluation must encompass the entire application, not just its major security components, because it cannot be assumed that security-relevant areas have been correctly identified. The reason for this comprehensiveness is that security deficiencies, which can occur almost anywhere, sometimes arise in unlikely places. This must be balanced against the facts that evaluation resources are usually extremely limited and that some areas (e.g., functions applicable only to nonsensitive assets) warrant less detailed coverage than do others (e.g., password management). The plan must achieve the proper blend of completeness and focused emphasis. In general, the greatest emphasis is placed on those assets, exposures, threats, and controls associated with areas of greatest expected loss or harm. In addition, less emphasis is placed on areas in which flaws are believed to be well known and understood. (Nevertheless, the existence of these flaws is addressed in the evaluation findings.)

Besides the IT auditor’s basic experience, many factors can influence the proper placement of emphasis. Problem areas might have been identified by prior certifications. Audit or evaluation findings, risk analysis findings, and violation reports can identify areas of weakness and help set priorities. Applications personnel themselves should point out weak areas.

Probably the single most difficult question during the performance of an evaluation is determining how much is enough. For most areas of an application, a basic (i.e., high-level, overview-type) evaluation is sufficient for an evaluation judgment. Some situations warrant detailed evaluations because of their sensitivity or because their fundamental security safeguards are beyond the range of a high-level review. Several criteria must be taken into consideration to determine the amount of detail needed in an evaluation. In most cases, the major criteria are application sensitivity, evaluation evidence, and control location. Other criteria are the amount of evidential detail, application size and complexity, and the amount of IT auditor experience.

Application Sensitivity. In general, the greater the sensitivity of an application or application component, the greater the desirable evaluation detail. Extremely sensitive applications, whose potential for loss is high, almost certainly require detailed evaluation. Similarly, basic evaluations should suffice for applications areas that are sensitive but not critically so and whose potential for loss is low. Between these extremes, there is great need for judgment.

Evaluation Evidence. This is a broad criterion. It includes prior evaluation findings, prior problem or violation reports (for operational reviews), and new evidence obtained during the evaluation (for both operational and developmental reviews). The first two indicate areas of past strength and weakness, suggesting the degree of evaluation detail currently needed. Evidence obtained during the evaluation might be the single most important criterion and also suggests the degree of detail. For example, the planning portion of an evaluation might reveal that the application has never addressed security and is therefore in a completely insecure state. In this case, the planning process itself could suffice for an evaluation, in which a basic evaluation is performed once the major problem areas have been resolved. A detailed evaluation is inappropriate in the face of gross or fundamental security inadequacies. A detailed evaluation could also be inappropriate if the planning process reveals application security safeguards to be highly effective and well managed. Judgment is needed here, but the objective is to minimize spending resources on applications with either highly effective or highly ineffective security safeguards. It is usually preferable to place more security testing attention on intermediate cases.

The detection of a potential problem area can necessitate an auditor’s more detailed analysis. This might be the case if the software development method has not provided adequate procedures for preventing and detecting errors. Although the implemented security functions seem acceptable, there is a need for more detailed evaluation to provide confidence that the entire implementation can be relied on.

Control Location. The issue here is the extent to which application security safeguards are internal. Several factors influencing this area include the degree to which:

  The application relies on programmed, as opposed to user, control
  Transactions are initiated externally or internally
  Transaction records are kept externally or internally

Applications in which control is external are typically evaluated at the basic level. Examples include externally controlled accounts-receivable or inventory applications, message processing applications, and automated teller applications. Applications in which control is primarily internal require a detailed evaluation. Examples include fully automated funds-disbursement and accounting applications and real-time control applications (e.g., air traffic control and automated production).

Determine Resource Allocation. The IT auditor plans the resources needed to accomplish the task (e.g., time, staff, administrative support, and technical tools) on the basis of the analysis of what must be done in the evaluation. Time estimates include not only the time required to perform the tasks but also the time needed to acquire the resources. General administrative support needs and technical tools should be defined. Other related forms of general support include copies of documents (e.g., policies and checklists), training, personnel clearances, and travel scheduling. The most difficult resource to obtain is usually personnel. Required personnel include consultants, technical writers, and couriers, in addition to security evaluators. For all resource estimates, underlying assumptions should be listed. The assumptions consider contingencies that could affect the availability of people or other resources.

Develop an Application Certification Plan. Once the analysis and resource definition have been performed, a plan for certifying the application must be drawn up and documented. This plan is usually issued by the IT auditor, who coordinates its elements with the involved parties before it is issued. The document should not be long or complex. The plan should be followed closely unless unforeseen problems arise that indicate a need to revise or modify it. The plan should schedule opportunities for such revisions or modifications. As the audit team gains experience in planning evaluations, these revisions can become less frequent.


Previous Table of Contents Next