Appendix I

Analytical Approaches for Comparing Test Protocols

Existing armor has proven effective on the battlefield. That is, while statistical rigor was lacking in the Army’s original FAT and LAT protocols, there is no known evidence that substandard body armor has been sent to the field. Thus, it is not unreasonable to assume that the manufacturers and their lots of body armor that passed the Army’s original first article testing (FAT) and lot acceptance testing (LAT) produced body armor that met the required (or at least necessary) performance standards.

Of course, it is possible that, given the lack of statistical rigor, substandard body armor passed both FAT and LAT or, conversely, fully acceptable body armor failed either FAT or LAT. It is also possible that body armor has failed in the field and the evidence of such failure has been lost, or that substandard body armor that inadvertently passed FAT and/or LAT simply has not been put to the ultimate test. These outcomes are all impossible to determine.

ASSESSING MANUFACTURER RISK

In spite of these unknowns, one way to assess the impact of the new Office of the Director, Operational Test and Evaluation (DOT&E) protocols for body armor on manufacturers is to use historical test data for body armor that passed earlier FAT and LAT tests. The idea is to draw on actual historical test data to gain some insight into how manufacturers would fare under the new DOT&E protocol. Such an analysis is based on the following assumption:

All body armor that successfully passed the Army’s original FAT and LAT protocols was, in fact, fit for use in the field.

Analytical Approach

Given that the foregoing assumption is correct, the effect of the DOT&E protocol on manufacturers can be assessed as follows:

• Simulate a test under the new DOT&E protocol by randomly drawing (with replacement) from an appropriate pool of historical test data (at a minimum, by manufacturer and type of plate) using only data from passed tests.

• Given the pseudo data, determine whether the manufacturer (lot) would have passed or failed the FAT (LAT).
• Repeat steps 1 and 2 as necessary and appropriate to estimate rate of pass/fail for a given manufacturer/type of plate combination.
• Compare and contrast the estimated rates to the historical passing rates to evaluate whether the new DOT&E protocol is likely to result in higher failure rates and, if so, under what conditions and by what magnitude.

This type of simulation cannot determine whether the new DOT&E protocols will decrease the government’s risk of buying substandard body armor. What it can do is to assess, from the manufacturers’ perspective, whether the new DOT&E protocol will result in higher FAT/LAT failure rates for body armor that probably would have passed under the previous Army protocol.

Discussion

From a manufacturers’ economic risk perspective, this is a very reasonable comparison since, from that perspective, the issue is not how the armor performs in the field but whether the new DOT&E protocol increases test failure rates (and thus costs) for existing products and processes.

Note that this is essentially a nonparametric approach to evaluating manufacturer risk. That is, by using actual historical data drawn from passed tests, one does not have to make any parametric assumptions about the distribution of backface deformation (BFD), nor estimate the probability of penetration, nor try to model whether and how the two measures are jointly distributed.

However, this approach does depend on having sufficient historical data from which to resample. If insufficient data are available, then a parametric approach may be taken in which distributions are fit to the BFD and penetration data, and those distributions are used to simulate future data.

COMPARING PROTOCOLS

Another way to assess the impact of the new DOT&E protocols also uses historical test data for body armor. This approach can be used to compare two protocols. Instead of considering only manufacturer and design combinations that passed the historic protocols, consider a representative range of manufacturer and design results. For illustration, the committee compares the new DOT&E protocol and a historic Army FAT protocol.

Analytical Approach

• Using historical data, simulate enough test data to satisfy the protocol with the largest sample size. This can be done either nonparametrically (by randomly drawing with replacement from an appropriate pool of historical test data) or parametrically (by using summary statistics from previous tests).
• Note that the simulations may require several summary statistics (or appropriate data to resample), including probability of first and second shot complete and partial penetrations by shot order and first and second shot BFD by shot order. Depending on the criteria for each protocol, data may be simulated (for example, data on partial penetrations) that will be used to assess only one protocol.
• If the two protocols require a different sample size, randomly select the smaller sample size from the simulated test data.
• Compute whether or not the test would have been passed or failed under each protocol.
• Repeat the simulation as necessary and appropriate to estimate the four pairs of probabilities: (1) pass under protocol 1/pass under protocol 2, (2) pass under protocol 1/fail under protocol 2, (3) fail under protocol 1/pass under protocol 2, or (4) fail under protocol 1/fail under protocol 2.
• Repeat the simulation for a range of representative manufacturer and design results.

Discussion

This type of analysis may point to specific design characteristics that are advantaged or disadvantaged by particular protocols.