Reliability: .02 Agreement

Tip 49: It is very possible to have 02 agreement (20 mg/100mls agreement) of results, yet the results are not accurate or not reliable. It is important to remember that the BAC readings on an approved instrument are measurement results. The calibration check readings that are read on an approved instrument are also measurement results. Whether one is considering 02 agreement between two BAC readings (20 mg/100 mls truncated) or ± 10 mg/100mls of target value agreement between two (or three) cal. check readings, symmetry of results may be deceptive.

Consider the beauty, symmetry, and precision of the design and construction of these arches. Close comparison of the arches reveals that they are exactly parallel and symmetrical.

If you look at the results in the context of a series of more than two arches the symmetry is even more striking.

However, notwithstanding beautiful design in a building, or symmetry in breath testing results, a poor foundation can result in inaccuracy. It isn't enough to simply assume that symmetry / precision imply accuracy. Architecture and breath measurement results need to be assessed in their full scientific context, if the results are ultimately to be considered reliable.

The underlying foundation of a building or the foundation of an approved breath instrument, that is, its calibration, may also shift over time. Shifts in the calibration of an instrument that happen over time are called "drift". No one builds a building or an Intoxilyzer® with the intention that it will not be vertically true or evidentially true. The designers of the building, the builders of the tower, the manufacturer of the instrument, and the Alcohol Test Committee may all have performed their duties well, but the foundation may have shifted over time. There may have been a laissez faire approach to maintenance, quality assurance, or quality control at some point in the process. An assessment of reliability at the time of subject tests analysis, requires full appreciation of a whole system of alcohol and other standards that are themselves reliable. Symmetry is not helpful in assessing reliability unless the alcohol and other standards are traceable to higher known and reliable standards.

As with the beautiful bell tower at the church in Pisa, apparently normal and symmetrical results do not necessarily imply accuracy and reliability in relation to the real external world. Use of external standards (including a plumb line) always proceeds from known to unknown, not the other way around. You don't use symmetry in a building to establish what is plumb and you don't use an Intoxilyzer® to establish the reliabilty of the simulator or the alcohol standard.

Significant errors can be repeated 17 minutes apart. Similar errors can be caused by the same instrumental deviation, or physiological or environmental presentation. Examples include:

1. bad linearity in an aging/drifting instrument run twice

2. mouth alcohol or chemical interferent bias twice

3. radio frequency interference/bias from the same mobile phone or repeater twice

The proper way to minimize the possibility of such errors is to establish and follow rules (see OIML R 126) respecting:

1. metrological supervision by a metrological authority (e.g. Measurement Canada and the Fairness at the Pumps Act)

2. metrological control

3. traceability of all measurement results to international SI units

   1. through certificates of calibration and re-calibration

   2. transparent reference standards used during calibration or re-calibration

4. evaluation and type approval (which we have in Canada)

5. initial verification

6. mandatory periodic inspection/verification/re-calibration

7. verification after repair

8. training of personnel

9. use of an alcohol standard control in the field (which we have in Canada)

10. adherence to Standard Operating Procedures, enforced by the metrological authority, adhered to by

    1. police services

    2. police officers

11. transparency in documentation including audit trails and continuity

The proper way to minimize the possibility of errors is to recognize the causes of error, minimize their possibility, and budget for them by acknowledging and controlling uncertainty of measurement.

The proper way is not to legislate an untruth: that approved instruments are "accurate and reliable" when they are not used properly.