The smallest change in a quantity being measured that causes a perceptible change in the instrument's indication. Resolution represents the finest increment an instrument can detect or display.
Resolution is the smallest distinguishable difference between two measurement values that an instrument can indicate. For a digital instrument, resolution is typically the smallest digit displayed (e.g., a digital caliper reading to 0.01 mm has a resolution of 0.01 mm). For an analog instrument, resolution is related to the smallest graduation on the scale and the observer's ability to interpolate between marks.
Resolution should not be confused with accuracy or precision. An instrument may have very fine resolution but poor accuracy if it is not properly calibrated. Conversely, a coarse-resolution instrument might be highly accurate at the values it can display. Resolution sets a lower bound on the measurement uncertainty — you cannot reliably measure changes smaller than the instrument's resolution.
In calibration management, resolution is a key specification when selecting instruments for a measurement task. The instrument's resolution must be significantly finer than the tolerance being measured to ensure meaningful results. A common guideline is that the instrument resolution should be at least one-tenth of the tolerance. Additionally, resolution contributes to the uncertainty budget as a Type B uncertainty component, typically modeled as a rectangular distribution with a half-width equal to half the resolution.
In aerospace calibration labs, resolution directly impacts measurement reliability for critical components. When calibrating a digital multimeter used to test avionics systems, the instrument's voltage resolution of 1 µV determines whether technicians can detect subtle signal variations in flight control circuits. If the DMM's resolution is insufficient for the 10 µV tolerance requirements, measurements become meaningless. Medical device manufacturers face similar challenges when calibrating pressure transducers for ventilators. A pressure calibrator with 0.01 mmHg resolution may appear adequate, but when validating pressure sensors requiring ±0.005 mmHg accuracy, the 2:1 resolution-to-tolerance ratio violates best practices. This insufficient resolution leads to measurement uncertainty calculations that exceed acceptable limits. Common audit findings include using instruments where resolution exceeds 10% of the measurement tolerance, making it impossible to demonstrate measurement capability. For example, using a torque wrench with 0.5 Nm resolution to verify 2.0 Nm specifications results in 25% resolution ratio, causing automatic non-conformance during AS9100 audits. Labs frequently discover resolution inadequacy only during uncertainty calculations, forcing expensive instrument replacements and recalibration of all affected work.
ISO/IEC 17025:2017 addresses resolution in section 6.4.4, requiring labs to ensure measurement equipment has adequate resolution for intended measurements. AS9100D references resolution through measurement system analysis requirements in section 7.1.5.1, mandating 10:1 discrimination ratios where feasible. ISO 13485:2016 section 7.6 requires medical device manufacturers to validate that measuring equipment resolution supports process requirements. The GUM (ISO/IEC Guide 98-3) discusses resolution as a Type B uncertainty component in section 4.3.7, requiring its evaluation in uncertainty budgets. ANSI/NCSL Z540.3-2006 section 4.2 establishes the 4:1 test accuracy ratio, inherently addressing resolution adequacy. ILAC-P14:01/2013 policy requires accredited labs to demonstrate measurement capability through appropriate resolution selection. Auditors specifically examine instrument specifications versus measurement requirements, uncertainty calculations including resolution components, and documented rationale for resolution adequacy. Non-conformances frequently arise when labs cannot justify resolution selection or when instruments lack sufficient resolution for stated measurement capabilities.
CalibrationOS captures instrument resolution specifications in the Equipment module's technical specifications section, automatically flagging potential resolution inadequacy when creating calibration procedures. The Uncertainty Calculator module incorporates resolution as a Type B uncertainty component, performing automatic calculations per GUM guidelines and alerting users when resolution exceeds recommended limits relative to measurement tolerances. During procedure creation, the software compares required measurement resolution against selected reference standards, generating warnings when ratios fall below configurable thresholds (typically 4:1 or 10:1). Certificate generation includes resolution statements in measurement capability sections, ensuring traceability documentation meets regulatory requirements. The Audit Compliance dashboard tracks resolution adequacy across all instruments, providing real-time visibility into potential non-conformances. Automated reports highlight instruments approaching resolution limits, enabling proactive equipment planning and preventing measurement capability gaps during audits.
Resolution is the smallest increment of change that a measuring instrument can detect or display. For digital instruments, it is the value of the least significant digit; for analog instruments, it relates to the smallest scale graduation.
Resolution contributes to measurement uncertainty because you cannot distinguish values within one resolution increment. It is typically included in the uncertainty budget as a rectangular distribution with a half-width of one-half the resolution.
This article is licensed CC BY-SA 4.0. Share, adapt, and reuse with attribution to calibrationos.com/glossary/resolution.
Industry benchmarks, best practices, and calibration tips — delivered to your inbox.
Free calibration management software with audit-ready tracking, uncertainty budgets, and compliance tools.
Start Free — No Credit Card