Step-by-Step Sampling Methodology to Audit Mohs Photo Archives Before an OCR Desk Review

Define Audit Objectives

Your audit should state exactly what you want to prove before any OCR desk review begins. Focus on whether the Mohs photo archive is complete, consistent, and OCR-ready, and whether evidence exists to trace every selection decision from source to result.

Scope and units of analysis

• Population: all Mohs procedure cases within the defined timeframe and sites.
• Units: case-level (entire procedure), stage-level (Mohs stages/maps), and image-level (individual photographs/scans).
• Systems: archive repository, EMR/EHR links, and any image capture devices contributing files.

Measurable outcomes and KPIs

• Archive Completeness Assessment: percent of sampled cases with all expected images (pre-op, stage maps, margins, post-op) present.
• Metadata Verification: percent of images with correct patient/case IDs, dates/times, stage numbers, device info, and surgeon/tech identifiers.
• Quality Assurance: percent meeting minimum resolution, focus, color fidelity, and orientation requirements for OCR readiness.
• Data Traceability: ability to reproduce the sample using documented queries, timestamps, and random seeds.
• Error Identification: rate and severity distribution of missing, mismatched, or low-quality images.

Risk prioritization

• Prioritize strata with high patient volume, multi-stage cases, or prior quality issues.
• Define critical vs. major vs. minor error categories and target thresholds for each.

Select Representative Sample

A representative sample reflects how images are distributed across time, sites, devices, and case complexity. Start by building a clean sampling frame that uniquely lists eligible units.

Build the sampling frame

• Extract a de-duplicated list of cases and linked images with key fields (site, surgeon, date, device, stage count).
• Exclude non-audit images (training, test exports) and corrupted or quarantined records that cannot be legally reviewed.
• Confirm each record’s lineage to original systems to preserve Data Traceability.

Define inclusion/exclusion rules

• Inclusion: all clinical Mohs cases within the date window and participating sites.
• Exclusion: cases lacking required authorization, research-only images, and known system duplicates.

Representation criteria

• Mirror the population across site, quarter/month, device model, surgeon, stage count, and complex scenarios (recurrent tumors, flaps/grafts).
• Ensure rare but critical categories are present (e.g., multi-stage cases), even if oversampling is needed.

Implement Sampling Techniques

Choose methods that minimize bias while remaining practical and reproducible. For Mohs photo archives, Stratified Sampling is usually preferred.

Primary method: Stratified Sampling

• Create strata by site, time period, device, and case complexity.
• Allocate sample sizes to strata proportionally to their population share or via Neyman allocation when variability differs.
• Within each stratum, select cases using simple random sampling without replacement.
• For each selected case, include all linked images required for the desk review scope.

Alternative/complementary methods

• Systematic sampling: select every k-th case after a random start if lists are time-ordered and not periodic.
• Cluster sampling: sample by clinic-day or surgeon-session when operationally easier, then adjust for design effect.
• Risk-based targeted sampling: add a focused subset for known problem areas to accelerate Error Identification.

Reproducibility and controls

• Record exact queries, extract timestamps, and RNG seeds to support Data Traceability.
• Log all inclusions/exclusions and any manual adjudication with reasons.

Ensure Sample Size Adequacy

Perform a Sample Size Calculation that matches your objective (estimate an error rate vs. detect a minimum defect rate) and your design (simple random vs. cluster/stratified).

Estimating an error rate with a margin of error

• Use n0 = z² × p × (1 − p) / e², where p is expected error rate, e is desired margin of error, and z is the confidence level.
• Example: with p = 0.05, e = 0.02, and 95% confidence (z ≈ 1.96), n0 ≈ 457.
• If the population is finite, apply the finite population correction: n = n0 / [1 + (n0 − 1) / N]. For N = 5,000, n ≈ 420. Always round up.

Detecting at least one defect above a minimum rate

• Use n = ln(1 − confidence) / ln(1 − r), where r is the minimum defect rate you want to detect.
• Example: to detect at least one defect if r = 1% with 95% confidence, n ≈ 299.

Allocating sample across strata

• Proportional allocation: n_h = n × (N_h / N) keeps the sample aligned with the population.
• Neyman allocation: n_h ∝ N_h × S_h gives more sample to strata with higher variability (e.g., devices with mixed image quality).
• Set a minimum per stratum (e.g., 20–30 cases or all available if smaller) to enable within-stratum analysis.

Design and practical adjustments

• Inflate n for cluster sampling using a design effect if clustering is strong.
• Over-sample by 10–20% to offset expected ineligible or missing records.

Conduct Pre-OCR Desk Review

Apply a consistent, auditable checklist to each sampled case and image to verify OCR readiness and identify remediation needs before automation begins.

Archive Completeness Assessment

• Confirm that all expected images per workflow (pre-op, stage maps, margins, post-op) are present and linked to the correct case/stage.
• Check for duplicates, orphan images, or mismatched links across repositories.

Metadata Verification

• Validate patient/case identifiers, dates/times, stage numbers, device/capture source, and staff identifiers where applicable.
• Ensure consistent naming conventions and time-zone handling; flag gaps and conflicts.

Image-level Quality Assurance

• Assess resolution, focus/blur, exposure, color balance, orientation, cropping, and presence of handwritten annotations that could impair OCR.
• Confirm acceptable file formats and absence of corruption or compression artifacts beyond policy limits.

Error Identification and triage

• Classify findings as critical (block OCR), major (requires fix but not blocking), or minor (informational).
• Record root cause hypotheses (device issue, workflow gap, system sync) and immediate corrective actions.

Recording results

• Use a standardized log capturing case ID, stratum, checks performed, outcomes, severity, and remediation status.
• Preserve evidence images and decisions to maintain Data Traceability.

Validate Sample Representation

After review, confirm that conclusions generalize to the full archive by testing how well the sample mirrors the population.

Distribution checks

• Compare sample vs. population proportions for site, quarter, device, surgeon, and complexity; set tolerance bands (for example, within 5 percentage points).
• Apply simple statistical tests where appropriate (e.g., chi-square for categorical balances) and document results.

Adjustments

• Weight estimates by stratum if sample proportions differ from population.
• Conduct targeted re-sampling to fill underrepresented strata or confirm unexpected patterns.
• Perform inter-rater reliability checks if multiple reviewers graded images.

Document Sampling Procedures

A clear record enables repeatability and defensibility of your step-by-step sampling methodology for Mohs photo archives.

What to document

• Purpose, scope, definitions, roles, and acceptance thresholds.
• Sampling frame creation steps, inclusion/exclusion rules, and data lineage.
• Chosen technique (e.g., Stratified Sampling), Sample Size Calculation details, formulas, parameters, and assumptions.
• Allocation method across strata, random seed(s), exact queries, and selection timestamps.
• Desk review checklist, severity rubric, remediation workflows, and sign-offs.
• Quality Assurance controls, Data Traceability artifacts, and retention plans.

Conclusion

By defining objectives, selecting a representative sample, applying robust sampling techniques, verifying sample size adequacy, and executing a disciplined pre-OCR desk review, you create a defensible, efficient path to automation. Thorough documentation, rigorous Metadata Verification, and focused Error Identification ensure reliable findings and actionable improvements before full OCR deployment.

FAQs.

What is the best sampling method for auditing photo archives?

Stratified Sampling is typically best because it preserves representation across key dimensions like site, time period, device, and case complexity. You can complement it with risk-based targeting for known problem areas and systematic sampling when lists are stable and non-periodic.

How do you ensure sample size adequacy in audits?

Match the Sample Size Calculation to your goal: use proportion estimation when you need a margin of error around an error rate, or detection-based sizing to ensure you can find at least one defect above a minimum rate. Apply finite population correction, adjust for design effects if clustering exists, and over-sample slightly to cover ineligibles.

What criteria determine a representative sample of Mohs photo archives?

A representative sample mirrors the population across site, time, device, surgeon, and case complexity, includes rare but critical categories, and maintains correct case-to-image linkages. Balance can be confirmed by comparing sample and population distributions and using simple goodness-of-fit checks.

How is validation performed after sampling?

Validate by comparing sample and population distributions, applying statistical checks where suitable, and using weighting or targeted re-sampling if imbalances appear. Review inter-rater agreement, verify Data Traceability of selections, and confirm that conclusions remain stable under plausible alternative samples.