Behind the scenes of EQA – characteristics, capabilities, benefits and assets of external quality assessment (EQA)

Buchta C., Aberle S.W., Görzer I., Griesmacher A., Müller M.M., Neuwirth E., Puchhammer-Stöckl E., Weseslindtner L., Camp J.V.

Summary Background External quality assessment (EQA) schemes provide objective feedback to participating laboratories about the performance of their analytical systems and information about overall regional analytical performance. The EQAs are particularly important during pandemics as they also assess the reliability of individual test results and show opportunities to improve test strategies. With the end of the COVID-19 pandemic, the testing frequency significantly decreased in Austria. Here, we analyzed whether this decrease had an effect on participation and/or performance in SARS-CoV‑2 virus detection EQAs, as compared to the pandemic era. Material and methods Identical samples were sent to all participating laboratories, and the EQA provider evaluated the agreement of the reported results with defined targets. The EQA was operated under two schemes with identical samples and therefore we analyzed it as a single EQA round. The performance of testing was reported as true positive ratios, comparing the post-pandemic data to previous rounds. Furthermore, subgroups of participants were analyzed stratified by laboratory type (medical or nonmedical) and the test system format (fully automated or requiring manual steps). Results While the frequency of false negative results per sample did not change during the 3 years of the pandemic (5.7%, 95% confidence interval [CI] 3.1–8.4%), an average per sample false negative ratio of 4.3% was observed in the first post-pandemic EQA (0%, 1.8%, and 11% for the 3 positive samples included in the test panel, n = 109 test results per sample). In this first post-pandemic EQA medical laboratories (average 0.4% false negative across 3 samples, n = 90) and automated test systems (average 1.2% false negative, n = 261) had lower false negative ratios than nonmedical laboratories (22.8%, n = 19) and manual test systems (16.7%, n = 22). These lower average ratios were due to a low concentration sample, where nonmedical laboratories reported 36.8% and manual test systems 54.5% true positive results. Conclusion Overall ratios of true positive results were below the mean of all results during the pandemic but were similar to the first round of the pandemic. A lower post-pandemic true positive ratio was associated with specific laboratory types and assay formats, particularly for samples with low concentration. The EQAs will continue to monitor the laboratory performance to ensure the same quality of epidemiological data after the pandemic, even if vigilance has decreased.

Buchta C., Gidske G., Henriksen G.M., Badrick T.

Wielders J.P., Porpiglia N.M., Schellenberg F., Deenmamode J., Delanghe J., Anton R.F., Bortolotti F., Siebelder C., Tagliaro F., Weykamp C., Helander A.

Carbohydrate deficient transferrin (CDT) is a biomarker for excessive alcohol consumption utilized in clinical and forensic medicine and workplace testing. Previously, many different analytical methods for CDT were used and the measurand varied considerably, making direct comparison of test results difficult. To end this confusion, the IFCC established a working group on CDT standardisation (WG-CDT) which completed its tasks in 2017. This IFCC position paper by the WG-CDT summarizes state of the art information about the measurand and the analytical methods and gives concise recommendations for its utilization. The results achieved by the CDT standardisation process led to accuracy improvements in national external quality assessment schemes over the years. A brief review of ROC based comparison studies with the traditional biomarkers (GGT, MCV, ALT and AST) discusses the bias resulting from inadequate study populations. In large groups of the general population the superior diagnostic performance of CDT is confirmed. The relationship between alcohol intake versus resulting CDT is discussed as well as the cutoff and measurement uncertainty. Concerning the application in practice, potential pitfalls are considered and recommendations handling both analytical and preanalytical caveats are given. Finally, some examples of serious misunderstandings in publications about CDT are addressed.

van Rossum H.H., Holdenrieder S., Ballieux B.E., Badrick T.C., Yun Y., Zhang C., Patel D., Thelen M., Song J., Wojtalewicz N., Unsworth N., Vesper H.W., Cui W., Ramanathan L.V., Sturgeon C., et. al.

Abstract Background The harmonization status of most tumor markers (TMs) is unknown. We report a feasibility study performed to determine whether external quality assessment (EQA) programs can be used to obtain insights into the current harmonization status of the tumor markers α-fetoprotein (AFP), prostate specific antigen (PSA), carcinoembryonic antigen (CEA), cancer antigen (CA)125, CA15-3 and CA19-9. Methods EQA sample results provided by 6 EQA providers (INSTAND [Germany], Korean Association of External Quality Assessment Service [KEQAS, South Korea], National Center for Clinical Laboratories [NCCL, China], United Kingdom National External Quality Assessment Service [UK NEQAS, United Kingdom], Stichting Kwaliteitsbewaking Medische Laboratoriumdiagnostiek [SKML, the Netherlands], and the Royal College of Pathologists of Australasia Quality Assurance Programs [RCPAQAP, Australia]) between 2020 and 2021 were used. The consensus means, calculated from the measurement procedures present in all EQA programs (Abbott Alinity, Beckman Coulter DxI, Roche Cobas, and Siemens Atellica), was used as reference values. Per measurement procedure, the relative difference between consensus mean for each EQA sample and the mean of all patient-pool–based EQA samples were calculated and compared to minimum, desirable, and optimal allowable bias criteria based on biological variation. Results Between 19040 (CA15-3) and 25398 (PSA) individual results and 56 (PSA) to 76 (AFP) unique EQA samples were included in the final analysis. The mean differences with the consensus mean of patient-pool–based EQA samples for all measurement procedures were within the optimum bias criterion for AFP, the desirable bias for PSA, and the minimum bias criterion for CEA. However, CEA results <8 µg/L exceeded the minimum bias criterion. For CA125, CA15-3, and CA19-9, the harmonization status was outside the minimum bias criterion, with systematic differences identified. Conclusions This study provides relevant information about the current harmonization status of 6 tumor markers. A pilot harmonization investigation for CEA, CA125, CA15-3, and CA19-9 would be desirable.

van Schrojenstein Lantman M., Grobben R., van Herwaarden A.E., van Berkel M., Schaap J., Thelen M.

Abstract Objectives To accurately evaluate non-ST-elevated acute cardiac syndrome (NSTE-ACS), the quality of high-sensitive cardiac troponin (hs-cTn) assays is of vital importance. The 2020 revision of the NSTE-ACS guideline includes clinical decision-limits (CDL’s) to both rule-in and rule-out NSTE-ACS for most commercially available platforms, providing both 0/1 h and 0/2 h delta limits. Our study evaluated whether laboratories are able to meet the analytical performance specifications for imprecision (APS) for hs-cTnT. Methods Results from external quality assurance (EQA) in commutable samples were used to evaluate the current and historic performance of analyzers. The performance of analyzers that either passed or failed to comply with 0/1 h-APS were used on a real-world dataset of first hs-cTnT-values to simulate 10.000 samples of t=0, t=1 and t=2 h values with multiple delta’s for all relevant CDL’s. We compared the simulated values to the input values to obtain the percentage of aberrant results simulated. Results The majority of analyzers complies with APS for rule-in in 2022 (0/1 h: 90.4 % and 0/2 h: 100 %), compliance for the 0/1 h rule-out is still far from optimal (0/1 h: 30.7 %, 0/2 h: 75.4 %), with improving compliance over the past years (rule-in p=<0.0001, rule-out p=0.011, χ2). Whilst 0/1 h-APS-passing analyzers have a minute risk to falsely rule-out patients whom should be ruled-in (0.0001 %), failing performance increases this risk to 2.1 % upon using 0/1 h CDL’s. Here, adopting 0/2 h CDL’s is favorable (0.01 %). Conclusions Laboratories that fail to meet hs-cTnT 0/1 h-APS should improve their performance to the required and achievable level. Until performance is reached clinics should adopt the 0/2 h CDL’s.

Buchta C., Zeichhardt H., Osterman A., Perrone L.A., Griesmacher A.

The shift to a colder season is generally accompanied by an annual increase in respiratory infections. In early September, 2023, European epidemiological surveillance systems reported the first annual signs of increased COVID-19 infections.1 Efforts to increase testing activity have started,2 and the expected rise in testing frequency calls for a precautionary reminder of the limitations of diagnostic testing and screening for infectious pathogens. Every diagnostic test system has unique performance characteristics, including its overall sensitivity and specificity, which reflect the assay design.

Restelli V., Vimalanathan S., Sreya M., Noble M.A., Perrone L.A.

In response to the coronavirus pandemic (COVID-19) and scale up of diagnostic testing, the Canadian Microbiology Proficiency Testing program created a new proficiency testing (PT) program for the molecular and antigen detection of SARS-CoV-2. The program was geared to point of care testing (POCT) sites located in each of the eight provincial Health Authorities across British Columbia, Canada, with the intention to monitor testing quality. The PT program consisted of 6 shipments in a year, each containing a set of 4 samples either positive for SARS-CoV-2 virus or negative. The program began with initial 23 sites enrolling in March 2021, expanding to >100 participants by December 2021. After the first two surveys, it was observed that testing performance (accuracy) was consistently acceptable for sites using nucleic acid technology (NAT), however performance by sites using rapid antigen detection (RAD) methods was poor, especially when testing the weakly positive samples. A root cause investigation of poor testing performance revealed gaps in the execution of testing methods and also in results interpretation. These quality issues were most commonly associated with new testers who lacked experience with diagnostic testing. Tester training and mentoring was reinforced as was retraining of personnel; sample processing instructions were modified, and a training video was also created for testing sites. As a result of these interventions, sites improved their testing accuracy and the performance of POCT sites using RAD methods came to more closely match the performance of sites utilizing NAT. Overall, the PT program was highly successfully and improved quality of testing in the province. This work demonstrates the critical value of an external quality assessment (EQA) partner towards improving patient and public health and safety, especially when testing is conducted outside of an accredited medical laboratory setting.

Sandberg S., Fauskanger P., Johansen J.V., Keller T., Budd J., Greenberg N., Rej R., Panteghini M., Delatour V., Ceriotti F., Deprez L., Camara J.E., MacKenzie F., Lyle A.N., van der Hagen E., et. al.

Abstract It is important for external quality assessment materials (EQAMs) to be commutable with clinical samples; i.e., they should behave like clinical samples when measured using end-user clinical laboratory in vitro diagnostic medical devices (IVD-MDs). Using commutable EQAMs makes it possible to evaluate metrological traceability and/or equivalence of results between IVD-MDs. The criterion for assessing commutability of an EQAM between 2 IVD-MDs is that its result should be within the prediction interval limits based on the statistical distribution of the clinical sample results from the 2 IVD-MDs being compared. The width of the prediction interval is, among other things, dependent on the analytical performance characteristics of the IVD-MDs. A presupposition for using this criterion is that the differences in nonselectivity between the 2 IVD-MDs being compared are acceptable. An acceptable difference in nonselectivity should be small relative to the analytical performance specifications used in the external quality assessment scheme. The acceptable difference in nonselectivity is used to modify the prediction interval criterion for commutability assessment. The present report provides recommendations on how to establish a criterion for acceptable commutability for EQAMS, establish the difference in nonselectivity that can be accepted between IVD-MDs, and perform a commutability assessment. The report also contains examples for performing a commutability assessment of EQAMs.

Byrne R.A., Rossello X., Coughlan J.J., Barbato E., Berry C., Chieffo A., Claeys M.J., Dan G., Dweck M.R., Galbraith M., Gilard M., Hinterbuchner L., Jankowska E.A., Jüni P., Kimura T., et. al.

Vierbaum L., Wojtalewicz N., Grunert H., Zimmermann A., Scholz A., Goseberg S., Kaiser P., Duehring U., Drosten C., Corman V., Niemeyer D., Rabenau H.F., Obermeier M., Nitsche A., Michel J., et. al.

AbstractThe COVID-19 pandemic illustrated the important role of diagnostic tests, including lateral flow tests (LFTs), in identifying patients and their contacts to slow the spread of infections. INSTAND performed external quality assessments (EQA) for SARS-CoV-2 antigen detection with lyophilized and chemically inactivated cell culture supernatant of SARS-CoV-2 infected Vero cells. A pre-study demonstrated the suitability of the material. Participants reported qualitative and/or quantitative antigen results using either LFTs or automated immunoassays for five EQA samples per survey. 711 data sets were reported for LFT detection in three surveys in 2021. This evaluation focused on the analytical sensitivity of different LFTs and automated immunoassays. The inter-laboratory results showed at least 94% correct results for non-variant of concern (VOC) SARS-CoV-2 antigen detection for viral loads of ≥ 4.75 × 106 copies/mL and SARS-CoV-2 negative samples. Up to 85% had success for a non-VOC viral load of ~ 1.60 × 106 copies/mL. A viral load of ~ 1.42 × 107 copies/mL of the Delta VOC was reported positive in > 96% of results. A high specificity was found with almost 100% negative SARS-CoV-2 antigen results for HCoV 229E and HCoV NL63 positive samples. Quantitative results correlated with increasing SARS-CoV-2 viral load but showed a broad scatter. This study shows promising SARS-CoV-2 antigen test performance of the participating laboratories, but further investigations with the now predominant Omicron VOC are needed.

Buchta C., Zeichhardt H., Badrick T., Coucke W., Wojtalewicz N., Griesmacher A., Aberle S.W., Schellenberg I., Jacobs E., Nordin G., Schweiger C., Schwenoha K., Luppa P.B., Gassner U.M., Wagner T., et. al.

European legislation defines as "near-patient testing" (NPT) what is popularly and in other legislations specified as "point-of-care testing” (POCT). Systems intended for NPT/POCT use must be characterized by independence from operator activities during the analytic procedure. However, tools for evaluating this are lacking. We hypothesized that the variability of measurement results obtained from identical samples with a larger number of identical devices by different operators, expressed as the method-specific reproducibility of measurement results reported in External Quality Assessment (EQA) schemes, is an indicator for this characteristic. Legal frameworks in the EU, the USA and Australia were evaluated about their requirements for NPT/POCT. EQA reproducibility of seven SARS-CoV-2-NAAT systems, all but one designated as "POCT", was calculated from variabilities in Ct values obtained from the respective device types in three different EQA schemes for virus genome detection. A matrix for characterizing test systems based on their technical complexity and the required operator competence was derived from requirements of the European In Vitro Diagnostic Regulation (IVDR) 2017/746. Good EQA reproducibility of the measurement results of the test systems investigated implies that different users in different locations have no recognizable influence on their measurement results. The fundamental suitability of test systems for NPT/POCT use according to IVDR can be easily verified using the evaluation matrix presented. EQA reproducibility is a specific characteristic indicating independence from operator activities of NPT/POCT assays. EQA reproducibility of other systems than those investigated here remains to be determined.

van Schrojenstein Lantman M., van de Logt A., Prudon-Rosmulder E., Langelaan M., Demir A.Y., Kurstjens S., van der Horst A., Kuypers A., Greuter A., Kootstra-Ros J., van der Hagen E., Oostendorp M., de Beer R., Ramakers C., Bakkeren D., et. al.

Abstract Objectives Measurement of plasma albumin is pivotal for clinical decision-making in patients with chronic kidney disease (CKD). Routinely used methods as bromocresol green (BCG) and bromocresol purple (BCP) can suffer from aselectivity, but the impact of aselectivity on the accuracy of plasma albumin results of CKD-patients is still unknown. Therefore, we evaluated the performance of BCG-, BCP- and JCTLM-endorsed immunological methods in patients with various stages of CKD. Methods We evaluated the performance of commonly used albumin methods in patients with CKD stages G1 through G5, the latter divided in two groups based on whether they received hemodialysis treatment. In total, 163 patient plasma samples were measured at 14 laboratories, on six different BCG and BCP-platforms, and four different immunological platforms. The results were compared with an ERM-DA-470k-corrected nephelometric assay. The implications on outcome is evaluated by the proportion of patient results <38 g/L for the diagnosis of protein energy wasting. Results Albumin results determined with BCP- and immunological methods showed the best agreement with the target value (92.7 and 86.2 %, respectively vs. 66.7 % for BCG, namely due to overestimation). The relative agreement of each method with the target value was platform-dependent, with larger variability in agreement between platforms noted for BCG and immunological methods (3.2–4.6 and 2.6–5.3 %) as opposed to BCP (0.7–1.5 %). The stage of CKD had similar effects on the variability in agreement for the three method-groups (0.6–1.8 % vs. 0.7–1.5 % vs. 0.4–1.6 %). The differences between methods cause discrepancies in clinical decision-making, as structurally fewer patients were diagnosed with protein energy wasting upon using BCG-based albumin results. Conclusions Our study shows that BCP is fit for the intended use to measure plasma albumin levels in CKD patients from all stages, including patients on hemodialysis. In contrast, most BCG-based platforms falsely overestimate the plasma albumin concentration.

Buchta C., Zeichhardt H., Aberle S.W., Camp J.V., Görzer I., Weseslindtner L., Puchhammer-Stöckl E., Huf W., Benka B., Allerberger F., Mielke M., Griesmacher A., Müller M.M., Schellenberg I., Kammel M.

During an epidemic, individual test results form the basis of epidemiological indicators such as case numbers or incidence. Therefore, the accuracy of measures derived from these indicators depends on the reliability of individual results. In the COVID-19 pandemic, monitoring and evaluating the performance of the unprecedented number of testing facilities in operation, and novel testing systems in use, was urgently needed. External quality assessment (EQA) schemes are unique sources of data reporting on testing performance, and their providers are recognised contacts and support for test facilities (for technical–analytical topics) and health authorities (for planning the monitoring of infection diagnostics). To identify information provided by SARS-CoV-2 genome detection EQA schemes that is relevant for public health microbiology, we reviewed the current literature published in PubMed between January, 2020, and July, 2022. We derived recommendations for EQA providers and their schemes for best practices to monitor pathogen-detection performance in future epidemics. We also showed laboratories, test facilities, and health authorities the information and benefits they can derive from EQA data, and from the non-EQA services of their providers.

Buchta C., Zeichhardt H., Griesmacher A., Schellenberg I., Kammel M.

The Lancet Commission on lessons for the future from the COVID-19 pandemic1 discusses a series of failures during this challenging time, and we acknowledge the authors' comprehensive and important work. However, the authors refrained from discussing the urgent need to closely monitor the performance of SARS-CoV-2 testing, and to act when underperforming testing facilities and test systems were identified.

Cobbaert C.M.

MacKenzie F.