For getting help during the practical activities access: Instructions and Interpretations.
Introductory Notions
The ROC (Receiver Operating Characteristics) curve is a two-dimensional curve that graphically shows the pair values of sensitivity (the OY axis, the true positives rate) and specificity (the OX axis, (1-Sp) = the false positives rate) and thus allowing the evaluation of a sign/diagnosis test. It was used for the first time in assessing a diagnostic test in 1960 by Lusted in the radiographic diagnosis of pulmonary tuberculosis [Lusted LB. Logical analysis in Roentgen diagnosis. Radiology. 1960; 74:178 – 93].
The ROC analysis is correctly applied if the evaluation is done by comparing it with a reference test (‘the golden test’) and the result of the diagnosis test is continuous (a measured variable) or ordinary with minimum 5 classes (definitely normal/probably normal/unsure/probably abnormal/definitely abnormal). [Hanley JA. Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imaging. 1989;29:307–35] . The purpose of the ROC analysis is to identify the optimum threshold value for differentiating a positive result from a negative result. One can use parametric statistical methods (the diagnostic test values follow a normal distribution) or non-parametric methods (there is no assumption about the distribution of diagnostic test values) to perform the ROC analysis and to obtain the values of the area under the curve (AUC = Area Under the curve). The method is used to evaluate a diagnostic test or to compare two different diagnostic methods, both applied to each patient.
The ROC curve is:
• A method for evaluating a diagnostic test.
• A graphical method of representation of threshold values.
• An independent method of the prevalence of the disease of interest.
The ROC analysis allows visualization of more than one diagnostic test, respectively identifying the value of the Se for the fixed value of the Sp.
The Area under Curve (AUC):
• A unidimensional Indicator that summarizes the "general " location of the entire ROC curve.
• A correct interpretation is done with the associated 95% trust/confidence interval: If AUC ≥ 0.9 the diagnostic accuracy is excellent, 0.8 ≤ AUC 0.9 indicates very good diagnostic accuracy and a AUC value less than 0.8 indicates a test with a worthless diagnostic accuracy.
• The maximum value is equal to 1 and indicates a perfect diagnostic test, able to correctly identify both sick and disease-free subjects.
• AUC = 0.5 means random discrimination; AUC = 0 means that the test classifies incorrectly all subjects as negative and all disease-free subjects as positive.
• It can be checked statistically whether AUC is statistically significant or not.
Possible statistical assumptions:
- a single Test diagnosis: H0: AUC = 0.5, vs. H1: AUC ≠ 0.5
where AUC = The area under the curve of the new diagnostic test.
If p 0.05 this means that the diagnostic test has the ability to identify sick and disease-free.
- a comparison of two diagnostic tests: H0: AUC1 = AUC2, vs. H1: AUC1 ≠ AUC2
where AUC1 = The area under the curve of the first diagnostic test, AUC2 = The area under the curve of the second diagnostic test.
If p 0.05, this means that there is a statistically significant difference in the accuracy of tests; The higher the AUC value, the better the test.
The identifying of the threshold values is frequently done using one of the following methods (Figure 1):
- D2 = minimum, where D2 = (1-Se) 2 + (1-Sp)2
- J = max (Se + Sp-1), where J = Youden Index
Figure 1. ROC curve: generic aspects. A less-performing diagnostic test has the graph closer to the diagonal.
THE SCENARIO OF THE STUDY - EXAMPLE
Acute appendicitis consists of inflammation of the appendix that untreated surgically can rupture and the infection will extend into the abdominal cavity causing peritonitis. There are 3 types of appendicitis: ① catarrhal (congestive) in which the appendix is turgescent, congested, red-purplish; ② phlegmonous (suppurated/festered) with an enlarged appendix, tensioned and brittle; and ③ gangrenous (necrotic-haemorrhagic), in which the appendicular wall presents devitalized areas with flaccid appearance and brown colour.
Cikot and his collaborators have investigated the usefulness of the calprotectin (an anti-microbial protein accumulated in neutrophils but also existing in lesser quantity at the level of the monocytes and macrophages). The scenario of this exercise is based on the article published by Cikot and his collaborators and the answer to questions is based on this article. This exercise will help evaluate the clinical usefulness of the evaluation of blood levels of calprotectin in the differentiation of acute appendicitis, that is, to differentiate between complicated appendicitis (perforation, empyema or abscess) and uncomplicated appendicitis.
THE PROTOCOL OF THE STUDY
In order to answer to these questions, one should read the different sections (specified for each individual question) of the article titled “Plasma Calprotectin Level: Usage in Distinction of Uncomplicated from Complicated Acute Appendicitis”, authors Cikot M, Peker KD, Bozkurt MA, Kocatas A , Kones O, Binboga S, et al., Published in the World Journal of Emergency Surgery 2016; 11: 7. Two: 10.1186 / s13017-016-0062-9. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4729141/
- Aim and Objective (Abstract and last paragraph from “Introduction”)
The aim: The evaluation of calprotectin as a biomarker in the diagnosis of acute appendicitis and respectively the discrimination between acute complicated and uncomplicated appendicitis.
The objectives of the study:
- The comparison of the routine inflammatory markers in medical practice (Reactive C protein - CRP, white blood cell level- WBC) with calprotectin plasma levels in patients with acute appendicitis.
- The role of the plasma calprotectin in the diagnosis of complicated and uncomplicated acute appendicitis.
- The accuracy of the calprotectin and the C-reactive protein as markers of acute appendicitis.
- Field of research: The evaluation of a diagnostic procedure
- Type of study: (Methods)
- according to the objectives of the study: analytical
- according to the results: interventional
- according to the technique used in choosing groups: unmatched
- has the masking technique been used to evaluate the applied tests? There is no information in the text of the article that will allow the evaluation of masking.
- The Accessible Population and the Study Sample (Methods)
- The accessible population: Patients with acute appendicitis treated at the Dr. Sadi Konuk Training and Research Hospital in Istanbul Bakırköy.
- The study sample:
o Case: Patients with acute appendicitis with ASA score (American Society of Anesthesiologists) I-III with laparoscopic treatment between January 2013 and May 2013. The subjects with perforated, gangrenous appendicitis or with the presence of an intra-abdominal abscess were included in the appendicitis group complicated.
o Control: Patients who were present at the same time at the same hospital for abdominal pain in the right flank, or right and left at the lower dial with a laparoscopic diagnosis of other acute appendicitis pathology. The patients with pelvic inflammatory pathologies (18 cases), hemorrhagic or broken ovary (16 cases), or diverticulitis (3 cases) were excluded from the control group.
- The method of collecting data (Methods)
- Depending on the study population: It is not specified in the article whether a sampling method has been used or not.
- Depending on the duration of data collection: Not clearly specified in the article.
- Depending on the composition of the group or groups of subjects: case-control (the two groups were defined according to the presence / absence of acute appendicitis).
- The diagnostic tests (Methods)
Applied diagnostic tests
- Calprotectin: the limit of identification = 46.8 ng / ml. The blood harvested from the patients was centrifuged for 15 minutes at 2000 × g. The plasma was stored at -80 ° C until determination. Determination was done with the Cal ELISA kit (East Biopharm, China)
- CRP normal values: 0.01-0.5 mg / dL
- WBC normal values: 4-11 × 103 / mm
The markers were determined after diagnosis and before treatment (surgery).
The reference test which has been used: The histopathological examination (information available only in the “Results” section. This information had to be found in the “Methods” section).
THE EXPECTED RESULTS. THE ANALYSIS, PRESENTATION AND INTERPRETATION OF DATA
- The statistical analysis and interpretation of the results
- The description of the sample investigated (gender, age, etc.) (Results, in text)
o M/F: 54/35 case group, respectively 20/10 control group.
o Years (average (amplitude)): 28 (19-45) case group, 31 (21-56) control group
- In the acute appendicitis sample, how many patients had complicated appendicitis and how many had uncomplicated appendicitis? (Results, in text)
o uncomplicated appendicitis (mucosal inflammation): 22 patients
o complicated appendicitis: 67 patients (48 patients with phlegmonous appendicitis, 19 patients with gangrenous appendicitis)
- What was Se, Sp, VPP, VPN for calprotectin? (Results, in text)
o for the threshold value of 46 ng / mL: Se = 98.88%, Sp = 83.33%, VPP = 94.64%, VPN = 96.15%
- Were the values of the markers investigated significantly different in the acute appendicitis group compared to the control group? (Results, Table 1)
p The investigated markers had significantly higher values in the acute appendicitis group compared to the control group:
- Were the values of the markers investigated significantly different in those with complicated appendicitis compared to those with uncomplicated appendicitis? (Results, Table 2)
o There were no statistically significant differences in white blood cells (WBC) (p = 0.164).
o The C-reactive protein (PCR) values were statistically significantly higher in the complicated appendicitis group (p = 0.014).
o Calprotectin had approximately 3-fold higher values in patients with complicated appendicitis compared to patients with uncomplicated appendicitis. The difference was highly significant from a statistical point of view (p = 0.0001).
- Is there a statistically significant difference between markers investigated in the diagnosis of patients with acute appendicitis? (Results, Table 4)
The comparison of the AUC values for pairs of markers revealed the absence of statistically significant difference (p> 0.9, see Table 4)
- Is there a statistically significant difference between the markers investigated in the complicated appendicitis versus the uncomplicated appendicitis? (Results, Table 5)
INTERPRETATION OF DATA. DISCUSSIONS
8. Discussions
• What is the most effective marker in identifying patients with acute appendicitis? (Results)
All the three investigated markers had values of the area under the curve (AUC) above 0.9, with the lower limit of the 95% confidence interval greater than 0.8. à All three markers are good tests in the diagnosis of acute appendicitis. (Table 4)
• What is the most effective marker in discriminating complicated appendicitis? (Results, Table 5 & Fig 2)
The only biomarker with the ability to discriminate complicated appendicitis compared to uncomplicated appendicitis is calprotectin because AUC = 0.935 and lower confidence interval> 0.8 indicate an excellent diagnostic test. The lower white blood cell count limit (WBC) is in the region of an inaccurate diagnostic test. The lower confidence level range for C-reactive protein (CRP) is less than 0.5.
- The Conclusion of the Study
• Any of the three markers investigated are useful in identifying patients with acute appendicitis.
• Calprotectin appears to be a useful marker in discriminating patients with complicated appendicitis but to support this claim `more studies are needed