Question & Answer
Question
I produced an ROC (Receiver Operating Characteristic) curve with the ROC command (Analyze>ROC Curve) in IBM SPSS Statistics. My State Variable (STATEVAR) is binary (0,1), with 1 specified as the "Value of State Variable". My Test variable is named TESTVAR. I took the defaults in the Options dialog of the ROC Curve procedure. Here is my syntax command: ROC testvar BY statevar (1) /PLOT=CURVE(REFERENCE) /PRINT=SE COORDINATES /CRITERIA=CUTOFF(INCLUDE) TESTPOS(LARGE) DISTRIBUTION(FREE) CI(95) /MISSING=EXCLUDE. In the graph that was produced, the ROC curve is below the diagonal reference line, projecting into the lower right half of the graph. The Area Under the Curve (AUC) is .260 and significantly different from .5. I expected the ROC curve to be above the reference line, projecting into the upper left half of the graph, with an AUC value above .5. What does it mean when the ROC curve is below the reference line? Can such a curve be interpreted? Does the AUC below .5 mean that the Test Variable is performing worse than chance?
Answer
When the ROC curve dips prominently into the lower right half of the graph, this is likely a sign that either the wrong State Value has been specified or the wrong Test-State association direction has been specified in the "Test Direction" area of the "ROC Curve:Options" dialog. You stated that 1 is the intended choice for the value of the State Variable.
The default setting for "Test Direction" ("Larger test result indicates more positive test") is based on the assumption that the likelihood of the State event increases with increases in the Test variable. The keywords "TESTPOS(LARGE)" in the /CRITERIA subcommand of your ROC command syntax indicate that this positive association direction was expected. However, the location of the curve in the graph indicates that the likelihood of the State Value (1, in this analysis) increases as the value of the Test Variable decreases.
To indicate that the likelihood of the State Value will increase as the Test Variable decreases,
change this specification to "TESTPOS(SMALL)", as in:
ROC testvar BY statevar (1)
/PLOT=CURVE(REFERENCE)
/PRINT=SE COORDINATES
/CRITERIA=CUTOFF(INCLUDE) TESTPOS(SMALL) DISTRIBUTION(FREE) CI(95)
/MISSING=EXCLUDE.
To change the specification from the dialog boxes, click the Options button in the main ROC Curve dialog. In the "ROC curve: Options" dialog, click the radio button beside "Smaller test result indicates more positive test".
If the ROC curve wandered back and forth across the diagonal reference line, this would be an indication that the Test Variable had little diagnostic value. With the curve well underneath the reference line as you experienced, this is an indication that the direction of association is incorrectly specified, as stated above. It is better to correct this setting than to try and interpret a curve that falls below the diagonal reference. The area under the curve is misleadingly small (.260 with TESTPOS(LARGE) in your results), whereas the AUC with TESTPOS(SMALL) will be .740 with your data. Note that (.740-.5) = (.5 - .260).
If a researcher has predicted that the relationship between the state and test variables was positive, so that TESTPOS(LARGE) should have produced a curve that projects into the upper left, i.e. over the diagonal reference line, but finds a curve that is under the diagonal, then the theory behind the prediction is apparently faulty. If the theory predicted a negative association between the state and test variables, then the researcher just needs to choose the TESTPOS direction consistent with that theory.
Related Information
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21573229