Without guidelines With guidelines   Department (pr>chisq) PGY level (pr>chisq) Department (pr>chisq) PGY level (pr>chisq) True Positive 0.069 0.092 0.338 0.305 True Negative 0.079 0.056 0.081 0.957 False Positive 0.052 0.044 0.098 0.897 False Negative 0.065 0.093 0.262 0.160

Table 4: Modeling true positive, false positive, true negative and false negative rate probabilities across all scenarios by specialty and PGY level with or without the use of a guide.