This paper reports the results of a comparison between two different methods of examining quality in outpatient services in a developing country. Data from rural and urban Tanzania are used to compare the measures of quality collected by direct clinician observation (DCO) (where clinicians are observed in the course of their normal consultations) and vignettes (unblind case studies with an actor). The vignettes are shown to exhibit a strong connection between the inputs provided during consultation (rational history taking, physical examination and health education) and the ability of the clinician to properly diagnose the presented illness. However, the inputs provided in vignettes are not well correlated with the inputs provided in DCO, suggesting that the inputs provided in the vignette are not well correlated with the inputs that would be provided in an actual consultation. We conclude that since vignettes do not appear to be measuring what would be provided in an actual consultation they are not a good measure of quality. Instead, we suggest that vignettes and DCO be used simultaneously. We show how the scores obtained using vignettes in conjunction with DCO can be used to improve the reliability of DCO and therefore our estimates of actual clinician quality.