Journal of Health Services Research & Policy > Volume 12, Number 1 > Pp. 42-47

This Article Full Text (PDF) An erratum has been published Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Dixon-Woods, M. Articles by Jones, D. Search for Related Content PubMed PubMed Citation Social Bookmarking                      What's this?

Original research

Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods

Mary Dixon-Woods, Alex Sutton, Rachel Shaw, Tina Miller, Jonathan Smith, Bridget Young, Sheila Bonas, Andrew Booth, David Jones

Department of Health Sciences, University of Leicester, Leicester, UK; Department of Health Sciences, University of Leicester, Leicester, UK; School of Life and Health Sciences, Aston University, Birmingham, UK; Department of Sociology and Social Policy, Oxford Brookes University, Oxford, UK; School of Psychology, Birkbeck, University of London, London, UK; Division of Clinical Psychology, University of Liverpool, Liverpool, UK; Department of Health Sciences, University of Leicester, Leicester, UK; ScHARR, University of Sheffield, Sheffield, UK; Department of Health Sciences, University of Leicester, Leicester, UK

Objective: Qualitative research is increasingly valued as part of the evidence for policy and practice, but how it should be appraised is contested. Various appraisal methods, including checklists and other structured approaches, have been proposed but rarely evaluated. We aimed to compare three methods for appraising qualitative research papers that were candidates for inclusion in a systematic review of evidence on support for breast-feeding.

Method: A sample of 12 research papers on support for breast-feeding was appraised by six qualitative reviewers using three appraisal methods: unprompted judgement, based on expert opinion; a UK Cabinet Office quality framework; and CASP, a Critical Appraisal Skills Programme tool. Papers were assigned, following appraisals, to 1 of 5 categories, which were dichotomized to indicate whether or not papers should be included in a systematic review. Patterns of agreement in categorization of papers were assessed quantitatively using statistics, and qualitatively using cross-case analysis.

Results: Agreement in categorizing papers across the three methods was slight ( =0.13; 95% CI 0.06–0.24). Structured approaches did not appear to yield higher agreement than that by unprompted judgement. Qualitative analysis revealed reviewers' dilemmas in deciding between the potential impact of findings and the quality of the research execution or reporting practice. Structured instruments appeared to make reviewers more explicit about the reasons for their judgements.

Conclusions: Structured approaches may not produce greater consistency of judgements about whether to include qualitative papers in a systematic review. Future research should address how appraisals of qualitative research should be incorporated in systematic reviews.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

				D. S. Collingridge and E. E. Gantt The Quality of Qualitative Research American Journal of Medical Quality, September 1, 2008; 23(5): 389 - 395. [Abstract] [PDF]

				D. J. Cohen and B. F. Crabtree Evaluative Criteria for Qualitative Research in Health Care: Controversies and Recommendations Ann. Fam. Med, July 1, 2008; 6(4): 331 - 339. [Abstract] [Full Text] [PDF]