ࡱ> z|wxybjbjVV .<<%X X 8dk^}"zXL(z||||||$D$zzD$D$(J*J*J*D$zJ*D$zJ*J*@@ g,(ĄFfݒ< 6(@@"b:J* !)D$D$D$D$X a:Chapter 4 Critical appraisal of qualitative research This chapter should be cited as: Hannes K. Chapter 4: Critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group, 2011. Available from URL  HYPERLINK "http://cqrmg.cochrane.org/supplemental-handbook-guidance" http://cqrmg.cochrane.org/supplemental-handbook-guidance Key points Critical appraisal of qualitative studies is an essential step within a Cochrane Intervention review that incorporates qualitative evidence. The overarching goal of critical appraisal in the context of including qualitative research in a Cochrane Intervention Review is to assess whether the studies actually address questions under meaning, process and context in relation to the intervention and outcomes under review. Review teams should use a critical appraisal instrument that is underpinned by a multi-dimensional concept of quality in research and hence includes items to assess quality according to several domains including quality of reporting, methodological rigour and conceptual depth and bread. Critical appraisal involves (i) filtering against minimum criteria, involving adequacy of reporting detail on the data sampling, -collection and-analysis, (ii) technical rigour of the study elements indicating methodological soundness and (iii) paradigmatic sufficiency, referring to researchers responsiveness to data and theoretical consistency. When choosing an appraisal instrument a Review teams should consider the available expertise in qualitative research within the team and should ensure that the critical appraisal instrument they choose is appropriate given the review question and the type of studies to be included. Reviewers need to clarify how the outcome of their critical appraisal exercise is used with respect to the presentation of their findings. The inclusion of a sensitivity analysis is recommended to evaluate the magnitude of methodological flaws or the extent to which it has a small rather than a big impact on the findings and conclusions. Introduction Considerable debate exists on whether or not concepts such as validity and reliability apply to qualitative research and if so how they could be assessed. Some researchers have stated that qualitative research should establish validity, reliability and objectivity. Others plead for an adjustment of these concepts to better fit the qualitative research design. As a consequence, critical appraisal instruments might differ in the criteria they list to complete a critical appraisal exercise. Some researchers consider appraisal instruments a tool that can be utilized as part of the exploration and interpretation process in qualitative research (Popay et al, 1998; Spencer, 2003). Edwards et al (2002) describes the use of a signal to noise approach, where a balance is sought between the methodological flaws of a study and the relevance of insights and findings it adds to the overall synthesis. Other researchers do not acknowledge the value of critical appraisal of qualitative research, stating that it stifles creativity (Dixon-Woods, 2004). While recognising that all these views have some basis for consideration certain approaches succeed in positioning the qualitative research enterprise as one that can produce a valid, reliable and objective contribution to evidence synthesis. It is these that may therefore have more potential to be generally accepted within the context of producing Cochrane Intervention Reviews. The Cochrane Collaboration recommends a specific tool for assessing the risk of bias in each included study in an intervention review, a process that is facilitated through the use of appraisal instruments addressing the specific features of the study design and focusing on the extent to which results of included studies should be believed. This suggest that in assessing the methodological quality of qualitative studies the core criterion to be evaluated is researcher bias. Believability in this context refers to the ability and efforts of the researcher to make his or her influence and assumptions clear and to provide accurate information on the extent to which the findings of a research report hold true. However, it is the actual audit trail provided by researchers that allows for an in-depth evaluation of a study. Most existing appraisal instruments use broader criteria that account for reporting issues as well. We suggest that these issues should be part of the appraisal exercise. Currently, there are four possibilities to make use of qualitative research in the context of Cochrane Intervention reviews: The use of qualitative research to define and refine review questions a Cochrane Review (informing reviews). The use of qualitative research identified whilst looking for evidence of effectiveness (enhancing reviews). The use of findings derived from a specific search for qualitative evidence that addresses questions related to an effectiveness review (extending reviews). Conducting a qualitative evidence synthesis to address questions other than effectiveness (supplementing reviews). The latter use (Supplementing) is beyond the scope of current Cochrane Collaboration policy (Noyes et al, 2008). Stand alone qualitative reviews that supplement Cochrane Intervention reviews need to be conducted and published outside of the Cochrane context. Critical appraisal applies to all of the above possibilities. Reviewers should bear in mind that narratives used in reports of quantitative research cannot be considered qualitative findings if they do not use a qualitative method of datacollection and analysis.Therefore, critical appraisal based on instruments developed to assess qualitative studies is not applicable toreports that do not meet the criteria of being a qualitative study.. This chapter breaks down in four sections. Section 1 addresses translated versions of core criteria such as validity, reliability, generalisibility and objectivity of qualitative studies. Section 2 presents an overview of different stages involved in quality assessment. Section 3 guides the researcher through some of the instruments and frameworks developed to facilitate critical appraisal and section 4 formulates suggestions on how the outcome of an appraisal of qualitative studies can be used or reported in a systematic review. Section 1: Core criteria for quality assessment Critical appraisal is the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision (Hill & Spittlehouse, 2003). Instruments developed to support quality appraisal usually share some basic criteria for the assessment of qualitative research. These include the need for research to have been conducted ethically, the consideration of relevance to inform practice or policy, the use of appropriate and rigorous methods and the clarity and coherence of reporting (Cohen & Crabtree, 2008). Other criteria are contested, such as the importance of addressing reliability, validity, and objectivity, strongly related to researcher bias. Qualitative research as a scientific process needs to be rigorous and trustworthy to be considered as a valuable component of Cochrane systematic review. Therefore an evaluation using such criteria is essential. Nevertheless we should acknowledge that the meaning assigned to these words may differ in the context of qualitative and quantitative research designs (Spencer et al, 2003). Does translation of terminology compromise critical appraisal? The concepts used in table 1 are based on Lincoln and Gubas (1985) translation of criteria to evaluate the trustworthiness of findings. Acknowledging the difference in terminology does not obviate the rationale or process for critical appraisal. There might be good congruence between the intent of meanings relevant to key aspects of establishing study criteria, as demonstrated in table 1. Table 1: Criteria to critically appraise findings from qualitative research AspectQualitative TermQuantitative TermTruth valueCredibilityInternal ValidityApplicabilityTransferabilityExternal Validity or generalisibilityConsistencyDependabilityReliabilityNeutralityConfirmabilityObjectivity This scheme outlines some of the core elements to be considered in an assessment of the quality of qualitative research. However, the concept of confirmability might not be applicable to approaches inspired by phenomenology or critical paradigms in which the researchers experience becomes part of the data (Morse, 2002). The choice of critical appraisal instruments should preferably be inspired by those offering a multi-dimensional concept of quality in research. Apart from methodological rigour, that would also include quality of reporting and conceptual depth and bread. What indications are we looking for in an original research paper? There are a variety of evaluation techniques that authors might have included in their original reports, that facilitate assessment by a reviewer and that are applicable to a broad range of different approaches in qualitative research. However, it should be stated that some of the techniques listed only apply for a specified set of qualitative research designs. Assessing Credibility: Credibility evaluates whether or not the representation of data fits the views of the participants studied, whether the findings hold true. Evaluation techniques include: having outside auditors or participants validate findings (member checks), peer debriefing, attention to negative cases, independent analysis of data by more than one researcher, verbatim quotes, persistent observation etc. Assessing Transferability: Transferability evaluates whether research findings are transferable to other specific settings. Evaluation techniques include: providing details of the study participants to enable readers to evaluate for which target groups the study provides valuable information, providing contextual background information, demographics, the provision of thick description about both the sending and the receiving context etc. Assessing Dependability: Dependability evaluates whether the process of research is logical, traceable and clearly documented, particularly on the methods chosen and the decisions made by the researchers. Evaluation techniques include: peer review, debriefing, audit trails, triangulation in the context of the use of different methodological approaches to look at the topic of research, reflexivity to keep a self-critical account of the research process, calculation of inter-rater agreements etc. Assessing Confirmability: Confirmability evaluates the extent to which findings are qualitatively confirmable through the analysis being grounded in the data and through examination of the audit trail. Evaluation techniques include: assessing the effects of the researcher during all steps of the research process, reflexivity, providing background information on the researchers background, education, perspective, school of thought etc. The criteria listed might generate an understanding of what the basic methodological standard is a qualitative study should be able to reach. However, a study may still be judged to have followed the appropriate procedures for a particular approach, yet may suffer from poor interpretation and offer little insight into the phenomenon at hand. Consequently, another study may be flawed in terms of transparency of methodological procedures and yet offer a compelling, vivid and insightful narrative, grounded in the data (Dixon-Woods et al, 2004). Defining fatal flaws and balancing assessment against the weight of a message remains a difficult exercise in the assessment of qualitative studies. As in quantitative research, fatal flaws may depend on the specific design or method chosen (Booth, 2001). This issue needs further research. Section 2: Stages in the appraisal of qualitative research Debates in the field of quality assessment of qualitative research designs are centred around a more theoretical approach to evaluating the quality of studies versus an evaluation of the technical adequacy of a research design. How far criteria-based, technical approaches offer significant advantages over expert intuitive judgement in assessing the quality of qualitative research is being challenged by recent evidence indicating that checklist-style approaches may be no better at promoting agreement between reviewers (Dixon-Woods, 2007). However, these appraisal instruments might succeed better in giving a clear explanation as to why certain papers have been excluded. Given the fact that few studies are completely free from methodological flaws, both approaches can probably complement each other. Is the use of a critical appraisal instruments sufficient in assessing the quality of qualitative studies enhancing Cochrane intervention reviews? Three different stages can be identified in a quality assessment exercise: filtering, technical appraisal and theoretical appraisal. The first stage links to the inclusion criteria of study types that should be considered to enhance or extent Cochrane Reviews and requires no specific expertise. The required expertise for the next two stages ranges from a basic understanding of qualitative criteria to be able to critically appraise studies to a more advanced level of theoretical knowledge on certain approaches used. Stage 1: Filtering: Within the specific context of enhancing or extending Cochrane Reviews, and viewing critical appraisal as a technical and paradigmatic exercise, it is worth considering limiting the type of qualitative studies to be included in a systematic review. We suggest restricting included qualitative research reports to empirical studies with a description of the sampling strategy, data collection procedures and the type of data-analysis considered. This should include the methodology chosen and the methods or research techniques opted for, which facilitates the systematic use of critical appraisal as well as a more paradigmatic appraisal process. Descriptive papers, editorials or opinion papers would generally be excluded. Stage 2: Technical appraisal: Critical appraisal instruments should be considered a technical tool to assist in the appraisal of qualitative studies, looking for indications in the methods or discussion section that add to the level of methodological soundness of the study. This judgement determines the extent to which the reviewers may have confidence in the researchers competence in being able to conduct research that follows established norms (Morse, 2002) and is a minimum requirement for critical assessment of qualitative studies. Criteria include but are not limited to the appropriateness of the research design to meet the aims of the research, rigour of data-collection and analysis, well-conducted and accurate sampling strategy, clear statements of findings, accurate representation of participants voices, outline of the researchers potential influences, background, assumptions, justifications of the conclusion or whether or not it flows from the data, value and transferability of the research project etc. For this type of appraisal one needs to have a general understanding of qualitative criteria. Involving a researcher with a qualitative background is generally recommended. Stage 3: Theoretical appraisal: In addition to assessing the fulfillment of technical criteria we suggest a subsequent, paradigmatic approach to judgment, with a focus on the research paradigm used in relation to the findings presented. Although some critical appraisal instruments integrate criteria related to theoretical frameworks or paradigms most of them are pragmatic. These do little to identify the quality of the decisions made, the rationale behind them or the responsiveness or sensibility of the researcher to the data. Therefore, a consideration of other criteria should be considered. This would e.g. include an evaluation of methodological coherence or congruity between paradigms that guide the research project and the methodology and methods chosen, an active analytic stance and theoretical position, investigator responsiveness and openness and verification, which refers to systematically checking and confirming the fit between data gathered and the conceptual work of analysis and interpretation (Morse et al, 2002). For this type of overall judgment a more in-depth understanding of approaches to qualitative research is necessary. It is therefore recommended that a researcher with experience of qualitative research -who can guide others through the critical appraisal process- is invited. Experienced methodologists may have valuable insights into potential biases that are not at first apparent. It should be mentioned though that the need for a paradigmatic input might depend on the type of synthesis chosen. The Cochrane Qualitative Research Methods group recommends stage 3 whenever the instrument chosen for stage 2 does not cover for a paradigmatic approach to judgment. Other considerations include involving people with content expertise for the evaluation exercise. They are believed to give more consistent assessments, which is in line with what the Cochrane Collaboration suggests for the assessment of risk of bias in trials (Oxman et al, 1993). Section 3: A selection of instruments for quality assessment A range of appraisal instruments and frameworks is available for use in the assessment of the quality of qualitative research. Some are generic, being applicable to almost all qualitative research designs; others have specifically been developed for use with certain methods or techniques. The instruments also vary with regard to the criteria that they use to guide the critical appraisal process. Some address paradigmatic aspects related to qualitative research, others tend to focus on the quality of reporting more than theoretical underpinnings. Nearly all of them address credibility to some extent. The list with examples presented below is not exclusive with many instruments still in development or yet to be validated and others not yet commonly used in practice. It draws on the findings of a review of published qualitative evidence syntheses (Dixon-Woods et al, 2007) and the ongoing update of it. Reviewers need to decide for themselves which instrument appears to be most appropriate in the context of their review and use this judgement to determine their choice. Researchers with a quantitative background also need to consider an input from a researcher familiar with qualitative research, even when an appraisal instrument suitable for novices in the field is opted for. Which instruments or frameworks are out there? Checklists embedded in a software program to guide qualitative evidence synthesis: Some evidence synthesis organisations have developed and incorporated a checklist in the software they make available to assist reviewers with the synthesis of qualitative findings. Typically, potential reviewers need to register to be able to use it. However, the instruments are also available outside the software program on the websites of both organisations. Examples: QARI software developed by the Joanna Briggs Institute, Australia URL:  HYPERLINK "http://www.joannabriggs.edu.au/services/sumari.php" http://www.joannabriggs.edu.au/services/sumari.php Used by: Pearson A, Porritt KA, Doran D, Vincent L, Craig D, Tucker D, Long L, Henstridge V. A comprehensive systematic review of evidence on the structure, process, characteristics and composition of a nursing team that fosters a healthy environment. International Journal of Evidence-Based Healthcare 2006; 4(2): 118-59. Rhodes LG et al.Patient subjective experience and satisfaction during the perioperative period in the day surgery setting: a systematic review. Int J Nurs Pract 2006; 12(4): 178-92. EPPI-reviewer developed by the EPPI Centre, United Kingdom URL:  HYPERLINK "http://eppi.ioe.ac.uk/eppireviewer/login.aspx" http://eppi.ioe.ac.uk/eppireviewer/login.aspx Used by: Bradley P, Nordheim L, De La Harpa D, Innvaer S & Thompson C. A systematic review of qualitative literature on educational interventions for evidence-based practice. Learning in Health & Social Care 2005: 4(2):89-109. Harden A, Brunton G, Fletcher A, Oakley A. Teenage pregnancy and social disadvantage: a systematic review integrating trials and qualitative studies. British Medical Journal Oct 2009. Other online available appraisal instruments: Most of the instruments in this selection are easily accessible and clearly define what is meant by each individual criterion listed. As such, they may be particularly useful if reviewers with little experience of qualitative research are required to complete an assessment. Examples: Critical Appraisal Skills Programme (CASP):  HYPERLINK "http://www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdf" http://www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdf Used by: Kane GA et al. Parenting programmes: a systematic review and synthesis of qualitative research. Child Care Health and Development 2007; 33(6): 784-793. Modified versions of CASP, used by: Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J. Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science and Medicine 2003; 56: 671-84. Malpass A, Shaw A, Sharp D, Walter F, Feder G, Ridd M, Kessler D. Medication career" or "Moral career"? The two sides of managing antidepressants: A meta-ethnography of patients' experience of antidepressants. Soc Sci Med. 2009; 68(1):154-68. Quality Framework UK Cabinet Office  HYPERLINK "http://www.gsr.gov.uk/downloads/evaluating_policy/a_quality_framework.pdf" http://www.gsr.gov.uk/downloads/evaluating_policy/a_quality_framework.pdf Used by: MacEachen E et al. Systematic review of the qualitative literature on return to work after injury. Scandinavian Journal of Work Environment & Health 2006; 32(4): 257-269. Evaluation Tool for Qualitative Studies http://www.fhsc.salford.ac.uk/hcprdu/tools/qualitative.htm Used by: McInnes RJ & Chambers JA. Supporting breastfeeding mothers: qualitative synthesis. Journal of Advanced Nursing 2008; 62(4): 407-427. Checklists developed by academics and commonly used in published qualitative evidence syntheses: Such checklists have been selected and utilised by other researchers in the specific context of an evidence synthesis. Examples: The Blaxter (1996) criteria for the evaluation of qualitative research papers, used by: Gately C et al. Integration of devices into long-term condition management: a synthesis of qualitative studies. Chronic Illn 2008; 4(2): 135-48. Khan N et al. Guided self-help in primary care mental health - Meta-synthesis of qualitative studies of patient experience. British Journal of Psychiatry 2007; 191: 206-211. The Burns (1989) standard for qualitative research, used by: Barrosso J, Powell Cope GM. Meta-synthesis of qualitative research on living with HIV infection. Qualitative Health Research 2000; 10: 340-53. Thorne S, Paterson B. Shifting images of chronic illness. Image: Journal of Nursing Scholarship 1998: 30; 173-8. Hildingh C et al. Women's experiences of recovery after myocardial infarction: a meta-synthesis. Heart Lung 2007; 36(6): 410-7. Howard AF, Balneaves LG, Bottorff JL. Ethnocultural womens experiences of breast cancer: a qualitative meta-study. Cancer nursing 2007 30(4): E27-35. The Popay et al (1998) criteria, used by: Attree P. Low-income mothers, nutrition and health: a systematic review of qualitative evidence. Maternal and Child Nutrition 2005 1(4): 227-240. Sim J & Madden S. Illness experience in fibromyalgia syndrome: A metasynthesis of qualitative studies." Social Science & Medicine 2008; 67(1): 57-67. Yu D et al. Living with chronic heart failure: a review of qualitative studies of older people. J Adv Nurs 2008; 61(5): 474-83. The Mays & Pope (2000) criteria, used by: Humphreys A et al. A systematic review and meta-synthesis: evaluating the effectiveness of nurse, midwife/allied health professional consultants. Journal of Clinical Nursing 2007; 16(10): 1792-1808. Metcalfe A et al. Family communication between children and their parents about inherited genetic conditions: a meta-synthesis of the research. Eur J Hum Genet 2008; 16(10): 1193-200. Robinson L. & Spilsbury K. Systematic review of the perceptions and experiences of accessing health services by adult victims of domestic violence. Health Soc Care Community 2008; 16(1): 16-30. Section 4: Integrating outcomes of critical appraisal in a systematic review. In a best case scenario a qualitative synthesis or primary study will achieve a positive assessment or score for each of the criteria against which it has been assessed according to the critical appraisal instrument used. However, this will most likely not be the case for the majority of studies and researchers need to be aware of the fact that the assessment or score might depend on the instrument that has been used, which increases the value of involving a researcher with a qualitative background in the appraisal process. For studies that fail to report sufficient information or it is clear that the study is weak when matched against a certain criterion e.g. because of a methodological flaw a decision needs to be made whether to include the study or not. How to use and report the critical appraisal outcome? To include or exclude a study: In this particular case, only high quality studies are included. The potential risk is that valuable insights are excluded from the synthesis. Studies rated as low quality because of methodological flaws or lack of reporting may nevertheless generate new insights, grounded in the data, while methodological sound studies may suffer from poor interpretation of data, leading to an insufficient insight into the phenomenon under study (Dixon-Woods, 2007). This approach was used by Carlsen et al (2007), who excluded 5 out of 17 studies following quality appraisal. Potential format of presentation: Study/Criterion*Study 1Study 2Study 3Study 4Study 5Crit 1x//x/Crit 2xx/xxCrit 3xxxxxCrit 4x?/xxCrit 5xxx?xQuality rating H/L**HJudge!***LHJudge!CommentsMotivate****MotivateMotivate *Authors may choose to give more weight to certain criteria and use this in their final judgment. ** H/L= High/Low *** For studies that are clearly on the verge between in- and exclusion researchers a judgement on whether to include or exclude should be made and discussed with potential co-reviewers. **** Authors should include a motivation for in- or exclusion, particularly for those cases where judgments are being made. To give more weight to studies that scored high on quality: In this particular case, all valuable insights remain included. However, it might be complex to report on the findings of the synthesis given the subgroups of studies. One strategy to cope with the issue of weighing studies is to report the findings of differently rated studies in separate sections. However, this has an impact on the potential richness of the presented synthesis, especially in those approaches generating new theory based on all of the relevant and illuminating findings. No fixed parameters currently exist to determine the weight of qualitative studies. Reviewers choosing this approach need to evaluate which methodological flaws have a substantial impact on the findings presented. The key issue to consider is the extent to which the quality of reporting and choices made by the authors is acceptable in terms of inclusion of the evidence in the Cochrane review which it aims to enhance. Potential format of presentation: Study/ Criterion*Study 1 Subgroup 1Study 2 Subgr. 1/2Study 3 Subgr. 2Study 4 Subgr. 1Study 5 Subgr. 1/2Crit 1x//x/Crit 2xx/xxCrit 3xxxxxCrit 4x?/xxCrit 5xxx?xQual. rating H/LHJudge!LHJudge!CommentsMotivateMotivateMotivate Both approaches could benefit from a sensitivity analysis evaluating what happens to the findings of a study when low or high quality studies are removed. Thomas et al (2004) conducted such a sensitivity analysis and found that the findings of three studies rated as low quality did not contradict those from studies of a higher quality. This was confirmed by Noyes and Popays (2007) study on directly observed therapy and tuberculosis. The synthesis would have come to the same conclusions with or without their inclusion. It indicates that there might be little to gain from including lower quality studies in a synthesis (Harden, 2008). To describe what has been observed without excluding any studies: In this particular case, all potential valuable insights remain included, because the worth of individual studies might only become recognisable at the point of synthesis rather than in the phase of appraisal. In this approach, the responsibility for evaluating the quality of the studies is devolved to the reader from the researcher. The Cochrane Qualitative Research Methods Group sees value in all of these approaches. However, in line with current Cochrane policy, when conducting a Cochrane Intervention review and integrating qualitative evidence we recommend the two first approaches emphasizing the methodological soundness of studies rather than their contribution to science in general. The decision lies with the review team. Regardless of the approach eventually chosen for the quality assessment stage of the review there is a need to preserve the transparency of the method through careful documentation of decisions made. The convention of using at least two researchers for the quality assessment process is a useful legacy from quantitative-based review processes; not so much for inter-rater consistency purposes but, at the very least, to open up the data to a broader range of possible interpretations. Conclusion Quality assessment of qualitative research studies remains a contested area. While considerable widespread debate continues around the feasibility and utility of critical appraisal it is nevertheless possible to make recommendations within the specific context of informing, enhancing and extending a Cochrane Review. Balancing assessment against the weight of a message is a difficult exercise. In assessing the quality of an original study reviewers should focus on quality of reporting, methodological rigour and conceptual depth and bread of studies. Review Teams should deliberately seek indications that demonstrate rigour truth-and trustworthiness in the studies (section 1). Filtering, technical appraisal and theoretical appraisal are the three main stages in a critical appraisal exercise. Review teams need to make sure that they have the necessary expertise on board to be able to complete each phase of the critical appraisal exercise (section 2). In choosing an assessment instrument review teams need to consider the appropriateness of their choice in the context of their review and be aware of the fact that whether or not a study meets the standard might depend on the instrument used (section 3). Review teams can opt to use the outcome of their critical appraisal to in- or exclude studies, to give high quality studies more weight or to inform themselves without excluding any. Regardless of the approach eventually chosen there is a need to preserve the transparency of the method through careful documentation of decisions made. The use of at least two researchers to complete the critical appraisal process is recommended (section 4). References Blaxter M. Criteria for evaluation of qualitative research. Medical Sociology News 1996; 22: 68-71. Booth A. Cochrane or cock-eyed? How should we conduct systematic reviews of qualitative research? Qual EBP conference, Coventry university, may 14-16, 2001. Burns N. Standards for Qualitative Research. Nurs Sci Q 1989; 2(44): 44-52. Carlsen B, Glenton C, Pope C. Thou shalt versus thou shalt not: a meta-synthesis of GPs' attitudes to clinical practice guidelines. Br J Gen.Pract 2007;57:971-8. Cohen DJ & Crabtree BF. Evaluative criteria for qualitative research in health care; controversies and recommendations. Annals of Fam. Med 2008; 6 (4): 331-39. Dixon-Woods M, Booth A, Sutton AJ. Synthesizing qualitative research: a review of published reports. Qual Res 2007; 7: 375-421. Dixon-Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative research. QSHC 2004;13: 223-225. Dixon-Woods M, Booth A, Sutton AJ. Synthesizing qualitative research: a review of published reports. Qualitative Research 2007; 7(3): 375-422. Dixon-Woods M, Sutton AF, Shaw RL, Miller T, Smith J, Young B, Bonas S, Booth A, Jones DR. Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods. Journal of Health Service Research and Policy 2007; 12: 42-7. Edwards A, Elwyn G, Hood K, Rollnick S. Judging the 'weight of evidence' in systematic reviews: introducing rigour into the qualitative overview stage by assessing Signal and Noise. J Eval Clin Pract. 2000 May;6(2):177-84. Lincoln YS, Guba EG. Naturalistic Inquiry, Sage, Newbury Park, CA, 1985 . Guba EG. & Lincoln YS. Fourth Generation evaluation. Newbury Park. CA: Sage. 1989. Harden A. Critical Appraisal and Qualitative Research: Exploring sensitivity analysis. ESRC Research Methods Festival, St. Catherines College Oxford, 30th June-3rd July, 2008. Hill A & Spittlehouse C. What is critical appraisal? Evidence-Based Medicine 2003; 3 (2): 1-8. Noyes J & Popay J. Directly observed therapy and tuberculosis: how can a systematic review of qualitative research contribute to improving services? A qualitative meta-synthesis. Journal of Advanced Nursing 2007; 57(3), 227243. Morse JM, Barett M, Mayan M, Olson K, Spiers J. Verification strategies for establishing reliability and validity in qualitative research. Int J of Qual Meth 2002; 1 (2):spring. Mays N, Pope C. Qualitative research in health care Assessing quality in qualitative research. BMJ 2000;320:50-52. Noyes J, Popay J, Pearson A, Hannes K, Booth A. Chapter 20: Qualitative research and Cochrane reviews. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane-handbook.org. Oxman AD, Guyatt GH. The science of reviewing research. Annals of the New York Academy of Sciences 1993; 703: 125-133. Popay, J.,& Williams, G. (1998). Qualitative research and evidence based healthcare. Journal of Research in Social Medicine,91(Suppl 35), 3237. Popay J, Rogers A, Williams G. Rationale and Standards for the Systematic Review of Qualitative Literature in Health Care. Qual Health Research 1998; 8: 341-51. Spencer L, Ritchie J, Lewis J et al. et al. Quality in Qualitative Evaluation: a framework for assessing research evidence: a framework for assessing research evidence. Government chief social researchers office 2003. Thomas J, Harden A, Oakley Am Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J. Integrating qualitative research with trials in systematic reviews. BMJ 2004; 328: 1010-1012.  URL: http://www.joannabriggs.edu.au/cqrmg/tools_3.html. A detailed guide on how to conduct a QARI supported Systematic Review, including a detailed explanation of the10 critical appraisal criteria, can be found on the JBI-website: http://www.joannabriggs.edu.au/pdf/sumari_user_guide.pdf URL for instrument on process evaluation: http://eppi.ioe.ac.uk/cms/default.aspx?tabid=2370&language=en-US. Tools exist which help to assess quality along three dimensions: quality of reporting, sufficiency of strategies for increasing methodological rigour, and the extent to which study methods and findings are appropriate to answer the review question (For an example, see Harden et al 2009 study).  The loss of potential valuable studies is less likely if one would opt for a critical appraisal instrument which assesses conceptual depth and breadth of findings as well as methodological rigour. Currently, there is no guidance on how these two aspects might be balanced out. Sensitivity analyses could be considered.     PAGE  PAGE 1     678WcZ ￫hKh*hAh_h_0J$6B*CJOJQJ^JaJfHphq 8h5(0J$B*CJOJQJ^JaJfHphq >h_h_0J$B*CJOJQJ^JaJfHphq Dh_h_0J$5B*CJOJQJ\^JaJfHphq 'h_h_0J$5CJOJQJ^JaJhvjh55CJ aJ mH sH hvjh i5CJ aJ mH sH hvjh_5CJ aJ mH sH hvjh]5CJ aJ mH sH 78I J K V W  {oy & FgdrpgdIgd]h^hgd5 & Fgdg & Fgd]  & F gd_gds?   G H I J K V W ˫ː|lZH:/$h_ht 2mH sH h_h4YmH sH h_h]56mH sH "hvjh]56CJaJmH sH "h_h3` 56CJaJmH sH h_h]5CJaJmH sH 'h_h_0J$5CJOJQJ^JaJ5h_h_0JCJOJQJ^JaJfHq >h_h_0J$B*CJOJQJ^JaJfHphq Gjh_h_0J$B*CJOJQJU^JaJfHphq h_h_CJOJQJ^JaJ N [     0z{#Ծ߾Ծub%hvjh$I>56CJ\aJmH sH h_h3` 56\mH sH h_h$I>5\mH sH h_h ~mH sH h_h$pmH sH h_hgmH sH h_h%OmH sH h_h]mH sH h_h$}PmH sH h_h9mH sH h_h,mH sH h_h imH sH h_h4YmH sH "6Gdg߽άάΛuߛdSS h_hkCJOJQJ^JaJ h_hHCJOJQJ^JaJ)h_hOB*CJOJQJ^JaJph h_hOCJOJQJ^JaJ h_h9CJOJQJ^JaJ h_h3LCJOJQJ^JaJ h_hIUCJOJQJ^JaJ h_hz}CJOJQJ^JaJ h_h$I>CJOJQJ^JaJh_h$I>56\mH sH   Qg.:BgͼޚޚޚމxމggVVgVE h_hW`CJOJQJ^JaJ h_h"4CJOJQJ^JaJ h_hkCJOJQJ^JaJ h_hb CJOJQJ^JaJ h_hHCJOJQJ^JaJ h_h,CJOJQJ^JaJ h_hOCJOJQJ^JaJ h_h3LCJOJQJ^JaJ h_h:CJOJQJ^JaJ h_h$I>CJOJQJ^JaJ h_h9CJOJQJ^JaJ3;>Mi %(GYZmncdw )ͼͼ޼wwwwfU h_h5)CJOJQJ^JaJ h_h XCJOJQJ^JaJ#h_hl<5CJOJQJ^JaJ h_hl<CJOJQJ^JaJ h_h,CJOJQJ^JaJ h_hrpCJOJQJ^JaJ h_h4CJOJQJ^JaJ h_h?]QCJOJQJ^JaJ h_hW`CJOJQJ^JaJ h_h"4CJOJQJ^JaJ!)AHJUZ[^iu23Wir޼淪ޅͅtctcR h_h7CJOJQJ^JaJ h_h4YCJOJQJ^JaJ h_h^tCJOJQJ^JaJ h_h:yCJOJQJ^JaJ h_hl<CJOJQJ^JaJ(h_h,CJOJQJ^JaJmH sH  h_h XCJOJQJ^JaJ h_h5)CJOJQJ^JaJ h_h,CJOJQJ^JaJ h_hrpCJOJQJ^JaJ3!!!""U&V&&&!("(#(o(p(w((( $Ifgd $Ifgd5gdMwhgds?h^hgd5gdl<gdIgdrp - K Z [ q s | } !?!D!M!N!W!X!!!!!!!!޻ުުwުުުfU h_h9CJOJQJ^JaJ h_h 9CJOJQJ^JaJ h_h:CJOJQJ^JaJ h_hkCJOJQJ^JaJ h_h7CJOJQJ^JaJ h_h ;SCJOJQJ^JaJ h_h<CJOJQJ^JaJ#h_h]5CJOJQJ^JaJ h_h]CJOJQJ^JaJ h_ht2CJOJQJ^JaJ!!!!!"""" ###D#^#k#n#############$$$$2$8$N$W$xmxmxmxmxmmmbmWxh_hs?mH sH h_h7mH sH h_hF mH sH h_hAmH sH h_h5mH sH h_h ;SmH sH h_hMwhmH sH h_h3` mH sH "hvjh3` 56CJaJmH sH "hvjh56CJaJmH sH "hvjh!56CJaJmH sH h_h3` 56mH sH h_h95mH sH "W$_$}$~$$$$$%%%%%%%&&+&<&M&R&T&V&&&&&&&&&&&'''q''' (!(ɾɳߨߨߨԝ~ԝsshhh_h7mH sH h_hQ";mH sH hvjhMwhCJaJmH sH hvjhMwh5CJaJmH sH h_hMwhmH sH h_hF mH sH h_hmH sH h_hs?mH sH h_hAmH sH h_h 9mH sH h_h<mH sH h_h!mH sH h_h5mH sH (!("(#(n(o(p(((( ))2)=)Y)Z)[)f)))))****++++++źźźźxm]M]xh_h3` 5CJaJmH sH h_h<5CJaJmH sH h_h WmH sH h_h4YmH sH h_hmH sH h_h mH sH h_hxmH sH h_h<mH sH h_hAhmH sH h~h<mH sH h~h<5mH sH h_hI5mH sH h_hs?5mH sH h_h3` 5mH sH h_hImH sH (((((qhh_ $Ifgd $Ifgd5kd$$IflF ,"   t0"6    44 layt~(((( )qhh_ $Ifgd $Ifgd5kd$$IflF ,"   t0"6    44 layt~ ) ))%)1)qhh_ $Ifgd $Ifgd5kd$$IflF ,"   t0"6    44 layt~1)2)=)L)X)qhh_ $Ifgd $Ifgd5kd$$IflF ,"   t0"6    44 layt~X)Y)Z)++++T-U-.qllgbbbbZ & Fgds?gds?gd WgdAhkd4$$IflF ,"   t0"6    44 layt~ ++, ,2,O,X,,,,4-@-S-T-U-_-j-l-w---------..P.`.b.o.q....ɽttiiti^th_hQ";mH sH h_hmH sH h_hs?mH sH h_h~snmH sH h_hx6mH sH h_hxmH sH h_hx5>*mH sH h_h 5>*mH sH h_hs?>*mH sH h_hImH sH h_hb%mH sH h_h 9mH sH h_hMV)mH sH h_h<mH sH $....///"/J/r/s///0X0000000011111111728292c2ø߸ߡߊߡtit^h_hE%mH sH h_hg mH sH h_hmH sH h_h9mH sH h_h96mH sH h_hQ";mH sH h_hmH sH h_h~sn6mH sH h_h~snmH sH h_hx5>*mH sH h_h 5>*mH sH h_hxmH sH h_hs?mH sH h_huFmH sH !..0022b4c477777;;;;===@^gd] & Fgd]gd7Pgd09h^hgdgd] h^hgd09gd 9gdx & Fgds?h^hgd5c2~22222222222s33333a4b467777ƸԭԖ߀ui]Oh_h3` 56mH sH h_hc`35mH sH h_h095mH sH h_hLmH sH h_hE%mH sH h_hmH sH h_hs?mH sH h_h96mH sH h_h9mH sH h_hx5>*mH sH h_h 5>*mH sH h_hxmH sH h_hmH sH h_hQ";mH sH h_h 9mH sH 777777G8M889::7:H:i::::;;;;o;;ʸujuju^jN>h_h3` 5CJaJmH sH h_h W5CJaJmH sH h_h09>*mH sH h_h3` mH sH h_h"4mH sH h_hmH sH h_h$JmH sH h_hK7mH sH h_h09mH sH h_h$B5mH sH "hvjh56CJaJmH sH "hvjh$B56CJaJmH sH "hvjh W56CJaJmH sH "hvjh3` 56CJaJmH sH ;;;;;;;5<D<O<s<<<<=E=o===========Ͽϴ|pbTIIh_h$BmH sH h_h$B5>*mH sH h_h3` 5>*mH sH h_h5mH sH h_h7P5mH sH h_h5)mH sH h_hi3mH sH h_h7PmH sH h_hRmH sH h_h3` mH sH h_h W5CJaJmH sH h_h3` 5CJaJmH sH h_h4Y5CJaJmH sH h_hb 5CJaJmH sH ===>>>$>`>a>e>v>y>>>>? ??"?/?0?7?H?J?K?Y?d?e?i?z??????@@@F@X@Z@@@@@ԾԳԝɨɨԨԨwh_h3` 5>*mH sH h_h>*mH sH h_h$B>*mH sH h_h  mH sH h_hImH sH h_h@jhmH sH h_h7PmH sH h_hmH sH h_h$BmH sH h_h5)mH sH h_hi3mH sH h_hRumH sH ,@@@@@@@@@*AAAAAAAAAA/B=BNBSBYBZBlBmB{BBBxDDD˪˔~~~~ssh]R]h_hmH sH h_hmH sH h_h_OmH sH h_h7PmH sH h_hRumH sH h_h%OmH sH h_h6mH sH h_h  mH sH h_hmH sH h_hAmH sH h_h3` mH sH h_h$BmH sH h_h$B5>*mH sH h_h3` 5>*mH sH h_hi35>*mH sH  @@`EEoKpKL4M5M6MsMtMRRRRR SwTxT`gdK7^gd] & FgdK7h^hgd5gdgd5KgdW` 7$8$H$^gd & Fgd]^gd & Fgd]DDERE]E^E_E`EiEtE~EEEEEEEEEɽo^MF?F8 h_hi h_h h_hL h_h]CJOJQJ^JaJ h_h]CJOJQJ^JaJ#h_h$B>*CJOJQJ^JaJ&h_h$B5>*CJOJQJ^JaJ&h_hi35>*CJOJQJ^JaJ&h_h 5>*CJOJQJ^JaJh_h$B>*mH sH h_hmH sH h_h6mH sH h_hAmH sH h_h  mH sH h_h$BmH sH EEEEEMFOFYFwFFFFvGGaItIuIwIIIII JJJDJEJJJJJJK9KnKoKpKLLֳȳȳ֬֜r(h_h WCJOJQJ^JaJmH sH (h_hCJOJQJ^JaJmH sH  *h_h5KmH nH sH tH h_hW` h_h h_h h_h h_hA h_h  h_h5K h_h h_h h_hi h_hL h_h$B&LLLLLM/M1M2M3M4M5M6M9M[MrMsMtM뛐p^L:3 h_h5"hvjh56CJaJmH sH "hvjhR56CJaJmH sH "hvjh56CJaJmH sH "hvjh5K56CJaJmH sH h_hW`56mH sH h_h]mH sH  h_hW`CJOJQJ^JaJ(h_hokCJOJQJ^JaJmH sH (h_h6CJOJQJ^JaJmH sH (h_h CJOJQJ^JaJmH sH (h_h5KCJOJQJ^JaJmH sH tMMMMMMMMMMMMMMN N'N:NBNoNxNNNNNNNNNNOOOPP$PGPPP~PPQQ@Q`QQɾ߾ԾԾԾԾԾԾ꒾|qqh_himH sH h_h[RmH sH h_hYmH sH h_h8[>mH sH h_h'mH sH h_h5KmH sH h_hj`mH sH h_hmH sH h_h09mH sH h_hLmH sH h_h  mH sH h_hmH sH h_hK7mH sH ,QQQQ#RRRRRRRRRSS SS.STNTtTuTwTTTTɾ}}rgVgJ}?h_hSAmH sH h_hK75mH sH !jh_h0J#UmH sH h_hmH sH h_h"XmH sH h_hK7mH sH h_h]>*mH sH h_hK7>*mH sH h_hK75>*mH sH h_hMwh5CJaJmH sH h_hMwhmH sH h_hpAmH sH h_h=mH sH h_h$JmH sH h_hLmH sH h_himH sH xTT@UV9W:WuWWXYYZZZ[7\\\ 7$8$H$^gd]^gdc`gd8[> & F gd[R^gd&_"^gd`gdc^gdK7 7$8$H$^gdK7^gd]`gdK7TT U U U>U?U@UHUIUsVtVVVVVW#W)W*W,W9W:WuWzW{WWWWWWWWW!XXXXY干vkh_h&_"mH sH h_hmH sH #jh_hK7UmH sH h_hK75mH sH h_hK7\h_h8CsmH sH h_h8Cs h_hK7$h_hK70J>*B*mH phsH #jh_hK7UmH sH h_hK7mH sH jh_hK7UmH sH &YYYYYYYYYYYYZ%Z*Z/Z6ZKZeZZZZZZZZZZZ·¡–‹€€‹th\Ph_hL5mH sH h_h-5mH sH h_h8[>5mH sH h_h[R5mH sH h_hLmH sH h_h\mH sH h_h`VmH sH h_h'mH sH h_h]mH sH h_h[RmH sH h_himH sH h_h55>*mH sH h_h]5>*mH sH h_h>*mH sH h_hK7mH sH ZZZZZO[P[Q[[[[[[[[[!\'\)\7\Y\\\@]A]d]]^ʼѯѨyyqeZSLSLS h_hO"z h_h0vh_h0vmH sH h_h0v5mH sH h_h5\ h_h8Cs h_h5h_h5mH sH h_h8CsmH sH h_hYmH sH h_hrh_hVg0JB*phj/h_hVgU h_hVgjh_hVgUh_h"mH sH h_hL5mH sH h_h"5mH sH ^ ^&^'^(^7^8^\^]^^^^^__ __M_l_________`"`/`7`<`U`t````````ϸϱ⚒xjh_hi5>*mH sH h_hi7mH sH h_h(B\ h_h(Bh_h5h_h(B5h_h5\ h_hY h_h5h_h]0Jjh_h]Ujh_h]Uh_h095 h_h] h_h0v h_hO"zh_h0v0J(\\7^8^\^____````awup & F^gd(B`gd] 7$8$H$^gd(B^gd] hh^h`hgdc^gdzJEƀ@&^gd0v ````a a aa-a:aVa_ajalaaaaaaaaaaaa6bUbbbhbibvb}bʾʜ|t|mfmfmf^mSh_hljmH sH h_h0v\ h_hlj h_h0vh_h0v5h_hY5h_hc5mH sH h_hi5mH sH h_h[R5mH sH h_himH sH h_hLmH sH h_h8[>>*mH sH h_hi>*mH sH h_hi5>*mH sH h_h0v5>*mH sH h_hr5>*mH sH aaaavb$c%cccccddze{ee7ffMgNgyg 7$8$H$^gd 7$8$H$`gdc 7$8$H$^gd44^gdeq,`gdc 7$8$H$^gd0v^gdc`gd8[>}bbbbccc$c%cacbccccdudddddddze{eeeeeeef"f'f(f)f7f?fFfIffffff g,g8g>g@g积ؘ篧wwh_h \ h_h h_h445]h_h445h_h=5 h_h5h_h44\ h_h44h_h`VmH sH h_heq,mH sH h_heq,5mH sH  h_h]h_h0v\ h_hlj h_h0vh_h0vmH sH .@gMgNgRgmgngxgygggg h'h-h/h@hRhhhhhhhiiiiiiiiiiiiiǼ}qch_h]56mH sH h_hk5mH sH h_heq,mH sH  h_heq,h_h \h_h(B\ h_h(Bh_h5\ h_hlj h_h5h_h5mH sH h_hljmH sH h_h445h_h445mH sH h_h=5mH sH h_h] h_h #yg@hhiiii j jmmmImJmooooo $Ifgdygdy hh^h`hgd U & Fgdi?8^8gdk^gd8[> 7$8$H$^gdY 7$8$H$^gd(B 7$8$H$^gd5ii j j jjjjjjPj]jhjvjjjjjj k8kok}kkkkkl"l6l8llllllʾ|qqfq[|qh_h UmH sH h_hMwhmH sH h_h4mH sH h_hmH sH h_hpAmH sH h_hDZmH sH h_hi?mH sH h_hLmH sH h_himH sH h_hi5mH sH "hvjh7956CJaJmH sH "hvjhMwh56CJaJmH sH "hvjhZ@t56CJaJmH sH #lllllllmmmmm)m,mImJmimmmm n nnn:nOn`noo3o4o5oɾɮɂwpipipbipipQ!jh_hOc0J#UmH sH  h_h4 h_hL h_hl#h_hLmH sH h_hi?>*mH sH h_hpA5CJaJmH sH h_hMwh5CJaJmH sH h_hA:-5CJaJmH sH h_ho<mH sH h_hMwhmH sH h_hDZmH sH h_hi?mH sH h_hpAmH sH h_h4mH sH 5oPoWoiooooooooooooooopp p ppppppp#p%p'p(p/p1p5p7p9p:pApCpGpIpKpLpSpUpYp[p]p^pppƺܮh~hb ->*mH sH h~hb -5mH sH h~hZmH sH h~hb -mH sH h_hy>*mH sH h_h U>*mH sH h_h UmH sH h_hOcmH sH h_hymH sH  h_he h_h:h_h:mH sH 2oooopp p pppppppp!p#p%p'p(p/p1p3p5p7p9p:pFfFfFf# Ff $Ifgdy:pApCpEpGpIpKpLpSpUpWpYp[p]p^pspupppppppppppFfFfFfL $Ifgdypprpspup{p~ppppppppppppppppppppqq*qar{l]lNh_hOcCJaJmH sH h_h.CJaJmH sH h_h_eCJaJmH sH h_h CJaJmH sH h_hb*CJaJmH sH h_hb*mH sH h~hb*>*mH sH h~h.mH sH h~hb*5mH sH h~hb ->*mH sH h~hOcmH sH h~hb -5mH sH h~hb -mH sH h~hb*mH sH ppppq*qqarbr7v8vZv[vmvuvvvvvvvv $Ifgd/Ygd_e^gd_eh^hgdi? & Fgdi?gdb -h^hgdb*gd.Ffu! $Ifgdyarbrcrrrrr1s2s;s*mH sH h_hb -mH sH h_h UmH sH h_h`mH sH h_hmH sH h_hZmH sH h_h.mH sH h_h<&'mH sH h_hl#mH sH h_ha mH sH h_hi?mH sH h_hDZ>*mH sH h_hi?>*mH sH h_h_emH sH "[vmvvvvvvvvvvvvvvvvvvvvvvvvvww wwwwwww w"w#w4w6w?wAwHwIwRwSwewfwowpwqwrw0xǻǻǯh_htmH sH h_hb ->*mH sH h_hmH sH h~h>*mH sH h~h5mH sH h~hmH sH h~h_e>*mH sH h~h mH sH h~h_e5mH sH h~h_emH sH 4vvvvvvvvvvvvvvvvvvvvvvvvvwwFf!/Ff+FfK(Ff$ $Ifgd/Yw w wwwwwwww w"w#w4w6w=w?wAwHwIwRwSw\wewfwowpwFf<Ffb9Ff5Ff2 $Ifgd/Ypwqwrwyy{{    gdP%~gd/gdPVgdi?h^hgdt & Fgd gdth^hgdi?h^hgdZ0xxxxy=y>yfyyyyyy0z8z9z:z;zszyzzzzz { { {#{2{ƺuj_j_T_Ih_hmH sH h_hmH sH h_hHmH sH h_h$}PmH sH h_ha mH sH h_hl#mH sH h_hi?mH sH h_ha >*mH sH h_hDZ>*mH sH h_hi?>*mH sH h_ht>*mH sH h_htmH sH h_hYB*ph333h_hr>B*ph333h_htB*H*ph333h_htB*ph3332{3{4{5{={@{B{G{_{h{j{u{z{{{{{{{{{{i|j|m|||}   ɽweS"hvjhl#56CJaJmH sH "hvjha 56CJaJmH sH h_ht56mH sH h_hk_mH sH h_ho<mH sH h_h$}PmH sH h_htmH sH h_ht>*mH sH h_h >*mH sH h_hmH sH h_hHmH sH h_hi?mH sH h_hl#mH sH h_ha!mH sH  W8ςڂσڃ݃̄Ԅ+ԽԽԽԲꐂp^L"hvjho<56CJaJmH sH "hvjhT56CJaJmH sH "hvjhv56CJaJmH sH h_h?]Q56mH sH h_h]mH sH h_h;5mH sH h_hr>mH sH h_h;mH sH h_h 5mH sH h_h$}PmH sH h_h mH sH h_hYmH sH h_hPVmH sH h_ha mH sH  |1:;(/01ՊǼLjǓrcrcrTrch_h[RmH nH sH tH h_h@mH nH sH tH h_hQhmH nH sH tH h_h@h_h@mH sH  h_hP%~h_hQh\h_hQh] h_hQhh_h=mH sH h_hQhmH sH h_hP%~mH sH h_h'mH sH h_h'B*]ph333h_h'B*ph333h_h'pmH sH ;<12‰܊݊Z[ no 7$8$H$gdgd/ 7$8$H$gdrp dd[$\$gd^gdRu 7$8$H$gdQhgdP%~gdQhՊڊ܊݊ыYZ[ $mnoTдti^S^S^Dh_hmH nH sH tH h_h'pmH sH h_h/mH sH h_hrpmH sH h_hrpmH nH sH tH h_hQhH* h_hQh h_hP%~h_hP%~mH sH h_h]mH nH sH tH h_hmH nH sH tH h_hRu0JCJaJh_hRuCJaJ h_hRuh_hQhmH nH sH tH h_hjmH nH sH tH TU }/0‘cd?@""gdgdO!gdOgdOdPgd*gd/gdP%~ 7$8$H$gdTU <=hkz{|}C/‘cdkzncWOOOh_hj\h_h,HI5mH sH h_h,HImH sH h_hXO5mH sH h_hOaJh_h6]mH sH h_h6mH sH h_h5)6] h_h5)h_h5)mH sH h_h*mH sH  h_hjh_h*\ h_h*h_h*]h_h'pmH sH h_hP%~mH sH h_hmH sH >?@*ڔ۔ƻxƳmiaiaiaiaiWjhG0JUjh S,Uh S,hOchGmH sH h"XhGmH sH hShGmH sH  *hZbhGmH sH h%OhGmH sH h%OhG h* hGhGmH sH h* hGmH sH hGjhG0J#Uh_hOB*ph333h_hLmH sH h_h mH sH h_h/mH sH   gdOh]hgd &`#$gdVz h_hOB*ph333h S,h S,0JmHnHuhGjhG0JU hG0J ,1h/ =!"#$% $$If!vh5 5 5 #v :V l t0"65 yt~$$If!vh5 5 5 #v :V l t0"65 yt~$$If!vh5 5 5 #v :V l t0"65 yt~$$If!vh5 5 5 #v :V l t0"65 yt~$$If!vh5 5 5 #v :V l t0"65 yt~ADyK 3http://www.joannabriggs.edu.au/services/sumari.phpyK fhttp://www.joannabriggs.edu.au/services/sumari.php-DyK .http://eppi.ioe.ac.uk/eppireviewer/login.aspxyK \http://eppi.ioe.ac.uk/eppireviewer/login.aspx}DyK Dhttp://www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdfyK http://www.phru.nhs.uk/Doc_Links/Qualitative Appraisal Tool.pdfDyK Jhttp://www.gsr.gov.uk/downloads/evaluating_policy/a_quality_framework.pdfyK http://www.gsr.gov.uk/downloads/evaluating_policy/a_quality_framework.pdfyX;H,]ą'c]$$If!vh5a5555k5#va#v#v#v#vk#v:V l  t2065t55 55ap2yt~kda$$Iflֈd E`:,"t   t20644 lap2yt~]$$If!vh5a5555k5#va#v#v#v#vk#v:V l  t2065t55 55ap2yt~kd $$Iflֈd E`:,"t   t20644 lap2yt~]$$If!vh5a5555k5#va#v#v#v#vk#v:V l  t2065t55 55ap2yt~kd'$$Iflֈd E`:,"t   t20644 lap2yt~]$$If!vh5a5555k5#va#v#v#v#vk#v:V l  t2065t55 55ap2yt~kd$$Iflֈd E`:,"t   t20644 lap2yt~]$$If!vh5a5555k5#va#v#v#v#vk#v:V l  t2065t55 55ap2yt~kd$$Iflֈd E`:,"t   t20644 lap2yt~]$$If!vh5a5555k5#va#v#v#v#vk#v:V l  t2065t55 55ap2yt~kdP$$Iflֈd E`:,"t   t20644 lap2yt~]$$If!vh5a5555k5#va#v#v#v#vk#v:V l  t2065t55 55ap2yt~kd$$Iflֈd E`:,"t   t20644 lap2yt~]$$If!vh5a5555k5#va#v#v#v#vk#v:V l  t2065t55 55ap2yt~kd $$Iflֈd E`:,"t   t20644 lap2yt~e$$If!vh5 55555#v #v#v#v#v#v:V l  t2065 55555ap2yt~kdy#$$Iflֈdp,"   t20644 lap2yt~e$$If!vh5 55555#v #v#v#v#v#v:V l  t2065 55555ap2yt~kd&$$Iflֈdp,"   t20644 lap2yt~e$$If!vh5 55555#v #v#v#v#v#v:V l  t2065 55555ap2yt~kdO*$$Iflֈdp,"   t20644 lap2yt~e$$If!vh5 55555#v #v#v#v#v#v:V l  t2065 55555ap2yt~kd-$$Iflֈdp,"   t20644 lap2yt~e$$If!vh5 55555#v #v#v#v#v#v:V l  t2065 55555ap2yt~kd%1$$Iflֈdp,"   t20644 lap2yt~e$$If!vh5 55555#v #v#v#v#v#v:V l  t2065 55555ap2yt~kd4$$Iflֈdp,"   t20644 lap2yt~e$$If!vh5 55555#v #v#v#v#v#v:V l  t2065 55555ap2yt~kd7$$Iflֈdp,"   t20644 lap2yt~e$$If!vh5 55555#v #v#v#v#v#v:V l  t2065 55555ap2yt~kdf;$$Iflֈdp,"   t20644 lap2yt~^% 2 0@P`p2( 0@P`p 0@P`p 0@P`p 0@P`p 0@P`p 0@P`p8XV~_HmHnHsHtH@`@ NormalCJ_HaJmH sH tH DA`D Default Paragraph FontRi@R  Table Normal4 l4a (k (No List jj I Table Grid7:V04 @4 Footer  !.)@.  Page NumberB'!B uFComment ReferenceCJaJ<2< uF Comment TextCJaJ@j12@ uFComment Subject5\HRH uF Balloon TextCJOJQJ^JaJ6U@a6 Vg Hyperlink >*B*phFVqF i7FollowedHyperlink >*B* phLL Rutitle19d[$^9CJaJmH sH tH ROR Ruauthors19dH^9CJaJmH sH tH PP Rusource19dx^9CJaJmH sH tH .O. Ru journalnamePP xbulletedZ CJOJQJ^JaJmH sH tH X^@X $A Normal (Web)Z CJOJQJ^JaJmH sH tH POP $AnumberedZ CJOJQJ^JaJmH sH tH *W* gStrong5\.X. gEmphasis6]@+@@ O Endnote Text! aJmH sH >@"> "X Footnote Text"CJaJ@&@1@ "XFootnote ReferenceH*8OA8 _apple-style-spanPK![Content_Types].xmlj0Eжr(΢Iw},-j4 wP-t#bΙ{UTU^hd}㨫)*1P' ^W0)T9<l#$yi};~@(Hu* Dנz/0ǰ $ X3aZ,D0j~3߶b~i>3\`?/[G\!-Rk.sԻ..a濭?PK!֧6 _rels/.relsj0 }Q%v/C/}(h"O = C?hv=Ʌ%[xp{۵_Pѣ<1H0ORBdJE4b$q_6LR7`0̞O,En7Lib/SeеPK!kytheme/theme/themeManager.xml M @}w7c(EbˮCAǠҟ7՛K Y, e.|,H,lxɴIsQ}#Ր ֵ+!,^$j=GW)E+& 8PK!Ptheme/theme/theme1.xmlYOo6w toc'vuر-MniP@I}úama[إ4:lЯGRX^6؊>$ !)O^rC$y@/yH*񄴽)޵߻UDb`}"qۋJחX^)I`nEp)liV[]1M<OP6r=zgbIguSebORD۫qu gZo~ٺlAplxpT0+[}`jzAV2Fi@qv֬5\|ʜ̭NleXdsjcs7f W+Ն7`g ȘJj|h(KD- dXiJ؇(x$( :;˹! I_TS 1?E??ZBΪmU/?~xY'y5g&΋/ɋ>GMGeD3Vq%'#q$8K)fw9:ĵ x}rxwr:\TZaG*y8IjbRc|XŻǿI u3KGnD1NIBs RuK>V.EL+M2#'fi ~V vl{u8zH *:(W☕ ~JTe\O*tHGHY}KNP*ݾ˦TѼ9/#A7qZ$*c?qUnwN%Oi4 =3ڗP 1Pm \\9Mؓ2aD];Yt\[x]}Wr|]g- eW )6-rCSj id DЇAΜIqbJ#x꺃 6k#ASh&ʌt(Q%p%m&]caSl=X\P1Mh9MVdDAaVB[݈fJíP|8 քAV^f Hn- "d>znNJ ة>b&2vKyϼD:,AGm\nziÙ.uχYC6OMf3or$5NHT[XF64T,ќM0E)`#5XY`פ;%1U٥m;R>QD DcpU'&LE/pm%]8firS4d 7y\`JnίI R3U~7+׸#m qBiDi*L69mY&iHE=(K&N!V.KeLDĕ{D vEꦚdeNƟe(MN9ߜR6&3(a/DUz<{ˊYȳV)9Z[4^n5!J?Q3eBoCM m<.vpIYfZY_p[=al-Y}Nc͙ŋ4vfavl'SA8|*u{-ߟ0%M07%<ҍPK! ѐ'theme/theme/_rels/themeManager.xml.relsM 0wooӺ&݈Э5 6?$Q ,.aic21h:qm@RN;d`o7gK(M&$R(.1r'JЊT8V"AȻHu}|$b{P8g/]QAsم(#L[PK-![Content_Types].xmlPK-!֧6 +_rels/.relsPK-!kytheme/theme/themeManager.xmlPK-!Ptheme/theme/theme1.xmlPK-! ѐ' theme/theme/_rels/themeManager.xml.relsPK] tL4g $$$' )!W$!(+.c27;=@DELtMQTYZ^`}b@gil5oppar[v0x2{ՊTMOPQRSTVWXY_`bcdefhijklnopqsuvxyz}(( )1)X).@xT\\aygo:ppvwpwNUZ[\]^agmrtw{|~GL M>MzOOORPSS\VVVXXXXX  '!!8@0(  B S  ? _Hlt221437677 _Hlt221437686 _Hlt221437687 _Hlt221437689 _Hlt221437690 _Hlt236454437 _Hlt236454438idB15artMOOOOVV|@@@@@@@MOOOOVV|EE E*urn:schemas-microsoft-com:office:smarttags PersonName9*urn:schemas-microsoft-com:office:smarttagsplaceB*urn:schemas-microsoft-com:office:smarttagscountry-region8*urn:schemas-microsoft-com:office:smarttagsCity $  ;< ir:J !=!K!!!******E#EMMNNOOOOO#OOP PPPPSSBUIUhUmUqUuUVV WWYYYY]ZaZc[k[c\k\\\\] ]]]]]]7^:^x^^3_7_``aaPgWghhhh(h,h:h>hLhPhnnnnnnnnnnnnnnnoooppvv}}o~s~~~~~~~  -2Հ܀ȃ̃ $y~_e{ˆȈ05‰lj=Ap Z!5588`==`QQuXXgh8nro ؋ 7I\VW <Χ "I$}_{;TB]lʳ0> QfE&ZH\$'$*(T |w)X8v.*!Z5 8&}nV:ZHg(?edZ@j Hk A a;E2qJ&L}Lؠ#L$ts(PpCUm_&lObTAAN3Hi԰q;u|8 xzf*}K\b~ 0&I>^`^`^`^`^`^`^`^`^`h^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh88^8`B*OJQJo(phhHmh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh88^8`OJQJo(hHh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh88^8`OJQJo(hHh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh88^8`OJQJo(hHh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh88^8`OJQJo(hH ^`hH.  L ^ `LhH.   ^ `hH. xx^x`hH. HLH^H`LhH. ^`hH. ^`hH. L^`LhH.h88^8`OJQJo(hHh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh88^8`OJQJo(hHh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hH ^`OJQJo("  ^`OJQJo("  pp^p`OJQJo("  @ @ ^@ `OJQJo("  ^`OJQJo("  ^`OJQJo("  ^`OJQJo("  ^`OJQJo("  PP^P`OJQJo(" h^`OJQJo(hHh^`OJQJo(hHhpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh^`OJQJo(hHhpp^p`OJQJ^Jo(hHoh@ @ ^@ `OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhPP^P`OJQJ^Jo(hHoh  ^ `OJQJo(hHh88^8`OJQJo(hHh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhpp^p`OJQJ^Jo(hHoh@ @ ^@ `OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhPP^P`OJQJ^Jo(hHoh  ^ `OJQJo(hH^`o(. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH.h88^8`B*OJQJo(phhHmh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh88^8`OJQJo(hHh^`OJQJ^Jo(hHoh  ^ `OJQJo(hHh  ^ `OJQJo(hHhxx^x`OJQJ^Jo(hHohHH^H`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHhhh^h`OJQJo(hHh88^8`OJQJ^Jo(hHoh^`OJQJo(hHh  ^ `OJQJo(hHh  ^ `OJQJ^Jo(hHohxx^x`OJQJo(hHhHH^H`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hH^`.^`.pp^p`.@ @ ^@ `.^`.^`.^`.^`.PP^P`.hhh^h`OJQJo(hHh88^8`OJQJ^Jo(hHoh^`OJQJo(hHh  ^ `OJQJo(hHh  ^ `OJQJ^Jo(hHohxx^x`OJQJo(hHhHH^H`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhpp^p`OJQJ^Jo(hHoh@ @ ^@ `OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhPP^P`OJQJ^Jo(hHoh  ^ `OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh^`OJQJo(hHhpp^p`OJQJ^Jo(hHoh@ @ ^@ `OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhPP^P`OJQJ^Jo(hHoh  ^ `OJQJo(hH^`CJOJQJo(^`CJOJQJo(pp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(^`o(. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH.^`o(. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH. g(?K\b~}L;E$<*(_8TB]lOb(PUfE&nV:"R#Lq;u!Z5Z@w)*}m_\$'.k A2qJN3Hi x0&0>           'q                                            ht                                            ht                 XHvۀ^f2KP&Wg^zht                                            'q                                            ht                                   /`l1((ddR0@L t@ v C5I 3uxxuh3!xx8hBxxuJB7xxZ8xxSz%-[`lhB~Bxx^`/E H.b3!]}vxxeq,$1(;p ddc9.v @ qh2~MKxxJB79xxy-&9fV~9qh2xx88>a ?A&p`sAfAB@~B;iHC5I`sA~MKbqxxG3UZxx-[+DafABH.br0b3xx&p(xd;pa ? ddbqL t/E ]}vG3UxxX;w88>zR0{jH~{c9.jH~{fV~;:l<9v1I7!{")3`  b ]q[R`Vbsx9x0vuFJv'p?AAbFmL (i7ST a!zg L!z!&_")'<&'5(5))G)MV)b* S,Y,eq,b -A:-t 2t2c`3i344456$7K7 909\9H::;Q";=$I>8[>r>i?s?$A/AopApA$B(B;BkB#D,HI$JKK3LO%O?O7P$}PXQ?]Q:JR ;SMSfS15UIU V WDPW X"X4Yg9YsYDZ\_k_@~_W`j`yAaZbOcPd_eke`f(gVgAh@jhMwh imXi[iok:lUn~sn]o prp[Aqs8CsZ@t^tRuNy:8yO"zP%~ ~ F `'pjr-"4t@Qh$pk1<hJc]=W`Y79:9gk7}TIIo<}9LhEHb%'GoW9O9,=L~=Rg4VA"(ySA*?!fOPVzrv8 )i U:ZX2XOa{Z5x/74a Y@l vj5K/Yelj 3H`{FXjS  .PV.<B/5 l* I:yE%c] iGl#1S0@ _OiJ/z}.gY7.FNnXmOdV][!K2~$@ ~ ~c ~ ~@Unknownu0039686G*Ax Times New Roman5Symbol3. *Cx Arial5. .[`)Tahoma?= * Courier New;Wingdings7. [ @VerdanaA BCambria Math"1h% چ@*fYFkvGkvG!4d 2qHP ?52!xxdGuidance: quality assessment of qualitative research: how to critically appraise qualitative studiesu0039686u0039686                           Oh+'00<H\ p|   hGuidance: quality assessment of qualitative research: how to critically appraise qualitative studies u0039686 Normal.dotm u00396868Microsoft Office Word@T@@aB@Xf,kvG4VT$mO  p &" WMFC> }lVT$m EMFp {#a   % %  Rp@Times New Roman8 d8 8 H8 Q?j8 8 08 8 lP?j8 8 qh8 8 |Grh+t.XG*Ax Times ew RomanFD8 -h08 08 zhX8 |Gdv% % %  TT._̇@*z@LP -!" !. TT._̇@*z@L.P12TT_̇@*z@L.P -!." " Rp{@Times New Roman ˒ $ϒ ̒ Q?j$ϒ ̒ ͒ ϒ lP?j$ϒ ̒ qh̒ $ϒ Grh+t.XG*Ax Times ew RomanxIDP̒ -h̒ ̒ zh̒ Gdv% % % Rp{ @Times New Roman$ϒ ̒ Q?j$ϒ ̒ ͒ ϒ lP?j$ϒ ̒ qh̒ $ϒ Grh+t.XG*Ax Times ew omanxIDP̒ -h̒ ̒ zh̒ Gdvdv% Rp{@Times New RomanGrh+"}w} }w88X*Ax >@0w7 l8c,wB7 6i88F o8ʒ Txڒ Uyw"}wa!}w }w! F `%8"8F `%8F %̒ }wq,w,wdv% ( Rp{ @Times New Roman*Ax >@0w7 l8"}w} }w88i888h >c l8c,wB`ʒ }whH `ʒ 9uu8OuuO8#H h@0w ʒ H xڒ u߇\Ouu"-wh8O!8q,w,wdv% ( Rp{@Times New Roman8h >c l8"}w} }w88ʒ }w8 >c l8c,wBʒ }w Wʒ 9uu8OuuO8# W@0w ʒ  Wxڒ u߇\Ouu"-w8O!8q,w,wdv% ( Rp{ @Times New Roman8 >c l8"}w} }w88ʒ }w8h >c l8c,wB`ʒ }whH `ʒ 9uu8OuuO8#H h@0w ʒ H xڒ u߇\Ouu"-wh8O!8q,w,wdv% ( % (   T|Z_̇@*z@L\Chapter `JCJ,;;!TTZ*_̇@*z@LP4CTT+ZK_̇@*z@+LP !TTJZ_̇@*z@LLP CTTZ_̇@*z@LP "TZ _̇@*z@LpCritical appraisal`;%,&;C%!CJJ;C%4C%TT Z _̇@*z@ LP !T Z_̇@*z@ L|of qualitative researchC,!KJC%%,C,%C;!;;4;C;<JTTZ_̇@*z@LP HRp@Times New Roman ʒ `Β ˒ Q?j`Β X˒ ̒ DΒ lP?j`Β X˒ qhX˒ `Β LGrh+t.XG*Ax Times ew RomanCD˒ -h˒ ˒ zh˒ LGdv% % %  TT#b_̇@*z@MLP 6!"     '% Ld^^ x!??% (   % % % Tb_̇@*z@LThis chapter should be cited asC8',828!,,'82888,,!,82'% % % Td _̇@*z@ Ld: Hannes K. H,22,'HT d _̇@*z@  LdChapter 4: C2-2,!2T d_̇@*z@ "LCritical appraisal of qualitative C!,,,22!,',2!22,,2,!"     '% LdZHm v!??% (   T|9H_̇@*z@1L\research!,',-!,2T:sH_̇@*z@:1JL. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C  2H40-'(B222HH,22,'HI,!2,3HH,!!'(<,H28;3,2H222C!" Rp@Times New Roman ˒ $ϒ ̒ Q?j$ϒ ̒ ͒ ϒ lP?j$ϒ ̒ qh̒ $ϒ lGrh+t.XG*Ax Times ew RomanPDP̒ -h̒ ̒ zh̒ lGdv% % %     '% LdF\Fo v!??% (   % % % TJq_̇@*z@ Ld(editors), !,22!'!% % % T<rJ _̇@*z@r(LSupplementary Guidance for Inclusion of 2222,I,22',H2222,,&" WMFC ]2'!2,2'223T Ju_̇@*z@ !LQualitative Research in Cochrane H222,,=,',2',23C2,2'22,!"     '% Ld. v!??% (   % % % T$._̇@*z@$LSystematic Reviews of Interventions.2,',H2,=-,,C'2!2,',,222'% % % TT._̇@*z@LP TH._̇@*z@*LVersion 1 (updated August 2011). Cochrane H-!'222!232,,2H312'2222!C2-2!,2,!"     '% Ld, , v!??% (   T0 _̇@*z@ALCollaboration Qualitative Methods Group, 2011. Available from URLC2,22!,22H2,,2,Y,222'H!2222222H2,,2,!!2NHD<TT 0 _̇@*z@ LP !"     '% Ldc v v!??% (   Rp@Times New Roman <͒ В Β Q?jВ ͒ ϒ В lP?jВ ͒ qh͒ В Grh+t.XG*Ax Times ew RomanUD͒ -hΒ Β zh0Β Gdv% % % % % %  Tp_̇@*z@LXhttp:/22T , _̇@*z@ L/cqrmg.cochrane.org/supplemental,2!N1,2,3!,2,3"1'222,N,2,TT- M _̇@*z@- LP-!T|N  _̇@*z@N L\handbook3,222232TT  _̇@*z@ LP-!T| c _̇@*z@ L\guidance122,3,-'% Ldc v !??% ( % % %  TTd  _̇@*z@d LP 6!" Rp@Times New Roman ˒ $ϒ ̒ Q?j$ϒ ̒ ͒ ϒ lP?j$ϒ ̒ qh̒ $ϒ LP .% % % TT S _̇@*z@> LP rh% % % Tx cU _̇@*z@> L\Review 2C,2,HTTd U _̇@*z@d> LPtT (U _̇@*z@> Lleams should use ,,N''22222'-T) U _̇@*z@)> 9La critical appraisal instrument that is underpinned by a ,,!,,,23!,',2'!3N,22,'222,!222,230,!"  TlV  _̇@*z@ LXmulti)N2TTV  _̇@*z@ LP-i!TV 1 _̇@*z@ GLdimensional concept of quality in research and hence includes items to 2N,2'22,,22,,22!22,02!,(,,",2-222,2,,2,23,',N'2!"  Tp : _̇@*z@# LXassess,'',''TT &: _̇@*z@# LP sT' : _̇@*z@'# ELquality according to several domains including quality of reporting, 222,0,,,3!2212',2,!,22N,2'2,232122,02!",22!21!"  T<  _̇@*z@ Llmethodological rN,222221,,Tp<  _̇@*z@ LXrigour!123!TT<  _̇@*z@ LP 2T< _̇@*z@ L|and conceptual depth and,32,22,,22,3,22,22TT < _̇@*z@ LP NTx < _̇@*z@ L\bread. 3!,,2TT < _̇@*z@ LP *z-!"  % % % TT & _̇@*z@ LP,.% % % TT & _̇@*z@ LP h% % % T  ( _̇@*z@ NLCritical appraisal involves (i) filtering against minimum criteria, involving C!,,,22!,',2222-'!!!,!21-1,2'N2N2N,!,!,222221!"  T) ~ _̇@*z@ Ladequacy of reporting detail,2,22-.02!",22!213-,TT)  _̇@*z@ LP T)  _̇@*z@ Lxon the data sampling, 222,2,,',N221TT ) ( _̇@*z@ LP-!T) ) [ _̇@*z@) Lhcollection and,2,,22,22TT\) } _̇@*z@\ LP-"T|~) &" WMFC  _̇@*z@~ L\analysis,2,0''Tx)  _̇@*z@ L\, (ii) ,!!!"  T  _̇@*z@ Lttechnical rigour of ,,22,,!122!3!Td  _̇@*z@ LTthe 2,T _̇@*z@ Lhstudy elements'230,,N,2'TT  _̇@*z@ LP !T   _̇@*z@ #Lindicating methodological soundnessc22,,21N,222221,,'22222,('TT  _̇@*z@ LP T`  _̇@*z@ LTandt,22TT  _̇@*z@ LP *z!"  T|'_̇@*z@iL\(iii) pa!!2,T'Q _̇@*z@'iLradigmatic sufficiency, ref2!,21N,,'2"!,-2-0!,"TxI _̇@*z@I i2Lerring to researchers responsiveness to data and ,!!312",',-!,3,!'!",'222'2,2,''22-,,22!"  T_̇@*z@L|theoretical consistency.2,2!,,,,22'',3.0TT_̇@*z@LP -!"  % % % TTl_̇@*z@WLP`%.% % % TTl_̇@*z@WLP h% % % T,H n_̇@*z@W%LWhen choosing an appraisal instrument_2,2,222'21,2,22!,',2'!2N,2TTI a n_̇@*z@I WLP RTb  n_̇@*z@b W L`a Review ,C,2,HTT  n_̇@*z@ WLPtCT`  n_̇@*z@ WLTeam,,NTT  n_̇@*z@ WLPsm'TT  n_̇@*z@ WLP T n_̇@*z@ WLtshould consider the '2222,22'2,!2-!"  T4o_̇@*z@QLavailable expertise in qualitative research within the team and should ensure thai,2,,2,,32,!',222-,2,!,',,",2H222,,,N,22'2222,2'2"-2,TXo_̇@*z@LPt !"  TdS_̇@*z@<LTthe 2,TS_̇@*z@<JLcritical appraisal instrument they choose is appropriate given the review ,!,,,22",',2'"2N,22-0,222',',23!22!,,12,22,!,2,I!"  TlUq _̇@*z@0Lquestion and the type of studies to be included.22,'22,222,03,2!'22,'22,2,22,2TTr U _̇@*z@r LP P-!"  % % % TT?_̇@*z@*LP*z.% % % TT?_̇@*z@*LP h% % % T8A_̇@*z@*RLReviewers need to clarify how the outcome of their critical appraisal exercise is C,2,H,!'2-,22,-!"022H2,22,2N,2!2-!,!,,,22",',,3,!,','!"  T8C _̇@*z@'Lused with respect to the presentation o,2',2H2!,'2-,22,2!,',2,222Tp C _̇@*z@ LXf thei!2,T C _̇@*z@ Lhr findings. T!!2221'=T C_̇@*z@ Lphe inclusion of a 2,3,2'222!,!"  T,'_̇@*z@PLsensitivity analysis is recommended to evaluate the magnitude of methodological ',2'2/,2,0'''!,-2NN,22,22,2,2,,3,N-1222,2!N,222221-,!" &WMFC TL)_̇@*z@ULflaws or the extent to which it has a small rather than a big impact on the findings !,H'2!2,,3,22H2,22,','N,!,2-!2,2,21N2,,222,!3221'!"  T _̇@*z@Lland conclusions.,22,22,2'22'TT _̇@*z@LP -!"  % % % TT_̇@*z@kLP 6 % % % TT_̇@*z@LP 2 % % % TOz_̇@*z@b LdIntroduction.A -;;@4 !;@TTPz_̇@*z@PbLP ; % % % TT}_̇@*z@LP i2 % % % T__̇@*z@HLConsiderable debate existsC22'2,!,2,2,2,,,3''TT1__̇@*z@HLP "T2T__̇@*z@2H<Lon whether or not concepts such as validity and reliability 22H2,2,!2!22,23-,2''2,2,'2,20,22!,,20 Taw _̇@*z@ELapply to qualitative research and if so how they could be assessed. *z,220222,,2,",',-!,2,22!'222H3.0,3222,,'','',2Tx aP_̇@*z@x LpSome researchers 82N,"-',,!,3,!' TZE_̇@*z@.Llhave stated that2,2,',,22,TT[sE_̇@*z@[.LP ,T0t4 E_̇@*z@t.&Lqualitative research should establish 22,,2,!,'-,!,2'2222,',2'2TT5 f E_̇@*z@5 .LPv2T$g E_̇@*z@g .$Lalidity, reliability and objectivity,2/!,,20,3222,,20TXE_̇@*z@.LP.  TTG5_̇@*z@LPO2HT6GU_̇@*z@6Lhthers plead fo2,!'2,,2"2TTVGv_̇@*z@VLPr,!TTwG_̇@*z@wLP TG _̇@*z@Lan adjustment of these conceptsn,2,22'N,22!2,',,22-,2'TT G _̇@*z@ LP 2T G _̇@*z@ Llto better fit th22,,!!2T G._̇@*z@ L|e qualitative research w,22,,2-!,',-!,2 Tp+_̇@*z@LXdesign2,'12T`<+_̇@*z@LT. ,T@=+_̇@*z@=SLAs a consequence, critical appraisal instruments might differ in the criteria they *zH',,22',23,2-,,!,,,23!,',2'!3N,2'N122!!,!22,,!,!,2.0% % 666666666666666666666666666666666666 6 66 6  6 66 6  6 66 6  6 66 6  6 66 6 66666666666666666666  d."System????????????????--@Times New Roman---  2 Zf ,f',  2  1 2    , ''@Times New Roman--- @Times New Roman-@Times New Roman- @Times New Roman-@Times New Roman- @Times New Roman-- 2 WZfChapter   2 Wf4 2 Wf  2 Wf 2 Wf &2 WfCritical appraisal   2 W(f .2 W,fof qualitative research  2 Wf @Times New Roman--- 2 eZf ,f' - @ !gZ- ---:2 sZfThis chapter should be cited as---2 s f: Hannes K.   2 s5 fChapter 4: >2 sk"fCritical appraisal of qualitative ,f' - @ !uZ- 2 Zfresearchz2 Jf. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C      ,f'@Times New Roman--- - @ !Z- ---2 Z f(editors), ---G2 (fSupplementary Guidance for Inclusion of  =2 S!fQualitative Research in Cochrane i ,f' - @ !Z- ---A2 Z$fSystematic Reviews of Interventions. --- 2 f J2 *fVersion 1 (updated August 2011). Cochrane  ,f' - @ !RZ- m2 ZAfCollaboration Qualitative Methods Group, 2011. Available from URL       2 f ,f' - @ !$Z- @Times New Roman------ 2 Zfhttp:/;2 t f/cqrmg.cochrane.org/supplemental   2 f-2 fhandbook 2 Nf-2 Rfguidance- @ !$Z----  2 ~f ,f'@Times New Roman--- 2 Zf @Times New Roman--- 2 Zf 2 Z fKey points  2 f @Times New Roman- - - 2 Zf @Symbol- - - @"Arial- - - - - - 2 lf- - - 2 rf ---j2 ~?fCritical appraisal of qualitative studies is an essential step &2 fwithin a Cochrane ,f'd2 ~;fIntervention review that incorporates qualitative evidence.t  2 f ,f'- - - 2 !lf- - - 2 !rf ---U2 !~1fThe overarching goal of critical appraisal in the 2 !cf =2 !f!fcontext of including qualitative n,f'.2 /~fresearch in a Cochrane 2 / fIntervention G2 /-(fReview is to assess whether the studies  ,f'2 =~Qfactually address questions under meaning, process and context in relation to the  ,f'G2 K~(fintervention and outcomes under review.    2 KFf ,f'- - - 2 Ylf- - - 2 Yrf ---2 Y~fReview e  2 Yft#2 Yfeams should use  a2 Y9fa critical appraisal instrument that is underpinned by a  ,f'2 g~fmultiw  2 gf-v2 gGfdimensional concept of quality in research and hence includes items to   ,f'2 u~fassess 2 uf s2 uEfquality according to several domains including quality of reporting, o ,f'"2 ~fmethodological 2 frigour 2 f /2 fand conceptual depth and 2 gf 2 jfbread. e 2 f ,f'- - - 2 lf- - - 2 rf ---2 ~NfCritical appraisal involves (i) filtering against minimum criteria, involving    ,f'52 ~fadequacy of reporting detail 2 f ,2  fon the data sampling,   2 sf- 2 wfcollection and 2 f-2 fanalysis2 f, (ii) s,f')2 ~ftechnical rigour of 2 fthe  2 fstudy elements  2 4f @2 7#findicating methodological soundness  2 f 2 fand  2 f ,f'2 ~f(iii) pa42 fradigmatic sufficiency, ref V2 2ferring to researchers responsiveness to data and ,f'/2 ~ftheoretical consistency. 2 f ,f'- - - 2 lf- - - 2 rf ---C2 ~%fWhen choosing an appraisal instrument   2 =f 2 @ fa Review   2 pft2 sfeami 2 fs 2 f )2 fshould consider the ,f'2 ~Qfavailable expertise in qualitative research within the team and should ensure tha  2 ft ,f'2 ~fthe z2 Jfcritical appraisal instrument they choose is appropriate given the review   ,f'S2 ~0fquestion and the type of studies to be included. 2 af ,f'- - - 2 lf- - - 2 rf ---2 ~RfReviewers need to clarify how the outcome of their critical appraisal exercise is   ,f'F2 ~'fused with respect to the presentation o 2 7ff thei 2 Pfr findings. T&2 fhe inclusion of a ,f'2 +~Pfsensitivity analysis is recommended to evaluate the magnitude of methodological    ,f'2 9~Ufflaws or the extent to which it has a small rather than a big impact on the findings     ,f'#2 G~fand conclusions. 2 Gf ,f'--- 2 Ulf - - - 2 bZf ---2 rZ fIntroduction 2 rf - - - 2 lf ---22 ZfConsiderable debate exists 2 f e2 <fon whether or not concepts such as validity and reliability s2 ZEfapply to qualitative research and if so how they could be assessed. %2 fSome researchers  #2 Zfhave stated that 2 f D2 &fqualitative research should establish  2 YfvA2 _$falidity, reliability and objectivity2 f.  2 ZfO 2 cfthers plead fo 2 fr 2 f :2 fan adjustment of these concepts  2 Df #2 Gfto better fit th.2 fe qualitative research 2 Zfdesign2 zf. i2 SfAs a consequence, critical appraisal instruments might differ in the criteria they s  --ffffffeeeeeeeeeeeeeeeedddddddddddddddd՜.+,D՜.+,T hp  5 KULEUVENG eGuidance: quality assessment of qualitative research: how to critically appraise qualitative studies Title 8@ _PID_HLINKSATvH Jhttp://www.gsr.gov.uk/downloads/evaluating_policy/a_quality_framework.pdf v @http://www.phru.nhs.uk/Doc_Links/Qualitative Appraisal Tool.pdf ] .http://eppi.ioe.ac.uk/eppireviewer/login.aspx !l3http://www.joannabriggs.edu.au/services/sumari.php 9http://cqrmg.cochrane.org/supplemental-handbook-guidance   !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnpqrstuv{Root Entry Fg,}Data >1Table;WordDocument.SummaryInformation(LDocumentSummaryInformation8oCompObjy  F'Microsoft Office Word 97-2003 Document MSWordDocWord.Document.89q