ࡱ> '` =9bjbj 4p=1$::::D:$W2;;;;;k<k<k<{W}W}W}W}W}W}W$.Yh[xW>k<k<>>W;;W@@@>;;{W@>{W@@MhN;; :j?NUW0WN\@\ N\Nk<<h@e=T= k<k<k<WW@k<k<k<W>>>>$$$DhD$$$h$$$  Title: Introduction to Survey Research Methods Author: Toni Peters, PhD & Julie Irish, PhD Category: General research methods for medical educators Intended Audience: Novice researchers who plan to use survey methods Goals: To understand how to design surveys to answer a focused research question To practice designing survey items and response options that will yield interpretable and usable results To understand how the structure of the survey as a whole i.e., the sum of the parts (individual items) lends itself to a coherent answer to a research question Overview: This workshop will walk participants through the steps needed to design survey items that will answer their research questions. Participants are invited to bring work in progress and to develop the survey further within the workshop. Discussion will focus on item construction, response options, and consequent methods of analysis to address the research question. Rationale: Surveys are the most frequently used method used in medical education research. However, because the medical community is the most frequently surveyed population in the US, response rates are typically low. Therefore, it is critically important that potential respondents perceive surveys as important and interesting so they are more likely to participate in the research. Not only must the items be interesting, they must be limited so that the survey is not a burden for busy people. If each item provides a critical piece of information and response rate is high, the researcher should be in a good position to answer his/her question. Resources Needed: Readings: Fowler FJ. Improving Survey Questions: Design and Evaluation. Applied Social Research Methods Series, Vol. 38. Thousand Oaks: Sage Publishing, 1995. VanGeest JB, Johnson TP, Welch VL. Methodologies for improving response rates in surveys of physicians: a systematic review. Eval Health Prof 2007;30:303-321. Faculty: An experienced survey researcher Materials: Participants may bring works in progress, or the group may use examples the facilitator provides. A room large enough to accommodate break out groups. White board or flip chart. Protocol: (60 minutes) Introduction and goal setting (5 minutes). Establish the extent of experience participants have with conducting surveys and adjust goals to suit their needs. Elicit from participants research questions they are currently working on. Together, sharpen the focus of each question when needed. Establish whether they intend to conduct interviews in person or by phone; mailed or web-based surveys, or some other form of survey. (10 minutes) Alone, elaborate upon the domain to be studied. List the variables involved. To do this, one might sketch out general questions or points of information of interest (e.g., gender of respondent, satisfaction with current salary). (5 minutes) Demonstrate for participants how items written to answer a research question failed or succeeded. Point out which response options were used, and explain why these were preferable to others. (5 minutes) Alone, generate 2 items based on the rough sketches generated previously. Select 3 different response options (e.g., Likert scale, rank order, forced option, open-ended) and consider how each would answer the question differently. (5 minutes) In dyads or small groups, present to others the research question and the 2 items written. Ask others what the responses to these questions tell them about your research question. Critique the items, trying to identify limitations to later interpretations. (10 minutes) Demonstrate for participants the pros and cons of item response consistency versus variation (speed of response with possible response habituation versus discrimination), as well as the potential for more powerful analyses given different response formats (e.g., scales versus categorization). Describe possible analyses beyond descriptions (i.e., frequency counts versus inter-correlations amongst items). (5 minutes) Ask individuals to volunteer to show how their two items are related and how they might be analyzed together (e.g., are women more satisfied with their salaries than men are?). (10 minutes) Summary: Comment on how one structures surveys to enhance response rate, such as by placing the most interesting items first to capture interest of respondents. Leave demographics for last. Ask only for demographics that are theoretically important to the question. Summarize points the participants have made during the workshop. (5 minutes) Types of ItemsConsiderationsPossible AnalysesRating scales: Likert (SA to SD) Importance 0 = None to 10 = Most Affect vs cognitive (Strongly vs Generally)How many options? Fewer if administered orally; more if range is meaningful Categorical or quasi-numerical? Are names for each category distinctly different? Is the N large enough? Will the anchors drive variation in response? Mid-points & neutral? How would a mid-point mean answer your question? X2 if large enough N in each cell. Means/medians if meaningful to convert categories to numbers. Correlations amongst items easier with more data pointsForced Choice: Either/or; no mid-pointIs your question one of opinion? This is a good way to discriminate between groups of subjects.Distribution of % of respondents In conjunction with numerical variables use t-test or with categorical variables use X2Rank Order: Rank all Rank top/bottom Q-sort for many itemsHow is test administered? Orally, use few items In person, use physical cards to sort Will respondents comply with instructions (no ties)? Computers can force choice Are only the extremes important? Mid-point choices difficult to differentiate and may not be important to hypothesisFrequency of each item ranked 1st (e.g.) Mean rank order of each item In conjunction with other ranked items, use rank order correlationOpen-ended:How many per survey? Usually no more than 2 b/c they take more time to answer & to analyze.Thematic analyses using qualitative methods Handout: Examples of Survey Items Response options Rating scales Primary care physicians are required to tolerate more uncertainty in their clinical work than specialists: Generally agree, generally disagree, dont know Strongly agree, agree, neither, disagree, strongly disagree On a scale of 0 to 10, how important is job security to you? Forced choice Which of the following would you say describes you better: A physician who primarily attends to the social and emotional aspects of patient care OR A physician who primarily attends to the technological and scientific aspects of patient care Rank order Rank order the following in terms of sources of job satisfaction for you: Autonomy Intellectual challenge Colleagues Academic rewards such as promotion Material rewards such as salary Rank order the following in terms of sources of job dissatisfaction for you: Autonomy Intellectual challenge Colleagues Academic rewards such as promotion Material rewards such as salary Open ended What aspect of your current job gives you the greatest satisfaction? Item with false underlying assumption I would like to know your feeling about primary care as a career option. For you, does dealing with the psychosocial problems of patients make primary care seem more attractive, less attractive or does it have no effect on your thinking about a primary care career? What conclusions can one draw from the responses? Analysis of inter-item relationships Are physicians more oriented to scientific aspects of patient care more likely to rank intellectual challenge high as a source of job satisfaction than physicians who are more oriented to psychosocial aspects of care? Source of job satisfactionMean rank order by orientation to CareScientificPsychosocialIntellectual challenge1.31.9Colleagues3.91.1Autonomy1.62.5Academic reward2.04.2Note: these data are fictitious. DEFINITIONS OF VALIDITY Validity has to do with the meaning of a measurement and the inference from that measurement to the construct the researcher intended to measure. Internal Validity: The extent to which one can claim that the independent variable caused the dependent variable. External Validity: The extent to which one can generalize the experimental effect. (Will a test of "tolerance of ambiguity" created at HMS for its students predict HMS students choice of a primary care career as well in 1995 as it did in 1985? For all American medical students?) Face Validity: The extent to which the measure seems plausibly or intuitively true to a reasonable person. Content Validity: The extent to which there is consistency between the content of the measurements and the content taught, for example. (Were the items on the pathophysiology exam covered in the lectures and/or in the assigned reading?) Construct Validity: The extent to which the measure truly reflects the underlying trait or construct as it might be measured at another time or place or using a different instrument. (Does the measure of reading comprehension as measured by the SAT correlate highly with LSAT scores taken 5 years later?) Concurrent Validity: The extent to which a new test correlates with a previously established test that purportedly measures the same construct. (Is there a correlation between the HMS-constructed anatomy final exam scores and the same students' scores on the anatomy subtest of USMLE-I?) Convergent Validity: The extent to which two independent measures of one construct are correlated and are, therefore, indicators of the construct. (Do observers' ratings of the quality of a student's interaction with a standardized patient correlate with preceptors' ratings of the student's interaction with real patients in the Medicine Clerkship?) Discriminant Validity: The extent to which the measure of one construct differs from the measure of a second, supposedly independent construct. (Do measures of socioemotional orientation and technoscientific orientation to the practice of medicine discriminate between students?) Predictive Validity: The extent to which a measure on one test (of one construct) predicts future performance on a different measure (of a different construct). (How well does a high score on "tolerance of ambiguity" or "patient interaction" predict choice of a primary care career?) CHECKLIST OF TASKS FOR SURVEY DEVELOPMENT Clarify the research question or hypothesis Why do you want to know? Has the research been done before? What will you do with the data? Who cares? Identify the subjects of interest Do you have access to them? Can you gain access to them? How many do you need to survey to get a meaningful answer? (power calculation) How many can you get to survey? What is the likely response rate? Determine what information will be needed to answer the question Brainstorm questions or items that might be included in the survey Differentiate between critical information & nice to know Decide how to administer the survey Paper Computer (e.g., Survey Monkey) Telephone Face to face Write questions appropriate to the means of administration, question, subjects, time 10-20 minutes is the usual tolerance level Oral surveys require subjects to remember question + response options so be clear and succinct Using the same response options speeds up response time but may lead to response set & lack of variation Using different response options heightens focus and variation Open-ended items provide depth but must be recorded carefully & lengthen survey time The more response options the more material the survey needs to be (e.g., many items to rank require being able to sort into preliminary and then more refined piles) Field test survey Invite people who are like target subjects but avoid dipping into pool of respondents Measure time to complete survey Ask respondents to identify terms or whole questions they did not understand Ask for general comments & suggestions Analyze & refine survey items Read responses carefully to determine whether data answer your question Go beyond analysis to write a preliminary paragraph describing responses If a response is ambiguous, rewrite and retest the item Readings: Fowler FJ. Improving Survey Questions: Design and Evaluation. Applied Social Research Methods Series, Vol. 38. Thousand Oaks: Sage Publishing, 1995. VanGeest JB, Johnson TP, Welch VL. Methodologies for improving response rates in surveys of physicians: a systematic review. Eval Health Prof 2007;30:303-321. 01289]_gh< = > F G ķĭĭ؟ĕ}obX؟hOJQJ^JhJhOJQJ^JhmEh5OJQJ^Jh5OJQJ^JhJhSOJQJ^JhSOJQJ^JhShJ5OJQJ^JhmEOJQJ^JhJhzFAOJQJ^JhJOJQJ^JhzFAOJQJ^JhJhJOJQJ^JhJhJ5OJQJ^JhJhLOJQJ^J"12^_0 = > BCUVagd & FgdSgdJ=9 | ACSV^_`a-~689COPUX~߹߯߯ߘyobyoyboybyhJhzFAOJQJ^Jh!^OJQJ^JhNeOJQJ^JhOJQJ^Jh bs5OJQJ^Jh -<OJQJ^JhJh[ OJQJ^JhJOJQJ^Jh|OJQJ^JhhJ>*OJQJ^Jh[ hJ5OJQJ^Jh[ OJQJ^JhSOJQJ^JhJhJOJQJ^J&8PA  o $$Ifa$gdibgdzFA & Fgd!^ & FgdzFAgdJ67@A^$$+e  .DM !1EN$noxzx˪hl5OJQJ^Jhibhib5OJQJ^JhibOJQJ^JhJhqzOJQJ^JhzFAOJQJ^Jh -<OJQJ^JhqzOJQJ^JhNeOJQJ^Jh!^OJQJ^JhJhzFAOJQJ^J8,E]zaneeeeeeeeee $IfgdzFAkd$$IflF T3  t036    44 laytl *+bcoyzawGVq,1KLMO̬̾ݞݞݐtݞj``ݐhDOJQJ^JhlOJQJ^JhDhib5OJQJ^JhibhibH*OJQJ^Jhibhib6OJQJ^Jhibhib5OJQJ^J#hibhib5CJOJQJ^JaJh*CJOJQJ^JaJ hibhibCJOJQJ^JaJhibOJQJ^JhibhibOJQJ^Jhib5OJQJ^J!aw FGVqekd{$$IflF T3  t036    44 laytl $IfgdzFA NO[g}ekd$$IflF T3  t036    44 laytl $IfgdzFA OZ[)Gh}~JKVWkl  ٷٷّxm[#h%h -<>*CJOJQJ^JaJh -<5OJQJ^Jh -<h -<5OJQJ^Jhl5OJQJ^JhOJQJ^Jhhib5OJQJ^JhhD5OJQJ^JhibOJQJ^JhDhDH*OJQJ^JhlOJQJ^JhDOJQJ^JhDhib5OJQJ^JhDhD5OJQJ^J )GhJKWekdq$$IflF T3  t036    44 laytl $IfgdzFA Wl  ( e`````X & Fgd -<gdzFAkd$$IflF T3  t036    44 laytl $IfgdzFA  ( =!K!="H" $2$p%%%%%%o&q&&&&&&&&&&&'' ' '''''$'%'(')','.'N'O'P'h''̺ީީީީީީީީީީީީީީީrh5hibOJQJ^Jh5hib5OJQJ^JhibCJOJQJ^JaJh3^LCJOJQJ^JaJ h%h%CJOJQJ^JaJ#h%h%>*CJOJQJ^JaJ#h%h3^L>*CJOJQJ^JaJ h%h3^LCJOJQJ^JaJ h%h -<CJOJQJ^JaJ,( !=!K!!!="H""""""#M#V#m#x#### $ $2$gd3^L & Fgd -< & Fgd3^L & Fgd3^L2$=%o%p%%%p&q&r&&&vj $$Ifa$gd%jkdg$$Ifl,"" t0644 layt% $Ifgd3^Lgd3^L & FgdJ & Fgd3^L &&&&&wkk $$Ifa$gd% $Ifgd3^Lkd$$Ifl40 ,"`  t0644 layt%&&&&&mddd $Ifgd3^Lkd#$$Ifl4F ,"    t06    44 layt3^L&&&&'neee $Ifgd3^Lkd$$IflF ,"   t06    44 layt3^L'' '''neee $Ifgd3^Lkd$$IflF ,"   t06    44 layt3^L''%')'-'neee $Ifgd3^LkdH$$IflF ,"   t06    44 layt3^L-'.'O'h'i'''n(o()ni]TTTTTT^gdib $^a$gdibgd3^Lkd$$IflF ,"   t06    44 layt3^L '(O(U(n(()))***,),5-J-..//0000$1445556,677IJ~m~m~m~m_hjCJOJQJ^JaJ h5hjCJOJQJ^JaJ h5hC8^CJOJQJ^JaJ h5h oCJOJQJ^JaJ#h5hib5CJOJQJ^JaJ#h5h o5CJOJQJ^JaJ h5hibCJOJQJ^JaJh5hib6OJQJ^Jh5hibOJQJ^Jh5hib5>*OJQJ^J!))))**,,5-6-..//0000$1=1`11111 & FgdC8^ & FgdC8^gd3^L$a$gd ogdib^gdib152U2w22283\3b333334w445t56,6666747|7 & Fgdj & Fgdj & FgdC8^ & Fgd| & FgdC8^|777788<9=9gd|gdj & Fgdj78=9 h|h|CJOJQJ^JaJ#h|h|>*CJOJQJ^JaJ21h:p|/ =!"#$% y$$If!vh5 55#v #v#v:Vl t365 55ytly$$If!vh5 55#v #v#v:Vl t365 55ytly$$If!vh5 55#v #v#v:Vl t365 55ytly$$If!vh5 55#v #v#v:Vl t365 55ytly$$If!vh5 55#v #v#v:Vl t365 55ytlM$$If!vh5"#v":Vl t65"yt%k$$If!vh5 5#v #v:Vl4 t6+5 5yt%e$$If!vh5 5 5 #v :Vl4 t6+5 yt3^L]$$If!vh5 5 5 #v :Vl t65 yt3^L]$$If!vh5 5 5 #v :Vl t65 yt3^L]$$If!vh5 5 5 #v :Vl t65 yt3^L]$$If!vh5 5 5 #v :Vl t65 yt3^L@@@ NormalCJ_HaJmH sH tH DAD Default Paragraph FontRi@R  Table Normal4 l4a (k(No Listj@j 3^L Table Grid7:V0=1p12^_0=>BCUVa8 P A o,E]zaw FGVqNO[g})GhJKWl (=K=HMVmx  2=oppqr %)-.Ohin o !!!!""$$5%6%&&''(((($)=)`)))))5*U*w***8+\+b+++++,w,,-t-.,..../4/|////00<1?10000000000 0 0 0000000000000000000 0 0 0 0 0 0 0 0 000 0 0 0 000000 000000 00000 0 00 00 00 0 0000 0000000 00000 0 0 00 0 0 00000 0 0 0( 0( 0 0 0= 0K 0K 0 0= 0H 0H 0H 0H 0H 0= 0 0 0 0 0 0 000 0 020000 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000000000000000000000000 0 0( 0( 0( 0( 0 0) 0) 0) 0) 0 0w* 0w* 0 08+ 08+ 08+ 08+ 0 0+ 0+ 0+ 0+ 0+ 0+ 0 0. 0. 0. 0. 0 0/ 0/ 0/0000012^_0=>BCUVa8 P A o,E]zaw FGVqNO[g})GhJKWl (=K=HMVmx  2=oppqr %)-.Ohin o !!!!""$$5%6%&&''(((($)=)`)))))5*U*w***8+\+b+++++,w,,-t-.,..../4/|////00<1?10000000000 0 0 0000000000000000000 0 0 0 0 0 0 0 0 000 0 0 0 000000 000000 00000 0 00 00 00 0 0000 0000000 00000 0 0 00 0 0 00000 0 0 0( 0( 0 0 0= 0K 0K 0 0= 0H 0H 0H 0H 0H 0= 0 0 0 0 0 0 000 0 020000 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000000000000000000000000 0 0( 0( 0( 0( 0 0) 0) 0) 0) 0 0w* 0w* 0 08+ 08+ 08+ 08+ 0 0+ 0+ 0+ 0+ 0+ 0+ 0 0. 0. 0. 0. 0 0/ 0/ 0/0000000000 O '7=9 "$'*37aW( 2$&&&''-')1|7=9!#%&()+,-./012456=98@0(  B S  ? P<$^Q<R<,\S<&T<~U<$V<tQW<\בX<ԣY<kkVV//w0w0?1 mm^^0000?1 B *urn:schemas-microsoft-com:office:smarttagscountry-region8*urn:schemas-microsoft-com:office:smarttagsCity9 *urn:schemas-microsoft-com:office:smarttagsplace O    tx OQlnoy""&&6'D'U'e'0011?1akl/cd~]yd!!O%U%&&000E0F0v0/161?13333333333333`acoxx ! !(())v*v*7+7+++..///<1?1oy?1yrx.MJ0yh%J&L%X.B0q7$^`o(. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH.h^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hH^`o(. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH.^`o(. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH.^`o(. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH.yrB0q.MJL%                                             t<3>~gj\ P &Yw%,w 1I W ~ 4 jm fg |{ + ! @  5$J|0gswU2>nARH0gl|V7}NA.S5W r.X9ff4)!+!#'#u# $h%NP&y&T (,(H(X}(Rf)*66+qg+BM,I- 3.Q.)S.e../P/R1a1b1u42Zh2@352.6y8h*8%9@@9)L9|O9\9Z:i:|: -<@<>!>r>p?@zFA5BKBnCCg?CuCaDE ]EdErFCG:qGxG HItI(&L3^LNF4N TOP&QAQRNSWSoSTU2 U?%V(VyVilYsZ&tZa[:[R[G\`\c\yg\n\]n]u]^!^C8^Aq`ahbibrc(co0c9c_djdNe!Ne>g|5gmg9hvhvj%l1mMmVRmn nWn o_poq7r bs:tEymyK{VM{2|}h~n~c6:bfUT@)4>EX|`i`eEY `xZ%-mE!%DSqz[  `h~@tEQ[ks86_&.mXP_y!dm'E]dV}T,2Ylb{@fb,uKKW4{ +7 *FC_N<j" D!c)h3$yQcwr4,bB(3Z_j~H Jua;Rq;y#P*Z&SQ`;$Y)zC'Aa 9_`8 %t ^kL3 n-aQ^_D7a&e , a o/LH99+v^ &CsiOe3S $gav%))0@&GMVru7MB 6clT\45HfCs *4y= hqYRto~^c y|G1WG0S6;q[E5Ybo2o +D0w-?Q{*<W$2K%nKlbJ9 Z*AF Nx[HSkxk,LzFGqNOJKWpq %)-.?1@ xxxscxx=1p@UnknownG: Times New Roman5Symbol3& : Arial?5 : Courier New;Wingdings"qhR&CcfbfY)YY)Y$24d$1$12QHP)?L2Introduction to Survey Researchasp4asp4      Oh+'0  ( H T ` lx Introduction to Survey Researchasp4Normalasp44Microsoft Office Word@0@؎@RVJ@Y)՜.+,0 hp  Harvard Medical SchoolY$1'  Introduction to Survey Research Title  !"#$%&'()*+,-./012345678:;<=>?@BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnoqrstuvwyz{|}~Root Entry FY Data 91TableA.\WordDocument4pSummaryInformation(pDocumentSummaryInformation8xCompObjq  FMicrosoft Office Word Document MSWordDocWord.Document.89q