PEARSON WORDS THEIR WAY: WORD STUDY IN ACTION …

PEARSON WORDS THEIR WAY: WORD STUDY IN ACTION INTERVENTION EFFICACY STUDY FINAL REPORT

Submitted by: COBBLESTONE APPLIED RESEARCH & EVALUATION, INC.

Rebecca M. Eddy, Ph.D., H. Todd Ruitman, M.A., Nancy Hankel, M.A., Monique H. Matelski, M.A., Mariana Schmalstig, M.A.

With special acknowledgement to Xiaoxia Newton, Ph.D. October 18, 2011

2120 Foothill Blvd. Ste. 202 La Verne, CA 91750 (800) 971-3891

Cobblestone Applied Research & Evaluation, Inc.

Acknowledgements We would like to thank all of the participating schools sites that worked with us to study how Words Their Way impacted student performance, especially the intervention teachers and their students, school principals, district staff and administration. We appreciate all of the work that you do and your diligent efforts to make the study a success and your continued work to ensure a quality educational experience for children. Special thanks to all site liaisons that helped to coordinate all study activities. We would also like to thank our entire team at Cobblestone, especially Namrata Mahajan and Natasha Schuman. We also appreciate everyone from Pearson Education including Laura Egan, Deena Meyer, Judy Nelson, Linda Duncan, Madeline Coutts, Halle Smith, Anne Thomasmeyer, Marcy Baughman, and especially Ann Vilcheck. Finally, thanks to our families for their ongoing support.

With gratitude,

Rebecca M. Eddy H. Todd Ruitman Nancy A. Hankel Monique H. Matelski Mariana Schmalstig

Pearson Words Their Way: Word Study in Action: Efficacy Study Final Report Page 2 of 74

Cobblestone Applied Research & Evaluation, Inc.

Table of Contents

EXECUTIVE SUMMARY .................................................................................................................................. 6 SECTION ONE: STUDY BACKGROUND, STUDY PURPOSE, AND PROGRAM DESCRIPTION .......................... 10

Efficacy Study Background ...................................................................................................................... 10 A Response to Intervention (RTI) Context .......................................................................................... 11

Study Purpose ......................................................................................................................................... 12 Program Description ............................................................................................................................... 13 Background, Study Purpose and Program Description Summary .......................................................... 14 SECTION TWO: DESCRIPTION OF STUDY DESIGN, SETTING, AND SAMPLE ................................................ 15 Study Design ........................................................................................................................................... 15 Site Selection........................................................................................................................................... 15 Site Demographic Characteristics ........................................................................................................... 16 Student Participants................................................................................................................................ 18

Attrition............................................................................................................................................... 19 Pretest Equivalence............................................................................................................................. 20 Intervention Teacher Participants .............................................................................................................. 22 Design, Setting, and Sample Summary ................................................................................................... 22 SECTION THREE: DESCRIPTION OF STUDY PROCEDURES AND MEASURES ................................................ 23 Study Procedures .................................................................................................................................... 23 Study Incentives .................................................................................................................................. 23 Training Activities................................................................................................................................ 23 Data Collection Measures: Curriculum Implementation ........................................................................ 24 Weekly Logs ........................................................................................................................................ 24 Observations ....................................................................................................................................... 25 Teacher Focus Groups and Individual Interviews ............................................................................... 25 Data Collection Measures: Outcomes .................................................................................................... 26 Standards-based Reading and Language Arts Assessment................................................................. 28 Student survey .................................................................................................................................... 28 Study Procedures and Measures Summary ............................................................................................ 29 SECTION FOUR: ASSESSMENT OF CURRICULUM IMPLEMENTION ............................................................. 31 Description of Program Implementation................................................................................................ 31 Fidelity of Implementation Results ......................................................................................................... 32 Classroom Observations ......................................................................................................................... 33 Implementation Summary ...................................................................................................................... 34 SECTION FIVE: RESULTS RELATED TO STUDENTS' ATTITUDES AND ACHIEVEMENT IN LANGUAGE ARTS .. 35 HLM Analyses of Outcome Measures ..................................................................................................... 35 Metropolitan Achievement Tests 8 Sounds and Print ........................................................................ 36 Metropolitan Achievement Tests 8 Spelling ....................................................................................... 38 Summary of HLM Results........................................................................................................................ 40 Additional Analyses of Outcome Measures............................................................................................ 41 Performance: Comparing Treatment and Control.............................................................................. 41 AIMSweb R-CBM Reading Fluency Assessment Results ..................................................................... 43 Summary of Research Findings ............................................................................................................... 46 SECTION SIX: PRODUCT SATISFACTION ...................................................................................................... 49 Student Satisfaction ................................................................................................................................ 49

Pearson Words Their Way: Word Study in Action: Efficacy Study Final Report Page 3 of 74

Cobblestone Applied Research & Evaluation, Inc.

Teacher Satisfaction................................................................................................................................ 51 Instructional Components ...................................................................................................................... 53 Response to Intervention Components .................................................................................................. 53 Product Satisfaction Summary................................................................................................................ 55 SECTION SEVEN: DISCUSSION ..................................................................................................................... 56 Efficacy Study Key Findings ..................................................................................................................... 56

Intervention teachers and Students Liked Using the Words Their Way Program .............................. 56 Likeability Was Not Correlated with Achievement ............................................................................. 57 Achievement Was Not Related to Student Attitudes about Reading................................................. 57 Study Summary and Limitations ............................................................................................................. 58 REFERENCES ................................................................................................................................................ 60 APPENDIX A: RESPONSE-TO-INTERVENTION TIERS .................................................................................... 62 APPENDIX B: NCRTI RUBRIC ........................................................................................................................ 63 APPENDIX C: PROGRAM QUALIFICATIONS FOR NCRTI REVIEW ................................................................. 64 APPENDIX D: NCRTI QUESTIONS AND ANSWERS........................................................................................ 65 APPENDIX E: TEACHER TRAINING DESCRIPTION......................................................................................... 68 APPENDIX F: IMPLEMENTATION GUIDELINES ............................................................................................ 69 APPENDIX G: DOSAGE OF TREATMENT IMPLEMENTATION ....................................................................... 70 APPENDIX H: PROGRAM COMPONENTS COMPLETED................................................................................ 71 APPENDIX I: RANDOM-INTERCEPT MODELS WITH COVARIATES ............................................................... 72

Tables

Table 1. School Level Demographic Characteristics for Participating Sites ................................................ 17 Table 2. Sample Sizes for Participating Study Groups ................................................................................ 19 Table 3. Sample Sizes for Students in the Program and Control Groups.................................................... 20 Table 4. Pretest Academic Performance Measures.................................................................................... 21 Table 5. Demographic Information for Participating Students................................................................... 21 Table 6. Summary of Intervention Teacher Characteristics........................................................................ 22 Table 7. Schedule of Study Activities .......................................................................................................... 24 Table 8. Distal Measures ............................................................................................................................. 27 Table 9. Proximal Measures........................................................................................................................ 27 Table 10. Student Survey Cronbach's Alpha Reliability Measures.............................................................. 29 Table 11. HLM Results for MAT8 Sounds and Print Scaled Scores ............................................................. 37 Table 12. HLM Results for Treatment Group MAT8 Sounds and Print Scaled Scores ................................ 38 Table 13. HLM Results for the MAT8 Spelling Scaled Scores ...................................................................... 39 Table 14. HLM Results for the MAT8 Sounds and Print Scaled Scores ....................................................... 40 Table 15. Illustration of the Significant Interaction Effect .......................................................................... 40 Table 16. MAT8 Sounds and Print: Treatment and Control Scaled Scores by Grade ................................. 42 Table 17. Pretest and Posttest MAT8 Sounds and Print Results for Minority Students ............................. 43 Table 18. Pretest vs. Posttest Scores for Students Using Words Their Way............................................... 45 Table 19. Student Survey Pretest and Posttest Results .............................................................................. 46 Table 20. Results for Proximal and Distal Outcome Measures................................................................... 47 Table 21. Product Satisfaction Ratings of Words Their Way Program Components by Grade Level ......... 51

Pearson Words Their Way: Word Study in Action: Efficacy Study Final Report Page 4 of 74

Cobblestone Applied Research & Evaluation, Inc.

Figures Figure 1.Conceptual Framework for Words Their Way: Word Study in Action .......................................... 14 Figure 2. Pretest and Posttest MAT8 Sounds and Print: Control vs Treatment ......................................... 42 Figure 3. Pretest and Posttest MAT8 Sounds and Print Results for Minority Students.............................. 43 Figure 4. AIMSweb R-CBM Median Words Read Correct: Raw Score Results ............................................ 44 Figure 5. AIMSweb R-CBM Median Words Read Correct: Percentile Score Results .................................. 44 Figure 6. Student Survey Pretest and Posttest Results............................................................................... 46 Figure 7. Students' Favorability of the Words Their Way Program ............................................................ 50 Figure 8. Teacher Feedback for Words Their Way Components that Enhance Student Learning ............. 52

Pearson Words Their Way: Word Study in Action: Efficacy Study Final Report Page 5 of 74

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download