U.S. DEPARTMENT OF EDUCATION Evaluating Online Learning

[Pages:80]U.S. DEPAR TMENT OF EDUCATION

Evaluating Online Learning

Challenges and Strategies for Success

I n n o v a t io n s i n E d u c a t io n

Evaluating Online Learning

Challenges and Strategies for Success

I n n o v a t io n s i n E d u c a t io n

Prepared by WestEd With Edvance Research, Inc.

For U.S. Department of Education Office of Innovation and Improvement

2008

This report was produced under U.S. Department of Education Contract No. ED-01-CO-0012, Task Order D010, with WestEd. Sharon Kinney Horn served as the contracting officer's representative. The content of this report does not necessarily reflect the views or policies of the U.S. Department of Education, nor does the mention of trade names, commercial products or organizations imply endorsements by the U.S. government. This publication also contains URLs for information created and maintained by private organizations. This information is provided for the reader's convenience. The U.S. Department of Education is not responsible for controlling or guaranteeing the accuracy, relevance, timeliness, or completeness of this outside information. Further, the inclusion of information or URL does not reflect the importance of the organization, nor is it intended to endorse any views expressed, or products or services offered.

U.S. Department of Education Margaret Spellings Secretary

Office of Innovation and Improvement Doug Mesecar Assistant Deputy Secretary

Technology in Education Programs Cheryl Garnette Director

July 2008

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the citation should be: U.S. Department of Education, Office of Innovation and Improvement, Evaluating Online Learning: Challenges and Strategies for Success, Washington, D.C., 2008.

To order copies of this report (order number ED004344P),

write to: ED Pubs, Education Publications Center, U.S. Department of Education, P.O. Box 1398, Jessup, MD 20794-1398;

or fax your request to: 301-470-1244;

or e-mail your request to: edpubs@inet.;

or call in your request toll-free: 1-877-433-7827 (1-877-4-ED-PUBS). Those who use a telecommunications device for the deaf (TDD) or a teletypewriter (TTY), should call 1-877-576-7734. If 877 service is not yet available in your area, call 1-800-872-5327 (1-800-USA-LEARN; TTY: 1-800-437-0833);

or order online at: .

This report is also available on the Department's Web site at:

On request, this publication is available in alternate formats, such as Braille, large print, or computer diskette. For more information, please contact the Department's Alternate Format Center at 202-260-0852 or 202-260-0818.

Contents

List of Illustrations

iv

Foreword

v

Acknowledgments

vii

Introduction

1

Common Challenges of Evaluating Online Learning (1) The Featured Evaluations (2) What's in This Guide (3)

Part I: Challenges of Evaluating Online Learning

7

Meeting the Needs of Multiple Stakeholders (7) Building on the Existing Base of Knowledge (12) Evaluating Multifaceted Online Resources (20) Finding Appropriate Comparison Groups (26) Solving Data Collection Problems (33) Interpreting the Impact of Program Maturity (40) Translating Evaluation Findings Into Action (43)

Part II: Recommendations for Evaluating Online Learning

49

General Recommendations (49) Recommendations for Meeting the Needs of Multiple Stakeholders (50) Recommendations for Utilizing and Building on the Existing Base of Knowledge (50) Recommendations for Evaluating Multifaceted Resources (52) Recommendations for Programs Considering a Comparison Study (52) Recommendations for Gathering Valid Evaluation Data (53) Recommendations for Taking Program Maturity Into Account (54) Recommendations for Translating Evaluation Findings Into Action (54)

Featured Online Learning Evaluations

55

Appendix A: Resources

59

Distance Learning (59) Evaluation Methods and Tools (60) Resources From Higher Education (61)

Appendix B: Research Methodology

63

Glossary of Common Evaluation Terms

65

Notes

67

iii

Illustrations

FIGURES

1. Excerpt From Digital Learning Commons' Meeting 21st Century Learning Challenges

in Washington State

24

2. Example of Tabulated Results From an Arizona Virtual Academy Online Parent Survey

About the Quality of a Recent Student Workshop

38

TABLES

1. Selected Variables of Profiled Online Learning Evaluations

4

2. Excerpt From Appleton eSchool's Online Program Perceiver Instrument

17

3. Evaluation Design Characteristics

28

4. Excerpts From AZVA's Next Steps Plan, In Response to Recommendations

From the K12 Quality Assurance Audit

47

iv

Foreword

Education in this country has evolved dramatically from the days of one teacher in a one-room schoolhouse. Today, student learning is no longer confined to a physical space. Computers and the Internet have broken through school walls, giving students greater opportunities to personalize their education, access distant resources, receive extra help or more-challenging assignments, and engage in learning in new and unique ways. Although online learning is a relatively new enterprise in the K?12 arena, it is expanding rapidly, with increasing numbers of providers offering services and more students choosing to participate. As with any education program, online learning initiatives must be held accountable for results. Thus, it is critical for students and their parents--as well as administrators, policymakers, and funders--to have data informing them about program and student outcomes and, if relevant, about how well a particular program compares to traditional education models. To this end, rigorous evaluations are essential. They can identify whether programs and online resources are performing as promised, and, equally important, they can point to areas for improvement. The evaluations highlighted in this guide represent a broad spectrum of online options, from programs that provide online courses to Web sites that feature education resources. The evaluations themselves range from internal assessments to external, scientific research studies. All demonstrate how program leaders and evaluators have been able to implement strong evaluation practices despite some challenges inherent to examining learning in an online environment. This guide complements another publication, Connecting Students to Advanced Courses Online, published last year by the U.S. Department of Education. Both are part of the Innovations in Education series, which identifies examples of innovative practices from across the country that are helping students achieve. My hope is that this guide will assist evaluators and program leaders who seek to use data to guide program improvement aimed at achieving positive outcomes for our nation's students.

Margaret Spellings, Secretary U.S. Department of Education

v

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download