A Developmental Writing Scale - Educational Testing Service

Research Report

A Developmental Writing Scale

Yigal Attali Don Powers

April 2008

ETS RR-08-19

Listening. Learning. Leading.?

A Developmental Writing Scale Yigal Attali and Don Powers ETS, Princeton, NJ

April 2008

As part of its educational and social mission and in fulfilling the organization's nonprofit charter and bylaws, ETS has and continues to learn from and also to lead research that furthers educational and measurement research to advance quality and equity in education and assessment for all users of the organization's products and services.

ETS Research Reports provide preliminary and limited dissemination of ETS research prior to publication. To obtain a PDF or a print copy of a report, please visit:



Copyright ? 2008 by Educational Testing Service. All rights reserved. E-RATER, ETS, the ETS logo, LISTENING. LEARNING.

LEADING., and TOEFL are registered trademarks of Educational Testing Service (ETS). CRITERION and TEST OF ENGLISH AS

A FOREIGN LANGUAGE are trademarks of ETS.

Abstract This report describes the development of grade norms for timed-writing performance in two modes of writing: persuasive and descriptive. These norms are based on objective and automatically computed measures of writing quality in grammar, usage, mechanics, style, vocabulary, organization, and development. These measures are also used in the automated essay scoring system e-rater? V.2. Norms were developed through a large-scale data collection effort that involved a national sample of 170 schools, more than 500 classes from 4th, 6th, 8th, 10th, and 12th grades and more than 12,000 students. Personal and school background information was also collected. These students wrote (in 30-minute sessions) up to 4 essays (2 in each mode of writing) on topics selected from a pool of 20 topics.

The data allowed us to explore a range of questions about the development and nature of writing proficiency. Specifically, this paper provides a description of the trajectory of development in writing performance from 4th grade to 12th grade. The validity of a single developmental writing scale is examined through a human scoring experiment and a longitudinal study. The validity of the single scale is further explored through a factor analysis (exploratory and confirmatory) of the internal structure of writing performance and changes in this structure from 4th grade to 12th grade. The paper also explores important factors affecting performance, including prompt difficulty, writing mode, and student background (gender, ethnicity, and English language background). Key words: Writing proficiency, Automated essay scoring, e-rater

i

Table of Contents Introduction..................................................................................................................................... 1

Essay Writing Assessments ..................................................................................................... 1 Automated Essay Scoring........................................................................................................ 3 The Present Study .................................................................................................................... 5 Method ............................................................................................................................................ 9 School Sampling Plan.............................................................................................................. 9 Topic Selection and Allocation to Classes ............................................................................ 10 School Invitation.................................................................................................................... 11 Procedures.............................................................................................................................. 12 Background Questionnaire .................................................................................................... 13 Results on the Participation of Classes and Students.................................................................... 13 Creating Essay Writing Scores ..................................................................................................... 18 The Writing Scale and Grade Norms............................................................................................ 20 Cross-Classified Random Modeling for Student and Topic Effects............................................. 22 Score Reliability............................................................................................................................ 30 Human Scoring Experiment.......................................................................................................... 32 Longitudinal Study........................................................................................................................ 36 Factor Analysis ............................................................................................................................. 39 Discussion ..................................................................................................................................... 48 Summary of Results............................................................................................................... 48 Possible Applications............................................................................................................. 50 Limitations............................................................................................................................. 51 References..................................................................................................................................... 52 Appendix....................................................................................................................................... 56

ii

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download