2018-2019 Annual Program Assessment Report Guide - CSUN



2018-2019 Annual Program Assessment Report GuidePlease submit report to your department chair or program coordinator, the Associate Dean of your College, and to james.solomon@csun.edu, Director of the Office of Academic Assessment and Program Review, by September 30, 2019. You may, but are not required to, submit a separate report for each program, including graduate degree programs, which conducted assessment activities, or you may combine programs in a single report. Please include this form with your report in the same file and identify your department/program in the file name.College, Department and Assessment Liaison Information College: Health and Human DevelopmentDepartment: Communication Disorders and SciencesAssessment liaison: Karen Kochis-Jennings, Ph.D., CCC-SLPPlease check off whichever is applicable:A. X Measured student work within program major/options.B. X Analyzed results of measurement within program major/options.C. X Applied results of analysis to program review/curriculum/review/revision major/options.D. _______ Focused exclusively on the direct assessment measurement of General Education Arts and Humanities student learning outcomesE. _ X_____ Other: Employer Survey 1. Overview of Annual Assessment Project(s). On a separate sheet, provide a brief overview of this year’s assessment activities, including:an explanation for why your department chose the assessment activities (measurement, analysis, application, or GE assessment) that it enactedif your department implemented assessment option A, identify which program SLOs were assessed (please identify the SLOs in full), in which classes and/or contexts, what assessment instruments were used and the methodology employed, the resulting scores, and the relation between this year’s measure of student work and that of past years: (include as an appendix any and all relevant materials that you wish to include)if your department implemented assessment option B, identify what conclusions were drawn from the analysis of measured results, what changes to the program were planned in response, and the relation between this year’s analyses and past and future assessment activitiesif your department implemented option C, identify the program modifications that were adopted, and the relation between program modifications and past and future assessment activitiesif your program implemented option D, exclusively or simultaneously with options A, B, and/or C, identify the basic skill(s) assessed and the precise learning outcomes assessed, the assessment instruments and methodology employed, and the resulting scoresin what way(s) your assessment activities may reflect the university’s commitment to diversity in all its dimensions but especially with respect to underrepresented groupsany other assessment-related information you wish to include, including SLO revision (especially to ensure continuing alignment between program course offerings and both program and university student learning outcomes), and/or thecreation and modification of new assessment instruments2.Preview of planned assessment activities for 2019-20. Include a brief description as reflective of a continuous program of ongoing munication Disorders and Sciences 2018-2019 Assessment ActivitiesIn the 2018-2019 academic year, the department of Communication Disorders and Sciences assessed graduate PLOs 2 and 4, and College of Health and Human Development SLO 2.1, conducted an employer survey as required by our credentialing body, the American Speech and Hearing Association, and attempted to assess undergraduate PLOs 2 and 4. Assessment of Graduate PLO 2 and College of Health and Human Development SLO 2.1Graduate PLO 2 The student will demonstrate cultural sensitivity and knowledge of the effects of cultural difference on communicationSLO 2: Students will actively engage in diverse local and global communities, demonstrating knowledge and awareness of multi-cultural differences and disparities of the health and well-being of individuals and families.Cultural Awareness 2.1 Recognizes and acts upon cultural factors that affect health and well-being of others.We assessed CDS graduate PLO 2 and HHD SLO 2.1 by analyzing results from the multicultural case study essay question and multicultural case study multiple choice questions on the CDS graduate comprehensive exam (CDS 697). In addition, we included two cultural competency questions from our employer survey. A total of one hundred graduate students, 42 from our residential program and 58 from our distance program, were assessed. The multicultural case study essay question consisted of a case study with several prompts. The case study provided client history and diagnostic information for a bilingual culturally diverse client. The multicultural case study multiple choice question consisted of one case study and five case related multiple choice questions. A rubric was used to grade the multicultural essay question. The multicultural case study multiple choice questions were graded by scantron. All essays were graded blindly and received a grade of high pass, pass, low pass, or fail. Any essay receiving a grade of fail was read by a second reader. The second reader received the failed essay along with a second ‘foil’ essay. Thus, the second reader was blinded to the essay that received the failing grade to ensure objective assessment. In the event that the first and second readers disagreed, a third reader read and graded the essays to break the tie. Analysis of the multicultural case study essay results showed that 86% of the CDS residential graduate students and 90% of CDS distance graduate students passed the multicultural essay and met graduate PLO 2 and HHD SLO 2.1. Students who did not pass the multicultural essay were able to retake the essay portion of the comprehensive exam by writing on a different multicultural essay question the following semester. All students passed on reexamination. The criterion for PLO2 for the comprehensive exam multicultural essay question is a passing rate of 80%, i.e. 80% of students must pass the essay. The criterion for PLO2 for the comprehensive exam multicultural case study multiple choice question is 70% of questions within a given disorder area must be answered correctly by 70% of the students. Performance on the multicultural case study multiple choice question was assessed by analyzing the number of questions that were answered correctly by 70% of the students. There were five multiple choice questions related to the multicultural case study. Data analysis showed that the students (n=100) answered four of the five case study related questions correctly more than 70% of the time, resulting in an overall score of 80% for area competency. Responses to two cultural competency questions from the employer survey were analyzed. Sixteen of twenty-eight employers responded to the survey. Employers rated the CSUN trained clinicians as very well prepared, adequately prepared, poorly prepared, or not applicable. Below are the cultural competency survey questions.Question 14: Rate the clinician’s cultural competence.Question 15: How well prepared was the clinician in bilingual or multilingual speech and language assessment?Forty percent of employers rated the CSUN trained clinicians as well prepared in cultural competency, 40% adequately prepared, 10% poorly prepared, and 10% not applicable. Thirty percent of employers rated the CSUN clinicians’ preparation in bilingual speech and language assessment as adequate, 10% poor, and 60% not applicable. Assessment of Graduate PLO4Graduate PLO4The student will integrate theoretical knowledge with clinical experience and application of research literature in clinical practice in order to problem solve clinical cases.We assessed CDS graduate PLO 4 by analyzing results from the case study multiple choice portion of the graduate comprehensive exam (CD697) and the 2018-2019 Employer’s Survey. A total of one hundred graduate students, 42 from our residential program and 58 from our distance program, were assessed. The case study multiple choice portion of the exam consists of 16 case studies from the following disorder areas: child language, articulation and phonology, fluency, voice, aphasia, traumatic brain injury, motor speech, and dysphagia. Each case study has four or five multiple choice questions related to the case. There are two cases per disorder category. Each case study question includes patient information such as case history, diagnosis, and assessment results. The responses from all 100 students to the 68 multiple choice questions were analyzed to determine the percentage of students who answered each individual question correctly, and the percentage of questions within a disorder area that 70% of students answered correctly. Our criteria was 70% of case study multiple choice questions within a given disorder area must be answered correctly by 70% of the students. Analysis of the data showed that the criteria was reached in only two areas, voice disorders and motor speech disorders. The following disorder areas received scores of 60% to 65%: traumatic brain injury, dysphagia, and aphasia. The remaining areas received scores of less than 50%. The possible reasons for scores outside the criteria will be discussed in a subsequent portion of the report. The 2018-2019 Employer Survey assesses employer perception of CSUN trained clinician preparedness. The survey asked employers to rate the clinicians’ level of preparedness in the following disorder areas: pediatric speech, pediatric language, adult neurogenic disorders, adult and pediatric dysphagia, voice, fluency, alternative and augmentative communication, head and neck cancer, and autism. We also asked the employers to rate the clinicians’ knowledge and use of evidence based practices. Employers rated the clinicians’ preparedness as ‘very well prepared,’ ‘adequately prepared,’ ‘poorly prepared,’ or ‘not applicable.’ Sixteen of twenty-eight known employers of CSUN trained clinicians responded. However, only ten completed the entire survey. In the area of pediatric speech disorders, 30% of employers rated CSUN trained clinicians very well prepared, 20% adequately prepared, and 50% not applicable. In the area of pediatric language disorders, 20% of employers rated CSUN clinicians very well prepared, 30% adequately prepared, and 50% not applicable. In the area of adult neurogenic disorders (aphasia, traumatic brain injury, motor speech disorders), 10% of employers rated CSUN trained clinicians very well prepared, 30% adequately prepared, 10% poorly prepared, and 50% not applicable. In the area of adult dysphagia, 30% of employers rated CSUN clinicians adequately prepared, 20% poorly prepared, and 50% not applicable. In the area of voice disorders, 20% of employers rated CSUN trained clinicians very well prepared, 60% adequately prepared, and 20% not applicable. In the area of fluency disorders, CSUN trained clinicians were rated as very well prepared by 10% of employers, adequately prepared by 20%, and not applicable by 70%. Eighty percent of employers agreed that CSUN trained clinicians ‘showed knowledge and use of evidence based practice,’ while 10% of employers disagreed, and 10% could not judge. The only disorder areas that received ratings of ‘poorly prepared’ were head and neck cancer (20%), pediatric and adult dysphagia (20%, 20%), and adult neurogenic disorders (10%). However, these ratings reflect only 1 (10%) or 2 (20%) of the total number of employers who completed the entire survey. Employer SurveyOur credentialing body, the American Speech and Hearing Association’s Council on Academic Accreditation, requires that we conduct employer surveys every several years. In the 2018-2019 academic year, we designed a new employer survey to assess employer perception of CSUN trained clinician preparedness. The survey contained nineteen questions, seventeen multiple choice and two open comment. Employers were requested to rate CSUN trained clinician preparedness as ‘very well prepared,’ ‘adequately prepared,’ ‘poorly prepared,’ or ‘not applicable.’ A rating of ‘not applicable’ would be assigned when a particular population or disorder type is not treated in the employer’s clinical setting. For example, a rating of ‘not applicable’ would be assigned to the area of adult neurogenic disorders by an employer who supervises clinicians in a public school. Invitations to participate in the survey were emailed to twenty-eight known employers of CSUN trained clinicians. Sixteen employers responded, but only 10 completed the entire survey. Historically, response to employer surveys is poor across many disciplines. In addition to assessing employer perception of CSUN trained clinician preparedness in all disorder areas (see assessment of graduate PLO4 page 4) and perceived cultural competency and bilingual assessment skills (see assessment of graduate PLO2 pages 2-4), we also assessed employer perception of clinician professionalism and comportment, and professional writing skills. CSUN trained clinician comportment and professionalism were rated ‘excellent’ and ‘very good’ by 60% and 40% of employers, respectively. Clinical writing skills were rated ‘excellent,’ ‘good,’ and ‘adequate’ by 20%, 30%, and 50% of employers, respectively. Disorder areas that are not currently assessed by the comprehensive exam case study essay or multiple choice case study questions, are pediatric dysphagia, augmentative and alternative communication, and head and neck cancer. In the area of pediatric dysphagia, clinicians were rated poorly prepared by 20% of employers, or not applicable (80%). In the area of augmentative and alternative communication, clinicians were rated adequately prepared by 50% of employers, poorly prepared by 10%, or not applicable by 40%. In the area of autism, clinicians were rate well prepared by 10% of employers, adequately prepared by 40%, or not applicable 50%. Pediatric dysphagia and head and neck cancer are areas of specialty and are not heavily covered in most CDS programs. Overall, we are pleased with the results of the employer survey. The area of autism spectrum disorders, once rated as an area of weakness within our program by employers and alumni, received a preparedness rating of adequate by 50% of employers and not applicable from 40% of employers. Undergraduate AssessmentIn the 2018-2019 academic year, we planned to assess undergraduate PLOs 2 and 4. The courses in which PLOs 2 and 4 are taught are CDS 405, CDS 410, CDS 415, CDS 445, CDS 450, CDS 451, CDS 462, and CDS 469A. With the exception of CDS 450, all identified courses have two sections and are taught by one full time faculty member and one part time faculty member. CDS 450 has only one section and is taught by one part time faculty. CDS 469A has two sections as well, but both sections are taught by the same fulltime faculty member. Instructors of the target courses were contacted regarding PLO assessment. However, we discovered that it would not be possible to assess PLOs 2 and 4 in the 2018-2019 academic year because no common assessment instrument existed that could be used to assess both sections of the given course. Academic freedom had been given to each instructor, fulltime and part time, to teach the course content in the manner they believed was best. While both sections of each course use the same syllabus, the instructors design their own exams and assignments, and sometimes use different textbooks. Thus, we postponed undergraduate assessment until the 2019-2020 academic year. Application of Assessment Data to Program Improvement and Curriculum ChangeFor some time, we have questioned whether the current version of the CDS graduate comprehensive exam (CDS 697) is truly assessing the knowledge and skills taught in all the graduate courses. In the past five years, faculty have retired and new faculty have been hired. Course content has shifted, and new faculty have brought their own teaching styles and approaches to the curriculum. Students have also changed. Over the past two years, the average score on the graduate comprehensive exam has dropped, yet these same students are passing their graduate courses with A’s and B’s. The comprehensive exam has been completely revised only once in the past nine years. However, there have been some minor revisions since that time. Using the graduate comprehensive exam to assess graduate PLO4 has provided valuable information and has resulted in a complete revision of CDS 697, the graduate comprehensive exam. We are updating the multiple choice disorders case study questions and have replaced the essay portion of the exam with an oral presentation component similar to ‘grand rounds.’ The essay portion of the comprehensive exam previously consisted of three case studies. The essay topic areas were ethics, counseling, and cultural diversity. In this, the fall 2019 semester, we will pilot the oral grand rounds presentation component of the graduate comprehensive exam. Each student will receive one pediatric case and one adult case representative of a communication disorder common in that population. The student will receive the cases and have one hour to prepare for the oral presentation. The student will then present the case to a panel of three faculty and respond to various prompts related to any of the following: assessment and diagnosis, treatment, ethical issues, counseling, and cultural considerations. Knowledge of evidence based practice and current research will be required. The presentations will be graded using case rubrics. Case studies and rubrics will be written this semester. We are pleased with the assessment results for graduate PLO2 and can report that PLO2 has been met. In addition, we are pleased with the results of the employer survey, but would have liked to receive more responses. Our discovery that sections of our undergraduate courses are not uniform has led to a plan to create assessment instruments that will be common between the courses sections regardless of instructor. 2019-2020 Assessment PlanIn the 2019-2020 academic year, we will assess the following CDS PLOs:Graduate PLO1 Demonstrate appropriate comportment and knowledge of professional standardsGraduate PLO3 Demonstrate professional entry level knowledge and clinical application of diagnostic and therapeutic techniques for speech, language, cognition, swallowing and auditory disorders across the lifespan.In addition to assessing graduate PLOs 1 and 3, we will design assessment instruments for the following undergraduate courses: CDS 410, CDS 415 and CDS 442 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download