Learning Outcomes Assessment

These assessment activities were developed by the SLAT Chair and staff, with approval by the SLAT Executive Council and the University of Arizona Office of Instruction and Assessment.  SLAT requires that students complete a first year qualifying exam, comprehensive examinations, Dissertation Proposal, and a defense of their Ph.D. dissertation.  Students also fill out a final self-report of their skills as part of an exit survey. Assessment data is collected for each learning objective as outlined in the matrix below:

First Year Qualifying Exam Assessment Rubric

Written and Oral Comprehensive Exams Assessment Rubric

Dissertation Proposal Assessment Rubric

Dissertation Defense Assessment Rubric

SLAT Student Exit Survey

As of the close of summer 2023, SLAT has collected data for eight academic years: AY 2016, 2017, 2018, 2019, 2020, 2021, and now 2022.  Upon initiation of assessment data collection in AY 2016, SLAT anticipated having measurable data and outcomes by the fifth year of data collection.  Not only do we now have a robust number of responses collected over time, but since SLAT is a five-year program, these eight years of data collection have shown us data sets that measure student cohorts that entered and exited SLAT during the time we were collecting data.  

To collect assessment data, faculty are emailed a survey with the rubric questions as students reach each milestone:  Qualifying Review, Comprehensive Exams, Proposal Presentation, and Ph.D. Defense.  The students were asked to complete an exit survey at the completion of their degree.  The participants gave a score of 1 to 5 based on the following criteria:

1-Needs improvement
2-Developing
3-Progressing
4-Achieving
5-Excellent

We then compile these scores into a table that allows us to observe two measures: 1) average score across all students for each student learning outcome at each milestone, 2) the number of individual responses where at least 80% of the responses scored at 3 or higher.  The average score allows us to observe trends in how our students are performing overall in the measured skills at each milestone.  For the second measure, SLAT defines success for any skill as 80% of responses rated 3 or higher.  This measure allows us to see if there are any areas where we do not have at least 80% of responses indicating that our students are achieving that skill at that particular milestone.  If our average score is low in any area, or if there are milestones where our faculty have not assessed 80% of our students as achieving that skill, then the SLAT Executive Council will meet to determine how to change our preparation and teaching to better prepare our students to meet that milestone.  

Below: Numbers highlighted in yellow indicate where fewer than 80% of responses came in at 4 or higher (achieving or excellent) for these measures. See discussion below for more details.

Our first year of assessment data collection took place during the 2015-2016 academic year.  At the end of that cycle, it was determined that our results were too preliminary, and that it would require at least 5 years of data collection to adequately determine what changes are required in response to our findings.  Now, at the end of the 2022-2023 academic year, we have completed eight years of data collection and can make some observations.

Notes and Observations for the 2022-2023 Assessment Year

Student confidence: Across the board, students score themselves very highly in all skills measured in the SLAT assessment rubrics.  SLAT collects this exit survey immediately after students post their degree, so some of this confidence may be due to completion euphoria.  One way to correct for this would be to attempt to collect a follow-up self-assessment 1 or more years after graduation.  Since post-graduation success is beyond the purview of in-program assessment, and since job placement is a measure of post-graduation success SLAT already collects, the decision was made not to create a follow-up survey.  We are happy to be graduating skilled, confident Ph.Ds.

Data Averages: In exploring the average numbers collected for all SLO measures at all milestones for the past five years, the SLAT Executive Council found no alarming numbers.  There are some expected trends such as an increase in overall knowledge of applied linguistics at the comprehensive exam phase, and some surprising trends such as a slightly slower rating for a student’s ability to conduct independent research at the dissertation phase compared to the proposal phase, but the numbers are not statistically significant.  All scores average out to between 3-5, which is acceptable to the committee.

Specific Measures of Success:  Based on our updated success measure which defined success as 80% of students scoring three or higher on the rubric, SLAT met our definition of success on for all of our Student Learning Outcomes.

Future steps:  The SLAT Executive Council will meet in Spring 2024 to discuss specific plans to address student preparation in research methodology, planning, and design.  Possible responses include changing research methodology requirements, adding another research class, creating additional workshops for SLAT students on the topic, etc.  Changes will be implemented as soon as reasonably possible, hopefully by the 2024-2025 academic year.  

Assessment data collection for the 2023-2024 academic year is ongoing and will be analyzed in September 2024.