Perceived Student Autonomy and Student Ratings of Instruction

Bryan W. Griffin
Georgia Southern University

Paper to be presented at the annual meeting of the American Educational Research Association, April, 2003. Correspondence concerning this manuscript should be addressed to Bryan W. Griffin; Department of Curriculum, Foundations, and Research; Georgia Southern University; P. O. Box 8144; Statesboro, GA 30460 (e-mail: bwgriffin@gasou.edu).

NOTE. This is a draft version; this is not the version to be presented at AERA. There are a number of wording and other errors that will be changed.

Abstract

Motivation theory suggests that autonomy supportiveness in instruction often leads to many positive outcomes in the classroom, such as higher levels of intrinsic motivation and better learning. Logically such outcomes should also reflect positively in students’ perceptions of teaching effectiveness as well. The purpose of this study was to determine whether perceived student autonomy in the college classroom correlates with various dimensions of student ratings of instruction. Data were collected from 39 undergraduate education courses and 754 students. Consistent with expectations, perceived autonomy in the classroom was positively associated with all 12 dimensions of instructional evaluations considered. This finding suggests that students perceive instructors who offer some level of flexibility and student discretion in course activities to be better at facilitating instruction.

Student ratings of instruction are widespread and a common tool for evaluating faculty. When asked, most faculty members approve of the use of student ratings of instruction for teaching improvement (Baxter, 1991; Moses, 1986; Schmelkin, Spencer, & Gellamn, 1997), but some are resistant to the use of student ratings for tenure, promotion, and merit decisions (Feldman, 1997; McKeachie, 1997). Many educators believe that student ratings are affected, or biased, by a number of factors unrelated to teaching performance (Marsh & Overall, 1979; Wilson, 1998), although much of the research in this area suggests that ratings provide a valid measure of teaching effectiveness (d’Apollonia & Abrami, 1997; Marsh, 1987; McKeachie, 1997).

Most studies of student ratings have focused on predictors such as instructor expressiveness (d’Apollonia & Abrami, 1997; Marsh & Roche, 1997), instructor reputation (Griffin, 2001), grading leniency (Greenwald & Gillmore, 1997a; Griffin, 2002; Marsh & Roche, 2000; Olivares, 2001), course workload and difficulty (Marsh 2001; Marsh & Roche, 2000), class size (Feldman, 1984), expected grades (Marsh, 1987; Marsh & Roche, 2000), and prior subject interest of the student (Howard & Maxwell, 1980; Marsh, 1987). Researchers have long recognized that motivation plays an important role in the dynamics of the classroom and may affect the way students perceive instruction. Furthermore, there is much evidence that more motivated students, such as those with higher levels of interest in the subject matter of the course, provide higher ratings when evaluating instructors (Howard & Maxwell, 1980; Marsh, 1987).

Another area related to motivation that has not received much attention by researchers of student ratings is student autonomy. Student autonomy refers to the degree to which students have some control in deciding course related activities (Pintrich & Schunk, 1996). Considerable research shows that autonomy fosters higher levels of intrinsic motivation, mastery orientation, perceived competence, and achievement (Deci, Hodges, Pierson, & Tomassone, 1992; Garcia & Pintrich, 1996; Stipek, 1998). In a recently published study, Filak and Sheldon (2003) modeled two dimensions of student ratings, overall instructor and overall course evaluations, using the self-determination needs of autonomy, competence, and relatedness. Results of their study showed that competence and autonomy were the two strongest predictors of ratings; this finding further suggests that autonomy is an important area of research for student ratings.

Given that autonomy correlates with a number of positive learning factors, and that at least one study found autonomy to be related to overall ratings of instruction (Filak & Sheldon, 2003), it seems plausible that perceived autonomy in the classroom may be an important factor in shaping multiple dimensions of instructional evaluations. Thus, the purpose of this study was to learn whether perceived autonomy is related to various measures of instructional effectiveness, and to determine whether such a relationship persists once other strong predictors of student ratings are controlled (e.g., prior subject interest, course workload, expected grade, etc.). It was expected that the higher the level of perceived autonomy within the classroom, the higher will be end-of-term ratings that instructors receive.

Method

Participants

A total of 754 undergraduate students enrolled in 39 education courses at a medium sized (14,000 students), regional university in the southeastern United States participated in this study. The classes ranged in size from 6 to 34 students. Undergraduate education students at this institution are predominately White (71%) and female (80%). Most respondents (76%) reported grade point averages in the range of 2.5 to 3.5 on a 4.0 scale. Data were collected during the fall and spring semesters of the 1998-1999 academic year.

Instrument and Variables

An instrument to assess student evaluations of instruction and course characteristics was developed drawing item and question wording from multiple sources (Abrami, d'Apollonia, & Rosenfield, 1997; Feldman, 1997; Marsh, 1987; Murray, 1997). To measure teaching effectiveness, 12 statements were used to assess multiple dimensions of instruction with ratings following a five-point scale. The 12 statements follow.

1. Overall, how would you rate this course?

2. Overall, how would you rate this instructor?

3. The instructor was dynamic and energetic in conducting the course.

4. The instructor presented the material in a clear and understandable manner.

5. Course materials were well prepared and organized.

6. Students were invited to share their ideas and knowledge.

7. The instructor made students feel welcome in seeking help/advice in or outside of class.

8. The content of this course is useful, worthwhile, or relevant to you.

9. Methods of evaluating student work were fair and appropriate.

10. The instructor seems to have a real interest in and concern for students.

11. The instructor gave students useful/helpful feedback on work.

12. The instructor is very knowledgeable in the subject of this course.

For the first 2 items, overall course and overall instructor, the scaled ranged from 1 “Poor” to 5 “Excellent” and for the remaining 10 items the scale ranged from 1 “strongly disagree” to 5 “strongly agree.”

Perceived autonomy was measured by student responses to three statements, “The instructor was willing to negotiate course requirements with students” (M = 3.50, SD = 1.22), “Students had some choice in course requirements or activities that would affect their grade” (M = 3.32, SD = 1.22), and “The instructor made changes to course requirements or activities as a result of student comments or concerns” (M = 3.57, SD = 1.21). Responses to these statements ranged from 1 (“strongly disagree”) to 5 (“Strongly agree”). Cronbach’s alpha was .86 for responses from these three items, with an average correlation of .68 among them. Factor analysis was performed on data from the 12 ratings statements and the 3 autonomy statements, using a number of extraction methods, and the autonomy statements formed a distinct factor from the 12 ratings statements in each instance.

In addition to perceived autonomy, other predictors of ratings were included in this study to serve as control variables. These included a measure of (a) grading leniency, which was assessed by students’ responses to this statement, “This instructor is a lenient/easy grader” (1 “strongly disagree” to 5 “strongly agree”), (b) the instructor’s reputation (1 “very bad” to 5 “very good”, and 6 “didn’t know about the instructor”), (c) course difficulty (1 “one of easiest” to 5 “one of most difficult”), (d) course workload (1 “very light” to 5 “very heavy”), (e) current GPA, and (f) prior subject interest (1 “no interest” to 5 “very interested”). Class size and instructor’s sex (coded 1 for males and 0 for females) were also included in the analysis. Instructor reputation was recoded into three groups, negative reputation (ratings of 1, 2, or 3), positive reputation (ratings of 4 or 5), and no reputation (rating of 6). To use this information in the analyses that follow, two dummy variables (Pedhazur, 1997) were created. For the positive reputation group, the dummy variable was coded 1 if the student rated the instructor as having a positive reputation, and 0 otherwise. The second dummy variable was labeled negative reputation and was coded 1 if student responses corresponded with the negative reputation category, and 0 for all others. Of the 754 sets of responses, 176 (23.3%) were classified into the positive reputation group, 420 (55.7%) into the no reputation group, and 158 (21%) into the negative reputation group.

In addition to the predictors listed above, students were asked to identify their expected grade for the course, “What grade do you think the instructor will assign you in this course?” and the grade they believe they deserve for the course, “What grade do you think you deserve in this course?” Both of these was scored on a four point scale such that any “A” grade equals 4, “B” equals 3, etc. The difference between these two represents a grade discrepancy measure, i.e., expected grade minus deserved grade. The majority of respondents, 623 (82.6%) indicated that the grade they expected was the same as the grade they deserved. For 118 (15.6%) students the difference was negative such that the expected grade was lower than the deserved grade, and for only 13 (1.7%) students the expected grade was higher than the deserved grade. To incorporate this negative grade discrepancy measure into statistical models used below, a dummy variable was created where 1 was coded for the 118 students who believed their expected grade was lower than their deserved grade, and a 0 was used for all others (n = 636).

Procedures

The evaluation instrument was administered during the last week of regular classes in the fall and spring semesters of the 1998 and 1999 academic year. Instructors were required to leave the classroom during evaluations. Students were told that evaluations would not be made available until after course grades had been assigned and would only be provided to instructors in aggregate form.

Results

Correlations and descriptive statistics for each of the student-level variables are provided in Table 1. Perceived autonomy had a mean of 3.46 (on a scale of 1 to 5) with a standard deviation of 1.08. The mean and standard deviation show that there was variability in responses to how students perceived autonomy within the classroom. For example, of the 39 courses evaluated, the means for autonomy ranged from low of 2.06 to a high of 4.70. Correlations between perceived autonomy and each of the 12 student ratings ranged from a low of .20 to a high of .42. For 9 of the 12 instructional rating items the zero-order correlation between perceived autonomy and the given instructional rating item was strongest of the student-level predictors considered. In the 3 situations in which the autonomy correlation was not strongest, it was either the second or third largest correlation. These correlations support the hypothesis that perceived autonomy and instructional ratings are positively related.

To statistically model ratings by autonomy and each of the control variables, multilevel regression (Bryk & Raudenbush, 1992) was used in an effort to examine variation in student ratings both within and across classes. Several researchers of student ratings of instruction (e.g., Cranton & Smith, 1990; Feldman, 1998; Gigliotti & Buchtel, 1990) have noted that the level of analysis, either student- or class-level, at which student ratings are examined could influence the nature of the relationships that are revealed. For example, the analysis of class means rather than student-level data may obscure important variation in ratings that result from individual student differences within the classroom. Multilevel analysis allows one to combine both levels of analysis to provide a more complete model of student ratings.

Incorporated into the multilevel analyses were covariates noted above. At the student level, these covariates included grading leniency, negative grade discrepancy dummy variable, instructor reputation dummy variables, course difficulty, course workload, prior subject interest, and expected grade in the course. At the class level, class size and instructor sex were included. A total of 12 models were estimated for each of the 12 student ratings items. Thus, the models examined were, with variables enclosed in parentheses, as follows:

Student-level.

(Student Rating of Instruction Item)_ij = b_0j + b₁ (Grade Leniency)_ij + b₂ (Neg. Grade Discrepancy)_ij

+ b₃ (Positive Reputation)_ij + b₄ (Negative Reputation)_ij + b₅ (Course Difficulty)_ij

+ b₆ (Course Workload)_ij + b₇ (Prior Subject Interest)_ij + b₈ (Expected Grade)_ij + e_ij

At the class-level, mean ratings of the instructor were modeled with class size and instructor sex:

Class-level.

b_0j = g₀₀ + g₀₁ (Instructor’s Sex)_j + g₀₂ (Class Size)_j + u_0j

Combining the student- and class-level equations yields the following model of instructor rating:

Combined.

(Student Rating of Instruction Item)_ij = g₀₀ + b₁ (Grade Leniency)_ij + b₂ (Neg. Grade Discrepancy)_ij

+ b₃ (Positive Reputation)_ij + b₄ (Negative Reputation)_ij + b₅ (Course Difficulty)_ij

+ b₆ (Course Workload)_ij + b₇ (Prior Subject Interest)_ij + b₈ (Expected Grade)_ij

+ g₀₁ (Instructor’s Sex)_j + g₀₂ (Class Size)_j + e_ij + u_0j

This combined model was used to estimate the regression coefficients for each of the 12 rating items presented above. Multilevel regression results, using full information maximum likelihood to obtain estimates (Hox, 1995), are presented in Table 2.

The regression results in Table 2 show that perceived autonomy was statistically and positively related to all 12 rating items. The weakest relation (b = .15) was with the instructor knowledge item (“The instructor is very knowledgeable in the subject of this course”), and the strongest relation (b = .29) was with the students could seek help item (“The instructor made students feel welcome in seeking help/advice in or outside of class”). The average partial regression coefficient for autonomy across the 12 instructional items was .22. To help frame the relevance of perceived autonomy for instructional ratings, consider the situation of examining the single overall instructor rating item for which the perceived autonomy regression estimate is b = .21. Assuming that all other factors are held constant, two instructors who differ only on perceived autonomy by one standard deviation (SD = 1.08, see Table 1) could expect an average mean difference of 1.08 × .21 ≈ .23 points on their overall instructor rating item. In the current sample, the extremes on perceived autonomy ranged from a mean of 2.06 to a mean 4.70. The predicted difference in overall rating for these two instructors would be: (4.70 - 2.06) × .21 = .55 points, or, for example, a difference in absolute ratings of 4.55 vs. 4.00.

For the other variables included in the models, results tended to replicate findings from previous studies. For example, expected grade, prior subject interest, and course difficulty and workload were all positively related to instructional ratings. Grading leniency was positively related to each of the 12 ratings items, though not always statistically significant. The average partial regression coefficient for grading leniency was only .07, which shows a positive, but relatively weak association with ratings (i.e., instructors judged more leniency in grading practices tend to receive higher ratings). As one might expect, grading leniency was mostly strongly associated with the item measuring perceived fairness of methods for evaluating student performance. The negative grade discrepancy dummy showed less robust associations with the ratings items than grading leniency, and was statistically significant in only 5 of 12 cases. Note, however, than in every case the association was negative which suggests that students who perceive their deserved grade is lower than their expected tended to rate lower their instructors on various dimensions of instructional ratings. For instructor reputation, the negative reputation dummy consistently showed a penalty rating for instructors, with an average partial coefficient of -.28. On average, instructors with negative reputations can expect to have their ratings about -.28 points lower than instructors with no reputations. The positive reputation dummy tended to show small and statistically insignificant differences from the no reputation group. This finding suggests that a penalty effect exists for instructors with negative reputations, but little benefit may be derived from positive reputations. Finally, class size had no discernable association with ratings, and male instructors tended to have slightly lower ratings than did their female counterparts.

Discussion

Results of this study show that perceived student autonomy within an undergraduate course positively, and strongly, relates to multiple dimensions of instructional effectiveness as judged by students. These findings are consistent with the prior expectation of a positive relation, and support the theoretical prediction, derived from various views of motivation theory (Deci, Vallerand, Pelletier, & Ryan, 1991; Stipek, 1998; Vallerand, 2000), that by allowing choice in important course activities, or perhaps responding in positive ways to students’ requests for choice, leads to better instructional and learning outcomes as manifested in instructional ratings in this study. Moreover, this association between perceived autonomy and instructional evaluations appears to be stronger than other previously investigated predictors of student ratings (e.g., prior subject interest, expected grade, grading leniency, course workload and difficulty, etc.).

There are several possible ways the positive relation between perceived autonomy and student evaluations of instruction can be interpreted. First, the perception by students that some level of autonomy is present within a course may be enough for students to reward instructors with higher ratings. For example, Griffin and Pool (1998) speculated that by simply allowing students the opportunity to voice concern in how a course is taught was enough to enhance end-of-term evaluations of instruction. Such a relationship would be important for manipulating ratings, but would be unfortunate if no higher levels of learning, or better instruction, occurred in the classroom. Second, autonomy may result in enhanced motivation and interest in course material, but may not result in better understanding. While this may seem unlikely, research by Garcia and Pintrich (1996) found evidence that autonomy was related to higher levels of intrinsic goal orientation, task value, and self-efficacy, but was not correlated with higher levels of achievement. Similarly, Filak and Sheldon’s (2003) study also noted that autonomy was related to overall ratings, but cautioned that empirical support is needed to clearly determine whether autonomy in the college classroom results in higher levels of learning. Third, autonomy may lead to higher levels of motivation among students, and this may result many positive academic outcomes, including better understanding of course material. If this latter relation proves true, then offering an autonomy-centered classroom would be beneficial for many positive reasons, such as sparking higher interest and motivation among students, and therefore correctly points to better instruction, which justifies the higher student ratings.

Additional study of student autonomy and student ratings of instruction is needed to confirm or reject the findings of this study and the study by Filak and Sheldon (2003). One important note about the findings Filak and Sheldon’s and this current study is that autonomy supportiveness is something that almost all higher education instructors can incorporate into their instruction. A simple change, such as soliciting feedback from students periodically during the semester on instruction, or offering some choice in required and graded activities, may prove beneficial in sparking higher interest in the course content and increasing learning. Further, Filak and Sheldon’s use of Deci and Ryan’s (1985) self-determination theory as a model for further exploration seems to hold much promise for student evaluation of instruction research. While not addressed in this current study, Filak and Sheldon’s finding that competence and relatedness also correlated with ratings offers rich research and practical potential. While it may be more difficult or require more effort to incorporate competence and relatedness, as compared to autonomy, into the college classroom, nevertheless it is possible. As result in this area unfolds, it will be interesting to learn which of these factors proves most beneficial, and how creative instructors manage to address each of these three needs.

References

Abrami, P. C., d'Apollonia, S., & Rosenfield, S. (1997). The dimensionality of student ratings of instruction: What we know and what we do not. In R. P. Perry & J. C. Smart (Eds.), Effective Teaching in Higher Education: Research and Practice (pp. 321-367). New York: Agathon.

Baxter, E. P. (1991). The TEVAL experience, 1983-88: The impact of a student evaluation of teaching scheme on university teachers. Studies in Higher Education, 16, 151-179.

Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.

Cranton, P., & Smith, R. A. (1990). Reconsidering the unit of analysis: A model of student ratings of instruction. Journal of Educational Psychology, 82, 207-212.

d’Apollonia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52, 1198-1208.

Deci, E. L., Hodges, R., Pierson, L., & Tomassone, J. (1992). Autonomy and competence as motivational factors in students with learning disabilities and emotional handicaps. Journal of Learning Disabilities, 25, 457-471.

Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in him behavior. New York: Plenum.

Deci, E. L., Vallerand, R. J., Pelletier, L. G., & Ryan, R. M. (1991). Motivation and education: The self-determination perspective. Educational Psychologist, 26, 325-346.

Feldman, K. A. (1984). Class size and college students' evaluations of teachers and courses: A closer look. Research in Higher Education, 21, 45-116.

Feldman, K. A. (1998). Reflections on the study of effective college teaching and student ratings: One continuing question and two unresolved issues. In J. C. Smart (Ed.) Higher Education: Handbook of Theory and Research (pp. 35-74). New York: Agathon.

Feldman, K. A. (1997). Identifying exemplary teachers and teaching: Evidence from student ratings. In R. P. Perry & J. C. Smart (Eds.), Effective Teaching in Higher Education: Research and Practice (pp. 368-395). New York: Agathon.

Filak, V. F., & Sheldon, K. M. (2003). Student psychological need satisfaction and college teacher-course evaluations. Educational Psychology, 23, 235-247.

Garcia, T., & Pintrich, P. R. (1996). The effects of autonomy on motivation and performance in the college classroom. Contemporary Educational Psychology, 21, 477-486.

Gigliotti, R. J., & Buchtel, F. S. (1990). Attributional bias and course evaluations. Journal of Educational Psychology, 82, 341-351.

Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52, 1209-1217.

Griffin, B. W. (2001). Instructor reputation and student ratings of instruction. Contemporary Educational Psychology, 26, 534-552.

Griffin, B. W. (2002, April). Grading leniency, grade discrepancy, and student ratings of instruction. Paper presented at annual meeting of the American Educational Research Association, 2002, New Orleans, LA.

Griffin, B. W., & Pool, H. (1998). Monitoring and improving instructional practices (and are student evaluations valid?). Journal of Research and Development in Education, 32, 1-9.

Howard, G. S., & Maxwell, S. E. (1980). Correlation between student satisfaction and grades: A case of mistaken causation? Journal of Educational Psychology, 72, 810-820.

Marsh, H. W. (1987). Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11, 253-388.

Marsh, H. W. (2001). Distinguishing between good (useful) and bad workloads on students’ evaluations of teaching. American Educational Research Journal, 38, 183-212.

Marsh, H. W., & Overall, J. U. (1979). Long-term stability of students' evaluations: A note on feldman's consistency and variability among college students in rating their teachers and courses. Research in Higher Education, 10, 139-147.

Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52, 1187-1197.

Marsh, H. W., & Roche, L. A. (2000). Effects of grading leniency and low workload on students’ evaluations of teaching: Popular myth, bias, validity, or innocent bystanders? Journal of Educational Psychology, 92, 202-228.

McKeachie, W. J. (1997). Good teaching makes a differenceand we know what it is. In R. P. Perry & J. C. Smart (Eds.), Effective Teaching in Higher Education: Research and Practice (pp. 396-408). New York: Agathon.

Moses, I. (1986). Student evaluation of teaching in an Australian universitystaff perceptions and reactions. Assessment & Evaluation in Higher Education, 11, 117-129.

Murray, H. G. (1997). Effective teaching behaviors in the college classroom. In R. P. Perry & J. C. Smart (Eds.), Effective Teaching in Higher Education: Research and Practice (pp. 171-204). New York: Agathon.

Olivares, O. J. (2001). Student interest, grading leniency, and teacher ratings: A conceptual analysis. Contemporary Educational Psychology, 26, 382-399.

Pedhazur, E. J. (1997) Multiple regression in behavioral research: Explanation and prediction, (3^rd edition). New York: Harcourt, Brace.

Pintrich, P. R., & Schunk, D. H. (1996). Motivation in education: Theory, research, and application. Columbus, Ohio: Prentice Hall.

Schmelkin, L. P., Spencer, K. J., & Gellman, E. S. (1997). Faculty perspectives on course and teacher evaluations. Research in Higher Education, 38, 575-592.

Stipek, D. (1998). Motivation to learn: From theory to practice (3 rd ed.). Boston: Allyn and Bacon.

Vallerand, R. J. (2000). Deci and Ryan’s self-determination theory: A view from the hierarchical model of intrinsic and extrinsic motivation. Psychological Inquiry, 11, 312-318.

Wilson, R. (1998, January 16). New research casts doubt on value of student evaluations of professors. The Chronicle of Higher Education, p. A12-A14.

Table 1. Descriptive Statistics and Correlations Among Student-level Variables

Note. Variables include: 1 = Overall Instructor Rating; 2 = Overall Course Rating; 3 = Dynamic/Energetic Rating; 4 = Presented Clearly Rating; 5 = Materials Organized Rating; 6 = Students Invited to Share Ideas Rating; 7 = Students Could Seek Help Rating; 8 = Course Content Worthwhile Rating; 9 = Fair Evaluations Rating; 10 = Instructor Show Interest in Students Rating; 11 = Feedback Helpful Rating; 12 = Instructor Knowledgeable Rating; 13 = Grading Leniency; 14 = Negative Grading Discrepancy (coded 1 if grade lower than deserved, 0 otherwise); 15 = Positive Reputation Dummy (1 if student rated instructor as having positive reputation, 0 otherwise); 16 = Negative Reputation Dummy (1 if student rated instructor as having negative reputation, 0 otherwise); 17 = Course Difficulty; 18 = Course Workload; 19 = Prior Subject Interest; 20 = Expected Grade; 21 = Perceived Autonomy.

All correlations larger than .071 in absolute value are statistically significant at the .05 level.

n = 754

Table 2. Multilevel Regression Results for Student Ratings of Instruction

Table 2. continued

Note. Negative Grading Discrepancy coded 1 if grade lower than believed deserved, 0 otherwise; Positive Reputation dummy coded 1 if student rated instructor as having positive reputation, 0 otherwise; and Negative Reputation dummy coded 1 if student rated instructor as having negative reputation, 0 otherwise.

* p < .05.

n = 754 students in 39 courses.

Copyright © 2003, Bryan W. Griffin

Last revised on 17 April 2003