Monitoring and Improving Instructional Practices
(and Are Student Evaluations Valid?)

Bryan W. Griffin
Georgia Southern University
Harbison Pool
Georgia Southern University

(Published in Journal of Research and Development in Education, 1998)

Bryan W. Griffin, Department of Curriculum, Foundations, and Research, and Harbison Pool, Department of Leadership, Technology, and Human Development, Georgia Southern University.

This manuscript is an updated version of a paper presented at the annual meeting of the American Educational Research Association, April 1995, San Francisco, California.

Correspondence concerning this paper should be addressed to Bryan W. Griffin, Department of Curriculum, Foundations, and Research; Georgia Southern University; P. O. Box 8144; Statesboro, Georgia; 30460 (Telephone: 912/681-0488; E-mail: bwgriffin@gasou.edu).

Abstract

The purpose of this research was to determine whether the use of multiple in-class student evaluations of instruction, coupled with instructor consultations with students, would subsequently lead to improved instruction as measured by direct observation and end-of-term student ratings of instructional performance. A nonequivalent control group design was used with 19 classes over a period of 3.5 years. One result of the multiple evaluations and consultations, as noted by student comments and instructor observations, was that students requested and received instructional alterations and activities that have been shown, through prior research, to lead to better comprehension of course concepts. This suggests that the use of multiple student evaluations and consultations led to an enhanced classroom learning environment. Further, end-of-term student evaluations were uniformly higher in all instructional areas rated by students exposed to the treatment, with an average effect size for the treatment that was substantial (d = 1.01). This finding implies that multiple student evaluations and consultations throughout the term provided an important means of responding to student needs and bettering instructional performance. These results, however, also raise interesting questions about the validity of student evaluations of instruction and whether student autonomy and other motivational variables influence student ratings.

Monitoring and Improving Instructional Practices
(and Are Student Evaluations Valid?)

Research has demonstrated that student evaluations can provide a valid measure of teaching effectiveness (Cohen, 1981; d’Apollonia & Abrami, 1997; Koon & Murray, 1995; Marsh & Dunkin, 1992; Marsh, Fleiner, & Thomas, 1975; Marsh & Roche, 1997; McKeachie, 1979). Given this evidence, various methods for using feedback taken from student evaluations have been developed to enhance instruction. Cohen (1980), L'Hommedieu, Menges, and Brinko (1990), and Marsh, Fleiner, and Thomas (1975) showed that when instructors used feedback obtained from midterm evaluations to alter their teaching, slight improvements in instruction resulted as evidenced by end-of-term evaluations. When student feedback from midterm evaluations were combined with consultation with other teachers or instructional experts, end-of-term evaluations of teaching often showed substantial improvements (Cohen, 1980; L'Hommedieu, Menges, & Brinko, 1990; Marsh & Roche, 1993, 1997; Overall & Marsh, 1979). The use of student feedback, combined with consultations with experts, to improve teaching effectiveness has become the focus of much research, and, as a result, a number of alternative models of consultation now exist (Brinko, 1990, 1991; Wilson, 1984, 1986).

In almost all circumstances, models of feedback and consultation have utilized a single midterm evaluation followed by consultation with either teachers or instructional experts (Marsh, 1987; Marsh & Dunkin, 1992). Few studies have considered the use of feedback from multiple evaluations administered throughout the term, or the use of students as consultants. Notable exceptions were studies by Kottke (1984) and Zahn (1993).

Kottke (1984) used a single midterm evaluation and met with a committee of class representatives to determine what students perceived to be her areas of weakness in teaching. She found that students provided both realistic and helpful suggestions for improvement, and she agreed to several changes in her lecture format as a result of her student consultations (e.g., "to proceed more slowly with complex material, to reiterate major points several times, to summarize each major section, to list lecture outlines on the blackboard" [p. 118]).

Zahn's (1993) procedure was to obtain written evaluations at the end of every regular class session during the semester from a random sample of 10% of the students in his class. He provided written responses to each student's evaluation and returned the evaluation the next class session. Zahn summarized responses on the evaluations, presented these results at the beginning of the next class session, and listed problem areas identified by at least 25% of the respondents. Zahn then initiated a brief discussion of these problems with the class.

The results obtained by Kottke (1984) and Zahn (1993) indicated that multiple evaluations, coupled with student consultations, appeared to lead to better instruction. As previously mentioned, most research on improving instruction has used a single midterm evaluation in conjunction with consultation with either other teachers or instructional experts. In our experiment, two or more evaluations were administered throughout the term. These evaluations were utilized to monitor and improve teaching performance, while teacher/student discussions of the evaluations, i.e., consultations, were used to identify solutions to perceived problems. Thus, this experiment was designed to determine whether the use of multiple evaluations and student consultations would lead to increased instructional effectiveness as demonstrated by direct observation of instructor activity, and by end-of-term student evaluations.

The use of student evaluations to judge whether better instruction resulted due to our manipulation assumes that such evaluations validly measure instructional effectiveness. The question of whether student evaluations provide valid measures has been debated for three decades. In the 1970’s, concern often focused upon whether student ratings of instruction were biased, or could be biased, by factors unrelated to instructional effectiveness, such as the manipulation of course grades or grading leniency (Greenwald, 1997). However, during the 1980’s and early 90’s, the number of published studies suggesting bias in student evaluations dropped sharply, and these were clearly outpaced by research providing evidence for the validity of student ratings (Greenwald, 1997). Recently, the validity of student evaluations of instruction has been questioned again, and this has renewed debate about the potential biasing effects of non-instructional factors (Wilson, 1998).

It is clear that for universities and colleges nationwide, student evaluations have become an important—some may argue too important—component in tenure, promotion, and merit decisions (McKeachie, 1997; Williams & Ceci, 1997; Wilson, 1998). This importance has placed new emphasis on determining which factors lead to better student evaluations. Many argue that student ratings are valid and tap multidimensional aspects of teaching effectiveness (Marsh & Roche, 1997). However, two recent studies provide evidence that student ratings may be affected by factors not necessarily related to good teaching. In an analysis of ratings from over 200 courses at the University of Washington, Greenwald and Gillmore (1997) found a relatively strong relationship between grading leniency and positive student evaluations. It is not clear whether the relationship was due to grading leniency or to better teaching. See Greenwald and Gillmore for a detailed analysis of this issue.

In another study that directly questioned the validity of student ratings, Williams and Ceci (1997) investigated whether changes in presentation style—increased enthusiasm—would lead to better student evaluations. Ceci taught two undergraduate psychology courses using the same content, lecture notes, syllabus, etc. (p. 16), but changed only the presentation style to include "more pitch variability and … more gestures while lecturing" (p. 15). Williams and Ceci found that for every category of instruction rated, those students exposed to the more enthusiastic lecture rated Ceci’s instruction and course higher. Indeed, the mean effect size found was 1.11, a very large increase. Williams and Ceci also noted that despite the much improved evaluations, students’ mean test scores in both classes were nearly identical. Further, and more troubling, Williams and Ceci noted that both groups of students were exposed to identical tests, syllabi, texts, course goals, and grading practices, yet the group that received the more enthusiastic lecture rated each of these higher. Abrami, Leventhal, and Perry (1982), in a review of the effects of enthusiastic teaching research, similarly found that more enthusiastic instructors received higher student ratings.

It is not clear whether student ratings of instruction are contaminated by factors unrelated to teaching effectiveness. Due to a design characteristic of our experiment, we were unexpectedly able to provide some evidence to help weigh the merits of student evaluations. Briefly, in this experiment one set of classes was exposed to the use of multiple evaluations and student consultations, our treatment, and another set was not. In an effort to maintain instructional fairness, we decided that any suggestions adopted for instructional improvement gleaned from the experimental class would be simultaneously implemented in both experimental and control classes. With this policy the only thing that intentionally differed between the two sets of classes was the treatment.

Given this, any variation in end-of-term student ratings could be attributed to the use of multiple evaluations and consultations. The question this situation poses is, why would the treatment lead to better end-of-term evaluations if instructional activities remained similar in the two groups (e.g., syllabi, texts, tests)? If differences in evaluations were to occur in our study, does this mean student ratings are invalid? Maybe not. Strom and Hocevar (1980), in their experiment on course structure and course satisfaction, found that students given more freedom in choice on course activities reported higher levels of satisfaction with the course, but that this effect was moderated by motivational factors. Relatedly, Overall and Marsh (1979) found that student feedback combined with teacher consultations resulted in higher affective outcomes and examination performance for students. Perhaps student ratings of instruction are affected by a combination of motivational attributes and course policies and structure. This and other possible explanations are explored in our discussion of the results below, and recommendations for future research are offered.

In summary, the purpose of this experiment was to learn whether the use of multiple student evaluations given periodically during a course, coupled with student consultations, would lead to better instruction as judged by observation of changes in instruction, and by end-of-term student ratings of instruction. Based upon findings from pervious research with feedback and consultations, we expected teaching practices to improve and for student evaluations to be higher. An interesting paradox arises from this expectation, however. If end-of-term evaluations are higher, then one must wonder why this occurred given that students were exposed to similar instruction, texts, tests, syllabi, and instructor expectations. We offer possible explanations in light of the validity question that surrounds student ratings of instruction and current motivational research.

Method

The sample for this study consisted of end-of-term student evaluations obtained from 19 graduate-level classes (approximately 400 students) taught from the Fall 1992 through the Spring 1996 by the first author. The 19 classes represented two courses: Introduction to Educational Research (13 classes) and Introductory Statistics (6 classes). All data were collected at a medium-sized (enrollment of 14,000), regional university in southeast Georgia.

Two instruments were used in this study. The Class Assessment Questionnaire (CAQ) was developed to collect anonymous student input on a session-by-session basis. The CAQ contained six one-sentence statements, each asking students to rate the instructor on a specific aspect of instruction. For example, one statement reads: "The session was well-planned and organized." For each statement a Likert scale was used, with ratings ranging from "1" (strongly disagree) to "5" (strongly agree). The CAQ also posed three open-ended questions to elicit students' narrative reactions and their specific suggestions for achieving a better class: (a) "What did you like least about this class?," (b) "What did you like most about this class?," and (c) "What do you think could be done to improve this class?" Students completed the CAQ within two minutes. The CAQ is presented in Appendix A.

The second instrument employed in the study was a standard student evaluation questionnaire used by the College of Education (COE) of the university at which the authors teach. This instrument includes nine Likert-type items covering a range of topics related to instruction, such as organization, enthusiasm, and clarity of presentation. In addition to the nine items, students also had an opportunity to provide anonymous, written comments about the course and instructor on the COE survey. The college required that this questionnaire be administered on the last regular class session of each quarter. The nine Likert-scaled items on this questionnaire are presented in Appendix B.

A posttest-only, nonequivalent control group design was used in this study. Of the 19 classes involved, 9 (6 Introduction to Educational Research and 3 Introductory Statistics) were exposed to the treatment (i.e., at least two administrations of the CAQ and corresponding student consultations [described below]), and 10 served as the control. The first author taught all 19 classes in the same manner to the extent possible—i.e ., with similar notes, handouts, lectures, and learning activities in all classes. However, instruction in the experimental courses changed as a result of student feedback on the CAQ and COE questionnaires. For consistency, any change in instructional procedure introduced in an experimental class was also utilized in the corresponding control class during the same and succeeding quarters.

The treatment consisted of (a) administering the CAQ at the end of regularly scheduled class sessions (sessions with examinations were excluded); (b) summarizing and tallying the responses; and (c) briefly reporting the responses at the beginning of the next regularly scheduled class session. Student/instructor discussions of the responses (i.e., consultations) also occurred during this third phase. During consultations, the students and instructor negotiated which actions, if any, the instructor would take to address student concerns. All consultations lasted approximately five minutes.

The following examples illustrate the type of consultation and negotiation that occurred. Frequently students wrote on the CAQ that the instructor's delivery of material was too rapid for adequate note taking. After instructor/student consultation, the class agreed to inform the instructor when the lecture was proceeding too quickly. Another concern students expressed on the CAQ was the lack of case studies, scenarios, and examples provided to illustrate various research or statistical topics. After consultations, the instructor agreed to develop case studies and scenarios for many topics covered in the introductory research and statistics courses.

Results

Tests of statistical significance were run to determine on which items the two groups differed. Results of these tests, summary statistics, and effect sizes (d, Cohen, 1988; Rosenthal, 1994) are reported in Table 1. Seven of the nine items show statistically significant differences between the experimental and control groups. In all nine instances the experimental group means are uniformly higher. This indicates that students exposed to the evaluation and consultation treatment regarded the instruction as more beneficial.

The ds provide a standardized measure of the evaluation and consultation effect for each of the COE questionnaire items. The ds range from a low of 0.56 to a high of 1.22. The average effect size is 1.01, and this represents a large effect. Since the effect sizes are all positive (i.e., the treatment means are larger than the control means on each item), this implies that the treatment resulted in instructional improvement as judged by students.

Based on interviews and student responses to open-ended questions on the COE instrument, it appears that students were appreciative of (a) the instructor’s willingness to solicit criticisms of the instructor’s teaching and (b) the instructor’s effort to improve class instruction. Moreover, students' written responses on both the CAQ and COE instruments showed that they perceived themselves to be empowered by the treatment; they believed they had the ability to change, to some extent, the instructional procedures used in class, as illustrated by the following statements from two students:

"Dr. [first author] made every effort to adjust his teaching to our style of learning. He explained questions from multiple viewpoints and adjusted future instruction based on our feedback. He would actively seek input on his teaching methods."

"Dr. [first author] demonstrated a variety of teaching strategies throughout the quarter. He would have us evaluate which we liked and which worked for us, and then he would use those strategies of instruction. I came into class scared of stats and left loving it (almost). It was fun."

We observed that two evaluations and consultation sessions per term—one conducted at the end of the second or third class session and one in the first or second session after the midterm point—enabled students to inform the instructor of any changes they believed were called for, and gave the instructor time to respond. Also, in a second or succeeding evaluation, students were able to indicate whether appropriate accommodation to their perceived needs had occurred and what, if any, new concerns may have arisen.

We also noted three commonly requested instructional changes that resulted from the use multiple evaluations and student consultations. First, students often asked that the instructor slow the pace of instruction. Second, students frequently requested additional examples of concepts discussed. Third, students wanted more hands-on activities. For example, students asked for more group work analyzing research situations, writing hypotheses, selecting appropriate sample and research designs, and other similar activities.

Discussion

Often end-of-term evaluations are used by instructors to determine appropriate course changes to improve instruction. Unfortunately, such evaluations do not provide immediate feedback, and this precludes the possibility of adjusting instruction to meet current students' needs. The results of this research suggest that the evaluation and consultation strategy outlined above enables instructors to gain access to information regarding students' instructional needs when it can best be used—during the course. Based upon our direct observations, it is appears that students offered good instructional advice. For example, students requested a slower paced lecture. A recent experiment on lecture rate by Robinson, Sterling, Skinner, and Robinson (1997) found that undergraduates taught using a slower lecture rate demonstrated higher levels of comprehension and rated the lecture topic as more important. Additionally, our students requested more examples and hands-on activities or case studies. Research on cognitive strategies shows that students are far more likely to master concepts if given examples and nonexamples to classify (Tennyson & Park, 1980), and that some students require more examples to understand abstract concepts (Park, 1984; Tennyson & Park, 1980). Given the parallel between our students’ requests and the empirical research cited, it seems that students’ requests generated through multiple evaluations and consultations may have led to a better learning environment since the instructor became aware of their specific needs in a timely manner.

The positive results of the end-of-term student evaluations found here show that frequent mid-term evaluations and consultations were judged by students to be worthwhile and resulted in a better learning environment. This finding, however, is suspect. As previously noted, all changes in instruction resulting from feedback and consultations in the experimental classes were simultaneously and subsequently introduced in the control classes as well; thus, both groups of classes were exposed to the same instruction, course activities (save the experimental treatment), syllabi, tests, and texts. As far as possible, the only difference was the use of midterm evaluations and consultations. Given this, one must question why, for example, students in the experimental group rated the instructor’s organizational and planning skills, knowledge of content, course expectations, and skill in evaluating student performance higher.

One could conclude that the end-of-term instructional rating instrument used in this study was invalid since it was affected by noninstructional factors. An alternative explanation is that students saw the midterm ratings and consultations as a form of classroom empowerment, or, more likely, students viewed the treatment as a manifestation of autonomy. Here we adopt Garcia and Pintrich’s (1996) definition of autonomy: "the degree to which the student perceives he or she shares in the decision-making regarding course policies" (p. 479). If our treatment was seen as a mechanism for changing course policies and practices, then maybe this colored students’ perception of the learning experience within the course. Garcia and Pintrich found that perceived autonomy within the college classroom was directly related to other student motivational variables, and indirectly related to final course grades. If Greenwald and Gillmore (1997) are correct in their assessment that expected course grades influence student evaluations of instruction, then it may be that students in this experiment viewed the autonomy offered by multiple evaluations and consultations as important enough to rate the learning experience within the classroom much higher. More research is needed to investigate the possibility that student evaluations are directly, or indirectly, linked to motivational variables, course structure, and both perceived and real autonomy within the classroom. Once this is done, then perhaps a better picture of the validity of student evaluations will emerge.

References

Abrami, P. C., Leventhal, L, & Perry, R. P. (1982). Educational seduction. Review of Educational Research, 52, 446-464.

Brinko, K. T. (1990). Instructional consultation with feedback in higher education. Journal of Higher Education, 61, 65-83.

Brinko, K. T. (1991). The interactions of teaching improvement. In M. Theall and J. Franklin (Eds.), New directions for teaching and learning, No. 48. Effective practices for improving teaching (pp. 39-49). San Francisco: Jossey-Bass.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Cohen, P. A. (1980). Effectiveness of student-rating feedback for improving college instruction: A meta-analysis. Research in Higher Education, 13, 321-341.

Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multi-section validity studies. Review of Educational Research, 51, 281-309.

d’ Apollonia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52, 1198-1208.

Garcia, T., & Pintrich, P. R. (1996). The effects of autonomy on motivation and performance in the college classroom. Contemporary Educational Psychology, 21, 477-486.

Greenwald, A. G. (1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, 52, 1182-1186.

Greenwald, A. G., & Gilmore, G. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52, 1209-1217.

Koon, J., & Murray, H. G. (1995). Using multiple outcomes to validate student ratings of overall teacher effectiveness. Journal of Higher Education, 66, 61-81.

Kottke, J. L. (1984). Organizational development, instructor evaluations, and feedback. Teaching of Psychology, 11, 118-119.

L'Hommedieu, R., Menges, R. J., & Brinko, K. T. (1990). Methodological explanations for the modest effects of feedback. Journal of Educational Psychology, 82, 232-241.

Marsh, H. W. (1987). Students' evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11, 253-388.

Marsh, H. W., & Dunkin, M. (1992). Students' evaluations of university teaching: A multidimensional perspective. Higher Education: Vol. 8. Handbook on Theory and Research. New York: Agathon.

Marsh, H. W., Fleiner, H., & Thomas, C. S. (1975). Validity and usefulness of student evaluations of instructional quality. Journal of Educational Psychology, 67, 833-839.

Marsh, H. W., & Roche, L. (1993). The use of students' evaluations and an individually structured intervention to enhance university teaching effectiveness. American Educational Research Journal, 30, 217-251.

Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52, 1187-1197.

McKeachie, W. J. (1979). Student ratings of faculty: A reprise. Academe, 65, 384-397.

McKeachie, W. J. (1997). Student ratings: The validity of use. American Psychologist, 52, 1187-1197.

Overall, J. U., & Marsh, H. W. (1979). Midterm feedback from students: Its relationship to instructional improvement and students’ cognitive and affective outcomes. Journal of Educational Psychology, 71, 856-865.

Park, O. (1984). Example comparison strategy versus attribute identification strategy in concept learning. American Educational Research Journal, 21, 145-162.

Robinson, S. L., Sterling, H. E., Skinner, C. H, & Robinson, D. H. (1997). Effects of lecture rate on students’ comprehension and ratings of topic importance. Contemporary Educational Psychology, 22, 260-267.

Rosenthal, R. (1984). Parametric measures of effect size. In H. Cooper and L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). Russell Sage.

Strom, B., & Hocevar, D. (1980, April). Influence of course structure on student affect: The structural affect hypothesis. Paper presented at the meeting of the American Educational Research Association, Boston, MA. (ERIC Document Reproduction Service No. ED 185 938)

Tennyson, R. D., & Park, O. (1980). The teaching of concepts: A review of instructional design research literature. Review of Educational Research, 50, 55-70.

Williams, W. M., & Ceci, S. (1997, September/October). "How’m I doing?" Problems with student ratings of instructors and courses. Change, 13-23.

Wilson, R. C., Douglas, D., & Harrington, D. (1984). Using consultation to improve teaching. Berkeley, University of California, Teaching and Evaluation Services. (ERIC Document Reproduction Service No. ED 242 271)

Wilson, R. C. (1986). Improving faculty teaching: Effective use of student evaluations and consultants. Journal of Higher Education, 57, 196-211.

Wilson, R. (1998, January 16). New research casts doubt on value of student evaluations of professors. The Chronicle of Higher Education, pp. A12-A14.

Zahn, D. A. (1993). Notes on the use of minute papers in teaching statistics courses. Unpublished manuscript, The Florida State University, Tallahassee, FL.

Appendix A

Instructions: Please circle the number which best represents your opinion on the statements below. All statements refer to the class session just completed.

Instructions: Please write in the space below each question. Your responses will be used to improve class, so please try to provide constructive comments, although all comments are welcome.

7. What did you like least about this class session?

8. What did you like most about this class session?

9. What, if anything, could be done to improve this class?

Appendix B

Table 1

Summary statistics for nine items on the college of education evaluation instrument.

Note. The numbers above the parentheses are means and the numbers in parentheses are standard deviations. d is defined as (M_experimental - M_control)/SD_within.

adf = 1,17.

*p< .10.

This page has been accessed 221 times since 3 October 1996.

Copyright © 2000, Bryan W. Griffin

Last revised on 08 December, 2000 02:46 AM