More ways to stamp out bad teaching

In a previous post I suggested using a strengthened faculty evaluation process as a tool to reward good teaching and punish bad teaching. But how do we know which faculty and departments are doing a lousy job teaching? Well here's an idea, something that I would imagine is being done at other colleges and universities but not, as far as I am aware, here.

Suppose the Provost's office were to collect evaluations from every course every semester in digital form. (This would require all of us to have students do course evaluations on line, or use Scantron sheets for in-class evaluation.) Data on course grades are compiled every semester as well. Staff in the Provost's office - preferably people with an NSA background - sift through the data looking for outliers: courses, faculty members or departments with an unusually high concentration of A's and A+'s; with an unusually high number of students giving overall instructor evaluations of "poor"; with low scores on a "degree of difficulty" question; etc.. A message goes out to the chair of the department concerned: look into this course or instructor to see if there's something wrong and report back to the Provost. Faculty are given an opportunity to address whatever problems are found; if they are not addressed in future semesters, the wrath of God is called down upon them as I suggested earlier.

One concern with this plan would be catching too many faculty in the dragnet. Student evaluations are notoriously unreliable indicators of quality of teaching. I would not want the Provost to be calling chairs every time an instructor got a few below-average course evaluations. So the process would have to be focused on rather extreme outliers. If half the class rates the professor "poor", my guess is there's a real problem that needs to be addressed.

Data-gathering in this way could help deliver a more positive message as well. The Provost's office could post summary statistics on course evaluations so that individual faculty could see how they match up to college averages. How many hours a week do students say they spend on my class relative to the college average? How difficult do they find my class? How good an instructor do they think I am? What are the cross-correlations between "difficulty" and "overall evaluation" scores? Obviously one would need to take this information with a grain of salt, but I for one would find it interesting and potentially useful, and it might help establish norms for what qualifies as effective teaching.