The AHSS Assessment Team was composed of eleven members representing the various departments within the College. For the purpose of reviewing the program reports, the Team was divided into four sub-committees, each of which evaluated five or six reports. Members of each sub—committee read and evaluated program reports individually before meeting to reach a consensus on each program report’s overall evaluation. The review process followed the one used last year. Each program report writer and department chair received a draft of the assessment report evaluation and had the opportunity to meet with the sub-committee to clarify their report or evaluation. Each report writer then had the opportunity to submit a revised report, and the sub-committee had the option of revising its evaluation before submitting it to the Dean’s office.  On 17 March the Assessment Team met to review and discuss the individual reports and evaluations. The table below indicates the final evaluations as reported by the sub-committees.

Table A: Summary of AHSS Program Evaluations: 2005

Summary of Results and Comparison Over The Years:

Of the twenty-two programs rated this year five, or 23%, were rated as Excellent; six (27%) were rated as Reasonable/Excellent or Excellent/Reasonable; nine, or 41%, were found to be Reasonable. Two programs were rated as Reasonable/Limited or below. A comparison of the overall ratings of programs over the past nine years (1998-2006) indicates that the general trend has been towards an increase in the number of programs appropriately engaged in Assessment activities (see Table B below). With very few exceptions, AHSS programs ranked in the range from Excellent to Reasonable. (The Assessment Team would like to stress that Excellent/Reasonable and Reasonable ratings indicate a positive evaluation of a program’s assessment efforts.)

As compared to the programs reviewed last year, the total number of programs evaluated during 2006 increased. Two programs (Public History and German Studies), did not submit reports because of staff changes. Both programs are re-evaluating their assessment activities and will be expected to report next year.

Table B:   Overall Ratings of AHSS Programs:   (year of evaluation) 1998-2006

Analysis of Results:

Table B supports the conclusion that the great majority of programs are successfully engaged in program assessment. Only two programs received overall scores of Reasonable/Limited or below as compared with three last year. One program that was evaluated Reasonable/Limited or below last year, Sociology, received a Reasonable rating this year, and the sub-committee that evaluated the report specifically commented on significant improvements over the previous years and noted that the report was “serious, competent, and moving in the right direction.” The Theater program, which also received a rating of Reasonable/Limited or below, showed dramatic improvements and now has a firm grasp on its assessment program. Meanwhile, five programs were evaluated as Excellent and seven as Excellent/Reasonable, as compared with four and six, respectively, last year.

Although fluctuations in the programs' ratings over the last three years have been much smaller than in the past, questions continue to be raised about rater reliability and validity. It is not clear that all sub-committees are using the same criteria to evaluate assessment reports. In an effort to ensure that Team members were “on the same page” when it came to evaluation reports, the Assessment Team held a norming session in mid-February. Prior to the session each Team member read a designated report from outside the College, and during the norming session we shared ideas about how the report should be evaluated. We believe that report writers, as well as Team members, should be advised to attend these norming sessions in future. Report writers should also understand that they should link their assessment tools to the program’s Goals and Objectives, and they should be aware of the criteria on which their reports will be evaluated. (In many of the reports we evaluated, writers appeared not to have read the criteria at all, making evaluation especially difficult.) It is unlikely that we will ever reach a point where there is complete consistency among individual Team members and between and among sub-committees. Some small fluctuations from year to year are inevitable.

There is a degree to which assessment is as much art as science, and will never be completely objective. The varying backgrounds of Team members tend to result in a variety of perspectives when it comes to evaluating reports. In an effort to widen the perspectives that are brought to individual reports, sub-committee members are rotated so that the same people are not evaluating the same reports year after year. Perhaps the ideal solution to problems of reliability and validity would be to have all twenty-plus AHSS reports read and evaluated by all members of the Team, and then discussed by the Team in a series of marathon meetings. Realistically however, if this were to be implemented, Team morale would become a serious problem. Under the circumstances, the steps taken above would seem to be the best approach.

The Assessment Team continued last year's practice of soliciting responses to draft evaluations. After receiving its evaluation from the subcommittee, a program could revise or clarify information in its report, in a face-to-face meeting with the sub-committee or via email, and in response the subcommittee could revise its comments and/or its rating/s. This year most programs and sub-committees communicated entirely by email. The Theater program had a face-to-face meeting with the sub-committee, and that meeting produced a very fruitful discussion. As a result, the Theater program produced a much stronger report that more clearly articulated the great strides they had made over the past year in terms of assessment. The process resulted in revisions of a few reports and evaluations, and most report writers and evaluators seemed satisfied with the outcome. (There was relatively little apparent acrimony between any of the subcommittees and any of the report writers.)

Strengths:

As stated, the majority of programs are engaged in successful assessment. A few of the highlights include:

• Some programs are eager to make changes based on both positive and negative assessment data (especially the History and Theater programs);
• Many programs collect several different types of assessment data (especially the MA in Technical and Expository Writing, which utilizes exit interviews, portfolios, employer surveys, and faculty focus groups);
• Many programs now have several years of data and are interpreting it effectively and using it to make positive changes in their programs;
• Many programs are striving to increase stakeholder involvement, particularly amongst faculty; many other programs have acknowledged that problems exist vis-à-vis faculty involvement and have committed to improving that in future;
• Many programs (especially English, DISLS, and History/Social Studies Ed) have developed such effective assessment procedures that assessment now operates in those programs like a “well-oiled machine”;
• Two programs (Sociology and Theater) that received somewhat low ratings in previous years have made great strides in developing their assessment programs, and both received ratings of Reasonable or better in all aspects of their assessment reports;
• Several programs (including Theater) make effective use of outside exports in evaluating their students and programs, which significantly enhances stakeholder involvement.

Based in part on feedback from the AHSS Assessment Team, the PAAG is in the process of revising report forms, evaluation forms, etc, in addition to producing an Assessment “how to” guide. These changes should help to rationalize the report writing and evaluating processes in the coming years and may also address some of the problems of evaluators not using similar criteria to evaluate reports.

Areas of concern:

Although, as we have seen above, most if not all programs have made great strides in their assessment activities, there is always room for improvement. Following are some “issues” that have become evident on the basis of the program assessment reports and evaluations:

• As indicated above, many programs have sought ways to improve stakeholder involvement; some programs, however, have still not achieved that goal, particularly when it comes to faculty involvement. Perhaps department chairs could be asked to impress on their faculty how vital assessment is, how seriously it is taken at both the College and University levels, and how important faculty cooperation and involvement are;
• Some program assessment reports do not clearly link their assessment activities with stated Goals and Objectives; more effort should be made to ensure that the classes in which assessment is conducted actually teach towards the program’s Goals and Objectives. Similarly, the assessment process should clearly be linked to those Goals and Objectives;

Recommendations:
The Assessment Team spent a great deal of time this year discussing ways that assessment could be done more efficiently and effectively, but also less burdensome. Many Team members feel that as assessment currently stands, it is extremely labor intensive, but there are few “payoffs” for those programs that are doing well, or consequences for those programs that are not doing well. Team members have offered several suggestions for improving the overall process:

• Severall team members expressed concern with the way program assessment reports are evaluated, because it has the potential of pitting one group of people against another; similarly, some sub-committees felt inept at evaluating reports in disciplines where student assessment is done in such vastly different ways. (This was particularly true for the sub-committee that evaluated the Theater report; the initial evaluation was unduly harsh, not because Theater was doing a bad job at assessment but because the evaluators did not appreciate the vastly different ways that the Theater program assesses student progress and the overall program.) Similarly, some Team members believe that the sub-committees are not in a position to advise others on how they should conduct certain aspects of their assessment (particularly when it comes to the involvement of stakeholders outside the university).
• Almost all members of the Team agree that submitting program assessment reports every year is an onerous process and probably accomplishes very little.

o        One suggestion has been made for a three-year reporting cycle for most programs. In this situation, programs would continue to file reports every year, but only one out of every three reports would present results, interpret them, and announce changes that emerge from these results. The other two reports would be much shorter, merely giving a general picture of departmental assessment activities. Such a cycle might allow programs to collect and analyze a critical mass of data before making any radical changes. However, this approach might not apply to programs with multiyear plans (such as Music, in which a given track is already assessed only every three years). It would also not apply to new programs, programs with significantly revised assessment plans or approaches (Goals, Objectives, Methods, and/or Rubrics), and programs with weaker ratings. On the other hand, this approach would be especially appropriate for those programs with very few majors (such as MA Art, International Studies, German, Anthropology, and French/Spanish).

o        Another suggestion is to have a staggered assessment process. Programs with overall scores in the range of Excellent to Reasonable would be required to submit reports every two to three years, while programs with Limited or Inadequate ratings would be reviewed by the Assessment Team to gauge progress over the course of the year.

• Some Team members expressed a belief that Employer and Alumni surveys should be conducted by the university as a whole, not by individual departments, in order to centralize data collection. This would also lighten some of the workload for individual departments.
• The team is divided on whether subcommittees ought to evaluate a program reports in part on the basis of the content of a program's objectives, rather than whether they are clearly stated and measured appropriately. Some members believe that a comment on content usually constitutes an inappropriate attempt to intrude in a discipline foreign to that of the subcommittee members. Others believe that comments and/or suggestions on content are appropriate, but that these comments and suggestions--and whether the department adopts them the following year--should play no role in the evaluation of the report, unless the report is so bad that the objectives seem irrelevant. These members believe that, except in extreme cases, subcommittee members should not advise a department as to what its goals or objectives should be. Others feel that comments and suggestions on content in some circumstances can be justified, and factored into the evaluation of a report. The team suggests that this issue be discussed by PAAG as well as by the AHSS Assessment Team, and that any recommendations be forwarded to all of the college assessment teams.
• The team is also divided on the issue of whether a program, in its report, needs to describe the primary traits that it measures, and its rubrics, in explicit, transparent fashion (i.e. so that someone outside the department and the discipline could understand these easily). A few members feel that as long as the primary traits and rubrics are clear to those within the program, there is less need to describe them precisely for outsiders; others disagree. We suggest that this could be a question for discussion by PAAG, as well as by the AHSS Assessment team.
• The team recommends the continuation of release times for assessment coordinators in each department and the continuation of financial assistance to defray some of the costs of program assessment (although at present the assessment coordinators in some departments, including history, do not receive course release for their assessment activities). 
• Several team members have expressed concern that individual programs are expected to meet specific goals and objectives, but that they are not given adequate resources to meet those goals and objectives. The university needs to appropriate monies in areas identified as critical for programs to achieve goals and objectives (including computer labs and software, funding for faculty development, smaller class sizes and teaching loads, etc.)

Appendix A:   AHSS Assessment Team, 2005-2006

Rolf Groesbeck, team leader (Music)
Ann Clemmer (Political Science)
Andrew Deiser (DISLS)
Andrew Eshleman (Philosophy and Liberal Arts)
Cheryl Harris (Rhetoric and Writing)
Yslan Hicks (Theater Arts)
Roslyn Knutson (English)
Moira Maguire, interim team leader (History)
Terry Richard (Sociology/Anthropology)
Carey Roberson (Art)
Roger Webb (Psychology)

Appendix B:   AHSS Program Evaluations for Calendar Year 2005

Last year's ratings (calendar year 2004) are provided in parentheses.   If the 2004 rating is lower than the 2005 rating, the score is indicated in bold .   If the 2004 rating is higher than the 2005 rating, the score is indicated in italics.

Excellent:
14 BA English (Excellent)

17b BA History/Secondary Education (Not rated)

22 BA Psychology (Reasonable/Excellent)
45b BA Professional and Technical Writing II (Excellent)

68 MA Technical and Expository Writing (Reasonable/Excellent)

Reasonable/Excellent:
12 BA Studio Art ( Reasonable / Excellent )
13 BA Art History (Reasonable)

15/16 French/Spanish (Reasonable)

20 BA Philosophy (Excellent)

21 BA Political Science (Reasonable/Excellent)
67 MA Applied Psychology (Reasonable/Excellent)

Reasonable:
17 BA History (Reasonable)

18 BA Music (Excellent)

23a BA Sociology (Limited/Inadequate)

24 BA Theater Arts (Reasonable/Limited)

26 BA International Studies (Reasonable)

27 BA Liberal Arts (Reasonable)

65 MA Art (Limited)

110 MA Liberal Studies (Reasonable)
115 MA Second Languages (Reasonable)

Reasonable/Limited:
23b BA Anthropology (Reasonable)

Limited:
None

Limited/Inadequate:
None

Inadequate:

18a BA Music/Secondary Education (Not rated)

Not Rated:
66 MA Public History  (Not rated)

119 BA German Studies (Reasonable/Excellent)

Back to Assessment Home Page