Assessing the Learning Environment

How do you know where your students are?  This is one of the most nagging questions for those who teach in a distance environment.  Without regular classroom contact, the informal ways of gathering student feedback disappear.  Teachers in distance learning settings need to design deliberate methods to find out how students are responding to the instruction.

The teaching-learning process is a dynamic one.  The learner must respond to the teacher and the teacher must keep in touch with the learner.  As one researcher describes it, educational relations are like a dance where teachers and students must synchronize their steps if either is to be successful.

As with dancers, as long as there is clear two-way communication people can perform well.  When the communication breakdown, someone's toes get stepped on.  The problem in every educational setting is that there are many factors that can break down the feedback process: the press of time, politics of grading, mis-communication, self-esteem concerns, to name a few.   In fact, the problem of getting useful feedback from students who are best served by disguising or biasing feedback makes it very difficult to be confident that teaching methods are working well.

In face to face encounters, the instructor can overcome part of this problem by reading the hidden signals or by tapping informal moments with students.  In a distance learning environment, these casual aids disappear.  Teachers need to construct more exact procedures to get the information they need to stay in touch with their students? progress.

There are two parts to an evaluation system that will get the teacher the feedback he or she needs to know how the students are doing and what they need next.  The first is often called the formative evaluation process.  The second is called the summative evaluation process.  The difference between these is described below.  The steps for creating an adequate system are included in each section.  The important initial point is that distance education teachers need to plan an evaluation system that corresponds with their teaching plan.

SUMMATIVE FEEDBACK
Summative evaluations are what come to mind when we talk about assessments.  They are methods we use to find out how well someone is doing when they have arrived at an end point.  They provide a judgment that summarizes someone's knowledge or ability at a given moment.  Usually, this takes the form of exams, quizzes, papers, or presentations.  The student's performance on the activities allows the teacher to rate the student's degree of success in mastering the learning objectives.

The purpose of the summative feedback process is to help us learn how well students know the material (knowledge, skills, or attitudes).  In most cases, it is better to not wait until the final day of class to create an occasion for evaluating what students know about the entire course.  Instead, teachers should plan a series of evaluation points.  Each of these points should serve a specific purpose, use appropriate practices, and lead to the right kinds of responses.  A discussion of what needs to be done is outlined below.

Evaluation Points
Teachers need to decide what the appropriate points for evaluating what the students? knowledge and abilities are.  The traditional mid-term and final points may or MAY NOT be suitable.  The principle question is: at what point do you have to assure students have met a learning objective before they can go on?

The answer to this question will change depending on the kind of course goal guiding the instruction. Generally speaking, an evaluation point needs to be developed for each of the developmental levels outlined by your developmental learning goals.   For those whose primary goal is content, evaluation points should occur at the end of each of the knowledge stages.  Skill-based instruction should establish summative evaluation points for each expected skill (or skill cluster).  Learning skill courses must evaluate at each level of skill development.  Those whose primary goal is personal growth must evaluate stages of growth.

To place too many summative evaluation points in a course will distract teacher and students from the learning process.  However, to use too few summative evaluation points means that the teacher will have to assume (often mistakenly) that students have mastered prior skills or abilities well enough that the instruction can continue.  You can use the developmental learning curve to map out the most efficient use of summative evaluation points for your course.  If you were able to create a developmental rhythm for your course (Last week's note explained how to map the developmental learning goals into the semester schedule to establish a ?rhythm? for the course.), then you will be able to identify those points where one section ends as good evaluation points.  Even though you may not know how you want to evaluate students at this point (to be discussed below), you want to mark these points as places where you want to create some kind of evaluation.

Evaluation Purpose
What an evaluation point looks like depends on the kind of goals the teacher establishes.  Formal authority teachers evaluate content knowledge; skill-oriented teachers assess performance; those who teach for learning skills measure knowledge processes; and teachers who promote growth need to know about personal development.  Each of these has a different pattern in the instruction and thus the evaluation points will look different.

The problem when we first approach designing evaluation systems is that the content-only paradigm is so compelling that we may use it at points where it doesn't fit.  We can get so wrapped up in measuring content that we may fail to assess the changes that tell us whether students are reaching our intended goals.  It is better to develop evaluation points according to the kinds of goals you have chosen.

The first step in creating such evaluation points is to clarify the various levels of ability you expect to find at these points.  Describe the criteria for low, medium, and high performance.  For content-oriented teachers, this will probably be defined in terms of amount of content that can be recalled and/or used to answer questions.  For skills-oriented teacher, this will be defined in terms of competencies mastered.  For learning-oriented teachers, the evaluation should measure the kinds of strategies used to address a problem.  For growth-oriented teachers, the measure is likely to be in the quality of reflections a student can provide.

Once you are clear about the purpose of the evaluation point, you can choose the kind of criteria that will tell you how ready the students are to move forward.  This process allows you to break from the assumption that only measures of content count as evaluations.  In fact, teachers in a distance environment need to have a more rich evaluation design so that they will get feedback on how their students are doing.  Deciding on the best method to gather reliable data on how students match up to these criteria is the next step.

Evaluation Practices
Evaluation depends on accurate data gathering and interpretation methods.  These vary by purpose and so, need to be appropriate to the purpose & criteria. There is, for example, an over-reliance on the standard exam format given under strictly controlled circumstances.  While this is a good, valid data collecting instrument, it is the most difficult under distance education conditions (and a great deal of energy is spent trying to resolve the related problems).  In fact, only a few teacher need to worry about this

For content-oriented teachers who decide that a traditional test is the best data collecting tool there are software programs that allow on-line testing.  Test security is an issue that is usually resolved by on-site supervision.  There are, of course, guidelines on writing effective test items.  There are other data collecting methods that these teachers might consider, including: essays, reports, and oral exams.

 There are other evaluation protocols used by other teaching approaches.  Skill-oriented teachers often develop tasks or performances that are monitored.  In distance education, the teacher can assess the final product, a recording of the student doing the task, or an outline of the steps followed (and results achieved) at each phase of a task.  Learning-oriented teachers may also observe a task ? which may be writing an essay, developing a presentation, doing a project ? but will focus more on measuring how well the student followed the learning skills presented.  A nice example of this model can be   Guy Bensusan, Arizona State University, has developed a number of ?Teaching Tools? that outline learning sequences students need to follow (the hexadigm and the ladder are two examples) and gives examples of how his assignments tell how well student applying these skills (see http://jan.ucc.nau.edu/~hgb/ttools.html).  Those teacher interested in personal growth must, like Bensusan, develop activities that illustrate what kind of developmental growth students have been accomplishing.  Eventually, teachers develop a repertoire of evaluation techniques that are appropriate to each evaluation point.

The main point, here, is that there is no need to rely on the monitored test in a distance environment.  There is a range of methods for getting students to show you what they can do.  Start with the criteria and then decide what technique is best for gathering information.

Evaluation Responses
There are two parts to any evaluation: rating & feedback.  The one provides a quality indicator, the other creates an additional, and essential, teaching moment.  The tendency is to focus on the ratings (grades) and to provide ?ritualized? responses that do not provide much teaching.  For distance education students this can shift motivation from learning to performing.  Without the other classroom supports, ?doing it for the grade? may not be sufficient.  Also, given limited feedback opportunities, less precise feedback can leave a student lost.

The distance educator needs to develop clear, helpful responses.  Because you have developed some clear criteria, students should be able to rate themselves (not that they will) when they submit their work.  They should be able to look at a checklist, for example, about what should be in an essay and determine whether or not they have included these items.  The less mystery about what is expected, the better.  The reasons for the grade should be obvious.

The more important role for the feedback is how it guides learners to continue their work.  Responses should identify which of the current material that is being handled successfully and what difficulties provide the next challenge.  You should highlight the pattern that reveal their mistake & direct the students to the steps for improvement.  Like a computer program debugger, the teacher wants to spot the point where an error occurs as a point that must be fixed.  Fortunately, experienced teachers can anticipate common errors and pre-program appropriate responses in order to save time.

FORMATIVE FEEDBACK
The formative feedback process aims to find out 1) how we are doing right and what we need to improve, and 2) what students are doing right & they need to improve.  The formative evaluation process is usually viewed as more informal.  Teachers ask students how they feel or they offer sample quizzes or review a student's work before it is due.  Even though these assessments seem informal, they  should not be casually planned.  In a distance learning environment, formative evaluations are crucial for assuring teacher and students that progress is being made.  The planning of a formative evaluation system follows the same decision process as described above.

Evaluation points
It is wise to build in formative evaluation as a part of each learning zone. Two key times to concentrate on are 1) around bottleneck points 2) when uncertain or problematic situations are likely to emerge.  Bottleneck points are those lessons that explain fundamental concepts or skills.  Students without a firm understanding of this information are likely to misunderstand what follows.  Experienced teachers tend to spend extra time on these points to assure that students have a good grasp on this subject matter.

There are also moments that an experienced teacher knows to expect confusion or uncertainty.  I know, for example, that the first few times I introduce the concept of democracy student will assume I mean ?absolute individual freedom? no matter how eloquently I explain it as a ?form of  regulated social living.?  I inevitably need to find out who is still using their prior conceptions about individual liberty before I know that my social concept has been learned.  In short, this is a good point for formative evaluation.

Formative evaluation is also a valuable tool to add into a course whenever unanticipated confusion or unexpected roadblocks seem to be hindering student progress.

Ironically, the most common use formative feedback is through end-of-semester course evaluations when, unfortunately, it is too late to adjust the instruction.  The best strategy, then, is to create multiple occasions for gathering informal feedback.  Depending on the purpose, feedback can be directed to the teacher, peers, or self.

Evaluation Purpose
There are a number of processes going on in the classroom that can inform your understanding of what is going on.  You may want to know what students know (without using graded activities).  You may want to allow students to check skills.  You may want to see how well they are processing information or how they are changing.  On the other hand, you may want to know how well your activities are going.  Was your web site clear enough?  Was it adequate?  Do you need examples?  Was the discussion useful?

It would be helpful to have a continual monitor on all of these dimensions all the time.  Obviously, this is not very practical.  Instead, the teacher need to strategically think through which of these areas would provide the most useful information -- that is, what information would most informative in deciding what was contributing to or restricting the success of the course.

Review the formative evaluation points you feel you will need in guiding students through the developmental learning curve you designed for your course.  Decide what kind of information you would like to have ? information about the teacher, the students, or the learning activities.  For example, would it help you to know if students were able to apply the reading during their small group discussion?  Or, would it help you to know what doubts students still have about the subject?  Do you want to know if students feel nervous or comfortable about performing a new procedure?  Or, do you want to know if students completing reading assignments at all?  Such information can be gathered by using formative evaluation practices.

Evaluation Practices
Formative evaluation practices follow a slightly different process than summative evaluations.  A cycle of change is implied in most formative processes that leading from problem identification to corrective action.  Some models, like quality improvement or action research, guide teacher and/or students through specific steps.  Others are more informal.  Most have data collecting and reporting strategies (they are usually anonymous, for example).

In order to gather and use formative information teachers need to develop a repertoire of techniques that will collect the data they desire efficiently.  Angelo and Cross, Classroom Assessment Techniques provide 50 examples of methods that can generate insights into almost any teaching goal an instructor may have.  There are a number of other methods for gaining feedback on lessons and learning activities, such as the Small Group Interaction Diagnosis.  More formal and involved methods involve classroom action research methods.

There is little enough attention to the need for formative evaluation in regular classrooms.  Faculty don't usually have many routines or  techniques.  Adding the time to conduct formative evaluations into crowded schedules is a problem.  Fortunately, asynchronous media allows the teacher to establish formative feedback points that can be accessed when convenient.  Email, discussion boards, and electronic forms can all serve to give the instructor a steady flow of information about how the course is going.  Even more helpful is the fact that any additional time spent conducting or reviewing formative data is usually recovered later, when class is running smoothly.

Evaluation Responses
The results of formative evaluations are meant to improve and enhance future action.  Thus, the results must be reported back to the teacher and/or the students in a format that will induce them to reflect on the information and adopt new behavior, where appropriate

The biggest problem is not with the expertness by which these evaluations are conducted but rather that the information is received too late to take action.  By incorporating formative processes throughout the course, this problem is resolved.  A second issue is whether or not the teacher should report the results back to the group and include them in making decisions about what changes to make.

In general, students react positively to any real effort made to addresses identified concerns, even the barest, most simple efforts.  When you plan your course in developmental stages, you can shape each section based on feedback from students in the prior section.  Without the casual feedback opportunities of the face-to-face classroom, teachers often feel cut-off from students and can become uncertain how they class is progressing.  Designing an effective formative evaluation system anticipates this problem and establishes a mechanism to assure that there is enough feedback opportunities to support learning efforts.

DISCUSSION QUESTION
Look over the course you are transforming and identify a formative or a summative evaluation point.  Think through how you might design this evaluation ? What is its purpose, what is a good practice, and what is the best way to give students a response?  Share your ideas with your discussion group.  Raise any questions or concerns you have about developing your own evaluation system