 |
Distance Education at a Glance
Guide #4
October 1995 |
Engineering
Outreach
College of Engineering
University of Idaho
|
Index | Guide 1 | Guide 2 | Guide 3 | Guide 4 | Guide 5 | Guide 6 | Guide
7 | Guide 8 | Guide 9
Guide 10 | Guide 11 | Guide 12 | Guide 13 | Glossary
Guide #4
Evaluation for Distance Educators
Effective teachers use a variety of means, some formal and others informal, to
determine how much and how well their students are learning. For example, to formally
evaluate student learning, most teachers use quizzes, tests, examinations, term papers,
lab reports, and homework. These formal evaluation techniques help the instructor to
evaluate student achievement and assign grades.
To evaluate classroom learning informally, teachers also use a variety of techniques.
For example, teachers pose questions, listen carefully to student questions and comments,
and monitor body language and facial expressions. Informal, often implicit evaluations
permit the teacher to make adjustments in their teaching: to slow down or review material
in response to questions, confusion, and misunderstandings; or to move on when student
performance exceeds expectations.
When teaching at a distance, educators must address a different teaching challenge than
when teaching in a traditional classroom. For example, instructors no longer have:
- A traditional, familiar classroom.
- A relatively homogeneous group of students.
- Face-to-face feedback during class (e.g. students' questions, comments, body language,
and facial expressions).
- Total control over the distance delivery system.
- Convenient opportunities to talk to students individually.
For these reasons, distance educators may find it useful to not only formally evaluate
students through testing and homework, but to use a more informal approach (see Angelo and
Cross, 1993) in collecting data to determine:
- Student comfort with the method used to deliver the distant instruction.
- Appropriateness of assignments.
- Clarity of course content.
- If class time is well spent.
- How a course can be improved.
Evaluation can be either formative, summative, or a combination of both.
Formative evaluation:
- Is an on-going process to be considered at all stages of instruction.
- Will enable the instructor to improve the course as he/she proceeds.
- Facilitates course and content adaptation.
- Will identify major gaps in the instructional plan or the need for minor adjustments.
Some strategies that educators can use to collect formative data from their distant
students include:
- Post cards - provide each student with prestamped and preaddressed postcards. On a
weekly basis, have students use the postcards to share their concerns or respond to
questions during the last three to five minutes of class.
- Electronic mail - Can be a very effective way for instructors and students to
communicate. Another plus, while the instructor is eliciting information about classroom
learning, students become familiar with the use of electronic mail, a valuable skill.
- Telephone - Call students often. Ask them open ended questions (e.g., "What snags
did you run into on the second writing assignment?") to let students voice their
concerns. Follow with probes (e.g., "Then, will you need more information
sources?"). Set phone-in office hours but be sure to welcome calls at other times.
Summative evaluation:
- Assesses overall effectiveness of the finished product or course.
- Can be a springboard in developing a revision plan.
- Can be a baseline of information for designing a new plan, program, or course.
- Will not help current students since it is conducted upon course completion.
Some questions that educators may want to ask students when collecting summative data
include:
- List five weaknesses of the course.
- List three (or five) strengths of the course.
- If you were teaching the course, what would you do differently?
- Student background information: age, level in school, number of distance delivered
courses taken prior to this one.
- What would you recommend to a friend planning to take this course?
- What did you think would be covered in this course but was not?
- Would you recommend this course to a friend? Why or why not?
Within the context of formative and summative evaluation, data may be collected through
quantitative and qualitative methods.
Quantitative evaluation:
- Involves asking questions which can be statistically tabulated and analyzed, frequently
using a scale, check list, or yes/no responses.
- Limits students to responding to the categories made available to them.
- Needs a large student sample for relevant statistical analyses.
Quantitative methods may be most useful for gathering information on large numbers of
respondents for whom more in-depth, personalized approaches are not feasible. However,
they do have some significant drawbacks:
- Many distance education courses have relatively small class sizes with students from
various backgrounds. These small, stratified populations typically defy relevant
statistical analysis.
- Quantitative surveys typically result in a rate of return of under 50 percent. A low
rate of return often suggests that only those feeling very positively or negatively about
the course responded to the evaluation.
- By definition and design, forced choice surveys offer respondents a limited number of
possible response options. Therefore, fresh insights and unique perspectives falling
outside the provided response categories go unreported.
- The cumbersome and often tedious nature of quantitative data collection can discourage
formative evaluation, and often results in an over-reliance on summative evaluation.
- Statistical analysis often results in an illusion of precision that may be far from
reality.
Qualitative evaluation:
- Is typically more subjective.
- Involves gathering a wider range and depth of information.
- Is more difficult to tabulate into neat categories.
- Will be less affected by typical small class size.
- Is a more flexible and dynamic method.
- Is not limited to pre-conceived topic of inquiry.
- Allows for student output of topics.
Can use:
- Open ended questioning -- with respondents asked to identify course strengths and
weaknesses, suggest changes, explore attitudes towards distance delivery methods, etc..
- Participant observation -- with the distance educator observing group dynamics and
behavior while participating in the class as an observer, asking occasional questions, and
seeking insights regarding the process of distance education.
- Non-participant observation -- with the distance educator observing a course (e.g., an
audioconference, interactive television class, etc..) without actually participating or
asking questions.
- Content analysis -- with the evaluator using predetermined criteria to review course
documents including the syllabus and instructional materials as well as student
assignments and course-related planning documents.
- Interviews -- with a facilitator or specially trained individual collecting evaluative
data through one-on-one and small-group interviews with students.
Consider the following areas:
- Use of technology - familiarity, concerns, problems, positive aspects, attitude toward
technology.
- Class formats - effectiveness of lecture, discussion, question and answer; quality of
questions or problems raised in class; encouragement given students to express themselves.
- Class atmosphere - conduciveness to student learning.
- Quantity and quality of interaction with other students and with instructor.
- Course content - relevancy, adequate body of knowledge, organization.
- Assignments - usefulness, degree of difficulty and time required, timeliness of
feedback, readability level of print materials.
- Tests - frequency, relevancy, sufficient review, difficulty, feedback.
- Support services - facilitator, technology, library services, instructor availability.
- Student achievement - adequacy, appropriateness, timeliness, student involvement.
- Student attitude - attendance, assignments submitted, class participation.
- Instructor - contribution as discussion leader, effectiveness, organization,
preparation,enthusiasm, openness to student views.
- Check out and adapt already published questionnaires; there’s no need to re-invent
the wheel.
- Draft and revise questions; change if necessary.
- Make use of follow-up probes:
- Alternate between instruction and interaction.
- Sequence your questions for best effect - go ahead and ask for suggestions for
improvement before asking for what is good. This will help convey sincerity for seeking
improvements.
- Place open ended questions after quick answer questions. This gives students built-in
thinking time.
- On summative evaluation, assure anonymity. This can be accomplished by having all
questionnaires sent to a neutral site where they would be removed from their envelopes and
forwarded to the instructor without a postmark.
- Establish rapport by being interested and supportive. Withhold judgmental responses.
- Adapt to the student in degree of formality and pace of communication.
- Use evaluation as a method for understanding teaching and learning.
- Try to get both positive and negative feedback. It is important not only to know what is
not working, but also what is working.
Angelo, T. & Cross, P. (1993). Classroom assessment techniques: A handbook for
college teachers. San Francisco: Josey-Bass Publishers.
Willis, B. (1993). Distance education: A practical guide. Englewood Cliffs, NJ:
Educational Technology Publications
This guide is one in a series developed by Barry
Willis and the University of Idaho Engineering Outreach staff highlighting
information detailed in Dr. Willis' books, Distance Education - Strategies and Tools
and Distance Education - A Practical Guide. Other guides in this series
include:
#1 Distance Education: An Overview
#2 Strategies for Teaching at a Distance
#3 Instructional Development for Distance Education
#4 Evaluation for Distance Educators
#5 Instructional Television
#6 Instructional Audio
#7 Computers in Distance Education
#8 Print in Distance Education
#9 Strategies for Learning at a Distance
#10 Distance Education: Research
#11 Interactive Videoconferencing in Distance Education
#12 Distance Education and the WWW
#13 Copyright and Distance Education
#14 Glossary of Distance Education Terminology
| Distance at a Glance Index | Engineering Outreach Home Page |
This guide was originally edited by Tania H. Gottschalk
EO web presence design by Robert A. Neer
University of Idaho
Engineering Outreach
Comments about this page to: outreach@uidaho.edu
Why Evaluate?