An electronic database of engineering educational courseware, called the National Engineering Education Delivery System (NEEDS), has been developed by the SYNTHESIS Coalition. The SYNTHESIS Coalition is a National Science Foundation sponsored coalition of eight schools dedicated to improving engineering education.[2] The premise of the SYNTHESIS Coalition is that a database of curricular materials in electronic form, or courseware, accessible by students and faculty alike, can enhance the learning experience in any context. While the content of the NEEDS database was originally created by members of the SYNTHESIS Coalition, NEEDS has expanded to include non-commercial courseware developed by other NSF-funded Coalitions, as well as other interested participants. This paper will review previous efforts in courseware review by others, and then discuss a peer-review system that is being implemented for courseware on the NEEDS database.
Engineering courseware is computer-based educational material that can be used to assist engineering students in their learning process and assist instructors in their pedagogy. Courseware can be used in lectures, during recitation sections, as self-paced study, as reference material for the student, or as exercises for the student to perform alone or in a group. Typically, courseware takes advantage of multiple media, such as graphics, photographic images, text, sound, video and animation to illustrate engineering concepts, devices, or practices. Courseware will often include features such as hyperlinks or hypertext which permit users to explore related information or greater depth of information as they are interested.
Since NEEDS is an electronic medium, it is highly flexible and subject to great variation. The quality of content and technical features of courseware on the NEEDS database is highly variable for several reasons; 1) the rapid changes in multimedia technology; 2) improved authoring environments; and 3) evolving practices of using educational courseware over the last few years. This diversity is a strength of the NEEDS database, since it encourages students, instructors, and courseware developers to explore the NEEDS database, experiment, download material, and incorporate the material into lectures or reports. While this diversity of material on NEEDS is an asset, it also poses a challenge for users who are looking for reliable, tested courseware that can be incorporated into a classroom with minimal effort.
Courseware on NEEDS, as with any textbook or journal publication, requires
review to assure quality. To implement this review, we first examined
existing schemes that evaluated courseware, then developed our own. We will
implement a peer review system
that results in a three-tiered ranking of
courseware on the NEEDS database: non-reviewed material, Endorsed courseware,
and Premier courseware.
Before discussing the three-tiered ranking in more
detail, I will first provide background on classification of courseware, review
existing evaluation methodologies, and then describe our initial attempts at
developing a detailed and thorough peer evaluation procedure for NEEDS. The
paper concludes with the final peer-review system we will implement.
Digital instructional material available on the NEEDS database has been
classified into three major categories, based upon its complexity and purpose.
These categories are 1) elements, 2) collections, and 3) courseware. An
element is the smallest form the piece of digital information can take, i.e. it
can be described and assessable only as a single entity.
Courseware is digitally-based material logically structured to convey at least
one coherent educational theme and/or concept. Courseware can be classified as
a "courseware module", which has a narrow scope of educational content, or a
"curricular unit", which is composed of multiple courseware modules that convey
a complete unit of curricular material. Examples of engineering courseware
modules include a discussion of Newton's Second Law, solution of a dynamics
problem, or a description of a measurement device. Examples of curricular
units include courseware covering topics in a course on dynamics, or a
case-study of the multidisciplinary design of a wheelchair.
The format of courseware on the NEEDS database varies considerably, including
case studies, tutorials, technical references, support for laboratory
experiments, educational games, demonstrations, and sets of practice problems.
At this point in time, the digital material on the NEEDS database consists of
approximately 230 elements, 20 collections, and roughly 210 courseware
modules/units. For example, an element might consist of a portrait of a
scientist, a video clip of a bridge, or a digitized audio record of an
interview. A collection is a set of individual elements that are related, with
documentation of each element and the relationship between elements in the
collection. Examples of a collection include a set of flow visualization
images, a set of bridge digital photographs, or a set of Mozart audio clips.
Most of the courseware on NEEDS would be classified as curricular units,
designed by an instructor to be used in his/her course. The quality varies
considerably, varying from incomplete "alpha" versions of untested courseware
to polished, well-documented functional courseware.
Establishing a review methodology for the NEEDS database has not been
straightforward since standard evaluation procedures have not been established
for courseware. Many different evaluation models for instructional courseware
have been proposed based on different philosophical views on education,
conceptualization of educational processes and products, and methodological
orientations. (Popham, 1974; Worthen and Sanders, 1973). To a large extent,
the lack of controlled evaluation procedures for computer-based instruction has
lead to disagreement regarding their success in improving student learning
(Baker, 1990). There is also great skepticism in the education community
regarding the efficacy of any evaluation scheme for software evaluation
(Tucker, 1989; Bates, 1981; Borich and Jemelka, 1981; Micceri et al, 1989).
These authors take the position that courseware evaluation schemes need
improvement and are highly subjective since they depend on the goals of the
designer and the context of use.
Bonekamp (1994), in an effort to identify a standard evaluation procedure for
courseware in Europe, conducted a study of 19 organizations within 10 member
states of the European Community and found a diversity of both formative and
summative evaluation activities. Formative evaluations, conducted by most of
the organizations, identify weaknesses in a product early enough to implement
design changes. The summative evaluations, which are conducted at the end of
major development to assess the various aspects of a finished product, varied
from a critical appraisal of a product by an expert, to extensive in-depth
testing.
Duchaste (1987) proposes three prototypical models for summative evaluations:
The first of these evaluation models, the product review, is highly subjective.
This methodology can offer, though, significant information to potential users
if a comparative review is conducted and the results of the review widely
published to potential adopters of the product.
The last of these prototypical evaluation models, experimental observation, has
greatest potential to offer in-depth, accurate assessment of the educational
value of the courseware. In fact, most evaluation models for educational
processes are based on an in-situ assessment in which the learning is
contrasted between a control group and an experimental group.
Shute and Regian (1993) provide a framework within which an experimental
evaluation methodology for Intelligent Tutoring Systems (ITS) could be
standardized. While ITS involve computer-based tutoring that is based on
artificial intelligence models, the framework these authors propose is also
applicable to the evaluation of interactive educational courseware. Shute and
Regian propose that if the following principles are followed, an effective
evaluation procedure can be designed, planned and implemented:
While implementing an experimental-based evaluation methodology following the
above guidelines has the potential to provide the best indication of the
effectiveness of a piece of courseware as a learning tool, this procedure is
extremely costly in terms of time and person-hours. Barker and King (1993)
propose an evaluation methodology based on a check-list in which they
identified "hallmarks of quality" that characterize good learning products.
They defined 12 basic categories that embody good learning design: engagement;
interactivity; tailorability; appropriateness of multimedia mix; mode and style
of interaction, quality of interaction; quality of end-user interface; learning
styles; monitoring and assessment techniques; built-in intelligence; adequacy
of ancillary learning support tools; and suitability for single
user/group/distributed use. This evaluation checklist was used to assess 43
wide-ranging types of computer-based learning and training products, and Barker
and King found that the quality of end-user interface design was of paramount
importance in producing a quality product. In addition, engagement,
interactivity, as well as tailorability, were found to be other important
hallmarks of quality.
In our opinion, the shortcoming of the check-list evaluation methodology
suggested by Barker and King is that it only addresses student learning on a
superficial level. Research in the area of cognitive science has indicated
that certain learning models and tools that may be present within courseware
have a greater potential to improve learning than others (Weyer, 1988;
Anderson, 1988; Kozma, 1991). By reviewing courseware for these features, the
potential for courseware to improve learning can be predicted without resorting
to an experimental user assessment.
NEEDS courseware was initially evaluated using questionnaires to avoid the high
cost associated with empirical testing of courseware in multiple contexts and
learning environments. A checklist or questionnaire methodology meets our
requirements since they can be filled out by peer reviewers at different
locations in a relatively short period of time. Another advantage of using a
detailed questionnaire is that it can guide peer-reviewers to focus on
important and critical features of courseware that contribute to a quality
product.
We initially chose to model the evaluation process for material on the NEEDS
database on the checklist developed by Barker and King (1993), as well as the
questionnaires by Mead and Gay (1995), and Huston et al (1994). We developed
a draft questionnaire, and solicited a panel of experts in the area of
computer-based education, courseware authoring and publishing to review our
questionnaire. This workshop was held at California Polytechnic State
University, San Luis Obispo, CA in May of 1995. Quality Review Workshop guests
included SYNTHESIS participants, as well as invited guests from other
Coalitions, engineering educational organizations, authors of courseware and
commercial publishers. Participants worked in groups to conduct mock reviews
of an assortment of courseware from the NEEDS database, as well as courseware
developed external to SYNTHESIS. This process helped identify and refine key
aspects of the evaluation instrument, namely the questionnaire.
The questionnaire included nine primary categories for evaluating courseware:
engineering content, engagement, impact on learning, user interface, user
interaction, multimedia design, instructional use, technical performance, and
accessibility from the NEEDS database. Each of these categories are described
briefly below, including a summary of the type of questions included in the
detailed questionnaire:
During the feedback sessions, we were convinced that the detailed questionnaire
was not going to be a successful tool for peer review of courseware. Comments
from participants included:
We re-evaluated our approach to the peer review process for the NEEDS database.
We realized that the questionnaire is too cumbersome, expecting more from both
authors and reviewers than we had intended. Requiring evidence of improved
learning over traditional teaching, whether this is based on a questionnaire or
an assessment of learning in the classroom, was too high a standard to set for
accepting courseware as reviewed on the NEEDS database.
Our goal, in the peer review process, is to establish that courseware has
correct technical content, is easy to use and is potentially useful to
instructors other than the author. To accomplish this in a straight-forward
manner, we are going to implement a two page peer-review form that provides a
"gestalt" review of the courseware. This form is shown in Appendix A. The
form includes one page of author supplied information specifying. among other
things, the educational goals of the courseware, its intended pedagogical use,
and the target student audience. The reviewer is asked to assess if the
engineering content is error-free, if the author's stated target audience and
educational goals are consistent with the courseware's content, if the
courseware is visually appealing and functional, if any copyright infringements
were observed, and to recommend if this courseware be Endorsed by the NEEDS
database.
However, in order to also recognize the exceptionally high quality pieces of
courseware that will be on the NEEDS database, we will implement an annual
award for "premier" courseware, hopefully in collaboration with a national
engineering education organization.
The NEEDS database will have a three-tiered structure for courseware,
based on the level of review the courseware has undergone.
1. Non-reviewed material.
Non-peer-reviewed material will be retained on the NEEDS database since
this will provide a large repository of elements, collections, and courseware
that can either be used "as-is" or modified as appropriate for the intended
academic application (depending on copyright restrictions of the author). The
non-reviewed materials on the database will provide a function similar to that
of "free-ware" in the computer-program domain, in which users may find a
diversity of quality and usefulness in the material available. In spite of the
disparity in quality we expect on NEEDS, it is important to keep this source of
"pioneering" courseware available to foster the creativity for the next
generation of courseware.
2. Endorsed courseware.
This courseware has undergone a peer review process to ensure the
engineering content is error-free, the package is complete with author
descriptions of content and recommended pedagogy, the courseware is
aesthetically appealing and is potentially useful to instructors other than the
author. While the review process does not determine if learning is enhanced by
using this courseware, potential users should expect the courseware to add
value to their instruction.
3. Premier courseware.
The Premier designation is reserved for the exceptional piece of courseware
that represents a particularly creative or innovative use of multimedia
courseware for instruction. This courseware must have documented evidence that
it has measurably increased student understanding of the given engineering
topic with respect to traditional lecture/textbook format. Authors must submit
student assessments of the courseware, a learning assessment, documentation of
effective pedagogical applications of the courseware (such as video clips or
published papers), and the courseware must have been used and assessed by at
least one instructor other than the author. We will annually recognize one or
two Premier courseware packages with an award ceremony at a national
conference.
When authors submit courseware for inclusion in the NEEDS Database through the
NEEDS Manager, they will have the option of requesting their courseware be peer
reviewed. The peer review process will determine if courseware will receive
the Endorsed category on the NEEDS database.
Below is a chart of the flow of courseware through the evaluation system for
the NEEDS database. All courseware submitted to the NEEDS database will be
reviewed for functionality by the NEEDS Manager and then either placed on the
NEEDS database, or returned to the author for modification if the courseware
was not functional. If the author has requested, the courseware will also be
sent on to the NEEDS Editorial Board for peer review. If the courseware is not
accepted by the peer evaluations for the Endorsed status, it will still remain
on the NEEDS database as non-reviewed courseware.
The peer-review process will be modeled after that of the professional
publications. A NEEDS Editorial Board will control the process. The Editorial
Board consists of a NEEDS Editor and Associate Editors, all of which are
engineering instructors with experience using and/or developing computer-based
instruction. After receiving the courseware, the NEEDS Editor will pass it to
an Associate Editor with expertise in the courseware's technical area. The
Associate Editor will solicit reviews from individuals both internal and
external to SYNTHESIS concerning the courseware's content and pedagogy. The
reviewers will complete the gestalt questionnaire (Appendix A) and provide a
written review. The courseware will then either be accepted as is, or
recommended to be revised and resubmitted for review. Once courseware is
accepted as Endorsed , the author will be sent the written reviews and will be
given the option to have this review appended to the courseware's bibliographic
record.
The submission and review process for courseware to be considered for the
Premier category has not yet been established. We expect that courseware would
be submitted by the author to an annual award competition, which would be
coordinated by NEEDS in conjunction with another national organization.
Winners of the annual engineering education courseware award would be placed on
the NEEDS database with a Premier designation.
The NEEDS Editor will solicit comparative reviews of similar courseware on the
NEEDS database as it becomes more populated. These reviews will assist
potential users in choosing the most appropriate courseware for their
instructional needs. The reviews will be available on-line, and be sent for
publication in print media.
In spite of the excellent work that has been done related to evaluation of
computer-based instruction, implementing in-depth peer-evaluation of digital
course material submitted to a database is extremely difficult. We recognize
that any peer-review process must consider the trade-off between the time and
expertise of peer reviewers with the time-consuming efforts necessary to assess
improved learning through utilizing the courseware. We have chosen to rely on
a global and general questionnaire methodology, since our objective of the
peer-review process is to identify reliable, easy-to-use courseware that has
correct engineering content, that can be integrated easily into a learning
environment and that has potential to enhance learning.
It is essential that a database of courseware set standards and implement a
peer-review system in order to establish credibility as a valuable resource for
sound educational material. We predict the number of users will drastically
increase as they become confident that material they are accessing within the
NEEDS database is complete, technically reliable and aesthetically appealing.
Moreover, the peer review process will enhance the recognition of
courseware-developers for the scholarly and creative effort they have expended
in developing the courseware. In addition, the peer review process will create
greater exposure of courseware on the NEEDS database to potential users via the
peer reviewers. Eventually, an extensive peer review system for courseware
will elevate the quality of all courseware developed in the academic community,
as reviewers gain innovative ideas to incorporate into their own development
efforts and users raise the standard of what they are willing to work with in
the classroom.
This paper describes a collaborative effort of many people within the SYNTHESIS
Coalition, especially the members of the NEEDS Editorial Board (Jeff Huston,
Iowa State; Bisi Oladipupo, Hampton University; and Martin Ramirez, University
of Detroit, Mercy) and Dave Martin from Iowa State. The author would
especially like to acknowledge the contributions of Sherry Hsi of UC Berkeley,
whose first concern is always whether the students' learning is improved.
Classification of Digital Material on NEEDS
A Review of Literature Regarding Evaluation Methodologies for
Courseware
Preliminary NEEDS Evaluation Methodology
NEEDS Review Criteria
Review Procedure for NEEDS Courseware
Figure 1. Flow of courseware through the evaluation system for the NEEDS
database.
Summary
Acknowledgments