W E L C O M E !

Evaluation

Evaluating the effectiveness of the NEEDS digital library is of primary importance to the users and sponsors of this resource. We intend to develop a thriving community of educators and learners actively engaged in the full lifecycle of engineering education resources, from creation to curation, dissemination and use. An expert in digital libraries and their evaluation - Prof. Nancy Van House at the University of California, Berkeley - will serve as our development and implementation consultant. We are constantly looking for ways to improve and make changes daily but we have also conducted, and continue to conduct, a number of evaluation activities to systematically assess the performance of the library in a number of key areas.

In addition to a number of past studies on the effectiveness of the NEEDS digital library, see bottom of page, we have a number of ongoing activities to provide us with greater insight into the user experience on NEEDS.

Performance and User Metrics

To perform user evaluations, we are drawing upon prior studies, and methods and instruments used by NEEDS, providing a useful benchmark against which to measure progress towards current objectives. To better understand user needs, specific quantitative goals will be developed through discussions with partner communities. We are expanding performance monitoring applications to automatically report access and performance measures of the system's infrastructure.

Our current infrastructure is instrumented to enable our collections, in accordance with our privacy policies, to track usage of a number of key metrics (downloads, queries, federated search queries, 'successful' recommendations, etc.). These usage metrics enable us to monitor the variations in performance and activity on NEEDS in response to, for example, changing monthly themes (e.g. live summary of monthly usage statistics).

Quantitative and Qualitative Evaluation

We are using qualitative evaluation techniques to understand the quality of the users' experience and how resources are used. We currently mine the server log data to provide a better understanding of how much time users spend with resources and how these resources are accessed. In addition to these in-depth metrics there are a number of ongoing activities that enable cross-referencing of these results.

Survey

We participated in the NSDL-wide DigiQUAL evaluation survey. DigiQUAL grew out of LibQUAL, an evaluation of traditional brick and mortar libraries. DigiQUAL is bringing the rigor and expertise of LibQUAL and applying it to digital repositories.

Thanks to all those who participated in the survey run from May 2005 through July 2005.

Heuristic Evaluation

We are also performing heuristic evaluations based on general web design heuristics and heuristics more specific to the design of digital repositories. Usability heuristics are general principles for user interface design that help to improve the user-centered approach and experience of a site's visitors. For an introduction to and some of the most widely used heuristics for web site evaluation see Nielsen's 10 usability heuristics and Bruce Tognazzini's first principles of interaction design.

Usability Studies

We are performing a number of task-based usability studies with select users from NEEDS' target user groups. The usability studies are focused by specific tasks to perform such as searching, registering, cataloging and commenting on resources.

Other Activities

In addition to the three main activities discussed above we will be performing a range from the following activities: Usability inspection analysis, competitive analysis and lead-user qualitative interviews

For an explanation of each usability assessment method we recommend the usability methods table at usabilitynet.org.

Advisory Committee

The co-chairs of our advisory committee are:

John Prados served as president of ABET (1991-92), senior education associate in the NSF Engineering Directorate (1994-97) and was founding editor-in-chief of the Journal of Engineering Education. In addition, he served as vice president of the University of Tennessee, where he received a Ph.D. in chemical engineering.

Barbara Waugh is a director for strategy in HP's University Relations and will work with Wayne Johnson, vice president for worldwide University Relations, on behalf of K-Gray sustainability. Developed Hp's breakthrough programs for women and minority recruiting and co-founded Hp's Sustainability Network and e-inclusion.

Read some of our past publications on evaluation

Teng X., J.G. Tront, B. Muramatsu, A. Agogino, Best Practices in the Design, Development and Use of Courseware in Engineering Education, to appear in Proceedings of the 2005 Frontiers in Education Conference, October 19-22, Indianapolis, Indiana

Shuang, S., A. Dong and A.M. Agogino. Modeling Engineering Information Needs. Journal of Computing and Information Science in Engineering, 2, No. 3, Sept. 2002, pp. 199-207.

Dong, A. and Agogino, A. M. Design principles for the information architecture of a SMET education digital library. Proceedings of the 2001 ACM/IEEE Joint Conference on Digital Libraries: June 2001, Roanoke, VA.

Puzniak, J., & McMartin, F. Building a digital learning community for faculty on the Internet. Proceedings of the 2000 American Society for Engineering Education Annual Conference: June 18-21, 2000, St. Louis, MO.

McMartin F., NEEDS User Study, Observations at the ASEE, June 2000

McMartin, F. Preliminary findings from "Science, Mathematics, Engineering, and Technology Education User Study" focus groups.

If you would like more information or have questions about our evaluation activities please contact jono {at} Berkeley [dot] edu