The Evaluation Station

Free instructions and downloadable materials

This activity is offered free of charge for your use in educational settings or World Usability Day events.

On this page: Activity Summary | Learning Goals | Preparation | Setup | Moderating

Activity Summary

Description:

Non-profit agencies and other community groups are invited to introduce their websites to a rotating team of usability professionals. These professionals have a subsequent round-robin evaluation discussion for one hour of the website's usability, information architecture, visual design, conceptual model, nomenclature, and task-appropriateness. This discussion is held in an open location where the public can observe and interject. The website is projected onto a screen where all can see it.

In Boston, MA in 2005 we ran two tables of reviewers and a the end of the day 15 community group websites were reviewed and over 75 usability professionals had participated in the reviews! We limited evaluators' individual participation to no more than two sessions and ensured that each review group had at least one "senior" professional.

Activity type:

Table-based activity

Good for:

Community based organizations, usability professionals, observers

Requires:

electricity, computer, Internet access, LCD projector, projection screen

Minimum activity time:

1 hour

Docents/Moderators:

A rotating team of six reviewers for every website/product examined, volunteer coordinator/scheduler

Equipment:

An environment where approximately eight persons can sit comfortably around a table or in a theatre-like format that can accommodate observers
An LCD projector
A computer or laptop
Internet access
Projection screen
Microphone and speaker (optional)
Moderator's packet
Some means of recording the session (optional)

Signage:

Download the Activity Signage (526K PDF)

Downloadable files:

Moderator Tally Sheets (MS Word, PDF)
Data collection spreadsheet (MS Excel, PDF)

Learning goals

To encourage individuals to think critically about the products they routinely encounter. To discuss ergonomics, display technologies, button types, and how these and other factors influence a product's ease of use. Encourages participants to consider function over form when considering products.This live demonstration of usability in action helps newcomers to the field to understand the importance of user centered design in accomplishing business goals, and provides a valuable free service to agencies who otherwise might never have the ability to engage seasoned usability professionals to advise them.

Observers are encouraged to think critically about the websites/products they encounter and participating organizations receive structured, actionable advice. Participating usability practitioners are discouraged from selling their organizations or services during the sessions, but are encouraged to continue to provide advice after the event.

Activity Preparation

Invite community groups and area non-profits to submit their websites for this live usability review.
Inform respondents of the nature of the event, the time they should arrive, the location they should go to, the form the discussion will take, and the types of information they will take home with them.
Recruit a number of usability professionals to participate in this activity.
Inform the professionals of the nature of the event and the need for generation of an actionable, coherent list of next steps.
Give each reviewer a moderator's packet for each site they will review. Encourage them to familiarize themselves with the website before the event, and to bring any notes/summary lists of suggestions to the event.

Activity Setup

Set up a table and chairs so that a seated group discussion may be had, but also so that observers may easily see/hear the reviewers, perhaps in a panel format or half-circle
Set up the LCD projector, computer, table, and projector screen so that the computer screen is projected and visible to all activity participants
Ensure that the computer has Internet access (otherwise, participants must bring prototype or disc versions of their websites)
Place an active microphone on the table to either capture the group discussion or to pass from speaker to speaker

Activity Moderation

This activity works well when it has an activity leader who can greet each community group representative as they arrive, introduce each hourly session, and who can facilitate the rotation of reviewers.

Invite all participants to take their seats. Inform observers that their participation will be welcome once the session is underway.
Nominate one reviewer to be the group's note taker. If at all possible, record the session in a format that the community group could take with them (videotape, Morae file, audio cassette).
At the beginning of each hour-long session, welcome and introduce that hour's community group.
Invite the community group representative(s) to briefly introduce themselves, state the organization they represent and the nature of their work.
Invite the professional reviewers to briefly introduce themselves and their areas of specialty (as appropriate)
Invite the community group representatives to give a five minute overview of the purpose of their website/product and the audiences it serves.
Invite the reviewers to spend no more than 5 minutes asking clarifying questions about the site's purpose and target audiences.
Starting with the homepage (as appropriate), the reviewers begin to discuss the product's usability strengths and weaknesses. Special emphasis is given to ensuring that reviewers use a minimum of jargon, are not unproductively critical, and that they identify actionable, prioritized suggestions for enhancing the product.
Note: the goal of this activity is not a "free for all firing squad" of criticism, but rather uncovering low-cost, high impact ways of enhancing the community groups' products. At the end of the session we hope to have opened their eyes to the value of user centered design, and to have at least a handful of "next steps" and straightforward enhancements they, or those they work with can make.
During the session, encourage the community group to participate in the discussion, ideally not in a defensive vein, but to help weigh the constraints they are laboring under versus the suggestions being made.
Fifteen minutes before the end of the hour, encourage audience members to contribute observations, ask questions, comment on the product, or respond to any of the comments made before that point.
Ten minutes before the close of the session, invite the group to summarize their findings and prioritize the action steps discussed. Ensure that the community group representatives are able to voice their reactions.
At the close of the hour, thank everyone involved and encourage the reviewers who will not be involved in the next session to step outside the activity area.
Repeat the process for each subsequent hour of reviewing.