July, 2006
After a bit of chapter business and recognition for their amazing support of World Usability Day (WUD), four representatives from the Museum of Science's Research and Evaluation department presented on the following topics:
Chapter Business
- Chris gave an update about WUD. MOS has agreed to sponsor us again, and we're working on a plan to have some activities become permanent and to entire other museums nationwide to adopt the same activities. The quest for big sponsorship money is on track. We're hoping to have a larger planning meeting the week of July 31-Aug 3.
- The Deborah Mayhew "Designing for Efficiency" Workshop is just around the corner on August 12. Sign up at www.upaboston.org.
- In August we will be hosting Jared Spool to give a talk about optional design teams at Molecular. We are collecting submissions for 10 minute talks for our September meeting at Staples. Be sure to send yours in!
- UPA 2007 is now seeking submissions and volunteers to read submissions. Check out http://www.upassoc.org/conference/2007/ to participate.
- PJ is seeking volunteers for AIR, an event in Boston in January. Chris presented the MOS team with a "Friends of UPA" award.
Visitor Studies Presentation
Christine Reich, Manager of Informal Education Research and Evaluation at MOS, gave us some insight into their role at MOS. She described it as a cross between educational research and usability.
During visitor studies, they study museum visitors to determine their learning needs and entertainment values so they can incorporate those into policies, practices, and products. They assess the needs, interests and concerns of existing and potential audiences. Efficiency isn't the goal; learning is. They serve a very broad range of people (i.e. general public). Because the museum learning experience is intrinsic, or by "free choice", they have to figure out what motivates people to learn, and what barriers to learning exist that can be removed. They follow activity theory, which has three parts: subject (i.e. visiting groups), object (content) and mediating artifacts. Together, these three pieces lead to learning for visitors.
Christine's group assists product development by providing knowledge (i.e. data) about visitors. They are an in-house group that works to identify cross project patterns, as they hear directly from professionals in different areas. They work on exhibits, programs (e.g. staged demos), multimedia, and visitor services. They do market research, formative and summative usability studies, and research in the field as a whole.
Christine described how her team did this in the context of an exhibition about models. They used interviews to determine what people think of as being a "model", and used that information (and things they didn't say) to both engage visitors and educate them. They performed usability testing activities to determine the best materials that would encourage visitors to create minimalist models (yet be feasible for the MOS) - this I'm referring to as the "pipe cleaner debacle", which engaged much lively conversation in the group. : Because the model exhibition had to take into account the general public (20% of which they believe is disabled in some way), universal design was important.
Folks can visit www.visitorstudies.org to learn more about their yearly conference.
Q: The audience wondered how they test the efficacy of exhibits.
A: While observing visitors, they check off from a list of "desired actions", then they interview people about the value of the exhibit. From this they learn whether the value they assumed was there is really there, and how they could improve the experience. They do surveys for programs. Everything is around testing whether the "learning goal" has been achieved.
Q: The audience wondered how they study such a wide population.
A: They think about the target audience in terms of "learning groups" (e.g. families, school groups, adults, etc.). Individual exhibits may be more specifically targeted, while exhibitions have a wider target.
Universal Design (UD) Presentation
Anna Lindgren-Streicher, Senior Research and Evaluation Assistant at MOS, talked with us about how the MOS employs universal design.
Using tactile, auditory and visual (text/images) mediums make a learning experience better for everyone, not just those with disabilities. Multimodal components eliminate barriers to learning. They consider input from people with disabilities who are active partners in exhibit development at MOS, and attempt to make sure the environment remains a social experience in which visitors can actively participate. They take into consideration those with learning disabilities, those with hearing and vision problems. They are trying to incorporate more for those who speak English as a second language.
Seniors don't often label themselves as disabled, but often benefit from what is done for the disabled population. This way they don't have to ask for something special. The MOS does purposeful sampling, take information from the extreme ends of disabilities and expect that what they do helps to cover those who have milder forms of the disability as well.
Copy in their designs is generally written at a 6th grade level but it depends on the audience. They have a maximum of 3-4 learning goals per exhibition, and communicate those messages in different ways using different activities.
Q: The audience wondered whether visitors ever suffer from info overload.
A: Since it's self directed learning, many skip over things that don't interest them. But if they try to do everything yes there have been studies of visitor fatigue.
Broken Exhibits Presentation
Elizabeth Kunz, Senior Research and Evaluation Assistant at MOS walked us through a research study they did to determine what exhibits at MOS were truly "broken".
The MOS was getting feedback on a good portion of visitor comment cards (13 %?) that many of their exhibits were broken. They wanted to find out what visitors thought of as "broken", how this matched up with what the maintenance dept thought was broken, and how they could reduce visitor disappointment.
They collected data from 225 visitors, using 5 galleries over 6 months. They used 5 data collection techniques: trends in comment card reports; timing/tracking maps (were the most popular exhibits the ones that were broken?); exit interviews; surveys/focus groups; and maintenance surveys (to compare with visitor comment cards).
Generally, when exhibits had design issues (i.e. they were difficult to use, they had label/content issues, or were confusing), visitors listed them as broken. Maintenance workers listed things like the exhibits being dirty. Although problem exhibits are often removed, some in older galleries caused more problems. They weren't designed using UD principles, had more wear and tear, and had computer issues. Everyone agreed that if the exhibit was nonfunctional, it was broken, but visitors didn't always consider exhibits that were partially nonfunctional as broken. If an exhibit appeared nonfunctional in its resting state, people thought it was broken.
Star Wars Multimedia Tour Design Presentation
Elissa Chin Senior Research and Evaluation Assistant at MOS gave us some insight into how effective the Star Wars Multimedia Tour was.
Visitors would walk around the exhibition and punch the number of the stop into the handheld, which would then give them additional information about it. There was also a capability to bookmark information that would later be emailed to them. The tour had closed captioning, and also an American Sign Language (ASL) version of the tour.
About 10% of the visitors to the Star Wars exhibit used the tour, and of that, about 10% bookmarked information for later viewing. 42% of those who bookmarked later retrieved their bookmarks.
The MOS discovered that there was a disconnect between the social experience the museum tries to encourage and use of the handheld devices. The content should have done better to promote interactivity and not be repetitive with what the visitor was seeing or reading already.
The ASL tour got many positive reactions because overall, non-hearing visitors were less reliant on others for their experience. It made them more comfortable being independent, although they didn't realize at first that such a version of the tour existed (given that the handhelds had headphones attached). There were, however, issues with timing, learning style, and cultural norms. Non-hearing visitors often had to choose whether to look at the exhibit or the handheld, could not see as many lines in the CC as they would have liked, and were not familiar with the keypad used.
Some audience members suggested that the handhelds might in the future include:
- A locator / map of the exhibition
- Surveys about what visitors are currently seeing
- Prompts to do certain things
Wrap Up
Q: How good is the relationship between the evaluation and product design departments?
A: There is healthy conflict built in. Many product designers are artists and don't like being told what they can't do. The evaluation department is positioning themselves and helping them do their job better. It also varies by team: some love having the data and others need convincing.