Using semantic differentials to evaluate visual design preferences

We recently completed a redesign for LoveToKnow, a site that provides advice on everything from beauty tips and pet health to travel recommendations and party-planning. LoveToKnow wanted a clean, clear design that welcomed users while also feeling believable and authoritative. The new designs had to reveal the site’s wealth of content without being overwhelming, encouraging people to spend more time browsing relevant articles, and ultimately, turning to LoveToKnow as a trusted source for help with all kinds of topics.

LoveToKnow wanted to test designs before finalizing them for launch. A/B testing was our first choice, but wasn’t an option for technical reasons. How could we confirm whether users preferred the new look and feel?

Visual design preferences are by definition subjective. However, tools like semantic differentials and Likert scales give people the terms they need to describe subjective opinions by choosing from a range of possible answers between two opposing statements. With this data, researchers can quantify what otherwise seems unquantifiable.

Sample semantic differential scales

Designing the test
LoveToKnow had questions they were hoping testing would help answer. We knew who our audience was: primarily middle- to upper-income women with some college education and moderate internet experience. We then broke the tests into three groups: one that set a baseline by evaluating the current designs before the new ones, and two that only saw the new designs, albeit in different orders in case viewing order affected preference.

We began by setting up a scenario so that test participants would all approach the designs with the same mindset: Imagine you have a puppy, and are wondering how much bigger it will get. You type “is my dog full-grown?” into a search engine, and click through to this page. Beginning with a scenario familiar to the dog owners we recruited immediately made the article page designs relevant, and a more realistic scenario is more likely to yield honest results.

Once users had the right mindset, we asked open-ended questions to gather general impressions of the page and website: what did people think the page offered? What did they think the site offered? What would they click on?

We also provided three semantic differential scales:

• credible / not believable
• inviting / unappealing
• helpful / waste of time

Finally, we provided the same scales for three home page design options to help us choose a winner.

Building the test
Designing the questions, as it happened, was the easy part. The much trickier part was creating a test that would help us get answers through remote testing. We’ve had good experiences with usertesting.com before and believed we could make it work here, too; it was just a matter of linking our questions and designs to guarantee that users would proceed in a straight line instead of accidentally meandering off into the woods.

We settled on a split-screen approach: the questions would be coded in SurveyMonkey and show up in a left-hand frame. In a larger, right-hand frame, we’d display the page designs so that people would have them to refer to as they answered the questions. We also set things up so that clicking anywhere on a design would take people to the next one to review.

Over and over through our dry runs, though, we found that people got off-track immediately. They couldn’t help but click on what looked like a real web page to them, and no amount of written instructions stopped that basic behavior. (A real-life reminder of the usability truism that people often ignore written instructions!)

What did eventually curb the clicks was presenting test participants with a thumbnail of the appropriate page design in the survey, and asking them to confirm they were looking at the right design before answering questions. The colorful image broke up the survey’s wall of text and grabbed enough attention for people to stop, read, and understand.

After that, it was just a matter of watching test videos and analyzing our SurveyMonkey data, which the site helpfully provides as charts and free text.

The final results
Some of what we discovered:

• The new article page design was slightly more likely to encourage exploring the site. While participants found both old and pages credible, design had little to no effect on perception of credibility, and there was scant difference in design appeal.
• However, the new design unquestionably helped people understand that LoveToKnow provided information far beyond just dog content, an important improvement.
• Results for all three home pages were largely positive and equal, but our second design had an edge over the others, particularly in terms of credibility.

Increasing appeal for LoveToKnow

LoveToKnow provides advice on everything from beauty tips and pet health to travel recommendations and party-planning. We helped LoveToKnow provide more consistency and a better user experience by first analyzing their taxonomy as part of a larger search engine marketing project with Nine By Blue. LoveToKnow returned to us to redesign their site to improve its appeal and apply the new taxonomy to an updated, easier-to-use menu system.

As we translated wireframes into visual design, we ran remote usability tests to determine whether our approach resonated with the target audience. LoveToKnow's revenue model is ad-driven, and we paid close attention to this as we wireframed possible design approaches and feature refinements. Test results confirmed that the new designs helped people better understand the breadth of LoveToKnow's content, and that the new article page design better achieved the goal of site exploration.

Work included:

• Taxonomy analysis
• Visual redesign
• Heuristic review
• Usability testing

Remote usability tests used a split screen to display a survey and relevant page designs (top: current design; bottom: new design).

Original home page page design.

New home page design.

New article page and fat navigation.

Focusing on the future – website planning for the International Development Innovation Network

The International Development Innovation Network, led by MIT’s D-Lab, provides training and resources for innovators worldwide to create new processes and products that improve the lives of people in poverty. IDIN works from the ground up by serving local innovators; community members are most familiar with their own situation, and most motivated to improve it.

We helped the IDIN team organize and narrow down its goals for a new site by developing user scenarios with the team and mapping goals to specific audience needs. Through an interview with the program's sponsors at USAID, as well as a paper prototype review session with summit alumni, we gathered audience feedback on valuable features and expected usage, which provided further insights into how to structure the site and design key interactions.

Moving from audience and feature analysis into information architecture, we led a collaborative workshop with IDIN staff to map content and features into page types and a high-level site organization. This groundwork helped prepare IDIN for usability testing and a full site redesign.

Work included:

• stakeholder and user interviews
• user scenarios – based on real people and interactions
• prototype development
• site map
• content inventory and planning
• high-level wireframes

Whiteboard notes from content mapping session.

Left: home page wireframe (mobile view). Right: project page wireframe (desktop view).

Juniper internal app research & concept

Juniper Networks uses multiple internal tools to keep track of sales-related technical documentation. In an effort to consolidate those tools, Juniper approached us to design a proof-of-concept for a new, mobile-friendly application. We interviewed sales engineers as well as the people who manage and support them to understand the different situations and patterns of use. Based on those interviews, we designed multiple approaches for an intuitive, intelligent, and responsive search interface that guided users to the most recent and relevant information.

Work included:

• User research
• UX design

Quickie "Valued Features" Prototype Test

We were recently part of a team redesigning a student services department website for MIT. A lot of ideas for home page features were discussed at the kickoff meeting. The question quickly became, which of these features, if any, would students value?

Since it was the end of the semester, we had limited access to students. The client team suggested that a lot of students would be at an upcoming event, so we designed a test that they could take in a few minutes at a table near the event entrance.

We created a very simple prototype using OmniGraffle that included a simple black and white mockup of a bare home page frame, and several cutouts of proposed paper features and content elements.





For the test, students selected the paper elements that interested them most and placed them on the paper home page frame while telling us why they'd chosen those elements. We photographed each student's selections as documentation and asked two direct questions from our test plan. With two test facilitators each conducting one-on-one tests, we had 15 responses in less than 20 minutes!



The students got the idea of the paper prototype right away and had no problems understanding the test or the interactivity suggested by the paper features. We heard stories of how they imagined using the new site and the proposed features. The test provided simple data for the "what" and the "why," which helped the group make decisions.

One test gem was that about half of the participants interpreted one of the features differently than intended – they saw a pair of small photos as staff member photos rather than student photos. Since they commented about how nice it would be to see a staff member and click on their contact information, we learned they valued a feature that we hadn't anticipated.

After the test, we tallied which features were selected most, reviewed the reasons students gave us for their choices, and created OmniGraffle wireframes showing proposed content and features based on what we learned.

This test could be adapted to be run remotely via screen-sharing software, such as GoToMeeting. Hand over the keyboard and mouse, and participants can place content and feature elements on an onscreen frame. Either way, it was a quick and effective method to determine what users value.

Notes from FutureM Boston: “The Future of the Web User Experience”

On October 6, I attended part of MITX's FutureM program, where Bill Albert, director of Bentley University's Design and Usability Center, and Moira Dorsey, an analyst at Forrester, spoke with Weymouth Design President and CEO Tom Anderson about what the future might hold for web user experience design and content. Here are my notes from the talks.


Moira Dorsey

• New forms of technology start by emulating older forms (like the "horseless carriage"), and then evolve.
• Online experience has yet to find its true form.
• 70% of US consumers prefer "rich" interactions (Forrester statistic).
• Framework for the future: CARS - Customized, Aggregated, Relevant, Social.
• Cool example: Wikitude augmented reality travel guide
• User research is the key to knitting together experiences across channels.
• Customer experience maps are a good tool used to help as well.

Moira's presentation is posted here on the Forrester site.

Bill Albert

• There are a lot of new tools to help collect quantitative and qualitative data – help designers and businesses get a clearer picture.
• New tools facilitate micro-studies – can test at the pixel-level.
• Do users trust your site? Find it attractive? The devil is in the details.
• In the UX industry, Bill sees a shift from usability to persuasion – usability beyond the web to all communications.
• "Usable" is no longer enough.
• Studies on user experiences – example of four ads for Fidelity. The promotion that was the least ad-like (no image, no background color) got the most clicks

Bill's presentation in a .pdf is here

Thoughts and question

These were very solid presentations that complemented each other well. Both talked about how design is no longer about the web – it goes beyond to all channels, and research with users is key to getting the UX across channels right.

At the end, I was thinking about the 70% that Moira mentioned prefer "rich experiences" and the majority of testers Bill mentioned who selected the Fidelity ad that was the most plain, and asked, "How can the difference between these seemingly extreme examples be reconciled?"

Bill and Moira's answer was that the plain ad was central to helping people accomplish a task – a serious one about retirement planning. It goes back to data. An example of "rich" is Google Maps. If the "richness" is too fancy that is a problem. My take away is this: data + rich + simplicity = happy users.

Why we don't design for "Innovation"

We love technology and the things it can help us do. We are excited by the new functionality and resources we see everyday. However, we regularly see and hear from people who wish the technology they already have met their needs better. Thinking back over our most recent research sessions the participants - executives, students and reporters - all want to access information that addresses their needs in a way that's easy to find.

Solving some of these problems is pretty easy, like designing a faceted search, creating a comprehensive structure for information and designing type and imagery that supports the content. Solving others can be tricky politically, like providing the names of appropriate people to contact if needed (creating transparency). A few more solutions can be time consuming, like developing summaries for a database of long documents or enabling personalization. All of these are manageable, however, and provide a significantly improved user experience.

It would be amazing to develop the next big thing, but we see a lot of room for improved user experiences using the tools we and our clients already have. Learning about what matters to people and providing it quickly and manageably never gets old.

Curaspan application user experience, user interface, and visual design

Curaspan Healthgroup, Inc., a leader in web-based healthcare administration software, wanted to expand and improve its product range, including breaking off modules into standalone items and making all their software easier to use.

After observing how hospitals, care facilities, and transportation providers used the existing application, we redesigned the flow of Curaspan's main application to reduce steps, and made the UI easier to scan in busy environments.

We provided:

• Heuristic review
• Contextual inquiry
• HTML prototype
• Visual and UI design
• Style guide and HTML/CSS templates

Find a Patient interface and sample results

Patient details screen

Transport request screen

Using card sorts with storytelling to get off to a great start

Working with non-profits often means large teams with multiple stakeholders, each with a list of desired features and improvements that may have been in the works for years. The expectation level is high – funding has finally been approved, a target launch date planned and everyone knows just what they want – in fact, a site map has been developed. While each stakeholder is trying to address the perceived needs of their audience, when it all gets put together the results can be unprioritized and sprawling. The team may have been meeting for months and are looking for the designers to pick up the ball and run. However, committee-driven IA can be so democratic that it can miss providing the information that site visitors are looking for in the way they expect to find it. Our favorite path to a successful project with multiple stakeholders is holding card sort/participatory design sessions with storytelling as soon as possible.

Traditional card sort sessions involve gathering data from participants by having them sort terms into groups that make the most sense to them. These sessions are a valuable tool for creating navigation that makes sense to the people who will use the site. A large numbers of participants (>30) is needed to provide the data, and software to sort the responses is helpful. While traditional sorts are not difficult to do, we have found that having sessions with fewer participants (5-12) and mixing approaches provides input that covers several areas beyond defining appropriate navigation categories.

When sessions include prompting participants to tell stories about a specific time they used a site, perform a card sort, and use cards to position content and features on a page the way they'd like to see it the data is deep and rich. The results provide a clear picture of each person's mindset, and how that mindset affects how they think about content, categories and what they value. Patterns emerge that point to pretty clear user group needs. While it doesn't provide deep statistical data, it does provide rich scenarios of use and an understanding of needs and values that is just what we need to get buy in from stakeholders to prioritize the feature list and focus the IA on the top issues and most common situations.