Using semantic differentials to evaluate visual design preferences

We recently completed a redesign for LoveToKnow, a site that provides advice on everything from beauty tips and pet health to travel recommendations and party-planning. LoveToKnow wanted a clean, clear design that welcomed users while also feeling believable and authoritative. The new designs had to reveal the site’s wealth of content without being overwhelming, encouraging people to spend more time browsing relevant articles, and ultimately, turning to LoveToKnow as a trusted source for help with all kinds of topics.

LoveToKnow wanted to test designs before finalizing them for launch. A/B testing was our first choice, but wasn’t an option for technical reasons. How could we confirm whether users preferred the new look and feel?

Visual design preferences are by definition subjective. However, tools like semantic differentials and Likert scales give people the terms they need to describe subjective opinions by choosing from a range of possible answers between two opposing statements. With this data, researchers can quantify what otherwise seems unquantifiable.

Sample semantic differential scales

Designing the test
LoveToKnow had questions they were hoping testing would help answer. We knew who our audience was: primarily middle- to upper-income women with some college education and moderate internet experience. We then broke the tests into three groups: one that set a baseline by evaluating the current designs before the new ones, and two that only saw the new designs, albeit in different orders in case viewing order affected preference.

We began by setting up a scenario so that test participants would all approach the designs with the same mindset: Imagine you have a puppy, and are wondering how much bigger it will get. You type “is my dog full-grown?” into a search engine, and click through to this page. Beginning with a scenario familiar to the dog owners we recruited immediately made the article page designs relevant, and a more realistic scenario is more likely to yield honest results.

Once users had the right mindset, we asked open-ended questions to gather general impressions of the page and website: what did people think the page offered? What did they think the site offered? What would they click on?

We also provided three semantic differential scales:

• credible / not believable
• inviting / unappealing
• helpful / waste of time

Finally, we provided the same scales for three home page design options to help us choose a winner.

Building the test
Designing the questions, as it happened, was the easy part. The much trickier part was creating a test that would help us get answers through remote testing. We’ve had good experiences with usertesting.com before and believed we could make it work here, too; it was just a matter of linking our questions and designs to guarantee that users would proceed in a straight line instead of accidentally meandering off into the woods.

We settled on a split-screen approach: the questions would be coded in SurveyMonkey and show up in a left-hand frame. In a larger, right-hand frame, we’d display the page designs so that people would have them to refer to as they answered the questions. We also set things up so that clicking anywhere on a design would take people to the next one to review.

Over and over through our dry runs, though, we found that people got off-track immediately. They couldn’t help but click on what looked like a real web page to them, and no amount of written instructions stopped that basic behavior. (A real-life reminder of the usability truism that people often ignore written instructions!)

What did eventually curb the clicks was presenting test participants with a thumbnail of the appropriate page design in the survey, and asking them to confirm they were looking at the right design before answering questions. The colorful image broke up the survey’s wall of text and grabbed enough attention for people to stop, read, and understand.

After that, it was just a matter of watching test videos and analyzing our SurveyMonkey data, which the site helpfully provides as charts and free text.

The final results
Some of what we discovered:

• The new article page design was slightly more likely to encourage exploring the site. While participants found both old and pages credible, design had little to no effect on perception of credibility, and there was scant difference in design appeal.
• However, the new design unquestionably helped people understand that LoveToKnow provided information far beyond just dog content, an important improvement.
• Results for all three home pages were largely positive and equal, but our second design had an edge over the others, particularly in terms of credibility.

Big data UX, UI and IA

For the past three and a half years, Nimble Partners has worked with a big data company to design two suites of web and mobile-friendly applications investigating multiple aspects of the labor market, including demand, supply, job posting, job search, and career exploration. The applications serve a wide user base: job seekers of all ages and backgrounds, the state and local government officials assisting them, public- and private-sector employees researching labor market opportunities, and others.
We worked directly with customers, product management, and developers to redesign some applications, while building others entirely from scratch. Our work included both the customer-facing user experience as well as the tools administrators rely on daily to help job seekers and manage the applications. Customers were delighted with the results: one said we’d “nailed Jell-O to a tree.”

We provided:

• User experience conceptualization and design
• Information architecture
• User interface design
• Visual design
• User testing

We can't show this work publicly. Contact us for access to samples via our demo site.

Product strategy vs. UX strategy – thoughts from UXPA Boston 2014

Note: this post was updated June 10, 2014. I read Jeff Gothelf's post "There is no such thing as UX strategy" after seeing tweets about it a while ago, and didn't think too much about it. Recently, however, I've been thinking about how (and if) UX strategy manifests in UI design decisions, so I decided to attend the UXPA Boston 2014 conference session "UX Strategy exists, but what is it?" with this in mind.

Gothelf's post, which claims "[T]here is no such thing as UX strategy. There is only product strategy," inspired the session, moderated by Diana DeMarco Brown. It was great to hear stories from panelists Sarah Bloomer, Eva Kaniasty, and Lori Landesman. During Q&A, one attendee said he agreed with Gothelf, but unfortunately, there was not much time left to explore the statement. With a night to sleep on it, I realized that statement was crucial to helping me define my own thoughts on the topic.

To me, UX strategy is the goals and plan established for helping and delighting end-users, as well as meeting the needs of stakeholders, and in the case of enterprise applications, purchasers. Stakeholders and purchasers are types of users. My bias as a designer is to go the extra mile for end users – the people who use what I design. In the case of a consumer app, the user and the purchaser are the same, so in these situations I agree that product and UX strategy are the same. In the case of enterprise applications, user and purchaser goals and needs are often vastly different. Like so many answers to UX design questions, the answer to the question of whether UX strategy is different from product strategy is, "it depends." It depends on who your users are, who the organization is, and what the goals for the product are.

Focusing on the purchaser has led to dreadful and compromised enterprise application experiences. I've worked on enterprise applications for ATG (now Oracle), Endeca, (also now Oracle), and Curaspan, to name a few. In those cases, UX was part of product strategy, both functionally and conceptually. That in itself was not a problem. However, while stakeholders and product managers believed in UX, when it came down to coding and shipping, their focus was on meeting the needs of buyers. This makes sense given how enterprise product management success is measured – in sales, obviously. Like an oldest child in a family, enterprise product strategy takes up a disproportionate amount of stakeholders' time and attention. UX is the younger sibling, wanted and adored, but left to its own devices due to the focus on the oldest child. With intermittent support, UX for enterprise applications often lacks the resources it needs to sustain it and help it thrive.

Follow up note - June 10, 2014: With more thought and reading, I want to add that I think enterprise product strategy can be the same as UX strategy, it is just that it often isn't. If/when enterprise product strategy focuses on providing value to purchasers by developing software that is designed to succeed by helping the people who use the applications do so effectively, efficiently, and with appropriate delight, product strategy is aligned with and subsumes UX strategy.

This brings up more questions: are stakeholder and purchaser needs for enterprise applications really so different from end users, and if so, why? Also, if business stakeholders and product purchasers are types of users (as I consider them to be), then is the discussion of UX vs. product strategy really about end user-focused strategy vs. stakeholder and purchaser-focused strategy specifically in the realm of enterprise applications?

Speaking the Language of Meta-Principles: Consistency, Hierarchy, and Personality

I recently returned from beautiful Portland, Oregon, where I spoke as part of the Computer-Human Interaction Forum of Oregon (CHIFOO) annual lecture series. The 2014 series focuses on storytelling, which can take many forms in user experience, user interface, and visual design.

My talk, "Speaking the Language of Meta-Principles: Consistency, Hierarchy, and Personality," covered how to use the meta-principles described in Visual Usability to effectively "tell a story" by using good design to guide users through their desired pathways and tasks in an application. We think of these meta-principles as if they’re part of a language. Consistency and hierarchy are the grammar people learn while using an application: the basic elements that define how a language is spoken. The “words” we speak – that is, the visual design characteristics we choose to convey a message – create an application’s personality. These principles are so fundamental to creating successful interfaces that we call them “meta-principles.” While technology that affects interfaces changes, the underlying meta-principles hold true.

I'd like to thank Emily Mahood Bowman, Clodine Mallinckrodt, Fellene Gaylord, and everyone else at CHIFOO for bringing me to Portland and giving me such a warm welcome. I'll be happy to return anytime!

Speaking the Language of Meta-Principles: Consistency, Hierarchy, and Personality from Nimble Partners

Redesigning Northeastern University's College of Professional Studies B.S. in Graphic Design Degree

Founded to advance the careers of working students, Northeastern University's College of Professional Studies (CPS) grants doctoral, master's and undergraduate degrees. I've enjoyed teaching Interactive Information Design 2 and advising CPS graduate digital media students for the past few years.

I love systems and design, and am anxious to help higher ed better prepare students for the complex design work that we do today. I was excited to be asked to participate in a review of CPS' BS in Graphic Design degree last summer, followed by a redesign effort in the fall. The review was required for accreditation purposes, and the redesign was long overdue.

While the accreditation process was written in academic language, much of the work involved what I think of as design thinking. I (re)defined a clear mission for the program based on graphic design skills that are needed today, outlined desired outcomes for students and a system for measuring results, reviewed current courses in a structured way for strengths and weaknesses against goals, and analyzed student performance data over time to understand trends and gaps. Once the required accreditation materials were complete, it was great to take all I had learned and defined and work on redesigning the degree program overall.

The result of the effort is a roadmap for a revised BS in Graphic Design degree program. There are six new courses and six revised courses, which will begin to be phased in this fall. The new program maintains its practical focus and provides a stronger foundation in the traditional "roots" of graphic design, such as typography. When in place, courses and course projects will help prepare students by beginning in black and white and two-dimensional design principles, and carry through to using visual design principles to help people understand, enjoy and use visual information in static and interactive applications.

Increasing appeal for LoveToKnow

LoveToKnow provides advice on everything from beauty tips and pet health to travel recommendations and party-planning. We helped LoveToKnow provide more consistency and a better user experience by first analyzing their taxonomy as part of a larger search engine marketing project with Nine By Blue. LoveToKnow returned to us to redesign their site to improve its appeal and apply the new taxonomy to an updated, easier-to-use menu system.

As we translated wireframes into visual design, we ran remote usability tests to determine whether our approach resonated with the target audience. LoveToKnow's revenue model is ad-driven, and we paid close attention to this as we wireframed possible design approaches and feature refinements. Test results confirmed that the new designs helped people better understand the breadth of LoveToKnow's content, and that the new article page design better achieved the goal of site exploration.

Work included:

• Taxonomy analysis
• Visual redesign
• Heuristic review
• Usability testing

Remote usability tests used a split screen to display a survey and relevant page designs (top: current design; bottom: new design).

Original home page page design.

New home page design.

New article page and fat navigation.

Focusing on the future – website planning for the International Development Innovation Network

The International Development Innovation Network, led by MIT’s D-Lab, provides training and resources for innovators worldwide to create new processes and products that improve the lives of people in poverty. IDIN works from the ground up by serving local innovators; community members are most familiar with their own situation, and most motivated to improve it.

We helped the IDIN team organize and narrow down its goals for a new site by developing user scenarios with the team and mapping goals to specific audience needs. Through an interview with the program's sponsors at USAID, as well as a paper prototype review session with summit alumni, we gathered audience feedback on valuable features and expected usage, which provided further insights into how to structure the site and design key interactions.

Moving from audience and feature analysis into information architecture, we led a collaborative workshop with IDIN staff to map content and features into page types and a high-level site organization. This groundwork helped prepare IDIN for usability testing and a full site redesign.

Work included:

• stakeholder and user interviews
• user scenarios – based on real people and interactions
• prototype development
• site map
• content inventory and planning
• high-level wireframes

Whiteboard notes from content mapping session.

Left: home page wireframe (mobile view). Right: project page wireframe (desktop view).