Recently, intranets and enterprise systems are being met with s host of new “social web” requirements. How are these new requirements bleeding into the corporate culture? How successfully are these requirements being integrated? What are the challenges, what are the risks and how do you define success?

Our study looks at internal company networks and how they are or are not employing social media as a means of increasing or aiding communication, collaboration, process management and productivity. Our initial responses are giving us a better idea of the importance of social media on the company intranet as well as where issues currently exist that might be preventing companies from making use of the technology.  We’re confident this will be a valuable report.

Our responses are coming from hundreds and potentially thousands of people at all levels of their organizations. We expect that this approach will give us a more accurate representation of the current conditions.

Survey participants will receive a pre-release version of the report when the results are compiled.  Take the survey!

The survey should only take a few minutes to complete. For all questions, there is a “n/a” (not applicable) answer if the question does not apply to you or your company.

About Limina

Our user experience research and design consultancy specializes in user research and complex information design which includes multi‐layered workflows and complex visualizations. We improve user effectiveness, make products easier to learn, operate, and more meaningful in their function.

If you have questions regarding this report and our research program, please contact Mimi Knowels (mknowles at limina-ao dot com).

How to better incorporate customer feedback into your engineering driven product.

Limina is often faced with products that have been in the market for years with little to no professional user centered and UI design methodology applied. It’s typically apparent in the interface before delving into the history of development through issues including but, not limited to the following examples:

• inconsistent UI patterning
• obtuse, or in many cases, non-existent workflows where users are either left to their own devices to develop their own approach to the system or spend time in manuals or training
• random color and graphic treatments with little or no usage rationale
• core features and functions hidden in right click menus with no alternative access
• extensive use of dialogs and workspace changes to complete primary tasks
• random and inconsistent screen layout
• chop-shop iconography, cut and pasted from other applications

Most of these issues can be attributed to an engineering driven culture where usability has not been a core part of the methodology. Sure, the code is clean, the feature works, QA and Unit Tests have passed with flying colors… but is it usable or useful from an end to end experience?

A typical engineering approach to incorporating customer feedback
As a stop-gap to implementing user centered design practices, we’ve heard: “We’re meeting with our customers regularly to hear what they need and we keep them happy by feeding these requirements directly to the engineers to implement.”

First of all, we applaud you for going directly to the users with your product and taking back their requests into the design… but this is a slippery slope.

Although you may know your users and even have some working for you, how well are you capturing their needs? Are you asking the right questions? Are your users able to articulate what they need? You may have voluminous user feedback, support call logs, or error log reports on file, but have you put the user data into a framework that generates an actionable set of UI enhancements?

Equally important; when you have captured their feedback, how do you use it? Without a clear definition or roadmap for incorporating the user feedback, your product is at risk of losing competitive ground. When a company has to spend additional revenue in extensive training sessions, help documentation, call centers and product revisions, the margin of return on investment can take a massive beating.

Here are some activities to help you to incorporate user feedback into your agile development practice:

1) Give the feedback some context – What were users doing or attempting to do when they encountered the issue, and what are their roles? Usability specialists conduct a set of contextual inquiries to interview and observe users as they are performing various tasks in the context of their workplace to determine both system and non-system based activities, documents, and tools that are used to complete their tasks.

2) Quantify and qualify the user data – Where do you see common issues reaching a critical mass? Which are the exceptions and how do you prioritize them? What issues constitute a completely new set of features and possibly a new product?

3) Organize it – Once you have synthesized and prioritized the issues, determine their relationship not only to the system, but also to each other. Which comments are related and which ones are specific to a given task or feature?

The result is a set of researched usability issues that can be organized into enhancements prioritized by issue, prevalence, technical complexity, business, and user benefit.

Such a framework should be employed when embarking on a product definition or enhancement process. It allows parties from marketing, product management, and engineering to uncover the root of software design issues that challenge usability, and ultimately to gain a deeper understanding of their users. Product managers and engineers who gather and process user feedback assist their company by developing the right products and tools for their customers.

This post includes an excerpt from Limina’s white paper “Nine Ways to Improve Software Usability and Increase Market Share“.

Skipping this step leads to an iterative path of organically distorting the original design of the system and frankin-hacking patches and appendages to the product to the point where rebuilding from scratch is easier than overhauling the UI when your users start running to your competitors.

There are a great many development teams using agile methods, SCRUMing it out, getting the features out the door to see what sticks. In many ways, this is a helpful model to beat the first-to-market and innovation clocks, but if the net result is revisiting the feature again and again or playing “pass the trouble ticket” from developer to developer as the feature enhancement is punted to the next iterations for months… something isn’t working.

We are all for engineering driven product teams. In most cases, massive leaps in technical innovation are paved by developer teams and individuals unhindered by business and user requirements. But we’re talking about products in the market that have ROI, user adoption, marketability, competitiveness and other business considerations to name a few.

There’s something to be said for dealing with the cost up-front; for taking the time to build a sustainable product whether in its initial incarnation or when producing the next generation of its kind, because the hidden cost of maintenance, marketing and training can work against you in the long run.

Give us your thoughts and share your experiences with us and our readers!

There have been some recent crowd-sourcing business models making their way on the Usability Research and User Experience Design scene.  The crowd source value proposition is, “High Volume Results – Cheap” – with some important variables like: Quality, Usefulness, Relevance, Focus, Strategy, and more.

How do you make the right decision on whether or not to crowd-source UX research for your project/product and where will you get the most yield for your time, money and energy?  Here’s a quick review of Feedback Army and Loop 11 as well as some tips for your back pocket.

FeedbackArmy

What is it?

We first heard of Feedback Army back in January ’09.  This site is almost exactly what you think it might be.

1. You post up a URL and a list of questions/criteria to evaluate against (3-6 recommended)
2. You select the number of responders to your posting (3 tiers – 10 users for $10, 25 users for $23 and 50 users for $40)
3. Make a payment, wait and watch the reviews roll in.

What do you get?

Just what the site claims you get: “Simple, Cheap ‘Usability Testing’ for your Website.”

Depending on your questions and the range of responses you select, you have some variable control on the quality of the responses.  The site allows you to reject responses that are not of the quality you feel is deserving of $1.00 (or less depending on how many you selected).  The site has some tips on usability testing and some guidance on how best to use the service with a nice little endorsement for Steve Krug’s “Don’t Make Me Think“.  For what you pay, you get a fair shake.

As part of my research, I read over comments in the sample reviews, I submitted my own request for review, assessed the responses and I also nosed around some discussion forums where Feedback Army was the topic de jour.

Certainly, this service has it’s benefits (particularly on your bottom line) but there are the typical responses from folks disappointed by their own misguided expectations.  Look, you can’t use a service like this, then complain when you’re not handed a glossy analysis of your user findings broken down by persona & scenario that map 1 to 1 with your research goals.  It just won’t happen.  So when you get shorthand “unintelligent” lol-speak responses you really can’t complain.  Some users may/may not follow your posting to the letter and may spout off whatever comes to mind…  that’s the level of expectation you should have going in.

What you don’t get…

User Demographics & Targeted Personas – you’re dreaming.  The reviewer pool comes from Amazon’s Mechanical Turk – a crowd-sourcing work-in-progress.  While there are advantages here, there is limited control over who is actually doing the work.  The m-turk pool is 70% American… combine that with Feedback Army’s English only UI framework, and you’re limited to US domestic testing.

Quantitative Metrics – You won’t get time to completion and conversion rates,  and industry benchmarks.  If you outfit your test environment with Google Analytics, you can get at some success metrics around goals, popular content, and bounce rates, but with limited specificity on who’s feedback maps to which metrics.

Qualitative Metrics  – You can get if you’re explicit about ratings, but you’ll have to compile your own report if you want the pretty charts.

A Usability Report – this one is all you – if you played your cards right, you can get some decent raw feedback to compile into a report, but this requires a lot of planning.

How to make up the difference:

What are your research goals, what candidate  features/functions to test, what evaluation criteria, etc?  Ideally, you run a series of these to arrive at a more comprehensive view of your product’s usability, and compile the report in the end.  Hiring a consultant or using an internal dedicated resource to own this task will help ensure the value added direction setting and iteration planning for your product post feedback solicitation.

Loop 11

More recently we took a look at Loop 11.  Currently in private beta, Loop 11 is hooking up some usability testing bells and whistles.  I used ‘quotes’ around “usability testing” on my FeedbackArmy review because it’s really just a feedback machine.  Loop 11, however, has scratched the surface on tackling the tough stuff: Targeted Personas, Quantitative Metrics, Industry Benchmarks, and more.

What do will you get?

To be honest, I can’t tell you everything…  Loop 11′s closed beta is by invitation only.  Here’s what the site claims:

Create a user test. This is a lightweight form, but it takes more thought and detail than simply posting a URL.  A 3 step set up walks you through adding test details, tasks & questions and additional test options. The demo suggests you can organize tests into “projects” and save tests as templates.  (nice touch)

Invite test participants. This looks like a nice set of options: Get link to user test: presumably, you can send it out to a predetermined list of users (the ideal scenario), create pop-up invitation for your site: this gives you random users which may or may not be what you’re looking for (less ideal) or purchase from their panel users (needs investigation).  The site claims separation of test participants, making data roll up and drill down more interesting.

Everyone loves dashboards… so why not, a nice dashboard to give you high level data on average page views, avg. time per page, avg. task completion rate and average industry completion rates… That’s right, I said Industry Benchmarks.  Now that’s a rich claim – noting their closed beta partners, they’ve picked Amazon, Ikea, HSBC, Toyota…  these will be your benchmarks folks!  Not a bad competitive pool.  Well done Loop.

Here is a list of metrics you can get in the dashboard:

What you don’t might not get.

You already know that I can’t get a good handle on the truth here based on their current closed beta status.  But here’s a list of assumption you can make based on what they’ve exposed.  I found a posting by Ann Smarty who somehow got into their beta, she posted a light review here.

Validated qualitative metrics – you may get ratings, but you miss out on non-explicit reactions.  The classic,”users will say one thing but do another” is always in effect- you’ll get their feedback, but miss facial expressions, eye tracking, mouse hovering, heat mapping and general behavior surrounding their remarks.

That’s about it, it looks like you get a good set of data collection and analysis features – you still have to set up your test(s) properly.  This means well thought out targeted test goals and participant recruitment.

Online User Testing Service/Tool Limitations:

If you’ve found yourself staring down the barrel of some usability crowd-source projects you’re most likely dealing with tight time-frames and or budgets and you’ve ruled  out a lengthy and potentially costly full blown usability study.  What tips can you learn from user research professionals to make the most of your crowd-sourced efforts and build a design strategy from your study outputs?

1) You can’t meet everyone’s needs.  Take some time to look over the feedback and group them into “UI themes” or “issue categories”.  There will always be outliers – if your study was targeted and you knew the demographic weight of missing the mark on an outlier, then you can factor them in – or, if this outlier hit the exact note that all of the others missed – the note you have been attempting to hit…  then factor them in, but be careful not to upset the balance of maintaining a clear grasp of mass appeal.  You can alway run multiple targeted feedback sessions once you know what your issue categories are. Try your hand at feedback and observation analysis you may find affinity diagramming or mental modeling useful, but don’t forget to segment and simplify your feedback – “verb + noun = atomic task”.

2) User segmentation and personas.  Getting at the psychographics and demographics of your user takes a little extra time & thought and has very real user experience implications.  While no two users on any given system are the same, you can loosely characterize their behavior and relationship to information, objects and tasks into 3-6 types.  ex. Novice, Intermediate, Advanced, Specialist, etc.  The more comprehensive your view of your users going into a study, the more focused your test and test results can be.

3) Reconciliation – User requirements and business requirements don’t always map1:1 to each other, and the technical architecture may or may not support all of the requirements. Map out your requirements into a functionality matrix where you look at all of the system functions and features, making sure that you account for all business and user requirements (using excel helps you stay concise and color coded).  Rank each item by business benefit, user benefit and technical complexity (H/M/L).  Use the matrix to build an iteration plan based on your ranking.

4) Mapping study results to information and interaction design strategy.  You may have a head for this, and if you do, you’ve most likely covered your bases, but it never hurts to get an outside opinion.  Great design is rarely achieved without a great deal of planning.  Knowing where you are, where you’ve come from and where you’re going at all points of development can keep your tests and iteration plans focused and practical.  Understanding how to meet the needs of your users in rapid order with a long range view of feature extensibility will go a long way towards keeping your product on track.

Additional on-line usability testing tools:

Lightweight Usability Checklist

Remote Eye Tracking Service

Concept Feedback

Web Review Community

Remote Task Analysis

Remote Usability Testing

Application Testing

Other (unrelated) Product Crowd-Sourcing Sites:

Graphic Design

Automobile Design (just because it’s cool)

Feedback

Freelancing

Happy testing all.  Remember: “Test early & test often”.  Don’t be afraid to admit you need help, we’re pretty good at what we do.

-Jon Fukuda

If you’ve gone shopping for orange juice anytime in 2009, you’ve been privy to one of the biggest branding blunders since New Coke. Pepsico released it’s new Tropicana brand to the market in January 2009 and shortly after, the public unleashed a firestorm of criticism over the new design.  On February 23rd, just under 2 months later, Pepsi announced they will be reverting back to the original branding.

While, for Pepsi, this has been an unfortunate loss of marketing spend and brand equity, there are some excellent lessons to take away from the experiment.

Lesson 1) Know thy consumer

At Limina we often tout the benefits of conducting user research and usability tests before, during and after the design phase of any project.  As we now know, this applies not only to pure user interfaces and product design, but to something as seemingly minor as a branding exercise.  Focus groups, as opposed to usability studies tend to remove the product and the user from context of use, and has the tendency to lose out on the rich data collection opportunities users will encounter in a live scenario.  Had the brand design team tested the new packaging in a grocery store and observed product selection behavior  of consumers with the new brand against a control study (of the original brand), they would have learned an incredible amount of qualitative and quantitative data about the impact of the new design.

My own personal observations and experience with the new design at the point of purchase consisted of stalled out consumers standing in front of the tropicana selections, heads tilted slightly  trying to figure out which one they used to buy.  On my first encounter with the new brand, I must have  spent 3 minutes crouched in front of the juice rack looking over the new and drastically reduced information based labeling of:  Some Pulp, No Pulp, Lots of Pulp, etc. etc. before grabbing the carton of choice.  Surely a brand manager would have noted this as indication of “missing the mark” and hit the drawing board for another pass before going into full blown package manufacturing.

Take Away: Test early, test often.  Context Matters

Lesson 2) Detailed  Design vs. Gestalt

The Gestalt notion,  “The whole is other than the sum of it’s parts”, could not have rung more truth in this event.
It’s one thing to look at the design of the new Tropicana carton as it sits on the table and admire some of the aesthetic choices of typography, pantones and graphics.

It’s truly another thing to put all of the variations of over a dozen cartons on the table and ask you to make a decision based on the design.  If you say you identified your preferred  household carton of choice without having to scan the selection more than once…  you’re either lying or you worked on the the brand strategy for this campaign.
Either way, running the same visual scan with the original branding is a far simpler task.  Here’s why:  The original branding had large color blocks as part of their information design system to color code a family of very similar products.  On the original design, the color coding  takes up nearly 1/4 of the visual field.  So, whether you’re a pulp, no pulp, some pulp consumer…  you knew which one was made just for you.  Looking at the new designs, the color coding system fails as it’s reduced to less than 10% of the visual field and while they added some redundant labels in case you missed the color coding, colored text on colored juice graphics don’t quite pop the way white text on color does.

Take Away: Avoid designing in a vacuum, think of how a single change ripples through the whole.

Lesson 3) Usability is for everyone

Most people who know me, but are unfamiliar with the usability field, often begin to glaze over when I start to break down what Limina does.  So I love a great anecdote that’s accessible enough for me to put people in the shoes of a frustrated user and get bobble-head nods back.  The Tropicana story is great because it’s not that they should have never touched the brand…  it’s more that they lacked an understanding of how to modify the brand without negatively impacting the information design system which worked so well.

I’ve even heard our clients say “we have a designer for that we only need you for the usability piece” or “we’d like you to run a usability study and give us the results, we can take it from there.”  It’s really not enough to see the poor ratings on a score card run with it or to take a wireframe and go nuts on the visual design.  It’s critical to see usability and information design as a holistic end-to-end process, any loss in translation of user requirements not only results in sunk cost on design/development/manufacturing, but potentially failure to take flight.  Tropicana is fortunate enough to have enough capital to absorb the blow and correct their course.

Take Away: If you’re not sure if usability applies to your product or software…  ask someone who knows something about it.

Lesson 4) Leveraging social tools

It’s no secret…  social networking tools are extremely effective in marketing feedback.  Great product managers have had their ears to the ground in social network response mechanisms for some time.  Good examples of consumer feedback mechanisms are Dell’s ideastorm.com and Starbucks’ mystarbucksidea.com, but more recently we’re seeing the strength of  Twitter and Facbook as focus group and user feedback mechanisms.  While you’re not going to get fine granularity of complex usability issues resolved in these forums, you can definitely gain perspective on broad stroke issues that ail your product.

Take Away: Word of Mouth marketing is a two way street, sure, you can promote yourself all over the social web, but take time to listen to what they’re saying in response.

Lesson 5) If it ain’t broke, don’t fix it

Needs no explanation.  Taking a page from Sony/Aiwa, Nissan/Infinity, Toyota/Lexux/Scion – if you want to test something new, create a test market or spin off a sub-brand.

I’ve heard rumors that this was a stunt to get consumer buzz.  I’m not sure I’m buying that, afterall..  they didn’t change their juice formula,  just the box design.  Whatever the case, this has been a great example of how not to mess with a good thing.

-Jon