We get a ton of hits on our Agile Approach page so I thought I’d take an opportunity to give some more background on our methods and share some of our experiences.  The Limina Team comes from a traditional Waterfall or Rational Unified Process software development background.  Our consulting history and practice methodology is an adaptation of best practices developed in-house by user experience professionals and collaborators who’s expertise run the full spectrum of user interface; research, analysis, strategy, management, interaction modeling, information architecture, design and development.  As a result, we have developed a suite of services that can be applied throughout the full span of the product development lifecycle.

As a practice, our methodology has been flexible enough to add value to every client engagement in our portfolio.  As our team engaged on increasingly more frequent Agile/SCRUM driven teams, a trend which began for us in 2006, we needed to make some adaptations to keep pace with rapid iterations.  The following is a rough breakdown of the adaptations we made by “activity”.

We tool a look at the activities and deliverables we execute in a more verbose and lengthy cycle and dissected each phase to  determine which tasks and deliverables dovetail with the Project, Incremental Release and Sprint cycles of an agile project life cycle. Each level invariably incorporates tasks and deliverables from the traditional Analysis, Definition, Design, Development phases.  Here’s what we came up with:

Product Cycle: Assuming 6-9 months.

Intense Observation and Analysis activity in the all important “Phase Zero” a period of three to six weeks.  In the absence of the waterfall lead time, this is the cycle where UX research seeks to identify the following in rank order:

• Task / Activity Model
• User Role Model
• Business Stakeholder Goals
• User Goals
• Competitive Analysis

Release planning: “Iteration Zero”.

While rough concepts are being established to determine technical frameworks and baseline use cases, the UX team takes 2 weeks to elaborate on the primary storyboards to cover feature definitions in the first iteration.

The At this phase, requirement gaps have been identified, rudimentary user typologies have been identified, development road map has been established based on technical complexity, feasibility, business benefit and user benefit.  Relevant personas for the user stories and features are drafted, usage scenarios are  drilled down, the draft interaction model is established and the associated process flows and wire frames are generated.  This iterative cycle is a lather, rinse,and repeat.

Ideally after 2 weeks iteration zero kicks off user stories, wire frames and mockups and will be available for itteration one development work.

While UX team is cranking out user stories and related assets for iteration two, custom asset creation and spot UI reviews run in parallel in support of iteration one.

This  completes the lather, rinse, repeat cycle.

Meanwhile, persona assets, user stories and related assets are aggregated up for incremental release review.  Any usability or user experience hurdles are triaged and assessed for re-insertion into the iteration plan.  Instructive text, user help documentation are written and evaluated for release.

Benefits and Lessons Learned

As a seasoned UX practitioner, I know the value of getting the requirements right before writing a line of code and my initial reaction to agile development was harsh to say the least.  It’s just a temporary jolt.  Once you get in the swing of rapid iteration and continuous design, you barely miss lengthy requirements gathering and documentation.  The clear benefit is low upfront project spend and near term return on investment.  In traditional models, upfront costs on analysis, strategy, definition and design don’t immediately translate to rapid deployment.  And the upfront cost is significantly higher to account for  end to end specification prior to development.

In an agile team, the analysis, strategy and definition are more light weight and design an development run in parallel.  If high yielding business benefits are addressed in the early release, you will be seeing a return on the investment   earlier than you would have if you staggered the design and development in waterfall fashion.

One major lesson learned for Agile UX in practice:  It is absolutely critical to get one or two iterations ahead of the development team. One slip, and you lose any runway for giving yourself the time  necessary to construct successful solution to meet the needs of your users.

Happy Sprinting UXers

-Jon Fukuda

Years ago, various UX gurus touted the coming of the ubiquitous computer– every device would have a computer built in and they would all talk to one another. Mark Weiser coined the term while at Xerox PARC in 1988 and Alan Kay of Apple called this the “Third Paradigm”. It began with mainframe computers (one compute -> many people), followed by the personal computer (one person -> one computer), bringing us to ubiquitous computing (one person -> many computers). Ubiquitous computing can be considered the opposite of virtual reality, rather than surrounding the person within a computer-generated environment, ubiquitous computing surrounds the individual with computers (in between, lies augmented reality). In other words, computers are made to live in our world and not the other way around. Donald Norman updated the term in 1999 and used the term invisible computer and information appliance to refer to the coming revolution of devices that that would have embedded computer chips and effortlessly communicate with one another.

To some degree that has happened, even if you disregard the multiple computers that most people own (desktop, laptop, netbook) even the simple electronic devices that surround us have considerable computing power. According to futurists, my toaster should be talking with my refrigerator to know what kind of bread I have and how I like it toasted or to notify my online grocer that I only had two slices left (some of the early predictions were pretty bizarre and stretched the definition of “usefulness”). But why did their prediction fail to materialize and we’re all living in remedial houses rather than the smarthouse we were promised? Even though my appliances are much smarter than the ones my parents had there is little chatter between my toaster, coffeemaker, or any other smart appliances in my household. Most of the things I own don’t really need to communicate with one another, nor do they have much to say to Net. In fact, most of devices don’t have much to say about anything. My refrigerator still doesn’t know when I’m almost out of orange juice, though, given Tropicana’s new packaging I wish it could tell me what kind I just bought and why it was not the one I wanted. Is ubiquitous computing just one more failed prediction like fifth generation computing (expert systems) or has ubiquitous computing shifting to something else?

Apple takes an interesting two-pronged approach to ubiquitous computing, instead of everything communicating with everything else, the iPhone says – “talk to me, and I’ll talk to those that need to know” and asks “no need for multiple devices, just give me a simple focused task and I can do that.” The latter approach is familiar to everyone who has ever downloaded an app from the iTunes Store, while the former is somewhat new and is made possible by the introduction of OS 3.0 and APIs that allow for easy communication with other devices. The application approach is akin to a universal Turing machine, and the API model is more like a universal interface. It’s not quite two-pronged, rather applications are developed that allow devices to integrate seamlessly with iPhone – I consider it a different enough model as to warrant it’s unique status. For example, Johnson & Johnson made a bit of splash at the Apple developer conference with their demo of LifeScan a tool that integrates a glucose monitor with the iPhone and provides a range of features from simple visualizations of historical data to food tracking to uploading data to your healthcare provider.

Whether or not one believes the hype surrounding LifeScan (is it vaporware or will it see production?), it’s easy to imagine a range of medical devices (or any category of devices) all communicating through the iPhone, all using the iPhone’s screen and its Samsung S5PC100 processor – one person, one user interface with many devices.

I’m not sure if Apple is going to be the one that finally surrounds us with computers or smart devices, but I think their model puts us in more control than the previous one of smart appliances “deciding” for us – when to order OJ, how to toast my bread, etc. My toaster can simply communicate with my iPhone, and I can set how the bread is toasted (silly ideas never really die, they just move around the ether). I think the future is looking less like Terminator Salvation and maybe more like Star Trek [Note: It is very hard to find good science fiction movies or shows that paint a bright picture].

–kipp

There have been some recent crowd-sourcing business models making their way on the Usability Research and User Experience Design scene.  The crowd source value proposition is, “High Volume Results – Cheap” – with some important variables like: Quality, Usefulness, Relevance, Focus, Strategy, and more.

How do you make the right decision on whether or not to crowd-source UX research for your project/product and where will you get the most yield for your time, money and energy?  Here’s a quick review of Feedback Army and Loop 11 as well as some tips for your back pocket.

FeedbackArmy

What is it?

We first heard of Feedback Army back in January ’09.  This site is almost exactly what you think it might be.

1. You post up a URL and a list of questions/criteria to evaluate against (3-6 recommended)
2. You select the number of responders to your posting (3 tiers – 10 users for $10, 25 users for $23 and 50 users for $40)
3. Make a payment, wait and watch the reviews roll in.

What do you get?

Just what the site claims you get: “Simple, Cheap ‘Usability Testing’ for your Website.”

Depending on your questions and the range of responses you select, you have some variable control on the quality of the responses.  The site allows you to reject responses that are not of the quality you feel is deserving of $1.00 (or less depending on how many you selected).  The site has some tips on usability testing and some guidance on how best to use the service with a nice little endorsement for Steve Krug’s “Don’t Make Me Think“.  For what you pay, you get a fair shake.

As part of my research, I read over comments in the sample reviews, I submitted my own request for review, assessed the responses and I also nosed around some discussion forums where Feedback Army was the topic de jour.

Certainly, this service has it’s benefits (particularly on your bottom line) but there are the typical responses from folks disappointed by their own misguided expectations.  Look, you can’t use a service like this, then complain when you’re not handed a glossy analysis of your user findings broken down by persona & scenario that map 1 to 1 with your research goals.  It just won’t happen.  So when you get shorthand “unintelligent” lol-speak responses you really can’t complain.  Some users may/may not follow your posting to the letter and may spout off whatever comes to mind…  that’s the level of expectation you should have going in.

What you don’t get…

User Demographics & Targeted Personas – you’re dreaming.  The reviewer pool comes from Amazon’s Mechanical Turk – a crowd-sourcing work-in-progress.  While there are advantages here, there is limited control over who is actually doing the work.  The m-turk pool is 70% American… combine that with Feedback Army’s English only UI framework, and you’re limited to US domestic testing.

Quantitative Metrics – You won’t get time to completion and conversion rates,  and industry benchmarks.  If you outfit your test environment with Google Analytics, you can get at some success metrics around goals, popular content, and bounce rates, but with limited specificity on who’s feedback maps to which metrics.

Qualitative Metrics  – You can get if you’re explicit about ratings, but you’ll have to compile your own report if you want the pretty charts.

A Usability Report – this one is all you – if you played your cards right, you can get some decent raw feedback to compile into a report, but this requires a lot of planning.

How to make up the difference:

What are your research goals, what candidate  features/functions to test, what evaluation criteria, etc?  Ideally, you run a series of these to arrive at a more comprehensive view of your product’s usability, and compile the report in the end.  Hiring a consultant or using an internal dedicated resource to own this task will help ensure the value added direction setting and iteration planning for your product post feedback solicitation.

Loop 11

More recently we took a look at Loop 11.  Currently in private beta, Loop 11 is hooking up some usability testing bells and whistles.  I used ‘quotes’ around “usability testing” on my FeedbackArmy review because it’s really just a feedback machine.  Loop 11, however, has scratched the surface on tackling the tough stuff: Targeted Personas, Quantitative Metrics, Industry Benchmarks, and more.

What do will you get?

To be honest, I can’t tell you everything…  Loop 11′s closed beta is by invitation only.  Here’s what the site claims:

Create a user test. This is a lightweight form, but it takes more thought and detail than simply posting a URL.  A 3 step set up walks you through adding test details, tasks & questions and additional test options. The demo suggests you can organize tests into “projects” and save tests as templates.  (nice touch)

Invite test participants. This looks like a nice set of options: Get link to user test: presumably, you can send it out to a predetermined list of users (the ideal scenario), create pop-up invitation for your site: this gives you random users which may or may not be what you’re looking for (less ideal) or purchase from their panel users (needs investigation).  The site claims separation of test participants, making data roll up and drill down more interesting.

Everyone loves dashboards… so why not, a nice dashboard to give you high level data on average page views, avg. time per page, avg. task completion rate and average industry completion rates… That’s right, I said Industry Benchmarks.  Now that’s a rich claim – noting their closed beta partners, they’ve picked Amazon, Ikea, HSBC, Toyota…  these will be your benchmarks folks!  Not a bad competitive pool.  Well done Loop.

Here is a list of metrics you can get in the dashboard:

What you don’t might not get.

You already know that I can’t get a good handle on the truth here based on their current closed beta status.  But here’s a list of assumption you can make based on what they’ve exposed.  I found a posting by Ann Smarty who somehow got into their beta, she posted a light review here.

Validated qualitative metrics – you may get ratings, but you miss out on non-explicit reactions.  The classic,”users will say one thing but do another” is always in effect- you’ll get their feedback, but miss facial expressions, eye tracking, mouse hovering, heat mapping and general behavior surrounding their remarks.

That’s about it, it looks like you get a good set of data collection and analysis features – you still have to set up your test(s) properly.  This means well thought out targeted test goals and participant recruitment.

Online User Testing Service/Tool Limitations:

If you’ve found yourself staring down the barrel of some usability crowd-source projects you’re most likely dealing with tight time-frames and or budgets and you’ve ruled  out a lengthy and potentially costly full blown usability study.  What tips can you learn from user research professionals to make the most of your crowd-sourced efforts and build a design strategy from your study outputs?

1) You can’t meet everyone’s needs.  Take some time to look over the feedback and group them into “UI themes” or “issue categories”.  There will always be outliers – if your study was targeted and you knew the demographic weight of missing the mark on an outlier, then you can factor them in – or, if this outlier hit the exact note that all of the others missed – the note you have been attempting to hit…  then factor them in, but be careful not to upset the balance of maintaining a clear grasp of mass appeal.  You can alway run multiple targeted feedback sessions once you know what your issue categories are. Try your hand at feedback and observation analysis you may find affinity diagramming or mental modeling useful, but don’t forget to segment and simplify your feedback – “verb + noun = atomic task”.

2) User segmentation and personas.  Getting at the psychographics and demographics of your user takes a little extra time & thought and has very real user experience implications.  While no two users on any given system are the same, you can loosely characterize their behavior and relationship to information, objects and tasks into 3-6 types.  ex. Novice, Intermediate, Advanced, Specialist, etc.  The more comprehensive your view of your users going into a study, the more focused your test and test results can be.

3) Reconciliation – User requirements and business requirements don’t always map1:1 to each other, and the technical architecture may or may not support all of the requirements. Map out your requirements into a functionality matrix where you look at all of the system functions and features, making sure that you account for all business and user requirements (using excel helps you stay concise and color coded).  Rank each item by business benefit, user benefit and technical complexity (H/M/L).  Use the matrix to build an iteration plan based on your ranking.

4) Mapping study results to information and interaction design strategy.  You may have a head for this, and if you do, you’ve most likely covered your bases, but it never hurts to get an outside opinion.  Great design is rarely achieved without a great deal of planning.  Knowing where you are, where you’ve come from and where you’re going at all points of development can keep your tests and iteration plans focused and practical.  Understanding how to meet the needs of your users in rapid order with a long range view of feature extensibility will go a long way towards keeping your product on track.

Additional on-line usability testing tools:

Lightweight Usability Checklist

Remote Eye Tracking Service

Concept Feedback

Web Review Community

Remote Task Analysis

Remote Usability Testing

Application Testing

Other (unrelated) Product Crowd-Sourcing Sites:

Graphic Design

Automobile Design (just because it’s cool)

Feedback

Freelancing

Happy testing all.  Remember: “Test early & test often”.  Don’t be afraid to admit you need help, we’re pretty good at what we do.

-Jon Fukuda

Big news! Apple just announced the much anticipated iPhone3G S, along with the new 3.0 platform. As usual, Apple has done a stellar job of hyping the release, providing the 3.0 SDK (Software Development Kit) to their huge following of app developers in advance, encouraging them to take advantage of the new capabilities such as copy/paste, voice dialing, compass positioning, video support with editing, and, of course, improved speed (brought to you by the letter “S”).

New iPhone3G S

These are exciting times for Mac fanatics, and for those who were waiting for the new and improved version to be released before jumping on the iPhone bandwagon. The pricing structure for newbies appears relatively reasonable and assures Apple of many new iPhone customers. However, for existing iPhone customers it’s a bit more confusing, and expensive. This is where the ever present Apple user experience gets interrupted. AT&T, in their ultimate, and very typical wisdom, are essentially punishing their customer base for upgrading to the new iPhone – by charging premiums of two or three hundred dollars for those users who haven’t yet fulfilled their 2-year contract; which by the way is practically impossible seeing as it hasn’t yet been 2 years since the iPhone was released! In other words, just when everyone is getting excited about the new iPhone, loving their experience with the brilliant Apple brand, they get hit in the face with the harsh, predictably bad experience provided by AT&T. Not only are they charging premiums for making the upgrade, but there are doubts about whether the AT&T network will actually support the speed at which the iPhone3G S is programmed to run. And, one of the key features that will be supported across the globe (but not in the U.S. thanks to AT&T) is tethering – a feature many customers have looked forward to since the beginning.

During the preview of the new iPhone there were many app demos that didn’t go as planned, and actually completely flopped, but that doesn’t come anywhere close to turning the fan base away from their beloved brand. But now, thanks to AT&T, the supreme user experience has been interrupted. The question is, given today’s economy, how many current iPhone users will go for the upgrade and sustain the blows from AT&T? I know at least one who is determined to keep up with the Jones’, but how many more will do the same? Unlike AT&T, Apple’s user experience has only improved over the past few years. Let’s see how many users overlook the grey cloud of AT&T to continue their journey with new products from Apple.

- Mimi Knowles

pssst! Limina has embarked on developing our own iPhone app – “Top Secret” for now – but be on the lookout for the official announcement!