These days we’re all under pressure to produce new software, new features and new interface improvements quickly. And the speed demanded by a market of disruptors and startups is ever-increasing.
Within this context, techniques such as agile and lean startup can help immensely to identify critical issues, bring people together in constructive forms and ensure a focus on delivery of software. However, in the rush to ideate, build an MVP and launch, we can still sometimes forget to validate assumptions and fail to incorporate the right kind of user input through selected contextual research. When this happens, sometimes the results can be frustrating; other times they can be disastrous.
Two recent instances have highlighted this. One is well known: Microsoft’s now-infamous Tay AI bot fiasco. The other is virtually unknown but personally frustrating to me: the recent relaunch of the public website for my son’s school Trinity Grammar.
Case Study 1: Tay
The other day Microsoft garnered world-wide headlines for all the wrong reasons when one of its public innovation exercises went horribly wrong.
Tay is an artificial intelligence program designed to communicate like a 19-year-old woman via social media (think Twitter).
And by communicate, I mean communicate: Tay is designed to directly respond to other Twitter users, answering questions and making comments. She is also designed to learn from her interactions.
Well, it didn’t take users long to figure out how to game the system: within 12-18 hours or so, Tay was spouting sexist, racist, anti-Semitic and other similar comments, and within 24 hours Microsoft decided she need to go to sleep for a while.
What went wrong here?
has explained that the Microsoft team conducted ‘extensive user studies with diverse user groups’ but that in this instance they failed to pick up a vulnerability that was exploited.
Others have remarked that Microsoft ‘should have known better than to let Tay loose on the raw uncensored torrent of what Twitter could direct her way‘.
I think the broader implication concerns the question of user testing, and specifically the selection of users for contextual research. Evidently Microsoft failed to pick up the potential for Tay to be attacked in this way: what might they have learnt had they focused on this issue with the right kind of user testing?
Case Study 2: Trinity Grammar
Some months back, my son’s school, Trinity Grammar, relaunched its public website. In many ways it’s a big improvement on the old site which hadn’t been updated for years. The new site is gorgeous and image-rich: it’s also mobile-responsive.
The new site’s target audience is new parents, and its singular focus is on engagement and acquisition of new customers (in many ways its a big ‘sales’ website). Existing customers (ie parents like me) are left with the old site. It’s been retained as a ‘Community’ site and given a new URL. Which is handy: despite its design limitations. it’s full of useful information, forms and contact details; just the kind of stuff a parent needs.
So ‘what’s wrong with this?’ you might ask. Well nothing too much… if only it were easy to get to the old site. However, it’s not. There are two problems.
The first is that there is very limited access to the old site from the new: the only way is via an obscure link at the bottom of the menu under ‘Contact Us’ via a link called ‘Current Families’. It took me a while to figure that one out.
You’d think there would be a nice prominent button on the home page… but no luck…
However it gets worse.
The other day I was searching for information about weekend sports fixtures on my phone. I knew I could find it via the old Community Site, which I recalled, was accessible via one of the menu options.
But do you think I could find it? It was nowhere to be seen…
Not under ‘Contact Us’ (I was sure it was there!).
Not under ‘Sport at Trinity’ (all I could find was ‘sales’ copy).
Then my finger slipped.
Instead of tapping on the menu link, I accidentally tapped on one of the right-hand chevrons.
And instead of being taken to the same sales page as when I tapped on the menu text, I was presented with a sub-menu!
All the secondary menu options (including the link to the Community site) are hidden under the chevrons. I think I screamed in frustration. Fortunately there was no sport that weekend.
So, what went wrong here? Three obvious things:
- firstly, the lack of a simple, obvious link to the old ‘Community’ site from the new home page
- secondly, the bizarre use of a right-hand-pointing disclosure indicator chevron to reveal a sub menu.
- thirdly, the way in which two actions are triggered from the one menu item: this runs directly counter to known interaction patterns and is not the expected behviour for this type of UI pattern. It can disorientate users, adds unnecessary cognitive load and cause frustrations (as I know) as it forces users to learn new patterns that need to be remembered just for this one instance
However the main issue here, I would suggest is user testing. I’m sure the new site is much more successful than the old site in engaging with potential new customers and attracting them to apply.
However I’m equally sure that the majority of users of Trinity’s online content/functionality are existing customers: parents like me. And if most of them are like me then they can’t be bothered remembering or saving a special URL for the Community site; instead they will expect to be able to access it easily from the main home page. Which you can’t. Unless you open up the menu and look down the bottom. And heaven help you if you try to find it on mobile.
(BTW, you can’t even find it through Google: search for ‘Trinity Community’ and you find something entirely different.)
Why testing with the right type of user is so critical
Each of these case studies shares a common theme: testing with the right type of user to validate assumptions and incorporating the right kind of user input.
In Microsoft’s case, the issue concerned how best to secure input from users likely to want to game the Tay AI algorithm. A hard job no doubt; and potentially not possible. However, when planning any project, it serves to be able to step back from it and gain as much perspective as possible, bring in people who are not part of the project to test it out or to at least sense check the scenarios and processes. Then after testing, considering it was going public, to plan for a small release and monitor how it performed with a controlled level of human interaction. With a project so high-profile and with the potential PR downside so great, it’s a salient case in point.
In the case of Trinity Grammar, I don’t know what contextual research or final user testing was done, but it surely can’t have been exhaustive and it surely can’t have can’t have involved too many existing parents. Had it done so then my frustration would, at least, have been placated.
Anthony Voigt 