Ep.4 Disasters (July 2 2020)

Okay it’s time for sticky notes chat episode — what are we at now — episode 4 yeah welcome to our episode on “Disasters”

Let me let me tell you what this episode is all about, because that was kind of –this isn’t like one of those disaster movies; this is about our experiences having stuff happen to us during research sessions that stopped us from going with our original plans and then how can you recover from those situations.

So I’m here with Cory Lebson and Amanda Stockwell, two other LinkedIn Learning authors who do the same kind of thing that I do on LinkedIn Learning. Also both are user researchers.

And I’ve got a story to start with, which was a disaster I could not recover from. Yes exactly! So we’re gonna talk about what can you do that is maybe not your original plan, but could still be a way of recovering the session. Here’s one that didn’t quite work out that way!

I was running sessions with IT administrators on the Monday, Tuesday, and Wednesday, I think it was of the week that one of the big viruses hit in the late 90s and I can’t remember now whether it was “I Love You” or “Wannacry” I’ve kind of erased this whole thing from my memory. But the first IT administrator comes in, we do the signing of the forms that you have to; the consent forms, we go through that whole spiel you do at the beginning of a session, you know, “This isn’t about testing you, you’re helping us test the product” and all of a sudden his pager (and this is in the days of pagers still) his pager goes off.

I said “You know, I know what your job is, so if you need to get that, you take a look.” So he takes a look at his pager and he says “I have to go!” and I’m like “Oh is there anybody else you can get to help? Do you you have to?” “Okay you have to go? Okay.” And he’s like “No, really, like I’m really sorry this is — this is important.”

So as we walk, as we’re kind of running back out to the front door for me to take him out again he says “There’s a big virus that’s pretty much taken down our entire network, I have to be in the office.” and so that was like “Oh well that’s a shame, well I’ll follow up with him later on and see what that’s all about.”

And then almost immediately the recruiting people in the organization called, like “Um we had a call from somebody who said they can’t make it to your session.” and we don’t even get calls from the other five, which is okay — by that point we knew inside our organization there was a problem and that was just the end of it. That was end of my session.

There’s not much you can do when you don’t have participants for your session that you’d expected to have! What about you two? Anything you can you’ve got that can top that one?

Well so so first off I’ll start by saying for six years I actually focused in in natural disasters and kind of implications on interfaces and things like that but then within that period I was doing a project not related to any of that — it was just completely unrelated — and I was in Boston they said “We’re going to get this snowstorm coming” I said “No problem! Whatever! I’ll do my research, scheduled for two days.”

Day one, half the people didn’t show up they’re getting a little nervous and then then you know it towards the end of day one they were like Oh we’re gonna get I don’t know how many feet it was … it was what they called “Snowmageddon” or or I forget what it was in Boston where they got like you know it was it was…

It was higher than houses wasn’t it?

I gotta tell you it takes a lot of snow to shut down that city!

Yeah, they’re pretty proud of their ability to cope, aren’t they!

So go you know I’ll head to the airport. Look at the airport, like every flights canceled. I’m stuck, I don’t have my second day of research, I only have half first day done I was like okay no problem I’ll take the train like the train start being like I watch them being cancel, cancel, cancel, getting towards mine. finally it’s like there’s a train available it jump on the train the train starts going back to back to Maryland where I am and it’s like I’m sorry the tracks are blocked we need to back up and then it goes back to Boston for a while me we’re taking other tracks whatever…

I think like ten hours later I was I was I was home and very grateful that I had you know had my own natural disaster you know in research that caused a bit of a disaster and you know the research was rescheduled and came back given I think maybe two months or something to actually finish it but uh yeah that probably was the number one for me

Amanda? Okay this is this is our bragging rights time what have you got? what have you got?

Well I don’t have exactly either of those two things And actually I wanted to kind of point out is that there’s like disasters with the studies and bells right like participants can’t show up technology doesn’t work that kind of thing but then there’s also just like personal or life or like other kinds of disasters so um this one particular comes mine actually I think this study turned out okay, but it was also in Boston.

I was meeting Cory and we were staying at a hotel very very very close by where we needed to be but they lost my reservation and it was the same night that I believe the Bruins were in like a final playoff or something there was some sort of big sporting event such that there was literally not an available hotel room in the Boston metropolitan area and I arrived at about 10:00 p.m. the night before because my flight was delayed

Well we’ve got this cardboard box for you

I took a car to a several miles away sketchy Beach motel for one evening. I switched to a slightly less sketch Air B&B the next day and then fortunately the last day that we were there we were able to get into the hotel where I was originally supposed to be which by that time though although when I had booked the room it was like 150 bucks a night they were they wanted to charge like a thousand because it was suddenly very popular

so I I we had three three nights while we were there conducting research and I stayed in three different places which of course is always great your sense of mental composure isn’t it, during that time? Yes, and your ability to focus and all of that kind of thing. So, it was not ideal but we made it through

Okay I think you two have got some pretty awesome stories there yeah at least I didn’t have any kind of personal suffering involved in mine I could just be like well I guess we’re not doing that one, then!

Yeah there’s definitely been quite a lot of those personal suffering you know from birds hitting engines during research you know where there are no other planes an airplane huh we were all not in the plane we were on the ground

But the plane couldn’t take you where you needed to go because it had had a bit of a bird incident

yeah But the positive is you can put pizza through the Helena Airport security line fYI if you ever get stuck you can order pizza

good to know yeah this is a funny little there should probably be some kind of website that set up by user researchers of all the crazy stuff you can do and all the places you get stuck when you can’t do your research y

eah yeah so I mean those I guess we started off with our worst possible moments We do have another one before I forget because this might actually be worse

you’re already already in the lead, come on what have you got?

wait so similar to Cory’s snowstorm reminded me, I was going to Tampa to get on a cruise ship for a client of mine

Stop, stop… “I was going to get on a cruise ship for a client”?? “I was working on a cruise ship”?? So cruise ships aren’t exactly my favorite way of spending my time but I’d imagine….

But you know already it’s like yeah I was researching… So it wasn’t like I was researching the umbrella drinks, I was working for the company that does the payment system for workers on cruise ships so I was

You were behind the scenes

Yeah I was below the sea. But however we had worked it out so that, oh and because and because my client is not the actual cruise line or was not the cruise line they had to coordinate with their contacts on the cruise line to get us on to ship, and said it was very important that you’d be there because cruises, obviously, they just leave when they leave and you can’t really like hop on later

so there was a snowstorm coming. I live in North Carolina where we are not very well equipped for the snow. There was a big storm coming so I was like well why not go ahead and leave the night before and hopefully I’ll I’ll get there on time and I can just spend an extra day in Tampa

so I did that in order to change my flight I had to take what had been a direct flight and get a connection and I made it to Atlanta I got out of North Carolina but then I got stuck in Atlanta, overnight.

It gets better…. in Atlanta is not a bad place to be stuck but yes, still… But it was the situation where I kept thinking my plane would leave so kept thinking kept I kept thinking finally it’s about midnight and they were like yeah no so anyway eventually next morning I make to Tampa and I’m just doing a little bit of prep trying to like wrap my brain around the client work um and I was not working on the booking system at all but I went into the booking system to see like okay exactly where is this dock and what time do I think I need to leave etc

and it turns out that we had been booked on the cruise ship out of New Orleans. Not out of Tampa. Well, I’m in the wrong city…

And for anybody who’s not really familiar with American North American geography that much, I mean if you’re talking about a map of America being this size we’re talking about that much of it pretty much to get across it it’s a good couple of states to the cross and it’s a several hour plane ride away

so it was a good thing that I left the day early because I then from Tampa managed to hop on a plane to New Orleans oh my goodness eventually I got on the cruise ship where I was intended to be and it was actually good that I had checked because I had a colleague joining me who had intended to fly to Tampa and did not get on that flight, got on a different one!

but so I began what is already so again I know it sounds like a fun thing but working on a cruise ship is actually very intense and hours are extremely long so I began my week on a cruise ship with about twenty four hours of awake time, so…

I think that raises the issue of starting research when you’re all discombobulated like you’re like oh yeah and then the first participant walks in saying like hi how are you thank you so much for coming you’re like I’m falling asleep I can’t focus and it’s really it’s a hard it’s a skill for any researchers out there you know who are kind of starting you know be prepared for that like this particularly the in-person research where you just kind of like you know participants walk in you have to have set the tone and the tone is not oh it’s awful it’s awful It’s “Oh Hi! How are you? I’m so glad you’re here” no I’m not I want to go back to sleep

oh yeah yeah yeah it’s it’s funny isn’t it I mean there’s we do everything to try and make our participants as comfortable as they can be even if that means kind of I don’t know drinking five extra cups of coffee just that we can stay awake during the session the the the the mental gymnastics and the physical kind of bends we have to put ourselves through just to make this stuff happen yeah it’s just to make it work for the clients but oh poor us I mean it’s what we chosen to do as a job I guess so yeah so those are I mean you actually, Amanda, you managed to recover from that one you managed to actually get there and that was through planning that’s because you’d actually said I’m gonna give myself an extra day to make this happen and oh I better actually double check all the travel plans.

It’s one reason why I really, really, try to do my own travel plans as much as I can there are some clients who kind of insist on doing it for you because it’s easier for them to do it through their booking system and they get better rates and stuff but wherever I can I try and do my I’m a bit of a control freak like that I prefer to do my own travel booking not sure about you – what about you yes prefer to do my own travel booking

I get all kind of nervous when the client’s like “we’ll book you” and are you going to book me on the cheapest flight that’s like like seven legs and you’re going here yeah but but but also the idea of a recovery day I mean if if I’m going within the US and only going like one two or three time zones you know I’ll just add a lot of buffer time hopefully an overnight just in case but when if I’m going like five or more time zones you know where I need recovery I’m just going to say to the client I’m like hey we need to build in a recovery day and they are like oh well it’s extra budget it doesn’t matter if you want this research to go well there’s got to be a time to sleep in you know it’s not it’s not like I’m charging them to sleep in but but there is a hotel charge you know there’s per diem, things like that so right right yeah

Amanda you book your travel?

Yeah a thousand percent and again I try to do the same thing in terms of leaving myself some some wiggle room I am notorious for trying to pack too many things in a time of that time that we were in [—] I was flying to Belize the next day for vacation so I’m a little bit notorious for that but I think it kind of goes to the one of the reasons I like to book my own travel and stuff is just I like to have a little bit of control or how much buffer time and you know learning which airports I can connect through fairly easily if I’m if I’m traveling somewhere and I think it kind of just goes to a more general sense that what seems like a very gracious recovery from some disaster is actually from like planning well and years of having things not go so well or learning from mistakes or learning from situations haven’t worked out so well and taking time really kind of yes it’s worth it for you to pay for an extra night in a hotel or for this that gets on earlier or whatever because it makes a big difference in the quality of research you’re able to do when you’re there

I think also there’s there’s something about learning how to travel – about packing light not having, only having carry-on if you can and and learning how airlines work like oh yeah my flights been delayed and they’re just trying to keep me at this gate as long as they can as long as they can as long as they can before they actually cancel it so let me right now get on the phone with the central booking place and find another flight even if it’s with another airline because they can do that and they will do that if they have to do that and it definitely helps if you’re doing a lot of travel for work and you end up with some kind of frequent flyer status – so yeah there’s there’s all these things you learn after doing this for several years that really help you work the system to your advantage so when you do find yourself in trouble you’ve got the experience of working out how to get out of trouble again

I think I think that’s true not just with travel but I think that that’s true with every aspect is when you get in trouble and you will get in trouble something will go wrong it’s not you know we started by talking about like disasters that kind of like that’s the end, you know. that was it. A lot of disasters are they start and then you need to stop them and regroup and continue the research no matter how crazy no matter what’s happening no matter how wrong the participants have been recruited you know it doesn’t matter you know whatever happens you regroup and you keep going and you smile for the stakeholder or the clients or whoever it is and you you put on your your calmest demeanor, keep going and then at the end of the day when you’re alone you said oh my god I just survived this this was such a pain

“I need beer” – exactly!

yeah you’ve been talking about sort of travel disasters but I’ve had also plenty of other sorts of disasters of prototypes not working or technology not working or you know and I think it kind of goes back to that all the things now that it looks like I easily recover from it’s probably because at some point it has gone actually wrong and I’ve learned over the years be like yeah really is to back that up from two ways to share a screen and all of that that kind of stuff.

There’s lots of disasters that weren’t, because of preparedness and kind of making sure that you’re equipped for a lot of the ways things can go wrong And because of previous pain to some degree as well right

yeah so I mean Cory already mentioned some of them: participants have not been recruited to what they need to be they’re not the right kind of people then things like the system that you’re supposed to test for some reason isn’t ready or isn’t accessible and then I mean just Amanda like you were saying the stuff that you backing up things communicating during the session having the equipment that you need to run the set and all those kinds of things and I’m thinking through that list and I’m thinking about times where all those things have affected me and how I’ve been able to recover from them in certain ways I mean let’s start with participants the participants the wrong participants

you get there you have you agreed with the client who you’re gonna recruit and what you’re gonna do and then you get there on the client says oh by the way we couldn’t quite get the people you wanted but we got these people instead we couldn’t okay here’s one of my favorites we couldn’t get people who actually work on the floor so we got their managers instead

yeah I guess we ought to explain what the problem is with that first what I mean I think so so I mean what basically I think what we want is that that context of use if we have a product whether it’s a screen based product or a physical product or a mobile product whatever we want people who legitimately can use the product if they can’t use product then it doesn’t make sense to research them using the product

and not it’s not it’s not functional testing because more than you know multiple times they say well all we need is you to generate the script you know do this do that do that anyone can do it so there’s you know but we’ll have kind of sorta legitimate users and they’re they’re walking through the script no no no no exactly the whole point is that we don’t make a script for them to walk through

Our script is do it you try it what do you think and and you know yes so basically basically you know finding those right people to do it then we know if they can do it and and that’s what’s about so the further we deviate from the are they legitimately can they do it the worse our research is going to be yeah and and that even if they can do it if they use it in a different context than what you’re trying to look for

I am I am such a stickler about recruiting the right participants that I made all think about it I’ve talked about it several times it’s it’s kind of a pill on the side because even if you know you’re well I’m making an e-commerce site anybody should be able to shop for it or would be looking for things for that but you will miss out on so much important context of is that what they wanted to do is that what they were looking for is there some particular sort of knowledge of fo you know the products that the group you were looking for had that the general population doesn’t have for that kind of thing

and I it also depends what kind of research you’re doing and kind of what your goals are but I I’m it’s such as a thorn in my side I often will – I don’t know if over index is the word but I will often very much focus with recruiting the right people I think more than some sometimes other people you know but I will cog that by saying that you can almost always learn something from somebody even if they weren’t who you intended to talk to in fact maybe what you can learn is how they’re different than who you intended to talk to you and if there’s a like a difference so I’m not saying they should turn people away they show up to research them they’re not not they ought to be you can definitely learn but it’s I have such a it’s such a thorn in my side so with it

with the managers versus actual workers thing I mean I’ve got two to two ways I typically deal with that and and one of them is we carry on running with the managers but often you’re not really gonna get what you need it because the managers used to do the job now they know what management wants and they’ve got a different set of tracking goals and a different set of ways of thinking about the system they’re probably people who got really good at using the system which is why they got promoted to being a manager a floor manager of the the set of people so their motivations are different and their skill levels are different from the people you wanted. And it might be okay to use them as for instance a benchmark of how quickly somebody can get through the system when they’re really motivated to use it which isn’t necessarily how people are going to be who are using it on a regular basis.

What I’ve done in those situations is actually not run the session in the labs at all but go out on the floor and watch people on the floor and and take people off of their rotation actually put the put what we want to do on a computer to this on the floor so that people don’t have to walk a long way or go a long way to do what what we need to do for instance I’m talking about in for some reason I seem to end up doing a lot of work in call centers and places like that so so for instance if you know if you’ve got call center staff here on rotation just setting up next to where they work pulling somebody out of rotation and getting them to work through with your system instead of their regular system you don’t have to keep them for as long as you regularly would for a whole usability session you might not have the time to do that the opportunity to do that there’s almost always a way around it

and clients don’t, are normally a bit uncomfortable with you doing that but if you go in with a confident enough air about it oh you know that’s not a problem we don’t this isn’t the end of the world let’s just go over here and do this and you’re walking as you already what they’re going carrying your laptop with the system on it look it doesn’t matter just come with me I found that that actually it’s a bit bit cheeky but it can work if you’re confident enough to do it

yeah it so it goes back to that smile yeah I mean if you had the flight from hell to get down you’re like this is the last thing I want to deal with

yeah it might be beer time at that time in the morning or something in your coffee.

I think it’s the ability of, not just in terms of you know like I’m tired and whatever but um and this is hard for new researchers and maybe even sometimes for experienced researchers but being able to adapt to whatever the system is or whatever the people are or whatever happening just because you’ve had a plan and it’s great to have one doesn’t mean you need to stick to it when you get there you know sometimes it’s sometimes it’s subtle like I was doing some usability testing for a marketing website a while back and there was um you know kind of existing users of one product, existing users of another product, then people who are sort of just exploring but it turned out one of the big findings was that the definition between the two products wasn’t all that clear and when people had self-identified as having used wasn’t totally right

and the first you know kind of time happened I was like well I’m just gonna fit in the scratch like none of these questions meant to you and the sort of that that flexibility I’ve been able to read what’s happening and making quick choices about will look and I do right now this particular person in this particular instance whether you’re on the floor of a call center or even just on like a remote call with somebody or you know whatever it is what can you learn and also having kind of reframe that so you can go back to your client and say well we didn’t get exactly what we thought we did…

yeah so I throw in there I think there’s there’s two kinds of participants that that could arrive besides the ones that are actually the right participant. There’s one set of participants that arrive that are not the right ones but you can you can work with them you’re like

and also by the way I like to keep a real-time chat with a client being like okay now I am going to talk about this you know keep keep them informed why am I going off-script well and go with it you know then talk about

the other kind is the kind participant where you’re like it’s not happening it’s just not happening I can think of one story where you know we’re doing a study of of adults older adults without health care for it was really the Affordable Care Act and such when that first came out and a man walks in you know first of all he his caregiver brought him in which was perhaps a tip-off and then it turned out he had very serious dementia and and so I I try I tried everything I tried you know you know reframing working with it nothing was working and finally I’m like you know what you know thank you so much for your participation this has been wonderful was like after ten minutes and I’m like here’s your incentive

bingo! That’s the important thing! They turned up, they get the money!

they definitely get thee incentive no matter what no matter if they scammed you they get the incentive you know no matter what but but anyway

but then so then he looks at me, he’s like I thought you wouldn’t want to talk with me so I brought this book that I had authored 20 years ago because once I was able to like produce things like this and I was hoping that you would that you were you know understand and such but you know so can you look at my book and so I looked at his book it was it was a children’s book whatever

but but it wasn’t it wasn’t going to happen and you have to deal with that too

but the number one thing I would say is you’ve got to set client expectations before and not just client, stakeholder if you’re in a company in house.

we’ve got a question about that are you able to write in the contract with the client the requirements about who needs to be recruited

Yes and no. Exactly! Yes you can that doesn’t mean you’ll get them

Well no. What I’ll also say is yes you can write in the contract I will manage the recruiting it doesn’t mean you have to do the recruiting but you can always subcontract or deal with the in-house recruit or whatever as the researcher, you should be writing a recruiting script like like basically a screener for the recruiter to use whether it’s you or somebody else or even electronic although electronic sometimes more people get in you don’t want but ideally want ideally I’d see you in a manual process

but you have got to take some control of that and then you don’t even know when you write the contract you really don’t or if you’re in house you don’t know when the project starts exactly who you want but that’s what you do as a project you hammer out who you want with the stakeholders and then you you make the the script for the recruiter the screener and you you take control of that

and then that could give you more power to effect change or affect the right people

hmm yeah it’s not necessarily the contract I mean the contract everybody has the right intentions coming into this typically they’re not trying to give you the wrong people it’s just yeah yeah

If people want to know more about recruiting scripts actually — okay course plug time — my course on usability testing has a whole section on recruiting scripts and how to write one that that gives you that walks you through walks your recruiters through the process of identifying whether somebody is the right person for for a session or not

yeah I mean so even with the best intention in the world even if you do get people who are right I had a guy come in it was pretty obvious he didn’t look drunk but he he was acting drunk and after a couple minutes he was like “you know what it probably wasn’t the best idea to come in immediately after having my wisdom teeth out”

He’d literally come from the dentist after having two wisdom teeth extracted I mean he was did I’m not sure they put him out for that but he was definitely not completely with it

I definitely had participants who were high and some ways they’re really good sessions! If you could do it then you could do it any of you could do it any time!

I had a particular participant who was drunk, he told me proudly, shortly before asking me on a date. He was approximately my grandparents age

so I would have to kind of point out that there’s there might be people who are the right fit for your study in that day you know do in fact use the kind of software you’re looking for they do have the right qualities but not every participant is going to be appropriate or respectful or yeah

so there I think that’s something else to look out for it’s something I’m particularly aware of when I’m in research by myself that you have to be prepared for whatever crazy stuff is gonna come someone’s mouth and you need to have some sort of of grace and poise to redirect the best you can but also to Cory’s point to to cut and run if you need to and to end sessions as briefly as possible and safely

yeah it’s it’s something that you have to be aware of and you’re doing pretty much any but particularly in-person research by yourself. It’s a thing.

And I mean in in a lab environment or an office environment there’s other people around. Typically one thing I do not do is site visit research where I’m going into people’s home or ethnographic research on my own if I’m going to be in a private place with somebody else

There’s several reasons for that. One is I want somebody there to to be a witness to what I’m doing, but I also want somebody else there to to be part of, to make it more of a social interaction than potentially a one-on-one interaction where people could misinterpret that.

When I set up research I try always to have at least a team of two you’re just as general best practice I just think it’s I think it’s just all around for many reasons including the back-up plan in case like you know I have a coughing fit even you know something but you know it there’s just so many reasons that if you can have a team of two you know built whether you’re internal to a company or a consultant build that into your budget yeah particularly in-person research.

And it’s not visit you’re right I mean it’s not just for safety or that kind of assurance I often try and take if I’m doing site work I will do my best to try and take somebody from the client with me because it’s a way for them to really experience … I find that when people do that and experience it firsthand it really brings drives home to them how different their users are from them and from what they expected their users to be like and it gives them stories they can take back to the rest of their organization to say no no really we did this thing and we saw this person who did this and it was so different from what we expect is that so got real stories they can use

but like you say Cory having two people from who have who are practitioners is brilliant because you might be caught in Tampa rather than in New Orleans you might be having a coughing fit you might anything could happen and it’s another form of redundancy is the wrong word backup for everything you’re doing plus there’s one person behind the scenes you can manage the client and say look I think you can probably tell why Cory’s going off script right now but it’s okay Cory’s good at this

I mean and it’s I mean just yeah for any number of reasons I think having two people is is good also I tell you having those two people visible because easy enough whether you’re you know on the ground in some random place where you can you could stick them and not you know make them invisible behind the scenes ,stream whatever, I do my best to have those two people me and another person both visible to the participant just it’s you know it just it somehow keeps them in control a little bit better in a way that there’s a backup I mean again the backup plan but it for me that’s that I feel I feel like like that redundancy really helps

I think um we it used to be I mean I remember when I was first ever doing usability we had boxes of equipment we’d take places and we’d set up cameras and we’d set up monitors and we’d have somebody looking at monitors with these big SVHS tape decks that we were recording things on and I can back where we were in the office we’d had one-way mirrors and and big labs that we did all the stuff in and that was almost an impediment to doing what we did

I found out subsequently I mean the technology’s moved on a bunch since then but now we’ve got much more ability to be flexible in what we do and to for instance have like you say Cory both the researchers sitting in the room maybe with somebody sitting in the far corner away from the participant but they’re still in the room doing what they’re doing is we’re not we’re not relying on the being behind the scenes messing with all the stuff that we’ve got and of the technology we’ve got to try and record the session or whatever

yeah I definitely hear you I remember being very stressed by the even we had what was called lab in a box where yeah it was yeah it was like this this metal box and there was I think I had I used the I remember using the Hi-8 version where you had to align and you would adjust the mixer and then you had to press a button and then you had to press the time stamp button at the same time or else the time stamp would be off and it was like and then if you messed up like you’re recording every time I’d be like “Check the recording – did it work this time? Did it not work this time?” oh my god that was it yeah that had its own little sad disasters with it

like yeah yeah and that kind of technology was more aimed at technologists than at people who wanted to run research studies and now I think we’ve got at the point where the technology easy enough for us to not have to think so much about that for running the studies which is amazing also you can carry it in your pocket rather than the carry in a an extra case which meant you had to check luggage for flights

I think it’s also less intimidating for for participants if they see you show up you know yeah I’ve got an electronic pen takes notes for you that feels very different than if maybe you pull up with briefcase of of gear and I think that certainly helps

wait are you using one of those pens that takes notes and then makes them digital later? Yes, but that also qualifies as a disaster because I had very poor luck with the one that I had. If anybody has had a good experience with one of those pens that they particularly like

Paper and pen.

Yeah yeah yeah I liked having I liked writing it down and then it recorded the audio and you can make like you know a little star whatever so that if there was a part of the conversation you particularly wanted to go back and listen to you I quite liked that when it works but it didn’t seem to work all that well

Oh you early adopter you

Yes so okay so we’ve talked about recruiting disasters we’ve kind of begun to touch on technological disasters and I think really those disasters are less disasters these days. If your camera dies you can still run the study unless you’re remote. Remote has its own set of issues that we can deal with in a separate session

But what other sets have we got?

What about when you don’t when the product you’re supposed to be testing was is not available to you for some reason? Either you can’t get into it because nobody gave you the passwords or because it’s down that day or because it was a beta release and all of a sudden the staging platform they put it on somebody wiped it overnight or something like that? All of those have happened to me by the way.

I’ve never had the wipe overnight but but yeah I’ve had I’ve had servers go down in mid test and I’ve lost you know half a day of research you know because the mid one session at you know you get through 30 minutes that’s it and then the rest of the afternoon gets cancelled and what I’d say to that is it’s only ultimately it’s only money and time and it’s and ultimately you know particularly when it’s the client system or the or or the internal system you know maybe it’s a little more money for another day of testing maybe it’s a little more of your time as a researcher but sometimes you you’ve lost a day you’ve lost two days you might in the snowstorm example the day was gone or the whole new travel you know and whole new flight up to Boston at that time but but it’s really just saying you know sigh take a sigh and and you know okay okay it’s I’ll spend a little more money I’ll spend a little more time and I’ll still get it done

Yep as long as you are not flying to Belize for your vacation the next day!

Yeah what the issue is to and sort of what the expectations are from the client or the internal stakeholder it’s it’s kind of in how you manage that also it’s if you have learned something but the second half gets cut off you know maybe you can do a preliminary report and make some recommendations and suggests you know doing longer research later or doing another round of things after a first group of things have been you know kind of worked on or

Or I have had I have showed literally like taking screenshots and showed a a PowerPoint presentation of things instead of a working prototype that right happened that that crashed about 10 minutes before section was supposed to start we had an extra long interview at the beginning of the session to give the team some time to run around and make it work again

So no I think you just you have to be prepared to be flexible and work with what you have and if you really truly have nothing if you’re supposed — usability tests are supposed to be showing people something you know you can still interview them about their needs or their motivations there you can probably maybe you can well you know we have been meaning to run that card sort so maybe we should do that now like you can probably find some way particularly particularly if you’ve recruited a of people who are hard getting contact with I really recommend trying to find some way to get value out of the sessions even if you can’t show what you were intending to or interact with what you are intending to

Because it’s easier to get a little bit of value and then make a case for more than it is just throw in the towel

Yes! Oh, your fault! You suck! Particularly as an outside person like a consultant no even if that’s true it doesn’t do you good to tell that to people

So so in the past what I’ve done is and I have do I have asked participants to run through the set of tasks that we had planned for our system with a competitor system you ever done that? And now obviously there are some clients were a bit antsy about doing that typically there should not be a problem if it’s a publicly available system you are just basically doing competitor testing there shouldn’t be any problem with it

Some clients get a bit worried about whether there’s some kind of issue with industrial espionage or something like that believe it or not you probably have a competitor insights group inside your organization already so you’re ready doing that then but yeah it might be an issue for some organizations but typically it’s not a problem and you can learn a lot

It’s not that you’re trying to copy what they’re doing. You can learn from their mistakes so you can actually get a ton of value from having somebody do your set of tasks with a competitor product. And we’ve actually learned more from that that we might have done for just an iteration of the product that we were that the client was developing themselves

Yeah so you know with respect to you know you’re saying that you’re doing this changing it up you know both of you changing it up I do want to point out I think what we’re talking about is the more formative more like design research

yeah I can imagine someone listening and saying but what about the what about the the the the power of the large data set but

But each participant and this is really every bit of research I do do and you know qualitative exclusively or almost exclusively it you each participant is a special set of data and that’s okay and the other thing is you know iterative studies are awesome where we’re you know we’ve got a set of participants pause do some changes do another iteration but most in my world most of the time it’s like a one-time project do do research get results maybe research a couple months later and to Amanda’s point earlier if you if you’re forced to do this iterative study that’s not a bad thing either even if a dataset of like three people it’s all good and that’s that’s the beauty I think also of you know this kind of early on your design research qualitative kind of very formative approach that we that we do

And to be clear there’s still rigor associated with that if you’re doing if you’re changing it up between participants. There’s actually a methodology for that called RITE. R-I-T-E developed by Dennis Wixon and Michael Medlock that kind of tells you how to go through that and still be — hmm — rigorous in a qualitative way for getting that research done in a proper way

And the same thing with if I was doing a competitor study I’d be applying the same rigor to that as I would be to my regular qual work I’d be still as formal with participants. I would be stepping away from and letting them do the task as much as I would do with normal with my with the clients’ own product so yeah it’s you’re still applying the same rigor to the research regardless of what it is you’re actually testing

Even if like Amanda said it’s PowerPoint screens, paper prototypes, whatever it is it’s the same you’re applying the same rigor to it

So yeah did we miss anything out in terms of disasters and how to recover from them? We’re getting to the end of the time we should be spending online, so is there anything else we should be saying to people? What would your parting thoughts be to somebody about dealing with disasters in your user research, Cory?

Well I would just say and and perhaps there’ll be more about this in a bit um hint be flexible you know be flexible we’ll be prepared to be flexible be prepared for anything because it’s gonna happen you know it’s it’s going to it’s gonna happen that things don’t go exactly as you expect, or not even close to exactly like nowhere near you expect, and you can get good stuff and you can do your job and you can come out with with with you know clients or stakeholders who are still happy with you know even though everything else kind of collapsed around you. So stay strong and and you know you can do it!

Yep. Amanda: Be flexible. What about you?

I very much echo that and I would say to balance being flexible with being rigorous I’m all for rigor but I am also all for pragmatism and you know it obviously depends on what the scenario is and sometimes it’s just a matter of learning is to go and and you know kind of picking up from what disaster and turning it into something

But I’ve gotten to a point where pretty much anything that gets thrown on run at me I can find a way to find some sort of value out of a situation and so …

You know what you’ve done to yourself there, don’t you you know what you’ve just done to yourself that’s the jinx if ever there was one!

Although I would say that encourage people to to balance that you know that sort of rigor with pragmatism and to try to look for value out of whatever situation they’re in even if it feels an utter failure because it was different than what you planned I really recommend people you know come looking for what they can get out of something and whether you’re you know exhausted from flying all over the place or prototype visit where the system is down or things didn’t get recorded or the screen share isn’t working there’s almost always a way to get something good out of a session

By the way we didn’t even talk about like unmoderated testing or like more quantitative stuff. We probably are towards the end of time we could probably have a whole other conversation about about that but even in those circumstances you know I was recently running a survey that the data got very messed up through no fault of ours that the collection wasn’t working properly but you know we still were able to look at what we did have and find valuable insights from it

So I would kind of say on top of being flexible really finding a way to find value no matter what happens

Yeah yeah and I mean so the other thing that I would summarize as well was something we covered was be prepared. If you’re prepared upfront all of the being flexible being rigorous all that stuff becomes much easier because you’ve you thought about what your backup plan is. You thought about what you’re going to do if things go wrong and and you’ve potentially got that extra set of batteries you need for the for the video camera or you’ve got the the extra internet connection or you’ve got your you know your Wi-Fi on your cell phone as your backup or all these other things that you do that that can help you be prepared to deal with the the inevitable disasters that happen when you start running this kind of research

Okay I think we’d better leave it at this at this point. Thank you both Cory and Amanda for being part of this again this week. This was a good one, I enjoyed this and we had a lot of people watching. I hope we get a lot of people watching after the event as well. If you’re interested in if you’re watching this now or later remember you can always find out more about this particular session. We’ll add some links for instance of some of those things we’ve talked about on our website sticky notes dot chat you can always get in touch with us through LinkedIn through twitter at sticky notes chat or through email yes we have all those forms of @communication chat at stickynotes.chat is our email. You’ll find all that information at the stickynotes.chat website.

Oh should we tell them what we’re talking about next week? Yeah! So next week back to our regular timeslot of 10:00 Pacific 1:00 Eastern time and we will be talking about “Change”. There always seems to be some new buzzword in user experience yeah whoo are you are you whatever so and I mean even user experience we never used to call it user experience it used to be like user research or usability at some some institutions around this or associations around this have even changed their names to deal with new buzzwords! Enough on that one… but what really changes? What really changes? Is it all the same as it was 20 years ago or have things actually moved on? So that’s the topic for next week and until then thank you very much for watching we’ll see you then!