The recent past has seen everyone adjusting to the ‘new normal’ of these challenging times. For UX research these adjustments have included adapting research methodologies to fit our new circumstances. Where once in-person qualitative research would have been a go-to, it is now often not feasible. At User Insight we’ve been drawing upon over 18 years of experience conducting remote research to adapt to this new, socially-distanced research paradigm. Remote user testing has always been an important tool for conducting UX research, but with the current circumstances around COVID-19, it has become even more relevant. Testing remotely has unique challenges but also has unique advantages.
Prep for Success
Ensure participants know how to use remote technology
Prior planning is always important in research, and when it comes to remote qualitative studies there are some specific steps that can help set you up for success. Recruiting and screening are always important, but for remote testing it can be helpful to add additional criteria you might not always include. Some indication of a respondent’s familiarity with the video conferencing platform can be helpful in anticipating how much help they might need prior to an interview. Additionally, be sure to screen for any hardware such as a specific platform or webcam that will be needed (and that they work!). If it makes sense for the research, limiting/standardizing what technology respondents are using can have the added benefit of making issues easier to troubleshoot if problems do occur.
Hire recruiters experienced in remote research
If circumstances around COVID have impacted your normal recruitment channels, another option to consider is outsourcing recruitment. At User Insight we are fortunate enough to have an in-house recruiting team and recently we’ve had several clients reach out for help with recruitment for their internal teams’ research, so I know this is an option folks are utilizing. Established recruiting channels can help you more easily stand up a remote research study if your existing methods have been impacted by COVID.
Conduct a prep meeting before the actual research session
When conducting in-person research, we often don’t meet with respondents prior to their session, but having a preparatory meeting with respondents can help things run smoothly during the actual research session. This early meeting should be focused on handling any technical or planning details, including ensuring they have any technology they need and that it works. While ideally this takes place as a separate session ahead of the main research session to allow time to fix problems or replace the respondent if needed, it could even be just some additional time at the start of a session if having a prior check-in isn’t practical. It can also be helpful at these check-ins to let respondents know generally what they’ll be doing the day of the research session, including specific expectations such as if they need to be visible on video chat (and if so be sure to test that capability). As an added benefit, respondents are less likely to no show for the research session after they’ve already put in the time and effort for the tech call.
Having an initial meeting proved very helpful during a recent project where a client wanted to test a new prototype of one of their apps. While in-person we could just install the prototype on one of our test phones, conducting the research remotely required that the respondents have the app installed on their phone. Respondents were screened and recruited for what phone they used in order to make sure it was compatible with the prototype. During the initial prep session with the respondents, we were able to walk them through the installation for the prototype, which required changing the permissions on their phone’s OS to allow the prototype to run. We also were able to walk them through the installation of the app we used to video conference, including how to enable screen sharing and test/confirm it worked. Getting all of this accomplished ahead of time allowed us to avoid any technical hiccups during the research session and allow the respondents to spend as much time as possible experiencing the prototype.
Capturing insights from a remote session
After all this effort to plan for success, you want to be sure you and your stakeholders get the data they need from the research. Plan from the beginning to have multiple methods of data capture whenever possible. With remote testing screen recording is relatively easy to do, being a native capability of most remote meeting software. It’s good practice to archive these sessions, and at User Insight these recordings are part of the typical deliverables for any project. Additionally, we capture data using mind-mapping software to take contemporaneous notes, providing a second method of data capture but also allowing us to begin to see patterns emerge as data is still being collected and reduce post-testing processing. We like to use Xmind for mind mapping because it has a free version, which makes it easier for clients to download and view, but there are lots of alternatives that work well. Mind mapping is also a nice notetaking format because it’s easy to share with teams between sessions during testing if there are questions or points of discussion during downtime. In terms of additional data capture, teams observing research in person often like to use Post-It notes or whiteboarding to organize feedback and ideas. In a remote testing paradigm, this can be replicated using a shared virtual whiteboard like Mural to allow for simultaneous communal feedback to be gathered and organized in much the same way. Mural is nice because it’s easy to use and has a good selection of premade templates available so there’s little work to get things set up.
Addressing issues during a remote test
Even with the best preparation, Murphy’s Law still holds and it’s likely you will run into the occasional issue during your remote research. Document problems as they arise so you can find a fix for them and avoid them or be better prepared to troubleshoot them on the fly in the future. Sharing this information across research teams within your organization can help you improve faster collectively as you do more remote research. Also, with Murphy’s Law in mind, build extra time into sessions to allow for potential complications to allow for unexpected hurdles. Even with the best planning, there will be the occasional bump in the road. The extra time can always be returned to the respondent or used to spend additional time on points of interest if things are going well. As much as you might prepare, interruptions will happen. We’ve all been on a call during these COVID-19 times when someone’s child or pet decides to make an unplanned appearance. Plan as best you can, and then just roll with the unexpected when it happens. Lastly, don’t forget to ask respondents to mute notifications and messages when they’re sharing their devices. In one recent remote study where respondents were screen sharing their phones, a respondent received an unexpected text message from their partner that was clearly not meant for sharing. Fortunately, everyone had a good chuckle, but definitely, lesson learned.
Technology needs and limitations
Unlike lab-based research, we are limited to what respondents have on hand, as well as their ability to use that technology. Consider your approach from the perspective of the respondent, looking for what will require the least of them both in terms of technology on hand as well as execution. This is where things like walking them through an installation instead of simply sending them a set of instructions can really be helpful. When designing the study, try to keep the tech requirements minimal for respondents and select technology respondents are likely to have.
Another approach we often leverage to streamline technology requirements, as well as retain more control, is providing the technology on the researcher’s side and giving the respondent control remotely. In one recent study, we used the remote desktop feature, just like the one from azure rds, of our conferencing software to allow respondents to control the program they needed to interact with remotely. This allowed us to install the program on our machine, ensure everything worked, and then set things up so that we could just share our screen and hand over mouse and keyboard control to the respondent when it was time for them to complete their tasks. Having more of the technological elements located on our end also means that it’s easier for us to troubleshoot problems or reset things if needed. If you are going to share a device with a respondent in this way, be sure to prepare that device ahead of time so that there are no interruptions from other programs on the machine or potential security or privacy concerns in handing over control of the machine to the respondent.
When it comes to remote conferencing software, we’ve most frequently been using Zoom because it’s relatively simple to use, and ticks most of the boxes for important functionality like screen sharing, video chat, direct messaging, cloud recording, and remote desktop access. It also has some nice to have features like the waiting room, which allows a respondent to join a meeting but be isolated where they can’t hear or see what’s going on in the meeting until the researcher is ready. Additionally, because of how prevalent Zoom has become during the pandemic, many respondents are now already familiar with how to use it and may already have it installed. Zoom has had some recent security concerns, so for some clients, we’ve opted to use their internal conferencing software. We’ve had success using most of the major options out there (GoToMeeting, RingCentral, Microsoft Teams, Google Meet, etc.). Each has its advantages and limitations (e.g. Google Meet only works with certain browsers), so just be sure you’re very familiar with how to run a meeting in whatever platform you choose. One added advantage of using an organization’s existing video conferencing software is that it makes joining the remote research sessions easy for other stakeholders.
Pandemic-specific observations
All of the usual benefits of remote testing remain unchanged; relatively low cost, ease of recruiting across geographic divides, time efficiency for respondents, and potential increased ecological validity for respondents performing tasks on their own devices in their own home. There are some additional benefits to remote testing in our current pandemic situation. Respondents are more often familiar with software that is used for remote testing because so many people are using videoconferencing tools to keep in touch with friends and family. They are also more likely to have already solved hardware problems related to using these tools, including have the needed hardware in the first place. With many activities canceled due to isolation, some respondents may have more available time in their schedules, making it easier to connect with them. With this additional free time and without a commute we’ve found respondents to be more likely to be on time or even early and very happy to chat.
Conclusion
Despite the limitations and disruptions posed by these troubled times, there is an opportunity for organizations to increase their level of remote testing to ensure their moderated qualitative research needs are satisfied. Now is the time to hone the tools and procedures that can make this research successful, until we can be face-to-face again.