We had recently launched our new app and were eager to see it in the wild (in hands of real users). So we decided to conduct usability tests.

Although the initial target was to conduct 50 tests, we halted the experiment after 25. We had started to see a lot of repeated feedback which we realised should be fixed before we conduct more such tests.

Disclaimer - I do not recommend doing this. It was very tiring going through 25+ 1 hour long calls in a 40 hour work week.

Why 25 tests in one week!?

I have always considered, talking to users as the weakest part of my PM skill set. And I wanted to get better at it. I had not really conducted proper usability tests for a long time and the ones I conducted were too far in between to give us any serious insights.

So, I forced myself to reach out to users and schedule as many calls as possible. That resulted in my calendar being booked from 7:30 on a Saturday morning to even on some occasions 8:30 in the evenings.

What did I learn from this experience?

1. I achieved what I set out to do. I think I have now built a better muscle for conducting such tests and actually just talking to users.

2. It is difficult to conduct usability tests remotely. We had to ask users to share screen from their phones so that we all could see the screen they are on.

3. Last but actually the primary purpose of conducting these tests, some pretty glaring misses from us on the usability side.

Tips for me (and possibly the readers) for future usability tests

1. Create a brief email that can be sent to customers beforehand. In this email, send a link of the app and a small description of what is expected out of them for this experiment.

2. Set the expectations on the call too, before you dive into the tests. I used these pointers for my calls

   • This is just the first version of the app so we can change directions easily
   • Don’t hold back while giving us feedback, we would appreciate you being brutally honest
   • I might ask you questions while you are exploring the application, to better understand your point of view

3. Silence is okay. Don’t rush to fill the silence with your words. Let the user break the silence. They’ll do it, don’t worry. If nothing else they ask “So what’s next?”

4. Record sessions. It might not be easy for you to come back and view all the recordings, but I found myself going back to specific sections in an interview. Might be to revalidate a bug or to confirm the answer to a question.

5. One question that always gave me good insights was this - “On a scale of 1 to 5, how helpful did you find this screen?” You should not miss the opportunity to ask a follow up question and understand what made them give that rating.

6. Keep your video on. And have a smile on your face. Basically act human and not like a scientist observing a lab rat.

7. Thank them always.

8. There are no stupid questions. Answer sincerely and note all of them down.

9. Have someone else take notes, but also have a notebook handy to jot down highlights.