Remote Testing

This has been sat in my drafts since 2020! Back when Covid was all we heard about and when working from home seemingly happened overnight. In this case mid way through a project where we were about to embark on testing.

At the time and after I recorded some observations and lessons learned from this experience as a reference point for future projects, and thought now is as good a time as any to share.

My most recent project was hit by the pandemic - like countless other projects - mid-way through UAT (User Acceptance Testing) meaning we had to think on our feet and look to methods of managing testing remotely.

I’m pleased to say that thanks to good collaboration tools and great buy-in from our business SMEs, it meant our testing process was pretty much unaffected.

What did we learn along the way?

Always one for reflection, I put together a brief lessons learned piece on what we learned as a team in this time to help other workstreams with their testing process. Here are some of the key points:

What worked well

• Single Master Sheet - Test script updated at the end of each day & recirculated the following morning. Single source of truth & easier to track progress & issues.

• Daily Teams call in the morning - Teams call each morning, no longer than 30mins, to review at a high-level previous day progress, agree plan for the day, discuss/resolve blockers & focus team.

• Breaking down steps by role - Highlighting for each script which role is responsible for a step or steps to identify the sequence for testing. Enables prioritisation and better planning for testing process.

• Motivation using % complete / steps remaining - Without being in the same room testers can soon lose focus. We found with our team they were motivated by knowing the % complete & steps remaining KPIs recorded during testing.

• IT resource on hand to resolve issues - IT resource(s) on hand to work with testers when issues come about to understand and resolve issues.

• Combination of Teams calls & email exchanges to manage issue resolution process.

• Sizing the tests - Applying an indicative size to each script & amending as we went. Enabled proactive management of testing, helped the project team determine the length of testing remaining and testing could be targeted to clear off smaller tests, for example.

• Business Owner as coordinator - Having the business owner coordinating the team helped significantly, as they could specify the scripts that could be done, those dependent on other scripts & where the team should focus next.

Challenges we came across

• Everything takes longer - Working from home, often using one screen and managing BAU / Contingency Plans meant testing was deprioritised and delayed. Most days testers only spent 2 – 3 hours in total testing as opposed to 4 – 7 like they were when in the office.

• Avoid email (where possible) - For most testers Teams had not been used before, meaning email was used as the primary tool. Long emails often can be misinterpreted or unclear as to what the tester is trying to communicate.

• Issue resolution - Without being in the same room we found issues took longer to 1) understand and 2) resolve. Multiple Teams calls and emails were needed to diagnose and obtain updates on issues. This became hard to track at times, with other issues (e.g. testers not understanding the process and limited commentary) causing steps to fail but upon closer inspection were not found to be issues.

• Misinterpretations of the test scripts happen more - We realised quickly that testers would wait and build up questions and send them all at once to the SME, again this caused delays. A lot of the time this was because the scripts themselves were not clear to the tester – if they were in the same room this would be clarified immediately.

• Avoid changing test resource mid-way through testing - Unfortunately, this was unavoidable for us but we found it disrupted the flow of testing and meant we were unable to walk the new tester through the test process face to face as they were already working from home.

• User inexperience using Teams - Underpinning most of the challenges already mentioned is the general inexperience of using Teams. This meant a lot of communication from testers was through email and caused more delays.  

What would we do differently next time (Lessons learned)

• Assign “chunks” of time for testing - With an understanding of the test scripts it should be possible to work with the testers to assign chunks of their time to focus primarily on testing (e.g. 2 hours for test script 1).

• Better commentary on testing sheet - Some scripts covered end to end processes where back office needed to reconcile specific payments. Next time we would instruct testers to note down specific deals even if the script doesn’t ask them to, and once they had completed their part of the process to share this information with the next role in the process.

• More daily catch ups including ad hoc - Morning catch ups worked well, but more catch ups would have helped keep a level of focus, ensure any blockers were raised in good time & could help with managing the testing.

• More rigour around issue logging & tracking - Improve the visibility of issues, review issues at the end of each day to verify whether an issue or not & provide testers with a clear process for qualifying and raising issues.

• Intro pack for testers - Having a document with instructions for users to access Citrix, the application in DEV / UAT and a description of the test scripts would have been useful to have to provide to testers, in particular those having to work from home.

• Focus on one script at a time - Some scripts had to be run over 2 – 3 days to check overnight schedulers would run and interest rates would be applied correctly. This broke up the flow of testing & next time we would identify these scripts & plan out these scripts to prevent delays.

• Centralised view of testing & issues - Utilising a feature or features in Teams to aid stand ups & calls would have been useful, particularly for the tracking of issues. This is something we would look in to for the next wave of testing.

• Teams user training - Possibly include Teams user training ahead of any testing to ensure users are familiar with Teams, how the issue resolution process will work, sharing screens, etc. (this need should lessen the more users get familiar with Teams).

A large portion of this is applicable to normal testing of course, but perhaps the biggest lesson we took away as a team is to communicate more and communicate clearly. Plus, having great buy-in from your stakeholders helps enormously! (as it does in general!)

Without knowing how long this pandemic will last, I’m pleased we have this remote testing experience to look back on and learn from for the next time.