Typical Challenges of UAT Sessions

While User Acceptance Tests (UATs) sound extremely straightforward in theory, reality is a different ball game - they come with many challenges. These challenges can however be easily surmounted if the right approach is adopted. This article explores 5 challenges of UAT and proffers recommendations on how they can be tackled.

1. Users are often required to carve out time from their busy schedules to participate in UAT sessions. How much time do these users really have to push the system to its limits?

Faced with time constraints and impending deadlines, users are only able to test a limited number of flows through the system. Even though UATs are limited to the functional aspects of software, users may not be able to cover all the critical paths through the system within the allotted time. Consequently, test results may turn out to be incomplete and superficial.

Recommendation: Though achieving 100% coverage during testing is a myth (See 7 Practical Software Testing Principles & Myths from Jerry Gao), you can improve the chances of substantial coverage by starting UAT as early as possible. UAT sessions should be scheduled to hold at convenient times and not hurried so that users have enough time to test the main flows through the system. Extra UAT sessions may need to be scheduled to ensure testing is reasonably complete.

2. The UAT phase is usually left until after other tests have been concluded and the system is “almost” ready to go live. It is often the case that by the time users are called in for UAT, the system would already have been tested with most of the major bugs uncovered and corrected. This usually happens because implementers do not want users to go away with a bad impression of the system or a lowered confidence in their ability to deliver a high-quality system.

Consequently, the UAT session can easily turn into a formality that involves seeing the system in action, making observations for improvement and not an “actual testing” of the system as would be expected.

Most UATs begin with the mindset, “We want the users to test the system but we don’t want them to find any major bugs since any such bugs at this stage can easily lead to project delays”. This kind of thinking can lead to a situation where the Business Analyst or the IT Department hands down test cases that have already been successfully tested to users to retest.

Recommendation: Users should show their commitment to the system/project by designing their own test cases. These test cases are likely to be based on scenarios that the BA may not even be aware of or have missed. The BA could however support this phase by: 1) reviewing test cases to ensure that major scenarios have been covered, 2) supporting the set-up of test data & environment, 3) guiding users during UAT and 4) ensuring test results are documented appropriately.

3. In situations where users are not required to design their own test cases, important scenarios may be missed thereby increasing the risk of deploying the application. This then begs the question, how much effort can one really expect users to put into UAT, especially since they have other major commitments/objectives outside testing? Unless there's a strong executive backing and a sense of ownership of the project outcome, it may be difficult to get users to have the right attitude and invest their time.

Recommendation: To increase the chances of success, management should demonstrate their support for the project, emphasize the importance of UAT and require that users test and sign off on their system as a mandatory requirement for moving any system into production.

4. In situations where users have not had enough time to test the system exhaustively, it can become quite difficult to obtain sign-off. This could potentially delay the project if obtaining sign-off is a requirement before go-live.

Recommendation: Ensure that users see prototypes of the system and are kept in the loop long before they are expected to test and sign off on the system – this will increase their sense of ownership and prevent delays resulting from not understanding the system.

5. Another common challenge in UAT sessions occurs when users don’t understand how the system works. On our team, we start with a demonstration of how the system has been designed to work and run a number of tests before asking users to execute their own test cases. In our situation, this demonstration is usually enough because the users are already familiar with the BPMS platform and only test the automation/flow of new processes.

Recommendation: Training users on how the system works is essential for effective UAT but should be organized separately or the UAT session will stand the risk of turning into a training event. Users should be trained on how to use the application as well as on the mindset required to test the system effectively.

Which of these challenges do you face?