User acceptance testing with Flawsnap

The problem: UAT findings are hard to act on

A finance team is accepting a new expense approval flow before it goes live. Six people from finance and two managers have a week to try it. They are good at knowing what is wrong for their job ('this should go to my deputy when I'm away') and less good at describing it in a way a developer can reproduce. Findings come back as a spreadsheet with a column called 'Issue' and no screenshots.

UAT has two kinds of finding: defects (it doesn't do what was agreed) and change requests (it does what was agreed, but that isn't what we need). Both need to be visible, and the business needs to see which are blocking sign-off.

Before testing starts

1. 1

   Write the test plan as a project doc

   Create a doc in the project with the scenarios: 'Submit an expense over the limit', 'Approve as a delegate', 'Reject with a comment'. Docs have a page tree, so each scenario can be its own page. Use @mentions to assign scenarios; mentions notify people in the app.

2. 2

   Invite testers as members, limited to this project

   Invite each business tester by email. Invites are valid for 7 days and can be resent or shared as a link, and testers can accept with Google. Restrict them to the UAT project so they see only that board and those docs.

3. 3

   Allow the UAT environment

   Add the UAT host, such as uat.example.com, to allowed domains. Leave production off this project if you don't want UAT findings mixed with live reports.

4. 4

   Turn on team sign-in

   With Let team members sign in from the widget and app on, testers sign in once from the widget and their name is on every report. The sign-in lasts 30 days, which covers a normal UAT window.

5. 5

   Set categories that separate findings

   Use Defect, Change request and Question as categories. Testers can answer that, and the reports list can be filtered by it later.

A ready-to-use list of what to prepare is in the UAT checklist.

During testing

Testers work through the scenarios in the UAT environment. When something is wrong, they click the button, draw a box around the problem, choose a category and a priority, and write what they expected. The screenshot, the page, the browser and any failed API calls are attached for them. Ask them to start the title with the scenario number, such as 'S3: delegate cannot approve', so reports map back to the plan.

Link each card to the scenario doc it came from. At the end of a day, the scenario pages show which ones have open cards.

Using the board as the sign-off record

Change requests that won't make go-live can stay in Backlog with a due date or a comment explaining the decision. The card's activity log shows who moved what and when, which is useful when the business owner asks why something was deferred. You can rename lists or add your own if your sign-off has more stages.

What a good UAT finding looks like

• The scenario number and what the tester was trying to do, in their own words.
• Expected against actual: 'Expected it to go to Priya while I'm on leave; it stayed with me.'
• A marked screenshot of the screen where it went wrong.
• The right category, so a change request is not counted as a defect.

Pitfalls

• Testers on the wrong environment. If someone reports from production, it won't be accepted unless production is listed. That is usually what you want; tell them the UAT link.
• Developers closing cards that testers should accept. Agree that only the original tester moves a card from QA to Done.
• Everything is critical. Business testers often mark every finding critical. Define critical as 'we can't go live with this' in the test plan doc.

Questions

Do UAT testers need training?

Very little. The widget is a button, a markup screen and a short form. A two-minute walkthrough on the first day, plus the test plan doc, is usually enough.

Can we share UAT results with a stakeholder who has no account?

Yes. Write the summary in a project doc and send its public share link. Anyone with the link can read that doc without signing in.

Does a tester's sign-in work on several UAT sites?

Sign-in gives that site a token for one project, valid for 30 days. A tester working on two projects signs in on each.

Related

• User acceptance testing (UAT) checklist
• Project docs that live next to the bugs
• Team roles, invites and project access
• Bug reporting for QA testing
• Install Flawsnap on any HTML website