We are rolling out a 3-part blog series that discusses how to test your clinical automation system effectively and accurately. Subscribe here so you don’t miss an article.

The user acceptance testing for the wet testing phase of your automation system project should be designed to exercise the complex rules and routing logic as well as the wide range of use cases and customized workflows to ensure that your lab’s automation system delivers on expected outcomes.

In part 1, Practical Approaches to Risk-Based Automation System Testing, of our 3-part Strategies for Testing blog series, we discussed how to determine the scope of automation user acceptance testing by applying a risk-based approach that reflects your real-world automation workflows.

Here, in part 2, we discuss how to plan a user acceptance validation testing program focused on creating workflow testing scenarios that target your key rules and business logic. Taking the time to build a program around your testing strategy helps you ensure that you have your bases covered In so doing you will optimize your processes, assure the quality of your rules-based program and reduce re-work when your system goes into production.

Create a Risk-Based Validation Protocol and Test plans

Your user acceptance validation testing program should include a ‘road-map or ‘recipe’ that describes the ‘how’ and ‘what’ of your testing event. This takes the form of a Risk-Based Validation Protocol document (the ‘how’) and the Test Plan document(s) (the ‘what’). Both documents should be written with an adequate amount of detail to describe how the testing event will be accomplished, and the strategy to get it done. Let’s dive into each of these documents separately.

Pro-Tip. The Risk-Based Validation Protocol and Test Plans should be maintained as foundational documentation that forms both the starting point for current testing as well as subsequent testing events. A library of validation protocol documents and test plans provides the risk-based framework for future testing that is specific to your system.

The Risk-Based Validation Protocol Document

Just like cooking a meal, planning is required to consider the ingredients (testing components), where and how to cook the meal (where and how to test), the cooking time and temperature (testing time, skills, resources), and what the outcome of the meal should look like (expected testing outcomes). Without adequate planning, your meal (testing event) may not turn out as planned. By preparing a Risk-Based Validation Protocol document you will not only cover the planning details of a typical validation protocol document, but you will also identify the testing strategy that enables you to target areas of your rules systems that might otherwise get missed during testing, including those that impact patient safety. A Risk-Based Validation Protocol document will also lead to a more streamlined and directed verification process that shortens the time to go live.

To help you keep your testing event contained and on track we recommend you complete a Risk-Based Validation Protocol document that addresses:

Program purpose and objectives – defines the purpose of your validation program, clear quality objectives, and timeline goals that are measurable and time-bound.

As you prepare later sections of your Validation Protocol document refer back to this section frequently to make sure that your developing plan still aligns with your original objectives.  Sometimes when you get down into your plan and start identifying resources, requirements, and test plans, the effort will balloon and could require you to go back and re-assess the original objectives.   

Testing strategy – sets out a summary of how the testing will be addressed. Part 1, Practical Approaches to Risk-Based Automation System Testing, of our 3-part Strategies for Testing blog series, describes an effective testing strategy that prioritizes the testing of complex (multi-rules) workflows over testing the simplest (no rules triggered) workflows. Our inverted pyramid diagram illustrates this testing approach.

Responsibilities, resources, departments, and personnel – a description of the human, IT, and physical components that will be needed to complete the testing. Here, you’ll want to document the major stages of your testing program, such as test environment preparation, test execution, approvals, and so forth. For each stage of the program, identify the requirements and pre-requisites such as set-up/tear down, interface connections needed, equipment needed, department approvals or assistance, and so forth. Then layer in the designated personnel skills required including their responsibilities and role. In addition to the testers, including the project leader, lab IT resources, and reviewers as well as the requirements for the quality assurance department approval. And finally, in this section, you can start to ascribe a timeline for each stage of the program. This section will help you confirm, ‘are the original objectives and goals still realistic for this testing program’?

Testing approach – A list of the approved testing tools or methods to execute the testing process. These would include both automated and manual procedures used to facilitate the testing activity.

Pro-Tip. If you utilize automated, semi-automated methods they may include screen-capture technology. Otherwise, for manual testing methods, software such as Microsoft® Paint or Microsoft® Snipping tool and Snagit® can be used.

Testing environment – A self-contained non-production environment, identical or near-identical to your production environment regarding software version, operating system, settings, rules, and LIS/instrument interfaces. Your test environment should be isolated from other applications and traffic that may impact performance or cause the test environment to inadvertently overwrite production patient data.

Pro-Tip. A separate document should be created with the connection information needed to connect the LIS and instrument interfaces. Typically, this is a listing of IP addresses and other settings (MAC addresses, Subnet Mask, Gateway addresses, etc.) required to connect and disconnect your instrument and server interfaces from the production to test and then back to the production system. The instructions should also include how the testers will access the test environment including passwords and any security settings needed to enter the test area.

Pro-Tip. A connection diagram is useful to visually understand the data flow between systems, their components, and interactions. Your IT team should be able to create a graphical diagram that depicts your test interface system design that includes your LIS, middleware server, and automation equipment.

Timelines – The Risk-Based Validation Protocol document should include an estimated timeline for the completion of the collection of test case scenarios included in the testing protocol document including time for data review and collation in the final documentation.

Acceptance criteria –Include a section in your Risk-Based Validation Protocol document that addresses the definition and criteria for pass/fail criteria. A simple narrative summary of the acceptance criteria should align with the Validation Protocol objectives and testing strategy as well as aid the testing team in recognizing what will be accepted as a pass or failed state.

Compliance –Get the team together to step through the document as a team. Stakeholders can discuss concerns, risk mitigation and align on scope and direction. Getting individual feedback and approval without a meeting can create inadvertent silos and less than optimal alignment as the project progresses.

In the end, a thoughtful risk-based approach to preparing the validation protocol document is beneficial because it organizes and directs the validation process in a way that reduces repetitive activities,  and, more importantly,  provides a higher degree of assurance that the testing will optimize rule-system performance.

Sustainable User Acceptance Test Plans

The Test Plan – download our example - refers to a detailed document that articulates for a particular testing challenge, the strategy, purpose, tools, resources, and specific testing instructions to complete a particular testing challenge. The Test Plan represents the ‘how-to’ or ‘recipe’ for executing the testing and ensuring your software and rules logic are working as expected.

Each Test Plan describes the testing exercises that are performed based upon the overall testing strategy approach described in your Validation Protocol document. If you completed the risk-based analysis from Part 1, Practical Approaches to Risk-Based Automation System Testing, of our 3-part Strategies for Testing series, it will guide you in developing the right Test Plan(s) based on the prioritized automation and clinical workflow scenarios.

Each Test Plan should stand on its own and not entirely duplicate the previous test plan. Each Test Plan should build on the other and introduce new features to approximate the user experience and workflows. The Test Plan elements set the blueprint for the testing that guides the thinking process and defines the details.

 A Test Plan should include these essential elements:

Scope – a synopsis of the specific features or logic that you want to challenge.

Test Plan identifier – a specific unique title or alphanumeric code that distinguishes it from another Test Plan. Smart identifiers help the team quickly identify them when they fall into a list or when added to a subject line of an email. It is helpful if identifiers can be constructed to incorporate the name (or abbreviations) of the project, Test Plan title, and version of the software.

Pro-Tip. It is recommended that if you retire a Test Plan that you avoid re-using its unique identifier so that traceability can be maintained

Overview – key information that describes the purpose and expectations of the Test Plan. The elements of the Test Plan overview include:

• Purpose or objective of this testing exercise and its alignment with the objectives stated in the Validation Protocol document

• Functionality, workflow, or logic to be tested

• Testing assumptions, pre-conditions, or risks anticipated

• The version of software to be tested

• Testing environment, including interfaces

• Automated tools that will be used for any step in the testing process

• Testing specimens needed to complete the Test Plan

• Testing exclusions or comments that will aid the tester and reviewer

• Responsibilities and requirements of the tester and reviewer

Pro-Tip. You may want to consider building your Test Plans as controlled documents so that they can be used again in the future. Include a section at the front or back of each Test Plan to capture document approval, author, contact information, and revision history.

Resources – description of the tester skills needed to execute the Test Plan and the related and appropriate review and approval resources from management, quality assurance, and others.

Testing inputs - testing supplies and/or inputs such as consumables, reagents, and specific clinical specimens. You will want to indicate in your Test Plan how many and what types of specimens (blood, plasma, urine) are required to generate or trigger specific clinical conditions.

Testing steps – a set of steps that represent the workflow actions to prove the intended outcome with a provision for marking each step pass or fail. This section should also include a way for the tester to notate that the test case is successfully executed, along with the ability to note defects or issues encountered, and re-testing, as necessary.

Pro-Tip. Once it has been determined what type of clinical specimens and values you need, your testing team should collect and store these previously analyzed specimens. Creating a listing of samples and any special clinical conditions required in a table matrix is an efficient method to direct your specimen collection activities.

Testing outcomes expected - a brief description of the pass and fail criteria (expected outcomes) that guide the tester in how to evaluate the testing outcomes including when actual results do not match expected results.

Re-testing criteria and approvals – a narrative that describes how the tester should assess or grade the ‘non-compliance’ of testing outcomes, a description of the most common remedies, and escalation instructions for issues that cannot be resolved by the tester. Generally, a Test Plan will pass when all test steps pass. If the tester cannot successfully complete all test steps within a Test Plan, it fails. Issue mitigation of a failed step may include a review with management or other appropriate reviewers to agree on possible software configuration changes or modifications as well as the subsequent re-testing required to pass the step.

Test execution sign-off and approval sign-off – a section that captures tester, reviewer, and approver information and their signatures. A final Test Plan pass/fail determination and related comments may also be captured here.

Looking Ahead

In our final blog, Simplify Your Validation Summary Documentation of our 3-part Strategies for Testing series, we will review the recommended process for documenting your Test Plan outcomes to meet quality, accreditation, and regulatory compliance as well as to prepare you for re-using them. Keeping clear, concise and easily retrievable records of your testing programs reduces the amount of time that reviewers and quality assurance teams need to spend approving them. In today’s environment, changes come fast; a library of readily available Test Plans will help you pivot faster.