In safety-critical software projects, software quality is paramount. Especially for Class C software, which must be certified according to strict standards such as IEC 62304 (medical technology), it is essential that every aspect of the software is thoroughly tested. Defects in such applications can not only result in economic losses but also pose real threats to human life.
Unit tests and integration tests are essential in this context. They ensure that each functional unit of the software functions correctly (unit tests) and that all components function reliably together (integration tests). In practice, however, these tests often prove to be time-consuming and resource-intensive, especially when test coverage is to be maximized.
The testing tool VectorCAST offers a solution to this challenge. It provides an environment for automating unit and integration tests and helps analyze test coverage to ensure that no critical parts of the code remain untested.
This article presents the potential uses of VectorCAST for conducting unit and integration tests. The focus is on how to design such tests efficiently while maintaining a high level of test coverage and quality assurance. Finally, a practical example is presented that illustrates the practical application of test coverage and probe points.
What is VectorCAST?
VectorCAST is a testing platform specifically designed for automating unit and integration tests in safety-critical software projects. It helps development teams ensure the quality and reliability of their code by streamlining the entire testing process—from test case creation to test coverage analysis.
A key feature of VectorCAST is its ability to fully automate and repeat tests. The tool integrates seamlessly into existing development environments and allows both individual code units and entire modules to be tested at various test levels. This flexibility enables errors to be identified and corrected early, before they become more complex in later development phases.
In addition, VectorCAST offers comprehensive test coverage analyses that provide a detailed overview of which parts of the code have been tested and which areas may still be untested. The probe point feature allows you to specifically monitor specific locations in the code to better understand runtime behavior and more precisely identify potential error sources.
VectorCAST supports various programming languages such as C, C++ and Ada and is used in many safety-critical industries, including automotive, aerospace and medical technology.
Unit and integration tests with VectorCAST
VectorCAST supports both unit tests and integration tests to ensure that each functional unit and its interaction within the overall system functions flawlessly. While unit tests focus on testing individual functions or methods within a class or module, integration tests are used to verify the interaction of multiple modules and ensure that they function correctly in combination.
In this example, we'll set up a test environment for the MotorStatusHandler class, which monitors the status of a stepper motor. The StepperMotorDriver class is used to receive motor data and process it further in the MotorStatusHandler. The goal of the unit tests is to ensure that the methods of these classes function as expected, especially with regard to motor data processing and error detection.
Let's start with the setup. In VectorCAST 2021, we create a new project and specify the compiler we want to use.
Next, we create a test environment.
When creating the test environment in VectorCAST, the coverage types to be selected are 'Statement+MC/DC' (Statement Coverage and Modified Condition/Decision Coverage). MC/DC ensures that every condition in a decision and its combinations are tested at least once. We also select the white-box testing option because we know the internal structure of the methods and specifically test for different conditions.
Once the test environment is created, it provides a list of class methods that need to be tested.
To cover the ProcessStepperMotorData method with tests, we use automatic test case generation.
The result is three tests that were automatically generated to ensure method coverage. It's worth renaming these tests accordingly so the names convey their meaning.
In the following figure you can see an automatically generated test case for the ProcessStepperMotorData() method, where we can configure the expected behavior and also change the return values in the “Stub Function” section.
After the test is executed, a window opens with the test coverage for the tested method, in which the code sections and logical conditions covered by the test case are displayed in green.
By performing the next test (Figure 11) we get additional coverage (to what we already have).
Then we set up the final test and run it until we get full test coverage for the method.
Using Probe Points in VectorCAST
A key aspect of testing safety-critical systems is precisely testing all branches and conditions in the code. Especially when working with if statements and dependencies on specific input values, it can happen that certain paths cannot be executed without additional control. This is where probe points come into play.
Let's look at an example from our test for the ProcessFaultDriverStatus() method (Figure 14). This code handles various motor fault states and issues error messages when certain conditions are met.
In the following test path generated by VectorCAST, we see that the first if statement evaluates to TRUE, but all subsequent conditions remain FALSE:
To ensure that this test path is completed without any of the error detection conditions being met, the value of the motorDriverData variable must be adjusted accordingly.
By placing a probe point before the line if ((motorDriverData & MotorDriverDataMask::eTmc249aOt) != 0U), we can change the value of motorDriverData so that the conditions in the if statements remain FALSE, correctly covering the test path. This way, we can ensure that the first if statement evaluates to TRUE and all subsequent ones evaluate to FALSE, without having to change the original code.
The following figure shows the use of a probe point to ensure that the test case covers the targeted test path.
Without the use of probe points, this test case could not be fully covered, since we have no direct influence on the value of motorDriverData at runtime.
Probe points allow us to intervene in the testing process in a targeted manner, changing values or gathering additional information without altering the original code. The use of probe points contributes significantly to ensuring that safety-critical software is fully tested according to standards, thus guaranteeing high reliability.
Best practices when using VectorCAST
To make the testing process efficient and achieve maximum test coverage, the following best practices should be observed:
- Early integration of tests
Tests should be integrated early in the development cycle to detect errors early and avoid costly fixes in later phases. - Use of Probe Points
Probe points help test hard-to-reach code paths by manipulating variables during test execution, ensuring that all conditions are checked. - Combination of coverage types
The combination of Statement Coverage and MC/DC (Modified Condition/Decision Coverage) ensures seamless coverage, especially for safety-critical applications. - Using stubs and UTT (Unit Test Targets)
Stubs simulate external dependencies to focus on the functional unit. It's important to choose between stubs and real implementations (UTT) depending on your needs. Stubs are used to simulate external dependencies, while UTT is used to test the real code. Stubs are recommended for external libraries, while UTT is ideal when functionality needs to be verified in the real-world environment. - Regular maintenance of test environments
Test cases and environments should be updated regularly to ensure that they always reflect the current state of development.
Conclusion
VectorCAST is a versatile and powerful tool for performing unit and integration tests in safety-critical applications such as medical technology. It offers extensive features for mapping complex test scenarios and meeting test coverage requirements.
Through the use of probe points, the flexible use of stubs or unit test targets, and comprehensive support for various coverage types such as MC/DC or statement coverage, even complex and safety-critical systems can be tested efficiently.