Automatic Test Case Generation: When It Makes Sense

Many automotive clients express a desire for fully automatic test case generation. But why? Their wishes typically encompass:

• Scalability: With automatic test case generation, the quality assurance process can scale up rapidly.
• 100% Coverage: Achieving comprehensive test coverage without exhaustive effort. 
• Reduced Manpower: The need for fewer testers. 
• Maintenance Elimination: The dream of reducing testing-related maintenance overhead.

The potential benefits of automatically generated tests are abundant. However, I believe that in the short to medium term, a significant portion of tests will still need to be crafted by human testers. 

“Why?”, you ask?

A test case consists of two core elements: 

1️⃣ Test Data: This involves stimulating the test subject within its environment. 

2️⃣ Expected behavior: A description of how the test subject should behave in specific situations. 

While test data can be generated in various ways, the automatic definition of expected values (for functional behavior) remains a challenging task. 

This is where skilled testers come in. A good tester isn’t just someone who knows testing methods and tools; they are also product experts who understand the ins and outs, the quirks, and the corners of the product. 

This product-specific knowledge is typically built through manually created, requirement-based tests. Test by test, a tester accumulates knowledge about the product. 

I firmly believe that manual test case creation won’t become obsolete. However, I also foresee the increasing adoption of automatically generated test data because it can significantly boost testing efficiency. Nevertheless, defining expected values will remain a tester’s task. 

In TPT, we’ve designed a feature to accelerate testing:  the ability to define expected values separately from test data. 

This approach offers several advantages: 

• Reusable Functional Descriptions: Once specified, functional behavior can be assessed in every test. 
• Simplified Updates: When behavior changes (e.g., due to new dependencies or conditions), you only need to update one description. 
• Evaluation Flexibility: Any test data, including automatically generated data, can be evaluated without additional annotations. 
• Clear Traceability: Direct and unambiguous links between requirements and expected values.

Separating stimulation from test data is, in our view, the enabler for fully automated test data generation. With a comprehensive description of expectations, you can create any scenario (via test data) and promptly detect issues. 

By the way, our TPT addon, TASMO, is by far the most beloved test data generation method among TPT users. TASMO not only measures coverage but also fills coverage gaps through automatic test data generation. And with TPT 19, TASMO went even further, as test cases can be created directly from formalized requirements. 

Stay curious, keep testing, and remember that while automation can accelerate processes, a tester’s expertise remains indispensable.