Speech:

Testing AI based systems

AI-Based systems are being developed for various applications and are becoming more prevalent and critical to our daily lives. This talk will discuss how testing of AI based systems is different from that of traditional systems and the challenges it introduces. We will look at key concepts from the ISTQB CT-AI syllabus and ISO/IEC 29119-11 to show how quality can be achieved with these systems. We will also consider the range of approaches suitable for the black-box testing of AI-based systems, introducing, in some detail, the metamorphic testing technique.  We will discuss how white-box testing can be applied to drive the testing and measure the test coverage of neural networks.

With more AI systems being used and as we get a better understanding of how to test these systems, it is becoming more evident that achieving quality of these systems requires more than just testing of the model itself. The input data, which is used to train these models and highly impacts the performance and reliability of the model must also be tested. In some cases, the framework used to create the models needs to be tested as well. We will briefly introduce the work being done by JWG2 to define common defect types found at each of these test levels and testing types that can be used to identify such defects.