Model-based testing relies on models of a SUT and its environment. Model traces are interpreted as test cases for the SUT: input and expected output. Because models need to be validated themselves, models and SUTs must reside at different levels of abstraction - otherwise, the effort of validating the model would match the effort of directly validating the SUT. As a consequence, in order to execute model-based tests, driver components for bridging the levels of abstraction are required. In addition to models and drivers, a third crucial ingredient is that of test selection criteria. Structural criteria lend themselves to automated generation and may be required by development standards; but coverage usually does not correlate with failure detection. In this survey talk, I will highlight several methodological, economical and technological characteristics and challenges of model-based testing. Since I consider the problem of defining "good" tests to be one of the fundamental issues, I will then briefly focus on ongoing work on generating tests from properties rather than structural criteria. Specifically, I'll show one approach to generating tests via model-based flaw injection. These tests (1) targeted security properties and attacks such as cross site scripting attacks rather than structural criteria, and (2) are turned into implementation-level tests that are run in a semi-automatic way and that often can reproduce attacks at the implementation level. [Slides]
Pathologies and Distributed Testing of Message Sequence Charts
Message Sequence Chart (MSC) is a popular language for modelling distributed systems. It is known that an MSC specification can contain different types of pathology. However, definitions of different types of pathology and the problems caused by pathologies are unclear, let alone the relationships between them. In testing systems with distributed interfaces/ports we may place a separate tester at each port. It is known that this approach can introduce controllability problems which have received much attention in testing from finite state machines. However, controllability problems in testing from MSCs have not been thoroughly investigated.
In this talk, the relationships between pathologies and controllability problems of distributed testing are discussed. This is achieved by following steps. Different types of pathology and the problems that they cause are introduced based on a novel MSC pathology framework. We show that race and non-local choice, two types of pathology in MSCs, cause synchronisation problems in distributed systems. They are orthogonal and causes different types of problematic scenarios. We identify two types of controllability problem in MSC-based testing: controllability problems of timing and controllability problems of choice. It transpires that each type of controllability problem is related to a type of MSC pathology. Controllability problems of timing are caused by races but not every race causes controllability problems; controllability problems of choice are caused by non-local choices and not every non-local choice causes controllability problems. Finally, algorithms are provided to tackle both types of controllability problems based on existing work on detecting race and non-local choices in MSCs. [Slides]
Model-based conformance testing of reactive and timed systems
After an introduction to conformance testing, we study the testing theories and automatic test generation from two kind of models. In a first part, we consider simple models of transition systems with inputs/outputs, named IOLTS, the underlying testing theory funded on the conformance relation ioco, and test generation in this context. In the second part, we consider timed models with inputs/outputs (TAIO), we study the underlying testing theory funded on the conformance relation tioco, and the more involved problems of test generation that follow. [Slides]
Higher Order Mutation Testing
Traditional Mutation Testing considers only first order mutants, created by the injection of a single fault. Often these first order mutants denote trivial faults that are easily killed. Higher order mutants are created by the insertion of two or more faults. This talk will briefly cover the history of Mutation Testing, considering why it has been traditionally a first order paradigm and giving motivations for a move to the higher order paradigm. The talk will show how Search Based Software Engineering (SBSE) can be used to seek out, from the impossibly large space of potential higher order mutants, those which may possess important and valuable properties that make them good at revealing faults that may otherwise go unnoticed. The talk will also show a mutation-based test data generation approach which targets strong mutation adequacy and is capable of killing both first and higher order mutants. [Slides]
Testing Software via Dynamic Symbolic Execution
Symbolic execution has gathered a lot of attention in recent years as an effective technique for generating high-coverage test suites and for finding deep errors in complex software applications. In this talk, I will discuss the main challenges of symbolic execution in terms of path exploration and constraint solving, and our experience building two practical symbolic execution tools, EXE and KLEE, which are able to automatically discover serious bugs and security vulnerabilities in a diverse set of software systems, including network servers, file systems, device drivers, packet filters, utility applications, and computer vision code. [Slides]
The long way from the requirement to test scenarios - How to derive consistent and coherent modular Features, Test Cases and Test Scenarios?
Requirements represent the starting point of long trip for the development as well as the validation of systems. Transparency, coherency, consistency of requirements, features and functions, system behaviour, test cases and test scenarios or sequences, behavioural models are highly desired - hopping the dream of it becomes true! From the experiences of creating a new test standard for a new train control system and its evolutionary story the lessons learned will be shown, i.e., figuring out a comprehensive testing methodology. Firstly, the question will be - how to derive features from a complex system specification which contains more than 4000 core requirements. Thereby the creation of hierarchical features from principle, functional and procedural requirements will be demonstrated on that real world example. Furthermore, the moving target problem of constantly changing requirements will be tackled when creating and maintaining a complex and detailed set of test cases containing around 1500 test cases. The next point will be the introduction of a generic Meta-model for real-time system behavior and how it increases the transparency, consistency and coherency on the microscopic and macroscopic level of the system behavior applied to test cases and test sequences. Last but not least we generate a model from the existing test cases using the starting and end conditions as states in order to get a model starting model-based testing. All these steps form a comprehensive testing methodology for a continuous test creation process independent if you would like to classically derive test cases directly from requirements or if you apply model-based testing as approach of tomorrow. [Slides]
Security testing: a key challenge for software engineering of web apps
Yves Le Traon
While important efforts are dedicated to system functional testing, very few works study how to specifically and systematically test security mechanisms. In this talk, we will present two categories of approaches.
The first ones aim at assessing security mechanisms compliance with declared policies. Any security policy is strongly connected to system functionality: testing function includes exercising many security mechanisms. However, testing functionality does not intend at exercizing all security mechanisms. We thus propose test selection criteria to produce tests from a security policy. Empirical results will be presented about access control policies and about Android apps permission checks.
The second ones concern the attack surface of web apps, with a particular focus on web browser sensitivity to XSS attacks. Indeed, one of the major threats against web applications is Cross-Site Scripting (XSS) that crosses several web components: web server, security components and finally the client's web browser. The final target is thus the client running a particular web browser. During this last decade, several competing web browsers (IE, Netscape, Chrome, Firefox) have been upgraded to add new features for the final users benefit. However, the improvement of web browsers is not related with systematic security regression testing. Beginning with an analysis of their current exposure degree to XSS, we extend the empirical study to a decade of most popular web browser versions.The results reveal a chaotic behavior in the evolution of most web browsers attack surface over time. This particularly shows an urgent need for regression testing strategies to ensure that security is not sacrificed when a new version is delivered.
In both cases, security must become a specific target for testing in order to get a satisfying level of confidence in security mechanisms [Slides]
FSM-based test derivation methods
The development of test cases is an important issue for testing software, communication protocols, digital circuits, and other reactive systems. Test suites derived using methods based on Finite State Machine (FSM–based) have been extensively used by research community in the last years, those are the W, Wp, UIO, UIOv, DS, H, HIS, and SPY test derivation methods. In this lecture, these methods and the relationship between them are overvied. Moreover, we briefly discuss how these methods can be adapted for various modifications FSM model such as nondeterministic FSMs, extended FSMs, timed FSMs etc. Experimental results of using those test suites for some protocols are provided. [Slides]
On Test Design
Since proper and adequate testing is a quality investment for any company producing software systems, it is essential to choose efficient, effective and applicable methods. Regardless of process, understanding these methods is fundamental to any software or system testing approach. Different test design techniques are helping us select test cases, since full path coverage is never feasible for most real industrial applications. I will describe our initial guidelines of test design techniques, common mistakes in test case writing, and look at some non-functional test approaches to illustrate some challenges that could give inspiration for further research. [Slides]
Tutorial: Automatic structural unit testing with PathCrawler
Automation of test-case generation brings obvious benefits. In critical systems processes where structural testing is required by the development norm, manually creating tests from the specification fails to achieve complete satisfaction of the coverage criterion. In this case, automatic methods help to reach the objectives which are not covered and provide corresponding path conditions that may be used to refine the specification if needed. They may also determine whether the objectives which are not yet covered are really infeasible. Even when the development process does not impose any structural testing activity, the use of a structural test generation tool is a way to increase the quality of the software with a very low cost overhead.
PathCrawler is an automatic tool developed at CEA LIST for the structural testing of C functions. The tutorial will take advantage of the latest version of PathCrawler which is in the form of a freely available web-based testing service: PathCrawler-online.com. This will enable participants to bring their own computers and interactively participate in the tutorial. After a brief presentation of PathCrawler, the tutorial will show participants how to easily and quickly test their code. Based on different examples of C code, users will be guided through the input parameters and shown how the outputs can help in debugging or justifying coverage. Some limitations and novel uses of structural testing will also be illustrated. [Slides]
Architecture-based Testing and Analysis: achievements, challenges, and potentials
Software Architectures (SA) have been advocated as a means for improving the dependability of software systems. In this light, different methods have been proposed for assessing the architectural decisions correctness with respect to system goals and requirements, and to drive a better system design and implementation.
Testing and analysis play a central role in architecting dependable systems. Conformance and regression testing techniques, as well as functional and non functional analysis methods have been proposed to improve the resultant software systems. Building on state-of-the art research, workshops and meetings that have been organized in this domain, this talk wants to briefly describe the current role of software architecture in the testing and analysis of complex software systems. Achievements, challenges, and potentials for future research will be presented. [Slides]