“Overloaded! ” — A model-based approach to database stress testing

As a new era of “Big Data” comes, contemporary database management systems (DBMS) introduced new functions to satisfy new requirements for big volume and velocity applications. Although the development agenda goes at full pace, the current testing agenda does not keep up, especially to validate non-functional requirements, such as: performance and scalability. The testing approaches strongly rely on the combination of unit testing tools and benchmarks. There is still a testing methodology missing, in which testers can model the runtime environment of the DBMS under test, defining the testing goals and the harness support for executing test cases. The major contribution of this paper is the MoDaST (Model-based Database Stress Testing) approach that leverages a state transition model to reproduce a runtime DBMS with dynamically shifting workload volumes and velocity. Each state in the model represents the possible running states of the DBMS. Therefore, testers can define state goals or specific state transitions that revealed bugs. Testers can also use MoDaST to pinpoint the conditions of performance loss and thrashing states. We put MoDaST to practical application testing two popular DBMS: PostgreSQL and VoltDB. The results show that MoDaST can reach portions of source code that are only possible with non-functional testing. Among the defects revealed by MoDaST, when increasing the code coverage, we highlight a defect confirmed by the developers of VoltDB as a major bug and promptly fixed.