DSpec2Test: Specification-Driven Test Generation in Dafny

Abstract

Verification-aware languages, such as Dafny, integrate logical constructs into code and enable automatic verification of program correctness. However, tests remain helpful in scenarios that verification alone does not address (e.g., to support test-driven development). Existing Dafny test generation tools are implementation-based, limiting their applicability in this context.
We present DSpec2Test, a specification-driven test generation tool for Dafny that automatically derives tests from formal specifications, without considering implementation details. Our tool extends Dafny’s generate-tests command with a new black-box mode based on Disjunctive Normal Form (DNF) equivalence class partitioning and optional Boundary Value Analysis (BVA). DSpec2Test relies on the Z3 SMT solver to synthesize inputs and expected outputs that meet the specification-derived constraints.
We evaluate DSpec2Test on programs from DafnyBench mutated using MutDafny and compare it against Dafny’s existing implementation-driven Block mode. DSpec2Test achieves a 93.9% mutation kill rate on a dataset of 131 mutants, outperforming Block’s 82.4%, and uniquely killing 17 mutants. These results suggest that specification-driven testing is an effective and complementary approach for testing Dafny programs.
Demo: https://youtu.be/mK1EeJfinRQ
Code: https://github.com/VeriFixer/DSpec2Test
Prebuilt docker image: https://doi.org/10.5281/zenodo.21191158

Alexandra Mendes

Assistant Professor

My research focuses on encouraging a wider adoption of software verification by creating tools and methods that hide the complexities of verifying software. Much of my most recent work overlaps with the area of software engineering. For more details, see selected publications and some of my projects. Follow me on Twitter or add me on LinkedIn. See also the Software Reliability Lab website.