Property-based Testing

software-engineering testing

Definition

Property-based Testing

Property-based testing is a testing technique where, instead of writing specific test cases with concrete inputs (Example-based testing), the developer defines properties (invariants) that the code must satisfy for any valid input.

A testing framework then automatically generates a large number of random inputs to try and falsify these properties.

Comparison

Feature	Example-based Testing	Property-based Testing
Input	Manually chosen concrete values (e.g., sum(1, 2)).	Randomly generated values within a domain (e.g., sum(x, y) for any int).
Verification	Check specific expected output (e.g., result == 3).	Check general properties (e.g., result == x + y, result >= x).
Goal	Verify known scenarios and regressions.	Explore the input space to find edge cases the developer missed.
Failure	Test fails once.	Framework finds a counterexample (minimal input that breaks the property).

Core Concepts

1. Generators: Tools that produce random data (integers, strings, lists) satisfying certain constraints (e.g., “non-null string”).
2. Properties: Boolean expressions that must hold true for all inputs.
3. Shrinking: When a failure is found (e.g., a list of 1000 items), the framework attempts to find the smallest input that still causes failure (e.g., a list of 1 item) to simplify debugging.

Common Issues

1. Expensive Generation: Generating complex data structures (e.g., unique sorted lists of prime numbers) can be computationally expensive or difficult to implement.
2. Boundary Definition: Failing to correctly express the boundaries of a property (e.g., handling NaN in float ranges) leads to false positives/negatives.
3. Data Distribution: Ensuring fair distribution of inputs is critical.
   • The framework tries to distribute inputs evenly (e.g., uniform distribution of integers).
   • Risk: Custom generators or filters might inadvertently skew this distribution, causing the test to miss entire classes of valid inputs (e.g., never generating 0 or negative numbers).

Common Properties

• Inverses: decode(encode(x)) == x
• Idempotence: sort(sort(x)) == sort(x)
• Invariant: size(filter(x)) <= size(x)
• Commutativity: add(x, y) == add(y, x)

Example

Requirement: A sort function.

Example-based:

@Test
void testSort() {
    assertEquals(List.of(1, 2, 3), sort(List.of(3, 1, 2)));
}

Property-based:

@Property
void sortedListIsOrdered(@ForAll List<Integer> list) {
    List<Integer> sorted = sort(list);
    // Property 1: Size remains the same
    assertEquals(list.size(), sorted.size());
    // Property 2: Every element is <= the next
    for (int i = 0; i < sorted.size() - 1; i++) {
        assertTrue(sorted.get(i) <= sorted.get(i + 1));
    }
}