Property-based Testing in Detail

Now that we have a high level idea about what property-based testing is all about it’s time to see how this works in a JavaScript setup.

The following examples are based on testcheck-js by Lee Byron, but there other alternatives like jsverify.

We have a handlePayment function that expects two arguments balance

const handlePayment = (balance, payment) =>
  (!payment || balance - payment <= 0) ? balance : balance - payment

In a traditional fashion you would start to write a couple of tests that verify certain expectations. For example your test might look something like this. You might test that handlePayment

import { assert } from 'chai'
import handlePayment from '../src/handlePayment'

describe('handlePayment', () => {
  it('should handle zero inputs', () => {
    assert.equal(handlePayment(0, 1), 0)
    assert.equal(handlePayment(1, 0), 1)
  })
  it
 
('should handle negative inputs', () => {
    assert.equal(handlePayment(-1, 1), -1)
    assert.equal(handlePayment(10, -1), 11)
  })
  it('should handle positive inputs', () => {
    assert.equal(handlePayment(200, 1), 199)
    assert.equal(handlePayment(10, 11), 10)
  })
})

We run the tests and see everything is green.

What property function would we need to define for verifying our assumptions?

Let’s verify that zero payment always returns balance.

import { check, property, gen } from 'testcheck';
import handlePayment from './handlePayment'

const result = check(property([gen.int], (x) => {
  return handlePayment(x, 0) === x
}))

console.log(result)

This is the log output:

{ result: true, ‘num-tests’: 100, seed: 1471133618254 }

testcheck-js ran one hundred tests against handlePayment and all cases passed. Now let’s verify if handlePayment works correctly when working with a balance of 100 and random payment inputs.

const result = check(property([gen.intWithin(0, 100)], x => {
  return handlePayment(100, x) === 100 - x;
}))

console.log(result)

The result output shows clearly that a case has failed.

{ result: false,
  'failing-size': 22,
  'num-tests': 23,
  fail: [ 100 ],
  shrunk: 
   { 'total-nodes-visited': 7,
     depth: 0,
     result: false,
     smallest: [ 100 ] } }

If we take a closer look, we’ll find a shrunk property. This is a powerful feature that most generative testing libraries encompass. Instead of returning a large dataset of cases, where maybe one or a couple of cases failed, testcheck-js will try to find the smallest failing test (see the smallest property inside shrunk) as soon as one case fails. In this specific case the smallest value that failed is 100. This gives us very specific data to find out if the problem is inside the predicate function verifying our handlePayment or in handlePayment itself or if the dataset we generated isn’t explicit enough.

The dataset set should be fine. Let’s check the handlePayment function.

Obviously the case where balance and payment might be equal, in this failing case it’s [100, 100], isn’t handled properly. We verified that balance should return balance when payment is larger than the balance but didn’t cover this specific case. Let’s fix updatePayment.

const handlePayment = (balance, payment) =>
  (!payment || balance - payment < 0 // fix. only less than zero
  ) ? balance : balance - payment

This will the solve the problem.

Let’s verify that paymentTransaction can handle a negative balance as well as a negative payment.

const { strictNegInt : strNegInt } = gen

const result = check(property([strNegInt, strNegInt], (x, y) => {
  return handlePayment(x, y) === x - y;
}), {seed: 50})

console.log(result)

If you take a closer look at our new property, we added an options object, where we defined a seed, to be able to reproduce the generated tests. (Thanks Sune Simonsen for highlighting the fact)

Running the newly added test results in a new case failing.

{ result: false,
  'failing-size': 1,
  'num-tests': 2,
  fail: [ -2, -1 ],
  shrunk: 
   { 'total-nodes-visited': 1,
     depth: 0,
     result: false,
     smallest: [ -2, -1 ] } }

A balance of-2 and a payment of -1 would have a positive outcome on the balance. We need to refactor handlePayment again.

const handlePayment = (balance, payment) =>
  (!payment ||
  (balance <= 0 && balance - payment < balance) ||
  (balance > 0 && balance - payment < 0)) ?
    balance : balance - payment

Running the test again finally results in all cases passing.

{ result: true, 'num-tests': 100, seed: 50 }

Let’s refactor the handlePayment one last time, just to make it more readable.

const handlePayment = (balance, payment) =>
  payment &&
  ((balance <= 0 && balance - payment > balance) ||
  balance - payment >= 0) ?
     balance - payment : balance

The tests still verify the property.

{ result: true, ‘num-tests’: 100, seed: 50 }

The two failed cases can be added to the previously defined unit tests.

We haven’t covered how to integrate testcheck-js with mocha or jasmine. This is out of scope for this writeup but there are specific mocha and jasmine testcheck-js wrappers available, see mocha-check and jasmine-check for more information on the topic. jsverify also has integration for mocha and jasmine.