Add Test.RunnerV2 module

[lydell]

This deprecates the Test.Runner module and adds Test.RunnerV2. This is a backwards-compatible change.

node-test-runner PR: rtfeldman/node-test-runner#686. The idea is to require a recent enough version of elm-explorations/test in node-test-runner (updating the package in projects is a no-brainer), but keep compatibility in this package, so we don’t leave elm-test-rs behind.

Changes in Test.RunnerV2:

• Removed seed distribution. All fuzz tests now run with the same seed. This means that moving a test, adding another fuzz test, or commenting out a fuzz test no longer can cause fuzz tests to behave differently, even though you passed a fixed seed. This also got rid of a lot of complexity.
• Unit tests and fuzz tests are now returned separately. This allows a runner to more cleverly distribute tests across threads. For example, spread the fuzz tests on multiple threads, while running all unit tests single threaded. This also allowed for more precise types. Previously it looked like you could get a DistributionReport for unit tests, while in practice you can’t, for example.
• Make it possible for runners to collect Debug.log from the run that caused a fuzz test to fail (and ignore Debug.log from exploratory runs).
• Make it possible for runners to run fuzz tests with the “fuzzer ints” (RandomRun) from a previous failure, allowing for an instant reproduction of a fuzz test failure.

[jfmengels]

Looks very good to me, great work!

I left a bunch of improvement suggestions, but none are blocking.

[jfmengels]

Could this function not return the result without throwing it away? The client of the API is here forced to call this function as if it's an explicit side-effect, which feels odd in Elm.

I imagine that the alternative is to return a value that is potentially a test pass, which is odd as well as the client would have to handle an impossible branch. Or could we potentially return a FailData (or whatever information we want to yield for a failure)?

[lydell]

It could, yes. But what would it be used for? We already have all the data that it would return. And as you say, it allows for impossible cases:

• It could somehow return a pass.
• It could somehow fail to even generate a given value.

It does feel odd to have a side-effect function like this in Elm. But on the other hand, it is all about side effects: Debug.logs!

[lydell]

I found this PR: #200

It made me realize that it’s possible to do stuff with Kernel code in this package, rather than doing ugly string replacements on the output in runners.

Maybe I can come up with a way that running a test includes the debug logs as data. Then it’s no need to expose this weird side-effect function.

[jfmengels]

Is wrapWithTryCatch mostly a rename of what was done before, in order to make it clearer what's happening?

[lydell]

Previously, the thunks weren’t “stored with a try-catch wrapper” (like I changed them to) – instead, when running a test thunk that happened in a try-catch. This doesn’t work with my new, more detailed fuzz test failures, though. Catching errors so late means that it’s much more trouble constructing a FuzzTestFailData. Whereas if we wrap early we get everything for free.

(Not sure if this explanation made sense at all.)