Use Test.RunnerV2

README.md

@@ -156,12 +156,30 @@ Start the runner in watch mode. Your tests will automatically rerun whenever you
 
     elm-test --watch
 
+### --no-clear-console
+
+By default, the console is cleared before each run in watch mode, so you only see the latest information. If you don’t like this, turn it off with `--no-clear-console`.
+
+    elm-test --watch --no-clear-console
+
+### --unbuffered-logs
+
+elm-test collects all `Debug.log` output while executing a test, and displays it all once the test in question is finished. This way elm-test can print _which_ test the logs came from.
+
+If the function you are testing gets into an infinite loop, it means that your debug logs will never show up. Then it can be useful to have the logs print _immediately_ instead (at the loss of no longer being able to label which tests the logs came from). To avoid confusion, use [Test.only](https://package.elm-lang.org/packages/elm-explorations/test/latest/Test#only) to isolate your test, or pass `--workers 1` to run in single-threaded mode to avoid oddly mixed output:
+
+    elm-test --unbuffered-logs --workers 1
+
+For _failing_ fuzz tests, elm-test only prints `Debug.log` output from the run of the fuzz test that produced the failure, which is usually what you want to debug. (Earlier, passing runs of the function with different input is just noise). For _passing_ fuzz tests, elm-test _ignores_ your `Debug.log` calls (and instead displays a note about this). Let’s imagine you are debugging a failing fuzz test. After a while it finally passes. There is no longer a failing run, so which one should we pick logs from? All of them? But would you really like to see the screen fill with 100+ repetitions of your logs at that point? Probably not. But if you actually _do_ want to show logs from all runs, you can use `--unbuffered-logs` for this use case, too. Also remember that you can make the test fail from anywhere using `Debug.todo` – that’s also a way to make logs appear!
+
 ### --seed
 
 Run with a specific fuzzer seed, rather than a randomly generated seed. This allows reproducing a failing fuzz-test. The command needed to reproduce (including the `--seed` flag) is printed after each test run. Copy, paste and run it!
 
     elm-test --seed 336948560956134
 
+On top of that, if you run elm-test without the `--seed` flag, elm-test will automatically use the same seed as the last run if there was a fuzz test failure, letting you reproduce errors without doing anything. It even tries to fast-forward you through the fuzzing. So if it took some time for the fuzzer to find the problem the first time, the next run should be instant.
+
 ### --fuzz
 
 Define how many times each fuzz-test should run. Defaults to `100`.
@@ -173,16 +191,18 @@ Define how many times each fuzz-test should run. Defaults to `100`.
 
 ### --workers
 
-Choose how many workers elm-test should use to run tests in parallel. Defaults to the number of “logical CPU cores” of the machine you run the tests on.
+Choose how many workers elm-test should use to run fuzz tests in parallel. Defaults to the number of “logical CPU cores” of the machine you run the tests on.
 
     elm-test --workers 4
 
-Your computer might say that it has 12 logical CPU cores. Then dividing up the tests between 12 parallel workers is the theoretical optimum for running the tests as quickly as possible. But in practice your tests might run faster with just 4 workers in parallel due to overhead. Play around with it and see what is the fastest for your test suite on your computer!
+Your computer might say that it has 12 logical CPU cores. Then dividing up the fuzz tests between 12 parallel workers is the theoretical optimum for running the tests as quickly as possible. But in practice your tests might run faster with just 4 workers in parallel due to overhead. Play around with it and see what is the fastest for your test suite on your computer!
 
 To see the number of logical CPU cores on your machine, run `node -p "os.cpus().length"` (it’s also shown in `elm-test --help`).
 
 If you pass `--workers 1`, elm-test won’t even start a new thread for running the tests in – it’ll do everything in the main thread (single-threaded mode).
 
+Currently, elm-test always executes unit tests on the main thread, and only uses separate threads for fuzz tests. Unit tests tend to execute so fast that the overhead of threads isn’t worth it. But fuzz tests often run long enough to benefit from parallelization.
+
 ### --report
 
 Specify which format to use for reporting test results. Valid options are:

elm/elm.json

@@ -9,12 +9,12 @@
             "elm/core": "1.0.5",
             "elm/json": "1.1.3",
             "elm/random": "1.0.0",
-            "elm/time": "1.0.0",
             "elm-explorations/test": "2.2.1"
         },
         "indirect": {
             "elm/bytes": "1.0.8",
             "elm/html": "1.0.0",
+            "elm/time": "1.0.0",
             "elm/virtual-dom": "1.0.3"
         }
     },

elm/src/Test/Reporter/Console.elm

@@ -7,7 +7,6 @@ import Test.Reporter.Console.Format exposing (format)
 import Test.Reporter.Console.Format.Color as FormatColor
 import Test.Reporter.Console.Format.Monochrome as FormatMonochrome
 import Test.Reporter.TestResults as Results exposing (Failure, Outcome(..), SummaryInfo)
-import Test.Runner exposing (formatLabels)
 
 
 formatDuration : Float -> String
@@ -36,6 +35,23 @@ pluralize singular plural count =
     String.join " " [ String.fromInt count, suffix ]
 
 
+formatLabels :
+    (String -> Text)
+    -> (String -> Text)
+    -> List String
+    -> List Text
+formatLabels formatDescription formatTest labels =
+    case labels of
+        [] ->
+            []
+
+        test :: descriptions ->
+            List.foldl
+                (\x acc -> formatDescription x :: acc)
+                [ formatTest test ]
+                descriptions
+
+
 passedToText : List String -> String -> Text
 passedToText labels distributionReport =
     Text.concat
@@ -147,7 +163,7 @@ getStatus outcome =
 
 
 reportComplete : UseColor -> Results.TestResult -> Value
-reportComplete useColor { labels, outcome } =
+reportComplete useColor { labels, outcome, hasDebugLogs } =
     Encode.object <|
         ( "type", Encode.string "complete" )
             :: ( "status", Encode.string (getStatus outcome) )
@@ -156,10 +172,18 @@ reportComplete useColor { labels, outcome } =
                         -- No failures of any kind.
                         case distributionReportToString distributionReport of
                             Nothing ->
-                                []
+                                if hasDebugLogs then
+                                    [ ( "message"
+                                      , passedLabelsToText labels
+                                            |> textToValue useColor
+                                      )
+                                    ]
+
+                                else
+                                    []
 
                             Just report ->
-                                [ ( "distributionReport"
+                                [ ( "message"
                                   , report
                                         |> passedToText labels
                                         |> textToValue useColor

elm/src/Test/Reporter/JUnit.elm

@@ -84,9 +84,9 @@ formatClassAndName labels =
             ( "", "" )
 
 
-encodeDuration : Int -> Value
+encodeDuration : Float -> Value
 encodeDuration time =
-    (toFloat time / 1000)
+    (time / 1000)
         |> String.fromFloat
         |> Encode.string
 
@@ -119,6 +119,7 @@ encodeExtraFailure _ =
                   }
                 , NoDistribution
                 )
+        , hasDebugLogs = False
         }
 
 

elm/src/Test/Reporter/Json.elm

@@ -28,7 +28,11 @@ reportComplete { duration, labels, outcome } =
         , ( "labels", encodeLabels labels )
         , ( "failures", Encode.list identity (encodeFailures outcome) )
         , ( "distributionReports", Encode.list identity (encodeDistributionReports outcome) )
-        , ( "duration", Encode.string <| String.fromInt duration )
+
+        -- Keep the "duration" field Int for backwards compatibility,
+        -- and also expose the new Float field for more precision.
+        , ( "duration", Encode.string <| String.fromInt (round duration) )
+        , ( "durationFloat", Encode.string <| String.fromFloat duration )
         ]
 
 

elm/src/Test/Reporter/TestResults.elm

@@ -3,13 +3,9 @@ module Test.Reporter.TestResults exposing
     , Outcome(..)
     , SummaryInfo
     , TestResult
-    , isFailure
-    , outcomeFromExpectations
     )
 
-import Expect exposing (Expectation)
 import Test.Distribution exposing (DistributionReport)
-import Test.Runner
 import Test.Runner.Failure exposing (Reason)
 
 
@@ -22,7 +18,8 @@ type Outcome
 type alias TestResult =
     { labels : List String
     , outcome : Outcome
-    , duration : Int -- in milliseconds
+    , duration : Float -- in milliseconds
+    , hasDebugLogs : Bool
     }
 
 
@@ -40,38 +37,3 @@ type alias Failure =
     , description : String
     , reason : Reason
     }
-
-
-isFailure : Outcome -> Bool
-isFailure outcome =
-    case outcome of
-        Failed _ ->
-            True
-
-        _ ->
-            False
-
-
-outcomeFromExpectations : List Expectation -> Outcome
-outcomeFromExpectations expectations =
-    case expectations of
-        -- The type of test runner functions says that they return `List Expectation`,
-        -- but in practice they only ever return lists with exactly one item:
-        -- https://github.com/elm-explorations/test/pull/244
-        -- That PR was reverted because it unfortunately was a breaking change for the package:
-        -- https://github.com/elm-explorations/test/commit/11f70d5fc0b6fdc88d7a34ea1d10f56969890493
-        -- But to keep things simpler here, we only support exactly one expectation.
-        [ expectation ] ->
-            case Test.Runner.getFailureReason expectation of
-                Nothing ->
-                    Passed (Test.Runner.getDistributionReport expectation)
-
-                Just failure ->
-                    if Test.Runner.isTodo expectation then
-                        Todo failure.description
-
-                    else
-                        Failed ( failure, Test.Runner.getDistributionReport expectation )
-
-        _ ->
-            Debug.todo ("A test somehow did not return exactly 1 expectation, it returned " ++ String.fromInt (List.length expectations) ++ "!")