docs(examples): add ML training and CI pull-request examples

.cspell.json

@@ -52,6 +52,14 @@
         "stdenv",
         "demisto",
         "toomanyrequests",
-        "envrc"
+        "envrc",
+        "MNIST",
+        "torchvision",
+        "makedirs",
+        "optim",
+        "argmax",
+        "randperm",
+        "shutil",
+        "pytest"
     ]
 }

README.md

@@ -14,12 +14,14 @@ You are encouraged to fork the repo and experiment with adjusting the workflows
 
 ## Examples
 
-| Example Name                                               | Description                                                                                                                                |
-|------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
-| [cronworkflow-example]( examples/cronworkflow-example/ )   | Runs a basic CronWorkflow.                                                                                                                 |
-| [dag-diamond]( examples/dag-diamond/ )                     | Runs a basic DAG Workflow.                                                                                                                 |
-| [external-logs]( examples/external-logs/ )                 | Workflows deploys a kubernetes job. You can see the logs from the job within Pipekit.                                                      |
-| [fan-out-fan-in]( examples/fan-out-fan-in/ )               | Shows how S3 artifact processing can be parallelized with Argo Workflows using a fan-out approach.                                         |
-| [get-versions](examples/get-versions/)                     | A workflow that outputs the versions of software installed in a Pipekit free trial cluster. Available as both a Hera and a native Workflow |
-| [hera-coinflip](examples/hera-coinflip/)                   | Runs a basic coinflip example using the Hera Python framework.                                                                             |
-| [hera-notebook-forecast](examples/hera-notebook-forecast/) | Press-play data job from a Jupyter notebook: aggregate energy demand and forecast the next day. Shows a platform abstraction over Hera.    |
+| Example Name                                               | Description                                                                                                                                           |
+| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [ci-pr-check](examples/ci-pr-check/)                       | A pull-request check pipeline: checkout, then lint and unit tests in parallel, with a combined status on exit. Runs in the free trial cluster.        |
+| [cronworkflow-example](examples/cronworkflow-example/)     | Runs a basic CronWorkflow.                                                                                                                            |
+| [dag-diamond](examples/dag-diamond/)                       | Runs a basic DAG Workflow.                                                                                                                            |
+| [external-logs](examples/external-logs/)                   | Workflows deploys a kubernetes job. You can see the logs from the job within Pipekit.                                                                 |
+| [fan-out-fan-in](examples/fan-out-fan-in/)                 | Shows how S3 artifact processing can be parallelized with Argo Workflows using a fan-out approach.                                                    |
+| [get-versions](examples/get-versions/)                     | A workflow that outputs the versions of software installed in a Pipekit free trial cluster. Available as both a Hera and a native Workflow            |
+| [hera-coinflip](examples/hera-coinflip/)                   | Runs a basic coinflip example using the Hera Python framework.                                                                                        |
+| [hera-notebook-forecast](examples/hera-notebook-forecast/) | Press-play data job from a Jupyter notebook: aggregate energy demand and forecast the next day. Shows a platform abstraction over Hera.               |
+| [pytorch-fashion-mnist](examples/pytorch-fashion-mnist/)   | A PyTorch training pipeline: prep, parallel train, and register the best model. Authored with the Argo Workflows Python SDK and as a native Workflow. |

examples/ci-pr-check/README.md

@@ -0,0 +1,47 @@
+[![Pipekit Logo](../../assets/images/pipekit-logo.png)](https://pipekit.io)
+
+# CI pull-request check
+
+A pull-request check pipeline that runs on Argo Workflows. It checks out a repository, runs lint and unit tests in parallel, and reports a combined pass or fail status.
+
+```text
+checkout  ->  lint + unit-test (in parallel)
+onExit:   report
+```
+
+- `checkout` clones the repository and passes the working tree to the next steps as an artifact.
+- `lint` and `unit-test` run in parallel, each against the checked-out code.
+- `report` is an exit handler, so it runs whether the checks pass or fail. It prints the overall result and, if given a GitHub token, posts a commit status.
+
+This is a simplified, self-contained version of the CI that builds this examples repository (see the `ci/` directory). It runs inside the free trial cluster with no secrets: by default it checks the public `pipekit/examples` repository. The full build, test, and deploy pipeline, including multi-cluster promotion, is described in the Pipekit docs under [CI/CD](https://docs.pipekit.io/use-cases/ci-cd).
+
+## Log into Pipekit via the CLI
+
+With the [CLI installed](https://docs.pipekit.io/reference/cli), log in once:
+
+```bash
+pipekit login
+```
+
+## Run the Workflow
+
+```bash
+pipekit submit -w --cluster-name=free-trial-cluster --pipe-name=ci-pr-check-example examples/ci-pr-check/workflow.yaml
+```
+
+## Check your own repository
+
+Override the parameters to point at your project and its commands:
+
+```bash
+pipekit submit -w --cluster-name=free-trial-cluster --pipe-name=ci-pr-check-example \
+  -p repo_url=https://github.com/your-org/your-repo.git \
+  -p ref=main \
+  -p lint_cmd="pip install --quiet ruff && ruff check ." \
+  -p test_cmd="pytest" \
+  examples/ci-pr-check/workflow.yaml
+```
+
+## Post a real commit status
+
+The `report` step posts a GitHub commit status when three values are present in the pod: `GITHUB_TOKEN`, `GITHUB_REPOSITORY` (as `owner/repo`), and `GIT_SHA`. Add the token as a [secret on the pipe](https://docs.pipekit.io/using-pipekit/pipes/edit/secrets) and pass the other two as parameters or environment values. Without them the step prints the status it would post, so the example still runs with no setup.

examples/ci-pr-check/workflow.yaml

@@ -0,0 +1,131 @@
+---
+# A pull-request check pipeline as a single Argo Workflow:
+#
+#   checkout  ->  lint + unit-test (in parallel)
+#   onExit:   report  (posts a combined pass/fail status)
+#
+# checkout clones a repository and passes the code to the next steps as an artifact.
+# lint and unit-test run in parallel. An exit handler reports the overall result, so the
+# status reflects the whole run whether it passed or failed.
+#
+# This is a simplified, self-contained version of the CI that builds this examples
+# repository (see ci/ci.yaml). It runs inside the Pipekit free trial cluster with no
+# secrets: by default it checks the public pipekit/examples repository. The full
+# build, test, and deploy pipeline is described in the Pipekit docs under CI/CD.
+apiVersion: argoproj.io/v1alpha1
+kind: Workflow
+metadata:
+  generateName: ci-pr-check-
+  namespace: argo
+spec:
+  entrypoint: main
+  onExit: report
+  serviceAccountName: argo-workflow
+  arguments:
+    parameters:
+      - name: repo_url
+        value: "https://github.com/pipekit/examples.git"
+      - name: ref
+        value: "main"
+      # The command each check runs, from the repository root. Override these to point
+      # at your own project's linter and tests.
+      - name: lint_cmd
+        value: "pip install --quiet ruff && ruff check ."
+      - name: test_cmd
+        value: "python examples/hera-notebook-forecast/test_demand_forecast.py"
+  templates:
+    - name: main
+      dag:
+        tasks:
+          - name: checkout
+            template: checkout
+          - name: lint
+            template: check
+            depends: checkout
+            arguments:
+              parameters:
+                - name: title
+                  value: "lint"
+                - name: command
+                  value: "{{workflow.parameters.lint_cmd}}"
+          - name: unit-test
+            template: check
+            depends: checkout
+            arguments:
+              parameters:
+                - name: title
+                  value: "unit-test"
+                - name: command
+                  value: "{{workflow.parameters.test_cmd}}"
+
+    # checkout: shallow-clone the repository and pass the working tree on as an artifact.
+    - name: checkout
+      container:
+        image: python:3.11-slim
+        command: [bash, -c]
+        args:
+          - |
+            set -e
+            apt-get update -qq && apt-get install -y -qq git >/dev/null
+            git clone --depth 1 --branch {{workflow.parameters.ref}} {{workflow.parameters.repo_url}} /tmp/src
+            echo "checked out {{workflow.parameters.repo_url}} at {{workflow.parameters.ref}}"
+        resources:
+          requests: { cpu: "250m", memory: 256Mi, ephemeral-storage: 256Mi }
+          limits: { cpu: "500m", memory: 512Mi, ephemeral-storage: 1Gi }
+      outputs:
+        artifacts:
+          - name: source
+            path: /tmp/src
+            s3:
+              key: "{{workflow.uid}}/source"
+
+    # check: run one command against the checked-out code. Reused by lint and unit-test.
+    - name: check
+      inputs:
+        parameters:
+          - name: title
+          - name: command
+        artifacts:
+          - name: source
+            path: /tmp/src
+            s3:
+              key: "{{workflow.uid}}/source"
+      container:
+        image: python:3.11-slim
+        workingDir: /tmp/src
+        command: [bash, -c]
+        args:
+          - |
+            set -e
+            echo "=== {{inputs.parameters.title}} ==="
+            {{inputs.parameters.command}}
+        resources:
+          requests: { cpu: "250m", memory: 256Mi, ephemeral-storage: 256Mi }
+          limits: { cpu: "1", memory: 768Mi, ephemeral-storage: 1Gi }
+
+    # report: exit handler. Prints the final status of the whole run. If a GitHub token
+    # is provided as the GITHUB_TOKEN secret on the pipe, it also posts a commit status;
+    # without one it prints the status it would post, so the example runs with no setup.
+    - name: report
+      container:
+        image: python:3.11-slim
+        command: [bash, -c]
+        args:
+          - |
+            set -e
+            STATUS="{{workflow.status}}"
+            if [ "$STATUS" = "Succeeded" ]; then STATE="success"; else STATE="failure"; fi
+            echo "pull-request check finished: $STATUS -> commit status '$STATE'"
+            if [ -n "${GITHUB_TOKEN:-}" ] && [ -n "${GITHUB_REPOSITORY:-}" ] && [ -n "${GIT_SHA:-}" ]; then
+              apt-get update -qq && apt-get install -y -qq curl >/dev/null
+              curl -sS -X POST \
+                -H "Authorization: Bearer ${GITHUB_TOKEN}" \
+                -H "Accept: application/vnd.github+json" \
+                "https://api.github.com/repos/${GITHUB_REPOSITORY}/statuses/${GIT_SHA}" \
+                -d "{\"state\":\"${STATE}\",\"context\":\"pipekit/ci\"}"
+            else
+              echo "GITHUB_TOKEN not set; skipping the GitHub commit status post"
+            fi
+        resources:
+          requests: { cpu: "100m", memory: 128Mi, ephemeral-storage: 128Mi }
+          limits: { cpu: "250m", memory: 256Mi, ephemeral-storage: 256Mi }

examples/pytorch-fashion-mnist/README.md

@@ -0,0 +1,88 @@
+[![Pipekit Logo](../../assets/images/pipekit-logo.png)](https://pipekit.io)
+
+# PyTorch Fashion-MNIST training pipeline
+
+A small machine-learning pipeline that trains an image classifier on Argo Workflows and picks the best model. It runs inside the Pipekit free trial cluster.
+
+The pipeline has four stages as one DAG:
+
+```text
+prep  ->  train (x3, in parallel)  ->  register
+```
+
+- `prep` downloads the Fashion-MNIST dataset once and saves it as an artifact.
+- `train` fans out over three hyperparameter configs. Each pod trains a small PyTorch model on the CPU and writes its model and metrics as a candidate artifact.
+- `register` reads every candidate, picks the most accurate, and writes the winning model and a model card to the artifact store.
+
+This is the standard "data prep, train, evaluate, register" shape, scaled down to fit the free trial cluster's resource limits. The training logic is generic, so you can swap in your own model and dataset.
+
+## Files
+
+- `workflow.py`: the pipeline authored with the Argo Workflows Python SDK. Data and ML teams usually author in Python, so this is the primary path.
+- `workflow.yaml`: the same DAG as a plain manifest, for the GitOps path or the CLI.
+- `model.py`: the model and training logic as plain PyTorch functions, the tested source of truth. The cluster steps mirror this logic.
+- `test_model.py`: a local test of that logic. Runs in milliseconds, no cluster.
+
+## PyTorch on the free trial cluster
+
+PyTorch is installed at run time from `download.pytorch.org` (the CPU-only wheel), so the only Docker Hub image is `python:3.11-slim`. The free trial cluster shares one egress IP, and pulling large images from Docker Hub there can hit the anonymous rate limit. Fetching the wheel from pytorch.org avoids that.
+
+The `torch_index` workflow parameter sets the pip index. Point it at an internal mirror to run offline or on a locked-down cluster.
+
+## Log into Pipekit via the CLI
+
+With the [CLI installed](https://docs.pipekit.io/reference/cli), log in once:
+
+```bash
+pipekit login
+```
+
+## Run it with the Python SDK
+
+From the repo root:
+
+```bash
+pip install 'hera' 'pipekit-sdk>=7.1.0'
+python examples/pytorch-fashion-mnist/workflow.py
+```
+
+The SDK package installs as `hera` today. The script submits the pipeline, prints a link to watch it live in the Pipekit UI, and exits. Set `PIPEKIT_CLUSTER` to target a cluster other than `free-trial-cluster`. Print the manifest without submitting with `RENDER_ONLY=1 python workflow.py`.
+
+## Run the native Workflow
+
+```bash
+pipekit submit -w --cluster-name=free-trial-cluster --pipe-name=fashion-mnist-example examples/pytorch-fashion-mnist/workflow.yaml
+```
+
+## What you get
+
+The run finishes in a few minutes. Each `train` pod logs its test accuracy, for example:
+
+```text
+train config 0: lr=0.05 hidden=64 accuracy=0.6280
+train config 1: lr=0.1 hidden=128 accuracy=0.6955
+train config 2: lr=0.2 hidden=256 accuracy=0.7290
+```
+
+The `register` step selects the most accurate model and exposes it as an output parameter, `selected`, so the choice is visible in the run graph even after the pods are cleaned up:
+
+```json
+{
+  "framework": "pytorch",
+  "dataset": "fashion-mnist",
+  "selected": { "idx": "2", "lr": 0.2, "hidden": 256, "accuracy": 0.723 }
+}
+```
+
+## Test the logic locally
+
+```bash
+pip install torch
+python examples/pytorch-fashion-mnist/test_model.py
+```
+
+The model and training functions are plain PyTorch, so you can test them in milliseconds before the job ever touches the cluster.
+
+## Scaling up
+
+The free trial version keeps the model and dataset small so it fits the trial cluster's limits. For the production shape of this pipeline, including a model registry and GPU training, see [ML Pipelines](https://docs.pipekit.io/use-cases/ml-pipelines) in the Pipekit docs.

examples/pytorch-fashion-mnist/model.py

@@ -0,0 +1,57 @@
+"""Pure PyTorch logic for the Fashion-MNIST classifier. No Argo, no cluster.
+
+These functions are the tested source of truth (see ``test_model.py``). The workflow
+steps in ``workflow.yaml`` and ``workflow.py`` mirror this logic so the same maths runs
+on the cluster. This is the same split the ``hera-notebook-forecast`` example uses:
+plain, tested Python here, and the cluster step inlines an equivalent.
+"""
+
+import torch
+from torch import nn
+
+CLASSES = 10
+PIXELS = 28 * 28
+
+
+def build_model(hidden):
+    """A small multi-layer perceptron: 784 inputs, one hidden layer, 10 outputs.
+
+    Big enough to learn Fashion-MNIST on a CPU, small enough to train in seconds.
+    """
+    return nn.Sequential(
+        nn.Flatten(),
+        nn.Linear(PIXELS, hidden),
+        nn.ReLU(),
+        nn.Linear(hidden, CLASSES),
+    )
+
+
+def train(model, images, labels, lr, epochs=1, batch_size=128, seed=0):
+    """Train the model in place with plain SGD and return it.
+
+    ``images`` is a float tensor shaped [N, 1, 28, 28] scaled to 0-1. ``labels`` is a
+    long tensor shaped [N]. The seed fixes the batch order so a given config is
+    reproducible.
+    """
+    torch.manual_seed(seed)
+    optimizer = torch.optim.SGD(model.parameters(), lr=lr)
+    loss_fn = nn.CrossEntropyLoss()
+    sample_count = images.shape[0]
+    model.train()
+    for _ in range(epochs):
+        order = torch.randperm(sample_count)
+        for start in range(0, sample_count, batch_size):
+            batch = order[start : start + batch_size]
+            optimizer.zero_grad()
+            loss = loss_fn(model(images[batch]), labels[batch])
+            loss.backward()
+            optimizer.step()
+    return model
+
+
+def accuracy(model, images, labels):
+    """Return the fraction of correct predictions on the given set."""
+    model.eval()
+    with torch.no_grad():
+        predictions = model(images).argmax(dim=1)
+        return float((predictions == labels).float().mean())