Skip to content

test(k8s): assert complete log messages in deployment tests - #218

Open
fedemaleh wants to merge 6 commits into
betafrom
chore/full-message-assertions-k8s-deployment
Open

test(k8s): assert complete log messages in deployment tests#218
fedemaleh wants to merge 6 commits into
betafrom
chore/full-message-assertions-k8s-deployment

Conversation

@fedemaleh

Copy link
Copy Markdown
Collaborator

Summary

Applies the full-message assertion convention to every test file in k8s/deployment/tests/. Follow-up to the review comment on #216:

siempre tenes que hacer un assert de mensajes completos (con emoji incluido) y en el caso de errores de la totalidad de los logs que indican el error, con las sugerencias y todo. Esto es porque los logs son la principal fuente de feedback con el cliente que usa esto.

19 files, 5 commits, 0 production code changed. Full k8s suite passes: make test-unit MODULE=k8s → exit 0, "All BATS tests passed!"

What changed

Every assertion on a log/message became an assertion of the complete message — emoji, indentation, trailing parentheticals, and for errors the entire 💡 Possible causes: / 🔧 How to fix: blocks. Where $output is exactly one message, assert_equal; where it sits inside a larger trace, assert_contains with the whole block via a quoted heredoc.

The suite got smaller while covering more. Fragment assertions collapse: several checks against one message become one check of that message.

before after
print_failed_deployment_hints 64 fragments 21 assert_equal
validate_alb_target_group_capacity 69 fragments 39 full blocks
verify_ingress_reconciliation 68 fragments 47 full blocks
verify_http_route_reconciliation 51 fragments 7 assertions
3 × gateway route_traffic 48 fragments 7 assertions

What it found

Product text that had no test at all, because assertions checked the line that identifies an error and never the lines that help the customer:

  • verify_http_route_reconciliation leaks a raw jq error to the customer. On the timeout path with empty parents, 📋 Current conditions: prints jq: error (at <stdin>:1): Cannot iterate over null (null) instead of the conditions. The customer is already looking at a route-reconciliation timeout and the diagnostic that should help them is a jq exception. This one is a real bug, not a wording issue — worth its own ticket. Not fixed here: fixing it changes the output and therefore the assertions in this PR.
  • route_traffic never tested gomplate's stderr. The failure test asserted "render failed" but not template: template.yaml:5: function 'undefined' not defined — the entire diagnostic value of that error.
  • notify_active_domains only tested the first of two domains. np fails for both and the loop continues, so the real output has a full error block per domain plus a completion line. None of it was asserted.
  • Silently untested suffixes and prefixes: ✅ Single target group on listener port 443 without (traffic is no longer split between deployments); ✅ Weights match without (GREEN: X, BLUE: Y); ✅ HTTPRoute successfully reconciled without (Accepted: True, ResolvedRefs: True); a missing ⏭️ emoji; and five lines missing the literal 3-space indent the scripts emit. Each of those was deletable with a green suite.
  • validate_alb_target_group_capacity: 11 tests asserted only the dynamic header of messages that also carry 💡/🔧 blocks, so those suggestion blocks were entirely uncovered. Two tests were missing whole sub-sections.
  • apply_templates sources the real k8s/backup/backup_templates, which emits 📋 Manifest backup is disabled, skipping in every non-dry-run test — never asserted.

Mock fidelity

Strengthening an assertion does not validate a mock — it promotes the mock's behaviour to specification. Three mocks did not model real kubectl, and full-output assertions would have cemented them:

  1. kill_instancekubectl get pod always returned 0, so the success path logged ⚠️ Pod still exists after deletion attempt (status: Running). The full-output assertion would have declared that warning expected on a successful kill, meaning the test would pass even if the product genuinely failed to delete the pod. Fixed: the mock now models a pod that is gone after delete succeeds, disambiguated by call order (the two bare get pod lookups are at kill_instance:66 pre-delete and :104 post-delete). Applied to all three mocks in the file, including one that was latent.
  2. delete_cluster_objects — the mock echoed kubectl delete $* while $1 was already delete, producing kubectl delete delete deployment,.... Fixed to kubectl $*.
  3. delete_ingress_finalizer — the mock echoes its own invocation where real kubectl patch prints <resource> patched. Not fixed, and deliberately not baked in: the assertions were left un-merged across that line rather than asserting fiction. Follow-up.

Scope

k8s/deployment/tests/ only. k8s/scope (678 assertions), k8s/diagnose (380), k8s/backup (42) and k8s/utils (16) are separate PRs.

Three files in this directory have zero assert_contains and were audited rather than assumed: ingress_template_shape.bats greps .tpl source text and never runs anything; wait_blue_deployment_active.bats and build_blue_deployment.bats test wrapper scripts that contain no log calls at all. None have message assertions. The assert_contains count used to scope this work did not miss anything here.

Not in this PR

  • The jq-error bug above, and the delete_ingress_finalizer mock.
  • Message-quality observations, listed for a separate pass: inconsistent emoji spacing (⚠️ two spaces vs ⚠️ one) across scripts; - versus bullets between the gateway scripts and the rest; translate_probe_message's fallback returning a bare "Startup probe failed" that repeats its own input and surfaces to the customer as "...health check at /health. Detected: Startup probe failed."; a missing sentence-final period on one diagnose_failure branch where every sibling branch has one; the certificate-error 🔧 How to fix: giving no concrete commands while the timeout block does.
  • Assertions on data rather than messages — kubectl/aws argv, jq values, rendered YAML, mock invocation logs — left untouched throughout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant