test(k8s): assert complete log messages in deployment tests - #218
Open
fedemaleh wants to merge 6 commits into
Open
test(k8s): assert complete log messages in deployment tests#218fedemaleh wants to merge 6 commits into
fedemaleh wants to merge 6 commits into
Conversation
…sertions-k8s-deployment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Applies the full-message assertion convention to every test file in
k8s/deployment/tests/. Follow-up to the review comment on #216:19 files, 5 commits, 0 production code changed. Full
k8ssuite passes:make test-unit MODULE=k8s→ exit 0, "All BATS tests passed!"What changed
Every assertion on a log/message became an assertion of the complete message — emoji, indentation, trailing parentheticals, and for errors the entire
💡 Possible causes:/🔧 How to fix:blocks. Where$outputis exactly one message,assert_equal; where it sits inside a larger trace,assert_containswith the whole block via a quoted heredoc.The suite got smaller while covering more. Fragment assertions collapse: several checks against one message become one check of that message.
print_failed_deployment_hintsassert_equalvalidate_alb_target_group_capacityverify_ingress_reconciliationverify_http_route_reconciliationroute_trafficWhat it found
Product text that had no test at all, because assertions checked the line that identifies an error and never the lines that help the customer:
verify_http_route_reconciliationleaks a raw jq error to the customer. On the timeout path with emptyparents,📋 Current conditions:printsjq: error (at <stdin>:1): Cannot iterate over null (null)instead of the conditions. The customer is already looking at a route-reconciliation timeout and the diagnostic that should help them is a jq exception. This one is a real bug, not a wording issue — worth its own ticket. Not fixed here: fixing it changes the output and therefore the assertions in this PR.route_trafficnever tested gomplate's stderr. The failure test asserted "render failed" but nottemplate: template.yaml:5: function 'undefined' not defined— the entire diagnostic value of that error.notify_active_domainsonly tested the first of two domains.npfails for both and the loop continues, so the real output has a full error block per domain plus a completion line. None of it was asserted.✅ Single target group on listener port 443without(traffic is no longer split between deployments);✅ Weights matchwithout(GREEN: X, BLUE: Y);✅ HTTPRoute successfully reconciledwithout(Accepted: True, ResolvedRefs: True); a missing⏭️emoji; and five❌lines missing the literal 3-space indent the scripts emit. Each of those was deletable with a green suite.validate_alb_target_group_capacity: 11 tests asserted only the dynamic header of messages that also carry💡/🔧blocks, so those suggestion blocks were entirely uncovered. Two tests were missing whole sub-sections.apply_templatessources the realk8s/backup/backup_templates, which emits📋 Manifest backup is disabled, skippingin every non-dry-run test — never asserted.Mock fidelity
Strengthening an assertion does not validate a mock — it promotes the mock's behaviour to specification. Three mocks did not model real
kubectl, and full-output assertions would have cemented them:kill_instance—kubectl get podalways returned 0, so the success path logged⚠️ Pod still exists after deletion attempt (status: Running). The full-output assertion would have declared that warning expected on a successful kill, meaning the test would pass even if the product genuinely failed to delete the pod. Fixed: the mock now models a pod that is gone afterdeletesucceeds, disambiguated by call order (the two bareget podlookups are atkill_instance:66pre-delete and:104post-delete). Applied to all three mocks in the file, including one that was latent.delete_cluster_objects— the mock echoedkubectl delete $*while$1was alreadydelete, producingkubectl delete delete deployment,.... Fixed tokubectl $*.delete_ingress_finalizer— the mock echoes its own invocation where realkubectl patchprints<resource> patched. Not fixed, and deliberately not baked in: the assertions were left un-merged across that line rather than asserting fiction. Follow-up.Scope
k8s/deployment/tests/only.k8s/scope(678 assertions),k8s/diagnose(380),k8s/backup(42) andk8s/utils(16) are separate PRs.Three files in this directory have zero
assert_containsand were audited rather than assumed:ingress_template_shape.batsgreps.tplsource text and never runs anything;wait_blue_deployment_active.batsandbuild_blue_deployment.batstest wrapper scripts that contain nologcalls at all. None have message assertions. Theassert_containscount used to scope this work did not miss anything here.Not in this PR
delete_ingress_finalizermock.⚠️two spaces vs⚠️one) across scripts;-versus•bullets between the gateway scripts and the rest;translate_probe_message's fallback returning a bare"Startup probe failed"that repeats its own input and surfaces to the customer as"...health check at /health. Detected: Startup probe failed."; a missing sentence-final period on onediagnose_failurebranch where every sibling branch has one; the certificate-error🔧 How to fix:giving no concrete commands while the timeout block does.kubectl/awsargv,jqvalues, rendered YAML, mock invocation logs — left untouched throughout.