Shubh/misc tasks - #166
Open
ShubhamRasal wants to merge 10 commits into
Open
Conversation
…-9_-]) Explicit bad input exits at startup. The hostname-derived default name is sanitized instead, since ".local" suffixes and spaces are normal there. Invalid values from the server are logged and ignored, not fatal.
Safety net before extracting the upload behind an interface. Covers the presign request shape, headers set verbatim on the PUT, gzip round-trip, max_bytes, PUT status handling, temp-file cleanup, and that a transport failure never leaks the signed URL's HMAC query.
Destinations() returns a slice so an agent can ship the same log to PDCP-managed storage and to customer-owned storage at once. The orchestrator gzips once and fans out; a failing destination does not stop the others. No change to the presign or PUT wire behavior. Log messages move from the "nuclei scan:" prefix to "scan-log:", and the upload line reports dest and location instead of object_path.
Agents silently pinned themselves to whatever template release they booted with. Three causes stacked: - InitNucleiProcess called DisableUpdateCheck(), a one-way global that short-circuits NeedsTemplateUpdate(), so every per-scan refresh was a no-op. Removed. - Freshness compared the installed version against nuclei's cached "latest", which the SDK refreshes only at first engine init, after the code that could act on it has run. An agent stale by three releases therefore reported "up to date". - Nothing verified files on disk, so a version-current set with missing templates was never repaired. Missing templates were skipped with an ERROR log while the scan reported success, so a vuln scan could silently omit checks. A chunk now fails when requested templates are absent after a repair; set PDCP_ALLOW_MISSING_TEMPLATES=true to scan with an incomplete set. runtools now resolves the newest release from GitHub (15-minute TTL), updates incrementally, falls back to a full reinstall when an update does not land, and stats the paths a chunk actually requested. Reinstall moves the old directory aside so a failed download keeps the agent usable. Private templates are absolute temp paths and are not verified.
An unreachable or rate-limited GitHub must not ground a fleet whose templates are already on disk. templateBootFatal separates "no usable set" (fatal) from "could not confirm the newest release" (warn and continue), and a failed lookup no longer discards the last tag it resolved, so agents still update toward it during an outage. Also: - drop templateBootTimeout: ctx reaches only the release lookup, which the client already caps at 30s, and the download is not ctx-aware - treat 403/429 carrying Retry-After as rate limiting; secondary limits leave X-RateLimit-Remaining untouched - unexport reinstallTemplates, which is only safe under templateMu
Chunk-level repair let every chunk of a scan wipe and re-download the shared template directory. A template the platform requests that no release carries made this unbounded: eight chunks produced eight full reinstalls, each one renaming the directory out from under whatever sibling chunks were loading templates. Those chunks then loaded a partial set and reported a clean scan, because the walk ignores I/O errors, the loader only logs unresolved paths, and the SDK drops the load error. Chunks now verify and report; installs happen only in the scan-level refresh, before any chunk launches: - VerifyTemplatesFor replaces EnsureTemplatesFor and never installs. A chunk with missing templates queues a repair and fails, or degrades when PDCP_ALLOW_MISSING_TEMPLATES is set. - RequestTemplateRepair is negative-cached per release: reinstalling the same release cannot produce a template it does not contain. - installFreshSet downloads into a sibling directory and swaps it in, so the live directory holds a complete set at all times and the previous one survives a failed download. An empty download is refused. Nuclei's configured path is written back after the staged install, or the next boot would look for a directory that no longer exists. - templateRW replaces templateMu: RunNuclei holds the read lock across engine setup and template load, so a swap cannot land mid-load. RunNuclei now loads templates explicitly and surfaces the error the SDK discards.
MissingTemplates approximated nuclei's resolver with os.Stat, so the two disagreed on any entry the resolver expands. A glob such as http/cves/2024/*.yaml was reported missing while nuclei resolves it to every matching template: the chunk hard-failed and queued a reinstall that could not help. disk.NewCatalog(dir).GetTemplatesPath is the same call the loader makes and returns the per-definition error map, so resolution now matches the scan exactly. Cost is 114ms for 7000 definitions against 17ms for the stat loop, once per chunk. Definitions that stay unresolved are still reported, but only relative ones queue a repair: private templates are absolute paths in a per-chunk temp dir, and reinstalling the public set cannot produce one.
Exercises the download-into-a-sibling-then-swap path against the actual template directory, which unit tests cannot reach: stubbing the config writer is what stops them rewriting nuclei's real config file. Asserts the swap keeps a full set, leaves no staging or retired directory, and that .templates-config.json still names the live path rather than the directory the install just deleted. Verified on darwin: 13617 -> 13619 templates in 5.6s, config intact.
POST /v1/agents/in carried os, arch, id, name and agent_network but not the running build, so the platform could only learn an agent's version by broadcasting a health-check RPC. Every heartbeat now sends version, which gives the server a value to keep beside the agent record it already refreshes on that call. Query building moves into heartbeatQuery so the wire contract is testable without a round trip. The registration log line names the version too. Also documents PDCP_ALLOW_MISSING_TEMPLATES and GITHUB_TOKEN, both added earlier on this branch.
Two unlocked readers and one unlocked writer remained around the staged install, which repoints config.DefaultConfig.TemplatesDirectory while it downloads. Readers: VerifyTemplatesFor ran per chunk with no lock, and execute.go read TemplateDir() for an error message. Both could observe the staging path mid-download or the gap between the two renames, and conclude every template was missing. TemplateDir now takes the read lock, callers under the write lock use the unlocked templateDir, and MissingTemplates is unexported so the locking entry point is the only way in. Writer: removing DisableUpdateCheck let nuclei's engine init call UpdateIfOutdated, which writes the shared directory without taking templateRW. A concurrent chunk could load a half-updated set and report a clean scan. The disable is restored, and our own updates no longer use nuclei's updater at all: a staged install is the only path that writes, always under the write lock. A full install measures ~6s against ~8s for an incremental one, so nothing is lost. Also: document that GetNucleiDefaultTemplateDir's cache is safe only because ensureNucleiTemplates primes it before any repoint; correct the NucleiOptions.Templates comment, which described template IDs that are a separate nuclei option; merge two consecutive imports in name_test.go; ignore local run artifacts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.