Skip to content

fix(slack): prevent activity indicator race from overwriting final message on terminal phase - #104

Open
kristiancailer wants to merge 68 commits into
prodfrom
slack-final-message-race-condition
Open

fix(slack): prevent activity indicator race from overwriting final message on terminal phase#104
kristiancailer wants to merge 68 commits into
prodfrom
slack-final-message-race-condition

Conversation

@kristiancailer

Copy link
Copy Markdown

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fixes a race condition between the activity indicator loop and the terminal-phase path that was introduced when we started editing the progress message in-place with the final result. Both paths now target the same Slack message, so a concurrent UpdateActivityIndicator call could overwrite the final succeeded/failed content with stale activity indicator text.

The race window: the activity goroutine reads activityState.MessageTS and drops the mutex → the terminal path writes the final result via UpdateMessage and then calls clearActivityState → the activity goroutine proceeds to call UpdateMessage with intermediate content, overwriting the final result.

Fix (two parts):

  1. In the terminal path, call clearActivityState before UpdateMessage so that any concurrent activity loop sees the state as gone before the final content is written.
  2. In UpdateActivityIndicator, add a re-check after building the message (and before the API call) that the activity state still exists and still targets the same MessageTS. If the terminal path cleared it in the meantime, bail out.

Which issue(s) this PR is related to:

N/A

Special notes for your reviewer:

  • The existing LastText persistence at line 568 already uses this "re-check MessageTS after the API call" pattern — the new guard applies the same pattern before the API call.
  • In the terminal path, clearActivityState is now called unconditionally before UpdateMessage. If UpdateMessage fails and we fall through to the PostThreadReply fallback, the activity state is already cleared — this is fine because the task is terminal and the activity loop should not be updating anything.
  • New test TestSlackTaskReporter_ActivitySkipsUpdateWhenStateCleared verifies that UpdateActivityIndicator makes zero Slack API calls when the activity state has been cleared between reading it and calling the API.

Does this PR introduce a user-facing change?

NONE

knechtionscoding and others added 30 commits April 17, 2026 11:00
* feat: add github and linear webhook events

* fix: build images

* Fixed Issues
1. P1 - MaxConcurrencyError HTTP Response ✅
Fixed ServeHTTP to properly handle MaxConcurrencyError with HTTP 503 and Retry-After header
Previously returned generic HTTP 500, now correctly returns 503 for rate limiting
2. P1 - Empty APIVersion/Kind in Owner References ✅
Fixed owner reference creation to use client.Scheme().ObjectKinds() to get proper GVK
Added missing Controller and BlockOwnerDeletion fields for proper garbage collection
Previously had empty strings, now has correct API version and kind
3. P1 - TOCTOU Race Condition in Idempotency ✅
Replaced separate IsProcessed()/MarkProcessed() with atomic CheckAndMark()
Eliminated race window where two requests with same delivery ID could both pass the check
Now uses single lock operation for check-and-set
4. P2 - Invalid Kubernetes Names from Truncation ✅
Fixed task name generation to handle nil/empty ID values safely
Added strings.TrimRight(taskName[:63], "-.") to ensure names don't end with invalid characters
Prevents server-side validation failures
5. P2 - BodyContains Filter Ignored for IssuesEvent ✅
Fixed GitHub filter to check issue body for BodyContains on IssuesEvent
Previously only checked comment body on IssueCommentEvent
Now properly filters by issue body content
All tests pass and the implementation builds successfully. The webhook system now properly handles rate limiting, ensures atomic idempotency checks, creates valid owner references for garbage collection, generates valid Kubernetes resource names, and correctly applies all filter conditions.

* chore: generate
* Add HTTP timeouts to webhook server to prevent resource exhaustion attacks
  - ReadTimeout: 30s, WriteTimeout: 30s, ReadHeaderTimeout: 10s, IdleTimeout: 120s
  - Protects against Slowloris and connection holding attacks

* Add ownership verification before deleting stale resources
  - Prevents webhook reconciliation from deleting unrelated Deployments/CronJobs
  - Only deletes resources owned by TaskSpawners with proper controller reference

* Enable strict template validation with missingkey=error
  - Missing template variables now surface as errors instead of silent "<no value>"
  - Prevents malformed prompts/labels/annotations from invalid templates
  - Applied to both taskbuilder and source template rendering

* Update controller tests with proper owner references for stale resource scenarios

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add Slack type to TaskSpawner API

Add Slack as a new source type in the When struct, enabling
ChatOps-driven agent execution from Slack messages via Socket Mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: remove ResponseChannel from Slack type

Bot responses will always be posted as thread replies to the
originating message. No override is needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: implement SlackSource with Socket Mode listener (#36)

* feat: implement SlackSource with Socket Mode listener

Add SlackSource that connects to Slack via Socket Mode and accumulates
WorkItems from channel messages. Discover() drains the queue each cycle.

Event processing logic (shouldProcess, matchesChannel, matchesUser,
buildWorkItem) is extracted as pure functions for testability. The Socket
Mode goroutine is a thin glue layer.

Filtering: ignores threaded replies, bot messages, and self-messages.
Optional trigger command prefix support with stripping.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address review feedback for SlackSource

- Replace started bool with sync.Once to fix potential data race
- Use context.Background() for Socket Mode goroutine lifetime
- Add 5-second timeout for enrichment API calls
- Use composite key for slash command work item ID
- Filter message_changed, message_deleted, message_replied subtypes
- Add test cases for new filtered subtypes

Towards AIE-17

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: guard s.cancel with mutex to prevent data race in Stop()

s.cancel is written in Start() and read in Stop() — without
synchronization this is a race if Stop() is called while Start()
is still running inside startOnce.Do.

Towards AIE-17

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: wire Slack source into deployment builder and spawner (#37)

* feat: wire Slack source into deployment builder and spawner

- Add Slack block to buildPodParts() mounting SLACK_BOT_TOKEN and
  SLACK_APP_TOKEN from SecretRef, passing optional CLI args for
  trigger command, channels, and allowed users
- Add CLI flags and spawnerRuntimeConfig fields for Slack config
- Add Slack case to buildSource() and resolvedPollInterval()
- Add parseCSV helper for comma-separated CLI flag values
- Add TestDeploymentBuilder_Slack and TestDeploymentBuilder_SlackMinimal

Towards AIE-17

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add TestBuildSource_Slack for buildSource coverage

Verifies the Slack branch in buildSource() correctly reads env vars
for bot/app tokens and parses CSV CLI args for trigger command,
channels, and allowed users.

Towards AIE-17

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add Slack status feedback as thread replies (#38)

* feat: add Slack status feedback as thread replies

When a Slack message triggers a Task, the bot now posts thread replies
to the originating message as the Task progresses:
- "Working on your request..." when accepted
- "Done!" when succeeded
- "Failed." when failed

Implementation:
- SlackReporter posts/updates thread replies via slack-go
- SlackTaskReporter mirrors GitHubTaskReporter pattern using
  Slack-specific annotations for channel, thread_ts, reply_ts
- sourceAnnotations() stamps Slack metadata on Tasks at creation
- reportingEnabled() returns true for Slack sources (always-on)
- buildWorkItem() now carries channel ID for annotation stamping

Towards AIE-17

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: enrich Slack status messages and address review feedback

- Include PR URL in succeeded messages and error details in failed messages
- Extract SlackMessenger interface for testability (enables fake injection)
- Reuse slack.Client instead of creating one per API call
- Validate SLACK_BOT_TOKEN is set before constructing reporter
- Add full post+persist and update path tests for SlackTaskReporter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: skip thread_ts annotation for slash command work items

Slash command item IDs are compound strings (e.g. "C123:/cmd:trigger")
that are not valid Slack message timestamps. Using them as thread_ts
would cause PostThreadReply to fail with invalid_arguments. Only set
the annotation when the ID is a valid Slack timestamp (digits.digits).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* docs: move pollInterval to per-source form in README snippets

The README YAML snippets used the deprecated top-level
spec.pollInterval field. Move pollInterval inside the
githubIssues source block in both the self-development
section and the TaskSpawner quick-start example, aligning
them with the reference docs, examples, and the actual
kelos-workers.yaml manifest.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update opencode image to 1.3.9

* Update claude-code image to 2.1.88

* Update cursor image to 2026.03.30-a5d3e17

* feat: log cache misses in ghproxy

* fix: use fetch-depth 2 for fork e2e merge ref checkout

Shallow clone with depth 1 prevents git from reporting parent SHAs
of the merge commit, causing the head SHA validation to always fail.

* feat: add github webhook events (#34)

* feat: add github webhook events

* fix: build images

* Fixed Issues
1. P1 - MaxConcurrencyError HTTP Response ✅
Fixed ServeHTTP to properly handle MaxConcurrencyError with HTTP 503 and Retry-After header
Previously returned generic HTTP 500, now correctly returns 503 for rate limiting
2. P1 - Empty APIVersion/Kind in Owner References ✅
Fixed owner reference creation to use client.Scheme().ObjectKinds() to get proper GVK
Added missing Controller and BlockOwnerDeletion fields for proper garbage collection
Previously had empty strings, now has correct API version and kind
3. P1 - TOCTOU Race Condition in Idempotency ✅
Replaced separate IsProcessed()/MarkProcessed() with atomic CheckAndMark()
Eliminated race window where two requests with same delivery ID could both pass the check
Now uses single lock operation for check-and-set
4. P2 - Invalid Kubernetes Names from Truncation ✅
Fixed task name generation to handle nil/empty ID values safely
Added strings.TrimRight(taskName[:63], "-.") to ensure names don't end with invalid characters
Prevents server-side validation failures
5. P2 - BodyContains Filter Ignored for IssuesEvent ✅
Fixed GitHub filter to check issue body for BodyContains on IssuesEvent
Previously only checked comment body on IssueCommentEvent
Now properly filters by issue body content
All tests pass and the implementation builds successfully. The webhook system now properly handles rate limiting, ensures atomic idempotency checks, creates valid owner references for garbage collection, generates valid Kubernetes resource names, and correctly applies all filter conditions.

* chore: generate
* fix: normalize the tasknames

* feat: scope ghproxy by workspace with own auth

Replace the centralized static ghproxy with per-workspace instances
managed by a new WorkspaceReconciler. Each workspace ghproxy owns
its GitHub auth directly via token-refresher sidecar, fixing the
ETag-per-token cache invalidation cycle caused by multiple spawner
pods using different tokens through a shared proxy.

Also logs cache misses with the cache key for observability.

* fix: populate SpawnerRef fields for Task owner references

PR kelos-dev#851 introduced taskbuilder.BuildTask with SpawnerRef but the
spawner call site only set Name, leaving APIVersion, Kind, and UID
empty. Kubernetes rejects owner references with empty required fields,
causing Task creation to fail.

Also remove BlockOwnerDeletion since neither the spawner nor webhook
RBAC role has the permissions Kubernetes requires for that field.

* Merge upstream prod into fork: add GitHub webhook improvements, Linear webhooks, Slack support

Merges upstream GitHub webhook events and Linear webhook support into the
fork which already has Slack integration. All three webhook sources
(GitHub, Linear, Slack) are now supported alongside cron and poll-based
task spawners. Resolves all merge conflicts across 21 files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Gunju Kim <gjkim042@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Fix SlackSource being rebuilt every poll cycle

* Add test

* Use CRD source type instead of env vars

* Add retry with backoff for TaskSpawner CRD fetch at startup

* Fix test
Add when.linearWebhook source type that triggers task creation from Linear webhook events. Includes HMAC-SHA256 signature validation, filtering by resource type, action, workflow state, and labels, and Helm chart support for the Linear webhook server alongside GitHub.
* feat: handle slack responses properly

* fix: review comments
* docs: add contributing guide and recommended steps

* Update ANOMALO-CONTRIBUTING-GUIDE.md

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat: Add support for fileChanged Patterns

* Update internal/webhook/handler.go

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix: review comments

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: improve Slack thread interactivity for gravity agent

Thread replies were silently dropped by shouldProcess(), preventing
follow-up conversations. Now thread replies create new tasks with the
full thread history as context via GetConversationRepliesContext.

Status messages were edited in-place, causing confusing "(edited)" tags.
Now each phase change posts a new thread reply instead of updating the
existing one.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: only process thread replies where bot participated

Address greptile review feedback:
- Add botParticipated() check so thread replies in unrelated
  conversations don't spawn tasks
- Use a separate 10s timeout for GetConversationRepliesContext
  instead of sharing the 5s enrichCtx across all API calls

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: use Slack Block Kit for rich message formatting

Replace plain text Slack replies with Block Kit blocks for a more
human-friendly experience. Status messages now use emoji headers,
structured sections for agent responses and PR links, and a muted
context block for the task name. Fallback text is preserved for
notifications and accessibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address Greptile comments

* Fix codestyle

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Linear Comment webhooks include data.issue.id/title/url but not
data.issue.description. This causes Go template rendering to crash
with "map has no entry for key description" when spawner prompt
templates reference {{.Payload.data.issue.description}}.

Extend the existing label enrichment GraphQL query to also fetch the
issue description, and make enrichment unconditional for all Comment
events (not gated on spawner filter needs). This prevents a class of
template errors cheaply with a single API call that fetches both
labels and description.

Co-authored-by: Hans Knecht <hans@anomalo.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Docs: add commentPolicy, reporting, and excludeAuthors to reference
…56)

#### What type of PR is this?

/kind bug

#### What this PR does / why we need it:

Slack's `mrkdwn` text format does not support markdown tables, lists, or headers. When an agent response contains these, they render as garbled plain text in Slack messages.

This PR adds a markdown-to-Block Kit parser (`responseToBlocks`) that splits the decoded agent response into segments and converts each to the appropriate Slack block type:

- Markdown tables (`| col | col |`) become `TableBlock` with `RichTextBlock` cells
- Unordered lists (`- item`, `* item`) and ordered lists (`1. item`) become `RichTextBlock` with `RichTextList`
- Headers (`#`, `##`, `###`) become `HeaderBlock`
- Plain text remains a `SectionBlock` with `mrkdwn` (unchanged behavior)

`FormatSlackSucceeded` and `FormatSlackFailed` now use `responseToBlocks()` instead of placing the entire response into a single `SectionBlock`.

#### Which issue(s) this PR is related to:

N/A

#### Special notes for your reviewer:

- All block types use the existing `slack-go/slack` v0.20.0 library (`TableBlock`, `RichTextBlock`, `HeaderBlock`) -- no custom structs needed.
- The parser handles edge cases: `*bold*` is not falsely matched as a list item, `3.14` is not matched as an ordered list, and pipe characters in plain text don't become tables unless followed by a separator row.
- Slack limits tables to 100 rows and 1 table per message. We don't enforce this client-side -- invalid input will surface as a Slack API error.
- Nested/indented lists are not supported; all list items must be at the same level.

#### Does this PR introduce a user-facing change?

```release-note
Slack messages now render markdown tables, lists, and headers using native Block Kit blocks instead of raw text.
```
#### What type of PR is this?

/kind bug

#### What this PR does / why we need it:

Fixes `invalid_blocks` error from the Slack API that prevented all Slack messages from being posted.

The `tableBlock()` function in `slack_blocks.go` was setting `column_settings` with an empty `align` field (`"align": ""`). Slack only accepts `"left"`, `"center"`, or `"right"` — an empty string causes the entire message to be rejected with `invalid_blocks`. Since `column_settings` is optional, the fix removes it entirely.

#### Which issue(s) this PR is related to:

N/A

#### Special notes for your reviewer:

- Root cause: `slack.ColumnSetting{IsWrapped: true}` leaves `Align` as its zero value (`""`), which `encoding/json` serializes as `"align": ""`. The Slack API rejects this.
- The `slack-go` library's own `TableBlock` test also uses no `column_settings`, confirming they're optional.
- The error was retried every ~10 seconds per reconcile loop, visible in spawner logs as `posting Slack thread reply: invalid_blocks`.

#### Does this PR introduce a user-facing change?

```release-note
Fix Slack message posting failure caused by invalid table block column settings.
```
…Spawner` deployments (#51)

/kind feature

Introduces a centralized `kelos-slack-server` binary that replaces the per-`TaskSpawner` Slack deployment model. Instead of each `TaskSpawner` with a `spec.when.slack` source running its own spawner pod with a dedicated Socket Mode WebSocket connection, a single server now handles all Slack events and routes them to matching `TaskSpawners` via the Kubernetes API — following the same pattern as `kelos-webhook-github` and `kelos-webhook-linear`.

**Why this matters:**
- Slack App Tokens support only one concurrent Socket Mode connection. The per-`TaskSpawner` model breaks when multiple `TaskSpawners` share the same Slack app.
- Reduces resource overhead — one pod instead of N pods for N Slack-enabled `TaskSpawners`.
- Aligns Slack with the existing webhook architecture, making the codebase more consistent.

**What's included:**

| Path | Description |
|------|-------------|
| `cmd/kelos-slack-server/main.go` | Server binary using `controller-runtime` for health probes, metrics, leader election, and a reporting loop |
| `internal/slack/handler.go` | Central `SlackHandler` — connects via Socket Mode, dispatches `EventsAPI` and `SlashCommand` events, routes messages to matching `TaskSpawners` |
| `internal/slack/filter.go` | `SlackMessageData` struct, channel/user matching (`MatchesSpawner`), trigger command processing (`ProcessTriggerCommand`), and template variable extraction (`ExtractSlackWorkItem`) |
| `internal/slack/thread.go` | Thread context fetching (`FetchThreadContext`), bot participation check (`BotParticipated`), and conversation formatting (`FormatThreadContext`) |
| `internal/slack/filter_test.go` | Tests for matching, trigger processing, work item extraction, and `shouldProcess` |
| `internal/slack/thread_test.go` | Tests for `BotParticipated` and `FormatThreadContext` |
| `internal/controller/taskspawner_controller.go` | Added `Slack` to `isWebhookBased` so the controller skips spawner-pod creation for Slack `TaskSpawners` |
| `api/v1alpha1/taskspawner_types.go` | Removed `SecretRef` and `PollInterval` from `Slack` struct — these are now server-side concerns |
| `internal/controller/taskspawner_deployment_builder.go` | Removed Slack args/env injection from `buildPodParts` |
| `cmd/kelos-spawner/main.go` | Removed all Slack flags, persistent source, reporting, and `buildSourceWithProxy` Slack branch |
| `cmd/kelos-spawner/reconciler.go` | Removed Slack config fields, reporting branch, and poll interval case |
| `internal/source/slack.go` | Deleted — per-`TaskSpawner` Slack source is no longer needed |
| `internal/source/slack_test.go` | Deleted |
| `internal/manifests/charts/kelos/templates/slack-server.yaml` | Helm template for the centralized `kelos-slack-server` `Deployment` |
| `internal/manifests/charts/kelos/templates/rbac.yaml` | Added `kelos-slack-server-role` `ClusterRole` and binding |
| `internal/manifests/charts/kelos/templates/serviceaccount.yaml` | Added `kelos-slack-server` `ServiceAccount` |
| `internal/manifests/charts/kelos/values.yaml` | Added `slackServer` values (`enabled`, `replicas`, `secretName`, `image`, `resources`) |
| `internal/manifests/charts/kelos/templates/crds/taskspawner-crd.yaml` | Removed `secretRef` and `pollInterval` from Slack schema, updated description |
| `internal/manifests/install-crd.yaml` | Same CRD changes as above (standalone install manifest) |

**Message routing flow:**
1. Socket Mode receives a Slack event
2. `handleEventsAPI` / `handleSlashCommand` parse and enrich the message
3. `routeMessage` lists all `TaskSpawners` with `spec.when.slack` set
4. For each spawner: checks suspended state, `maxConcurrency`, channel/user filters, and trigger command
5. Creates a `Task` with Slack reporting annotations (`kelos.dev/slack-reporting`, `kelos.dev/slack-channel`, `kelos.dev/slack-thread-ts`)

N/A

- The reporting loop in `main.go` lists **all** `Tasks` cluster-wide (not scoped to a single `TaskSpawner`), matching the centralized model. Each `Task`'s Slack annotations determine whether reporting applies.
- Thread interactivity: thread replies only trigger tasks if the bot has previously participated in the thread (`BotParticipated` check), preventing noise from unrelated conversations.
- The Helm chart follows the same pattern as `webhook-server.yaml` — `Deployment` with `ServiceAccount`, `ClusterRole`, and `ClusterRoleBinding`. The Slack server additionally needs `update` on `tasks` (for reporting annotations) and `leases`/`events` (for leader election).

```release-note
Add centralized `kelos-slack-server` for routing Slack messages to matching `TaskSpawners`, replacing the per-`TaskSpawner` Socket Mode deployment model. Remove `secretRef` and `pollInterval` from the `Slack` CRD struct. Add Helm chart support via `slackServer.enabled`.
```
…erver` image (#60)

/kind bug

The `kelos-slack-server` binary was added in #59 but was missing a `Dockerfile` and wasn't included in the `IMAGE_DIRS` list in the `Makefile`. CI wouldn't build or push the `kelos-slack-server` Docker image, so it can't be deployed.

This adds:
- `cmd/kelos-slack-server/Dockerfile` (same `distroless/static:nonroot` pattern as the other server images)
- `cmd/kelos-slack-server` to `IMAGE_DIRS` in the `Makefile`

N/A

The `Dockerfile` is identical in structure to `cmd/kelos-webhook-server/Dockerfile`.

```release-note
NONE
```
anomalogravity Bot and others added 3 commits April 21, 2026 15:53
* fix(slack): show activity indicator on in-progress messages

Progress messages were posted as text-only (no Block Kit blocks),
causing appendActivityContext to skip the update since it requires
blocks to attach the activity context element. Add FormatProgressMessage
to wrap progress text in blocks (section + context), so the activity
indicator continues working after the first progress snapshot replaces
the accepted message as the update target.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove activity indicator line from older messages

---------

Co-authored-by: Gravity <gravity@anomalo.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Kristian Cailer <kristian@anomalo.com>
#### What type of PR is this?

/kind bug

#### What this PR does / why we need it:

When a task completes, the final result (succeeded/failed) was always posted as a new thread reply — even when a progress snapshot message already existed. This meant the thread had three messages (ack, progress snapshot, final result) where the last two were nearly identical, since the progress snapshot captured the agent's near-final output.

This PR changes `ReportTaskStatus` to edit the existing progress message in-place with the final `FormatSlackTransitionMessage` content instead of posting a new reply. The result is a cleaner two-message thread: the initial ack and a single response message that evolves from progress snapshot to final result.

If no progress message exists (e.g. the task finished before the first 30-second snapshot), the behavior is unchanged — a new reply is posted as before.

#### Which issue(s) this PR is related to:

N/A

#### Special notes for your reviewer:

- The production change is a single block in `ReportTaskStatus`: on terminal phases (`succeeded`/`failed`), check `getProgressTS` and call `UpdateMessage` instead of `PostThreadReply`.
- If `UpdateMessage` fails (e.g. Slack API error), we fall back to posting a new reply so the final result is never lost.
- The ack message is never at risk of being overwritten — it's tracked via `setActivityTarget` (the `activity` map), while this code uses `getProgressTS` (the `progressTS` map). The two maps are completely separate.
- Two new tests:
  - `TestSlackTaskReporter_EditsProgressMessageOnTerminalPhase` — verifies the progress message gets edited with final content and no new reply is posted.
  - `TestSlackTaskReporter_FallsBackToPostOnUpdateFailure` — verifies that when `UpdateMessage` fails, a new reply is posted as fallback.

#### Does this PR introduce a user-facing change?

```release-note
NONE
```
@greptile-apps

greptile-apps Bot commented Apr 21, 2026

Copy link
Copy Markdown

Greptile Summary

This PR fixes a race condition in the Slack reporter where the activity-indicator goroutine could overwrite the final succeeded/failed message by calling UpdateMessage with stale intermediate content after the terminal path had already written the result. The fix has two parts: (1) clearActivityState is now called before UpdateMessage in the terminal path so any concurrent activity loop sees the state as gone first, and (2) UpdateActivityIndicator adds a re-check under the mutex after building the message but before the API call to bail out if the state was cleared in the meantime.

Confidence Score: 5/5

Safe to merge — the fix correctly closes the primary race window; the only remaining concern is a narrow TOCTOU gap that is an inherent limitation of the unlocked-API-call pattern.

All findings are P2. The core logic is sound and matches established patterns in the codebase. The new test covers the deterministic scenario for the fix. No data-integrity or blocking-path issues were found.

No files require special attention beyond the noted residual TOCTOU window in watcher.go.

Important Files Changed

Filename Overview
internal/reporting/watcher.go Adds clearActivityState before UpdateMessage in the terminal path and introduces a re-check guard in UpdateActivityIndicator; a narrow TOCTOU window between the re-check and the API call remains.
internal/reporting/watcher_test.go Adds TestSlackTaskReporter_ActivitySkipsUpdateWhenStateCleared to verify zero Slack API calls when state is pre-cleared; deterministic test for the fix, coverage is appropriate for a unit-test suite.

Sequence Diagram

sequenceDiagram
    participant A as ActivityGoroutine
    participant T as TerminalPath
    participant S as SlackAPI

    Note over A,S: Happy path after fix
    A->>A: lock, read messageTS, unlock
    A->>A: build msg
    T->>T: clearActivityState
    T->>S: UpdateMessage with final content
    A->>A: re-check, state nil, bail out

    Note over A,S: Residual narrow race
    A->>A: lock, read messageTS, unlock
    A->>A: build msg
    A->>A: re-check passes, unlock
    T->>T: clearActivityState
    T->>S: UpdateMessage with final content
    A->>S: UpdateMessage with activity, may overwrite
Loading

Fix All in Claude Code

Prompt To Fix All With AI
This is a comment left during a code review.
Path: internal/reporting/watcher.go
Line: 566-573

Comment:
**Residual TOCTOU window between re-check and API call**

After the re-check mutex is released (line 571) and before `UpdateMessage` is called (line 573), the terminal path can still: (1) `clearActivityState`, then (2) complete its own `UpdateMessage` with the final content — leaving the activity goroutine's subsequent `UpdateMessage` to overwrite the final message.

```
Activity:  lock → check passes → unlock → [window] → UpdateMessage(activity) ← last-write wins
Terminal:                                   clearState → UpdateMessage(final)
```

Because both calls are Slack network round-trips (~100 ms each), the window is narrow but real. The fix meaningfully narrows it (the earlier gap between the first mutex drop at line 548 and the API call was much wider), so this is an improvement — but the residual TOCTOU is inherent to the check-then-act pattern across an unlocked API call. If this bites in practice, one mitigation would be a monotonic generation counter on `activityState` that the activity goroutine captures and re-validates just before the API call, though that adds complexity for a cosmetic concern.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(slack): prevent activity indicator r..." | Re-trigger Greptile

Comment on lines +566 to 573
tr.mu.Lock()
if s := tr.activity[task.UID]; s == nil || s.MessageTS != messageTS {
tr.mu.Unlock()
return
}
tr.mu.Unlock()

if err := tr.Reporter.UpdateMessage(ctx, channel, messageTS, msg); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Residual TOCTOU window between re-check and API call

After the re-check mutex is released (line 571) and before UpdateMessage is called (line 573), the terminal path can still: (1) clearActivityState, then (2) complete its own UpdateMessage with the final content — leaving the activity goroutine's subsequent UpdateMessage to overwrite the final message.

Activity:  lock → check passes → unlock → [window] → UpdateMessage(activity) ← last-write wins
Terminal:                                   clearState → UpdateMessage(final)

Because both calls are Slack network round-trips (~100 ms each), the window is narrow but real. The fix meaningfully narrows it (the earlier gap between the first mutex drop at line 548 and the API call was much wider), so this is an improvement — but the residual TOCTOU is inherent to the check-then-act pattern across an unlocked API call. If this bites in practice, one mitigation would be a monotonic generation counter on activityState that the activity goroutine captures and re-validates just before the API call, though that adds complexity for a cosmetic concern.

Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/reporting/watcher.go
Line: 566-573

Comment:
**Residual TOCTOU window between re-check and API call**

After the re-check mutex is released (line 571) and before `UpdateMessage` is called (line 573), the terminal path can still: (1) `clearActivityState`, then (2) complete its own `UpdateMessage` with the final content — leaving the activity goroutine's subsequent `UpdateMessage` to overwrite the final message.

```
Activity:  lock → check passes → unlock → [window] → UpdateMessage(activity) ← last-write wins
Terminal:                                   clearState → UpdateMessage(final)
```

Because both calls are Slack network round-trips (~100 ms each), the window is narrow but real. The fix meaningfully narrows it (the earlier gap between the first mutex drop at line 548 and the API call was much wider), so this is an improvement — but the residual TOCTOU is inherent to the check-then-act pattern across an unlocked API call. If this bites in practice, one mitigation would be a monotonic generation counter on `activityState` that the activity goroutine captures and re-validates just before the API call, though that adds complexity for a cosmetic concern.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Claude Code

@anomalogravity

Copy link
Copy Markdown

Risk Assessment: Low

  • Small, focused diff: 45 additions, 1 deletion across 2 files (one source, one test)
  • No critical paths touched: Changes are confined to Slack reporting logic — no DB connectors, migrations, public API, ML algorithms, auth, or infrastructure
  • Race condition fix: Reorders an existing call to run before (rather than after), and adds a mutex-guarded re-check in before the API call — both are minimal, surgical changes
  • Test coverage: New test directly verifies the fix, using established test patterns in the file
  • Greptile confidence: 5/5: Greptile rated this safe to merge. The only finding (P2) is a residual TOCTOU window inherent to the unlocked-API-call pattern — acknowledged in the PR description and consistent with existing codebase patterns
  • No security, privacy, or external service implications: Pure internal concurrency fix for a cosmetic Slack message ordering issue

@anomalogravity anomalogravity Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: PR assessed as low risk by Gravity.

@kristiancailer

Copy link
Copy Markdown
Author

@claude, not sure if Greptile's comment requires addressing but please take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants