Skip to content

Detect Otto and GitHub Copilot as calling agents - #249

Merged
jlaneve merged 2 commits into
mainfrom
worktree-detect-otto
Aug 3, 2026
Merged

Detect Otto and GitHub Copilot as calling agents#249
jlaneve merged 2 commits into
mainfrom
worktree-detect-otto

Conversation

@jlaneve

@jlaneve jlaneve commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

_detect_invocation_context reads a fixed list of environment variables to work out which coding agent ran the command. Otto is not on it and sets no marker of its own, so every af command Otto runs arrives with no agent and looks like a person typing.

This matters more for af than for astro. Otto installs its own pinned af wrapper and puts it first on PATH, so in an Otto session essentially all af traffic is Otto-driven — and none of it is labelled.

There is a second effect. Otto hands its environment to child processes untouched, so a user who starts Otto from inside another coding agent passes that agent's variable straight through to af, and the commands get credited to it. Some share of the claude-code volume in af telemetry is really Otto.

What this does

Adds "OTTO": "otto" to agent_env_vars, first in the dict. The loop breaks on the first match and dicts keep insertion order, so position is the fix for the second problem: with OTTO ahead of CLAUDECODE, an Otto session started inside Claude Code reports otto, the nearer caller.

Companion changes: astronomer/otto#379 sets OTTO=1; astronomer/astro-cli#2234 and astronomer/astro-cli-internal#81 do the same for the astro CLI. Until the Otto change ships this is inert.

Testing

test_detects_otto and test_otto_wins_over_inherited_marker, plus OTTO added to the clean_env fixture so it cannot leak in from the environment running the suite. 45 tests pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NKSW26NfBtfq8hVeeLdfLj

Update: also adds GitHub Copilot detection

Second commit on this branch. Copilot was missing from the same list, so anything run from Copilot arrived unlabelled too.

Variable Surface Source
COPILOT_CLI Copilot CLI — also the harness behind the cloud coding agent and, since 2026, Copilot for JetBrains github's own gh keys on it: cli/cli internal/agents/detect.go
COPILOT_AGENT VS Code agent mode terminals microsoft/vscode toolTerminalCreator.ts, where it is commented as backward compatibility for this kind of detection
AI_AGENT VS Code agent mode (github_copilot_vscode_agent), Copilot desktop app (github_copilot_app_agent) microsoft/vscode aiAgentEnv.ts, which calls it a wire contract

AI_AGENT is a different shape from everything else in the list: the value names the agent, not the variable. It is read after the list so a specific marker still decides, and only prefixes we recognise are mapped.

Values are deliberately not passed through, which is where this differs from gh. gh returns any value matching ^[a-zA-Z0-9_-]+$. Some vendors put a version in theirs — Claude Code's is claude-code_2-1-156_agent — so passing values through would spread one agent across a new telemetry value on every release and make the column useless for grouping. A new vendor adopting the convention is a one-line addition.

Two candidates dropped after checking the sources

COPILOT_AGENT_SESSION_ID is widely cited but appears in no Copilot CLI changelog entry through 1.0.76, and gh does not read it. COPILOT_CLI covers the same surface with a first-party source behind it.

VSCODE_AGENT is named in the VS Code 1.121 release notes but does not exist in the source — only VSCODE_AGENT_ZSH_FIXUPS and VSCODE_AGENT_HOST_*. The release note is wrong and at least one other library has copied the error.

Also avoided: COPILOT_MODEL, COPILOT_ALLOW_ALL, COPILOT_GITHUB_TOKEN. Some detectors key on these, but they are user config the CLI reads, so anyone with export COPILOT_MODEL=… in a shell profile would be misdetected forever.

Worth a follow-up, not in this PR

CODEX_API_KEY in the existing list has that same flaw — it is a key the user sets, not a marker Codex exports. CODEX_THREAD_ID or CODEX_SANDBOX are what Codex actually sets per execution, and are what gh reads.

@jlaneve jlaneve changed the title Detect Otto as a calling agent in af telemetry Detect Otto and GitHub Copilot as calling agents Jul 30, 2026
@jlaneve

jlaneve commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

The Type checking job failure here is pre-existing and unrelated to this PR.

It fails on skills/analyzing-data/scripts/cli.pyCannot resolve imported module click`` — a file this branch does not touch. This PR changes only astro-airflow-mcp/src/astro_airflow_mcp/telemetry.py and its test.

I reproduced it on an untouched checkout of current origin/main, same error:

error[unresolved-import]: Cannot resolve imported module `click`
  --> cli.py:26:8

The cause is the job itself. .github/workflows/ci.yml:85-96 runs uvx ty check in skills/analyzing-data/scripts with no dependency install step, so click is only ever resolvable by accident, and ty is unpinned so every run picks up the latest release. Main looks green because its last run was 2026-07-20, before the ty version that reports this.

Two ways to fix, whichever the owners prefer: run it as uv run ty check (or add a uv sync step) so the project's own dependencies are on the path, or pin ty so the workflow stops drifting. Happy to open a separate PR for that — it touches a shared workflow, so it did not belong in this one.

kaxil pushed a commit that referenced this pull request Aug 3, 2026
`Type checking` fails on `main` as it stands. Any PR touching Python in
this repo hits it —
[#249](#249) is how I found it.
I reproduced it by running the job's own command against an untouched
checkout of `origin/main`:

```
error[unresolved-import]: Cannot resolve imported module `click`
  --> cli.py:26:8
```

`main` reads green only because its last CI run was 2026-07-20, before
the ty release that changed this.

## What broke

The job at `.github/workflows/ci.yml:85-96` runs `uvx ty check`,
unpinned, so every run picks up the latest ty.

ty `0.0.62` (2026-07-21) shipped this, under Configuration:

> Respect `rules` and `analysis` in PEP 723 script metadata
configurations

`cli.py` carries a PEP 723 inline metadata block. As of that release the
block configures the file, and `ty.toml` no longer reaches it. The
`unresolved-import = "ignore"` rule stopped applying and the `click`
import surfaced as an error.

Bisected:

| ty | result |
|---|---|
| `0.0.61` | passes |
| `0.0.62` | fails ← behaviour change |
| `0.0.65` (what CI pulls today) | fails |

## What this does

Repeats the suppression in the script's own metadata, which is where ty
now reads it from, plus a note in `ty.toml` so the next person
understands the duplication.

The dependency is not actually missing — `cli.py` declares
`click>=8.0.0` in its PEP 723 header and runs via `uv run cli.py`. ty
does not install dependencies, which is the reason `ty.toml` suppressed
the rule in the first place. This keeps that decision, it just states it
where ty will now look.

`cli.py` is the only PEP 723 script in the checked directory, so it is
the only file affected.

## Testing

`uvx ty check` passes on ty `0.0.65` and on `0.0.61`, so it works either
side of the change. `uv run cli.py --help` still works, confirming the
added block does not disturb PEP 723 execution.

## One thing for the owners to decide

I did not pin ty, because that is a policy call rather than part of the
fix. Worth considering though: an unpinned tool in CI means an upstream
release can turn `main` red without anyone touching the repo, which is
exactly what happened. A pin would make the job reproducible at the cost
of someone bumping it. Happy to add it if you want.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01NKSW26NfBtfq8hVeeLdfLj

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
jlaneve and others added 2 commits August 3, 2026 14:29
…rkers

_detect_invocation_context reads a fixed list of environment variables to
work out which coding agent ran the command. Otto was missing, so the af
commands Otto runs came through with no agent at all and looked like a
person typing. Otto drives af heavily, so this is a real gap.

Otto passes its environment to children untouched, so an Otto session
started from inside another agent still carries that agent's marker. OTTO
goes first in the dict, ahead of CLAUDECODE and the rest, because Otto is
the nearer caller. A matching change in astronomer/otto sets it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSW26NfBtfq8hVeeLdfLj
Copilot was missing from the list, so any af command run from Copilot
looked like a person typing.

Two variables carry it. COPILOT_CLI comes from the Copilot CLI, which is
also the harness behind the cloud coding agent and, since 2026, Copilot
for JetBrains, so one entry covers three surfaces; github's own gh reads
the same variable. COPILOT_AGENT is set on the terminals VS Code builds
for agent mode, where the source calls it backward compatibility for this
kind of detection.

AI_AGENT is a newer convention where the value names the agent rather
than the variable, so VS Code reports github_copilot_vscode_agent and the
desktop app github_copilot_app_agent. It is read after the list, so a
specific marker still decides, and only prefixes we know are mapped.
Values are not passed through: some vendors put a version in theirs,
which would spread one agent over dozens of values in the telemetry.

Two candidates were dropped on inspection. COPILOT_AGENT_SESSION_ID is
in no changelog and gh does not read it. VSCODE_AGENT is named in the
VS Code 1.121 release notes but is nowhere in the source; the note is
wrong, and other tools have copied the mistake.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSW26NfBtfq8hVeeLdfLj
@kaxil
kaxil force-pushed the worktree-detect-otto branch from 41526f8 to 80e8d50 Compare August 3, 2026 13:29
@jlaneve
jlaneve merged commit 445b418 into main Aug 3, 2026
11 checks passed
@jlaneve
jlaneve deleted the worktree-detect-otto branch August 3, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants