Skip to content

fix(runner/gcp): correct gke_logs filter and reshape gke_traces BigQuery output - #527

Open
mayankpande88 wants to merge 4 commits into
mainfrom
fix/runner-gcp-logs-and-bq
Open

fix(runner/gcp): correct gke_logs filter and reshape gke_traces BigQuery output#527
mayankpande88 wants to merge 4 commits into
mainfrom
fix/runner-gcp-logs-and-bq

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Two parity fixes for the GCP primitives:

  • gke_logs: the filter matched resource.type="gce_instance" by zone,
    which returns VM syslog rather than node-pool scaling events. Query
    the GKE cluster-autoscaler-visibility logs instead (k8s_cluster
    resource, project_id + location labels, the autoscaler-visibility
    logName, severity>=DEFAULT), with a zone->region fallback since those
    logs are keyed by the cluster's location (zonal vs regional).

  • gke_traces: pass the dataset location (explicit location, else
    derived from zone) in the BigQuery job, and reshape the raw BQ REST
    response ({schema.fields[], rows[].f[].v}) into the
    {data, columns, column_types} envelope the backend warehouse consumer
    expects (mirroring the legacy run_bigquery).

Note: the BigQuery reshape key names mirror the legacy agent; the
agent_warehouse consumer contract should be confirmed against the
backend repo before release.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01L2HAXQH1gEtX7h4sUJsi9j

@mayankpande88
mayankpande88 requested a review from a team as a code owner July 14, 2026 19:28

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the GCP client to query GKE cluster-autoscaler visibility logs instead of GCE instance logs, implementing a fallback from zone to region when no logs are found. It also updates QueryBigQuery to accept a location parameter and reshape responses into a {data, columns, column_types} envelope. Feedback on these changes includes a correction to zoneToRegion to prevent incorrect truncation of region names (e.g., converting us-central1 to us-central), and a performance optimization to check if cell values are non-empty before unmarshaling them.

Comment thread runner/pkg/observability/gcp/client.go
Comment on lines +205 to +209
var v any
// BigQuery cell values are JSON scalars (usually strings); keep the
// decoded value, or nil on an unexpected shape.
_ = json.Unmarshal(cell.V, &v)
vals[i] = v

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To avoid unnecessary CPU overhead and potential errors from parsing empty or nil json.RawMessage values, we should check if cell.V is non-empty before calling json.Unmarshal.

Suggested change
var v any
// BigQuery cell values are JSON scalars (usually strings); keep the
// decoded value, or nil on an unexpected shape.
_ = json.Unmarshal(cell.V, &v)
vals[i] = v
var v any
if len(cell.V) > 0 {
_ = json.Unmarshal(cell.V, &v)
}
vals[i] = v

RamanKharchee
RamanKharchee previously approved these changes Jul 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Image Tags Updated

I've automatically updated the image tags in `charts/nudgebee-agent/values.yaml` to the latest versions from GHCR for the `main` branch.

The image tags are now synchronized with the latest builds and ready for release.

@mayankpande88
mayankpande88 force-pushed the fix/runner-gcp-logs-and-bq branch from ba048f1 to 18afad8 Compare August 11, 2026 09:50
claude and others added 3 commits September 3, 2026 10:37
…ery output

Two parity fixes for the GCP primitives:

- gke_logs: the filter matched resource.type="gce_instance" by zone,
  which returns VM syslog rather than node-pool scaling events. Query
  the GKE cluster-autoscaler-visibility logs instead (k8s_cluster
  resource, project_id + location labels, the autoscaler-visibility
  logName, severity>=DEFAULT), with a zone->region fallback since those
  logs are keyed by the cluster's location (zonal vs regional).

- gke_traces: pass the dataset location (explicit `location`, else
  derived from `zone`) in the BigQuery job, and reshape the raw BQ REST
  response ({schema.fields[], rows[].f[].v}) into the
  {data, columns, column_types} envelope the backend warehouse consumer
  expects (mirroring the legacy run_bigquery).

Note: the BigQuery reshape key names mirror the legacy agent; the
agent_warehouse consumer contract should be confirmed against the
backend repo before release.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L2HAXQH1gEtX7h4sUJsi9j
(cherry picked from commit 048912d)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
(cherry picked from commit bc0ef44)
Deriving it from `zone` pinned jobs to a region, which fails for datasets in
the US/EU multi-regions that work today by omitting location. (Legacy
run_bigquery passed the raw compute zone, which is not a valid BigQuery
location either.) Forward an explicit `location` and otherwise let BigQuery
resolve it from the referenced dataset.

The zone->region fallback is unchanged on the logs path, where it matches the
legacy locations_to_try behavior.
@mayankpande88
mayankpande88 force-pushed the fix/runner-gcp-logs-and-bq branch from d9dcdfa to 7116327 Compare September 3, 2026 05:07
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

📦 Image Tags Updated

I've automatically updated the image tags in `charts/nudgebee-agent/values.yaml` to the latest versions from GHCR for the `main` branch.

The image tags are now synchronized with the latest builds and ready for release.

@mayankpande88

Copy link
Copy Markdown
Contributor Author

Rebased onto main (both commits preserved, including the zoneToRegion suffix fix).

Verified the core claim against the legacy agent: gcloud_client.py:49-52 really does query resource.type="k8s_cluster" with the cluster-autoscaler-visibility logName, so the gce_instance filter on main cannot ever have returned autoscaler logs. The zone→region fallback list and the {data, columns, column_types} envelope also match legacy exactly.

One fix while rebasing: the BigQuery location is no longer derived from zone. BigQuery locations are multi-regions (US, EU) or regions (us-central1); pinning a job to a derived region breaks datasets in the US/EU multi-regions that work today by omitting location. (Legacy passed the raw compute zone, which is not a valid BigQuery location either — so neither side was right.) Now an explicit location param is forwarded and otherwise BigQuery resolves it from the referenced dataset. Covered by TestGkeTraces_LocationOnlyWhenExplicit.

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

📦 Image Tags Updated

I've automatically updated the image tags in `charts/nudgebee-agent/values.yaml` to the latest versions from GHCR for the `main` branch.

The image tags are now synchronized with the latest builds and ready for release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants