[Power BI] Address ARG payload size errors

[Copilot]

Problem

Users with large Azure environments are encountering payload size limit errors when attempting to load the Governance and Workload Optimization Power BI reports. The error message appears as:

Response payload size is 48439993, and has exceeded the limit of 16777216. Please consider querying less data at a time and make paginated call if needed.

This occurs because both reports use Azure Resource Graph queries with [resultTruncated = false] to retrieve comprehensive resource details. For organizations with millions of resources, the response payload exceeds Azure Resource Graph's 16 MB limit, preventing the reports from loading.

Root Cause

Azure Resource Graph enforces a 16 MB (16,777,216 bytes) response payload limit per query. The Governance and Workload Optimization reports are designed to show detailed resource-level information and are optimized for small to medium-sized environments. When used in large enterprise environments with millions of resources, the unfiltered queries exceed this limit.

Solution

This PR includes both a functional fix and documentation. The functional fix paginates Azure Resource Graph queries by subscription so a single query is split into multiple smaller batches that each stay under the 16 MB limit, then combines the results. The documentation gives users manual mitigation options (filtering by subscription/tags, trimming columns) for cases where the default batching still isn't enough, or where they want more control.

Functional changes

• Added ftk_QueryARG and ftk_ARGBatchSize (in Shared.Dataset/definition/expressions.tmdl, both the kql and storage datasets) — a shared M function that queries Resource Graph for the tenant's subscriptions, splits them into batches (default 100 subscriptions per batch, user-configurable via ftk_ARGBatchSize), runs the caller's query once per batch with a subscriptionId in (...) filter, and combines all batch results.
• Adopted ftk_QueryARG across the Resource Graph-backed tables used by the Governance and Workload Optimization reports (Advisor recommendations, disks, network interfaces/security groups, policy assignments/states, public IPs, resources, SQL databases, virtual machines).
• Hardened ftk_QueryARG against invalid ftk_ARGBatchSize() values (null, non-numeric, non-integer, or less than 1 — falls back to the documented default of 100) and against the subscription-list Resource Graph query failing or returning an unexpected shape (falls back to an empty batch list instead of throwing).

Documentation changes

1. Enhanced Error Documentation (help/errors.md)

Expanded the existing "Response payload size exceeded" error section with detailed, step-by-step mitigation strategies for cases where the built-in batching isn't sufficient:

Option 1: Filter by Subscription

| where subscriptionId in~ ('subscription-id-1', 'subscription-id-2')

Option 2: Filter by Tags

| where tags.Environment in~ ('Production', 'Staging')
| where tags.CostCenter == '12345' and tags.Environment =~ 'Production'

Option 3: Remove Unnecessary Columns

• Guidance on identifying and removing columns that aren't used in report visuals

Option 4: Disable Problematic Queries

• Instructions for disabling specific failing queries while keeping others functional

Each option includes step-by-step instructions, KQL code examples, and links to official Microsoft documentation.

2. Updated Report Documentation

Added "Known limitations" sections to both affected reports (power-bi/governance.md, power-bi/workload-optimization.md) explaining the 16 MB Azure Resource Graph payload limit and linking to the error reference guide.

3. Changelog Entry

Added an entry to the v15 Power BI reports section of changelog.md (the next unreleased version).

Impact

• ⚠️ Includes functional Power Query (M) changes to the Shared.Dataset expressions and Resource Graph-backed tables — not documentation-only.
• ✅ No breaking changes — batching is transparent to report consumers; existing table schemas are unchanged.
• ✅ Improved reliability — reports in large environments no longer fail outright on the 16 MB payload limit.
• ✅ Self-service guidance — users can further tune batch size or apply manual filters if their environment still needs it.

References

• Working with large Azure resource data sets
• Troubleshoot errors using Azure Resource Graph
• Azure Resource Graph Power BI connector quickstart
• Guidance for throttled requests

Testing

• Verified all documentation links are valid
• Confirmed markdown formatting follows repository standards
• Validated KQL syntax examples against Azure Resource Graph documentation
• Traced the M (Power Query) batching and error-handling logic against the M language specification since there's no local Power BI Desktop/CI harness to execute it directly
• Merged current dev and resolved conflicts, restoring a review-flagged regression (a dropped reserved-instance-specific filter in AdvisorReservationRecommendations) in the process

Known limitations / follow-ups

• Subscription-list batch size (ftk_ARGBatchSize) is user-configurable but has no upper cap; an extremely large custom value could still hit the 16 MB limit in very large tenants.
• Per-batch Resource Graph query failures (e.g., mid-refresh throttling) still propagate for that table, matching pre-existing behavior for the inner batch loop.

---

Fixes: #1768

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Governance Report is not loading says payload size limit is exceeding the limits.</issue_title>
<issue_description><!--
⚠️⚠️⚠️ BEFORE YOU SUBMIT ⚠️⚠️⚠️

1. Confirm there isn't an issue already. If so, vote it up (👍) and add comments.
2. Complete all TODO items below and remove the TODO lines after.
3. Internal: Add applicable labels: Type, Micro PR, Area
   -->

🐛 Problem

TODO: Governance report is not loading - With payload size limits exceeding the limits. Currently using V0.11 version of Finops hub. MCA customer with billing account admin privileges is trying to pull out data in the governance report, below is the error reported.

👣 Repro steps

TODO: Add repro steps below:

1. Go to Governance.Pbit
2. Click on Load
3. See error

🤔 Expected

TODO: All the ADF pipelines are successful and remaining cost summary, workload optimization reports are loading ok., where-as governance report is not at all loading at this moment.

📷 Screenshots

TODO: Attached is the screenshot.

ℹ️ Additional context

TODO: Governance Reports fails to load in the customer environment due to payload size limits issue.

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.
   </issue_description>

<agent_instructions>Triage the issue, research using mslearn, provide proposed solutions along with links/citations to authorative sources.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

Fixes #1768

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

[flanakin]

🤖 [AI][Claude] PR Review

Summary: This documentation-only PR adds helpful mitigation guidance for Azure Resource Graph payload size limit errors. The expanded error documentation is a significant improvement. A couple of items to address before merging.

⚠️ Should fix (1)

1. Changelog entries are placed under "Implementing FinOps guide v14" but these are Power BI report documentation changes. They should be in their own section (e.g., a new ### [Power BI reports](power-bi/reports.md) v14 section or a standalone ### Documentation improvements section). Entries also missing issue link #1768.

💡 Suggestions (1)

1. "small- and medium-sized" uses a suspended hyphen that's grammatically correct but unusual in Microsoft docs — consider "small and medium-sized" for readability.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 43 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/power-bi/kql/Shared.Dataset/definition/expressions.tmdl:158

• tostring(split(id, '/')[2]) only yields a real subscription ID for IDs that start with /subscriptions/{guid}/.... For tenant- or management-group-scoped policy definitions, it produces incorrect values (for example, /providers/Microsoft.Authorization/... becomes Microsoft.Authorization). Use an extract that returns null when the ID is not subscription-scoped.

		      subscriptionId = tostring(split(id, '/')[2]),

src/power-bi/storage/Shared.Dataset/definition/expressions.tmdl:565

• tostring(split(id, '/')[2]) only yields a real subscription ID for IDs that start with /subscriptions/{guid}/.... For tenant- or management-group-scoped policy definitions, it produces incorrect values (for example, /providers/Microsoft.Authorization/... becomes Microsoft.Authorization). Use an extract that returns null when the ID is not subscription-scoped.

		        subscriptionId = tostring(split(id, '/')[2]),

src/power-bi/kql/Shared.Dataset/definition/tables/PolicyAssignments.tmdl:372

• tostring(split(id, '/')[2]) yields incorrect values for non-subscription-scoped IDs (tenant or management group scope). Use an extract that only returns a subscription ID when the resource ID contains /subscriptions/{guid}/... so you don't populate bogus IDs like Microsoft.Authorization.

				    | extend subscriptionId = tostring(split(id, '/')[2])

src/power-bi/kql/Shared.Dataset/definition/tables/PolicyStates.tmdl:347

• tostring(split(id, '/')[2]) yields incorrect values for non-subscription-scoped IDs (tenant or management group scope). Use an extract that only returns a subscription ID when the resource ID contains /subscriptions/{guid}/... so you don't populate bogus IDs like Microsoft.PolicyInsights.

				    | extend subscriptionId = tostring(split(id, '/')[2])

[MSBrett]

tostring(split(id, '/')[2]) only yields a real subscription ID for IDs that start with /subscriptions/{guid}/.... For tenant- or management-group-scoped policy definitions, it produces incorrect values (for example, /providers/Microsoft.Authorization/... becomes Microsoft.Authorization).

Confirmed as a real bug in code this PR introduced (this subscriptionId derivation for the policy tables was added here to support the new subscription batching; it did not exist before). Fixed in all 4 flagged locations (kql/expressions.tmdl, storage/expressions.tmdl, PolicyAssignments.tmdl, PolicyStates.tmdl) by guarding the split so non-subscription-scoped IDs now produce an empty value instead of a misleading label like Microsoft.Authorization or Microsoft.PolicyInsights.

One residual limitation this fix does not solve, disclosed for transparency: ftk_QueryARG appends a | where subscriptionId in (...) filter per subscription batch, so rows with an empty/non-subscription-scoped subscriptionId (tenant- or management-group-scoped policy definitions/assignments/states) will not match any batch and are excluded from these three tables' results, both before and after this fix. That is a consequence of the subscription-based batching design itself, not something a value-correctness fix can resolve. If org-wide policy visibility matters for this report, that would need a separate change (for example, including an empty-subscriptionId branch in exactly one batch). I have not made that change here since it touches the shared ftk_QueryARG function used by every batched table and is a bigger design change than this fix; happy to file a follow-up issue if that is wanted.

[MSBrett]

1. High — subscription batching drops built-in and management-group Azure Policy data

PolicyDefinitions and PolicyAssignments derive subscriptionId from split(id, '/')[2], then call ftk_QueryARG, which appends where subscriptionId in ('<subscription GUIDs>'). Built-in definition IDs use /providers/Microsoft.Authorization/policyDefinitions/<id>, so the derived value is Microsoft.Authorization; management-group IDs similarly derive Microsoft.Management. Neither can match a subscription GUID, so those rows are removed. Keep tenant/management-group records outside subscription batching and union them with batched subscription-scope records, or apply the filter only to subscription-scoped IDs. Add regression coverage for built-in, management-group, and subscription IDs in both dataset copies.

2. Medium — the reported NSG payload case remains insufficiently bounded and unproven

The issue specifically identifies the NSG query and its mv-expand properties.securityRules as exceeding 16 MB, but NetworkSecurityGroups.tmdl retains that expansion and uses ftk_QueryARG(query, 20). The documented default therefore creates 2,000-subscription batches, and the documented integer minimum of 1 still produces 20 subscriptions. No measurement or validation demonstrates that this stays below ARG’s 16 MB limit. Remove or reduce the multiplier, or expose a true per-table batch setting, and validate the selected bound against a large NSG-rule tenant.

3. Medium — required validation evidence for the core M change is absent

There is no Power BI test, .tmdl/M CI validation, or reported manual refresh for the new batching behavior. The PR only reports an M-spec trace, while the repository’s build-and-test guidance says Power BI reports require manual build/verification and that manual verification is always expected. Record a Power BI Desktop refresh against a representative large tenant, including NSG expansion and policy scopes, and add a deterministic boundary check for batching and ID scoping if automated Power BI validation remains unavailable.

[MSBrett]

Thanks for the detailed review, @MSBrett. Addressed in cdd13b2c:

1. High: subscription batching drops built-in and management-group policy data

Confirmed and fixed. ftk_QueryARG now takes an optional includeUnscoped flag. When set, it runs one extra, unbatched ARG query for rows whose derived subscriptionId is blank (| where isempty(subscriptionId)) and unions that in with the batched results. PolicyDefinitions (kql + storage), PolicyAssignments, and PolicyStates now pass true for this flag, so built-in definitions and management-group-scoped assignments/states are no longer silently excluded. All other tables are unaffected (the parameter defaults to off), so this doesn't touch subscription-scoped tables that don't need it.

I did not add a union of every possible non-subscription scope pattern beyond "blank subscriptionId" - the fix relies on the existing iff(...) guard that already normalizes any non-/subscriptions/... id shape to '', so it covers tenant and management-group scope uniformly rather than enumerating each scope type.

2. Medium: NSG payload case insufficiently bounded

Agreed the 20x multiplier (2,000 subs/batch default, 20 subs/batch even at the documented minimum) was backwards. NetworkSecurityGroups had the largest multiplier of any table despite its mv-expand of properties.securityRules being the highest per-row payload expansion in the whole dataset, and despite this table historically being the original motivating case for reducing payload size (changelog issue #1540). I removed the multiplier entirely, so it now falls back to the 100-subs/batch default like Resources - a 20x reduction from before, and the documented minimum of ftk_ARGBatchSize() = 1 now genuinely yields 1-subscription batches.

I could not validate this bound against a live large tenant. This sandboxed environment has no Azure Resource Graph access, no Power BI Desktop, and (confirmed by checking Package-Toolkit.ps1) no CLI-based way to refresh or measure a .tmdl dataset - PBIX files in this repo are explicitly a manual "open in Power BI Desktop and save" step. The new default is a reasoned, conservative choice based on the historical evidence above, not a measured one. If you have access to a tenant with a large number of NSG rules, a manual refresh to confirm the new default (or tune it further) would still be valuable.

3. Medium: validation evidence for the core M change is absent

This is a fair gap I can't close from here for the same reason as #2 - there's no automated Power BI/M test harness in this repo (Build-PowerBI.ps1 doesn't validate query correctness, and per Package-Toolkit.ps1, .pbip projects require manual Power BI Desktop steps). What I can offer is: I hand-verified the M syntax (paren/bracket balance, control flow) of every change in this PR, and reasoned through the batching/scoping logic against the known ARG resource ID shapes. I have not performed, and cannot perform, a live refresh against a representative tenant. That verification still needs a human with Power BI Desktop and ARG access before merge.

[flanakin]

🤖 [AI][Claude] PR Review

Summary: The batching approach is sound and the hardening from earlier review rounds (batch-size validation, try...otherwise on the subscription list, restored RI filter) is solid. Three items need attention before merge — most importantly registering the two new functions in the Power BI build script, and the direction of the batchMultiplier argument. A few items are confirmation requests rather than defects.

🚫 Blockers (3)

1. Build-PowerBI.ps1 needs ftk_QueryARG and ftk_ARGBatchSize added to the Governance and WorkloadOptimization Expressions lists so they survive the build filter.
2. batchMultiplier scales batches up rather than down, so the heaviest tables issue the largest requests.
3. Confirm the storage .tmdl → symlink conversion is intentional and works on Windows + the build scripts.

⚠️ Should fix (3)

4. AdvisorRecommendations changed resultTruncated from true to false.
5. In-query subscriptionId filtering trims results but doesn't reduce Resource Graph's scan scope.
6. Confirm the AdvisorReservationRecommendations column/typing additions are intended to ship in this PR.

💡 Suggestions (2)

7. Reference the default batch size instead of duplicating the 100 literal.
8. Apply the same try...otherwise guard to the UnscopedSource query.

[RolandKrummenacher]

Overview

The batching approach is sound and ftk_QueryARG is correctly written M. My concerns are about failure modes, request volume, and scope rather than the core idea.

I've left inline comments on the specific lines. Summary of what I think blocks merge:

1. A failed subscription enumeration silently produces an empty report rather than an error — strictly worse than the error it replaces.
2. ~6x more ARG requests with no retry, in exactly the large tenants this targets. Trades a deterministic 16 MB error for a nondeterministic 429.
3. Resource-group-scoped readers lose data, because they can't see the subscription containers the batch list is built from.

And a scope request: this is at least three PRs. Beyond batching it rewrites AdvisorReservationRecommendations' schema (+16 columns, lookbackPeriod string→int64, coalesce(…,0) on savings amounts, a new recommendedActions fallback), switches properties['category'] to properties.category, converges the entire storage dataset onto the kql dataset's content, and churns several hundred lineageTag GUIDs. Those may all be wanted, but they're unreviewable next to the batching change and will make bisecting a future refresh regression painful. I'd like the AdvisorReservationRecommendations schema work split out.

Note that every finding on src/power-bi/kql/.../expressions.tmdl applies identically to the duplicated copy in src/power-bi/storage/.../expressions.tmdl — I only commented once per issue to keep this readable.

Verified as fine

So these don't come up again: changelog placement is correct (v15 is still 15.0.0-dev.0; the "Released June 2026" header is a placeholder). The errors.md anchor resolves. Every batched query either uses extend-only against resources/advisorresources/policyResources — which carry subscriptionId natively — or explicitly projects it, so the appended filter always resolves; I checked all twelve. The broken symlinks flagged earlier are gone. ManagementGroups correctly stays unbatched. {0..-1} is a valid empty range in M, so the zero-subscription path doesn't throw.

Testing

@MSBrett's note that none of this ran against a live tenant is the crux. Pester passes but never touches these files, and findings 1–3 are all runtime-only failure modes that no static check would catch. A manual refresh against a genuinely multi-batch tenant (>100 subscriptions, ideally >500) is the real gate here — specifically watching for 429s during concurrent partition refresh, and confirming the tables aren't silently empty.