feat(transformations): deploy OneLake external data sources - #3099
feat(transformations): deploy OneLake external data sources#3099Khyat-Cognite wants to merge 15 commits into
Conversation
Wire TransformationExternalDataSourcesAPI, ExternalDataSourceIO, ACLs, ext_onelake() dependency ordering, dump support, and unit tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Add example module YAML/SQL under examples/transformations and a Fabric-gated integration lifecycle test. Co-authored-by: Cursor <cursoragent@cursor.com>
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. |
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for Fabric OneLake external data sources used by transformations, adding the TransformationExternalDataSourcesAPI client, the ExternalDataSourceIO resource handler, corresponding ACLs, and integration with TransformationIO to automatically detect dependencies via SQL query parsing. The review feedback suggests adhering to the repository style guide by using absolute imports, safely retrieving external IDs from dictionaries to prevent KeyErrors, optimizing the retrieve method with an early return for empty inputs, and correctly supporting dataset filtering in _iterate to avoid breaking CLI commands like dump or pull.
Use format as a Pydantic field, fix annotation shadowing on Python 3.12, harden ExternalDataSourceIO retrieve/iterate/get_id, add ACL Action enum, and fix test lint issues. Co-authored-by: Cursor <cursoragent@cursor.com>
Fix Python 3.12 annotation shadowing, harden ExternalDataSourceIO, add ACL Action enum, and use absolute imports where requested. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request adds support for Fabric OneLake external data sources used by transformations, introducing the TransformationExternalDataSourcesAPI client, TransformationsExternalDataSourcesAcl capability, and ExternalDataSourceIO resource handler. Feedback on the changes highlights an issue in ExternalDataSourceIO where self.client.tool is incorrectly used to access datasets instead of using self.client.data_sets directly.
| data_sets = self.client.tool.datasets.retrieve( | ||
| [ExternalId(external_id=data_set_external_id)], ignore_unknown_ids=True | ||
| ) | ||
| if not data_sets: | ||
| raise ToolkitRequiredValueError(f"DataSet {data_set_external_id!r} does not exist") | ||
| data_set = data_sets[0] |
There was a problem hiding this comment.
The self.client.tool object does not have a datasets attribute. Core CDF resources like datasets should be accessed directly via self.client.data_sets. Additionally, retrieving a single dataset by external ID can be simplified using self.client.data_sets.retrieve(external_id=...) which returns None if not found.
| data_sets = self.client.tool.datasets.retrieve( | |
| [ExternalId(external_id=data_set_external_id)], ignore_unknown_ids=True | |
| ) | |
| if not data_sets: | |
| raise ToolkitRequiredValueError(f"DataSet {data_set_external_id!r} does not exist") | |
| data_set = data_sets[0] | |
| data_set = self.client.data_sets.retrieve(external_id=data_set_external_id) | |
| if not data_set: | |
| raise ToolkitRequiredValueError(f"DataSet {data_set_external_id!r} does not exist") |
Only wire ExternalDataSourceIO via ext_onelake dependencies, scope dump to referenced sources, exclude ACL from dataset_configurations until SDK ships, and update complete_org loader coverage test. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove examples and integration test from the deploy PR branch; they belong in the stacked docs PR. Co-authored-by: Cursor <cursoragent@cursor.com>
Use Sequence in the paginated return annotation so mypy does not resolve list to the API method on Python 3.12. Co-authored-by: Cursor <cursoragent@cursor.com>
Add API client, ExternalDataSourceIO, ext_onelake() deploy ordering, dump support, and unit tests. Skip auth/ACL checks when the capability is unavailable or no resources are loaded. Co-authored-by: Cursor <cursoragent@cursor.com>
Merge clean deploy restack with optional client_secret, auth prerequisite_warning, and skip ACL validation when no resources are loaded. Co-authored-by: Cursor <cursoragent@cursor.com>
Skip ExternalDataSourceIO in cdf auth required-capability checks because transformationsExternalDataSourcesAcl is not yet in FlatCapabilities. Co-authored-by: Cursor <cursoragent@cursor.com>
☂️ Code Coverage
Overall Coverage
New Files
Modified Files
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## feat/toolkit-onelake-edc-models #3099 +/- ##
===================================================================
- Coverage 86.02% 85.99% -0.04%
===================================================================
Files 477 479 +2
Lines 44215 44336 +121
===================================================================
+ Hits 38036 38125 +89
- Misses 6179 6211 +32
🚀 New features to boost your workflow:
|
Co-authored-by: Cursor <cursoragent@cursor.com>
1 participant
Description
Add deploy/pull support for Fabric OneLake external data sources in transformations.
Slice 2/3 — stacked on #3098.
Bump
Changelog
Added
TransformationExternalDataSourcesAPI(list/upsert/delete/verify_usability) nested undertool.transformationsExternalDataSourceIOfortransformations/*.ExternalDataSource.yamldeploy/dump/pullext_onelake()parsing in transformation SQL for dependency orderingTransformationsExternalDataSourcesAcland access validationMade with Cursor