diff --git a/fs_attachment_azure/README.rst b/fs_attachment_azure/README.rst new file mode 100644 index 0000000000..81213372d7 --- /dev/null +++ b/fs_attachment_azure/README.rst @@ -0,0 +1,209 @@ +=================== +Fs Attachment Azure +=================== + +.. + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + !! This file is generated by oca-gen-addon-readme !! + !! changes will be overwritten. !! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + !! source digest: sha256:c01d32f225802fc30d7d79a6130c073016c0d98c69ba20dd6d6c0d1213e9f92d + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png + :target: https://odoo-community.org/page/development-status + :alt: Beta +.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png + :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html + :alt: License: AGPL-3 +.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fstorage-lightgray.png?logo=github + :target: https://github.com/OCA/storage/tree/17.0/fs_attachment_azure + :alt: OCA/storage +.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png + :target: https://translation.odoo-community.org/projects/storage-17-0/storage-17-0-fs_attachment_azure + :alt: Translate me on Weblate +.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png + :target: https://runboat.odoo-community.org/builds?repo=OCA/storage&target_branch=17.0 + :alt: Try me on Runboat + +|badge1| |badge2| |badge3| |badge4| |badge5| + +This module extends the functionality of +`fs_attachment `__ +to better support Azure storage. It includes features such as: + +- Special handling of X-Accel-Redirect headers for Azure storages. +- Options for using signed URLs in X-Accel-Redirect. (This is required + to be able to serve files from a private Azure Blob Storage using + X-Accel-Redirect without exposing the files publicly.) +- Bulk deletion of the orphaned files during the garbage collection, + using Azure blob batch requests instead of one request per file. + +**Table of contents** + +.. contents:: + :local: + +Configuration +============= + +On the Odoo instance, go to *Settings* > *Technical* > *Storage* > *File +Storage*. + +When you create a new storage for Azure or modify an existing one, when +you activate the option "Use X-Sendfile To Serve Internal Url", 3 +additional fields will appear: + +- **Azure Uses Signed URL For X-Accel-Redirect**: If checked, the + X-Accel-Redirect path will be a signed URL, which is useful for Azure + storages that require signed URLs for access. +- **Azure Signed URL Expiration**: The expiration time for the signed + URL in seconds. This field is only relevant if the previous option is + checked. By default, it is set to 30 seconds but it could be less + since the url generated into the X-Accel-Redirect process is directly + used by the web server to serve the file. +- **Azure Delegation Key Expiration**: The lifetime of the user + delegation key, in seconds. This field is only relevant when the + storage authenticates with an identity (see below). By default it is + set to 1 hour, and Azure does not allow more than 7 days. + +The value of these fields can also be set in the server environment, by +installing the *fs_attachment_azure_environment* glue module and using +the keys: + +- *azure_uses_signed_url_for_x_sendfile* +- *azure_signed_url_expiration* +- *azure_delegation_key_expiration* + +When the option "Use X-Sendfile To Serve Internal Url" is enabled, the +system will generate an X-Accel-Redirect header in the response to a +request to get a file. In the case of Azure storages, it will follow the +format: + +.. code:: text + + X-Accel-Redirect: /fs_x_sendfile/{scheme}/{host}/{path with query if any} + +Where: + +- ``{scheme}``: The URL scheme (http or https). +- ``{host}``: The host of the Azure storage. +- ``{path with query if any}``: The path to the file in the Azure + storage, including any query parameters. (Query parameters are set + when the ``azure_uses_signed_url_for_x_sendfile`` option is enabled.) + +In order to serve files using X-Accel-Redirect, you must ensure that +your web server is configured to handle these headers correctly. This +typically involves setting up a location block in your web server +configuration that matches the X-Accel-Redirect path and proxies the +request to the Azure storage. + +For example, if you are using Nginx, you would add a location block like +this: + +.. code:: nginx + + + location ~ ^/fs_x_sendfile/(.*?)/(.*?)/(.*) { + internal; + set $url_scheme $1; + set $url_host $2; + set $url_path $3; + set $url $url_scheme://$url_host/$url_path; + + proxy_pass $url$is_args$args; + proxy_set_header Host $url_host; + proxy_ssl_server_name on; + + } + +Unlike the standard implementation of X-Accel-Redirect on non Azure +storages, the Azure implementation does not require a base URL to be set +in the storage configuration. The X-Accel-Redirect path is constructed +directly from the Azure storage's URL defined for the connection, the +directory name as bucket name, and the file path. + +Signing with an identity +------------------------ + +Signed URLs are generated with the account shared key when the storage +is configured with a connection string or an account name/key pair. +Otherwise (managed identity, workload identity, service principal, ...) +they are signed with a *user delegation key*, which requires the +identity to have the **Storage Blob Delegator** role on the storage +account, on top of a data plane role such as *Storage Blob Data Reader*. + +Delegation keys are obtained from Azure with an extra request, so they +are kept in the cache of the Odoo registry for **Azure Delegation Key +Expiration** seconds. They are requested from Azure for slightly longer +than that, so that a key served from the cache still covers the URLs +signed with it. + +A higher value means fewer requests to Azure, but also a longer window +during which the key of a revoked identity remains usable. Note that a +key already issued by Azure stays valid until it expires anyway, +whatever Odoo does with its copy. + +The cache is only kept in memory, so it is never shared between +processes nor persisted in the database, and it is dropped whenever Odoo +clears the cache of the registry, which modifying a storage does. +Reconfiguring a storage is therefore applied right away. + +Changelog +========= + +17.0.1.0.0 (2026-07-13) +----------------------- + +- This module was "forked" from fs_attachment_s3 v17.0.1.2.1 + +Bug Tracker +=========== + +Bugs are tracked on `GitHub Issues `_. +In case of trouble, please check there if your issue has already been reported. +If you spotted it first, help us to smash it by providing a detailed and welcomed +`feedback `_. + +Do not contact contributors directly about support or help with technical issues. + +Credits +======= + +Authors +------- + +* ACSONE SA/NV +* Camptocamp + +Contributors +------------ + +- Laurent Mignon laurent.mignon@acsone.eu (https://www.acsone.eu) +- Stéphane Bidoul stephane.bidoul@acsone.eu (https://www.acsone.eu) +- Akim Juillerat akim.juillerat@camptocamp.com + +Maintainers +----------- + +This module is maintained by the OCA. + +.. image:: https://odoo-community.org/logo.png + :alt: Odoo Community Association + :target: https://odoo-community.org + +OCA, or the Odoo Community Association, is a nonprofit organization whose +mission is to support the collaborative development of Odoo features and +promote its widespread use. + +.. |maintainer-grindtildeath| image:: https://github.com/grindtildeath.png?size=40px + :target: https://github.com/grindtildeath + :alt: grindtildeath + +Current `maintainer `__: + +|maintainer-grindtildeath| + +This module is part of the `OCA/storage `_ project on GitHub. + +You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute. diff --git a/fs_attachment_azure/__init__.py b/fs_attachment_azure/__init__.py new file mode 100644 index 0000000000..0650744f6b --- /dev/null +++ b/fs_attachment_azure/__init__.py @@ -0,0 +1 @@ +from . import models diff --git a/fs_attachment_azure/__manifest__.py b/fs_attachment_azure/__manifest__.py new file mode 100644 index 0000000000..8244fc6974 --- /dev/null +++ b/fs_attachment_azure/__manifest__.py @@ -0,0 +1,22 @@ +# Copyright 2025 ACSONE SA/NV +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl). + +{ + "name": "Fs Attachment Azure", + "summary": """Store attachments into Azure Blob storage""", + "version": "17.0.1.0.0", + "license": "AGPL-3", + "author": "ACSONE SA/NV,Camptocamp,Odoo Community Association (OCA)", + "website": "https://github.com/OCA/storage", + "depends": ["fs_attachment"], + "external_dependencies": { + "python": [ + "adlfs", + ], + }, + "data": [ + "views/fs_storage.xml", + ], + "maintainers": ["grindtildeath"], +} diff --git a/fs_attachment_azure/models/__init__.py b/fs_attachment_azure/models/__init__.py new file mode 100644 index 0000000000..2dbbbdd091 --- /dev/null +++ b/fs_attachment_azure/models/__init__.py @@ -0,0 +1,3 @@ +from . import fs_storage +from . import fs_file_gc +from . import ir_attachment diff --git a/fs_attachment_azure/models/fs_file_gc.py b/fs_attachment_azure/models/fs_file_gc.py new file mode 100644 index 0000000000..e1d945489e --- /dev/null +++ b/fs_attachment_azure/models/fs_file_gc.py @@ -0,0 +1,106 @@ +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl). +import logging + +from odoo import models + +from .fs_storage import AZURE_MAX_BLOBS_PER_BATCH + +_logger = logging.getLogger(__name__) + + +class FsFileGc(models.Model): + _inherit = "fs.file.gc" + + # One batch request is sent per iteration, so Azure's own limit is the + # most files we can handle at once. + _GC_BATCH_SIZE = AZURE_MAX_BLOBS_PER_BATCH + + def _gc_files_unsafe(self) -> None: + """Collect the Azure storages by batch before the file by file cleanup. + + Deleting blobs one by one is one request per file, which does not + scale to a large backlog. Azure deletes up to 256 blobs per request, + so the Azure storages are collected that way first and ``super()`` is + left with the rest: the other storages, and the blobs the batch could + not delete, which it retries one by one. + """ + self._gc_azure_bulk_delete() + return super()._gc_files_unsafe() + + def _gc_azure_bulk_delete(self) -> None: + """Delete the orphaned blobs of every Azure storage, by batch.""" + # autovacuum_gc is not a stored field, so the storages are filtered + # in memory. + storages = ( + self.env["fs.storage"] + .search([]) + .filtered( + lambda storage: storage.autovacuum_gc and storage.is_azure_storage + ) + ) + for storage in storages: + try: + self._gc_azure_bulk_delete_storage(storage) + except Exception: + _logger.exception( + "GC: could not batch delete the blobs of the storage %s", + storage.code, + ) + + def _gc_azure_bulk_delete_storage(self, storage) -> None: + """Delete the orphaned blobs of one Azure storage, one batch at a time.""" + while True: + self._cr.execute( + """ + SELECT + store_fname + FROM + fs_file_gc + WHERE + fs_storage_code = %s + AND NOT EXISTS ( + SELECT 1 + FROM ir_attachment + WHERE store_fname = fs_file_gc.store_fname + ) + LIMIT %s + """, + (storage.code, self._GC_BATCH_SIZE), + ) + store_fnames = [row[0] for row in self._cr.fetchall()] + if not store_fnames: + return + blob_names = [ + store_fname.partition("://")[2] for store_fname in store_fnames + ] + _logger.info( + "GC: batch deleting %s blobs of the storage %s", + len(blob_names), + storage.code, + ) + collected = set(storage._azure_delete_blobs(blob_names)) + deleted = [ + store_fname + for store_fname, blob_name in zip(store_fnames, blob_names, strict=True) + if blob_name in collected + ] + if deleted: + self._cr.execute( + """ + DELETE FROM + fs_file_gc + WHERE + store_fname = ANY(%s) + """, + (deleted,), + ) + if not self._is_test_mode(): + # Commit each batch, so that the progress is kept even if a + # later batch fails, and the locks taken by _gc_files are not + # held for the whole backlog. + self._cr.commit() # pylint: disable=invalid-commit + if len(deleted) < len(store_fnames): + # The blobs that could not be deleted would be selected again + # by the query above: leave them to the file by file cleanup. + return diff --git a/fs_attachment_azure/models/fs_storage.py b/fs_attachment_azure/models/fs_storage.py new file mode 100644 index 0000000000..5d9c7bac97 --- /dev/null +++ b/fs_attachment_azure/models/fs_storage.py @@ -0,0 +1,173 @@ +# Copyright 2025 ACSONE SA/NV +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl). + +import datetime + +import fsspec.asyn + +from odoo import _, api, fields, models +from odoo.exceptions import ValidationError + +from ..tools import ormcache_expiring + +# Start times are backdated by this many seconds, to tolerate clock skew +# between this host and Azure. +AZURE_CLOCK_SKEW_TOLERANCE = 300 + +# Azure refuses to issue a user delegation key valid for more than 7 days. +AZURE_MAX_DELEGATION_KEY_EXPIRATION = 7 * 24 * 3600 + +# Azure accepts at most 256 subrequests in a single blob batch request. +AZURE_MAX_BLOBS_PER_BATCH = 256 + + +async def _azure_delete_blobs_batch(service_client, container_name, blob_names): + """Delete blobs in one batch request, returning a status code per blob. + + The container client is used as a context manager, the same way adlfs + does when it needs a blob client. + """ + async with service_client.get_container_client(container_name) as container_client: + responses = await container_client.delete_blobs( + *blob_names, + # A single blob that cannot be deleted must not discard the whole + # batch: the status of each subrequest is inspected instead. + raise_on_any_failure=False, + ) + return [response.status_code async for response in responses] + + +class FsStorage(models.Model): + _inherit = "fs.storage" + + azure_uses_signed_url_for_x_sendfile = fields.Boolean( + string="Use signed URL for X-Accel-Redirect", + help="If checked, the storage will use signed URLs for attachments " + "when using X-Accel-Redirect. This is useful for Azure storage where the " + "file path is not directly accessible without authentication.", + ) + azure_signed_url_expiration = fields.Integer( + string="Signed URL Expiration (seconds)", + default=30, + help="The expiration time for the signed URL in seconds. " + "Default is 30 seconds.", + ) + azure_delegation_key_expiration = fields.Integer( + string="Delegation Key Expiration (seconds)", + default=3600, + help="How long the user delegation key used to sign URLs is cached, when " + "the storage authenticates with an identity instead of a shared key. A " + "higher value means fewer calls to Azure, but a longer window during " + "which the key of a revoked identity remains usable. Azure does not " + "issue keys valid for more than 7 days, which this and the signed URL " + "expiration must leave room for.", + ) + + @api.constrains("azure_delegation_key_expiration", "azure_signed_url_expiration") + def _check_azure_delegation_key_expiration(self): + for rec in self: + if rec.azure_delegation_key_expiration <= 0: + raise ValidationError( + _("The delegation key expiration must be at least 1 second.") + ) + # See _azure_get_user_delegation_key for the lifetime the key is + # requested for. + requested = ( + rec.azure_delegation_key_expiration + + rec.azure_signed_url_expiration + + AZURE_CLOCK_SKEW_TOLERANCE + ) + if requested > AZURE_MAX_DELEGATION_KEY_EXPIRATION: + raise ValidationError( + _( + "Azure does not issue delegation keys valid for more than " + "7 days. The delegation key expiration, the signed URL " + "expiration and a %(skew)s seconds clock skew tolerance " + "must not add up to more than %(max)s seconds.", + skew=AZURE_CLOCK_SKEW_TOLERANCE, + max=AZURE_MAX_DELEGATION_KEY_EXPIRATION, + ) + ) + + @property + def is_azure_storage(self): + """Check if the storage is an Azure storage.""" + self.ensure_one() + fs = self._get_root_filesystem(self.fs) + protocol = getattr(fs, "protocol", []) + return self.protocol in protocol + + @api.model + def _azure_call_synchronous(self, azure_client_function, *args, **kwargs): + # adlfs uses asynchronous client + # We need to run the async function in a synchronous context. + return fsspec.asyn.sync( + fsspec.asyn.get_loop(), + azure_client_function, + *args, + timeout=None, + **kwargs, + ) + + @ormcache_expiring( + "self.id", + "service_client.url", + expiration="self.azure_delegation_key_expiration", + ) + def _azure_get_user_delegation_key(self, service_client): + """Return a user delegation key to sign URLs for this storage. + + The key is cached for ``azure_delegation_key_expiration`` seconds and + shared by all the attachments of the storage. It is requested for a + bit longer than that, so that it still covers the signatures generated + by the very last call served from the cache. + + Getting such a key requires the identity to have the "Storage Blob + Delegator" role on the storage account, on top of a data plane role. + """ + self.ensure_one() + now = datetime.datetime.now(datetime.timezone.utc) + skew = datetime.timedelta(seconds=AZURE_CLOCK_SKEW_TOLERANCE) + expiry_time = ( + now + + datetime.timedelta(seconds=self.azure_delegation_key_expiration) + + datetime.timedelta(seconds=self.azure_signed_url_expiration) + + skew + ) + return self._azure_call_synchronous( + service_client.get_user_delegation_key, + key_start_time=now - skew, + key_expiry_time=expiry_time, + ) + + def _azure_delete_blobs(self, blob_names): + """Delete the given blobs from this storage's container. + + The deletion is sent as a single batch request, so the caller is the + one splitting larger lists, one batch at a time. + + :return: the blobs that are gone, either because they were deleted or + because they were already missing, which is as good as deleted. + :raise ValueError: if more blobs than a batch can hold are given. + """ + self.ensure_one() + if not blob_names: + return [] + if len(blob_names) > AZURE_MAX_BLOBS_PER_BATCH: + raise ValueError( + f"A blob batch request holds at most " + f"{AZURE_MAX_BLOBS_PER_BATCH} blobs, got {len(blob_names)}." + ) + root_fs = self._get_root_filesystem() + status_codes = self._azure_call_synchronous( + _azure_delete_blobs_batch, + root_fs.service_client, + self.get_directory_path(), + blob_names, + ) + return [ + blob_name + for blob_name, status_code in zip(blob_names, status_codes, strict=True) + if 200 <= status_code < 300 or status_code == 404 + ] diff --git a/fs_attachment_azure/models/ir_attachment.py b/fs_attachment_azure/models/ir_attachment.py new file mode 100644 index 0000000000..4b90298f06 --- /dev/null +++ b/fs_attachment_azure/models/ir_attachment.py @@ -0,0 +1,100 @@ +# Copyright 2025 ACSONE SA/NV +# Copyright 2025 XCG SAS +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl). +import datetime +from urllib.parse import urlparse + +from adlfs.spec import BlobSasPermissions, generate_blob_sas + +from odoo import models + + +class IrAttachment(models.Model): + _inherit = "ir.attachment" + + def _get_x_sendfile_path(self): + self.ensure_one() + storage = self.fs_storage_id + if storage.is_azure_storage: + return self._get_azure_x_sendfile_path() + return super()._get_x_sendfile_path() + + def _fs_use_x_sendfile(self): + self.ensure_one() + storage = self.fs_storage_id + if storage.is_azure_storage: + return storage.use_x_sendfile_to_serve_internal_url + return super()._fs_use_x_sendfile() + + def _get_azure_x_sendfile_path(self): + """Generate the X-Accel-Redirect path for Azure storage. + + This method is used to generate the path for Azure storage when using + X-Accel-Redirect. It constructs the path based on the Azure container and + file path, ensuring that it is compatible with the Azure storage + configuration and the Odoo file storage system. + + Args: + attachment (IrAttachment): The attachment record for which the + X-Accel-Redirect path is being generated. + Returns: + str: The X-Accel-Redirect path for the Azure storage. + + The path is formatted as: + /fs_x_sendfile/// + + where: + - `` is the scheme of the base URL (e.g., 'https'). + - `` is the netloc of the base URL + (e.g., 'myaccount.blob.core.windows.net'). + - `` is the path to the file in the Azure container, including the + container name + """ + fs, storage_code, file_path = self._get_fs_parts() + storage = self.env["fs.storage"].sudo().get_by_code(storage_code) + root_fs = storage._get_root_filesystem(fs) + azure_client = root_fs.service_client + container_name = storage.get_directory_path() + blob_client = azure_client.get_blob_client(container_name, file_path) + if storage.azure_uses_signed_url_for_x_sendfile: + if root_fs.connection_string or ( + root_fs.account_name and root_fs.account_key + ): + # AzureBlobFileSystem.url() signs with the shared key. + file_url = root_fs.url( + f"{container_name}/{file_path}", + expires=storage.azure_signed_url_expiration, + ) + else: + # Ideally we would be able to call root_fs.url() as it is calling + # generate_blob_sas. However, it expects to use an account shared key + # (i.e either a connection string or account name/key pair). + # For this we need a user delegation key first. + delegation_key = storage._azure_get_user_delegation_key(azure_client) + # Then we can call generate_blob_sas. No start time is given, so + # that the signature is valid as soon as Azure receives it + # whatever the clock skew between this host and Azure. This is + # also what adlfs does when signing with a shared key. + expiry_time = datetime.datetime.now( + datetime.timezone.utc + ) + datetime.timedelta(seconds=storage.azure_signed_url_expiration) + sas_token = generate_blob_sas( + account_name=blob_client.account_name, + container_name=container_name, + blob_name=file_path, + user_delegation_key=delegation_key, + permission=BlobSasPermissions(read=True), + expiry=expiry_time, + ) + file_url = f"{blob_client.url}?{sas_token}" + else: + file_url = blob_client.url + + parsed_url = urlparse(file_url) + path = parsed_url.path.strip("/") + query = parsed_url.query + redirect_path = f"/fs_x_sendfile/{parsed_url.scheme}/{parsed_url.netloc}/{path}" + if query: + redirect_path += f"?{query}" + return redirect_path diff --git a/fs_attachment_azure/pyproject.toml b/fs_attachment_azure/pyproject.toml new file mode 100644 index 0000000000..4231d0cccb --- /dev/null +++ b/fs_attachment_azure/pyproject.toml @@ -0,0 +1,3 @@ +[build-system] +requires = ["whool"] +build-backend = "whool.buildapi" diff --git a/fs_attachment_azure/readme/CONFIGURE.md b/fs_attachment_azure/readme/CONFIGURE.md new file mode 100644 index 0000000000..6c9b3f2d62 --- /dev/null +++ b/fs_attachment_azure/readme/CONFIGURE.md @@ -0,0 +1,91 @@ +On the Odoo instance, go to *Settings* > *Technical* > *Storage* > *File Storage*. + +When you create a new storage for Azure or modify an existing one, when you activate +the option "Use X-Sendfile To Serve Internal Url", 3 additional fields will appear: + +- **Azure Uses Signed URL For X-Accel-Redirect**: If checked, the X-Accel-Redirect + path will be a signed URL, which is useful for Azure storages that require + signed URLs for access. +- **Azure Signed URL Expiration**: The expiration time for the signed URL in seconds. + This field is only relevant if the previous option is checked. By default, + it is set to 30 seconds but it could be less since the url generated into + the X-Accel-Redirect process is directly used by the web server to serve the file. +- **Azure Delegation Key Expiration**: The lifetime of the user delegation key, in + seconds. This field is only relevant when the storage authenticates with an + identity (see below). By default it is set to 1 hour, and Azure does not allow + more than 7 days. + +The value of these fields can also be set in the server environment, by installing +the *fs_attachment_azure_environment* glue module and using the keys: + +- *azure_uses_signed_url_for_x_sendfile* +- *azure_signed_url_expiration* +- *azure_delegation_key_expiration* + +When the option "Use X-Sendfile To Serve Internal Url" is enabled, the system will +generate an X-Accel-Redirect header in the response to a request to get a file. +In the case of Azure storages, it will follow the format: + +```text +X-Accel-Redirect: /fs_x_sendfile/{scheme}/{host}/{path with query if any} +``` + +Where: + +- `{scheme}`: The URL scheme (http or https). +- `{host}`: The host of the Azure storage. +- `{path with query if any}`: The path to the file in the Azure storage, + including any query parameters. (Query parameters are set when the + `azure_uses_signed_url_for_x_sendfile` option is enabled.) + +In order to serve files using X-Accel-Redirect, you must ensure that your +web server is configured to handle these headers correctly. This typically +involves setting up a location block in your web server configuration that +matches the X-Accel-Redirect path and proxies the request to the Azure storage. + +For example, if you are using Nginx, you would add a location block like this: + +```nginx + + location ~ ^/fs_x_sendfile/(.*?)/(.*?)/(.*) { + internal; + set $url_scheme $1; + set $url_host $2; + set $url_path $3; + set $url $url_scheme://$url_host/$url_path; + + proxy_pass $url$is_args$args; + proxy_set_header Host $url_host; + proxy_ssl_server_name on; + + } +``` + + +Unlike the standard implementation of X-Accel-Redirect on non Azure storages, +the Azure implementation does not require a base URL to be set in the storage +configuration. The X-Accel-Redirect path is constructed directly from the +Azure storage's URL defined for the connection, the directory name as +bucket name, and the file path. + +## Signing with an identity + +Signed URLs are generated with the account shared key when the storage is configured +with a connection string or an account name/key pair. Otherwise (managed identity, +workload identity, service principal, ...) they are signed with a *user delegation +key*, which requires the identity to have the **Storage Blob Delegator** role on the +storage account, on top of a data plane role such as *Storage Blob Data Reader*. + +Delegation keys are obtained from Azure with an extra request, so they are kept in +the cache of the Odoo registry for **Azure Delegation Key Expiration** seconds. They +are requested from Azure for slightly longer than that, so that a key served from the +cache still covers the URLs signed with it. + +A higher value means fewer requests to Azure, but also a longer window during which +the key of a revoked identity remains usable. Note that a key already issued by Azure +stays valid until it expires anyway, whatever Odoo does with its copy. + +The cache is only kept in memory, so it is never shared between processes nor +persisted in the database, and it is dropped whenever Odoo clears the cache of the +registry, which modifying a storage does. Reconfiguring a storage is therefore +applied right away. diff --git a/fs_attachment_azure/readme/CONTRIBUTORS.md b/fs_attachment_azure/readme/CONTRIBUTORS.md new file mode 100644 index 0000000000..cb7b22f28c --- /dev/null +++ b/fs_attachment_azure/readme/CONTRIBUTORS.md @@ -0,0 +1,3 @@ +- Laurent Mignon (https://www.acsone.eu) +- Stéphane Bidoul (https://www.acsone.eu) +- Akim Juillerat \ No newline at end of file diff --git a/fs_attachment_azure/readme/DESCRIPTION.md b/fs_attachment_azure/readme/DESCRIPTION.md new file mode 100644 index 0000000000..2f24666552 --- /dev/null +++ b/fs_attachment_azure/readme/DESCRIPTION.md @@ -0,0 +1,8 @@ +This module extends the functionality of [fs_attachment](https://github.com/OCA/storage/tree/16.0/fs_attachment) +to better support Azure storage. It includes features such as: + +- Special handling of X-Accel-Redirect headers for Azure storages. +- Options for using signed URLs in X-Accel-Redirect. (This is required to be able to serve files from a private Azure Blob Storage + using X-Accel-Redirect without exposing the files publicly.) +- Bulk deletion of the orphaned files during the garbage collection, using Azure + blob batch requests instead of one request per file. diff --git a/fs_attachment_azure/readme/HISTORY.md b/fs_attachment_azure/readme/HISTORY.md new file mode 100644 index 0000000000..9ec3de838e --- /dev/null +++ b/fs_attachment_azure/readme/HISTORY.md @@ -0,0 +1,3 @@ +## 17.0.1.0.0 (2026-07-13) + +- This module was "forked" from fs_attachment_s3 v17.0.1.2.1 diff --git a/fs_attachment_azure/readme/newsfragments/.gitkeep b/fs_attachment_azure/readme/newsfragments/.gitkeep new file mode 100644 index 0000000000..e69de29bb2 diff --git a/fs_attachment_azure/static/description/icon.png b/fs_attachment_azure/static/description/icon.png new file mode 100644 index 0000000000..3a0328b516 Binary files /dev/null and b/fs_attachment_azure/static/description/icon.png differ diff --git a/fs_attachment_azure/static/description/index.html b/fs_attachment_azure/static/description/index.html new file mode 100644 index 0000000000..75ac480e96 --- /dev/null +++ b/fs_attachment_azure/static/description/index.html @@ -0,0 +1,548 @@ + + + + + +Fs Attachment Azure + + + +
+

Fs Attachment Azure

+ + +

Beta License: AGPL-3 OCA/storage Translate me on Weblate Try me on Runboat

+

This module extends the functionality of +fs_attachment +to better support Azure storage. It includes features such as:

+
    +
  • Special handling of X-Accel-Redirect headers for Azure storages.
    • +
  • Options for using signed URLs in X-Accel-Redirect. (This is required +to be able to serve files from a private Azure Blob Storage using +X-Accel-Redirect without exposing the files publicly.)
    • +
  • Bulk deletion of the orphaned files during the garbage collection, +using Azure blob batch requests instead of one request per file.
    • +
+

Table of contents

+ +
+

Configuration

+

On the Odoo instance, go to Settings > Technical > Storage > File +Storage.

+

When you create a new storage for Azure or modify an existing one, when +you activate the option “Use X-Sendfile To Serve Internal Url”, 3 +additional fields will appear:

+
    +
  • Azure Uses Signed URL For X-Accel-Redirect: If checked, the +X-Accel-Redirect path will be a signed URL, which is useful for Azure +storages that require signed URLs for access.
    • +
  • Azure Signed URL Expiration: The expiration time for the signed +URL in seconds. This field is only relevant if the previous option is +checked. By default, it is set to 30 seconds but it could be less +since the url generated into the X-Accel-Redirect process is directly +used by the web server to serve the file.
    • +
  • Azure Delegation Key Expiration: The lifetime of the user +delegation key, in seconds. This field is only relevant when the +storage authenticates with an identity (see below). By default it is +set to 1 hour, and Azure does not allow more than 7 days.
    • +
+

The value of these fields can also be set in the server environment, by +installing the fs_attachment_azure_environment glue module and using +the keys:

+
    +
  • azure_uses_signed_url_for_x_sendfile
    • +
  • azure_signed_url_expiration
    • +
  • azure_delegation_key_expiration
    • +
+

When the option “Use X-Sendfile To Serve Internal Url” is enabled, the +system will generate an X-Accel-Redirect header in the response to a +request to get a file. In the case of Azure storages, it will follow the +format:

+
+X-Accel-Redirect: /fs_x_sendfile/{scheme}/{host}/{path with query if any}
+
+

Where:

+
    +
  • {scheme}: The URL scheme (http or https).
    • +
  • {host}: The host of the Azure storage.
    • +
  • {path with query if any}: The path to the file in the Azure +storage, including any query parameters. (Query parameters are set +when the azure_uses_signed_url_for_x_sendfile option is enabled.)
    • +
+

In order to serve files using X-Accel-Redirect, you must ensure that +your web server is configured to handle these headers correctly. This +typically involves setting up a location block in your web server +configuration that matches the X-Accel-Redirect path and proxies the +request to the Azure storage.

+

For example, if you are using Nginx, you would add a location block like +this:

+
+location ~ ^/fs_x_sendfile/(.*?)/(.*?)/(.*) {
+    internal;
+    set $url_scheme $1;
+    set $url_host $2;
+    set $url_path $3;
+    set $url $url_scheme://$url_host/$url_path;
+
+    proxy_pass $url$is_args$args;
+    proxy_set_header Host $url_host;
+    proxy_ssl_server_name on;
+
+}
+
+

Unlike the standard implementation of X-Accel-Redirect on non Azure +storages, the Azure implementation does not require a base URL to be set +in the storage configuration. The X-Accel-Redirect path is constructed +directly from the Azure storage’s URL defined for the connection, the +directory name as bucket name, and the file path.

+
+

Signing with an identity

+

Signed URLs are generated with the account shared key when the storage +is configured with a connection string or an account name/key pair. +Otherwise (managed identity, workload identity, service principal, …) +they are signed with a user delegation key, which requires the +identity to have the Storage Blob Delegator role on the storage +account, on top of a data plane role such as Storage Blob Data Reader.

+

Delegation keys are obtained from Azure with an extra request, so they +are kept in the cache of the Odoo registry for Azure Delegation Key +Expiration seconds. They are requested from Azure for slightly longer +than that, so that a key served from the cache still covers the URLs +signed with it.

+

A higher value means fewer requests to Azure, but also a longer window +during which the key of a revoked identity remains usable. Note that a +key already issued by Azure stays valid until it expires anyway, +whatever Odoo does with its copy.

+

The cache is only kept in memory, so it is never shared between +processes nor persisted in the database, and it is dropped whenever Odoo +clears the cache of the registry, which modifying a storage does. +Reconfiguring a storage is therefore applied right away.

+
+
+
+

Changelog

+
+

17.0.1.0.0 (2026-07-13)

+
    +
  • This module was “forked” from fs_attachment_s3 v17.0.1.2.1
    • +
+
+
+
+

Bug Tracker

+

Bugs are tracked on GitHub Issues. +In case of trouble, please check there if your issue has already been reported. +If you spotted it first, help us to smash it by providing a detailed and welcomed +feedback.

+

Do not contact contributors directly about support or help with technical issues.

+
+
+

Credits

+
+

Authors

+
    +
  • ACSONE SA/NV
    • +
  • Camptocamp
    • +
+
+ +
+

Maintainers

+

This module is maintained by the OCA.

+ +Odoo Community Association + +

OCA, or the Odoo Community Association, is a nonprofit organization whose +mission is to support the collaborative development of Odoo features and +promote its widespread use.

+

Current maintainer:

+

grindtildeath

+

This module is part of the OCA/storage project on GitHub.

+

You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

+
+
+
+ + diff --git a/fs_attachment_azure/tests/__init__.py b/fs_attachment_azure/tests/__init__.py new file mode 100644 index 0000000000..a5227618d9 --- /dev/null +++ b/fs_attachment_azure/tests/__init__.py @@ -0,0 +1,2 @@ +from . import test_fs_attachment_azure +from . import test_fs_file_gc diff --git a/fs_attachment_azure/tests/common.py b/fs_attachment_azure/tests/common.py new file mode 100644 index 0000000000..6d7c12454f --- /dev/null +++ b/fs_attachment_azure/tests/common.py @@ -0,0 +1,47 @@ +# Copyright 2025 ACSONE SA/NV (http://acsone.eu). +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). +from odoo.tests.common import TransactionCase + +from odoo.addons.base.tests.common import DISABLED_MAIL_CONTEXT + + +class TestFSAttachmentAzureCommon(TransactionCase): + @classmethod + def setUpClass(cls): + super().setUpClass() + cls.env = cls.env(context=dict(cls.env.context, **DISABLED_MAIL_CONTEXT)) + cls.azure_backend_config = { + "name": "Azure Storage", + "protocol": "az", + "code": "azure", + "directory_path": "test-blob", + } + cls.azure_backend = cls.env["fs.storage"].create(cls.azure_backend_config) + cls.ir_attachment_model = cls.env["ir.attachment"] + + cls.fake_attachment_azure = cls.env["ir.attachment"].create( + { + "name": "fake_azure_file.txt", + "fs_storage_id": cls.azure_backend.id, + } + ) + cls.fake_attachment_azure.flush_recordset() + # update the attachment into database since we don't have a real blob storage + cls.env.cr.execute( + """ + UPDATE + ir_attachment + SET + store_fname = 'azure://dir/sub/fake_azure_file.txt', + fs_filename = 'fake_azure_file.txt', + fs_storage_code = 'azure', + checksum = 234, + file_size = 1234, + fs_storage_id = %s + WHERE + id = %s + """, + (cls.azure_backend.id, cls.fake_attachment_azure.id), + ) + cls.fake_attachment_azure.invalidate_recordset() diff --git a/fs_attachment_azure/tests/test_fs_attachment_azure.py b/fs_attachment_azure/tests/test_fs_attachment_azure.py new file mode 100644 index 0000000000..f638428bdd --- /dev/null +++ b/fs_attachment_azure/tests/test_fs_attachment_azure.py @@ -0,0 +1,281 @@ +# Copyright 2025 ACSONE SA/NV (http://acsone.eu). +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). +import base64 +import datetime +import time +from contextlib import contextmanager +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch +from urllib.parse import parse_qsl + +from adlfs import AzureBlobFileSystem +from azure.storage.blob import UserDelegationKey + +from odoo.exceptions import ValidationError + +from odoo.addons.fs_attachment_azure import tools +from odoo.addons.fs_attachment_azure.models.fs_storage import ( + AZURE_CLOCK_SKEW_TOLERANCE, + AZURE_MAX_DELEGATION_KEY_EXPIRATION, +) + +from .common import TestFSAttachmentAzureCommon + +PROTOCOL = "https" +ACCOUNT_NAME = "myaccount" +ACCOUNT_KEY = "123456789" +DOMAIN = "blob.core.windows.net" +CONTAINER = "test-blob" +PATH = "dir/sub" +FILENAME = "fake_azure_file.txt" +ACCOUNT_URL = f"{PROTOCOL}://{ACCOUNT_NAME}.{DOMAIN}" +FILE_PATH = f"{PATH}/{FILENAME}" +BASE_URL = f"{ACCOUNT_URL}/{CONTAINER}/{FILE_PATH}" +REDIRECT_PATH = ( + f"/fs_x_sendfile/{PROTOCOL}/{ACCOUNT_NAME}.{DOMAIN}/{CONTAINER}/{FILE_PATH}" +) +TOKEN = "1111-2222-3333-4444" +CONNECTION_STRING = f"DefaultEndpointsProtocol={PROTOCOL};AccountName={ACCOUNT_NAME};AccountKey={ACCOUNT_KEY};BlobEndpoint={PROTOCOL}://{DOMAIN}/{ACCOUNT_NAME};" +DELEGATION_KEY_OID = "00000000-0000-0000-0000-000000000001" + + +def _fake_delegation_key(): + key = UserDelegationKey() + key.signed_oid = DELEGATION_KEY_OID + key.signed_tid = "00000000-0000-0000-0000-000000000002" + key.signed_start = "2026-01-01T00:00:00Z" + key.signed_expiry = "2026-01-02T00:00:00Z" + key.signed_service = "b" + key.signed_version = "2023-11-03" + # The value is used as an HMAC key, so it must be base64 decodable. + key.value = base64.b64encode(b"delegation-key-secret").decode() + return key + + +def _install_service_client(fs): + mock_blob_client = MagicMock() + mock_blob_client.url = BASE_URL + mock_blob_client.account_name = ACCOUNT_NAME + mock_service_client = MagicMock() + mock_service_client.url = ACCOUNT_URL + mock_service_client.get_container_client.return_value = MagicMock() + mock_service_client.get_blob_client.return_value = mock_blob_client + mock_service_client.close = AsyncMock(return_value="ok") + mock_service_client.get_user_delegation_key = AsyncMock( + return_value=_fake_delegation_key() + ) + fs.service_client = mock_service_client + + +def _fake_do_connect_shared_key(self): + """Connect with an account shared key: adlfs can sign URLs by itself.""" + self.connection_string = CONNECTION_STRING + _install_service_client(self) + + +def _fake_do_connect_identity(self): + """Connect with an identity: signing requires a user delegation key.""" + # Reset whatever the environment variables may have provided, so that the + # tests do not depend on the host configuration. + self.connection_string = None + self.account_name = None + self.account_key = None + _install_service_client(self) + + +class TestFSAttachementAzure(TestFSAttachmentAzureCommon): + def setUp(self): + super().setUp() + # The filesystem instances and the delegation keys are cached process + # wide: reset them so that each test connects with its own mock and + # doesn't leak into the next one. + self._clear_caches() + self.addCleanup(self._clear_caches) + + def _clear_caches(self): + # The delegation keys live in the registry cache, the filesystem + # instances in the fsspec one. + self.env.registry.clear_cache() + AzureBlobFileSystem.clear_instance_cache() + + @contextmanager + def _time_advanced_by(self, seconds): + """Run the block as if ``seconds`` had passed, for the caches. + + Only the clock the cache reads is moved: patching time.monotonic + itself would also move the one the asyncio event loop runs on. + """ + later = time.monotonic() + seconds + with patch.object(tools, "time", SimpleNamespace(monotonic=lambda: later)): + yield + + def _get_service_client(self): + """Return the mocked service client used by the storage. + + Must be called while ``do_connect`` is still patched. + """ + fs_storage = self.env["fs.storage"] + fs = fs_storage.get_fs_by_code(self.azure_backend.code) + return fs_storage._get_root_filesystem(fs).service_client + + def _enable_signed_url(self, expiration=60, **vals): + self.azure_backend.write( + { + "azure_uses_signed_url_for_x_sendfile": True, + "azure_signed_url_expiration": expiration, + **vals, + } + ) + + def test_get_x_sendfile_path_azure(self): + """Test the X-Accel-Redirect path generation.""" + with patch.object(AzureBlobFileSystem, "do_connect", _fake_do_connect_identity): + url = self.fake_attachment_azure._get_x_sendfile_path() + service_client = self._get_service_client() + + self.assertEqual( + url, + REDIRECT_PATH, + f"The X-Accel-Redirect path should match the expected format. ({url})", + ) + service_client.get_user_delegation_key.assert_not_awaited() + + def test_get_x_sendfile_path_azure_signed_shared_key(self): + """With a shared key, the URL is signed by adlfs itself.""" + self._enable_signed_url() + with patch.object( + AzureBlobFileSystem, "do_connect", _fake_do_connect_shared_key + ), patch.object( + AzureBlobFileSystem, "url", return_value=f"{BASE_URL}?{TOKEN}" + ) as mock_url: + url = self.fake_attachment_azure._get_x_sendfile_path() + service_client = self._get_service_client() + + mock_url.assert_called_once_with(f"{CONTAINER}/{FILE_PATH}", expires=60) + self.assertEqual(url, f"{REDIRECT_PATH}?{TOKEN}") + # No delegation key is needed to sign with a shared key. + service_client.get_user_delegation_key.assert_not_awaited() + + def test_get_x_sendfile_path_azure_signed_delegation_key(self): + """Without a shared key, the URL is signed with a delegation key.""" + self._enable_signed_url() + before = datetime.datetime.now(datetime.timezone.utc) + with patch.object(AzureBlobFileSystem, "do_connect", _fake_do_connect_identity): + url = self.fake_attachment_azure._get_x_sendfile_path() + service_client = self._get_service_client() + + path, _sep, query = url.partition("?") + self.assertEqual(path, REDIRECT_PATH) + params = dict(parse_qsl(query)) + self.assertEqual( + params.get("skoid"), + DELEGATION_KEY_OID, + f"The signature should be built from the delegation key. ({url})", + ) + self.assertIn("sig", params) + self.assertIn("se", params) + self.assertNotIn( + "st", + params, + "The signature should not have a start time, so that it is valid " + f"as soon as Azure receives it. ({url})", + ) + + key_args = service_client.get_user_delegation_key.await_args.kwargs + self.assertLess( + key_args["key_start_time"], + before, + "The delegation key start time should be backdated to tolerate " + "clock skew.", + ) + self.assertEqual( + key_args["key_expiry_time"] - key_args["key_start_time"], + datetime.timedelta( + seconds=self.azure_backend.azure_delegation_key_expiration + + self.azure_backend.azure_signed_url_expiration + + 2 * AZURE_CLOCK_SKEW_TOLERANCE + ), + "The key must cover the whole time it is cached, plus the " + "signatures generated by the last call served from the cache.", + ) + + def test_delegation_key_is_cached(self): + """The delegation key is fetched once and reused.""" + self._enable_signed_url() + with patch.object(AzureBlobFileSystem, "do_connect", _fake_do_connect_identity): + urls = [ + self.fake_attachment_azure._get_x_sendfile_path() for _i in range(3) + ] + service_client = self._get_service_client() + + self.assertEqual(service_client.get_user_delegation_key.await_count, 1) + for url in urls: + self.assertIn("skoid", url) + + def test_delegation_key_refreshed_once_expired(self): + """The key is fetched again by the first call made after it expired.""" + self._enable_signed_url() + with patch.object(AzureBlobFileSystem, "do_connect", _fake_do_connect_identity): + self.fake_attachment_azure._get_x_sendfile_path() + service_client = self._get_service_client() + self.assertEqual(service_client.get_user_delegation_key.await_count, 1) + # Right before the end of the caching duration, the key is reused. + with self._time_advanced_by( + self.azure_backend.azure_delegation_key_expiration - 1 + ): + self.fake_attachment_azure._get_x_sendfile_path() + self.assertEqual(service_client.get_user_delegation_key.await_count, 1) + # Past it, a new one is requested. + with self._time_advanced_by( + self.azure_backend.azure_delegation_key_expiration + 1 + ): + self.fake_attachment_azure._get_x_sendfile_path() + + self.assertEqual(service_client.get_user_delegation_key.await_count, 2) + + def test_delegation_key_dropped_when_storage_is_written(self): + """Reconfiguring the storage drops the key cached for it. + + This is what makes a new configuration effective right away instead of + at the end of the caching duration. + """ + self._enable_signed_url() + with patch.object(AzureBlobFileSystem, "do_connect", _fake_do_connect_identity): + self.fake_attachment_azure._get_x_sendfile_path() + service_client = self._get_service_client() + self.assertEqual(service_client.get_user_delegation_key.await_count, 1) + self.azure_backend.azure_delegation_key_expiration = 7200 + self.fake_attachment_azure._get_x_sendfile_path() + + self.assertEqual(service_client.get_user_delegation_key.await_count, 2) + + def test_delegation_key_expiration_constraint(self): + """Azure only issues delegation keys valid for up to 7 days.""" + for expiration in (0, -1, AZURE_MAX_DELEGATION_KEY_EXPIRATION): + with self.subTest(expiration=expiration), self.assertRaises( + ValidationError + ), self.env.cr.savepoint(): + self.azure_backend.azure_delegation_key_expiration = expiration + # The signed URL expiration and the clock skew tolerance are part of + # the lifetime the key is requested for, so they leave less room. + with self.assertRaises(ValidationError), self.env.cr.savepoint(): + self.azure_backend.write( + { + "azure_delegation_key_expiration": ( + AZURE_MAX_DELEGATION_KEY_EXPIRATION - AZURE_CLOCK_SKEW_TOLERANCE + ), + "azure_signed_url_expiration": 1, + } + ) + # The largest configuration Azure accepts. + self.azure_backend.write( + { + "azure_delegation_key_expiration": ( + AZURE_MAX_DELEGATION_KEY_EXPIRATION + - AZURE_CLOCK_SKEW_TOLERANCE + - 30 + ), + "azure_signed_url_expiration": 30, + } + ) diff --git a/fs_attachment_azure/tests/test_fs_file_gc.py b/fs_attachment_azure/tests/test_fs_file_gc.py new file mode 100644 index 0000000000..b33210ebba --- /dev/null +++ b/fs_attachment_azure/tests/test_fs_file_gc.py @@ -0,0 +1,184 @@ +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). +from types import SimpleNamespace +from unittest.mock import patch + +from odoo.addons.fs_attachment_azure.models.fs_storage import ( + AZURE_MAX_BLOBS_PER_BATCH, +) + +from .common import TestFSAttachmentAzureCommon + +CONTAINER = "test-blob" + + +class FakeBatchResponses: + """Async iterator of subresponses, as ``delete_blobs`` returns one.""" + + def __init__(self, status_codes): + self._status_codes = list(status_codes) + + def __aiter__(self): + return self + + async def __anext__(self): + if not self._status_codes: + raise StopAsyncIteration + return SimpleNamespace(status_code=self._status_codes.pop(0)) + + +class FakeContainerClient: + """Minimal stand-in for an asynchronous ContainerClient.""" + + def __init__(self, status_codes=None, error=None): + # blob name -> status code to answer, anything else is deleted (202) + self.status_codes = status_codes or {} + self.error = error + self.calls = [] + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc_info): + return None + + async def delete_blobs(self, *blobs, **kwargs): + self.calls.append((blobs, kwargs)) + if self.error: + raise self.error + return FakeBatchResponses([self.status_codes.get(blob, 202) for blob in blobs]) + + +class FakeServiceClient: + def __init__(self, container_client): + self.container_client = container_client + self.container_names = [] + + def get_container_client(self, container_name): + self.container_names.append(container_name) + return self.container_client + + +class TestFsFileGcAzure(TestFSAttachmentAzureCommon): + def setUp(self): + super().setUp() + self.gc_file_model = self.env["fs.file.gc"] + + def _mark_for_gc(self, *store_fnames): + for store_fname in store_fnames: + self.gc_file_model._mark_for_gc(store_fname) + + def _gc_azure_bulk_delete(self, container_client): + """Run the bulk delete against a fake Azure container.""" + root_fs = SimpleNamespace(service_client=FakeServiceClient(container_client)) + storage_class = type(self.azure_backend) + code = self.azure_backend.code + with patch.object( + storage_class, + "is_azure_storage", + new=property(lambda storage: storage.code == code), + ), patch.object(storage_class, "_get_root_filesystem", return_value=root_fs): + self.gc_file_model._gc_azure_bulk_delete() + return root_fs.service_client + + def _gc_rows(self, *store_fnames): + return self.gc_file_model.search( + [("store_fname", "in", list(store_fnames))] + ).mapped("store_fname") + + def test_batch_size_is_azure_limit(self): + """A batch cannot hold more blobs than Azure accepts.""" + self.assertEqual(self.gc_file_model._GC_BATCH_SIZE, AZURE_MAX_BLOBS_PER_BATCH) + self.assertEqual(AZURE_MAX_BLOBS_PER_BATCH, 256) + + def test_delete_blobs_rejects_oversized_batch(self): + """The storage refuses more blobs than a batch request can hold.""" + blob_names = [f"blob_{i}" for i in range(AZURE_MAX_BLOBS_PER_BATCH + 1)] + with self.assertRaises(ValueError): + self.azure_backend._azure_delete_blobs(blob_names) + + def test_delete_blobs_without_blobs_sends_no_request(self): + """Nothing to delete means no request at all.""" + self.assertEqual(self.azure_backend._azure_delete_blobs([]), []) + + def test_bulk_delete_removes_orphaned_files(self): + """Orphaned blobs are deleted in one request, referenced ones are kept.""" + orphan_1 = "azure://dir/sub/orphan_1.txt" + orphan_2 = "azure://dir/sub/orphan_2.txt" + referenced = self.fake_attachment_azure.store_fname + self._mark_for_gc(orphan_1, orphan_2, referenced) + + container_client = FakeContainerClient() + service_client = self._gc_azure_bulk_delete(container_client) + + self.assertEqual(len(container_client.calls), 1) + blobs, kwargs = container_client.calls[0] + self.assertCountEqual(blobs, ["dir/sub/orphan_1.txt", "dir/sub/orphan_2.txt"]) + self.assertFalse( + kwargs["raise_on_any_failure"], + "One blob that cannot be deleted must not discard the batch.", + ) + self.assertEqual(service_client.container_names, [CONTAINER]) + self.assertEqual(self._gc_rows(orphan_1, orphan_2, referenced), [referenced]) + + def test_bulk_delete_collects_missing_blobs(self): + """A blob that is already gone is collected as if it was deleted.""" + orphan = "azure://dir/sub/orphan.txt" + self._mark_for_gc(orphan) + + container_client = FakeContainerClient(status_codes={"dir/sub/orphan.txt": 404}) + self._gc_azure_bulk_delete(container_client) + + self.assertFalse(self._gc_rows(orphan)) + + def test_bulk_delete_keeps_rows_of_failed_blobs(self): + """A blob that Azure refuses to delete is left to the file by file pass.""" + deleted = "azure://dir/sub/deleted.txt" + failed = "azure://dir/sub/failed.txt" + self._mark_for_gc(deleted, failed) + + container_client = FakeContainerClient(status_codes={"dir/sub/failed.txt": 500}) + self._gc_azure_bulk_delete(container_client) + + # A partial failure must not loop on the rows it cannot delete. + self.assertEqual(len(container_client.calls), 1) + self.assertEqual(self._gc_rows(deleted, failed), [failed]) + + def test_bulk_delete_keeps_rows_when_request_fails(self): + """A failing request is logged and leaves every row in place.""" + orphan = "azure://dir/sub/orphan.txt" + self._mark_for_gc(orphan) + + container_client = FakeContainerClient(error=Exception("Azure is unavailable")) + with self.assertLogs( + "odoo.addons.fs_attachment_azure.models.fs_file_gc", level="ERROR" + ): + self._gc_azure_bulk_delete(container_client) + + self.assertEqual(self._gc_rows(orphan), [orphan]) + + def test_bulk_delete_sends_one_request_per_batch(self): + """More orphans than a batch holds are deleted in several requests.""" + orphans = [f"azure://dir/sub/orphan_{i}.txt" for i in range(3)] + self._mark_for_gc(*orphans) + + container_client = FakeContainerClient() + with patch.object(type(self.gc_file_model), "_GC_BATCH_SIZE", 2): + self._gc_azure_bulk_delete(container_client) + + self.assertEqual( + [len(blobs) for blobs, _kwargs in container_client.calls], [2, 1] + ) + self.assertFalse(self._gc_rows(*orphans)) + + def test_bulk_delete_ignores_storages_without_autovacuum(self): + """A storage with the autovacuum disabled is not collected.""" + self.azure_backend.autovacuum_gc = False + orphan = "azure://dir/sub/orphan.txt" + self._mark_for_gc(orphan) + + container_client = FakeContainerClient() + self._gc_azure_bulk_delete(container_client) + + self.assertFalse(container_client.calls) + self.assertEqual(self._gc_rows(orphan), [orphan]) diff --git a/fs_attachment_azure/tools.py b/fs_attachment_azure/tools.py new file mode 100644 index 0000000000..2308dfafb8 --- /dev/null +++ b/fs_attachment_azure/tools.py @@ -0,0 +1,83 @@ +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl). +import logging +import time +from inspect import Parameter, signature + +from odoo.tools import ormcache + +_logger = logging.getLogger(__name__) + +unsafe_eval = eval + + +class ormcache_expiring(ormcache): + """An :class:`~odoo.tools.ormcache` whose entries expire. + + ``odoo.tools.ormcache`` keeps an entry until something clears the registry + caches, which is not enough for a value that is only valid for a while, + such as a credential obtained from a remote service. + + ``expiration`` is the number of seconds an entry remains usable, given + either as a number or, like the cache key parameters, as an expression + evaluated against the signature of the decorated method:: + + @ormcache_expiring("self.id", expiration="self.token_lifetime") + def _get_token(self): + ... + + Expiring is not invalidating: an outdated entry is simply treated as a + miss, so the method is called again by the first caller that needs it + after the entry expired. + """ + + def __init__(self, *args, expiration, **kwargs): + super().__init__(*args, **kwargs) + self.expiration = expiration + + def __call__(self, method): + lookup = super().__call__(method) + self.determine_expiration() + return lookup + + def determine_expiration(self): + """Determine the function that computes the lifetime of an entry.""" + if not isinstance(self.expiration, str): + expiration = self.expiration + self.compute_expiration = lambda *args, **kwargs: expiration + return + # Same approach as ormcache.determine_key: build a lambda over the + # signature of the decorated method and evaluate the expression in it. + args = ", ".join( + str(param.replace(annotation=Parameter.empty, default=Parameter.empty)) + for param in signature(self.method).parameters.values() + ) + self.compute_expiration = unsafe_eval(f"lambda {args}: {self.expiration}") + + def lookup(self, method, *args, **kwargs): + d, key0, counter = self.lru(args[0]) + key = key0 + self.key(*args, **kwargs) + now = time.monotonic() + try: + expiry, value = d[key] + if now < expiry: + counter.hit += 1 + return value + # The entry outlived its expiration: recompute it, below. + counter.miss += 1 + except KeyError: + counter.miss += 1 + except TypeError: + _logger.warning("cache lookup error on %r", key, exc_info=True) + counter.err += 1 + return self.method(*args, **kwargs) + value = self.method(*args, **kwargs) + d[key] = (now + self.compute_expiration(*args, **kwargs), value) + return value + + def add_value(self, *args, cache_value=None, **kwargs): + """Override to store the expiry along with the value.""" + d, key0, _counter = self.lru(args[0]) + key = key0 + self.key(*args, **kwargs) + expiry = time.monotonic() + self.compute_expiration(*args, **kwargs) + d[key] = (expiry, cache_value) diff --git a/fs_attachment_azure/views/fs_storage.xml b/fs_attachment_azure/views/fs_storage.xml new file mode 100644 index 0000000000..deb724de62 --- /dev/null +++ b/fs_attachment_azure/views/fs_storage.xml @@ -0,0 +1,28 @@ + + + + + + fs.storage + + + + + + + + + + + diff --git a/fs_attachment_azure_environment/README.rst b/fs_attachment_azure_environment/README.rst new file mode 100644 index 0000000000..f5dd396dfb --- /dev/null +++ b/fs_attachment_azure_environment/README.rst @@ -0,0 +1,87 @@ +=================================== +Filesystem Attachment Backend Azure +=================================== + +.. + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + !! This file is generated by oca-gen-addon-readme !! + !! changes will be overwritten. !! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + !! source digest: sha256:64046497ed3ebd366cf8ceba5795bc4ef7fe763aecb0aa4a81b56f8cd9590f72 + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png + :target: https://odoo-community.org/page/development-status + :alt: Beta +.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png + :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html + :alt: License: AGPL-3 +.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fstorage-lightgray.png?logo=github + :target: https://github.com/OCA/storage/tree/17.0/fs_attachment_azure_environment + :alt: OCA/storage +.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png + :target: https://translation.odoo-community.org/projects/storage-17-0/storage-17-0-fs_attachment_azure_environment + :alt: Translate me on Weblate +.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png + :target: https://runboat.odoo-community.org/builds?repo=OCA/storage&target_branch=17.0 + :alt: Try me on Runboat + +|badge1| |badge2| |badge3| |badge4| |badge5| + +Glue module for fs_attachment_azure to use fs_storage_environment + +**Table of contents** + +.. contents:: + :local: + +Bug Tracker +=========== + +Bugs are tracked on `GitHub Issues `_. +In case of trouble, please check there if your issue has already been reported. +If you spotted it first, help us to smash it by providing a detailed and welcomed +`feedback `_. + +Do not contact contributors directly about support or help with technical issues. + +Credits +======= + +Authors +------- + +* ACSONE SA/NV +* Camptocamp + +Contributors +------------ + +- Laurent Mignon laurent.mignon@acsone.eu (https://www.acsone.eu) +- Stéphane Bidoul stephane.bidoul@acsone.eu (https://www.acsone.eu) +- Akim Juillerat akim.juillerat@camptocamp.com + +Maintainers +----------- + +This module is maintained by the OCA. + +.. image:: https://odoo-community.org/logo.png + :alt: Odoo Community Association + :target: https://odoo-community.org + +OCA, or the Odoo Community Association, is a nonprofit organization whose +mission is to support the collaborative development of Odoo features and +promote its widespread use. + +.. |maintainer-grindtildeath| image:: https://github.com/grindtildeath.png?size=40px + :target: https://github.com/grindtildeath + :alt: grindtildeath + +Current `maintainer `__: + +|maintainer-grindtildeath| + +This module is part of the `OCA/storage `_ project on GitHub. + +You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute. diff --git a/fs_attachment_azure_environment/__init__.py b/fs_attachment_azure_environment/__init__.py new file mode 100644 index 0000000000..1a9a001cf7 --- /dev/null +++ b/fs_attachment_azure_environment/__init__.py @@ -0,0 +1,2 @@ +from . import models +from .hooks import post_init_hook, uninstall_hook diff --git a/fs_attachment_azure_environment/__manifest__.py b/fs_attachment_azure_environment/__manifest__.py new file mode 100644 index 0000000000..9f9a6304e1 --- /dev/null +++ b/fs_attachment_azure_environment/__manifest__.py @@ -0,0 +1,18 @@ +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). + +{ + "name": "Filesystem Attachment Backend Azure", + "summary": "Allows to use server environment with fs storage attachment Azure", + "version": "17.0.1.0.0", + "category": "FS Storage", + "website": "https://github.com/OCA/storage", + "author": "ACSONE SA/NV,Camptocamp,Odoo Community Association (OCA)", + "license": "AGPL-3", + "development_status": "Beta", + "installable": True, + "depends": ["fs_attachment_environment", "fs_attachment_azure"], + "post_init_hook": "post_init_hook", + "uninstall_hook": "uninstall_hook", + "maintainers": ["grindtildeath"], +} diff --git a/fs_attachment_azure_environment/hooks.py b/fs_attachment_azure_environment/hooks.py new file mode 100644 index 0000000000..883f3b9c11 --- /dev/null +++ b/fs_attachment_azure_environment/hooks.py @@ -0,0 +1,23 @@ +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). + +from odoo.addons.server_environment.uninstall import restore_env_managed_columns + +ENV_MANAGED_FIELDS = [ + "azure_delegation_key_expiration", + "azure_signed_url_expiration", + "azure_uses_signed_url_for_x_sendfile", +] + + +def post_init_hook(env): + env["fs.storage"]._preserve_not_env_managed_data(ENV_MANAGED_FIELDS) + + +def uninstall_hook(env): + """Restore database columns dropped by server.env.mixin.""" + restore_env_managed_columns( + env, + "fs.storage", + ENV_MANAGED_FIELDS, + ) diff --git a/fs_attachment_azure_environment/models/__init__.py b/fs_attachment_azure_environment/models/__init__.py new file mode 100644 index 0000000000..349bb0495a --- /dev/null +++ b/fs_attachment_azure_environment/models/__init__.py @@ -0,0 +1 @@ +from . import fs_storage diff --git a/fs_attachment_azure_environment/models/fs_storage.py b/fs_attachment_azure_environment/models/fs_storage.py new file mode 100644 index 0000000000..0db6acea44 --- /dev/null +++ b/fs_attachment_azure_environment/models/fs_storage.py @@ -0,0 +1,21 @@ +# Copyright 2026 Camptocamp SA +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). + +from odoo import models + + +class FsStorage(models.Model): + _inherit = "fs.storage" + + @property + def _server_env_fields(self): + """Override to include Azure specific fields.""" + fields = super()._server_env_fields + fields.update( + { + "azure_uses_signed_url_for_x_sendfile": {}, + "azure_signed_url_expiration": {}, + "azure_delegation_key_expiration": {}, + } + ) + return fields diff --git a/fs_attachment_azure_environment/pyproject.toml b/fs_attachment_azure_environment/pyproject.toml new file mode 100644 index 0000000000..4231d0cccb --- /dev/null +++ b/fs_attachment_azure_environment/pyproject.toml @@ -0,0 +1,3 @@ +[build-system] +requires = ["whool"] +build-backend = "whool.buildapi" diff --git a/fs_attachment_azure_environment/readme/CONTRIBUTORS.md b/fs_attachment_azure_environment/readme/CONTRIBUTORS.md new file mode 100644 index 0000000000..cb7b22f28c --- /dev/null +++ b/fs_attachment_azure_environment/readme/CONTRIBUTORS.md @@ -0,0 +1,3 @@ +- Laurent Mignon (https://www.acsone.eu) +- Stéphane Bidoul (https://www.acsone.eu) +- Akim Juillerat \ No newline at end of file diff --git a/fs_attachment_azure_environment/readme/DESCRIPTION.md b/fs_attachment_azure_environment/readme/DESCRIPTION.md new file mode 100644 index 0000000000..6940407d26 --- /dev/null +++ b/fs_attachment_azure_environment/readme/DESCRIPTION.md @@ -0,0 +1 @@ +Glue module for fs_attachment_azure to use fs_storage_environment diff --git a/fs_attachment_azure_environment/static/description/icon.png b/fs_attachment_azure_environment/static/description/icon.png new file mode 100644 index 0000000000..3a0328b516 Binary files /dev/null and b/fs_attachment_azure_environment/static/description/icon.png differ diff --git a/fs_attachment_azure_environment/static/description/index.html b/fs_attachment_azure_environment/static/description/index.html new file mode 100644 index 0000000000..85e89b449e --- /dev/null +++ b/fs_attachment_azure_environment/static/description/index.html @@ -0,0 +1,428 @@ + + + + + +Filesystem Attachment Backend Azure + + + +
+

Filesystem Attachment Backend Azure

+ + +

Beta License: AGPL-3 OCA/storage Translate me on Weblate Try me on Runboat

+

Glue module for fs_attachment_azure to use fs_storage_environment

+

Table of contents

+ +
+

Bug Tracker

+

Bugs are tracked on GitHub Issues. +In case of trouble, please check there if your issue has already been reported. +If you spotted it first, help us to smash it by providing a detailed and welcomed +feedback.

+

Do not contact contributors directly about support or help with technical issues.

+
+
+

Credits

+
+

Authors

+
    +
  • ACSONE SA/NV
    • +
  • Camptocamp
    • +
+
+ +
+

Maintainers

+

This module is maintained by the OCA.

+ +Odoo Community Association + +

OCA, or the Odoo Community Association, is a nonprofit organization whose +mission is to support the collaborative development of Odoo features and +promote its widespread use.

+

Current maintainer:

+

grindtildeath

+

This module is part of the OCA/storage project on GitHub.

+

You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

+
+
+
+ + diff --git a/requirements.txt b/requirements.txt index 275c9dfb40..0bc35526f8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,5 @@ # generated from manifests external_dependencies +adlfs fsspec>=2024.5.0 fsspec>=2025.3.0 fsspec[s3]