forked from Azure/Azurite
-
Notifications
You must be signed in to change notification settings - Fork 0
Adding Blob Versioning Support to Azurite #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
67 commits
Select commit
Hold shift + click to select a range
a1d7dc0
starting work on blob versioning. Added account model and modified cr…
rorozcov e394be7
working tests with successful blob create with versioning
rorozcov a84fc37
improved account model usage outside store
rorozcov 2aa7cfb
working test. now I need to add more tests and develop rest of functi…
rorozcov d4a8080
went back and added versioning support to all of loki blob metadata s…
rorozcov 0571013
updated files to get build working. looking getting db tests working
rorozcov 70f15cb
basic tests and project builds. Must audit then make more tests
rorozcov ef14de9
ccaught bugs. Basic tests passing. Added tests to new utils
rorozcov fb63f7d
addressed bugs, added more versioning tests
rorozcov 903fab5
more tests and bugs fixed. More work to do before API changes
rorozcov 1279659
added disable to enable tests. Missing enable to disable and listblobs
rorozcov ed70b4c
squashed more bugs on blob operations. ready for v1. missing list blo…
rorozcov db20834
fixed bugs and added more tests to verify delete blob and snapshot logic
rorozcov 94d91fb
updated APIs. Tests still pass. Time for APi testing
rorozcov 056d639
fixing handler tests
rorozcov 721e60d
fixed copy bugs. missing block blob operations bugs
rorozcov 7b7f33a
fixed bugs and now all pre-existing block blob tests are passing
rorozcov 8c57e6b
all tests passing
rorozcov 7350ef8
Adding appendblob tests. They need auditing
rorozcov d24739c
added pageblob versioning tests, but only copy pasted non-versioning …
rorozcov fe3fafb
finished versioning enabled pageblob tests
rorozcov 82354f4
prod comparison tests
rorozcov 301b646
fixed bug in properties and added parity tests against storage account
rorozcov 36bf787
adding mutually exclusive check and tests for it
rorozcov 56e6f57
adding valid date checks
rorozcov 25457ca
allow iso range due to js vs blob storage differences
rorozcov 69cb5aa
renaming to prod
rorozcov 0edbad6
fixed API return values and added parity tests on azurite
rorozcov 63ae1a7
appendblob now checks versionId is undefined when versioning is off
rorozcov ac5867e
Adding basic blob tests against versionId being undefined when versio…
rorozcov d07c1ac
Revert "appendblob now checks versionId is undefined when versioning …
rorozcov a363e72
adding basic versionId undefined test
rorozcov 9888af9
added listblob versioning logic. missing full test suite with version…
rorozcov bb18539
refactoring lokidb testing files. next step: add list blob files
rorozcov 0625af0
added lokidb versioning tests for listBlobs
rorozcov 481b636
massively improved block blob API tests with meaningful test cases
rorozcov 8c63db7
cleaner append blob tests
rorozcov 0378faa
same for pageblob
rorozcov d6f607f
removing settings.json
rorozcov 4af1312
Switched completely to blob config filejson/path
rorozcov b6fbadc
adding comment about model compat
rorozcov e19ff17
fixing package.json
rorozcov eda253a
removing formatting changes to improve reviewability
rorozcov 4d7b5ac
adding env functions tests
rorozcov 44143e0
removing snapshot string
rorozcov 5cf4060
adding new parity test. IfNoneMatch should always fail to overwrite, …
rorozcov c52aa46
Updating readme
rorozcov 655468c
updating readme
rorozcov 87e8d25
added versioning to paginated listBlobs and filterBlobs
rorozcov 5b28f1b
Finished pagewithdelimeter logic
rorozcov 966a38d
Multi-account support works in unit testing. Missing manual testing
rorozcov 0413872
manually tested. Multi-account works
rorozcov d21d5ed
removing comment
rorozcov 363339b
Adding comment back
rorozcov 2c338c6
addressing api query param validation comment
rorozcov 7fc72de
added comment on blob insertion
rorozcov 4b13904
improving account model init, adding required comment
rorozcov b8e96f1
appendblob test comments addressed
rorozcov 8e36ede
blob test comments addressed
rorozcov 2dc9daf
addressing more comments
rorozcov d5318c2
updating client in use
rorozcov a24fa0c
adding docs
rorozcov 29a371c
filter blobs ignore tags in previous versions and only uses current. …
rorozcov 840fc68
matching prod behaviour
rorozcov 18cdd51
verified deletion behaviours
rorozcov 1f9f80d
addressing comments
rorozcov 12a07e5
Merge upstream main into blob versioning branch
rorozcov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -206,6 +206,8 @@ Following extension configurations are supported: | |
| - `azurite.disableProductStyleUrl` Force parsing storage account name from request URI path, instead of from request URI host. | ||
| - `azurite.inMemoryPersistence` Disable persisting any data to disk. If the Azurite process is terminated, all data is lost. | ||
| - `azurite.extentMemoryLimit` When using in-memory persistence, limit the total size of extents (blob and queue content) to a specific number of megabytes. This does not limit blob, queue, or table metadata. Defaults to 50% of total memory. | ||
| - `azurite.accountConfigFilePath` Path to a JSON file containing AccountModel configuration for blob versioning settings. See [Use Blob Versioning](#use-blob-versioning) for details. | ||
| - `azurite.accountConfigAsJson` Inline JSON string containing AccountModel configuration for blob versioning settings. See [Use Blob Versioning](#use-blob-versioning) for details. | ||
| - `azurite.disableTelemetry` Disable telemetry data collection of this Azurite execution. By default, Azurite will collect telemetry data to help improve the product. | ||
|
|
||
| ### [DockerHub](https://hub.docker.com/_/microsoft-azure-storage-azurite) | ||
|
|
@@ -239,7 +241,7 @@ docker run -p 10000:10000 -p 10001:10001 -v c:/azurite:/data mcr.microsoft.com/a | |
| #### Customize all Azurite V3 supported parameters for docker image | ||
|
|
||
| ```bash | ||
| docker run -p 7777:7777 -p 8888:8888 -p 9999:9999 -v c:/azurite:/workspace mcr.microsoft.com/azure-storage/azurite azurite -l /workspace -d /workspace/debug.log --blobPort 7777 --blobHost 0.0.0.0 --blobKeepAliveTimeout 5 --queuePort 8888 --queueHost 0.0.0.0 --queueKeepAliveTimeout 5 --tablePort 9999 --tableHost 0.0.0.0 --tableKeepAliveTimeout 5 --loose --skipApiVersionCheck --disableProductStyleUrl --disableTelemetry | ||
| docker run -p 7777:7777 -p 8888:8888 -p 9999:9999 -v c:/azurite:/workspace mcr.microsoft.com/azure-storage/azurite azurite -l /workspace -d /workspace/debug.log --blobPort 7777 --blobHost 0.0.0.0 --blobKeepAliveTimeout 5 --queuePort 8888 --queueHost 0.0.0.0 --queueKeepAliveTimeout 5 --tablePort 9999 --tableHost 0.0.0.0 --tableKeepAliveTimeout 5 --loose --skipApiVersionCheck --disableProductStyleUrl --accountConfigAsJson "{\"isBlobVersioningEnabled\":true}" --disableTelemetry | ||
| ``` | ||
|
|
||
| Above command will try to start Azurite image with configurations: | ||
|
|
@@ -272,6 +274,10 @@ Above command will try to start Azurite image with configurations: | |
|
|
||
| `--disableProductStyleUrl` force parsing storage account name from request URI path, instead of from request URI host. | ||
|
|
||
| `--accountConfigFilePath /workspace/accountModel.json` configures blob versioning using an AccountModel JSON file mapped to the docker workspace. See [Use Blob Versioning](#use-blob-versioning) for details. | ||
|
|
||
| `--accountConfigAsJson "{\"isBlobVersioningEnabled\":true}"` configures blob versioning using an inline JSON string. See [Use Blob Versioning](#use-blob-versioning) for details. | ||
|
|
||
| `--azurite.disableTelemetry` disable telemetry data collection of this Azurite execution. By default, Azurite will collect telemetry data to help improve the product. | ||
|
|
||
| > If you use customized azurite parameters for docker image, `--blobHost 0.0.0.0`, `--queueHost 0.0.0.0` are required parameters. | ||
|
|
@@ -501,6 +507,86 @@ server thus resetting the storage completely. | |
| Note that if many hundreds of megabytes of content (queue message or blob content) are stored in-memory, it can take | ||
| noticeably longer than usual for the process to terminate since all the consumed memory needs to be released. | ||
|
|
||
| ### Use Blob Versioning | ||
|
|
||
| #### How it works | ||
|
|
||
| Blob Versioning was implemented to follow the exact guidelines outlined in the [Azure Blob Storage versioning documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/versioning-overview), excluding interactions with soft delete, blob expiration, and SAS URIs. For detailed implementation information, see the [blob versioning design document](docs/designs/2025-12-blob-versioning.md). | ||
|
|
||
| #### How to use it | ||
|
|
||
| ##### Single Account support | ||
|
|
||
| Optional. By default, this is disabled. To enable it, there are two CLI args you can use: accountConfigFilePath, accountConfigAsJson. | ||
|
|
||
| Blob versioning is enabled by leveraging the [AccountModel](src/blob/AccountModel.ts). The account model is an abstraction to configure the storage account. Currently, it only supports configuring blob versioning. | ||
|
|
||
| accountConfigFilePath lets you pass in the path to a json file modeled after the AccountModel, which is then used to configure Azurite. | ||
|
|
||
| ```bash | ||
| azurite --accountConfigFilePath "./myAccountModel.json" | ||
|
rorozcov marked this conversation as resolved.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be better add a debug log when the azurite start, to print out the account config. This can help the debug of Azurite issues. |
||
| ``` | ||
|
|
||
| Example contents of `myAccountModel.json`: | ||
|
|
||
| ```json | ||
| { | ||
| "isBlobVersioningEnabled": true | ||
| } | ||
| ``` | ||
|
|
||
| accountConfigAsJson allows you to pass a json string as a CLI arg to configure the account as well. | ||
|
|
||
| ```bash | ||
| azurite --accountConfigAsJson "{ \"isBlobVersioningEnabled\": true }" | ||
|
rorozcov marked this conversation as resolved.
rorozcov marked this conversation as resolved.
rorozcov marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| By default, Azurite will always use whatever version of the account model already exists in its databases. However, if any of these parameters are passed in and are valid, the existing account model will be overwritten. | ||
|
|
||
| ##### Multi-account AccountModel support | ||
|
|
||
| Both `accountConfigFilePath` and `accountConfigAsJson` support configuring multiple accounts with different versioning settings. | ||
|
|
||
| **Using accountConfigFilePath with multiple accounts:** | ||
|
|
||
| ```bash | ||
| azurite --accountConfigFilePath "account1:/path/to/config1.json,account2:/path/to/config2.json" | ||
| ``` | ||
|
|
||
| Where `config1.json` might contain: | ||
|
|
||
| ```json | ||
| { | ||
| "isBlobVersioningEnabled": true | ||
| } | ||
| ``` | ||
|
|
||
| And `config2.json` might contain: | ||
|
|
||
| ```json | ||
| { | ||
| "isBlobVersioningEnabled": false | ||
| } | ||
| ``` | ||
|
|
||
| **Using accountConfigAsJson with multiple accounts:** | ||
|
|
||
| ```bash | ||
| azurite --accountConfigAsJson "account1:{\"isBlobVersioningEnabled\":true},account2:{\"isBlobVersioningEnabled\":false}" | ||
| ``` | ||
|
|
||
| **Backward compatibility:** | ||
|
|
||
| For single-account configuration, you can omit the account name prefix (defaults to `devstoreaccount1`): | ||
|
|
||
| ```bash | ||
| azurite --accountConfigFilePath "./myAccountModel.json" | ||
| # or | ||
| azurite --accountConfigAsJson "{\"isBlobVersioningEnabled\":true}" | ||
| ``` | ||
|
|
||
| > **Important:** Declaring an account in the AccountModel configuration only sets the versioning behavior for that account. You still need to configure authentication for these accounts using the `AZURITE_ACCOUNTS` environment variable (see [Customized Storage Accounts & Keys](#customized-storage-accounts--keys-1)) to actually use them. Without proper authentication setup, requests to these accounts will fail authentication. **Furthermore, if you want to configure all accounts, you must configure each account individually. If you follow the single-account flow or configure only one account, the other accounts will not be configured.** | ||
|
|
||
| ### Command Line Options Differences between Azurite V2 | ||
|
|
||
| Azurite V3 supports SharedKey, Account Shared Access Signature (SAS), Service SAS, OAuth, and Public Container Access authentications, you can use any Azure Storage SDKs or tools like Storage Explorer to connect Azurite V3 with any authentication strategy. | ||
|
|
@@ -1020,14 +1106,17 @@ Latest release targets **2025-11-05** API version **blob** service. | |
| Detailed support matrix: | ||
|
|
||
| - Supported Vertical Features | ||
|
|
||
| - CORS and Preflight | ||
| - SharedKey Authentication | ||
| - OAuth authentication | ||
| - Shared Access Signature Account Level | ||
| - Shared Access Signature Service Level (Not support response header override in service SAS) | ||
| - Container Public Access | ||
| - Blob Tags (preview) | ||
| - Blob versioning (Only in LokiDb instances of Azurite, which is the default. Does not support SAS URIs) | ||
| - Supported REST APIs | ||
|
|
||
|
rorozcov marked this conversation as resolved.
|
||
| - List Containers | ||
| - Set Service Properties | ||
| - Get Service Properties | ||
|
|
@@ -1062,7 +1151,6 @@ Detailed support matrix: | |
| - Soft delete & Undelete Blob | ||
| - Incremental Copy Blob | ||
| - Blob Query | ||
| - Blob Versions | ||
| - Blob Last Access Time | ||
| - Concurrent Append | ||
| - Blob Expiry | ||
|
|
@@ -1073,6 +1161,7 @@ Detailed support matrix: | |
| - Encryption Scope | ||
| - Get Page Ranges Continuation Token | ||
| - Blob Immutability Policy and Legal Hold | ||
| - SAS URIs for Blob Versions | ||
|
rorozcov marked this conversation as resolved.
|
||
|
|
||
| Latest version supports for **2025-11-05** API version **queue** service. | ||
| Detailed support matrix: | ||
|
|
@@ -1156,4 +1245,4 @@ provided by the bot. You will only need to do this once across all repos using o | |
|
|
||
| This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). | ||
| For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or | ||
| contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. | ||
| contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| # Add Blob Versioning Support | ||
|
|
||
| - Author Name: Rodolfo Orozco Vasquez ([@rorozcov](https://github.com/rorozcov)) | ||
| - GitHub Issue: [Azure/Azurite#665](https://github.com/Azure/Azurite/issues/665) | ||
|
|
||
| ## Summary | ||
|
|
||
| This design adds support for Azure Blob Storage versioning to Azurite, allowing blobs to maintain previous versions automatically when they are modified or deleted. Versioning is implemented following the [Azure Blob Storage versioning guidelines](https://learn.microsoft.com/en-us/azure/storage/blobs/versioning-overview) as closely as possible, with some limitations based on features not yet supported in Azurite. | ||
|
|
||
| ## Motivation | ||
|
|
||
| Blob versioning is a critical feature in Azure Blob Storage that automatically maintains previous versions of a blob. This enables users to: | ||
|
|
||
| - Recover from accidental blob modifications or deletions | ||
| - Maintain a history of blob changes over time | ||
| - Access and restore previous versions of blobs | ||
|
|
||
| Without blob versioning support in Azurite, developers cannot fully test applications that rely on this feature locally, limiting their ability to validate version-aware workflows before deploying to Azure. | ||
|
|
||
| ## Explanation | ||
|
|
||
| ### Functional explanation | ||
|
|
||
| Blob versioning in Azurite is controlled through the `AccountModel` abstraction, which allows configuration of storage account-level settings. When enabled, blob versioning automatically creates a new version of a blob whenever it is modified or overwritten. | ||
|
|
||
| Two command line options are available to configure blob versioning: | ||
|
|
||
| 1. **`--accountConfigFilePath`** - Path to a JSON configuration file | ||
| 2. **`--accountConfigAsJson`** - Inline JSON string configuration | ||
|
|
||
| We also support multi-account configuration since Azurite supports multiple accounts. | ||
|
|
||
| ### Technical explanation | ||
|
|
||
| Blob versioning is implemented using the `AccountModel` type which is stored in the metadata database: | ||
|
|
||
| ```typescript | ||
| export interface AccountModel { | ||
| key: string; | ||
| isBlobVersioningEnabled?: boolean; | ||
| } | ||
| ``` | ||
|
|
||
| When versioning is enabled for an account: | ||
|
|
||
| - **For block blobs:** All write operations trigger the creation of a new version, except for the Put Block operation | ||
| - **For page blobs and append blobs:** Only a subset of write operations triggers version creation: | ||
| - Put Blob | ||
| - Put Block List | ||
| - Set Blob Metadata | ||
| - Copy Blob | ||
| - **Operations that do NOT trigger version creation:** | ||
| - Put Page (page blob) | ||
| - Append Block (append blob) | ||
| - Each version is assigned a unique version ID in ISO 8601 date-time format | ||
| - **Note:** Azurite's version IDs end in 3 digits + Z (e.g., `2024-12-06T10:30:45.123Z`) due to JavaScript's Date implementation, while Azure's version IDs end in 7 digits + Z (e.g., `2024-12-06T10:30:45.1234567Z`). If your application relies on this specific format, plan accordingly. | ||
| - Previous versions are immutable and can be accessed using the version ID | ||
| - The `List Blobs` operation can include versions when the `includeVersions` parameter is set to true | ||
| - Specific versions can be retrieved, downloaded, or deleted using the `versionId` query parameter | ||
|
|
||
| The configuration is parsed through `EnvironmentFunctions.parseAccountModelFlags()` which supports: | ||
|
|
||
| - Single account configuration | ||
| - Multi-account configuration with comma-separated entries | ||
| - Both file-based and inline JSON configurations | ||
| - Proper JSON parsing with support for nested objects and escaped characters | ||
|
|
||
| ### Integration with Authentication | ||
|
|
||
| > **Important:** The `AccountModel` configuration only controls the versioning behavior. To actually use the configured accounts, they must also be set up in the `AZURITE_ACCOUNTS` environment variable for authentication. See [Customized Storage Accounts & Keys](https://github.com/Azure/Azurite#customized-storage-accounts--keys-1) for details. | ||
|
|
||
| ### Limitations | ||
|
|
||
| The following Azure Blob Storage versioning features are **not** currently supported: | ||
|
|
||
| - Soft delete integration with versioning | ||
| - Blob expiration with versioning | ||
| - SAS URIs for specific blob versions | ||
| - Version-level immutability policies (Version Level WORM) | ||
|
|
||
| ### VS Code Extension Support | ||
|
|
||
| Similar configuration options are available in the VS Code extension settings: | ||
|
|
||
| - `azurite.accountConfigFilePath` - Path to account configuration file | ||
| - `azurite.accountConfigAsJson` - Inline JSON configuration string | ||
|
|
||
| ## Azure Documentation on Blob Versioning | ||
|
|
||
| This implementation follows the Azure Blob Storage versioning specification as documented in the official Microsoft documentation: | ||
|
|
||
| - [Blob versioning overview](https://learn.microsoft.com/en-us/azure/storage/blobs/versioning-overview) | ||
| - [Enable and manage blob versioning](https://learn.microsoft.com/en-us/azure/storage/blobs/versioning-enable) | ||
|
|
||
| The design aligns with Azure's behavior where: | ||
|
|
||
| - Versioning is a storage account-level setting | ||
| - Version IDs are automatically assigned timestamps | ||
| - Previous versions are immutable | ||
| - The current version is mutable |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| export interface AccountModel { | ||
| key: string; | ||
| isBlobVersioningEnabled: boolean; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.