Add Azure Blob Storage Gen2 (ADLS HNS) support to BlobFileStore by Skrypt · Pull Request #19014 · OrchardCMS/OrchardCore

Skrypt · 2026-03-16T17:34:16Z

Make BlobFileStore adaptive: it now auto-detects whether the storage account has Hierarchical Namespace (HNS) enabled and uses native DataLake APIs for atomic moves, real directories, and efficient listing when available. Falls back to standard flat-namespace blob operations otherwise. HNS detection can be overridden via configuration.

All operations go through separate, unified API endpoints that adapt server-side

The Vue app calls the same endpoints regardless of storage backend:

Method	Gen2 (HNS) path	Gen1 (flat) path
`GetDirectoryInfoAsync`	`DataLakeFileSystemClient.GetDirectoryClient.ExistsAsync`	blob hierarchy listing
`GetDirectoryContentFlatAsync`	reads `hdi_isfolder` metadata	infers dirs from blob paths
`TryCreateDirectoryAsync`	`DataLakeDirectoryClient.CreateIfNotExistsAsync`	creates marker file
`TryDeleteDirectoryAsync`	`DataLakeDirectoryClient.DeleteAsync(recursive: true)`	iterates and deletes blobs individually
`MoveFileAsync`	`DataLakeFileClient.RenameAsync` (atomic)	`CopyFileAsync` + `TryDeleteFileAsync`

TODO

Implement a set of integration tests against BlobFileStore (Gen 1 and Gen2) with Azurite
Azurite needs to support ADLS; see: ADLS Gen 2 Azure/Azurite#2635

Make BlobFileStore adaptive: it now auto-detects whether the storage account has Hierarchical Namespace (HNS) enabled and uses native DataLake APIs for atomic moves, real directories, and efficient listing when available. Falls back to standard flat-namespace blob operations otherwise. HNS detection can be overridden via configuration. - Add IFileStoreCapabilities interface and FileStoreCapabilities impl - Add GetFilesAsync, GetDirectoriesAsync, Capabilities to IFileStore - Add UseHierarchicalNamespace option to BlobStorageOptions - Add Azure.Storage.Files.DataLake package dependency - Add MediaCreatedFileAsync and MediaCopiedFileAsync event hooks - Wire up EnsureCapabilitiesAsync at startup in Media.Azure module

Add integration tests that run against Azurite to verify BlobFileStore behavior in both flat-namespace (Gen1) and hierarchical-namespace (Gen2) modes. Tests cover capabilities detection, file CRUD, directory operations, move/copy, and directory content listing. Gen2 tests that use DataLake APIs will fail until Azurite supports the DFS endpoint (--dfsPort flag). This is intentional to track when a newer Azurite version enables full Gen2 testing. - Add abstract BlobFileStoreTestsBase with 26 test methods - Add BlobFileStoreGen1Tests and BlobFileStoreGen2Tests subclasses - Add AzuriteFactAttribute to skip tests when Azurite is unavailable - Add OrchardCore.FileStorage.AzureBlob reference to test project - Start Azurite in main_ci and pr_ci workflows on ubuntu runners

Add DfsEndpoint option to BlobStorageOptions for local emulators where the DFS endpoint runs on a separate port. Parse storage credentials from the connection string to construct the DataLakeServiceClient when an explicit DFS endpoint is configured. Fix Gen2 code paths to handle 404 RequestFailedException from DataLakePathClient.ExistsAsync, which can throw instead of returning false when the path does not exist.

Reverted both workflows back to docker run steps (instead of services) since that lets us pass --skipApiVersionCheck directly. The command now uses your image with the correct flags:

Piedone · 2026-03-21T02:32:01Z

I'll try to review this over the weekend.

Piedone

Will #14256 use the new APIs for better UX?

…dia startup Move BlobFileStore.EnsureCapabilitiesAsync() from the DI singleton factory (where it blocked via GetAwaiter().GetResult()) into MediaBlobContainerTenantEvents.ActivatingAsync(), which already runs asynchronously at tenant startup. BlobFileStore is now registered as its own singleton so it can be injected. Also remove the unused StorageProvider member from IFileStoreCapabilities and FileStoreCapabilities, and remove the default implementation of IFileStore.Capabilities so every provider must declare capabilities explicitly. FileSystemStore and AwsFileStore updated accordingly.

…s cleanup

Skrypt · 2026-03-29T16:16:34Z

    private readonly ILogger<FileSystemStore> _logger;
    private readonly string _fileSystemPath;

+    public IFileStoreCapabilities Capabilities { get; } = new FileStoreCapabilities(hasHierarchicalNamespace: false, supportsAtomicMove: false);


I'm changing this configuration to report that the local FileStorage uses HierarchicalNamespace and supports atomic move. It technically should not break any custom implementations.

…lesystem behavior FileSystemStore was configured with hasHierarchicalNamespace: false and supportsAtomicMove: false, which does not match its actual implementation. The local filesystem uses real Directory.* APIs for first-class directory operations and File.Move() which maps to an atomic OS rename syscall. Updated both flags to true to align with the implementation and be consistent with how BlobFileStore Gen2 reports the same native behaviors.

…tification Add a StorageName default interface member to IFileStore that returns a human-readable name for the underlying storage backend. Implementations: FileSystemStore ("Local"), BlobFileStore ("Azure Blob (Gen1)"/"Azure Blob (Gen2)" based on detected HNS), AwsFileStore ("Amazon S3"). DefaultMediaFileStore delegates to the inner store. This enables the media UI to display which storage provider is active at runtime, including distinguishing Azure Gen1 from Gen2.

…tegration project Extract BlobFileStore Gen1/Gen2 tests from OrchardCore.Tests into a new OrchardCore.Tests.Integration project. This separates integration tests that require external services (Azurite) from unit tests, avoiding unnecessary Docker container startup and extra execution time on every PR commit. The new project is built and run by a dedicated integration_tests.yml workflow that only triggers on review submission, push to main/release, or manual dispatch — matching the same pattern as functional_all_db.yml. Removed Azurite setup from pr_ci.yml and main_ci.yml.

github-actions · 2026-04-02T06:30:33Z

This pull request has merge conflicts. Please resolve those before requesting a review.

Skrypt · 2026-04-02T06:31:08Z

@Piedone Same here, moved everything to a new project and also new GH workflow.

github-actions · 2026-04-07T21:33:17Z

This pull request has merge conflicts. Please resolve those before requesting a review.

# Conflicts: # .github/workflows/integration_tests.yml # test/OrchardCore.Tests.Integration/OrchardCore.Tests.Integration.csproj

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Gen2 tests now create their containers using DataLakeFileSystemClient, which sends x-ms-namespace-enabled: true and stamps the container as an HNS filesystem — matching how Gen2 filesystems are created in production Azure. Gen1 tests continue using BlobContainerClient. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Gen1: TryCreateDirectoryAsync writes a marker blob (OrchardCore.Media.txt) to simulate directories in flat namespace storage. Gen2: TryCreateDirectoryAsync uses the DataLake API to create a real directory object — no marker blob is written. These tests assert the raw blob listing directly, proving Azurite is routing to the correct pipeline for each storage generation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Skrypt · 2026-04-30T23:56:14Z

I updated the Azurite ADLS pull request and the related docker image to clean up this PR.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Skrypt · 2026-05-01T15:01:25Z

@sebastienros I'm done with changes. Mostly needed to fix the Azurite emulator to simplify the code here. The remaining HierarchicalNamespace fields are there as a fallback for when the automatic discovery fails. This can happen when using a connection string to a specific container and that the storage account is protected. The GetAccountInfo() Azure blob storage / Azurite function could return a 403 since the user account would not be allowed to do a GetAccountInfo() to the root of the storage. Something like it ... keeping the fields was suggested by AI for that matter. Maybe it is completely wrong but we would need to do tests with a real Azure Blob Storage and see how it behaves with that use case.

I don't like needing an override from appsettings like this but that's a fallback because we have auto-discovery of the Gen type. I may investigate further when I get time.

Skrypt · 2026-05-21T18:24:48Z

@sebastienros I did not find on Google or Github the other alternative to Azurite. If you can share it here I will take a look at it. I think though that my PR on Azurite has had positive feedback so far, but no one from Microsoft did ever comment on it yet. Seems like a repository that doesn't get much attention and is only on "maintenance" mode.

github-actions · 2026-06-01T19:12:46Z

This pull request has merge conflicts. Please resolve those before requesting a review.

# Conflicts: # Directory.Packages.props # src/OrchardCore/OrchardCore.Media.Core/DefaultMediaFileStore.cs # test/OrchardCore.Tests.Integration/OrchardCore.Tests.Integration.csproj

The merge from main introduced a sibling OrchardCore.Tests.Integration.Azure namespace, which shadowed the global Azure SDK namespace and broke the fully-qualified Azure.Storage.Blobs.Models references in these tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds a New features entry to the 4.0.0 release notes and expands the Azure Media reference docs with the UseHierarchicalNamespace setting and its auto-detection/override behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-07-12T00:47:13Z

This pull request has merge conflicts. Please resolve those before requesting a review.

# Conflicts: # Directory.Packages.props

…nant activation GetDirectoryContentByHierarchyAsync (used by the Media Library's folder/file browsing) never checked the hdi_isfolder metadata that ADLS Gen2 uses to represent empty directories, so a freshly created empty folder on a Gen2 account showed up as a phantom 0-byte file instead of a directory until a file was uploaded into it. Also wrap the HNS capability probe in ActivatingAsync so a failed detection (restricted SAS, transient network error) degrades to flat-namespace operations instead of failing the whole tenant's shell activation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The current (soon to be replaced) Media Library gallery hasn't been fully verified against ADLS Gen2 / Hierarchical Namespace storage. Add a filter that surfaces a notification on the Media Library page recommending Gen1 until the upcoming gallery ships, and a separate warning when the account type couldn't be auto-detected and flat-namespace operations are used as a fallback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6eacd0ab-84d2-4cce-b526-aee041dc1239 Co-authored-by: Sébastien Ros <1165805+sebastienros@users.noreply.github.com>

Skrypt · 2026-07-16T19:57:10Z

@sebastienros Can I merge this now?

Skrypt · 2026-07-17T13:15:27Z

I'm merging this and the OpenApi pull requests today unless someone disagrees. @kevinchalet @hishamco @MikeAlhayek @gvkries

gvkries · 2026-07-17T13:22:23Z

Ship it! 👍

main squash-merged the OpenApi (#19016) and Azure Blob Gen2 (#19014) PRs, whose content vue-3 already contains via branch merges, so the add/add conflicts in those areas resolve to our (newer) versions. yarn.lock was re-resolved with yarn install; the duplicate Microsoft.AspNetCore.OpenApi 10.0.5 entry from main was dropped in favor of the existing 10.0.9 pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Skrypt marked this pull request as draft March 16, 2026 17:54

Skrypt added 2 commits March 16, 2026 15:07

Skrypt mentioned this pull request Mar 17, 2026

ADLS Gen 2 Azure/Azurite#2635

Open

Skrypt added 3 commits March 18, 2026 04:11

Update unit tests after updating Azurite

7be25b8

Update GH workflows with temp Azurite package repo

71ced65

skipApiVersionCheck

256835f

Reverted both workflows back to docker run steps (instead of services) since that lets us pass --skipApiVersionCheck directly. The command now uses your image with the correct flags:

Skrypt requested a review from Piedone March 19, 2026 20:46

Merge branch 'main' into skrypt/media-azure-gen2

121557d

Piedone requested changes Mar 25, 2026

View reviewed changes

Skrypt added 3 commits March 28, 2026 01:34

Merge branch 'main' into skrypt/media-azure-gen2

03c0980

fix: Remove StorageProvider from test broken by IFileStoreCapabilitie…

7d16f4f

…s cleanup

Skrypt commented Mar 29, 2026

View reviewed changes

Skrypt added 5 commits March 29, 2026 12:19

fix build

aa477e6

add documentation on added configuration

56b294a

github-actions Bot added the merge conflict label Apr 2, 2026

Merge branch 'main' into skrypt/media-azure-gen2

7c01e5c

github-actions Bot added merge conflict and removed merge conflict labels Apr 2, 2026

Merge branch 'main' into skrypt/media-azure-gen2

3a44b79

# Conflicts: # .github/workflows/integration_tests.yml # test/OrchardCore.Tests.Integration/OrchardCore.Tests.Integration.csproj

github-actions Bot removed the merge conflict label Apr 14, 2026

Skrypt and others added 4 commits April 30, 2026 19:35

Fix CA2254: split ternary log message into separate LogInformation calls

fb6bcb6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge branch 'main' into skrypt/media-azure-gen2

d91f49b

Fix dotnet test syntax: use --project flag

f572bd2

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions Bot added the merge conflict label Jun 1, 2026

Merge branch 'main' into skrypt/media-azure-gen2

2bc7244

# Conflicts: # Directory.Packages.props # src/OrchardCore/OrchardCore.Media.Core/DefaultMediaFileStore.cs # test/OrchardCore.Tests.Integration/OrchardCore.Tests.Integration.csproj

github-actions Bot removed the merge conflict label Jul 8, 2026

Skrypt and others added 2 commits July 7, 2026 23:21

github-actions Bot added the merge conflict label Jul 12, 2026

Merge branch 'main' into skrypt/media-azure-gen2

dfb0b53

# Conflicts: # Directory.Packages.props

github-actions Bot removed the merge conflict label Jul 15, 2026

Skrypt and others added 2 commits July 15, 2026 19:02

sebastienros mentioned this pull request Jul 16, 2026

Fix ADLS Gen2 review findings #19565

Merged

sebastienros and others added 2 commits July 16, 2026 15:44

Fix ADLS Gen2 review findings (#19565)

1ef754d

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6eacd0ab-84d2-4cce-b526-aee041dc1239 Co-authored-by: Sébastien Ros <1165805+sebastienros@users.noreply.github.com>

Merge branch 'main' into skrypt/media-azure-gen2

271d657

Skrypt merged commit 6b7e61a into main Jul 17, 2026
7 checks passed

Skrypt deleted the skrypt/media-azure-gen2 branch July 17, 2026 18:20

Skrypt mentioned this pull request Jul 17, 2026

OpenApi #19016

Merged

1 task

Conversation

Skrypt commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

All operations go through separate, unified API endpoints that adapt server-side

TODO

Uh oh!

Piedone commented Mar 21, 2026

Uh oh!

Piedone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Skrypt Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 2, 2026

Uh oh!

Skrypt commented Apr 2, 2026

Uh oh!

github-actions Bot commented Apr 7, 2026

Uh oh!

Skrypt commented Apr 30, 2026

Uh oh!

Skrypt commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Skrypt commented May 21, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

Skrypt commented Jul 16, 2026

Uh oh!

Skrypt commented Jul 17, 2026

Uh oh!

gvkries commented Jul 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Skrypt commented Mar 16, 2026 •

edited

Loading

Skrypt Mar 29, 2026 •

edited

Loading

Skrypt commented May 1, 2026 •

edited

Loading