Operating system and version: Windows 10 / .NET Core 3.1.17
Library Version: (NuGet) 2.1.3
Issue encountered: Different Enumeration results (Disk vs S3)
Expected behavior: Listing of objects
Steps to reproduce: Call Blob.Enumerate() while using DiskStorage

Sample code encapsulating the problem:

DiskSettings diskSettings = new DiskSettings("storage");
var BlobClient = new Blobs(diskSettings);
var blobs = await BlobClient.Enumerate();

foreach (var blobMeta in blobs.Blobs)
    Console.WriteLine(blobMeta.Key);

Exception details: N/A

I noticed while using DiskStorage that enumerating over the files returns odd results compared to S3.
For example, when enumerating over a S3 bucket, I get back blobs with keys using the full path without having to provide a prefix.

When I switch to DiskStorage, if I enumerate, I only get back directories, not files.
This is the same regardless if I provide a prefix or not.

I believe the issue is related to the flags passed to Directory.EnumerateFiles.
I also believe the call to Directory.EnumerateDirectories is unneeded as Directory.EnumerateFiles would provide the same information.

In my own testing, using SearchOption.AllDirectories instead returns what I'd expect when enumerating.

I'm opening this as an issue incase I'm misunderstanding the purpose of DiskStorage.

Ref:
https://github.com/jchristn/BlobHelper/blob/9953f09ab6455bf5086f1d5dad8c98cf292d29c5/BlobHelper/Blobs.cs#L1085
https://github.com/jchristn/BlobHelper/blob/9953f09ab6455bf5086f1d5dad8c98cf292d29c5/BlobHelper/Blobs.cs#L1089

Hi!

First of thank you for BlobHelper it is delicous of use.

I can't find a way to set permission when write to blob. For exemple:
CannedACL = S3CannedACL.PublicRead

Has a way to do that?

Regards

First of all, this is a great wrapper around blob storage engines! I was going to have to write this from scratch, so I am glad I found this.

I am writing a Storage copy command line utility (happy to add it to this repo once finished if you like). Basically it asks for a source server and a destination server, then iterates through the blobs on the source server and adds them to the destination server. This is great for downloading from a cloud server to a local server for testing.

I pointed the source server to a random local directory with sub directories, and to my surprise, I found it iterated through all the files in all directories. I then tried to add these "blobs" into an empty directory without the corresponding sub directories, and it crashed on the first directory. Looks like the target sub directories are not being created on a write.

So the question is, should it iterate through sub directories on read, or should it create sub directories on write? Or neither?

Totally understand if sub directories are not supported. But in that case, it should probably not enumerate sub directories.

And thanks again for this very nicely designed library.

Hi,

first of all, thanks for creating this library.

I didnt find a way of creating empty folders. Is there a possibility of adding this in a future release?

Thanks,

Would have been nice, to see such a helper for the New Azure.Storage.Blobs Package.
Seems like this Repo uses the old "Microsoft.WindowsAzure.Storage;" package - which is depricated and completly different to handle then the new one.

Hi,
Trying to get continuation token back from DiskEnumerate()
Looking at the call of DiskBuildContinuationToken(int start, int count)

The call is made with these parameters:
ret.NextContinuationToken = DiskBuildContinuationToken(startIndex + count, count);

First line in DiskBuildContinuationToken()
if (start >= count) return null;

So looks like only negative values on startIndex on any count can ever return a result that is not null?

we can create interface that contains all methods like Blobs.cs and create concrete implementations for each blob provider.

1. Blobs.cs will use this interface, base on storage type (now there is too many code in this class), we can separate them
2. we can use this interface to inject in DI container. (as far as i know there is no support for DI Container for now)

public interface IBlobClient
    {
        Task<byte[]> GetAsync(string key, CancellationToken token = default);
        Task<BlobData> GetStreamAsync(string key, CancellationToken token = default);
        Task<BlobMetadata> GetMetadataAsync(string key, CancellationToken token = default);
        Task WriteAsync(string key, string contentType, string data, CancellationToken token = default);
        Task WriteAsync(string key, string contentType, byte[] data, CancellationToken token = default);
        Task WriteAsync(string key, string contentType, long contentLength, Stream stream, CancellationToken token = default);
        Task WriteManyAsync(List<WriteRequest> objects, CancellationToken token = default);
        Task DeleteAsync(string key, CancellationToken token = default);
        public Task<bool> ExistsAsync(string key, CancellationToken token = default);
        string GenerateUrlAsync(string key, CancellationToken token = default);
        Task<EnumerationResult> EnumerateAsync(string prefix = null, string continuationToken = null, CancellationToken token = default);
        Task<EmptyResult> EmptyAsync(CancellationToken token = default);
    }

public static class ServiceCollectionExtensions
    {
        public static IServiceCollection AddAzureBlobStorage(this IServiceCollection services,
            AzureSettings azureSettings, ServiceLifetime serviceLifetime = ServiceLifetime.Scoped)
        {
            var azureBlobClientServiceDescriptor = new ServiceDescriptor(typeof(IBlobClient),
                _ => new AzureBlobClient(azureSettings), serviceLifetime);
            services.Add(azureBlobClientServiceDescriptor);
            return services;
        }
    }

checkout this branch: https://github.com/Revazashvili/BlobHelper/tree/refactor P.S code is just changed, nothing is tested

Ciao ragazzi,

I have a case where I can't change the code and my stream is at the end position.

Would you mind seeking to the begin of the stream, e.g.

if (stream.CanSeek && stream.Length == stream.Position)
{ 
    stream.Seek(0, System.IO.SeekOrigin.Begin); 
}

in the method

private async Task DiskWrite(string key, long contentLength, Stream stream, CancellationToken token)

?

Thanks a lot in advance

Best regards,

Florian