aloneguid / stowage Goto Github PK

Hi,

Thanks for this library, it is really useful.

Would you entertain the idea of creating strong named / signed assemblies. Currently I am getting the following warning:
CSC : warning CS8002: Referenced assembly 'Stowage, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' does not have a strong name

I would like to have anonymous access to S3. In my scenario access is authorized through VPC policies and doesn't require signed requests. However the software will automatically sign the AWS request even if I don't specify access keys.

Would it be possible to support anonymous S3 calls (ie. without the AWS signature)

Minio. support S3 compatible interface so it would be nice that we could use it.

Even AWS CLI supports custom endpoints.
aws --profile minio --endpoint-url=http://localhost:9001 s3 ls

Library is great - documentation is .....spotty at best.....

I'm trying to use Stowage against a local directory for development, and Azure Blob container for production. But I can't even get the local disk part to work...

I have tried this code (and a great many variations of it - with or without trailing backslash, with or without file pattern, using full path in the .Of.LocalDisk() call or only when doing .Ls()) - so far, without any success:

using (IFileStorage fs = Files.Of.LocalDisk(@"D:\"))
{
    var entries = await fs.Ls(@"\projects\mailtemplates", false);
}

I keep getting this error:

ArgumentException: path needs to be a folder (Parameter 'path')

What am I doing wrong? I would like to get all the files (or preferably: all files matching a certain pattern) from this folder ....

Hi 👋

The previous version Storage.Net is supporting the emu account while in development. It doesn't seem like stowage is supporting it. I looked at the source code and couldn't find any way of connecting, easily, to the emulator. Therefore, I revised the current implementation and do have one issue.

To address the issue, I used the same approach as @iamkoch which is allowing to override the azure endpoint. It worked great; however, it looks like using the http://127.0.0.1:10000/devstoreaccount1 endpoint with a container (emu in this example) lead to an issue with the emulator not looking at the right account. To address that, I had to set my ContainerName to devstoreaccount1/emu.

Here is the code I changed to get things working. If you prefer, I can create a pull request to start a discussion.

Files.cs

public static IFileStorage AzureBlobStorage(this IFilesFactory _, string accountName, string sharedKey, string containerName = null)
{
   return AzureBlobStorage(_, new Uri($"https://{accountName}.blob.core.windows.net"), accountName, sharedKey, containerName);
}

public static IFileStorage AzureBlobStorage(this IFilesFactory _, Uri endpoint, string accountName, string sharedKey, string containerName = null)
{
   return new AzureBlobFileStorage(endpoint, containerName, new SharedKeyAuthHandler(accountName, sharedKey));
}

AzureBlobFileStorage.cs

public AzureBlobFileStorage(Uri endpoint, string containerName, DelegatingHandler authHandler) : base(endpoint, authHandler)
{
   if(endpoint is null)
      throw new ArgumentNullException(nameof(endpoint));
   if(string.IsNullOrEmpty(containerName))
      throw new ArgumentException($"'{nameof(containerName)}' cannot be null or empty", nameof(containerName));
   if(authHandler is null)
      throw new ArgumentNullException(nameof(authHandler));

   _containerName = containerName;
}

I guess I could keep it that way, but I find it a bit confusing to put the storage account in the container name when in development. Haven't tested in production with Azure Storage yet.

Thanks,

Certain S3 compatible storages do not support multi-part upload and require to use PUT in all cases.
Could we please add support for this?

The title says it all.
I'm going to submit a pull request for this one.

https://github.com/aloneguid/stowage/blob/92612a6d12c0f78c93d38b65f3bb280bc07f6f1a/src/Stowage/Impl/Microsoft/SasAuthHandler.cs#L8C11-L8C26

The implementation seems to exist but I can't find any place that actually uses that class, and since it is all internal I can't call it myself either?

Hi,

may I ask what is the recommendation for injecting custom delegating handlers? I can't find how to do this without reflection.
If it's not currently available, I would like to kindly raise a feature request 😊
It would be very helpful to be able to implement retries, logging, throttling and other useful features.

Thank you,
Martin

Hi,

I am testing with S3 compatible storage (Minio) and I am experiencing issues where I am getting 403 Forbidden during authentication. Error code is SignatureDoesNotMatch.
I have found 2 causes:

1. if endpoint contains port other than 80 or 443, the port must be included in the host header
2. if the path ends with a path separator it must not be url encoded (I verified this issue also with AWS S3)

Best,
Martin

Hi,

I am writing large blobs to Azure blob storage and I keep getting response 400 Bad Request (The specified blob or block content is invalid). I tracked down the reason to a wrong block ID.
The documentation says (https://learn.microsoft.com/en-us/rest/api/storageservices/put-block?tabs=azure-ad):

For a specified blob, all block IDs must be the same length. If a block is uploaded with a block ID of a different length than the block IDs for any existing uncommitted blocks, the service returns error response code 400 (Bad Request).

The stream write fails always on blob with ID 10, so once we don't get single digit ID, we are braking the requirement.
Since there is a limit of 50k blocks, I suggest to pad the ID to a 6 digit string.

Thanks,
Martin