Giter Club home page Giter Club logo

aws-signature-version-4's Introduction

AwsSignatureVersion4

CI/CD Codecov NuGet version SemVer NuGet downloads License

The buttoned-up and boring, but deeply analyzed, implementation of Signature Version 4 (SigV4) in .NET.

Package: AwsSignatureVersion4
Platforms: .NET Standard 2.0, .NET 6

Table of contents

Introduction

This project is unique for me. It's my first that isn't a labor of love.

Having to sign requests in AWS I went through a series of emotions. My first was disappointment, directed at Amazon for not including a Signature Version 4 signer in their AWS SDK for .NET. The functionality is listed on Open Feature Requests for the AWS SDK for .NET but I haven't seen any actions towards an implementation yet.

My second emotion was being overwhelmed. The signing algorithm involved many more steps than I'd thought be possible, and I knew I'd have to spend a lot of time getting conformable with the algorithm.

So here we are, my attempt at implementing the Signature Version 4 algorithm in .NET. Please lets hope that AWS SDK soon releases this functionality so we can deprecate this piece of code...

Super simple to use

The best API is the one you already know. By extending both HttpClient and IHttpClientFactory we hope you'll get an easy integration.

Integration with HttpClient

This project extends the class HttpClient by providing additional overloads to DeleteAsync, GetAsync, GetStringAsync, PatchAsync, PostAsync, PutAsync, SendAsync, and the new synchronous addition to .NET 5 and onwards, Send. These overloads accept the following additional arguments.

  • regionName - The name of the AWS region, e.g. us-west-1
  • serviceName - The name of the service, e.g. execute-api for an AWS API Gateway
  • credentials - The AWS credentials of the principal sending the request

These overloads are built to integrate with HttpClient, i.e. HttpClient.BaseAddress and HttpClient.DefaultRequestHeaders will be respected when sending the request.

The following example is demonstrating how one would send a GET /resources request to an IAM authenticated AWS API Gateway on host www.acme.com.

// Don't specify credentials in source code, this is for demo only! See next chapter for more
// information.
var credentials = new ImmutableCredentials("<access key id>", "<secret access key>", null);

var client = new HttpClient();
var response = await client.GetAsync(
  "https://www.acme.com/resources",
  regionName: "us-west-1",
  serviceName: "execute-api",
  credentials: credentials);

Please see the tests directory for other examples.

Integration with IHttpClientFactory

This project supports IHttpClientFactory by means of providing a custom HTTP message handler called AwsSignatureHandler. Once injected into the pipeline of the HTTP client factory it will sign your requests before sending them over the network.

AwsSignatureHandler takes an instance of AwsSignatureHandlerSettings as its only constructor argument, thus you will have to configure your dependency injection container to sufficiently resolve both these classes.

The following example is demonstrating how one would configure the ASP .NET Core service collection to acknowledge a HTTP client named my-client. For more information regarding configuration, please see Dependency injection in ASP.NET Core.

// Don't specify credentials in source code, this is for demo only! See next chapter for more
// information.
var credentials = new ImmutableCredentials("<access key id>", "<secret access key>", null);
services
  .AddTransient<AwsSignatureHandler>()
  .AddTransient(_ => new AwsSignatureHandlerSettings("us-west-1", "execute-api", credentials));

services
  .AddHttpClient("my-client")
  .AddHttpMessageHandler<AwsSignatureHandler>();

Please see the tests directory for other examples.

Credentials

We've come a long way, but let's back up a step. Credentials should not be specified in source code, so where do they come from?

It all starts with a principal, i.e. an entity identifying itself using authentication. In some situations the principal is a IAM user and in other situations it is an entity assuming a IAM role. Whatever your principal is, it has the capability of providing credentials.

How the credentials are provided depend on where you run your code. If you run your code in a ECS Task you get your credentials using ECSTaskCredentials. Other runtime will require other credential providers, all of them are listed in the namespace Amazon.Runtime.

The pledge

This project comes with a pledge, providing transparency on supported and unsupported scenarios.

  • ✅ ~200 unit tests are passing before a release
  • ✅ ~700 integration tests targeting an IAM authenticated AWS API Gateway are passing before a release
  • ✅ ~300 integration tests targeting an IAM authenticated AWS S3 bucket are passing before a release
  • ✅ No steps of the signing algorithm have deliberately been left out
  • AWSSDK.Core is reused as much as possible, thus the dependency
  • ✅ Signature Version 4 Test Suite scenarios are passing, with the following exceptions:
    • General
      • get-utf8 - The signing algorithm states the following: 'Each path segment must be URI-encoded twice except for Amazon S3 which only gets URI-encoded once.'. This scenario does not URL encode the path segments twice, only once.
      • normalize-path/get-space - The signing algorithm states the following: 'Each path segment must be URI-encoded twice except for Amazon S3 which only gets URI-encoded once.'. This scenario does not URL encode the path segments twice, only once.
      • post-x-www-form-urlencoded - This scenario is based on the fact that we need to specify the charset in the Content-Type header, e.g. Content-Type:application/x-www-form-urlencoded; charset=utf-8. This is not necessary because .NET will add this encoding if omitted by us. We can safely skip this test and rely on integration tests where actual content is sent to an AWS API Gateway.
      • post-x-www-form-urlencoded-parameters - This scenario is based on the fact that we need to specify the charset in the Content-Type header, e.g. Content-Type:application/x-www-form-urlencoded; charset=utf-8. This is not necessary because .NET will add this encoding if omitted by us. We can safely skip this test and rely on integration tests where actual content is sent to an AWS API Gateway.
    • API Gateway
      • get-vanilla-query-unreserved - This scenario defines a request that isn't supported by AWS API Gateway
      • post-sts-token/post-sts-header-before - This scenario is based on the fact that the signing algorithm should support STS tokens, e.g. by assuming a role. This scenario is already covered by numerous other integration tests and can because of this safely be ignored.
    • S3
      • normalize-path/get-slash-pointless-dot - This scenario defines a request that isn't supported by AWS S3.
      • post-header-key-case - This scenario defines a request that isn't supported by AWS S3.
      • post-header-key-sort - This scenario defines a request that isn't supported by AWS S3.
      • post-header-value-case - This scenario defines a request that isn't supported by AWS S3.
      • post-sts-token/post-sts-header-after - This scenario defines a request that isn't supported by AWS S3.
      • post-sts-token/post-sts-header-before - This scenario defines a request that isn't supported by AWS S3.
      • post-vanilla - This scenario defines a request that isn't supported by AWS S3.
      • post-vanilla-empty-query-value - This scenario defines a request that isn't supported by AWS S3.
      • post-vanilla-query - This scenario defines a request that isn't supported by AWS S3.
      • post-x-www-form-urlencoded - This scenario defines a request that isn't supported by AWS S3.
      • post-x-www-form-urlencoded-parameters - This scenario defines a request that isn't supported by AWS S3.
  • ✅ All characters are supported in S3 object keys with the following exceptions:
    • ❌ Plus (+)
    • ❌ Backslash (\)
    • ❌ Left curly brace ({)
    • ❌ Right curly brace (})
    • ❌ Left square bracket ([)
    • ❌ Right square bracket (])
    • ❌ 'Less Than' symbol (<)
    • ❌ 'Greater Than' symbol (>)
    • ❌ Grave accent / back tick (`)
    • ❌ 'Pound' character (#)
    • ❌ Caret (^)
    • ❌ Percent character (%)
    • ❌ Tilde (~)
    • ❌ Vertical bar / pipe (|)
    • ❌ Non-printable ASCII characters (128–255 decimal characters)
    • ❌ Quotation marks
  • Authentication method
    • ✅ HTTP header authentication is supported
    • ❌ Query string authentication is not supported
  • HTTP version
    • ✅ HTTP/1.1 is supported
    • ❌ HTTP/2 is not supported, please create an issue if you wish it to be supported

Install via NuGet

If you want to include AwsSignatureVersion4 in your project, you can install it directly from NuGet.

To install AwsSignatureVersion4, run the following command in the Package Manager Console.

PM> Install-Package AwsSignatureVersion4

You can also install AwsSignatureVersion4 using the dotnet command line interface.

dotnet add package AwsSignatureVersion4

Credit

Thank you JetBrains for your important initiative to support the open source community with free licenses to your products.

JetBrains

aws-signature-version-4's People

Contributors

daniel-rck avatar dependabot[bot] avatar fantasticfiasco avatar github-actions[bot] avatar mungojam avatar renovate-bot avatar renovate[bot] avatar timovzl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

aws-signature-version-4's Issues

PostAsync issues

When using PostAsync we are getting the following error

"Please specify data to sign. (Parameter 'data')"

This is our code

var credentials = new ImmutableCredentials("xxxxxxxxxxx", "xxxxxxxxxxxxxxxxx", null);
var client = new HttpClient();
var content = JsonConvert.SerializeObject(<Object>);
var stringContent = new StringContent(content, Encoding.UTF8, "application/json");

var response = await client.PostAsync(url, stringContent,
    "eu-west-2",
    "",
    credentials);

I'm assuming we are doing something wrong but unable to see where, grateful for any help. Thanks,

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

  • chore(deps): update aws sdk (.net) (AWSSDK.Core, AWSSDK.S3, AWSSDK.SecurityToken)
  • fix(deps): update dependency aws-cdk-lib to v2.138.0
  • fix(deps): update dependency constructs to v10.3.0

Pending Status Checks

These updates await pending status checks. To force their creation now, click the checkbox below.

  • chore(deps): update eslint packages (js) to v7.7.1 (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
  • chore(deps): update xunit-dotnet monorepo to v2.8.0 (xunit, xunit.runner.visualstudio)

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

dockerfile
infrastructure/.devcontainer/Dockerfile
  • mcr.microsoft.com/devcontainers/javascript-node 1-20-bullseye
github-actions
.github/workflows/ci-cd.yml
  • actions/checkout v4
  • codecov/codecov-action v4
.github/workflows/greetings.yml
  • actions/first-interaction v1
.github/workflows/infrastructure.yml
  • actions/checkout v4
  • actions/setup-node v4
.github/workflows/update-copyright-years-in-license-file.yml
  • actions/checkout v4
  • FantasticFiasco/action-update-license-year v3
npm
infrastructure/package.json
  • aws-cdk-lib 2.131.0
  • constructs 10.1.175
  • source-map-support 0.5.21
  • @types/aws-lambda 8.10.137
  • @types/node 20.12.7
  • @typescript-eslint/eslint-plugin 7.7.0
  • @typescript-eslint/parser 7.7.0
  • eslint 8.57.0
  • prettier 3.2.5
  • rimraf 5.0.5
  • typescript 5.4.5
  • ansi-regex >=5.0.1
  • pac-resolver >=5.0.0
  • vm2 >=3.9.4
scripts/package.json
  • @octokit/rest 20.1.0
nuget
src/AwsSignatureVersion4.csproj
  • Microsoft.SourceLink.GitHub 8.0.0
  • AWSSDK.Core 3.7.303.14
test/AwsSignatureVersion4.Test.csproj
  • xunit.runner.visualstudio 2.5.8
  • xunit 2.7.1
  • Shouldly 4.2.1
  • Microsoft.NET.Test.Sdk 17.9.0
  • Microsoft.Extensions.Http.Polly 8.0.4
  • Microsoft.Extensions.Http 8.0.0
  • Microsoft.Extensions.DependencyInjection 8.0.0
  • coverlet.collector 6.0.2
  • AWSSDK.SecurityToken 3.7.300.75
  • AWSSDK.S3 3.7.307.15

  • Check this box to trigger a request for Renovate to run again on this repository

X-Amz-Content-SHA256 should be present when querying Amazon OpenSearch Serverless

Describe the bug

Update: Issue described below with the VPCE is due to the VPCE mutating a signed header. X-Amz-Content-SHA256 header is still a relevant issue but not the one described below.

Only reproducible when querying an Amazon OpenSearchServerless collection with a private network policy (accessibly only via VPCE). I am unable to reproduce on a collection with a public network policy.

Method: GET, RequestUri: 'https://host-id.us-west-2.aoss.amazonaws.com/_cat/indices', Version: 1.1, Content: System.Net.Http.StringContent, Headers:

{

  Accept: */*
  accountid: 855676708012
  User-Agent: curl/8.4.0
  x-forwarded-for: 15.248.7.84
  x-forwarded-port: 443
  x-forwarded-proto: https
  X-Amz-Date: 20240422T182608Z
  x-amz-security-token:  <redacted>
  Host:host-id.us-west-2.aoss.amazonaws.com
  Authorization: AWS4-HMAC-SHA256 Credential=ASIA4OOSOMSWA3GBWWEE/20240422/us-west-2/aoss/aws4_request, SignedHeaders=accept;accountid;host;user-agent;x-amz-date;x-amz-security-token;x-forwarded-for;x-forwarded-port;x-forwarded-proto, Signature=ec37d19eaf6227efe60c26a8690242c2126ef5449ed29a3755105e939d93004b
  Content-Length: 0
  Content-Type: application/json; charset=utf-8

} with  made by arn:aws:sts::855676708012:assumed-role/InsightsStack-EtlStorageOpenSearchServerlessApiProx-UMqk35Yx0VO4/InsightsStack-EtlStorageOpenSearchServerlessApiPro-p6lr3QP2XtH5 failed with StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:

{

  X-Request-ID: d8d04efc-0f9e-97d6-bf07-bc655fd42594
  Date: Mon, 22 Apr 2024 18:26:08 GMT
  x-aoss-response-hint: X01:gw-helper-deny
  Server: aoss-amazon
  Content-Type: application/json
  Content-Length: 121
} Forbidden is not successful with '{"status":403,"request-id":"d8d04efc-0f9e-97d6-bf07-bc655fd42594","error":{"reason":"403 Forbidden","type":"Forbidden"}}

--

Based on the documentation,

The following requirements apply when signing requests to OpenSearch Serverless collections when you construct HTTP requests with another clients.
You must specify the service name as aoss.
The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. If there's a request payload, set the value to its Secure Hash Algorithm (SHA) cryptographic hash (SHA256). If there's no request payload, set the value to e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, which is the hash of an empty string.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-clients.html#serverless-signing

Expected Behavior

X-Amz-Content-SHA256 should be present when service identifier is 'aoss'

Current Behavior

X-Amz-Content-SHA256 is not present when querying Amazon OpenSearch Serverless

Reproduction Steps

Infra
Client -> APIG -> Lambda -> VPCE -> AOSS (private)

Via the APIGateway Service Console:

  1. Create an API Gateway with a Lambda proxy integration.

Via the Lambda Service Console:

  1. Create a Lambda function

Via the OpenSearch Service Console:

  1. Create an Amazon OpenSearchServerless (AOSS) collection. When creating...
  2. Configure the collection with a "private" network policy.
  3. Create VPCE to access the private instance. Ensure the VPCE and Lambda are in the same subnet.
    1. Add an Inbound / Outbound rule for the Lambda SG
    2. Add an Inbound / Outbound rule for the AOSS SG
  4. Configure the collection with a data access policy. Add the Lambda execution role to the policy.

Sample Lambda Code

using System.Net;
using System.Net.Security;
using System.Security.Authentication;
using System.Text;
using System.Text.Json;

using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Amazon.Lambda.Serialization.SystemTextJson;
using Amazon.Runtime;

using AWS.Lambda.Powertools.Logging;

using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;

using EnvironmentVariables = Insights.Shared.Variables.EnvironmentVariables;
using HttpMethod = System.Net.Http.HttpMethod;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(DefaultLambdaJsonSerializer))]

namespace Functions;

//You can simplify to just query aoss directly in the lambda. This function is a generic proxy.
public class Function
{
    private const string ServiceKey = "awsService";
    private const string ServiceUriKey = "awsUri";
    private static readonly string Region = "us-west-2";
    /// <summary>
    /// Filter out query parameters used to configure the proxy request as these may cause the receiving service to fail. e.g. OpenSearch returns BadRequest when there are excess parameters
    /// </summary>
    private static readonly IReadOnlySet<string> ReservedQueryParameterKeys = new HashSet<string>([ServiceKey, ServiceUriKey]);
    /// <summary>
    /// Headers which should not be forwarded.
    /// See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection
    /// See https://www.rfc-editor.org/rfc/rfc7230#section-6.1
    /// </summary>
    private static readonly IReadOnlySet<string> HopByHopHeaders = new HashSet<string>([
        //sigv4
        "authorization",
        //standard
        "host",
        "connection",
        "keep-alive",
        "proxy-authenticate",
        "proxy-authorization",
        "te",
        "trailer",
        "transfer-encoding",
        "upgrade"
    ]);

    private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(30);
    private static HttpClient CreateHttpClient()
    {
        var handler = new SocketsHttpHandler
        {
            PooledConnectionLifetime = Timeout * 2,
            ConnectTimeout = Timeout,
            SslOptions = new SslClientAuthenticationOptions
            {
                EnabledSslProtocols = SslProtocols.None
            }
        };
        return new HttpClient(handler)
        {
            Timeout = Timeout
        };
    }

    public HttpClient HttpClient { get; init; } = CreateHttpClient();
    public ILogger Logger { get; init; } = AWS.Lambda.Powertools.Logging.Logger.Create<Function>();
    public Func<AWSCredentials> CredentialProvider { get; init; } = FallbackCredentialsFactory.GetCredentials;
    //Absurd levels of indirection to mock things out
    public Func<HttpClient, HttpRequestMessage, string, string, AWSCredentials, Task<HttpResponseMessage>> SignAndSendAsyncFunc { get; init; } = SignAndSendAsync;

    [Logging(LogEvent = true)]
    public async Task<APIGatewayHttpApiV2ProxyResponse> HandleRequestAsync(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)
    {
        try
        {
            using HttpResponseMessage response = await this.SendRequestAsync(request);
            string content = await response.Content.ReadAsStringAsync();
            if (response.IsSuccessStatusCode == false)
            {
                string requestContent = await (response.RequestMessage?.Content?.ReadAsStringAsync() ?? Task.FromResult(string.Empty));
                this.Logger.LogError("'{Request}' with '{RequestContent}' failed with '{Response}' '{Content}'", response.RequestMessage, requestContent, response, content);
            }
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = (int) response.StatusCode,
                Headers = response.Headers.ToDictionary(kvp => kvp.Key, kvp => string.Join(",", kvp.Value)),
                Body = content,
                IsBase64Encoded = false
            };
        }
        catch (HttpRequestException e)
        {
            this.Logger.LogError(e, "Failed to proxy request to AWS service");
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = (int) (e.StatusCode ?? HttpStatusCode.InternalServerError),
                Headers = new Dictionary<string, string>
                {
                    { "Content-Type", "application/json; charset=utf-8" }
                },
                Body = JsonSerializer.Serialize(new ErrorResponse(e.StatusCode ?? HttpStatusCode.InternalServerError, e.Message, request.RequestContext.RequestId), new JsonSerializerOptions().ConfigureDefaults()),
                IsBase64Encoded = false
            };
        }
        catch (Exception e)
        {
            this.Logger.LogError(e, "Failed to proxy request to AWS service");
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = StatusCodes.Status400BadRequest,
                Headers = new Dictionary<string, string>
                {
                    { "Content-Type", "application/json; charset=utf-8" }
                },
                Body = JsonSerializer.Serialize(new ErrorResponse(HttpStatusCode.BadRequest, e.Message, request.RequestContext.RequestId), new JsonSerializerOptions().ConfigureDefaults()),
                IsBase64Encoded = false
            };
        }
    }

    private async Task<HttpResponseMessage> SendRequestAsync(APIGatewayHttpApiV2ProxyRequest apiRequest)
    {
        var headers = apiRequest.Headers
            .Select(kvp => new KeyValuePair<string, string>(kvp.Key.ToLowerInvariant(), kvp.Value))
            //Remove 'x-amz' and 'authorization' SigV4 headers as these will cause the signing to fail
            .Where(kvp => kvp.Key.StartsWith("x-amz") == false)
            .Where(kvp => HopByHopHeaders.Contains(kvp.Key) == false)
            .GroupBy(kvp => kvp.Key)
            .ToDictionary(group => group.Key, group => string.Join(',', group.Select(kvp => kvp.Value)));

        if (string.IsNullOrWhiteSpace(apiRequest.RequestContext.Http.Method))
        {
            throw new InvalidOperationException("APIGatewayHttpApiV2ProxyRequest.RequestContext.Http.Method is empty / not set");
        }
        var method = HttpMethod.Parse(apiRequest.RequestContext.Http.Method);

        bool isContentTypeRequired = HttpMethod.Put.Equals(method) || HttpMethod.Post.Equals(method) || HttpMethod.Patch.Equals(method);
        if (headers.TryGetValue("content-type", out string? contentType) == false || string.IsNullOrWhiteSpace(contentType))
        {
            if (isContentTypeRequired)
            {
                throw new InvalidOperationException($"APIGatewayProxyRequest.Header 'content-type' is empty / not set and required for method '{method}'");
            }
        }
        if (apiRequest.PathParameters.TryGetValue("proxy", out string? path) == false || string.IsNullOrWhiteSpace(path))
        {
            throw new InvalidOperationException("APIGatewayHttpApiV2ProxyRequest.PathParameters 'proxy' is empty / not set. Cannot forward request");
        }

        if (apiRequest.QueryStringParameters.TryGetValue(ServiceKey, out string? service) == false || string.IsNullOrWhiteSpace(service))
        {
            throw new InvalidOperationException(
                $"APIGatewayHttpApiV2ProxyRequest.QueryStringParameters '{ServiceKey}' is empty / not set. Cannot forward request. Expect service authorization token. " +
                $"e.g. 'aoss'. See https://docs.aws.amazon.com/service-authorization/latest/reference/reference_policies_actions-resources-contextkeys.html");
        }

        if (apiRequest.QueryStringParameters.TryGetValue(ServiceUriKey, out string? baseUri) == false || string.IsNullOrWhiteSpace(baseUri))
        {
            throw new InvalidOperationException(
                $"APIGatewayHttpApiV2ProxyRequest.QueryStringParameters '{ServiceUriKey}' is empty / not set. Cannot forward request. Expect absolute Uri. e.g. 'https://domain-id.us-west-2.aoss.amazonaws.com'");
        }

        var parameterList = apiRequest.QueryStringParameters
            .Where(kvp => ReservedQueryParameterKeys.Contains(kvp.Key) == false)
            .Select(kvp => $"{kvp.Key}={kvp.Value}")
            .ToList();
        string parameters = parameterList.Count == 0 ? string.Empty : "?" + string.Join("&", parameterList);
        //We need to add '/' because the proxy path parameter does not include it
        var requestUri = new Uri($"{baseUri}/{path}{parameters}", UriKind.Absolute);
        string body = (apiRequest.IsBase64Encoded ? Encoding.UTF8.GetString(Convert.FromBase64String(apiRequest.Body)) : apiRequest.Body) ?? string.Empty;
        var request = new HttpRequestMessage(method, requestUri)
        {
            Content = new StringContent(body)
        };

        foreach (KeyValuePair<string, string> header in headers)
        {
            //Content headers must go into the content headers else it throws
            if (header.Key.StartsWith("content"))
            {
                _ = request.Content.Headers.Remove(header.Key);
                _ = request.Content.Headers.TryAddWithoutValidation(header.Key, header.Value);
            }
            else
            {
                //Remove request defaults because .NET hates humanity
                _ = request.Headers.Remove(header.Key);
                _ = request.Headers.TryAddWithoutValidation(header.Key, header.Value);
            }
        }
        this.Logger.LogInformation("Sending {HttpRequestMessage} to {Service} at {Region}", request, service, Region);
        return await this.SignAndSendAsyncFunc.Invoke(this.HttpClient, request, Region, service, this.CredentialProvider.Invoke());
    }

    private static async Task<HttpResponseMessage> SignAndSendAsync(HttpClient client, HttpRequestMessage request, string region, string service, AWSCredentials credentials) =>
        await client.SendAsync(request, region, service, credentials);

    private sealed record ErrorResponse(HttpStatusCode StatusCode, string Message, string RequestId);
}

Possible Solution

Update https://github.com/FantasticFiasco/aws-signature-version-4/blob/master/src/Private/Signer.cs#L138

Additional Information/Context

Context

I am attempting to proxy requests through API Gateway to our private AOSS collection via VPCE.

Infra
Client -> APIG -> Lambda -> VPCE -> AOSS

Security

  1. APIG
    1. IAM Authorizer
  2. Lambda
    1. Execution Role
      1. Allows all aoss operations
    2. Verified it is in the same VPC / Subnets as VPCE
  3. VPCE
    1. Security Group
      1. Allows Task Ingress / Egress
      2. Allows Lambda Ingress / Egress
      3. Allows AOSS Ingress / Egress
  4. AOSS
    1. Network Policy - insightsstack-network-3cd87f61b1
      1. Allows VPCE
    2. DataAccess Policy - insightsstack-data-access-0c0bf1
      1. Allows ECS Task Role
      2. Allows Lambda Task Role

AWS .NET SDK and/or Package version used

3.7.303.38

Targeted .NET Platform

.NET 8

Operating System and version

MacOS 13.6.1 (22G313), Amazon Linux

HttpClient.Send (synchronous API) is not intercepted

Describe the bug

The package correctly intercepts HttpClient.SendAsync() (and all the extension methods that invoke it, such as GetAsync() or PostAsync()), adding the Authorization header and such.

However, the synchronous API is not properly intercepted. HttpClient.Send() does not cause the necessary headers to be added.

Presumably, this is because this API was added in .NET 5. A new target would be needed.

To Reproduce

In .NET 5, create an HttpClient configured with this package's message handler, which normally signs any request made. Construct an HttpRequestMessage and send it using HttpClient.Send(). After sending, confirm the absence of headers on the HttpRequestMessage. Compare this to HttpClient.SendAsync(), which causes the headers to be added as expected.

Expected behavior

On send, the necessary headers are expected to be added to the HttpRequestMessage, such as the Authorization header.

Environment

.NET 5.

Additional context

It would probably make sense to include .NET 5 as an additional target platform, so that the synchrous API can be intercepted overridden.

There are significant use cases for the HttpClient's synchronous API. The original proposal contains extensive explanations and discussions about it. It was finally added in .NET 5.

Stephen Toub explains:

The key is [sync-over-async] ends up requiring many more threads in order to sustain throughput. And it can take a long time for the thread pool to ramp up to the necessary level, while in the interim the system can essentially be deadlocked. That means either the app-level dev needs to explicitly set a high min thread count to force the ramp up, or we need to make the pool way more aggressive at thread injection, which harms other cases.

My own experience confirms that thread pool exhaustion happens extremely quickly with sync-over-async - much more so than one might expect. This limits the number of simultaneous operations to a fraction of what is achievable with a purely synchronous API.

In a .NET 5 preview blog, Stephen Toub further summarizes the addition of a synchronous API:

While HttpClient was designed for asynchronous usage, we have found situations where developers are unable to utilize asynchrony, such as when implementing an interface method that’s only synchronous, or being called from a native operation that requires a response synchronously, yet the need to download data is ubiquitous. In these cases, forcing the developer to perform “sync over async” (meaning performing an asynchronous operation and then blocking waiting for it to complete) performs and scales worse than if a synchronous operation were used in the first place. As such, .NET 5 sees limited new synchronous surface area added to HttpClient and its supporting types. dotnet/runtime does itself have use for this in a few places. For example, on Linux when the X509Certificates support needs to download a certificate as part of chain building, it is generally on a code path that needs to be synchronous all the way back to an OpenSSL callback; previously this would use HttpClient.GetByteArrayAsync and then block waiting for it to complete, but that was shown to cause noticeable scalability problems for some users…

CancellationToken parameters must come last

According to CA1068 the cancellation token must come last. This would be applicable to both private/internal and public methods.

Since this is a breaking change, we will have to wait with fixing this issue.

Default Request Headers are double added on Android

Describe the bug
A HttpClient with client.DefaultRequestHeaders.Add("x-api-key", myApiKey); is signed correctly, but the client itself adds the header twice into the request:

"Headers": [
{
  "Key": "x-api-key",
  "Value": [
    "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  ]
},
{
  "Key": "X-Amz-Date",
  "Value": [
...
}

To Reproduce
Steps to reproduce the behavior:
Try this code:

var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", accessToken);

// do a random request
var response = await client.GetAsync(requestUri, cancellationToken, regionName, serviceName, credentials);

You can see the request object before sending here.

Compare to the request object that is return in response.RequestMessage.

Expected behavior
The Request Header should contain the header "x-api-key" only once. Like this:

"Headers": [
{
  "Key": "x-api-key",
  "Value": [
    "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  ]
},
{
  "Key": "X-Amz-Date",
  "Value": [
...
}

Environment (please complete the following information):

  • AWS API Gateway]
  • OS: Android 8.0
  • .NET version: Mono.Android

Additional context
I have tested an .NET Core 2.1 Console application where everything runs fine.

Example Request: use with retrieving credentials from running instance

Hi! I'm being tasked with providing some examples for showing signature requests in multiple languages for our dev teams since I'm the one who suggested we use signed API-GATEWAY requests for a new project :)

This led me to find this package as .NET AWS SDK does not apparently have its own.

In our use case (and I imagine many others) the credentials used to sign the request will be coming from the running role of ec2/eks/ecs/lambda etc.

I've been looking for an example on .NET of how to retrieve the running instance session credentials but not having much luck.

If you could provide an example on how to sign using this package and the IAM ROLE provided credentials that would help me (and I assume many ohters!) out quite a bit.

Thanks!

Working sample for simple GET request

New to github so sorry if this is the wrong place to ask. I am pretty confused with AWS singing and I have to talk to one of their new APIs (no SDK available) without any previous experience. Do you have a working example showing a full simple get request using your library to sign it? I am struggling getting off the ground with their docs and no experts to ask for help. Hoping for a little nudge.

Support Unity game engine

Discussed in #474

Originally posted by AUEzzat August 19, 2021
Have anyone tried using it with unity?
I have an issue in il2cpp related to System.Web dll

Error: called non-existent method System.Collections.Specialized.NameValueCollection System.Web.HttpUtility::ParseQueryString(System.String)

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

File: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Invalid regExp for packageRules[0].packagePatterns: ^@aws-cdk,^aws-cdk``

Overloads of GetStringAsync

Is your feature request related to a problem? Please describe.
The GetStringAsync() extension on HttpClient is exactly what I need, but it's not currently supported with this (super) package

Describe the solution you'd like
I'd like it to work just like the other overloads, taking the AWS credentials as a parameter

Describe alternatives you've considered
I can call GetAsync() instead but then I need to work a bit harder with the response status codes etc.

Usage of IHttpClientFactory won't work in scenario where HttpClient is reused over longer time than passed token is valid

Describe the bug

In one of the projects that I am working on, I have used the approach described in https://github.com/FantasticFiasco/aws-signature-version-4#integration-with-ihttpclientfactory. Unfortunately, I believe it won't work in cases where for example I am passing a temporary token valid for a short amount of time, and the HttpClient is going to be reused after the token was expired.

To Reproduce

  1. Register services with AwsSignatureHandlerSettings that get token valid for a limited amount of time.
  2. Reuse HttpClient after the credentials passed to AwsSignatureHandlerSettings expired.

Expected behavior

It would be great to have some sort of refresh mechanism where we could update credentials passed to AwsSignatureHandlerSettings after a certain amount of time.

Actual behavior

Calls using expired token are responded with The security token included in the request is expired

Add PatchAsync overload

Is your feature request related to a problem? Please describe.
One of our APIs uses PATCH which isn't directly supported by this library.

Describe the solution you'd like
PATCH overload that would be very similar to the POST overload that already exists.

Describe alternatives you've considered
I'll use your SendAsync overload for now.

Additional context
I'll hopefully contribute more to this library in future, but lots going on at the moment. Thanks again for making it!

How To Mock SendAsyncExtensions.SendAsync()?

Since SendAsyncExtensions.SendAsync() is an extension method, it cannot be mocked with Moq. What is the recommendation for mocking this method in unit tests?

I tried to mock the underlying method that is being called, but it looks like it never gets invoked when calling this extension method.

_mockHttpClient .Setup(x => x.SendAsync(It.IsAny<HttpRequestMessage>(), It.IsAny<CancellationToken>()))

Support Amazon S3

This package does not support Amazon S3 (Simple Storage Service). The decision for excluding S3 is based on the fact that it is a unique service when it comes to Signature Version 4, with exceptions to the rules of the signing algorithm.

Please give a thumbs up for this feature if you are requiring support for S3, and with enough votes, it will be implemented.

Don't push coverage on pull requests

Integration tests are not run on pull requests, thus it always looks like pull requests will lower the coverage.

We can run coverage on pull requests if we wan't, but we don't have to push the result.

Struggling to make API Gateway call

Just trying to write a simple console app to call API gateway using IAM authentication pulled from the IAM role associated with the EC2 instance the code is running on.

But I can't even get the code to compile, I must be missing something obvious but I have read the examples and cannot see what I am doing wrong. It's like .NET doesn't see the HttpClient() extensions your library provides.

I looked through the test folder and also Google'd for examples but came up empty.

We are using .NET Core 2.1 at present, but soon to be .NET Core 3.1

using System;
using System.Net.Http;
using System.Threading.Tasks;
using Amazon.Runtime;
using AwsSignatureVersion4;

namespace console
{
    class Program
    {
        static void Main(string[] args)
        {
          MainAsync().GetAwaiter().GetResult();
        }

      private static async Task MainAsync()
      {
        var credentials = new InstanceProfileAWSCredentials();
        var client = new HttpClient();
        client.DefaultRequestHeaders.Add("x-apigw-api-id","#####");
        var response = await client.GetAsync(
        "https://#####.execute-api.ap-southeast-2.vpce.amazonaws.com/Prod/api/values",
        regionName: "ap-southeast-2",
        serviceName: "execute-api",
        credentials: credentials);
      }
    }
}

NOTE: I removed the specifics of our environment.

Build error is

Program.cs(23,9): error CS1739: The best overload for 'GetAsync' does not have a parameter named 'regionName' [/home/ubuntu/code/console/console.csproj]

Any assistance would be greatly appreciated.

Deadlock when calling the code from an asp.net 4.8 web forms code behind

Describe the bug
Deadlock when calling the code from an asp.net 4.8 web forms code behind which is a synchronous code.

To Reproduce
Steps to reproduce the behavior:

  1. Create an asp.net web forms project
  2. Put following code in page_load event of a default.aspx page:
  3.  var content = new StringContent("..."
             , Encoding.UTF8, "application/json");
         var request = new HttpRequestMessage
         {
             Method = HttpMethod.Post,
             RequestUri = new Uri("..."),
             Content = content,
         };
         var credentials = new ImmutableCredentials("...", "...", null);
         var client = new HttpClient();
         var response = client.SendAsync(
           request: request,
           regionName: "...",
           serviceName: "execute-api",
           credentials: credentials).ConfigureAwait(false).GetAwaiter().GetResult();
    
  4. Code hangs

Expected behavior
Method should not hang

Environment (please complete the following information):

  • AWS service [AWS API Gateway]
  • OS: [Windows 10]
  • .NET version: [4.8]

Binary content does not POST. 408 error returned.

Passing a binary content will return 408 from the endpoint after a long wait.

To Reproduce

The following code will return a 408 from the service

var credentials = new ImmutableCredentials("AKIA2WNEXAMPLE", "rObND507EXAMPLE", null);

var client = new HttpClient();

byte[] fileBytes = File.ReadAllBytes(@"C:\File.pdf");
MemoryStream destination = new MemoryStream(fileBytes);
destination.Seek(0, SeekOrigin.Begin);

HttpContent content = new StreamContent(destination);

content.Headers.Add("x-api-key", "apikey");

var response = await client.PostAsync(
  "https://uat-customamazonservice.domain.net/v2/extract?inpFormat=PDF",
  content: content,
  regionName: "us-east-1",
  serviceName: "execute-api",
  credentials: credentials);

Checking web debugging tool (e.g. fiddler) you can see the content of the request is empty.

Expected behavior

The request should be sent and processed.

Desktop (please complete the following information):

Windows 11 .net 6 console client.

Additional context

If you send base64 encoded StringContent rather than StreamContent, the request is successful.

The fix is to reset the stream back to the beginning. I suggest doing this when the content hash is created by adding the following at line 30 in ContentHashcs.

contentStream.Seek(0, System.IO.SeekOrigin.Begin);

Could not load file or assembly 'AwsSignatureVersion4, Version=4.0.0.0

I am making API calls to Lambda function url with HttpClient in an IntegrationTestSuite. When run independently, it works fine but when I call the IntegrationTestSuite.dll from another project, I get the below exception
System.IO.FileNotFoundException: 'Could not load file or assembly 'AwsSignatureVersion4, Version=4.0.0.0, Culture=neutral, PublicKeyToken=9830731f15ae0355'. The system cannot find the file specified.'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.