cloudydeno / deno-aws_api Goto Github PK

code:

const s3 = new ApiFactory({
  credentials: {
    awsAccessKeyId: Deno.env.get("S3_ACCESS_KEY_ID")!,
    awsSecretKey: Deno.env.get("S3_SECRET_KEY")!,
  },
  fixedEndpoint: "https://s3.us-west-004.backblazeb2.com",
  region: "us-west-004",
}).makeNew(S3);

  await s3.createBucket({
    Bucket: s3Bucket,
  });

error:

header 'x-amz-content-sha256' must be included in signature [Request ID: 6df5c7d7d56e595d

sha256 is not sent alongside requests to b2 (b2 requires sha256's)

I was looking to use getSignedUrl for uploading directly to s3, but didn't see the method in the generated s3 service. I do see it on the s3 service on the aws sdk referenced on main

Am I looking in the wrong place? Thanks for your help!

There is some choice of style here:

1. Do the funcs yield each page's response in its entirety?
2. Do the funcs yield each inner item individually, hiding the page concept?
3. Should both styles be implemented for extra flexibility?

My instinct is that it's pretty easy to write a custom loop if visibility of the full pages are required, so the convenience functions should hide the pages. But I could be convinced otherwise I think.

Looking at s3.upload(), that specific function is implemented by the AWS.S3.ManagedUpload class. It appears to chop your stream into individual 5MB segments and upload them with a Multipart Upload strategy. This is actually separate from streaming request bodies because each 'part' is buffered. So I will track managed/multipart uploads in a separate issue 😅

Originally posted by @danopia in #24 (comment)

When AWS_STS_REGIONAL_ENDPOINTS is regional then use the region sts:AssumeRoleWithWebIdentity endpoint instead of the global one.

Variable is set by this: aws/amazon-eks-pod-identity-webhook@c1fc358#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R90

The Glacier module didn't actually work because of a Glacier header quirk which aws-sdk-js handles directly. There is a whole directory of these: https://github.com/aws/aws-sdk-js/tree/master/lib/services

Some of these quirks are just extra headers. Others are changes to hashing, or additional checksums. Cloudfront adds a custom signer for making cookies. S3's is basically a rewrite of the client, 1000+ extra lines.

Every service in this folder should be triaged for whether the quirk is blocking, how important it is, if the service needs to be marked "broken" until the quirk is implemented, etc.

We're trying to use this package in a deno app deployed via ECS and we're getting credential errors when trying to connect to dynamodb. On inspection of lib/client/credentials.ts, I noticed that ECSCredential handling is commented out/not included.

Adding support for this would be really helpful.

With the official AWS SDK, I can switch accounts/roles by defining profiles in ~/.aws/config. All these profiles may inherit from the same "credential" profile, specified through source_profile.

The Deno AWS API, however, only reads profiles from ~/.aws/credentials and doesn't seem to support role-arn or reading the config file at all.

Just ran into this:

error: Uncaught (in promise) Error: Failed to load any possible AWS credentials:
    - EnvironmentCredentials('AWS') Error: AWS environment variables not set
    - EnvironmentCredentials('AMAZON') Error: AMAZON environment variables not set
    - SharedIniFileCredentials() Error: Profile dev not found in credentials file
    - EcsTaskCredentials() Error: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI not set
    - TokenFileWebIdentityCredentials() Error: No WebIdentityToken file path is set
    - EC2MetadataCredentials() Error: Instance Metadata Timeout: 1000ms
    return Promise.reject(new Error([
                          ^
    at CredentialsProviderChain.getCredentials (https://deno.land/x/[email protected]/client/credentials.ts:44:27)
    at eventLoopTick (ext:core/01_core.js:168:7)
    at async XmlServiceClient.signingFetcher [as signedFetcher] (https://deno.land/x/[email protected]/client/client.ts:89:29)
    at async XmlServiceClient.performRequest (https://deno.land/x/[email protected]/client/client.ts:179:22)
    at async S3.listObjectsV2 (https://aws-api.deno.dev/v0.4/services/s3.ts:1601:18)

An example:

~/.aws/credentials:

[default]
aws_access_key_id = ABCDEFG
aws_secret_access_key = abcdefg

~/.aws/config:

[profile dev]
region = eu-west-1
output = json
role_arn = arn:aws:iam::123456:role/abc
source_profile = default

[profile staging]
region = eu-west-1
output = json
role_arn = arn:aws:iam::654321:role/def
source_profile = default

Furthermore, the Deno AWS API implementation also violates the following:

Depending on the file, profile section names use the following format:

• Config file: [default] [profile user1]
• Credentials file: [default] [user1]
  Do not use the word profile when creating an entry in the credentials file.

Ref: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

First of all thank you for making this module, it is very nice! I noticed that on windows there is an error when trying to load the AWS credentials. My credentials are located in the default location for the file on windows C:\Users\<MyUser>\.aws\credentials and this is an example of code that errors:

import { ApiFactory } from 'https://deno.land/x/[email protected]/client/mod.ts';
import { CloudFormation } from "https://aws-api.deno.dev/v0.3/services/cloudformation.ts";

const cfn = new ApiFactory({
    region: 'eu-west-3'
}).makeNew(CloudFormation);

const result = await cfn.describeStacks();

console.log(result);

Error:

deno run -A  .\life.ts
error: Uncaught (in promise) Error: Failed to load any possible AWS credentials:
    - ()=>new EnvironmentCredentials('AWS') Error: AWS environment variables not set
    - ()=>new EnvironmentCredentials('AMAZON') Error: AMAZON environment variables not set
    - ()=>new SharedIniFileCredentials() NotFound: The system cannot find the path specified. (os error 3)    
    - ()=>new EcsTaskCredentials() Error: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI not set
    - ()=>new TokenFileWebIdentityCredentials() Error: No WebIdentityToken file path is set
    - ()=>new EC2MetadataCredentials() TypeError: error sending request for url (http://169.254.169.254/latest/api/token): error trying to connect: tcp connect error: A socket operation was attempted to an unreachable network. (os error 10051)
    return Promise.reject(new Error([
                          ^
    at CredentialsProviderChain.getCredentials (https://deno.land/x/[email protected]/client/credentials.ts:44:27)
    at async QueryServiceClient.signingFetcher [as signedFetcher] (https://deno.land/x/[email protected]/client/client.ts:89:29)
    at async QueryServiceClient.performRequest (https://deno.land/x/[email protected]/client/client.ts:179:22)   
    at async CloudFormation.describeStacks (https://aws-api.deno.dev/v0.3/services/cloudformation.ts:606:18)  
    at async file:///C:/Users/GJZwiers/repos/deno_deno/life.ts:8:16

I also use WSL (Ubuntu) with credentials at ~/.aws/credentials and on there it works as normal.

I see this note on the getObject implementation for S3 and I know #24 exists, but seems to be focused on uploading objects, not getting objects. Being able to stream objects down from S3 would be awesome.

My use case:

My S3 buckets are locked down and can't be publicly accessed, so I would like to stream an object from S3, through my server, to the client, without needing to buffer the entire object. A workaround would be to create a presigned url for retrieving the object from s3 and the client using that, instead of my server, I just don't like exposing the underlying cloud infra, if that makes sense.

I would like to add AWS Transfer Family as a service. How do I go about doing this? How does 'codegen' work?

My immediate goal is to use an SFTP Connector to bring csv files into Supabase storage bucket using the S3 protocol.

Also, what is the general state of this module? Are we keeping in sync with the latest aws-sdk? Or have we abandoned this module as Deno supports more and more of Node and we use the aws-sdk v3 directly?

Different types:

• AWS Govcloud, AWS China (so-called AWS partitions)
• dualstack endpoint for EC2, e.g. https://api.ec2.eu-west-1.aws/status
• Localstack/minio (single origin endpoints, possibly unencrypted localhost)
• FIPS endpoints (some sort of different encryption)

In the first example in the readme, the path to import one of the URLs is incorrect:

import { STS } from 'https://deno.land/x/aws_api/services/sts.ts'; results in 404 Not Found.

The URL should instead be: https://deno.land/x/aws_api/services/sts/mod.ts

I tried getting a path like /2013-04-01/hostedzone/%2Fhostedzone%2FZ1LZDA11I5YD8/rrset by accident, and it broke the signature:

AwsServiceError: InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

Probably need to consult more docs and/or see if the aws-fetch project handles this case better.

Most important feature is showing a list of API actions and letting the user manually check individual ones and/or add glob patterns and show which/how many actions the globs are including.

• Documentation selection level (none/short/full)
• Checkboxes for waiters and paginators - should be good with just toggling whether they are emitted overall, and as now they'll only emit if the action they wrap is also emitted
• List of checkboxes or similar for all actions
• Text Input[s] for action name globs, which dynamically check & gray out all matching actions in the above list
• Button to load the new file and display the file size
• Download button (gives .ts)
• Should be accessible & pre-filled from any URL that can be imported into Deno
• Can depend on Javascript :)

Usage (for B2) looks something like this:

import { ApiFactory } from "https://raw.githubusercontent.com/cloudydeno/deno-aws_api/666991bff325933f28954bc4878fca6aff2f89a1/lib/client/mod.ts";
import { S3 } from "https://aws-api.deno.dev/v0.3/services/s3.ts";

export const minio = new ApiFactory({
  credentials: {
    awsAccessKeyId: Deno.env.get("S3_ACCESS_KEY_ID")!,
    awsSecretKey: Deno.env.get("S3_SECRET_KEY")!,
  },
  fixedEndpoint: "https://s3.us-west-004.backblazeb2.com",
  region: "us-west-004",
}).makeNew(S3);

Currently, there's no version that includes fixed B2 support. The version should be probably bumped..?

Code

#!/usr/bin/env -S deno run --unstable --allow-env --allow-run --allow-net

import { ApiFactory } from "https://deno.land/x/[email protected]/client/mod.ts";
import { S3 } from "https://aws-api.deno.dev/v0.2/services/s3.ts";

const makeS3Client = async () => {
  const factory = new ApiFactory({
    region: AWS_REGION,
    credentials: {
      awsAccessKeyId: AWS_ACCESS_KEY_ID,
      awsSecretKey: AWS_SECRET_ACCESS_KEY,
      sessionToken: AWS_SESSION_TOKEN,
      expiresAt: AWS_SESSION_EXPIRATION
    }
  });

  await factory.ensureCredentialsAvailable();
  return factory.makeNew(S3);
}

(async function () {
  const s3 = makeS3Client()
})()

Expected

No error

Actual

error: TS18022 [ERROR]: A method cannot be named with a private identifier.
  async #performRawRequest(opts: {
        ~~~~~~~~~~~~~~~~~~
    at https://deno.land/x/[email protected]/client/instance-metadata.ts:65:9

Here's the link to the javascript api for createPresignedPost.

Its possible that these functions in javascript skd have been manually added these since they're not actually api endpoints. Is there some utility class in here where I could effectively sign urls still?

For context, in case you're not aware, these two singing apis will create a url with an encrypted token in it which you can then hand off to someone else, including a browser, and it can then be used to fetch or upload a file directly from the browser. This is how you'd manage access to private buckets and also its a pretty slick way to handle file uploads without having to go through your api server at all.

1. EC2 InstanceNetworkInterfaceSpecification is slightly incorrect comparing to https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-network-iface-embedded.html. Fields such as Ipv4Prefixes, Ivp4Addresses are optional according to documentation. Groups are actually spelled GroupSet.
2. Running runInstnaces with NetworkInterfaces supplied in request lead to exception from AWS

Interrupted AwsServiceError: UnknownParameter: The parameter networkInterface is not recognized [Request ID: 4453910d-60c4-4c8a-8edb-169631ff8fdf]

We already have AssumeRoleWithWebIdentity support, so the credential fetching/refreshing will likely be similar. But the base credential is a different credential instead of a token file. So AssumeRole would likely not be in the default credential chain.