bojanbass / nx-aws Goto Github PK

Right now, the only way of skipping the remote cache and using the local one is via the NXCACHE_AWS_DISABLE environment variable.

Ideally, we would like to disable the remote cache by default for local development, but have it enabled for CI. Since we need to set an environment variable to disable the cache, that'd mean we need .env support or using something like crossenv to set the environment var in a cross-os compatible way.

If we add this as an option, we could just do nx build --skip-aws-cache=false on CI and have the default value be true in our NX config.

Happy to make a PR if you think this would be acceptable.

Using Latest NX version and RxJS causes the following issues:

$runner.subscribe is not a function

The issues is isolated to the latest version, and using the older 0.0.5 version with the same NX and RxJS still works.

I think it's because previous version wraps $runner in a from($runner), which creates an observable that can be subscribed.

Current Behavior

Currently, when the task runner is configured to use the @nx-aws-plugin/nx-aws-cache it always prints USING REMOTE CACHE in the terminal.

https://github.com/bojanbass/nx-aws/blob/master/packages/nx-aws-cache/src/tasks-runner/runner.ts#L43

However, when the --skip-nx-cache Nx ignores all the caches, local and remote.

Expected Behavior

When the --skip-nx-cache CLI option is specified the USING REMOTE CACHE should not be printed in the terminal.

Implementation Suggestion

Use the options.skipNxCache to condition the USING REMOTE CACHE log.

Problem

I'm running code on different nodes on CI pipeline execution.
During execution, an attempt of loading the remote cache ends up with the error, as nx does not allow the use of cache from different machines.

Proposed solution

Extend config with trustForeignMachineCache?: boolean property, which will allow to reuse cache generated on machines different than current (by removing source file from downloaded tar).

Workarounds

There is a known workaround by setting REJECT_UNKNOWN_LOCAL_CACHE=0 env variable, but this approach is discouraged. It should be the responsibility of RemoteCache provider (like nx-aws) instead of nx

Please share your POV about this idea. If you have no red flags, let me know and I will prepare a PR for that.

Hi,

thank you for this great project!

Locally, it works perfectly for me, but I have been struggling to run it on AWS CodeBuild. It did not pickup the credentials correctly. The current version 1.1.0 uses fromIni which is not working. Your current master already uses fromNodeProviderChain which works in AWS CodeBuild.

Could you please publish a new version to npm that uses fromNodeProviderChain?

Hey there! Thanks for the plugin, seems like exactly what I was looking for.

I've noticed that in my CI runs, i am definitely producing cache files, I can see them showing up in my bucket. But on subsequent reruns I can see output from the builds, and I'm wondering if I'm actually hitting the cache? I've also tested turning the caching off, and it seems about the same speed.

Do you have any advice for debugging and determining that I am in fact using the cache?

I'm on NX v17
Using Github Actions

Thanks!

I need to set NXCACHE_AWS_DISABLE=true in a .env file and I see the codebase supports that. Every developer on the team is going to want to set this flag for local development, so we want to commit this to the repo. We then flip this using an env var in github actions for CI.

The .env file is also read by another tool for username/login tokens which I would like to not commit.

I currently cannot change where either of these .env files are read from, would it be possible to add API surface to allow passing a different path to a .env file in?

Alternatively, would it be possible to change/flip the nxcache_aws_disabled=true without a .env file?

I saw that in the package.json, there are these peerDependenies entries:

  "peerDependencies": {
    "@angular-devkit/core": ">=11.0.1",
    "@angular-devkit/schematics": ">=11.0.1",
    "@nrwl/workspace": ">=9.2.2"
  },

Since I don't use angular, and since nx14 does not need @nrwl/workspace, can all of these peers be optional?

Related:

• #127
• #126

Since updating to version 3.2.0 we have the following error:

Error: endpointV2Middleware is not found when adding regionRedirectEndpointMiddleware middleware before endpointV2Middleware
    at /node_modules/@smithy/middleware-stack/dist-cjs/index.js:154:17
    at Array.forEach (<anonymous>)
    at getMiddlewareList (/node_modules/@smithy/middleware-stack/dist-cjs/index.js:143:31)
    at Object.resolve (/node_modules/@smithy/middleware-stack/dist-cjs/index.js:297:32)
    at PutObjectCommand.resolveMiddleware (/node_modules/@aws-sdk/client-s3/dist-cjs/commands/PutObjectCommand.js:39:22)
    at S3Client.send (/node_modules/@smithy/smithy-client/dist-cjs/index.js:110:29)
    at Upload.__uploadUsingPut (/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:66:57)
    at Upload.__doConcurrentUpload (/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:128:39)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 0)


 >  NX   Storage Cache: Upload error - Error: endpointV2Middleware is not found when adding regionRedirectEndpointMiddleware middleware before endpointV2Middleware

The config used is:

NXCACHE_AWS_ACCESS_KEY_ID=key
NXCACHE_AWS_SECRET_ACCESS_KEY=key
NXCACHE_AWS_BUCKET=bucket
NXCACHE_AWS_REGION=eu-west-1

When rolling back to 3.1.0, everything works.

Following the instructions in the README, I ran in to this error:

 >  NX   Cannot find module '@nrwl/workspace/tasks-runners/default'

   Require stack:
   - <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/runner.js
   - <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/index.js
   - <repo>/node_modules/nx/src/tasks-runner/run-command.js
   - <repo>/node_modules/nx/src/command-line/run-many.js
   - <repo>/node_modules/nx/src/command-line/nx-commands.js
   - <repo>/node_modules/nx/bin/init-local.js
   - <repo>/node_modules/nx/bin/nx.js

I'm using nx 14 with no plugins, so the default task runner (in nx.json) is: "runner": "nx/tasks-runners/default",
Should that new nx runner path should be used instead?

The breaking change from nx 16.0.0-beta.1.

When migrations are run @nrwl/devkit is replaced by @nx/devkit

> git -c user.useConfigOnly=true commit --quiet --allow-empty-message --file -

 >  NX   Affected criteria defaulted to --base=main --head=HEAD


 >  NX   Cannot find module '@nrwl/devkit'

   Require stack:
   - /../../node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/runner.js
   - /../../node_modules/@nx-aws-plugin/nx-aws-cache/src/index.js
   - /../../node_modules/nx/src/tasks-runner/run-command.js
   - /../../node_modules/nx/src/command-line/affected.js
   - /../../node_modules/nx/src/command-line/nx-commands.js
   - /../../node_modules/nx/bin/init-local.js
   - /../../node_modules/nx/bin/nx.js
   Pass --verbose to see the stacktrace.

husky - pre-commit hook exited with code 1 (error)

Caused by 16069

does this library support IAM roles? seems to always fail for me and falls back to using the default task runner.

the issue i see is checking the config in the AwsCache class

i've tested IAM role support bypassing this check and everything seems to work as expected

I'd like to try out this package, but I'm getting the following output:

$ nx generate @nx-aws-plugin/nx-aws-cache:init


>  NX  Generating @nx-aws-plugin/nx-aws-cache:init

✔ AWS Region (optional - can be defined using Environment variables) · us-east-1
✔ AWS Bucket (optional - can be defined using Environment variables) · nx-distributed-cache

 >  NX   You must use Nx >= 8.0 to enable Storage Cache

   Pass --verbose to see the stacktrace.

$ nx --version
15.7.2

am I missing something to be able to make this work?

A recent bug in my pipeline was causing some files to be missing from the cache. After investigating I narrowed it down to a missing folder called source. Renaming that folder made the cache work as expected.

Investigating further I found the root cause, which was the introduction of this filter:

I am reproducing the issue in this repo, but I may take a while to set the AWS part to make it a full reproduction. Anyway, I hope it helps to illustrate the issue.

The short-term solution was to rename the folder called source to something else, but I wonder how could we better identify if the filtered source is the actual file produced by NX.

I use a self hosted aws, to use that I need the possibility to set the endpoint, otherwise s3.eu-de.amazonaws.com is always used
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/s3clientresolvedconfig.html#endpoint

Would it be possible to extend this solution to support an S3-compatible API like Backblaze B2 or Wasabi?

It seems the only thing missing is the ability to provide an endpoint variable in the configuration and pass it on when creating the S3 object. For backblaze this is explained here.

If you want I can provide a PR.

The only S3 actions that are explicitly used are GetObject, PutObject, PutObjectACL, and GetObjectAttributes, However, the code is written in a way that implicitly requires ListBucket

  private async checkIfCacheExists(hash: string): Promise<boolean> {
    const tgzFileName = this.getTgzFileName(hash),
      params: clientS3.HeadObjectCommand = new clientS3.HeadObjectCommand({
        Bucket: this.bucket,
        Key: this.getS3Key(tgzFileName),
      });

    try {
      await this.s3.send(params);

      return true;
    } catch (err) {
      if ((err as Error).name === 'NotFound') {
        return false;
      } else if (err instanceof CredentialsProviderError) {
        return false;
      }

      throw new Error(`Error checking cache file existence - ${err}`);
    }
  }

if the HeadObjectCommand is used, and the principle has ListBucket permissions, then NotFound will be returned. However if they do not have ListBucket permissions, then 403 will be returned. So caching will fail

See API reasoning https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/HeadObjectCommand/

Permissions
You need the relevant read object (or version) permission for this operation. For more information, see [Actions, resources, and condition keys for Amazon S3 ](https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html)
. If the object you request doesn't exist, the error that Amazon S3 returns depends on whether you also have the s3:ListBucket permission.

If you have the s3:ListBucket permission on the bucket, Amazon S3 returns an HTTP status code 404 error.

If you don’t have the s3:ListBucket permission, Amazon S3 returns an HTTP status code 403 error.

Solution. Check if a 403 was returned, and return false in that case

Thanks for this plugin. Trying to set it up and got this question 😄

Hi, I'm trying out this package and ran into this issue while running init:

NX (0 , devkit_1.readRootPackageJson) is not a function

From what I can tell, that function doesn't exist in the @nx/devkit package. Could be a version mismatch? Not sure, but mine is v16.9.1

I was able to fix/bypass it by changing it to use readJsonFile('package.json') instead

So I tried using the plugin, but no matter which version I use (latest, 16.2.0 or 17.0.0) I always get errors, that there is no compatibility or that nx couldn't be found.

That's my package.json

"dependencies": {
    "@nx-aws-plugin/nx-aws-cache": "^3.1.0",
    "@nx/devkit": "16.2.1",
    "nx": "16.2.1"
  },

Running yarn nx generate @nx-aws-plugin/nx-aws-cache:init --verbose gives the following output:

  
✔ AWS Access Key Id · 
✔ AWS Secret Access Key · 
✔ AWS configuration profile to use · 
✔ AWS Endpoint of the web service if a custom endpoint is needed (e.g. when using MinIO) · 
✔ AWS region to which this client will send requests · us-east-1
✔ AWS S3 Bucket, where the cached Nx outputs will be stored · nx-rempte-cache.myblubilub.com
✔ Whether to force path style URLs for S3 objects (e.g. when using MinIO) (y/N) · false

Error: You must install Nx to enable Storage Cache
    at isCompatibleVersion (/Users/blabliblub/projects/myproject/node_modules/@nx-aws-plugin/nx-aws-cache/src/generators/init/generator.js:10:15)
    at /Users/blabliblub/projects/myproject/node_modules/@nx-aws-plugin/nx-aws-cache/src/generators/init/generator.js:42:14
    at Generator.next (<anonymous>)
    at /Users/blabliblub/projects/myproject/node_modules/tslib/tslib.js:167:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/Users/blabliblub/projects/myproject/node_modules/tslib/tslib.js:163:16)
    at default_1 (/Users/blabliblub/projects/myproject/node_modules/@nx-aws-plugin/nx-aws-cache/src/generators/init/generator.js:41:20)
    at Object.<anonymous> (/Users/blabliblub/projects/myproject/node_modules/nx/src/command-line/generate/generate.js:250:36)
    at Generator.next (<anonymous>)
    at fulfilled (/Users/blabliblub/projects/myproject/node_modules/tslib/tslib.js:164:62)
    

Any ideas or which versions are compatible with this?

 >  NX   Cannot find module '@angular-devkit/schematics'

   Require stack:
   - <repo>/node_modules/@nrwl/workspace/src/utils/ast-utils.js
   - <repo>/node_modules/@nrwl/workspace/src/utils/project-type.js
   - <repo>/node_modules/@nrwl/workspace/index.js
   - <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/logger.js
   - <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/aws-cache.js
   - <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/runner.js
   - <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/index.js
   - <repo>/node_modules/nx/src/tasks-runner/run-command.js
   - <repo>/node_modules/nx/src/command-line/run-many.js
   - <repo>/node_modules/nx/src/command-line/nx-commands.js
   - <repo>/node_modules/nx/bin/init-local.js
   - <repo>/node_modules/nx/bin/nx.js

I don't want to install @angular-devkit/*, because I'm not using angular 😅

I'm using nx@14, and ran in to this error after installing @nrwl/workspace after running into this error: #126

First of all, thank you very much for the plugin.

Problem:
Unfortunately, I'm developing on a Windows computer and have a problem where "/" is normalized to "" in this function https://github.com/bojanbass/nx-aws/blob/ddcc209d6a6f7612e0de30485edb865ed8cdbdb0/packages/nx-aws- cache/src/tasks-runner/aws-cache.ts#L247 .

Possible Solution:
The easiest way is to insert a replace("\","/") after the "join". This means that Windows can work properly and the files do not end up on the S3 storage from a Windows computer with “xyz\123.tar.gz”.