Giter Club home page Giter Club logo

gcp-storage-wagon's Introduction

Maven GCP Storage Wagon Build Status Maven Central Coverage Status

Maven wagon provider for Google Cloud Storage. It allows publishing and downloading of artifacts from Google Cloud Storage bucket.

Usage

1. To allow the plugin to access the bucket configure access to the GCP by following instructions here. The easiest way to do so is by using IAM Service Accounts and configuring environment variable GOOGLE_APPLICATION_CREDENTIALS.

2. Configure the target project to use maven wagon for handling transport via pom.xml:

  <build>
      <extensions>
          <extension>
              <groupId>com.lahsivjar</groupId>
              <artifactId>gcp-storage-wagon</artifactId>
              <!-- Replace gcp.storage.wagon.version with the correct version -->
              <version>${gcp.storage.wagon.version}</version>
          </extension>
      </extensions>
  </build>

Maven loads parent pom before loading any wagon thus if the parent pom also relies on wagon plugin we have to use .mvn/extensions.xml file released in Maven 3.3.1 in the root of the target project. Example:

<?xml version="1.0" encoding="UTF-8"?>
<extensions>
    <extension>
        <groupId>com.lahsivjar</groupId>
        <artifactId>gcp-storage-wagon</artifactId>
        <!-- Replace gcp.storage.wagon.version with the correct version -->
        <version>${gcp.storage.wagon.version}</version>
    </extension>
</extensions>

3. Setup distribution management in the target project to deploy artifacts to:

  <distributionManagement>
      <snapshotRepository>
          <id>gcp-bucket-snapshot</id>
          <!-- Replace gcp.project-id and gcp.maven.bucket with correct values -->
          <url>gs://${gcp.project-id}#${gcp.maven.bucket}/snapshot</url>
      </snapshotRepository>
      <repository>
          <id>gcp-bucket-release</id>
          <!-- Replace gcp.project-id and gcp.maven.bucket with correct values -->
          <url>gs://${gcp.project-id}#${gcp.maven.bucket}/release</url>
      </repository>
  </distributionManagement>

NOTE: Check Project Id Resolution section to find all supported ways of resolving GCP project id.

4. To use the deployed modules in another project add extension as described in step 2 to the target project and specify a repository with the wagon configured:

  <repositories>
      <repository>
          <id>gcp-bucket-snapshot</id>
          <!-- Replace gcp.project-id and gcp.maven.bucket with correct values -->
          <url>gs://${gcp.project-id}#${gcp.maven.bucket}/snapshot</url>
      </repository>
      <repository>
          <id>gcp-bucket-release</id>
          <!-- Replace gcp.project-id and gcp.maven.bucket with correct values -->
          <url>gs://${gcp.project-id}#${gcp.maven.bucket}/release</url>
      </repository>
  </repositories>

NOTE: Check Project Id Resolution section to find all supported ways of resolving GCP project id.

GCP Project Id resolution

Project id of GCP can be resolved in the following two ways(in decreasing priority):

  1. By specifying it as part of the repository url. For example: gs://${gcp.project-id}#${gcp.maven.bucket}/
  2. By setting environment variable WAGON_GCP_PROJECT_ID. In this case the url must be of the form gs://${gcp.maven.bucket}/...

NOTE: It is priority based so if project id can be resolved via a higher priority resolver then the lower priority won't be considered

Environment variable interpolation

Hash based project id resolution scheme supports environment variable interpolation by specifying the repository url as gs://${env.GCP_PROJECT_ID}#${env.BUCKET_NAME} with GCP_PROJECT_ID and BUCKET_NAME exported as environment variables.

Issues

Report any issues or bugs to https://github.com/lahsivjar/gcp-storage-wagon/issues

Changelog

3.0

  • Update module to support Java 11

2.1

  • (Bugfix #19) Handle small files in single blob request

2.0

  • Environment variable interpolation for hash separated project id resolution

1.0

  • Add support for environment variable based project id resolution

0.1

  • Add basic wagon for GCP storage
  • Support resolving project id via the repository url

License

The project is licensed under Apache-2.0 - see the LICENSE file for details.

gcp-storage-wagon's People

Contributors

joshfischer1108 avatar lahsivjar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

danhelending

gcp-storage-wagon's Issues

Idea: How to include or excludue the files that we move to GCP

Hey!! I would like to know if there is a way, I can send only the ".jars" to our Google Cloud Storage. Currently it is transferring all the files including the com , src folders and few unnecessary files.

Basically, my development team requires only the .jar files that mvn build provides.

Please let me know your thoughts on the same. Thank you!

Timeout waiting for connection from pool after uploading/putting 10 files successfully

It seems like there is a leak in the pool that maintains connections towards Google Cloud Storage. After successfully uploading/putting 10 files, the process hangs for a number of minutes, until the below stacktrace is printed:
[WARNING] Failed to upload checksum IOPG/IOPG-swagger-jaxrs/maven-metadata.xml.md5: Failed to transfer wagon-74871373d82f459ca54746808aa2c89c4789050654382797971.tmp to IOPG/IOPG-swagger-jaxrs/maven-metadata.xml.md5 org.apache.maven.wagon.TransferFailedException: Failed to transfer wagon-74871373d82f459ca54746808aa2c89c4789050654382797971.tmp to IOPG/IOPG-swagger-jaxrs/maven-metadata.xml.md5 at com.lahsivjar.GcpStorageWagon.put (GcpStorageWagon.java:153) at org.eclipse.aether.transport.wagon.WagonTransporter$PutTaskRunner.run (WagonTransporter.java:686) at org.eclipse.aether.transport.wagon.WagonTransporter.execute (WagonTransporter.java:435) at org.eclipse.aether.transport.wagon.WagonTransporter.put (WagonTransporter.java:418) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum (BasicRepositoryConnector.java:557) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums (BasicRepositoryConnector.java:532) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask (BasicRepositoryConnector.java:513) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:360) at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put (BasicRepositoryConnector.java:298) at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:320) at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:223) at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy (DefaultRepositorySystem.java:384) at org.apache.maven.artifact.deployer.DefaultArtifactDeployer.deploy (DefaultArtifactDeployer.java:139) at org.apache.maven.plugin.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:167) at org.apache.maven.plugin.deploy.DeployMojo.execute (DeployMojo.java:157) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290) at org.apache.maven.cli.MavenCli.main (MavenCli.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) Caused by: com.google.cloud.storage.StorageException: Timeout waiting for connection from pool at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate (HttpStorageRpc.java:220) at com.google.cloud.storage.spi.v1.HttpStorageRpc.open (HttpStorageRpc.java:758) at com.google.cloud.storage.BlobWriteChannel$2.call (BlobWriteChannel.java:69) at com.google.cloud.storage.BlobWriteChannel$2.call (BlobWriteChannel.java:66) at com.google.api.gax.retrying.DirectRetryingExecutor.submit (DirectRetryingExecutor.java:89) at com.google.cloud.RetryHelper.run (RetryHelper.java:74) at com.google.cloud.RetryHelper.runWithRetries (RetryHelper.java:51) at com.google.cloud.storage.BlobWriteChannel.open (BlobWriteChannel.java:66) at com.google.cloud.storage.BlobWriteChannel.<init> (BlobWriteChannel.java:37) at com.google.cloud.storage.StorageImpl.writer (StorageImpl.java:497) at com.google.cloud.storage.StorageImpl.writer (StorageImpl.java:492) at com.google.cloud.storage.StorageImpl.writer (StorageImpl.java:81) at com.lahsivjar.GcpStorageWagon.put (GcpStorageWagon.java:138) at org.eclipse.aether.transport.wagon.WagonTransporter$PutTaskRunner.run (WagonTransporter.java:686) at org.eclipse.aether.transport.wagon.WagonTransporter.execute (WagonTransporter.java:435) at org.eclipse.aether.transport.wagon.WagonTransporter.put (WagonTransporter.java:418) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum (BasicRepositoryConnector.java:557) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums (BasicRepositoryConnector.java:532) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask (BasicRepositoryConnector.java:513) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:360) at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put (BasicRepositoryConnector.java:298) at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:320) at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:223) at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy (DefaultRepositorySystem.java:384) at org.apache.maven.artifact.deployer.DefaultArtifactDeployer.deploy (DefaultArtifactDeployer.java:139) at org.apache.maven.plugin.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:167) at org.apache.maven.plugin.deploy.DeployMojo.execute (DeployMojo.java:157) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290) at org.apache.maven.cli.MavenCli.main (MavenCli.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking (ConnPoolByRoute.java:414) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry (ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection (ThreadSafeClientConnManager.java:243) at org.apache.http.impl.client.DefaultRequestDirector.execute (DefaultRequestDirector.java:422) at org.apache.http.impl.client.AbstractHttpClient.doExecute (AbstractHttpClient.java:835) at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:108) at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:56) at com.google.api.client.http.apache.ApacheHttpRequest.execute (ApacheHttpRequest.java:65) at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:981) at com.google.cloud.storage.spi.v1.HttpStorageRpc.open (HttpStorageRpc.java:748) at com.google.cloud.storage.BlobWriteChannel$2.call (BlobWriteChannel.java:69) at com.google.cloud.storage.BlobWriteChannel$2.call (BlobWriteChannel.java:66) at com.google.api.gax.retrying.DirectRetryingExecutor.submit (DirectRetryingExecutor.java:89) at com.google.cloud.RetryHelper.run (RetryHelper.java:74) at com.google.cloud.RetryHelper.runWithRetries (RetryHelper.java:51) at com.google.cloud.storage.BlobWriteChannel.open (BlobWriteChannel.java:66) at com.google.cloud.storage.BlobWriteChannel.<init> (BlobWriteChannel.java:37) at com.google.cloud.storage.StorageImpl.writer (StorageImpl.java:497) at com.google.cloud.storage.StorageImpl.writer (StorageImpl.java:492) at com.google.cloud.storage.StorageImpl.writer (StorageImpl.java:81) at com.lahsivjar.GcpStorageWagon.put (GcpStorageWagon.java:138) at org.eclipse.aether.transport.wagon.WagonTransporter$PutTaskRunner.run (WagonTransporter.java:686) at org.eclipse.aether.transport.wagon.WagonTransporter.execute (WagonTransporter.java:435) at org.eclipse.aether.transport.wagon.WagonTransporter.put (WagonTransporter.java:418) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum (BasicRepositoryConnector.java:557) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums (BasicRepositoryConnector.java:532) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask (BasicRepositoryConnector.java:513) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:360) at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put (BasicRepositoryConnector.java:298) at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:320) at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:223) at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy (DefaultRepositorySystem.java:384) at org.apache.maven.artifact.deployer.DefaultArtifactDeployer.deploy (DefaultArtifactDeployer.java:139) at org.apache.maven.plugin.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:167) at org.apache.maven.plugin.deploy.DeployMojo.execute (DeployMojo.java:157) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290) at org.apache.maven.cli.MavenCli.main (MavenCli.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Unable to download artifacts from GCS

Hi @lahsivjar ,

I am facing a rather odd issue.
I am using this storage wagon plugin to manage dependency on one of my projects. Few of the dependencies are internal and created by us and hosted in GCS.
This works perfect for the most part. Although I am seeing some weird issue where sometimes the artifacts fail to download because of 401 error .
Unauthorized [ERROR] { [ERROR] "code" : 401, [ERROR] "errors" : [ { [ERROR] "domain" : "global", [ERROR] "location" : "Authorization", [ERROR] "locationType" : "header", [ERROR] "message" : "Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.", [ERROR] "reason" : "required" [ERROR] } ], [ERROR] "message" : "Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object." [ERROR] }

This happens only on a specific setup.
I have windows VM outside of my GCP network on which remote builds happen. I am using jenkins for the remote builds.
If there a new version of dependency deployed to my GCP maven repo, the consuming project fails to download that dependency.

I am able to have this resolved if I restart my jenkins slave as Admin on that server and still be logged into that server using RDP until the dependency is downloaded. Once downloaded, it works fine for all subsequent builds till a new version is deployed.

If i restart slave as admin and close RDP, dependency does not download and gives above error.
If I RDP into the server and retry mu build it works.

I am using a service account which has admin access to my private GCS maven repo.
Somehow ability to download a new version is linked to RDP.

Any thoughts or comments on what could be causing this and how to resolve would be appreciated. Thanks in advance.

I can provide more details if needed.

Anonymous caller does not have storage.objects.get access

I am using this plugin to download artifacts from my maven repo hosted in buckets.
I am setting GOOGLE_APPLICATION_CREDENTIALS to a service account json which has admin access to this repo.
Is there some config I am missing?

Setup:
I have a library project that I have deployed using maven to google buckets using this plugin - worked.
In a second project I am using dependency of first project which is throwing below error when I am building using jenkins.
I have a jenkins slave setup on a client side server who is not in my GCP network thereby causing the need to specify service account to access maven repo.

This works fine inside my GCP network VMs because they are authenticated internally.
Thoughts?

Is there a way by which I can specify in my pom the path to service.json which is part of project 2?

I have check and GOOGLE_APPLICATION_CREDENTIALS is set correctly.
I have rechecked the permissions on that service account.

- members:
  - serviceAccount:[email protected]
  role: roles/storage.admin
{
[ERROR]   "code" : 401,
[ERROR]   "errors" : [ {
[ERROR]     "domain" : "global",
[ERROR]     "location" : "Authorization",
[ERROR]     "locationType" : "header",
[ERROR]     "message" : "Anonymous caller does not have storage.objects.get access to myrepo/snapshot/myproject/project/2.6.3/project-2.6.3.pom.",
[ERROR]     "reason" : "required"
[ERROR]   } ],
[ERROR]   "message" : "Anonymous caller does not have storage.objects.get access to mymavenreporsitory/....."
[ERROR] }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.