A simple Maven plugin to make your build byte-for-byte reproducible
Home Page: http://zlika.github.io/reproducible-build-maven-plugin/
License: Apache License 2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Configure relative paths (support multiple), eliminate the property type file timestamp in any zip type package
I get the following stacktrace when I try to create an maven report:
[INFO] Could not build project for: reproducible-build-maven-plugin:Error resolving project artifact: Could not find artifact io.github.zlika:reproducible-build-maven-plugin:pom:0. in central (https://repo.maven.apache.org/maven2) for project io.github.zlika:reproducible-build-maven-plugin:pom:0.
org.apache.maven.project.ProjectBuildingException: Error resolving project artifact: Could not find artifact io.github.zlika:reproducible-build-maven-plugin:pom:0. in central (https://repo.maven.apache.org/maven2) for project io.github.zlika:reproducible-build-maven-plugin:pom:0.
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:355)
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:287)
at org.apache.maven.report.projectinfo.PluginManagementReport$PluginManagementRenderer.renderSectionPluginManagement(PluginManagementReport.java:206)
at org.apache.maven.report.projectinfo.PluginManagementReport$PluginManagementRenderer.renderBody(PluginManagementReport.java:171)
at org.apache.maven.reporting.AbstractMavenReportRenderer.render(AbstractMavenReportRenderer.java:80)
at org.apache.maven.report.projectinfo.PluginManagementReport.executeReport(PluginManagementReport.java:65)
at org.apache.maven.reporting.AbstractMavenReport.generate(AbstractMavenReport.java:251)
at org.apache.maven.plugins.site.render.ReportDocumentRenderer.renderDocument(ReportDocumentRenderer.java:230)
at org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.render(DefaultSiteRenderer.java:349)
at org.apache.maven.plugins.site.render.SiteMojo.renderLocale(SiteMojo.java:198)
at org.apache.maven.plugins.site.render.SiteMojo.execute(SiteMojo.java:147)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact io.github.zlika:reproducible-build-maven-plugin:pom:0. in central (https://repo.maven.apache.org/maven2)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:223)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:294)
at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:344)
... 32 more
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact io.github.zlika:reproducible-build-maven-plugin:pom:0. in central (https://repo.maven.apache.org/maven2)
at io.takari.aether.connector.AetherRepositoryConnector$2.wrap(AetherRepositoryConnector.java:893)
at io.takari.aether.connector.AetherRepositoryConnector$2.wrap(AetherRepositoryConnector.java:1)
at io.takari.aether.connector.AetherRepositoryConnector$GetTask.flush(AetherRepositoryConnector.java:673)
at io.takari.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:310)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:520)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:421)
... 36 more
[INFO] Downloading: https://repo.maven.apache.org/maven2/org/codehaus/mojo/build-helper-maven-plugin/3.0.0/build-helper-maven-plugin-3.0.0.pom
[INFO] Downloaded: https://repo.maven.apache.org/maven2/org/codehaus/mojo/build-helper-maven-plugin/3.0.0/build-helper-maven-plugin-3.0.0.pom (6 KB at 62.7 KB/sec)
Seems like there is a problem getting a required pom from central. Im not sure what that error is.
Since Maven now defines the project.build.outputTimestamp property as the default build time, would it be possible to use this value if present for the zipDateTime parameter instead of 2000-01-01 00:00:00?
E.g. A simple include pattern to try and find jars starting with tika seems to fail to identify the files.
In fact the only pattern I've got working so far seems to be .*\.jar
<plugin>
<groupId>io.github.zlika</groupId>
<artifactId>reproducible-build-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>strip-jar</goal>
</goals>
<configuration>
<includes>
<include>tika.*\.jar</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
Looking at debug console it looks like the include patterns aren't applied to everything that goes into the final jar, but rather to content of the actual maven project being built..... This is contrary to the zipDateTime which gets applied to everything in the final packaged jar.
My hope was to set the modified time for a bunch of jars (identified by include pattern) within the final jar to a specific timestamp.
Is there a way of achieving this?
Hello,
bit of a long shot, but would it be possible for your plugin to do the same strip-jar process it does for jar artefacts for andrioid apk & apklib build artefacts?
We build these using the com.simpligility.maven.plugins.android-maven-plugin, which generates a META-INF folder and adds in a copy of an AndroidMainifest.xml file during the build which are date-time stamped with the machine time of the build, which means the md5sum of the resulting artefact (apklib/apk) isn't reproducible.
Is this something your plugin could help with?
Kind regards
Pete.
Hello.
It appears that the JDK 9 javac seems to get very upset when it encounters the timestamps produced by the plugin. Here's part of the output for a build of one of my projects, it's compiling against a jar file that has been stripped:
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ com.io7m.jnull.documentation ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/someone/git/com.github/io7m/jnull/com.io7m.jnull.documentation/target/classes
An exception has occurred in the compiler (9). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.time.DateTimeException: Invalid value for MonthOfYear (valid values 1 - 12): 0
at java.base/java.time.temporal.ValueRange.checkValidValue(ValueRange.java:311)
at java.base/java.time.temporal.ChronoField.checkValidValue(ChronoField.java:714)
at java.base/java.time.LocalDate.of(LocalDate.java:269)
at java.base/java.time.LocalDateTime.of(LocalDateTime.java:336)
at jdk.zipfs/jdk.nio.zipfs.ZipUtils.dosToJavaTime(ZipUtils.java:109)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.cen(ZipFileSystem.java:1950)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:1937)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getEntry(ZipFileSystem.java:1324)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.newInputStream(ZipFileSystem.java:550)
at jdk.zipfs/jdk.nio.zipfs.JarFileSystem.isMultiReleaseJar(JarFileSystem.java:91)
at jdk.zipfs/jdk.nio.zipfs.JarFileSystem.<init>(JarFileSystem.java:67)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:134)
at jdk.compiler/com.sun.tools.javac.file.JavacFileManager$ArchiveContainer.<init>(JavacFileManager.java:517)
at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.getContainer(JavacFileManager.java:319)
at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:715)
at jdk.compiler/com.sun.tools.javac.code.ClassFinder.list(ClassFinder.java:722)
at jdk.compiler/com.sun.tools.javac.code.ClassFinder.scanUserPaths(ClassFinder.java:655)
at jdk.compiler/com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:529)
at jdk.compiler/com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:293)
at jdk.compiler/com.sun.tools.javac.code.Symbol.complete(Symbol.java:633)
at jdk.compiler/com.sun.tools.javac.code.Symbol$PackageSymbol.members(Symbol.java:1120)
at jdk.compiler/com.sun.tools.javac.code.Symtab.listPackageModules(Symtab.java:810)
at jdk.compiler/com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:344)
at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:529)
at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:285)
at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:300)
at jdk.compiler/com.sun.tools.javac.comp.Enter.complete(Enter.java:570)
at jdk.compiler/com.sun.tools.javac.comp.Enter.main(Enter.java:554)
at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:1052)
at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:923)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:100)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:142)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:96)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:90)
at org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:126)
at org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174)
at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1075)
at org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:168)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
I'm not sure what the most pleasant workaround would be for this. Perhaps allow the specification of a custom timestamp to be used in all stripped zip/jar files?
Is there a way to increase the limit?
My file named is not really long... but because it's nested deep into the folders hence it's complaining to be lengthy file name.
Execution default of goal io.github.zlika:reproducible-build-maven-plugin:0.5.2:strip-jar failed: file name 'nested/folder/path/fileName.extension' is too long ( > 100 bytes)
Thanks :)
Hello, first I'd like to thank you again for this great plugin - I'm long time user and it helps us a lot to get reproducible artifacts.
Now getting back to issue. Right now docs mention that
Recent versions of the main Maven plugins have been modified to allow reproducible builds without the use of this plugin
I feel that by reading this, potential user might get puzzled - "What should I choose then?". Would be great to list some use-cases when stock maven functionality will be enough (I suppose it is something like building plain jar file in regular env) and scenarios when it will not be enough and using this plugin will be way more easier (or the only way to do things).
I assume this is related with some processing of custom files like properties (e.g. spring.factories) or fixing permissions of archive entries that was added in #24, but still.
ManifestAttributes tag How to configure ？
Thank you
I just upgraded our project from plugin version 0.3 to 0.5 to benefit from the .tar support.
After upgrading, some .war files fail to start, the exception is:
SEVERE: ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/app-ws]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:153)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:939)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1812)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@6ed89845]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:153)
at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4928)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5058)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
... 10 more
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.JarResourceSet@71310103]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:136)
at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:699)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
... 13 more
Caused by: java.lang.IllegalArgumentException: java.io.IOException: invalid manifest format
at org.apache.catalina.webresources.JarResourceSet.initInternal(JarResourceSet.java:96)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
... 16 more
Caused by: java.io.IOException: invalid manifest format
at java.util.jar.Manifest.read(Manifest.java:225)
at java.util.jar.Manifest.<init>(Manifest.java:69)
at java.util.jar.JarFile.getManifestFromReference(JarFile.java:194)
at java.util.jar.JarFile.getManifest(JarFile.java:180)
at org.apache.catalina.webresources.JarResourceSet.initInternal(JarResourceSet.java:94)
... 17 more
unfortunately the logs are not giving much details :-(
I noticed this commit, which could be probable cause for the above error, but I can't see anything wrong with it.
Also I tried inspecting the MANIFEST.MF file from the war META-INF dir, but that specific file does not appear to be the problem, so it must be somewhere else.
I want to use reproducible-build-maven-plugin with a jar outside of a maven project.
When i run strip-jar goal as a standalone command i got the following errors:
$ mvn 'io.github.zlika:reproducible-build-maven-plugin:0.2:strip-jar' -DoutputDirectory=/tmp
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
...
[ERROR] Failed to execute goal io.github.zlika:reproducible-build-maven-plugin:0.2:strip-jar (default-cli): Goal requires a project to execute but there is no POM in this directory (/tmp). Please verify you invoked Maven from the correct directory. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MissingProjectException
Can you allow strip-jargoal to run without requiring a maven project?
I had originally followed the Maven guide for reproducible builds and set the project's project.build.outputTimestamp to a fixed value. I was able to create reproducible builds without having to use this plugin.
However, once I added module-info, I began receiving different hashes. Using diffoscope I determined the timestamp across 2 builds was constant for all but the module-info.class:
│ -rw---- 2.0 fat 862 bl defN 20-Oct-17 00:40 space/arim/libertybans/api/select/SelectionOrder.class
│ -rw---- 2.0 fat 1113 bl defN 20-Oct-17 00:40 space/arim/libertybans/api/select/SelectionOrderBuilder.class
│ -rw---- 2.0 fat 2285 bl defN 20-Oct-17 00:40 META-INF/maven/space.arim.libertybans/bans-api/pom.xml
│ -rw---- 2.0 fat 74 bl defN 20-Oct-17 00:40 META-INF/maven/space.arim.libertybans/bans-api/pom.properties
│ --rw---- 2.0 fat 557 bl defN 20-Oct-25 12:39 module-info.class
│ +-rw---- 2.0 fat 557 bl defN 20-Oct-25 12:41 module-info.class
Note the + and - which are diffoscope's way of indicating the difference between the .jar files. Here the project.build.outputTimestamp is on 17 October. As shown, though, module-info has a rebellious timestamp.
Relation to reproducible-build-maven-plugin
This seemed a bug in the maven-compiler-plugin, so I decided to see if this plugin had a solution for this. However, adding the plugin yielded the same issue – the current timestamp is applied on module-info.class even though all the other files in the jar have the timestamp defined in project.build.outputTimestamp.
Additional Information
I considered JDK-8240734 as a possible cause of the issue. However, this problem occurs both on JDK 11 (build 11.0.8+10) and JDK 15 (build 15+36) and with and without reproducible-build-maven-plugin on each JDK version. module-info is consistently the culprit.
I'm planning to report this same bug to the maven-compiler-plugin.
The plugin should provide a goal to capture the current build environment characteristics and write them down to a file (somehow similar in scope to Debian's .buildinfo file).
The characteristics that should be captured are (at least):
This file could then, for instance, be version controlled so that the environment used to build a specific version can be reproduced by anyone willing to verify the project's binary artifacts.
Right know when building a project with subprojects, the strip-jar goal is skipped.
Your plugin does not produce identical artifacts on my multi-module project, where one sub-module has the packaging "war" defined. This sub-module also has code, it first assembles a jar, but not in the target root folder, but in target\eas-admin-4.0-SNAPSHOT\WEB-INF\lib\eas-admin-4.0-SNAPSHOT.jar.
Running "mvn clean install -DskipTests" shows on console:
--- maven-war-plugin:3.3.0:war (default-war) @ eas-admin ---
[INFO] Packaging webapp
[INFO] Assembling webapp [eas-admin] in [C:\Development\eascs2.0\trunk\eas-admin\target\eas-admin-4.0-SNAPSHOT]
[INFO] Processing war project
[INFO] Copying webapp webResources [C:\Development\eascs2.0\trunk\eas-admin\src/main/resources/int] to [C:\Development\eascs2.0\trunk\eas-admin\target\eas-admin-4.0-SNAPSHOT]
...
[INFO] Building jar: C:\Development\eascs2.0\trunk\eas-admin\target\eas-admin-4.0-SNAPSHOT\WEB-INF\lib\eas-admin-4.0-SNAPSHOT.jar
[INFO] Building war: C:\Development\eascs2.0\trunk\eas-admin\target\EasAdminService.war
[INFO]
[INFO] --- reproducible-build-maven-plugin:0.12:strip-jar (default) @ eas-admin ---
[INFO] Stripping C:\Development\eascs2.0\trunk\eas-admin\target\EasAdminService.war
As you see, only the generated war file is stripped by your plugin, but not embedded jar file. Comparing two builds indeed shows that the jar files does not produce the same hash, and therefore neither the stripped war.
Is there a workaround to strip also the jar file of this sub-module?
When running the plugin on a build in parallel with maven mvn --threads 1C I got this warning
[WARNING] *****************************************************************
[WARNING] * Your build is requesting parallel execution, but project *
[WARNING] * contains the following plugin(s) that have goals not marked *
[WARNING] * as @threadSafe to support parallel building. *
[WARNING] * While this /may/ work fine, please look for plugin updates *
[WARNING] * and/or request plugins be made thread-safe. *
[WARNING] * If reporting an issue, report it against the plugin in *
[WARNING] * question, not against maven-core *
[WARNING] *****************************************************************
[WARNING] The following plugins are not marked @threadSafe in Apache Beam :: Parent:
io.github.zlika:reproducible-build-maven-plugin:0.3
``
Maybe it is a good idea to ensure that the plugin is threadsafe and add the annotation if it is the case.
Hello
Please support jenkins plugin .hpi package difference elimination
1.MANIFEST.MF
2.pom.properties
3. *.stapler
#Fri Sep 28 09:38:28 GMT+08:00 2018
thank you very much
Would you be open to the idea of splitting the reproducible-build-maven-plugin into a 'maven plugin' part and a 'library' part?
I hacked up an sbt-reproducible-builds plugin just by wrapping the reproducible-build-maven-plugin strippers, and while I'm sure this is not completely sufficient, it's already looking quite promising. Right now I just depend on the reproducible-build-maven-plugin, it would of course be a bit neater to depend on a "reproducible-build" library that contains the core of the plugin and can be shared among various build systems.
I'd be happy to help!
The javax.inject.Named annotation adds a newline into a generated text file that ends up in the META-INF of a jar. The newline is 0x0D 0x0A on Windows and 0x0A on Mac.
Snippets of code from: https://github.com/apache/royale-compiler/blob/develop/royale-maven-plugin/src/main/java/org/apache/royale/maven/trust/DefaultTrustHandler.java
import javax.inject.Named;
@Named
public class DefaultTrustHandler {
}
Results in:
META-INF/sisu/javax.inject.Named
Which has different newlines on Windows vs Mac
This seems to be very similar to this issue: #41
When using the Maven shade plugin and Spring Boot version 2.4.7, I'm getting unreproducible builds. After looking in diffoscope, it seems to be due to a timestamp that's being included in the META-INF/spring.factories file.
#Merged by PropertiesMergingResourceTransformer
#Thu Dec 16 11:23:36 EST 2021
...
I saw in the other PR that you added a couple checks for build-info.properties and git.properties to strip comments. If you think the same approach is reasonable here, I can try putting together a PR to address this file as well.
Have you considered lowering the JRE requirement to 6 or 7 (maybe using retrolambda, just the runtime, not the JDK)?
I am trying to use this plugin (btw, excellent job and thanks a lot for creating it) to make the Apache Beam builds reproducible and I just found an issue because the plugin does not detect correctly one case.
Caused by: java.io.FileNotFoundException: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/apex/target/testCreateJar (Is a directory)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:213)
at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:192)
at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:153)
at io.github.zlika.reproducible.ZipStripper.strip(ZipStripper.java:62)
at io.github.zlika.reproducible.StripJarMojo.strip(StripJarMojo.java:81)
It seems it does not check if the matched path is a directory or not and it tries to unzip it which in the case of a directory will fail.
As title, i spent a lot time to test v0.9 ,finally it turns out this version cannot make maven package reproducible.but any of other can .
It looks like strip-jar goal breaks entries in TARs:
This behavior makes strip-jar goal unwanted when I build TARs which I use with Dockerfile ADD directive to place files with correct permissions (my solution of https://medium.com/@lmakarov/the-backlash-of-chmod-chown-mv-in-your-dockerfile-f12fe08c0b55 issue) - refer to mabrarov/dockerfile-test.
Refer to feature/reproducible-buids branch of mabrarov/dockerfile-test for reproducing this issue. When project is built using this branch, I get:
$ docker run --rm -it -p 8080:8080 abrarov/dockerfile-test
docker: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"/app/bin/run.sh\\\": permission denied\"\n".
because execute permission of start script - /app/bin/run.sh - is removed by reproducible-build-maven-plugin.
It would be helpful if strip-jar goal could:
when processing TARs .
Advantage of reproducible-build-maven-plugin is that it makes usage of TARs more Docker cache friendly and results in a better utilization of Docker layers.
I'd really appreciate if there would also be the possibility to explicitly define a list of files (ideally including wildcard support) to be processed instead of processing all files of the outputDirectory.
Thanks in advance!
In order for this plugin to work out of the box, I believe it should also remove:
Generated on: timestamp from headers in all the generated classes. Example://
// This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.11
// See <a href="http://java.sun.com/xml/jaxb">http://java.sun.com/xml/jaxb</a>
// Any modifications to this file will be lost upon recompilation of the source schema.
// Generated on: 2018.10.25 at 02:29:45 PM CEST
//
jaxb/META-INF/sun-jaxb.episode:This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.11
See <a href="http://java.sun.com/xml/jaxb">http://java.sun.com/xml/jaxb</a>
Any modifications to this file will be lost upon recompilation of the source schema.
Generated on: 2018.10.25 at 01:50:10 PM CEST
Both above features are turned on by default in jaxb2-maven-plugin, but can be turned off by setting:
<generateEpisode>false</generateEpisode> - though here the whole file (containing some mapping metadata) gets lost (jaxb episode documentation)<noGeneratedHeaderComments>true</noGeneratedHeaderComments>I was using below configuration when experiencing this issue:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>jaxb2-maven-plugin</artifactId>
<version>2.3.1</version>
<executions>
<execution>
<id>xjc</id>
<goals>
<goal>xjc</goal>
</goals>
<configuration>
<sources>
<source>src/src/resources/schemas/xsd</source>
</sources>
<outputDirectory>target/generated-sources/jaxb</outputDirectory>
</configuration>
</execution>
</executions>
<configuration>
<enableIntrospection>true</enableIntrospection>
</configuration>
</plugin>
<plugin>
<groupId>io.github.zlika</groupId>
<artifactId>reproducible-build-maven-plugin</artifactId>
<version>0.7</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>strip-jar</goal>
<goal>strip-jaxb</goal>
</goals>
</execution>
</executions>
</plugin>
The maven-bundle-plugin (and the underlying bnd tool) typically generates additional manifest entries which are not reproducible.
Bnd-LastModifiedToolExample for those entries:
Bnd-LastModified: 1520115124085
Tool: Bnd-3.3.0.201609221906
It looks like the former is already stripped (in v 0.5.0), but the latter is not.
By adding some optional configuration point to configure additional manifest headers to be stripped, the plugin could be easily adapted to other unknown setups.
Mr. Lorblanchès, I've tried the latest version and found that the jar resource is not reproducible then. Finally it proves that git.properties file is time different for each package. Something like this:
diff -aNru user1/BOOT-INF/classes/git.properties user2/BOOT-INF/classes/git.properties
--- user1/BOOT-INF/classes/git.properties 2019-05-09 12:16:14.000000000 +0800
+++ user2/BOOT-INF/classes/git.properties 2019-05-09 12:17:28.000000000 +0800
@@ -1,8 +1,8 @@
#Generated by Git-Commit-Id-Plugin
-#Thu May 09 12:16:14 CST 2019
+#Thu May 09 12:17:29 CST 2019
git.branch=chore-apm
git.build.host=rick-book
-git.build.time=20190509041614
+git.build.time=20190509041729
git.build.version=1.1.41
and my pom like this:
<build>
<plugins>
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<executions>
<execution>
<id>retrieve-git-info</id>
<phase>prepare-package</phase>
<goals>
<goal>revision</goal>
</goals>
</execution>
</executions>
<configuration>
<injectAllReactorProjects>true</injectAllReactorProjects>
<runOnlyOnce>true</runOnlyOnce>
<skipPoms>false</skipPoms>
<dateFormat>yyyyMMddHHmmss</dateFormat>
<dateFormatTimeZone>UTC</dateFormatTimeZone>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<!--<groupId>org.apache.maven.plugins</groupId>-->
<!--<artifactId>maven-jar-plugin</artifactId>-->
<configuration>
<archive>
<manifestEntries>
<!--<Last-Commit-Id>05c1678b70e36593fa262d3c8d2c89783abd473b6f66b691e2b47a50cfbe74b4</Last-Commit-Id>-->
<!--<Last-Commit-Time>20190509005059</Last-Commit-Time>-->
<Last-Commit-Id>${git.commit.id}</Last-Commit-Id>
<Last-Commit-Time>${git.commit.time}</Last-Commit-Time>
<Reproducible-Build>true</Reproducible-Build>
</manifestEntries>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>io.github.zlika</groupId>
<artifactId>reproducible-build-maven-plugin</artifactId>
<version>0.9</version>
<executions>
<execution>
<id>strip-jaxb</id>
<goals>
<goal>strip-jaxb</goal>
</goals>
</execution>
<execution>
<id>strip-jar</id>
<goals>
<goal>strip-jar</goal>
</goals>
<configuration>
<zipDateTime>${git.commit.time}</zipDateTime>
<!-- Set custom date/time format pattern, "yyyyMMddHHmmss" by default -->
<!-- <zipDateTimeFormatPattern>yyyyMMddHHmmss</zipDateTimeFormatPattern> -->
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>Thanks.
First: Thanks for writing this Maven plugin, it is a great help to make our Java builds more reproducible. 👍
However, there is one last obstacle that prevents the build from being reproducible in our case: the manifest files contain an entry like:
OpenIDE-Module-Build-Version: 201805291337
This is basically a timestamp of the build time (29th May 2018, 13:37). Since this entry changes with every build, I would appreciate it if your plugin could remove such an entry from the manifest.mf. Maybe this could also be achieved by providing a way to configure manifest entries that will be removed by the plugin, as suggested in #17 (Additional strippable manifest headers by configuration).
In addition to zip and tar it would be nice to also support cpio archives used by rpm packages (this format is handled by commons-compress).
All properties files (.properties) written by Java's java.util.Properties has a timestamp in it
Next, a comment line is always written, consisting of an ASCII # character, the current date and time (as if produced by the toString method of Date for the current time), and a line separator as generated by the Writer.
- https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html
Can those lines be stripped?
For example, a basic Spring Boot generated jar has these two files:
META-INF/build-info.properties:
#Properties
#Tue May 21 09:44:39 EDT 2019
build.artifact=todo
build.group=com.isobar
build.name=Todo
build.version=0.0.1-SNAPSHOT
BOOT-INFO/classes/git.properties
#Generated by Git-Commit-Id-Plugin
#Tue May 21 09:44:39 EDT 2019
git.branch=master
git.build.host=craigatwork
[email protected]
git.build.user.name=Craig Andrews
git.build.version=0.0.1-SNAPSHOT
git.closest.tag.commit.count=
git.closest.tag.name=
git.commit.id=7238c52581615f796a2dbb216cbca10ef142b9bc
git.commit.id.abbrev=7238c52
git.commit.id.describe=7238c52-dirty
git.commit.id.describe-short=7238c52-dirty
git.commit.message.full=Initial commit
git.commit.message.short=Initial commit
git.commit.time=2019-05-17T17\:05\:43-0400
[email protected]
git.commit.user.name=Craig Andrews
git.dirty=true
git.remote.origin.url=<redacted>
git.tags=
git.total.commit.count=1
The date headers cause the jar to not be reproducible.
Sorting the keys and ensuring the proper line endings would also be a nice to ensure reproducibility.
Thanks!
Hi,
When running the strip-jar command after maven has build the jar file for the spring boot project the jar is corrupt when the following configuration is added to the spring boot plugin.
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<executable>true</executable>
</configuration>
</plugin>
<plugin>
<groupId>io.github.zlika</groupId>
<artifactId>reproducible-build-maven-plugin</artifactId>
<version>0.10</version>
<executions>
<execution>
<goals>
<goal>strip-jar</goal>
</goals>
</execution>
</executions>
</plugin>this only occurs when the executable field is set
Hi,
Would be cool to add the support of stripping dates also from tar.gz tar.bz archives that are often generated in maven build with assembly plugin.
This can be easily achieved building a Stripper using TarArchiveEntry
Bye
Hello!
When compiling code on newer JDKs (seems to be 10+ here, although builds seem to differ), if you create a module descriptor such as:
module com.example {
requires java.compiler;
}
The version of the java.compiler module will be inserted into the compiled module-info.class. This makes the code inherently non-reproducible. Right now I don't think anything depends on the version information, so it could be possible to strip it out... It seems like quite a bad thing to do, though. Not sure what consequences losing that version information could have.
The produced JAR is not always specification conform. According to the spec, the manifest entry must be the first entry of the JAR file.
Currently, if there are entries in the JAR with a name which is alphabetically before the "META-INF/MANIFEST.MF" entry e.g. "Changelog.txt", they will be inserted before the JAR Manifest.
If the MANIFEST.MF is not the first entry, reading the MANIFEST via the java.util.jar.JarInputStream#getManifest() will return null.
the jar files created by jar plugin have system dependend bytes.
the difference is found in "External file attributes" (4 bytes, offset 38) in "Central directory" entries.
this effect is not caused by reproducible-build-maven-plugin, the question is wheather is can be fixed.
At least an according hint would be helpful.
use attached minimal project from zip file reproducible-test.zip
mvn clean install
mvn exec:java
(exec:java implements "sha256sum target/reproducible-test-1.0-SNAPSHOT.jar")
umask 0022
chmod 644 pom.xml
mvn clean install
mvn exec:java
jar hash: C4FD5F3404C2B65A56F0F99B6775AF0E54B67409A94BADCFEAF2119898F9833E
umask 0002
chmod 664 pom.xml
mvn clean install
mvn exec:java
jar hash: 25CE429882647ECF67BFFBC2317001B9BB190429D093F36F1C1049DE5AF8726A
mvn clean install
mvn exec:java
jar hash: C4FD5F3404C2B65A56F0F99B6775AF0E54B67409A94BADCFEAF2119898F9833E
update: changed test commands to be "reproducible" (pom.xml will get packaged so permissions also change jar result)
Hello all,
Can the current version of the plugin be considered stable? If not, is there a roadmap up to v1.0?
While trying to fix this concerning idem-potency builds for Wildfly Swarm I came across this neat plugin and it worked instantly! So I already proposed it to the Wildfly-Swarm team, but I guess (as an outsider) there needs to be a certain level of trust before I can integrate this plugin with the Wildfly Swarm Maven Plugin.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.