Done criteria

• Regular progress updates are provided about file parse
  • During backup
  • During restore
• File-parse respects the thread count configuration
  • During backup
  • During restore
• Improved speed of finding previous versions during incremental backup
• File metadata parser/setter is not attempting to use POSIX permissions on Windows

Done criteria

• Reusable progress listener class is implemented
• Progress listener is integrated into
  • Backup
  • Restore
  • Merge
  • Inspect increments
  • Inspect increment contents operations
• Change API to avoid similar breaking changes (caused by new optional parameters) in the future
  • API changes done in prod code
  • API changes done in tests
  • Documentation updated
• New functionality is covered with tests

Notes

• Use high-level steps for each long running operation
• When possible give precise progress within the steps
• Avoid large jumps in progress when possible (for example during backup, avoid file count based estimate and favor data size based progress to avoid causing uneven progress when the file sizes are not perfectly unified)

Symptoms

Windows backups are not finding any paths in backup sources (probably due to a regression issue of the cross platform restore)

Tasks

• Issue is investigated
• Root cause found and documented
• Fix implemented
• Tests updated (if necessary)

Root cause

During the implementation of #94 , the method which is used creating the glob pattern changed implicitly due to the type change from Path to BackupPath. As a result, Windows paths were no longer using Windows separators (as Backup Paths are using UNIX separators everywhere).

Changes

• Fixed the affected method
• Added an additional verification step to fail the whole backup when no files are found. This should make the already existing Windows tests to fail if the bug would come back.

Done criteria

• New command is added to delete selected backup increments
• Deletion is removing all components of the backup
  • Manifests
    • Encrypted cargo
    • Plain from .history
  • Index
  • Data files
• Tests updated
• Documentation updated

Done criteria

• Full restore
• Partial restore
• Benefits of allowing/preventing deletion
• Optimal thread-count
  • Mention network mounts

References

• https://github.com/nagyesta/file-barj/wiki/Restore-configuration-tips

Done criteria

• Backup paths are no longer using the OS File System's Path implementation
  • Backup config format is not changed
  • Backup manifest format is not changed
  • Backup manifest is using new Path type
  • Restore Targets is using the new Path type in case of the source parameter
    • Mapping produces OS File System specific paths
  • File metadata is using the new Path type
• Backup/Restore functionality is not affected (existing features remain working)
• Tests are updated
  • Updated existing tests
  • New test added restoring UNIX backup on Windows
  • New test added restoring Windows backup on UNIX
• Documentation is updated if necessary

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Done criteria

• The name of the OS is saved in the backup
• Enums are defined for UNIX, Mac, Windows (parsing the full names)
• OsUtil is using the new enums
• The inspect increments feature shows the OS as well
• The restore logs contain the name of the OS
• Tests are added/updated

Done criteria

• Implement incremental backup
  • Create
  • Restore
• Add new test cases
• Update documentation

Done criteria

• JSR-303 validation added for each part of the backup manifest. Validating:
  • Backup configuration
  • File metadata
  • Archive metadata
  • App version
  • Set of increments
• Validation is triggered with the right group setting when reading/writing manifests
• Tests updated

Done criteria

• New commands added to
  • List the available backup increments of the matching backup files
  • List the metadata of the backup contents from a selected backup increment
• Core component is updated to implement the logic of the new job commands
• Tests are added/updated
• Documentation is updated

Done criteria

• Added new optional argument to the --restore command
  • When provided the value is used for identifying the latest backup we should look for during the restore (allowing the selection of the backup increment we want to restore)
• Added tests
• Updated documentation
  • Core readme
  • Job readme

Vulnerable Library - checkstyle-9.3.jar

Path to dependency file: /build.gradle.kts

Path to vulnerable library: /home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.0.1-jre/119ea2b2bc205b138974d351777b20f02b92704b/guava-31.0.1-jre.jar,/home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.0.1-jre/119ea2b2bc205b138974d351777b20f02b92704b/guava-31.0.1-jre.jar,/home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.0.1-jre/119ea2b2bc205b138974d351777b20f02b92704b/guava-31.0.1-jre.jar,/home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.0.1-jre/119ea2b2bc205b138974d351777b20f02b92704b/guava-31.0.1-jre.jar

Found in HEAD commit: 25f32ffada2cdcedc7e1659a0b39ad96f52a92b9

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in (checkstyle version)	Remediation Possible**
CVE-2023-2976	High	7.1	guava-31.0.1-jre.jar	Transitive	10.12.1	❌

**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation

Details

CVE-2023-2976

Vulnerable Library - guava-31.0.1-jre.jar

Guava is a suite of core and expanded libraries that include utility classes, Google's collections, I/O classes, and much more.

Library home page: https://github.com/google/guava

Path to dependency file: /file-barj-stream-io/build.gradle.kts

Path to vulnerable library: /home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.0.1-jre/119ea2b2bc205b138974d351777b20f02b92704b/guava-31.0.1-jre.jar,/home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.0.1-jre/119ea2b2bc205b138974d351777b20f02b92704b/guava-31.0.1-jre.jar,/home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.0.1-jre/119ea2b2bc205b138974d351777b20f02b92704b/guava-31.0.1-jre.jar,/home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.0.1-jre/119ea2b2bc205b138974d351777b20f02b92704b/guava-31.0.1-jre.jar

Dependency Hierarchy:

• checkstyle-9.3.jar (Root Library)
  • ❌ guava-31.0.1-jre.jar (Vulnerable Library)

Found in HEAD commit: 25f32ffada2cdcedc7e1659a0b39ad96f52a92b9

Found in base branch: main

Vulnerability Details

Use of Java's default temporary directory for file creation in FileBackedOutputStream in Google Guava versions 1.0 to 31.1 on Unix systems and Android Ice Cream Sandwich allows other users and apps on the machine with access to the default Java temporary directory to be able to access the files created by the class.

Even though the security vulnerability is fixed in version 32.0.0, we recommend using version 32.0.1 as version 32.0.0 breaks some functionality under Windows.

Mend Note: Even though the security vulnerability is fixed in version 32.0.0, maintainers recommend using version 32.0.1 as version 32.0.0 breaks some functionality under Windows.

Publish Date: 2023-06-14

URL: CVE-2023-2976

CVSS 3 Score Details (7.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Local
  • Attack Complexity: Low
  • Privileges Required: Low
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: High
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-7g45-4rm6-3mm3

Release Date: 2023-06-14

Fix Resolution (com.google.guava:guava): 32.0.1-android

Direct dependency fix Resolution (com.puppycrawl.tools:checkstyle): 10.12.1

Step up your Open Source Security Game with Mend here

Describe the bug

Restoring on Windows throws an exception

To reproduce

Steps to reproduce the behavior:

1. Create a backup on Windows
2. Restore the backup to another directory on Windows

Expected behavior

The archive is restored.

Actual behavior

An exception is thrown:

Exception in thread "main" com.github.nagyesta.filebarj.io.stream.exception.ArchiveIntegrityException: The following paths are not normalized: [...]
	at com.github.nagyesta.filebarj.io.stream.BarjCargoArchiveFileInputStreamSource.validateEntityIndexes(BarjCargoArchiveFileInputStreamSource.java:378)
	at com.github.nagyesta.filebarj.io.stream.BarjCargoArchiveFileInputStreamSource.<init>(BarjCargoArchiveFileInputStreamSource.java:67)
	at com.github.nagyesta.filebarj.core.restore.pipeline.RestorePipeline.getStreamSource(RestorePipeline.java:601)
	at com.github.nagyesta.filebarj.core.restore.pipeline.RestorePipeline.lambda$restoreContent$14(RestorePipeline.java:323)
	at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
	at com.github.nagyesta.filebarj.core.restore.pipeline.RestorePipeline.restoreContent(RestorePipeline.java:314)
	at com.github.nagyesta.filebarj.core.restore.pipeline.RestorePipeline.restoreFiles(RestorePipeline.java:123)
	at com.github.nagyesta.filebarj.core.restore.pipeline.RestoreController.execute(RestoreController.java:89)
	at com.github.nagyesta.filebarj.job.Controller.doRestore(Controller.java:94)
	at com.github.nagyesta.filebarj.job.Controller.run(Controller.java:56)
	at com.github.nagyesta.filebarj.job.Main.main(Main.java:19)

The command you used

java -jar file-barj-job-0.3.0.jar --restore --backup-source D:/test-backups/barj-d-file-backup --prefix d-file-backup-gzip-unencrypted --target-mapping D:/=D:/test-backups/barj/restore-d --threads 3

Environment

• OS: Windows 10
• Version: 0.3.0
• Java version 17

Additional context

The faulty code can be found here
https://github.com/nagyesta/file-barj/blob/64013dff559d5d91594d13b53b995aef0d9c1a8a/file-barj-stream-io/src/main/java/com/github/nagyesta/filebarj/io/stream/BarjCargoArchiveFileInputStreamSource.java#L368C1-L380C6

Also, when there are a lot of files in the archive (and usually there are), the error message scrolls out important information on the console. Logging config should be adjusted.

Done criteria

• New flag added to --backup command
• When the flag is true, a full backup is created (regardless of the automatic evaluation)
• Tests updated
• Documentation updated

Done criteria

• Archive index format can store the version of the format which was used (the version of the library which created it)
• Typo in chunk metadata properties is fixed in a backward compatible manner
  • Newly created format uses the correct name
  • Old format is still supported for entries created with older specifications
• Tests updated
• Documentation updated

Notes

• The typo:

  file-barj/file-barj-stream-io/src/main/java/com/github/nagyesta/filebarj/io/stream/BarjCargoUtil.java

  Lines 27 to 35 in 7fdc1fb

  	public static final String LAST_CHUNK_INDEX_PROPERTY = "last.cnunk.index";
  	/**
  	* The name of the property storing the size of the last chunk.
  	*/
  	public static final String LAST_CHUNK_SIZE_PROPERTY = "last.cnunk.size";
  	/**
  	* The name of the property storing the maximum chunk size.
  	*/
  	public static final String MAX_CHUNK_SIZE_PROPERTY = "max.cnunk.size";

• The documentation: https://github.com/nagyesta/file-barj/wiki/About-the-BaRJ-Cargo-format#barjindexcargo

Done criteria

• Improve performance
• Save relative link values during backup instead of forcing absolute
• Address issues around relative symbolic links
• Create missing parent directories (in case of file backup sources)

Done criteria

• Adds new option to --restore command
  • When added, deletes the left-over files which would have been in the backup scope but were not in the backup increment which was restored
• Adds new tests
• Updates documentation
  • Core readme
  • Job readme

Done criteria

• Additional command option added to --restore command
  • When provided, only selected files/folders are restored from the backup
• New tests are added
• Documentation is updated

Done criteria

• File BaRJ is implemented, supporting the following features:
  • Full backup
  • Keeping each file in case of duplicates
  • No support for hard links (backup contains each of them as files)
  • Backup encryption using RSA key pair
  • Chunk size limit can be defined
  • Compressed archive
  • File properties are saved (POSIX permissions, owner, group, hidden status)
  • Content hashing is supported for integrity verification
  • Full restore
    • to original location or alternative root directory
    • from original backup location or any alternative directory containing the backup archive
    • avoiding unnecessary overwrites
    • only without deletion of extra files already present in the restore target
  • Verify archive integrity before restore
  • Verify restored content using original hashes
  • CLI job runner
• File format is defined in new module
  • Coverage is uploaded to Codecov
  • Flag is set in Codecov config
• Documentation created
  • Readme for Stream IO module
  • Readme for Core module
  • Readme for Job module
  • Wiki page about the file format
  • Wiki page about backup configuration
  • Wiki page about how the information is stored in a backup archive
• Release pipeline set up
  • Release to GitHub
  • Release to Maven Central
  • Regular release is set to the 25th of each month

Out of scope

• Backup configuration generator
• Incremental backup
• Merge archives

Logs from Windows CI agents

> Task :file-barj-job:test

ControllerIntegrationTest > testEndToEndFlowShouldWorkWhenCalledWithValidParameters() FAILED
    java.lang.RuntimeException at ControllerIntegrationTest.java:182
        Caused by: java.nio.file.FileSystemException at ControllerIntegrationTest.java:182

42 tests completed, 1 failed
> Task :file-barj-job:test FAILED

Task

• Deletion related code reviewed
• Tests fixed

Done criteria

• Duplicate handling strategies are no longer ignored
  • KEEP_ONE_PER_INCREMENT is deleted
  • KEEP_ONE_PER_BACKUP is implemented
• Both full and Incremental backups respect the duplicate handling configuration option
• New test cases added
• Documentation updated

Background

Restoring a Unix Backup on Windows is not working well due to the following factors:

1. If the path of two files differs only in case (as Unix is case sensitive) there is a collision on Windows (where the file names are case in-sensitive).
2. The POSIX permissions will not be restored properly. Due to this, a long list of errors will list all files at the end as not fully restored
3. The restore path transformation logic assumes that there is a drive letter on the source path side (although there was none in case of the original Unix backup where each path starts with /) (Solved by #127)

Done criteria

• Case sensitivity issues are reported (errors avoided)
• POSIX permission errors are listed in a user-friendly format
• Allow setting new command line option to ignore permission and owner/group differences during checks
• Tests are updated
• Documentation is updated

Done criteria

• Increments of the same backup can be merged together
  • If multiple increments are in the same archive, then it cannot be split later (but it can be merged with other increments)
  • The merge must be using a continuous range of increment versions, such as 0 to 4 (including 0,1,2,3,4)
  • The merge process verifies the hashes of the merged archived streams
• There is a way to automatically rename/delete obsolete backup files and their manifests (those which were the originals for the merge)
  • In the folder where the backup files are
  • In the .history folder if there is one
• New command options added to allow merges through CLI
• Tests are added
• Documentation is updated

Symptoms

23:20:27.435 [main] ERROR c.github.nagyesta.filebarj.job.Main - Execution failed:
java.lang.NullPointerException: null
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Caused by: java.lang.NullPointerException: null
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Caused by: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because the return value of "com.github.nagyesta.filebarj.core.model.FileMetadata.getFileSystemKey()" is null
        at com.github.nagyesta.filebarj.core.restore.pipeline.RestorePipeline.lambda$copyRestoredFileToRemainingLocations$19(RestorePipeline.java:295)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at com.github.nagyesta.filebarj.core.restore.pipeline.RestorePipeline.copyRestoredFileToRemainingLocations(RestorePipeline.java:291)