steffenfritz / bagit Goto Github PK
View Code? Open in Web Editor NEWgobagit is a library and command line tool to work with the BagIt format
License: MIT License
gobagit is a library and command line tool to work with the BagIt format
License: MIT License
I recently discovered that the BagIt 1.0 specification requires that CR
, LF
, and %
in file paths within manifest files are percent-encoded, and that there isn't a single BagIt implementation that does this correctly. Implementations either only encode CR
and LF
but not %
or they encode nothing.
This implementation does not encode paths in the manifest, which means that it would fail to validate BagIt 1.0 bags that include file paths containing CR
, LF
, or %
. Likewise, it would create bags that would fail BagIt 1.0 validation in the case that there are paths that naturally contain percent-encoded characters.
For example, let's say a bag contains the file data/file%0A1.txt
. This file should be written to the manifest per the spec as data/file%250A1.txt
. However, this implementation writes it as data/file%0A1.txt
. This means, that when this implementation validates a properly constructed 1.0 bag it will look for the file data/file%250A1.txt
which does not exist. Similarly, if another implementation that follows the spec attempts to validate a bag produced by this implementation, it would look for data/file\n1.txt
, which does not exist.
It would seem desirable to me to move the ecosystem in the direction of properly implementing the 1.0 specification, while at the same acknowledging that there are a large number of 1.0 bags in existence that may then become invalid.
As such, it may be prudent to, when validating bags, fall back on a series of tests. You may want to first attempt to validate per the spec, and then, if a file cannot be found, attempt to locate it by either only decoding the CR
and LF
or leaving the path unchanged, ideally validating all of the files using the same method.
I have not examined fetch.txt
implementations, but the same encoding requirements exist for paths in that file as well. This is potentially a thornier problem to address in a backward compatible way as it is unclear if the path data/file%250A1.txt
is supposed to create data/file%250A1.txt
(incorrect) or data/file%0A1.txt
(correct).
Finally, I created a related ticket against the spec discussing this encoding problem, in particular how it breaks checksum utility compatibility.
What:
Add a command line flag that inserts a fetch.txt file into the bag.
Why:
BagIt allows the usage of a fetch.txt file that references remote resources.
Things to consider:
The referenced resources MUST be listed in all payload manifest files. So it is necessary to download all resources temporary and calculate all hashsums on creation time.
Describe the bug
Validation fails if there are more whitespaces between hashsum and file path. BagIt 1.0 allows more than one whitespaces.
To Reproduce
"The purpose of the BagIt Profiles Specification is to allow creators and consumers of Bags to agree on optional components of the Bags they are exchanging. Details of the profile are instantiated in a JSON file that both the producing and consuming applications..."
The code base is three years old and needs some re-work.
BagIt 1.0 supports multiple manifest files for same payload or tagfiles with different hashing algorithms.
Add suport to create and validate
BagIt version 1.0 allows optional tagmanifest files
Describe the solution you'd like
Add a command line argument for creating at least one optional tagmanifest file
Replace standard library flag with spf13's flag package for POSIX/GNU-style --flags.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.