Instead of the URL path (current draft), we should use URI to be sure that this is not happening: http://blog.markfeeney.com/2010/11/java-uri-vs-url.html

Hi @magicDGS,

I hope things are going well! I'm looking at adding native support for URL's to GATK and I was going to use your library as a starting point. Do you have any interest/availability in continuing work on this? I'm going to fork the project and start doing some work on it. If you're interested I'm happy to work with you together on it, or I can just do the work on my own.

In #48, we change our plugin to git versioning to com.palantir.git-version; nevertheless, this does not handle the tagging system, nor help with the release proces (thus, it requires some scripting). There is a plugin that could handle this: https://github.com/ajoberstar/reckon

We should explore that option and add it before we move forward to version 0.1.0

With the following fields:

• HttpFileSystem originating the URL (to get the separator and the scheme from the provider)
• Authority part of URL (to re-construct the URL)
• Path within authority as byte array (to operate over the paths) with offsets array (int[])
• Optional fields (may be null): quey/reference from URL (for re-create the complete URL)

The implementation of the path part (byte[]) should be similar to other implementations of java.nio.Path (e.g., UnixPath or HadoopPath).

The methods implemented for this simple HttpPath should be:

• Constructors from URL/URI
• getFileSystem()
• toUri()
• toString() / hashCode / equals / compareTo
• isAbsolute() / toAbsolutePath() - should return always true/this in the first implementation and we shouldn't allow relative HTTP/S paths

The rest should still throw UnsupportedOperationException

This will be better for the community

I should claim the com.github.magicdgs id in Maven Central (this will be useful for other projects too) to upload the artifacts for v0.1.0.

More info here: https://central.sonatype.org/pages/ossrh-guide.html

This requires that the map holding the FS in the FSP should be thread-safe.

Once the first point release is out, we should implement some integration tests to work with:

• java.nio.file.Paths
• java.nio.file.Files

We should implement start/end with methods for other HTTP/S paths. We should also decide, although it can be later and moved to a different issue, if we want the following behavior for the String version:

• The contract suggest to throw InvalidPathExceptionif the String cannot be converted into a Path, but this does not allow to return true for HttpPath("http://example.com/file.txt").endsWITH("file.txt"). Nevertheless, if using the string "file://file.txt" it should throw because there is a mismatch with the providers. I don't know how much strict we should follow the contract here...
• If we want to test HttpPath("http://example.com/file.txt").startWith("file.txt"), this method should fail with InvalidPathException because the scheme and authority aren't provided and thus it cannot be tested. Maybe we can relax this by returning false by anything that cannot be converted into an HttpPath with the instance FileSystem to mimic the behaviour of endsWith described before.

Although we are always normalizing duplicated slashes, we should also implement other non-trivial normalization. Some examples are in https://en.wikipedia.org/wiki/URL_normalization

Instead of a FS for each authority, as currently designed, it will be nice to have one por page to being able to retrieve "root" directories, etc.

It requires:

• HttpPath without authority, but stored in the FS field
• FileSystem for each authority (base URI with only authority and protocol)
• FileSystemProvider keeping all the generated FileSystems.

This might be better before going further in the direction of a single FS per protocol.

If Git for linux is configured to convert EOL to CRLR, then the tests in the current master branch fail (see #39), because some of the tests check at the byte level the content of the file and thus there is a mismatch between the hosted file in GitHub pages and the local one.

We should add recommendations to clone the repository in Windows to make tests not failing.

It would be good if the library could be used with the Java NIO FileSystems API. See:
https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystems.html

please try following code with your library:
(after creating the usual "java.nio.file.spi.FileSystemProvider" file in META-INF)

final URI uri = new URI("http://localhost:8000/");
final Path rootPath = FileSystems.getFileSystem(uri).getPath("/test/test.txt");

it will crash with a "java.nio.file.FileSystemNotFoundException".

Could you please fix following code in your "AbstractHttpFileSystemProvider":

    @Override
    public final HttpFileSystem getFileSystem(final URI uri) {
        final HttpFileSystem fs = fileSystems.get(checkUri(uri).getAuthority());
        if (fs == null) {
            throw new FileSystemNotFoundException("URI: " + uri);
        }
        return fs;
    }

it should be like following to work:

    @Override
    public final HttpFileSystem getFileSystem(final URI uri) {
        final HttpFileSystem fs = fileSystems.get(checkUri(uri).getAuthority());
        if (fs == null) {
            try {
                return newFileSystem(uri, new HashMap<>());
            } catch (final IOException e) {
                throw new RuntimeException(e);
            }
        }
        return fs;
    }

thanx!, greetings.

Currently, I think that it is using the default (java6)

Using Circle-CI, where if I remember correctly I have a free account for public projects.

We should implement a relative HttpPathto get parts of the Path (file name, directory name, etc), and also iterate over paths. This support should include:

• Add offsets field (int[]) for easier splitting the directory (as in other path implementations)
• isAbsolute() returning false for this paths
• toAbsolutePath() should return an absolute path sited on the root
• getFileName
• getParent
• getNameCount
• getName(int)
• subpath
• iterator - using getName(int)

In addition, this relative Paths should fail for constructing an URI (no relative paths there), with a meaningful error.

We should check if there is any exception when establishing a connection with an URL that throws when the file does not exists. Here is a tracker for that. By now, we only catch the UnknownHostException, but it can be other reasons to say that the file does not exists.

Closing the FS might be useful to get rid of all connections with an HTTP/S host, but that requires that we track all the open connections and also remove from the map of the FSP.

If a path such http://example.com/foo/bar is tested for endsWith("/bar"), the contract of Path::endsWith says that it should return false; in our case, it returns true.

Implement the following methods to be able to resolve paths and get an absolute path from a relative ones:

• resolve and resolveSibling
• relativize

Maybe cache this values in a volatile variable (after #15) might be important for some users (e.g., printing Path in logs or using them in a map).

For both the javadoc and the log, the HttpFileSystem methods should have a better documentation, because we are not going to support close yet.

To avoid the NPE and throw a more informative exception (e.g., IllegalArgumentException or IllegalStateException.

It only remains for testing in #15

Including:

• Basic attributes
• Other attributes
• Attributes view

Could I get your email address or something to ask you about the Hapflk tool? I'm so sorry for issuing it here

Requires #54

Currently java8

And design an issue/pull request template

If using git for Windows, the EOL is converted and thus tests relying on reading all bytes from a file are not working. There are two possibilities to fix this problem:

• Change tests to do not check bytes but lines, removing the system-specific EOL.
• Add .gitattributes specific for the docs folder, where the line endings are kept as they are (also important for formatting docs, I guess)

In addition, we should test this on the CI (see #38)

Requires a functional HttpFileSystem (#19 and #22). For this being functional, we need to implement two parts (might be two different PRs):

• getFileSystem / newFileSystem: should create a FileSystem for each host in a Map and re-use
• getPath: should get the FS associated with the host from the map and construct the path from it
• newByteChannel returning a URLSeekableByteChannel (and failing for a writable channel), using Path.toUri().toUrl()
• checkAccess for checking for existance of files (using a HEAD request and checking HttpURLConnection.HTTP_OK.

This is the minimal for working with http paths that it is required for a functional version from my point of view.

By now, all the FS will be always open and a no-op for closing. We should implement also the getPath method for having a functional FS (requires #15).

Other methods will be implemented later.

We should decide if extra checking is required for it, because most likely it won't be used out of the package.