Excluding files & folders about gphotos-uploader-cli HOT 14 CLOSED

Same request! I‘m using this on my Synology DSM which contains huge number of @eadir folder and .thumbs.db . So it would be nice if we can excluding those folder.

from gphotos-uploader-cli.

Same request!
Also upload photos from my Synology DSM.

from gphotos-uploader-cli.

I'd like to propose a user interface for selecting and ignoring files to upload.

Add excludePattern and includePattern to configuration inside jobs, both parameters are an array of multiple patterns.

• If neither excludePattern nor includePattern is specified, all files are uploaded.
• If only includePattern is specified, only the items that match the given pattern are uploaded.
• If only excludePattern is specified, all items except those that match the given pattern are uploaded.
• If both excludePattern and includePattern are specified, only those items that matches with includePattern and not matches with excludePattern are uploaded. So excludePattern takes priority over includePattern.

BEFORE WAS:

• If both excludePattern and includePattern are specified, only those items are uploaded that either
  • don't match any exclude pattern
  • match an exclude pattern and at least one include pattern

As for the pattern, I'd like to implement something that is easy to use. Some examples:

• /proc matches all items below /proc, e.g. /proc/kcore
• foo/* matches all items in a subdirectory of a directory called foo, e.g. /home/user/foo/test.txt
• foo/**/bar*/*.png matches all items with the extension .png in a directory whose name begins with bar that somewhere has a parent directory called foo.

So, the rules are roughly:

• If the pattern starts with a slash, the following name is meant absolute
• If the pattern matches a directory, it matches all subdirs/items below, e.g. /proc is the same as /proc/*
• Similar to shell globbing, * matches everything

Usage examples:

• Upload everything except all items finishing with -thumb:

excludePattern: [ "*-thumb"] 

• Upload everything except all items in a subdirectory of a directory called @eadir or .thumbs.db/*:

excludePattern: [ "@eadir/*", ".thumbs.db/*"] 

• Upload only PNG files ignoring everything else:

includePattern: [ "*.png" ]

• Upload all PNG files, excluding everything that begins with Screenshoot-:

excludePattern: [ "Screenshoot-*" ]
includePattern: [ "*.png" ]

I think this can be implemented fairly easy by using filepath.Match.

What do you think about this approach? Are you aware of any use cases that aren't covered?

@OTZro, @j6s , @stig3 , @Goodwu , @tonymet, @nmrshll , @avidrissman

connects: #82, #66

from gphotos-uploader-cli.

I’d <3 an include/exclude system. Meanwhile…

If both excludePattern and includePattern are specified, only those items are uploaded that either

• don't match any exclude pattern
• match an exclude pattern and at least one include pattern

I’m not sure that’s quite the right semantics; with the first rule it sounds like you’d upload things that weren’t on the include list, and with the second rule I’m not sure how I successfully exclude anything (e.g. “include /photos” “exclude .DS_Store” would upload .DS_Store even though I’m trying to exclude them). Even your example seems broken with your semantics:

• Upload all PNG files, excluding everything that begins with Screenshoot-:

excludePattern: [ "Screenshoot-*" ]
includePattern: [ "*.png" ]

With your rules, a PNG file starting with Screenshoot- wouldn’t be excluded; your second rule is to upload if it “match[es] an exclude pattern and at least one include pattern” which this one does.

My version of implementing include/exclude (and which works for your examples) would be:

for (item in items)
  if there is an include filter
    if the item doesn’t match the include filter, skip the item
  if there is an exclude filter
    if the item matches the exclude filter, skip the item
  upload the item

Surely there are existing programs with include/exclude semantics that we can borrow here.

from gphotos-uploader-cli.

If both excludePattern and includePattern are specified, only those items are uploaded that either

• don't match any exclude pattern
• match an exclude pattern and at least one include pattern

Yeah, I was confused by this too. If both these patterns are included, it should upload everything in the include pattern except whatever is in the exclude pattern. In other words, exclude takes priority over include.

As for how patterns should actually work and match, isn't there a Go library that you could use that works exactly the same as shell globbing? That would be the best solution IMO.

from gphotos-uploader-cli.

Perfect! This was my intention, but I didn't explain it good. Now It's clear to everyone (I've modified the description).

@rfgamaral I was thinking to use filepath.Match (see here) that implement these patterns.

from gphotos-uploader-cli.

@rfgamaral I was thinking to use filepath.Match (see here) that implement these patterns.

I'm no expert in Go but looking at the documentation that option doesn't feel like the best one for me. Although this library only has URL examples, it probably works with file paths too. This was just my first find, there might be better libraries out there, not sure.

With this one we could have an include path like /photos/**/*.{jpg,png}, which would upload all JPG and PNG files inside /photos, recursively. I don't think you can specify such pattern with filepath.Match, as easily at least. Please correct me if I'm wrong.

from gphotos-uploader-cli.

Hi @rfgamaral

I've just started to code and it's more or less doable, see filter package. But I'm going to take a look to the one you have shared.

from gphotos-uploader-cli.

How do you whitelist a list of extensions in one line? Is that even possible with that implementation?

from gphotos-uploader-cli.

I don't think so... You should use something like this:

excludePattern: [ "*.png", "*.jpg" ]

from gphotos-uploader-cli.

I guess it's still possible, albeit with a different syntax. Personally I would explore the globbing possibility, I think it gives us great flexibility in a short and easy to understand syntax. But maybe that's just me. Up to you now :)

from gphotos-uploader-cli.

I feel the same with my approach. I think that's easier than the globbing one:

excludePattern: [ "*.png", "*.jpg" ]

instead of:

excludePattern: [ "*.{png,jpg}" ]

IMO It's easier the first one... bear in mind that it's very improbable to use full directory pattern since the directory is set in sourceFolder.

from gphotos-uploader-cli.

Easier is subjective (it always depends on one's skills and experience), that's why I used the word "flexibility" instead.

Actually, implementing globbing shouldn't mean you can't still do this:

excludePattern: [ "*.png", "*.jpg" ]

If you add support for an array of globbing patterns, you can specify the include/exclude filters the way each user is more comfortable with. You want to put every single extension as 1 item per array? Fine, do it. You want to have a single item with a globbing expression? Cool, you can do that too.

Personally, that's how I'd do it :)

from gphotos-uploader-cli.

This issue has been completed in PR #88. It's going to be available in version 0.4.0 after release.

from gphotos-uploader-cli.