Add option to remove empty directories not present in metadata (`-E`, `--remove-empty-dirs`) about metastore HOT 3 CLOSED

I agree this option is needed. But the -e option is clearly marked "experimental", so reusing that option (instead of -E) for this when combined with --apply, is a natural enhancement. I don't think a new option is warranted.

from metastore.

It's true that you can read in -e option description:

This is currently an experimental feature.

But it's in such state for at least few years already, so I believe that changing such behavior now would be detrimental. Behavior breaking should be avoided if possible, especially by tools used in batch scripts. Let's give people ways to improve what they do, but let's not enforce it, because we not necessarily always know better.

In your case removing empty directories is maybe natural, but for someone else - not. That said, introducing config file (#7) will surely help users tune metastore settings to their own needs (if defaults don't meet their needs).

Thus I am against expanding -e meaning to cover removing empty directories too.

from metastore.

Done.

from metastore.