incorrect results when filtering non-master branch about git-filter-repo HOT 16 CLOSED

Thanks for the report! It's a very interesting case. I believe you mean --path rather than --paths-from-file, but other than that, I can reproduce.

After playing around with it for a while, I found this bit interesting in regards to merge commit ceb0d792f328d1fc0692197d8856a43c3936a571:

$ git ls-tree -r ceb0d792f328d1fc0692197d8856a43c3936a571:pkg/ilist
100755 blob 18a239fcf2ff42a2a71a60f1095410e034006600 ilist_state_autogen.go
100755 blob 940c2d3f633d3db94b97608620f99b0c6616b8c3 interface_list.go
$ git ls-tree -r ceb0d792f328d1fc0692197d8856a43c3936a571^1:pkg/ilist
fatal: Not a valid object name ceb0d792f328d1fc0692197d8856a43c3936a571^1:pkg/ilist
$ git ls-tree -r ceb0d792f328d1fc0692197d8856a43c3936a571^2:pkg/ilist
100644 blob dbd65ab12f9c7be313e1ea081107e61916f279ef BUILD
100644 blob 019caadca501c90873b5796e90aec1ef65df03db list.go
100644 blob 3f9abfb5607366671a36ccdf870500806863f6a2 list_test.go

Also, git diff-tree mishandles displaying this commit:
$ git diff-tree -r -c --combined-all-paths --raw ceb0d792f328d1fc0692197d8856a43c3936a571 | grep pkg/ilist
::000000 000000 100755 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 18a239fcf2ff42a2a71a60f1095410e034006600 AA pkg/ilist/ilist_state_autogen.go pkg/ilist/ilist_state_autogen.go pkg/ilist/ilist_state_autogen.go
::000000 000000 100755 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 940c2d3f633d3db94b97608620f99b0c6616b8c3 AA pkg/ilist/interface_list.go pkg/ilist/interface_list.go pkg/ilist/interface_list.go

Something is wrong where it doesn't display the three deletion entries.

I'm also annoyed that 'git log origin/go -- pkg/ilist/BUILD' shows no entries at all (while 'git rev-list --objects origin/go' will show them); it feels like history simplification is excessively aggressive to me but history simplfication has hairy edge and corner cases and I've always been leery of digging into that mess.

Anyway, I managed to get this down to a simple testcase with three commits. The first commit is a root commit with no pkg/ilist directory, the second commit is also a root commit but with a pkg/ilist directory, and the third is a merge commit merging the previous two (in the order given) with a pkg/ilist directory containing different files than what its second parent had in that directory.

Finally, adding --prune-empty=never works around the problem, though it leaves an awful lot of empty commits around that you might not want. If you don't mind the empty commits, this is a reasonable interim workaround while I work on the proper fix.

from git-filter-repo.

Any luck digging into this?

from git-filter-repo.

Aside from digging in to find the cause, I haven't had much time to work on it. I know it's not a trivial fix, but I think (fingers crossed) it shouldn't be too involved either. Hoping to get a couple days later this week.

from git-filter-repo.

Sorry for the delay here. I've got a fix staged in the empty-pruning-fix branch. Care to give it a try?

from git-filter-repo.

I went ahead an merged the changes. It works with both for the small testcase I came up with and the repository and flags you pointed out, so I'll go ahead and close, but please do report any further issues you might run into.

Thanks again for testing and reporting!

from git-filter-repo.

I have the same issue where calling git filter-repo with a single --path leaves many other subdirectories of the repo root.

As you already said, --prune-empty=never works around this issue but leaves many unwanted commits.

However, what I noticed is that when I run the command twice, the remaining unwanted directories are removed on the second run.

from git-filter-repo.

@newren I can confirm @akuntsch's report - this is still an issue.

Can you please reopen this ticket?

from git-filter-repo.

Sure, can either of you provide steps to reproduce? What repo does it happen on? Also, what version (git filter-repo --version) are you using?

from git-filter-repo.

Here's a self-contained reproduction script (note --force is needed because I'm downloading git-filter-repo into the target repository).

$ git clone https://github.com/google/gvisor -b go && cd gvisor && \
  echo before && ls && \
  curl -sfSLO https://raw.githubusercontent.com/newren/git-filter-repo/master/git-filter-repo && \
  python3 git-filter-repo --path AUTHORS --force && \
  echo after && ls
Cloning into 'gvisor'...
remote: Enumerating objects: 169, done.
remote: Counting objects: 100% (169/169), done.
remote: Compressing objects: 100% (158/158), done.
remote: Total 35908 (delta 68), reused 102 (delta 11), pack-reused 35739
Receiving objects: 100% (35908/35908), 15.26 MiB | 33.24 MiB/s, done.
Resolving deltas: 100% (26852/26852), done.
before
total 88K
drwxr-x---  5 tamird primarygroup 4.0K Dec 12 14:32 .
drwxr-xr-x 16 tamird primarygroup  40K Dec 12 14:32 ..
-rw-r-----  1 tamird primarygroup  365 Dec 12 14:32 AUTHORS
drwxr-x---  8 tamird primarygroup 4.0K Dec 12 14:32 .git
-rw-r-----  1 tamird primarygroup  838 Dec 12 14:32 go.mod
-rw-r-----  1 tamird primarygroup 2.0K Dec 12 14:32 go.sum
-rw-r-----  1 tamird primarygroup  12K Dec 12 14:32 LICENSE
drwxr-x--- 40 tamird primarygroup 4.0K Dec 12 14:32 pkg
-rw-r-----  1 tamird primarygroup  175 Dec 12 14:32 README.md
drwxr-x--- 10 tamird primarygroup 4.0K Dec 12 14:32 runsc
Parsed 2738 commits
New history written in 0.39 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
HEAD is now at 42bafc62 Merge release-20190806.1-356-gfe2e076 (automated)
Enumerating objects: 533, done.
Counting objects: 100% (533/533), done.
Delta compression using up to 72 threads
Compressing objects: 100% (365/365), done.
Writing objects: 100% (533/533), done.
Total 533 (delta 152), reused 375 (delta 138), pack-reused 0
Completely finished after 0.79 seconds.
after
total 232K
drwxr-x---  6 tamird primarygroup 4.0K Dec 12 14:32 .
drwxr-xr-x 16 tamird primarygroup  40K Dec 12 14:32 ..
-rw-r-----  1 tamird primarygroup  365 Dec 12 14:32 AUTHORS
drwxr-x--- 10 tamird primarygroup 4.0K Dec 12 14:32 .git
-rw-r-----  1 tamird primarygroup 155K Dec 12 14:32 git-filter-repo
-rw-r-----  1 tamird primarygroup  822 Dec 12 14:32 go.mod
drwxr-x--- 34 tamird primarygroup 4.0K Dec 12 14:32 pkg
-rw-r-----  1 tamird primarygroup  177 Dec 12 14:32 README.md
drwxr-x---  8 tamird primarygroup 4.0K Dec 12 14:32 runsc
drwxr-x---  3 tamird primarygroup 4.0K Dec 12 14:32 third_party

from git-filter-repo.

So, my original fix would have only worked in special cases; and in some cases, not only would it fail to fix the problem but it'd actually make it worse. Ick. However, I pushed a few more commits up that should fix this problem. Since I got it wrong once, though, I'm going to leave this open and ask if either of you two can duplicate the problems with the new version.

Also, @tamird, you're awesome. Thanks so much for providing great testcases and clear instructions. Very, very helpful.

My apologies for taking so long to get back to this.

from git-filter-repo.

I was using version 2.24.0 of git-filter-repo today and experienced a serious bug when using "--path" and "--prune-empty always" at the same time. Specifically, many files that were outside my specified path were retained in some commits, and those commits should have been pruned empty commits instead.

However, after I upgraded to the current master branch of git-filter-repo, I was no longer able to reproduce the bug. I hope this data point is helpful in deciding whether or not to release the fix.

from git-filter-repo.

@dglynch: thanks for the feedback! I'm glad the fix works for you. I'll probably make a new release in the next week and a half to two weeks. Before then, I'm hoping to also hear back from @akuntsch and @tamird to verify that the fix works for their usecases as well. (And I'd also like to be able to fix #37 before the next release, but there aren't any available steps to reproduce that one yet...)

from git-filter-repo.

The fix seems to work for my use case. I could sadly not provide any steps to reproduce since this happened in corporate repository.

from git-filter-repo.

Looks good to me!

from git-filter-repo.

Awesome, thanks for verifying @akuntsch and @tamird!

from git-filter-repo.

[...] And I'd also like to be able to fix #37 before the next release, but there aren't any available steps to reproduce that one yet...

I am going to re-export my repo from #37 today, I'll let you know if the problem still exists. I also experienced #24 there, it might happen that working around that will also somehow solve other problems.

from git-filter-repo.