Comments (17)
@sebix do you know any good solution to solve this problem?
from intelmq.
Have there been any big files in the past?
To keep the number of commits small, please don't push to master directly. When looking through master I often see small commits and following commits correcting some minor bug, typo. Please use 1) branches and optionally 2) push them to your fork and review the PR another day, and 3) use git rebase
and git commit --amend
to stash commits. Have a look here for some hints on rebasing: http://stackoverflow.com/questions/6934752/combining-multiple-commits-before-pushing-in-git
Commits represent small independent changes, this is good. but also don't exaggerate when fixing minor things in last commits.
from intelmq.
yes, thats why I ask. Do you know how to remove that old file that was committed by mistake?
Ok, thank you for the explanation. Always learning heheh
from intelmq.
You can use the BFG Repo cleaner, remove the file src/bots/experts/geoip/GeoLite2-City.mmdb
or do bfg --strip-blobs-bigger-than 10M intelmq.git
. You need to force the push (-f
) as remote has more commits than local and would be rejected otherwise.
This will reduce the size of the repo from 19MB to 2MB.
from intelmq.
On Aug 10, 2015, at 9:20 AM, Sebastian [email protected] wrote:
You can use the https://rtyley.github.io/bfg-repo-cleaner/, remove the file src/bots/experts/geoip/GeoLite2-City.mmdb or do bfg --strip-blobs-bigger-than 10M my-repo.git. You need to force the push (-f) as remote has more commits than local and would be rejected otherwise.
This will reduce the size of the repo from 19MB to 2MB.
@SYNchroACK: okay? Do you want to do this or should I?
—
Reply to this email directly or view it on GitHub.
from intelmq.
The file is still there:
I'm just trying to get my fork right. Updating master and v0.9 branches are quite easy, but I'm struggling with the feature branch. Here's what I have yet to get the history fuck up right:
First, check if you have the upstream repository configured:
git remote -v
If there's no upstream repository from certtools, add it with SSH or HTTPS:
git remote add upstream [email protected]:certtools/intelmq.git
git remote add upstream https://github.com/certtools/intelmq.git
Fetch the upstream branches without merging them, then reset your master branch to upstream master:
git fetch upstream
git reset --hard upstream/master
And update your remote origin:
git push -f origin master
To delete branches locally:
git branch -d v1.0-beta
And remotely on origin:
git push origin :v1.0-beta
from intelmq.
If you have a feature branch with unmerged commits, first make a full backup or write your changes to patch files.
git checkout feature
git format-patch master
Then we can use rebasing. Find the last common commit:
git merge-base feature v1.0-final
Then do interactive rebasing starting with this commit:
git rebase -i [commit-id from above]
Alternatively, it can be useful to use:
git rebase -i master
rebase -i
will open an editor asking you what commits should be picked or combined. Delete all lines that are not part of your feature branch. Conflicts should only occur if really necessary.
git push -f origin feature
Hope it helps!
from intelmq.
$ java -jar bfg-1.12.4.jar --strip-blobs-bigger-than 10M intelmq.git/
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Using repo : /tmp/intelmq.git
This repo has been processed by The BFG before! Will prune repo before proceeding - to avoid unnecessary cleaning work on unused objects...
Completed prune of old objects - will now proceed with the main job!
Scanning packfile for large blobs: 17756
Scanning packfile for large blobs completed in 114 ms.
Found 1 blob ids for large blobs - biggest=30167713 smallest=30167713
Total size (unpacked)=30167713
Found 169 objects to protect
Found 139 commit-pointing refs : HEAD, refs/heads/master, refs/heads/v0.9, ...
Protected commits
-----------------
These are your protected commits, and so their contents will NOT be altered:
* commit 237ea694 (protected by 'HEAD')
Cleaning
--------
Found 2452 commits
Cleaning commits: 100% (2452/2452)
Cleaning commits completed in 974 ms.
Updating 134 Refs
-----------------
Ref Before After
-----------------------------------------
refs/pull/1/head | 40ec6c80 | cf43a8d1
refs/pull/10/head | 21d31914 | 8d968db6
refs/pull/101/head | 825e15ac | d8c99d7a
refs/pull/102/head | f553cd98 | 872ecd9b
refs/pull/11/head | 813788a0 | ab38df99
refs/pull/11/merge | 40c70aa8 | 376a5649
refs/pull/110/head | ea9f0713 | dca2b052
refs/pull/12/head | 7c4f0347 | ea59fac8
refs/pull/12/merge | a633342a | a90333eb
refs/pull/13/head | 0b44e032 | e5d9f925
refs/pull/134/head | fd6f5b7d | 5254f315
refs/pull/14/head | 53a3b7dc | d71e524d
refs/pull/141/head | d0284b16 | ee53d51c
refs/pull/143/head | 5e879858 | c97d148c
refs/pull/144/head | c06ce862 | bd749864
...
Updating references: 100% (134/134)
...Ref update completed in 69 ms.
Commit Tree-Dirt History
------------------------
Earliest Latest
| |
DDmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
D = dirty commits (file tree fixed)
m = modified commits (commit message or parents changed)
. = clean commits (no changes to file tree)
Before After
-------------------------------------------
First modified commit | ef24878a | 5c294afd
Last dirty commit | 52b92b40 | 54522af2
Deleted files
-------------
Filename Git id
---------------------------------------
GeoLite2-City.mmdb | aeee4b45 (28.8 MB)
In total, 1361 object ids were changed. Full details are logged here:
/tmp/intelmq.git.bfg-report/2015-08-12/17-22-52
BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive
Has the BFG saved you time? Support the BFG on BountySource: https://j.mp/fund-bfg
$ cd intelmq.git
$ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 7748, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7625/7625), done.
Writing objects: 100% (7748/7748), done.
Total 7748 (delta 5006), reused 2308 (delta 0)
$ git push
Counting objects: 1550, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (653/653), done.
Writing objects: 100% (1550/1550), 293.41 KiB | 0 bytes/s, done.
Total 1550 (delta 1034), reused 1378 (delta 886)
To https://github.com/certtools/intelmq.git
! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)
! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref)
! [remote rejected] refs/pull/101/head -> refs/pull/101/head (deny updating a hidden ref)
! [remote rejected] refs/pull/102/head -> refs/pull/102/head (deny updating a hidden ref)
! [remote rejected] refs/pull/11/head -> refs/pull/11/head (deny updating a hidden ref)
! [remote rejected] refs/pull/11/merge -> refs/pull/11/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/110/head -> refs/pull/110/head (deny updating a hidden ref)
! [remote rejected] refs/pull/12/head -> refs/pull/12/head (deny updating a hidden ref)
! [remote rejected] refs/pull/12/merge -> refs/pull/12/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/13/head -> refs/pull/13/head (deny updating a hidden ref)
! [remote rejected] refs/pull/134/head -> refs/pull/134/head (deny updating a hidden ref)
! [remote rejected] refs/pull/14/head -> refs/pull/14/head (deny updating a hidden ref)
! [remote rejected] refs/pull/141/head -> refs/pull/141/head (deny updating a hidden ref)
! [remote rejected] refs/pull/143/head -> refs/pull/143/head (deny updating a hidden ref)
! [remote rejected] refs/pull/144/head -> refs/pull/144/head (deny updating a hidden ref)
! [remote rejected] refs/pull/145/head -> refs/pull/145/head (deny updating a hidden ref)
! [remote rejected] refs/pull/146/head -> refs/pull/146/head (deny updating a hidden ref)
! [remote rejected] refs/pull/148/head -> refs/pull/148/head (deny updating a hidden ref)
! [remote rejected] refs/pull/149/head -> refs/pull/149/head (deny updating a hidden ref)
! [remote rejected] refs/pull/149/merge -> refs/pull/149/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/15/head -> refs/pull/15/head (deny updating a hidden ref)
! [remote rejected] refs/pull/150/head -> refs/pull/150/head (deny updating a hidden ref)
! [remote rejected] refs/pull/158/head -> refs/pull/158/head (deny updating a hidden ref)
! [remote rejected] refs/pull/158/merge -> refs/pull/158/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/16/head -> refs/pull/16/head (deny updating a hidden ref)
! [remote rejected] refs/pull/17/head -> refs/pull/17/head (deny updating a hidden ref)
! [remote rejected] refs/pull/170/head -> refs/pull/170/head (deny updating a hidden ref)
! [remote rejected] refs/pull/171/head -> refs/pull/171/head (deny updating a hidden ref)
! [remote rejected] refs/pull/171/merge -> refs/pull/171/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/172/head -> refs/pull/172/head (deny updating a hidden ref)
! [remote rejected] refs/pull/172/merge -> refs/pull/172/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/173/head -> refs/pull/173/head (deny updating a hidden ref)
! [remote rejected] refs/pull/177/head -> refs/pull/177/head (deny updating a hidden ref)
! [remote rejected] refs/pull/178/head -> refs/pull/178/head (deny updating a hidden ref)
! [remote rejected] refs/pull/178/merge -> refs/pull/178/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/179/head -> refs/pull/179/head (deny updating a hidden ref)
! [remote rejected] refs/pull/18/head -> refs/pull/18/head (deny updating a hidden ref)
! [remote rejected] refs/pull/180/head -> refs/pull/180/head (deny updating a hidden ref)
! [remote rejected] refs/pull/180/merge -> refs/pull/180/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/182/head -> refs/pull/182/head (deny updating a hidden ref)
! [remote rejected] refs/pull/183/head -> refs/pull/183/head (deny updating a hidden ref)
! [remote rejected] refs/pull/184/head -> refs/pull/184/head (deny updating a hidden ref)
! [remote rejected] refs/pull/184/merge -> refs/pull/184/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/19/head -> refs/pull/19/head (deny updating a hidden ref)
! [remote rejected] refs/pull/191/head -> refs/pull/191/head (deny updating a hidden ref)
! [remote rejected] refs/pull/191/merge -> refs/pull/191/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/192/head -> refs/pull/192/head (deny updating a hidden ref)
! [remote rejected] refs/pull/192/merge -> refs/pull/192/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/194/head -> refs/pull/194/head (deny updating a hidden ref)
! [remote rejected] refs/pull/194/merge -> refs/pull/194/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/195/head -> refs/pull/195/head (deny updating a hidden ref)
! [remote rejected] refs/pull/195/merge -> refs/pull/195/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/196/head -> refs/pull/196/head (deny updating a hidden ref)
! [remote rejected] refs/pull/196/merge -> refs/pull/196/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/197/head -> refs/pull/197/head (deny updating a hidden ref)
! [remote rejected] refs/pull/197/merge -> refs/pull/197/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
! [remote rejected] refs/pull/20/head -> refs/pull/20/head (deny updating a hidden ref)
! [remote rejected] refs/pull/201/head -> refs/pull/201/head (deny updating a hidden ref)
! [remote rejected] refs/pull/201/merge -> refs/pull/201/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/202/head -> refs/pull/202/head (deny updating a hidden ref)
! [remote rejected] refs/pull/204/head -> refs/pull/204/head (deny updating a hidden ref)
! [remote rejected] refs/pull/204/merge -> refs/pull/204/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/21/head -> refs/pull/21/head (deny updating a hidden ref)
! [remote rejected] refs/pull/211/head -> refs/pull/211/head (deny updating a hidden ref)
! [remote rejected] refs/pull/213/head -> refs/pull/213/head (deny updating a hidden ref)
! [remote rejected] refs/pull/215/head -> refs/pull/215/head (deny updating a hidden ref)
! [remote rejected] refs/pull/215/merge -> refs/pull/215/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/216/head -> refs/pull/216/head (deny updating a hidden ref)
! [remote rejected] refs/pull/220/head -> refs/pull/220/head (deny updating a hidden ref)
! [remote rejected] refs/pull/223/head -> refs/pull/223/head (deny updating a hidden ref)
! [remote rejected] refs/pull/224/head -> refs/pull/224/head (deny updating a hidden ref)
! [remote rejected] refs/pull/227/head -> refs/pull/227/head (deny updating a hidden ref)
! [remote rejected] refs/pull/229/head -> refs/pull/229/head (deny updating a hidden ref)
! [remote rejected] refs/pull/229/merge -> refs/pull/229/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/232/head -> refs/pull/232/head (deny updating a hidden ref)
! [remote rejected] refs/pull/234/head -> refs/pull/234/head (deny updating a hidden ref)
! [remote rejected] refs/pull/234/merge -> refs/pull/234/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/235/head -> refs/pull/235/head (deny updating a hidden ref)
! [remote rejected] refs/pull/236/head -> refs/pull/236/head (deny updating a hidden ref)
! [remote rejected] refs/pull/236/merge -> refs/pull/236/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/237/head -> refs/pull/237/head (deny updating a hidden ref)
! [remote rejected] refs/pull/237/merge -> refs/pull/237/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/238/head -> refs/pull/238/head (deny updating a hidden ref)
! [remote rejected] refs/pull/240/head -> refs/pull/240/head (deny updating a hidden ref)
! [remote rejected] refs/pull/243/head -> refs/pull/243/head (deny updating a hidden ref)
! [remote rejected] refs/pull/245/head -> refs/pull/245/head (deny updating a hidden ref)
! [remote rejected] refs/pull/262/head -> refs/pull/262/head (deny updating a hidden ref)
! [remote rejected] refs/pull/263/head -> refs/pull/263/head (deny updating a hidden ref)
! [remote rejected] refs/pull/265/head -> refs/pull/265/head (deny updating a hidden ref)
! [remote rejected] refs/pull/268/head -> refs/pull/268/head (deny updating a hidden ref)
! [remote rejected] refs/pull/269/head -> refs/pull/269/head (deny updating a hidden ref)
! [remote rejected] refs/pull/269/merge -> refs/pull/269/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/270/head -> refs/pull/270/head (deny updating a hidden ref)
! [remote rejected] refs/pull/271/head -> refs/pull/271/head (deny updating a hidden ref)
! [remote rejected] refs/pull/273/head -> refs/pull/273/head (deny updating a hidden ref)
! [remote rejected] refs/pull/274/head -> refs/pull/274/head (deny updating a hidden ref)
! [remote rejected] refs/pull/275/head -> refs/pull/275/head (deny updating a hidden ref)
! [remote rejected] refs/pull/277/head -> refs/pull/277/head (deny updating a hidden ref)
! [remote rejected] refs/pull/282/head -> refs/pull/282/head (deny updating a hidden ref)
! [remote rejected] refs/pull/282/merge -> refs/pull/282/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
! [remote rejected] refs/pull/32/head -> refs/pull/32/head (deny updating a hidden ref)
! [remote rejected] refs/pull/33/head -> refs/pull/33/head (deny updating a hidden ref)
! [remote rejected] refs/pull/36/head -> refs/pull/36/head (deny updating a hidden ref)
! [remote rejected] refs/pull/37/head -> refs/pull/37/head (deny updating a hidden ref)
! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref)
! [remote rejected] refs/pull/40/head -> refs/pull/40/head (deny updating a hidden ref)
! [remote rejected] refs/pull/43/head -> refs/pull/43/head (deny updating a hidden ref)
! [remote rejected] refs/pull/48/head -> refs/pull/48/head (deny updating a hidden ref)
! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref)
! [remote rejected] refs/pull/50/head -> refs/pull/50/head (deny updating a hidden ref)
! [remote rejected] refs/pull/51/head -> refs/pull/51/head (deny updating a hidden ref)
! [remote rejected] refs/pull/52/head -> refs/pull/52/head (deny updating a hidden ref)
! [remote rejected] refs/pull/55/head -> refs/pull/55/head (deny updating a hidden ref)
! [remote rejected] refs/pull/58/head -> refs/pull/58/head (deny updating a hidden ref)
! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
! [remote rejected] refs/pull/63/head -> refs/pull/63/head (deny updating a hidden ref)
! [remote rejected] refs/pull/65/head -> refs/pull/65/head (deny updating a hidden ref)
! [remote rejected] refs/pull/69/head -> refs/pull/69/head (deny updating a hidden ref)
! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
! [remote rejected] refs/pull/71/head -> refs/pull/71/head (deny updating a hidden ref)
! [remote rejected] refs/pull/75/head -> refs/pull/75/head (deny updating a hidden ref)
! [remote rejected] refs/pull/79/head -> refs/pull/79/head (deny updating a hidden ref)
! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
! [remote rejected] refs/pull/80/head -> refs/pull/80/head (deny updating a hidden ref)
! [remote rejected] refs/pull/81/head -> refs/pull/81/head (deny updating a hidden ref)
! [remote rejected] refs/pull/83/head -> refs/pull/83/head (deny updating a hidden ref)
! [remote rejected] refs/pull/86/head -> refs/pull/86/head (deny updating a hidden ref)
! [remote rejected] refs/pull/88/head -> refs/pull/88/head (deny updating a hidden ref)
! [remote rejected] refs/pull/89/head -> refs/pull/89/head (deny updating a hidden ref)
! [remote rejected] refs/pull/9/head -> refs/pull/9/head (deny updating a hidden ref)
! [remote rejected] refs/pull/91/head -> refs/pull/91/head (deny updating a hidden ref)
! [remote rejected] refs/pull/95/head -> refs/pull/95/head (deny updating a hidden ref)
! [remote rejected] refs/pull/98/head -> refs/pull/98/head (deny updating a hidden ref)
! [remote rejected] refs/pull/99/head -> refs/pull/99/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/certtools/intelmq.git'
@sebix do you know how to solve it?
from intelmq.
@sebix https://github.com/sebix do you know how to solve it?
Yes:
You need to force the push (|-f|) as remote has more commits than
local and would be rejected otherwise.
git push -f origin master
This will break the git history once again, but that's unavoidable.
Don't forget to make the backup.
from intelmq.
@sebix that command doesnt work.
$ git push -f origin master
error: --mirror can't be combined with refspecs
from intelmq.
Then leave out the refspec master
.
from intelmq.
$ git push -f origin
same result :S
from intelmq.
But you have just deleted your last commits from remote. The last commit in github repo was 2 days ago.
from intelmq.
When doing a normal clone of the repo now, it only has 2MB, or 5.4MB with extracted data. Doing a mirror clone, it results in 17MB. It seems we cannot change this, as there are read-only refs, see this comment in BFG:
It's a fiddly problem - you can update all the 'real' refs in your repo, but all the ones beginning 'refs/pull' are synthetic read-only refs created by GitHub - you can't update (and therefore 'clean') them if they're from outside your repository [like Pull Requests].
So, if you're pushing your updated refs up to your repository, all the non-pull-request refs are accepted and fixed, but the Pull Request ref updates will be rejected.
I'm really not sure if there's anyway the BFG can make this experience nicer... as an open question, can youthink of something you would like to happen?
from intelmq.
@sebix I think we can live with the current scenario. Do you agree?
from intelmq.
On 08/14/2015 11:06 PM, Tomás Lima wrote:
@sebix https://github.com/sebix I think we can live with the current
scenario. Do you agree?
Yes. The only other possibility is to delete the repo and create a new
one. This is not a viable option.
from intelmq.
yap. agreed.
from intelmq.
Related Issues (20)
- Missing docs for SMTP batch output (lost in conversion) HOT 1
- Twitter collector issues
- Redis Pipeline error HOT 3
- Schedule bots directly in IntelMQ HOT 7
- Revisit documentation on supported OS versions and packages HOT 2
- deprecate readthedocs.io
- crontab is missing shadowserver update schema script
- Shadowserver parser docs missing part on feedname HOT 2
- intelmq.lib.exceptions.KeyExists HOT 5
- Feature Request: Shadowserver parser setting feed.documentation link HOT 1
- Upgrades error HOT 9
- Can we change it to check for empty "" instead of None. Shadowserver api collector. HOT 5
- documentation build workflow fails in forks
- Initial fetch of shadowserver-schema.json fails even in auto_update mode
- On Windows 10 using Msys2 / Mingw64 - ModuleNotFoundError: No module named 'grp' HOT 3
- intelmq_gen_feeds_conf: yaml.load deprecated HOT 3
- Allow sieve reuse data from the event
- CLI command in ShadowServer API collector/parser to list available report types HOT 2
- pyyaml PendingDeprecationWarning: you should no longer specify 'unsafe' -> test failure
- Shadowserver API bot HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from intelmq.