Comments (17)
In my view it shouldn't be used to update applications in production (maybe in dev), but to update data used by an application and separating concerns (ssh + git) into their own container.
People can push markdown in the git repo and the site will get regenerated on our side too. It's just another way to do it. Wasn't pushing for it being the better solution, just a workaround until this gets cleared up.
from git-sync.
you should be able to inotify on the symlink itself - hugo might want to learn how to do that.
from git-sync.
As a sidenote:
We are building the page via ci from each commit and then push the static version into a new repo $repo_static. This static repo will get pulled in by git-sync and works like a charm.
Positive things: hugo only needed during build, static repo only contains the page, less moving parts in production.
from git-sync.
That's fine, but as far as I understood, that's the whole idea of a git-sync
sidecar container. In this case/demo was to prove you can update an application without redeploying.
So that people can just push markdown in the git repo, and the site will get regenerated.
from git-sync.
I agree that it's not something you want to do in prod. :)
Anyway, my feeling is that we move away from the subject which is this specific issue.
I set up a watch with inotify-tools
on the the synced
directory. This are the logs with what inotify sees that happened on the FS after updating the git repo (I just added a word in one of the files):
# inotifywait -m -r -e modify,attrib,close_write,move,create,delete /src/git_files
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/src/git_files/site/content/ DELETE index.md
/src/git_files/site/content/post/ DELETE hugoisforlovers.md
/src/git_files/site/content/post/ DELETE creating-a-new-theme.md
/src/git_files/site/content/post/ DELETE migrate-from-jekyll.md
/src/git_files/site/content/post/ DELETE goisforlovers.md
/src/git_files/site/content/ DELETE,ISDIR post
/src/git_files/site/ DELETE,ISDIR content
/src/git_files/site/ DELETE config.toml
/src/git_files/site/static/ DELETE .gitkeep
/src/git_files/site/ DELETE,ISDIR static
/src/git_files/site/ DELETE themes
/src/git_files/ DELETE,ISDIR site
/src/git_files/ DELETE .gitignore
/src/git_files/ DELETE .git
So basically this means that what Hugo sees is accurate. I'll try to look some more into it any maybe come up with a PR.
from git-sync.
Maybe it would be a better idea to use rsync
between the checked out directory and clean directory instead of symlinks ?
from git-sync.
That would introduce rsync as additional dependency. Not sure, if that's worth it. Also it does not give you an atomic switching operation. Currently the files are pulled and the symlink is used as atomic operation so inconsistent state is prevented. Rsync wouldn't have that atomic operation.
from git-sync.
Well, the symlink doesn't work for this specific case. And I expect it to be the same for other cases.
Got any suggestions for a fix, @stp-ip ?
from git-sync.
Do you mind sharing a redacted yml/json file for the specific use case? Symlinks in gerenal should be supported by hugo, when they are on the root level. gohugoio/hugo#1857
One way to "solve" symlinking issues would be to add a "GIT_SYNC_ATOMIC" false option, which disables the symlinking behaviour and instead pulls in new changes perhaps. That will lead to other issues with inconsistent state, therefore I am not seeing it as the best way to go forward.
from git-sync.
As for a specific use case, just try launching what's in the demo
folder and try to update the repository to see the changes on the (newly) generated website. It won't work.
As for the logs from different places, I added them to the issue, including what the inotify-tools
sees on the filesystem if I put a watch on that directory. So it's not just a specific hugo problem.
from git-sync.
My vague assumption (don't know most of the internals of inotify) is, that inotify follows the symlink and basically adds the watch on the worktree, which is removed during cleanup. Therefore the filesystem actually changes atomically, but inotify (which hugo is also using) is only seeing the DELETE as it watches the rev-* directory instead of just the symlink.
from git-sync.
@stp-ip Yes, I think you're right about the problems but nothing pops in my head on how to fix it nicely.
I did a rough implementation that adds rsync functionality on top of the symlinks here: ursuad/git-sync/tree/feature. There are a few issues with it, but it got the job done and it can be used as a starting point for implementing something better.
• If GIT_SYNC_ADD_NONATOMIC
is true (it's false by default), it will add rsync
functionality on top of the symlink stuff. (it was faster add something than changing existing behaviour based on conditions )
• Rsync will put it's files in GIT_SYNC_NONATOMIC_DEST
, which is mandatory to define if you enable first option
• You can specify GIT_SYNC_NONATOMIC_EXCLUDE
- a pattern if you don't want to sync some files/directories from git
I added the last option because rsync is running with --delete
and I wanted to be able to exclude the themes
directory (or other directories which might be created by other containers) . In this case the themes
directory was created by the Hugo container.
Also, I agree with the idea that the rsync operation is nonatomic and it might cause some problems but just like symlinks, it will work for some. :)
Again, this is just a draft implementation.
For Hugo, it worked. You can see the logs from Hugo during the rsync
operation below, so you can see the transitional files created:
Change detected, rebuilding site
2016-10-27 16:16 +0000
Source changed /src/site/content/.index.md.KKpChN
Source changed /src/site/content/post/.My-New-Post.md.kjrq26
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/.My-New-Post.md.kjrq26
ERROR: 2016/10/27 16:16:06 site.go:583: Error reading file /src/site/content/post/.My-New-Post.md.kjrq26 ; open /src/site/content/post/.My-New-Post.md.kjrq26: no such file or directory
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/.My-New-Post.md.kjrq26
ERROR: 2016/10/27 16:16:06 site.go:583: Error reading file /src/site/content/post/.My-New-Post.md.kjrq26 ; open /src/site/content/post/.My-New-Post.md.kjrq26: no such file or directory
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/My-New-Post.md
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/hive_blog.md
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/.my-personal-test.md.xS56xK
ERROR: 2016/10/27 16:16:06 site.go:583: Error reading file /src/site/content/post/.my-personal-test.md.xS56xK ; open /src/site/content/post/.my-personal-test.md.xS56xK: no such file or directory
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/my-personal-test.md
INFO: 2016/10/27 16:16:06 site.go:1489: found taxonomies: map[string]string{"tag":"tags", "category":"categories"}
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/tags/test/page/1" translated to "tags/test/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/tags/heating-patterns/page/1" translated to "tags/heating-patterns/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/tags/hive/page/1" translated to "tags/hive/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/categories/new-things/page/1" translated to "categories/new-things/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/tags/personal/page/1" translated to "tags/personal/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/categories/another-category/page/1" translated to "categories/another-category/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/categories/haething-patterns/page/1" translated to "categories/haething-patterns/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/post/page/1" translated to "post/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
WARN: 2016/10/27 16:16:06 site.go:2373: "404.html" is rendered empty
Built site for language en:
2 of 2 drafts rendered
0 future content
0 expired content
4 pages created
0 non-page files copied
9 paginator pages created
4 tags created
3 categories created
total in 54 ms
from git-sync.
I don't know what Hugo is doing, but in this case I don't really see any other way to update files atomically. This is how git-sync works. I really don't like the idea of having two very different operating modes - this is already too tricky. Can't we just fix hugo?
from git-sync.
@thockin it's not just hugo. As said above it seems that it is the used inotify, which follows symlinks and watches the dereferenced files/dirs instead of the symlinks.
@ursuad why use rsync and not just copy? Would at least prevent the additional dependency.
Overall I am still not convinced it's the best course of action. It complicates the side car container and basically hacks around a problem, we should first dive deeper, if there is a fix possible. I will do my best to dive deeper on my side next week or during the weekend.
Thanks @ursuad for the PoC with rsync. Always appreciate such work.
from git-sync.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
Prevent issues from auto-closing with an /lifecycle frozen
comment.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or @fejta
.
/lifecycle stale
from git-sync.
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten
.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or @fejta
.
/lifecycle rotten
/remove-lifecycle stale
from git-sync.
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen
.
Mark the issue as fresh with /remove-lifecycle rotten
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
from git-sync.
Related Issues (20)
- GitLab via ssh key HOT 3
- How to disable commit hash in the target folder? HOT 2
- Allow the submodule functionality for repos copied using vcs tool HOT 1
- Temporarily missing files when using $GITSYNC_LINK HOT 4
- Remediate security vulnerabilities on 4.1.0 HOT 5
- High vulnerability CVE-2023-6246 and CVE-2023-6779 HOT 1
- Issue with PYTHONPATH Resolution in git-sync 4.0.0 HOT 2
- v4 slack webhook HOT 6
- exechook behavior HOT 4
- Remediate 4.2.2 security vulnerabilities HOT 1
- Support GitHub App authentication HOT 7
- TODO: infra to make #878 work HOT 1
- Curious about real example raising race condition when using v3 HOT 1
- exechook working HOT 8
- TODO: shellcheck test_e2e.sh and test_git.sh HOT 5
- /bin/sh: ./build/test.sh: not found
- test_e2e.sh: use `local` for all variables defined inside functions HOT 2
- git-sync Multiple Vulnerabilities HOT 3
- how can put specific html folder from my git repo to nginx pod? HOT 2
- git-sync produced an intermediate file,the path of command is wrong HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from git-sync.