puppetfile install does not respect the :path module option about r10k HOT 28 CLOSED

+1. Does r10k rely on the eventual modulepath being a working tree, or could a repo be checked out and a subdir be copied?

from r10k.

It would appear that it does. I can use tags successfully and there is repo metadata in the resulting moduledir. Seems a very similar use case to path.

from r10k.

Not sure if I misread, but what's the solution to this? This is critical. Edit: I'm just using normal root-level modules for now.

from r10k.

+1 on this. Any workaround?

from r10k.

+1. Essentially r10k seems to work ONLY with a repo-per-module structure.

from r10k.

Adding a +1 here. We had researched before trying to implement and docs seemed to indicate path was supported. We can't use r10k until this works. (We are starting with a monolithic repo, and plan to migrate over the next year or so, but want to start using r10k/Puppetfile today to manage external dependencies.

from r10k.

This is nontrivial for r10k in conjunction with Git based modules because Git cannot directly check out subdirectories. It's possible to do sparse checkouts in a repository but that leaves the full path checked out.

I can see two solutions for this.

First, when :path is specified for a module, then r10k can rip apart the git repository and have a non-git copy checked out. This will make it much more difficult to create the module without blowing away the existing module, and will require a major reworking of the existing module logic to determine if a full clone should be used or if a non-git subdirectory should be used. In addition this approach makes it extremely hard to determine if an update needs to be applied for a module since the history has been stripped; resolving this would probably only be doable by recreating that directory every time or tracking other metadata. It would also be impossible to determine if someone had made local changes to the checked out files.

In addition people use r10k to generate a copy of modules within an environment for use in development, by running r10k puppetfile install within an environment. This clashes horribly with the subdirectory stripping approach laid out above.

The other option is to use the git-subtree command to synthesize a git repository for the given subdirectory. This is a CPU intensive operation, but allows a full git repository to be generated and checked out in the module directory.

So in short, here are the options I see:

1. Stop using git to check out working copies, and rewrite the module on every deploy
2. Selectively use git for modules without :path, and fall back to the previous case for modules using :path.
3. Use git-subtree to synthesize subdirectory checkouts.

Thoughts?

from r10k.

+1. I need this to support a separate puppet-master for Windows platform.

Since puppet-master exists only on Linux/Unix platforms, I have to borrow some code from Linux repository to manage puppet-master itself, so I want to extract only 'site' modules and not other sub-trees, which cause conflicts.

from r10k.

@adrienthebo you could use git archive to archive a subdirectory of a git repo and then pipe that to tar. For example: git --git-dir=/path/to/repo/.git archive HEAD path/to/module|tar xv

The tar options to extract that directory to the root depends on tar implementation AFAIK. But perhaps that part can be done using minitar or something instead.

from r10k.

@dalen there are definitely a number of ways to get a subdirectory, but all of them make it much more challenging to determine the state of the module afterwards. Since it's not a Git repo you can't check to see what HEAD is, so either r10k needs to manually store what version it checked out and not worry about people modifying files in place, or read that directory into an index and check to see what SHA the resulting tree has. It's definitely a solvable problem, but it's not a problem that can be solved easily without sacrificing functionality or adding complexity.

from r10k.

@adrienthebo well, another possibility is to checkout the full git repo somewhere outside the modulepath, and then just symlink in the specified :path into the modulepath as the module. Then you can check which checked out version it is and update using regular git commands. It does add complexity though, but IMHO it would be a very useful feature.

from r10k.

@dalen that's not too shabby of an idea. The thing that I don't really love is that if we just do this for modules with :path specified then we're inconsistent, and if we do the separate clone/symlink for everything that seem strange. The latter option seems better though. Where do you think the actual clone location should live?

from r10k.

Not sure. /var/lib/r10k/modules/ or something like that I suppose.

On Fri, May 8, 2015, 20:24 Adrien Thebo [email protected] wrote:

@dalen https://github.com/dalen that's not too shabby of an idea. The
thing that I don't really love is that if we just do this for modules with
:path specified then we're inconsistent, and if we do the separate
clone/symlink for everything that seem strange. The latter option seems
better though. Where do you think the actual clone location should live?

—
Reply to this email directly or view it on GitHub
#47 (comment).

from r10k.

For some time yet I'm going to be stuck with a repository containing many modules. I'd really like to see r10k support this.

• I like the idea of keeping a single clone of the remote repository to avoid having network traffic downloading it for each module.
• The symlink approach sounds OK so long as what's checked out is the same. E.g. no variations in branches, tags, refs, or whatever. But maybe that's not going to be the case.
• If r10k (when used with :path) copies all the module code out of the repo and copies over local changes, that's fine by me. In fact it would probably be politically useful for getting any module that's being worked on into its own repo. Supporting a monolithic repository is about smoothing the transition from a legacy situation.

from r10k.

+1 for the symlink approach.

from r10k.

+1 We have many modules which are linked together. It would be nice to have this feature to avoid having many git repo

from r10k.

No movement on this, was an alternative approach reached?

from r10k.

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

from r10k.

unstale

from r10k.

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

from r10k.

bad bot!!

from r10k.

what about turning off this bot or at least increase the timeout. comming back every 60days just to write "non-stale" into an issue is not a good UX. this issues wouldn't magically be fixed just because of 60days have passed.

from r10k.

Am I understanding correctly that the desired feature is a :path option that can be configured for git modules and specifies the directory from the git repo that you want to install, as opposed to the entire repo? Would this always be a single directory or would it need to support multiple directories?

brainstorming, so this might be a terrible idea, but would it be possible to clone the whole repo, make a commit to remove everything but the desired path, and then rebase after every fetch?? there would de be some gnarly merge conflicts, so maybe that's not great..

from r10k.

@mwaggett Only single directories are required for any given module, but if (as is likely) multiple modules are being installed from sub-directories of a single repository, then for efficiency you'd want to check the repo out once.

I'm not sure what the benefit is that you see in cloning and manipulating repositories, but it sounds inefficient. I prefer @dalen 's symlinking idea.

from r10k.

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

from r10k.

from r10k.

Generally speaking, when people want a monorepo, they store the modules in the control repo. Does that approach not work for this case?

from r10k.

Hi @lavaman, thanks for opening this! r10k is now considered to be feature complete and we are unfortunately discontinuing development. As a result, we are closing all open issues and pull requests. I sincerely apologize that we’re unable to address this issue.

from r10k.