Comments (14)
TheKangaroo
commented on August 30, 2024
1
@xakaitetoia
We'll still use the workaround I briefly mentioned in my last post.
We configure the HelmRepositories as follows:
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: myhelmrepo
namespace: flux-system
spec:
interval: 1m
url: https://token:${helm_pull_token}@gitlab.example.com/api/v4/projects/1234/packages/helm/stableWe run a script in our pipeline that recursively replaces the helm_pull_token in all files in a given subdirectory with an actual token created at runtime:
#!/bin/bash
folder_path="$1"
replace_variable_in_file() {
file_path="$1"
envsubst '$helm_pull_token' < "$file_path" > temp.txt
mv temp.txt "$file_path"
}
search_files_recursive() {
local current_folder="$1"
for file in "$current_folder"/*; do
if [ -f "$file" ]; then
replace_variable_in_file "$file"
elif [ -d "$file" ]; then
search_files_recursive "$file"
fi
done
}
search_files_recursive "$folder_path"Not a pretty solution, but it gets the job done.
from flux-local.
allenporter
commented on August 30, 2024
Hi, I'm not sure that error is related to the private secret. It seems more like it's just having a problem managing helm local cache directory? Perhaps you could run with --log-level=DEBUG (before other params) and pull out any relevant helm lines? It's supposed to make all the needed directories for helm.
from flux-local.
allenporter
commented on August 30, 2024
Oh I see, it can't pull the helm chart from the repo, you are right. I misunderstood thinking we were talking about the release, not repo.
from flux-local.
allenporter
commented on August 30, 2024
Do you have an example of how to use helm cli with secrets like this? Are there fields in the repository config or something else?
from flux-local.
TheKangaroo
commented on August 30, 2024
Hi @allenporter, sorry for the late response.
I build a minimal setup with just one Kustomization with a HelmRepo and HelmRelease from the private HelmChart.
These are the complete logs:
$ flux-local --log-level=DEBUG build --enable-helm --skip-crds .
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:flux_local.git_repo:Processing cluster with selector ResourceSelector(path=PathSelector(path=PosixPath('.'), sources=None), cluster=MetadataSelector(enabled=True, name='flux-system', namespace='flux-system', skip_crds=True, skip_secrets=True, visitor=None), kustomization=MetadataSelector(enabled=True, name=None, namespace=None, skip_crds=True, skip_secrets=True, visitor=ResourceVisitor(func=<bound method ContentOutput.call_async of <flux_local.tool.visitor.ContentOutput object at 0x1041fd290>>)), helm_repo=MetadataSelector(enabled=True, name=None, namespace=None, skip_crds=True, skip_secrets=True, visitor=ResourceVisitor(func=<function HelmVisitor.repo_visitor.<locals>.add_repo at 0x105215e40>)), helm_release=MetadataSelector(enabled=True, name=None, namespace=None, skip_crds=True, skip_secrets=True, visitor=ResourceVisitor(func=<function HelmVisitor.release_visitor.<locals>.add_release at 0x105216020>)), cluster_policy=MetadataSelector(enabled=True, name=None, namespace=None, skip_crds=True, skip_secrets=True, visitor=None))
DEBUG:git.util:Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')
DEBUG:git.cmd:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/private/tmp/flux, universal_newlines=False, shell=None, istream=None)
DEBUG:flux_local.command:Running command: (/private/tmp/flux) kustomize cfg grep kind=Kustomization .
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.git_repo:roots=[Kustomization(name='demo', namespace='flux-system', path='./base', helm_repos=[], helm_releases=[], cluster_policies=[], source_path='kustomization.yaml', source_kind='GitRepository', source_name='flux-system', source_namespace='flux-system')]
DEBUG:flux_local.git_repo:No clusters found; Processing as a Kustomization: .
DEBUG:flux_local.git_repo:Building cluster cluster .
DEBUG:flux_local.git_repo:Visiting path (.) .
DEBUG:flux_local.command:Running command: (/private/tmp/flux) kustomize cfg grep kind=Kustomization .
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.git_repo:Found 1 Kustomizations (1 unique)
DEBUG:flux_local.git_repo:Kustomization 'demo' sourceRef.kind 'GitRepository' of 'flux-system'
DEBUG:flux_local.git_repo:Visiting path (.) base
DEBUG:flux_local.command:Running command: (/private/tmp/flux/base) kustomize cfg grep kind=Kustomization .
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.git_repo:Found 0 Kustomizations (0 unique)
DEBUG:flux_local.git_repo:Processing kustomization 'demo': base
DEBUG:flux_local.command:Running command: (/private/tmp/flux/base) kustomize build --load-restrictor=LoadRestrictionsNone
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'kind=^(CustomResourceDefinition|Secret)$' --invert-match
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'kind=^(HelmRepository|HelmRelease|ClusterPolicy)$'
DEBUG:flux_local.tool.visitor:Waiting for cluster inflation to complete
DEBUG:flux_local.tool.visitor:Inflating Helm charts in cluster .
DEBUG:flux_local.helm:Updating 1 repositories
DEBUG:flux_local.command:Running command: (None) helm repo update --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml
DEBUG:flux_local.tool.visitor:Waiting for tasks to inflate .
DEBUG:flux_local.command:Running command: (None) helm template demo flux-system-demo/demo --namespace flux-system --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml
ERROR:flux_local.command:Command '(None) helm template demo flux-system-demo/demo --namespace flux-system --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml' failed with return code 1
Error: no cached repo found. (try 'helm repo update'): open /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w/flux-system-demo-index.yaml: no such file or directory
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/flux_local.py", line 60, in main
asyncio.run(action.run(**vars(args)))
File "/opt/homebrew/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/build.py", line 94, in run
await helm_visitor.inflate(
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/visitor.py", line 296, in inflate
await asyncio.gather(*tasks)
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/visitor.py", line 324, in inflate_cluster
await asyncio.gather(*tasks)
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/visitor.py", line 221, in inflate_release
await visitor.func(cluster_path, pathlib.Path(""), release, cmd)
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/visitor.py", line 135, in call_async
content = await cmd.run()
^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/kustomize.py", line 114, in run
return await run_piped(self._cmds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/command.py", line 104, in run_piped
result = await _run_piped_with_sem(cmds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/command.py", line 96, in _run_piped_with_sem
out = await asyncio.wait_for(cmd.run(stdin), _TIMEOUT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 479, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/flux_local/command.py", line 87, in run
raise self.exc("\n".join(errors))
flux_local.exceptions.HelmException: Command '(None) helm template demo flux-system-demo/demo --namespace flux-system --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml' failed with return code 1
Error: no cached repo found. (try 'helm repo update'): open /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w/flux-system-demo-index.yaml: no such file or directory
flux-local error: Command '(None) helm template demo flux-system-demo/demo --namespace flux-system --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml' failed with return code 1
Error: no cached repo found. (try 'helm repo update'): open /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w/flux-system-demo-index.yaml: no such file or directory
You can reproduce the same behavior with helm when adding the repo without a password:
$ helm repo add myrepo https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable
Error: looks like "https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable" is not a valid chart repository or cannot be reached: failed to fetch https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable/index.yaml : 401 Unauthorized
The same command succeeds with username and password:
$ helm repo add myrepo https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable --username token --password <token>
"myrepo" has been added to your repositories
Interestingly, helm already fails when adding the repository. flux-local only terminates with an error when I also add a helm release.
from flux-local.
TheKangaroo
commented on August 30, 2024
@allenporter is there something I can further assist with?
from flux-local.
allenporter
commented on August 30, 2024
Thanks for the extra detail -- sorry this fell off my radar. The way this works now in flux local is that instead of adding the repo one at a time, it makes the repository config file:
Line 75 in b15d571
help template command Line 205 in b15d571
--password and a --username flag so that looks like it can be used.
I think the steps we need are:
- when parsing the
HelmRepositorytrack that it uses a secret hereas fieldsflux-local/flux_local/manifest.py
Line 180 in b15d571
- when creating the
helm templatecommand pass the secret information as flags object hereLine 205 in b15d571
I realize though we need a way to pass a secret for that repository... Maybe another command line flag similar to --sources, or maybe an enviroment variable, or maybe it can find it from another object in the cluster (e.g. you create a fake Secret yaml file while running under test. There is prior art for something like this in real CI systems... Curious if you have an opinion or know best practices here.
from flux-local.
TheKangaroo
commented on August 30, 2024
@allenporter Thanks for the explanation of what happens in the code. I'm going to take some time to get my head around the options here.
But before I found out that instead of
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
[...]
url: https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable
secretRef:
name: helm-pull-token
I can also pass the token via the URL and postBuild variable substitution like this
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
[...]
url: https://token:<token>@gitlab.example.com/api/v4/projects/1234/packages/helm/stable
However, I do not think variable substitution is supported by flux-local atm, but perhaps this is an easier approach to implement.
from flux-local.
TheKangaroo
commented on August 30, 2024
I haven't come up with a perfect solution yet, but I've managed to get it to work with variables substitution before running flux-local.
First of all, I think the easiest way would be to use private OCI charts instead of helm chart artefacts, as you could simply run helm registry login before running flux-local. Unfortunately there is no such thing as helm chart login.
I'm still not sure how we should pass credentials to flux-local, I think the best way would be to pass username and password to the helm repo add command based on the secretRef in the HelmRelease resource. This would require inventing some sort of "virtual secret" to pass these credentials.
As for the environment variable part, I'm not sure about the naming of the variables. In my case, only one token is needed for each chart, but I think a solution should support different tokens per helm repository. This would require environment variables like HELM_REPO_<helm-repo-name>_USERNAME and HELM_REPO_<helm-repo-name>_PASSWORD.
In this case, we could check for a matching set of environment variables for the current helm repository and pass the --username and --password parameters to the helm command.
from flux-local.
allenporter
commented on August 30, 2024
Maybe we can add a flag where secretes are passed in as key/value pairs, then referenced as needed.
from flux-local.
xakaitetoia
commented on August 30, 2024
Hello all,
Was wondering if there was an update on this issue in general, or maybe there is another way to skip specific helmreleases/apps that require helm registry login with something like --ignore ?
from flux-local.
allenporter
commented on August 30, 2024
In #717 it was proposed to add a --repository-config which is now available 5.4.0. Perhaps this will let you specify authentication for a private repository?
from flux-local.
TheKangaroo
commented on August 30, 2024
I think this should work, although we'll be moving to OCI charts in the near future, so we'll end up with the exact same setup that was fixed in #717. We can close this issue from my point of view.
from flux-local.
allenporter
commented on August 30, 2024
Thanks!
from flux-local.
Related Issues (20)
- Provide a way to parse out images from rendered manifests and Helm Releases HOT 7
- Cache pip in github action HOT 1
- Provide a way to output results (`--output-file`) to a file
- Running with relative path outside of git repo fails
- Move away from deprecated set-output commands in actions
- Required fields in Helm templates causes templating of helm charts to not work HOT 4
- Concurrency issues with flux build without kustomization.yaml HOT 8
- Failing to find HelmRepository HOT 13
- Validate dependsOn rules HOT 2
- Error when starting the application HOT 5
- Support GitRepository for HelmRelease sources HOT 17
- HelmRelease Error: template due to substituteFrom not being supported HOT 11
- Github actions diff - Argument list too long HOT 1
- Test does not work with OCI repositories HOT 4
- No diff when comparing two versions of helm charts HOT 4
- Error with diff in github actions HOT 10
- [v5.3.0] GitHub Actions fail to install requirements HOT 4
- User report: Replace: id matched 2 resources HOT 3
- Private OCI chart repositories not working HOT 4
- Placeholder raises ConstructorError HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flux-local.