Comments (11)
manugarg
commented on June 13, 2024
1
You should be able to workaround this by using the flag: --cloud_metadata=none.
from cloudprober.
manugarg
commented on June 13, 2024
1
I think we can go by this list:
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#gke_mds
I'll send a fix. Thanks again for reporting this.
from cloudprober.
tbuchier
commented on June 13, 2024
Edit Closing as it's similar to #554
from cloudprober.
manugarg
commented on June 13, 2024
@tbuchier Were you able to find a workaround for this?
Re-opening this issue as even though it is similar to #554, it is in different path.
from cloudprober.
tbuchier
commented on June 13, 2024
Hello,
I tried to disable the cloudlogging in case it came from the stackdriver logger setup but still same issue.
I'll have a better look next week :)
from cloudprober.
manugarg
commented on June 13, 2024
I'll implement a proper for this:
- Detect if running on GKE. If running on GKE, don't try to get the instance name when getting cloud metadata.
from cloudprober.
manugarg
commented on June 13, 2024
This should now be fixed, if you use the docker image built from the head (:master).
from cloudprober.
tbuchier
commented on June 13, 2024
Hello,
It seems that it's still try to access instance metadata (ip)
Using master image i'm now getting:
I0714 05:01:24.382697 1 logger.go:152] [cloudprober.sysvars] Running on GCE. Logs for cloudprober.sysvars will go to Cloud (Stackdriver).
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa55197]
goroutine 1 [running]:
cloud.google.com/go/logging.(*Logger).Log(0x0, 0x0, 0x0, 0x0, 0xc8, 0x1265c00, 0xc0000b4250, 0x0, 0x0, 0x0, ...)
/home/runner/go/pkg/mod/cloud.google.com/go/[email protected]/logging.go:723 +0x37
github.com/google/cloudprober/logger.(*Logger).log(0xc0004f13b0, 0xc8, 0xc0006bfb40, 0x1, 0x1)
/home/runner/work/cloudprober/cloudprober/logger/logger.go:245 +0x138
github.com/google/cloudprober/logger.(*Logger).Infof(0xc0004f13b0, 0x15ad4c6, 0x3f, 0xc0006bfc18, 0x1, 0x1)
/home/runner/work/cloudprober/cloudprober/logger/logger.go:302 +0xa9
github.com/google/cloudprober/logger.(*Logger).EnableStackdriverLogging(0xc0004f13b0, 0x1779d08, 0xc00003e030, 0xc0006bfcc0, 0x1)
/home/runner/work/cloudprober/cloudprober/logger/logger.go:202 +0x4e5
github.com/google/cloudprober/logger.New(0x1779d08, 0xc00003e030, 0xc000132048, 0x13, 0x1, 0x1573baf, 0x7)
/home/runner/work/cloudprober/cloudprober/logger/logger.go:153 +0x185
github.com/google/cloudprober/logger.NewCloudproberLog(0x1573baf, 0x7, 0xc00005c800, 0x0, 0xc0006bfd90)
/home/runner/work/cloudprober/cloudprober/logger/logger.go:137 +0xd8
github.com/google/cloudprober.InitFromConfig(0xc0000da700, 0x645, 0x0, 0x0)
/home/runner/work/cloudprober/cloudprober/cloudprober.go:139 +0xb4
main.main()
/home/runner/work/cloudprober/cloudprober/cmd/cloudprober.go:180 +0x7aa
I re tried with --disable_cloud_logging and now getting:
F0714 05:05:13.685537 1 cloudprober.go:182] Error initializing cloudprober. Err: sysvars_gce: error while getting internal_ip from metadata: metadata: GCE metadata "instance/network-interfaces/0/ip" not defined
from cloudprober.
manugarg
commented on June 13, 2024
I see what's going on with the first error. We check for "l.logc" being before logging remotely, while "l.logger" is used to log remotely. With the recent change, we created a situation where we are trying to log when l.logger is nil while l.logc is not. Its fix is very simple. I've a PR for it: #625. Thank you so much for trying the "master" docker image and reporting this issue.
Regarding the second error, it seems that more metadata variables are not available on some GKE setups. Which metadata variables are available and which are not is not really documented well. Do you happen to know if there is anything special about your GKE setup? Is metadata access completely disabled?
from cloudprober.
manugarg
commented on June 13, 2024
It should all be fixed now. @tbuchier, Can you please give it another try? Thanks in advance and sorry for the trouble.
from cloudprober.
tbuchier
commented on June 13, 2024
Thanks a lot, it does work well now :)
At first I was getting some logging permission errors:
2021/07/16 12:33:15 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/07/16 12:33:16 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/07/16 12:33:51 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/07/16 12:33:56 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
...
The gke logging is set to system only that's probably why so I just added the disable_cloud_logging flag
from cloudprober.
Related Issues (20)
- Running On Kubernetes pod encounter an error on OpenStack HOT 6
- Support allowed_metrics_regex in Prometheus surfacer HOT 1
- Unable to connect gRPC for dynamic configuration of cloudprober
- Support specifying prober interval / timeouts as durations HOT 1
- RDS Kubernetes Endpoints include pod name HOT 5
- HTTP probe with `file_targets` can begin first iteration with 0 targets HOT 1
- Default behaviour of RDS Filter and Probing multiple matched services HOT 4
- Datadog surfacer makes cloudprober binary too big HOT 2
- [Documentation] Document file surfacer output HOT 1
- Reduce resource consumption when using file discovery with same file in multiple probes HOT 23
- Document templating language HOT 5
- Implement caching option in RDS protocol HOT 1
- Allow debugging configuration templates, and playbook HOT 3
- file_targets for probe options HOT 5
- Metrics not updated on external probe timeout HOT 6
- `additional_label`'s are missing in custom metrics of `EXTERNAL` probe HOT 3
- Socket: permission denied when run in Kubernetes HOT 2
- Failed to publish metrics to cloudwatch: MissingRegion: could not find region configuration HOT 3
- Improve documentation for cloudwatch surfacer
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudprober.