Comments (16)
My bad, that actually is #38.
However, it has been merged, so you should be able to pull the new version of the image.
from promgen.
Ah, this looks like partly a bug with my sample files and partly a bug with my poor documentation :(
If you look at the example Prometheus configuration file there is this block
scrape_configs:
- job_name: 'promgen'
file_sd_configs:
- files:
- "/etc/prometheus/promgen.json"
relabel_configs:
- source_labels: [__shard]
regex: docker-demo
action: keep
The regex for "docker-demo" needs to match the name of the shard which actually defaults to "Default". There are a few options
- Change the config to "regex: Default"
- Change the shard name from Promgen to "docker-demo"
- Delete the "relabel_configs" block under scrape_configs
I'll see if I can fix up the example files and documentation. Sorry for the confusion.
from promgen.
Thanks for the quick reply. I went into the prometheus docker and checked whether promgen writes anything to /etc/prometheus/promgen.json and that file seems to be missing. Maybe I made some mistake with the database configurations and the information isn't being fetched by the worker?
EDIT: my redis password wasnt set, maybe that was it (no auth required though), testing
from promgen.
I imagine if it was a database config issue, then there would be a more visible error. If you run the cli, does anything get output? (about to head home for the day so I don't recall the docker command exactly)
promgen targets # Print out Prometheus targets
promgen rules #print out rules
from promgen.
docker exec -it worker /bin/bash
> promgen targets
gives me breadcrumbs and
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 274, in get_new_connection
conn = Database.connect(**conn_params)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/__init__.py", line 86, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2005, "Unknown MySQL server host 'mysql' (-2)")
Ill look into the database configuration. The web docker definitely has a mysql connection, so this is wierd to me.
EDIT: my bad, the worker is supposed to have the link to mysql and it didnt
from promgen.
updated the docker file to include the mysql link for the worker
ubuntu@ip-172-31-38-136:~/promgen$ docker exec -it worker /bin/bash
bash-4.3$ promgen targets
[
{
"labels": {
"__farm_source": "promgen",
"__shard": "Default",
"farm": "Farm",
"job": "node",
"project": "Project",
"service": "Default"
},
"targets": [
"nodeexporter:9100"
]
}
]
bash-4.3$ promgen rules
Traceback (most recent call last):
File "/usr/local/bin/promgen", line 11, in <module>
load_entry_point('Promgen', 'console_scripts', 'promgen')()
File "/usr/src/app/promgen/manage.py", line 20, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/usr/src/app/promgen/management/commands/rules.py", line 35, in handle
self.stdout.write(prometheus.render_rules(version=kwargs['version']))
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 107, in write
if ending and not msg.endswith(ending):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str
Thanks for the help so far, I managed to commit prometheus targets successfully:
Endpoint | State | Labels | Last Scrape | Error
-- | -- | -- | -- | --
http://nodeexporter:9100/metrics | UP | farm="Farm" instance="nodeexporter:9100"project="Project" service="Default" | 2.393s ago
I still have issues with defining alert rules (internal server errors). And the promgen rules command returns the exception listed above. Any idea where the problem might be?
from promgen.
Your problems with generating the rules file may be related to #39.
from promgen.
Thanks for the suggestion, I applied the changeset in that patch, yet it does not seem to help.
I added logging to file in the settings.py (if sentry isnt configured) and got these errors in the log:
The main thing that seems to be the issue is:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/promtool'
Somehow the promtool executable does not get executed when building the image.
from promgen.
Ye, I just noticed that in a recent commit, thanks, that seems to be the issue :) Ill just work with the 0.25 release from now on, sorry about that. Thanks nevertheless.
EDIT: this bug seems to be in the 0.25 release aswell, gotta go back to the latest
from promgen.
I got to the point where I actually see rules rendered in the UI and can add/edit them etc. However it seems prometheus does not register an update. When i look at the promgen.rule file in the prometheus docker:
groups:
- name: Default » Default » Project
rules:
- alert: UAAAAAAAAAAAAAAAAAAA
annotations:
rule: http://example.com/rule/1/edit
expr: node_load1 > 2
for: 30s
labels:
project: Project
It does display the new rule file, however only when I update it manually (i have to specify the path) via:
docker exec web promgen rules --reload /etc/prometheus/promgen.rule
I checked the promgen.yml config file (the rule file points to /etc/prometheus/promgen.rule) so Im not sure whats wrong at this point. The functionality is there, but somehow some configuration is messing something somewhere up.
when I render the rules without a specific path, the function crashes (because writing to stdout):
$ docker exec web promgen rules
Traceback (most recent call last):
File "/usr/local/bin/promgen", line 11, in <module>
load_entry_point('Promgen', 'console_scripts', 'promgen')()
File "/usr/src/app/promgen/manage.py", line 20, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/usr/src/app/promgen/management/commands/rules.py", line 35, in handle
self.stdout.write(prometheus.render_rules(version=kwargs['version']))
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 107, in write
if ending and not msg.endswith(ending):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str
Note: I patched my repository with #39
from promgen.
This seems like a similar error as #39: The return of render_rules
is treated as str
, although it is of type bytes
. You could patch line 35 in promgen/management/commands/rules.py
to convert the returned value.
I don't understand what went wrong yet. If the rules files is correctly written but the rules are not showing up in prometheus, check if the prometheus configuration points to the correct rules file.
from promgen.
I don't understand what went wrong yet. If the rules files is correctly written but the rules are not showing up in prometheus, check if the prometheus configuration points to the correct rules file.
Nono, maybe I wasnt clear:
It does display the new rule file, however only when I update it manually (i have to specify the path) via:
It updates the rule file only when i update it manually.
I found a workaround for now at least, when i removed the CELERY_BROKER_URL: redis://redis:6379/0
variable to let promgen to fallback to not using redis it started working.
from promgen.
Thanks for specifying.
You didn't post how you bootstrap your setup, but you may be missing some worker queue configuration:
The worker must subscribe to a queue (-l
flag) with the hostname of the prometheus instance you registered with promgen register …
.
from promgen.
@Tommassino @seoester thank you a lot for your help so far trying to figure things out. Looks like I need to do some work on the installation instructions to make things more clear ^^;
from promgen.
@Tommassino I made some changes in #44 which I think will explain things a bit better and work a bit better. Can you take a look and tell me what you think? This should also take into account the changes @seoester did to include promtool into the same image.
from promgen.
Hey, I consider this issue closed.
from promgen.
Related Issues (20)
- When do prometheus alert set via promgen, It will occur 'reload' action? HOT 1
- run docker run --rm -it -v ~/.config/promgen:/etc/promgen/ line/promgen bootstrap error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte
- please link the document in the README.md
- promgen web worker bad request HOT 2
- it can not work well HOT 3
- Add feature to allow slack notification to use proxy
- Docker-compose.yml TypeError HOT 1
- [QUESTION] Prometheus Parameter HOT 3
- docker-compose setup results in broken UI HOT 3
- Not work well
- [QUESTION] Promgen ALLOWED_HOSTS HOT 1
- Can I add a host with domain name? HOT 1
- How to configure farm hosts when my application deploy in k8s cluster? HOT 2
- Errors on worker HOT 1
- Change field type of url field HOT 1
- Prometheus rules seems not generating correctly. HOT 2
- [Question]How can I receive an alert message?
- arm64 or armhf version HOT 5
- Documentation is not updated since 2021
- Some page loading is slow (docker compose) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from promgen.