promgen compose about promgen HOT 16 CLOSED

My bad, that actually is #38.

However, it has been merged, so you should be able to pull the new version of the image.

from promgen.

Ah, this looks like partly a bug with my sample files and partly a bug with my poor documentation :(

If you look at the example Prometheus configuration file there is this block

scrape_configs:
  - job_name: 'promgen'
    file_sd_configs:
      - files:
        - "/etc/prometheus/promgen.json"
    relabel_configs:
    - source_labels: [__shard]
      regex: docker-demo
      action: keep

The regex for "docker-demo" needs to match the name of the shard which actually defaults to "Default". There are a few options

1. Change the config to "regex: Default"
2. Change the shard name from Promgen to "docker-demo"
3. Delete the "relabel_configs" block under scrape_configs

I'll see if I can fix up the example files and documentation. Sorry for the confusion.

from promgen.

Thanks for the quick reply. I went into the prometheus docker and checked whether promgen writes anything to /etc/prometheus/promgen.json and that file seems to be missing. Maybe I made some mistake with the database configurations and the information isn't being fetched by the worker?

EDIT: my redis password wasnt set, maybe that was it (no auth required though), testing

from promgen.

I imagine if it was a database config issue, then there would be a more visible error. If you run the cli, does anything get output? (about to head home for the day so I don't recall the docker command exactly)

promgen targets # Print out Prometheus targets
promgen rules  #print out rules

from promgen.

docker exec -it worker /bin/bash

> promgen targets

gives me breadcrumbs and

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 274, in get_new_connection
    conn = Database.connect(**conn_params)
  File "/usr/local/lib/python3.5/site-packages/MySQLdb/__init__.py", line 86, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2005, "Unknown MySQL server host 'mysql' (-2)")

Ill look into the database configuration. The web docker definitely has a mysql connection, so this is wierd to me.

EDIT: my bad, the worker is supposed to have the link to mysql and it didnt

from promgen.

updated the docker file to include the mysql link for the worker

ubuntu@ip-172-31-38-136:~/promgen$ docker exec -it worker /bin/bash
bash-4.3$ promgen targets
[
  {
    "labels": {
      "__farm_source": "promgen",
      "__shard": "Default",
      "farm": "Farm",
      "job": "node",
      "project": "Project",
      "service": "Default"
    },
    "targets": [
      "nodeexporter:9100"
    ]
  }
]
bash-4.3$ promgen rules
Traceback (most recent call last):
  File "/usr/local/bin/promgen", line 11, in <module>
    load_entry_point('Promgen', 'console_scripts', 'promgen')()
  File "/usr/src/app/promgen/manage.py", line 20, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/src/app/promgen/management/commands/rules.py", line 35, in handle
    self.stdout.write(prometheus.render_rules(version=kwargs['version']))
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 107, in write
    if ending and not msg.endswith(ending):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

Thanks for the help so far, I managed to commit prometheus targets successfully:

Endpoint | State | Labels | Last Scrape | Error
-- | -- | -- | -- | --
http://nodeexporter:9100/metrics | UP | farm="Farm" instance="nodeexporter:9100"project="Project" service="Default" | 2.393s ago

I still have issues with defining alert rules (internal server errors). And the promgen rules command returns the exception listed above. Any idea where the problem might be?

from promgen.

Your problems with generating the rules file may be related to #39.

from promgen.

Thanks for the suggestion, I applied the changeset in that patch, yet it does not seem to help.

I added logging to file in the settings.py (if sentry isnt configured) and got these errors in the log:

debug.log

The main thing that seems to be the issue is:

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/promtool'

Somehow the promtool executable does not get executed when building the image.

from promgen.

Ye, I just noticed that in a recent commit, thanks, that seems to be the issue :) Ill just work with the 0.25 release from now on, sorry about that. Thanks nevertheless.

EDIT: this bug seems to be in the 0.25 release aswell, gotta go back to the latest

from promgen.

I got to the point where I actually see rules rendered in the UI and can add/edit them etc. However it seems prometheus does not register an update. When i look at the promgen.rule file in the prometheus docker:

groups:
- name: Default » Default » Project
  rules:
  - alert: UAAAAAAAAAAAAAAAAAAA
    annotations:
      rule: http://example.com/rule/1/edit
    expr: node_load1 > 2
    for: 30s
    labels:
      project: Project

It does display the new rule file, however only when I update it manually (i have to specify the path) via:

docker exec web promgen rules --reload /etc/prometheus/promgen.rule

I checked the promgen.yml config file (the rule file points to /etc/prometheus/promgen.rule) so Im not sure whats wrong at this point. The functionality is there, but somehow some configuration is messing something somewhere up.

when I render the rules without a specific path, the function crashes (because writing to stdout):

$ docker exec web promgen rules
Traceback (most recent call last):
  File "/usr/local/bin/promgen", line 11, in <module>
    load_entry_point('Promgen', 'console_scripts', 'promgen')()
  File "/usr/src/app/promgen/manage.py", line 20, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/src/app/promgen/management/commands/rules.py", line 35, in handle
    self.stdout.write(prometheus.render_rules(version=kwargs['version']))
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 107, in write
    if ending and not msg.endswith(ending):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

Note: I patched my repository with #39

from promgen.

This seems like a similar error as #39: The return of render_rules is treated as str, although it is of type bytes. You could patch line 35 in promgen/management/commands/rules.py to convert the returned value.

I don't understand what went wrong yet. If the rules files is correctly written but the rules are not showing up in prometheus, check if the prometheus configuration points to the correct rules file.

from promgen.

I don't understand what went wrong yet. If the rules files is correctly written but the rules are not showing up in prometheus, check if the prometheus configuration points to the correct rules file.

Nono, maybe I wasnt clear:

It does display the new rule file, however only when I update it manually (i have to specify the path) via:

It updates the rule file only when i update it manually.

I found a workaround for now at least, when i removed the CELERY_BROKER_URL: redis://redis:6379/0 variable to let promgen to fallback to not using redis it started working.

from promgen.

Thanks for specifying.

You didn't post how you bootstrap your setup, but you may be missing some worker queue configuration:

The worker must subscribe to a queue (-l flag) with the hostname of the prometheus instance you registered with promgen register ….

from promgen.

@Tommassino @seoester thank you a lot for your help so far trying to figure things out. Looks like I need to do some work on the installation instructions to make things more clear ^^;

from promgen.

@Tommassino I made some changes in #44 which I think will explain things a bit better and work a bit better. Can you take a look and tell me what you think? This should also take into account the changes @seoester did to include promtool into the same image.

from promgen.

Hey, I consider this issue closed.

from promgen.