I'm happy if mackerel-agent accepts include configuration to load additional configuration files and merge them.

include = /etc/mackerel-agent/conf.d/*

It's familiar to some automation tools such like Chef, Puppet, if user wants to new module with new metrics, User can write new mackerel configuration file for the module and put into conf.d directory.

But with current implementation, user have to implement merging multiple configuration files in their puppet, chef manifests.

Thoughts?

After #58, device-mapper metrics is no longer sent.
But some users want to this.

So, implement some option to send device-mapper metrics.

If you install mackerel-agent on a Windows 11 host, it is registered with Mackerel as a Windows 10 host.

The result of obtaining os name by Get-Computerinfo command is shown below.

PS C:\Users\arthur> Get-ComputerInfo | Select OsName, OsDisplayVersion, OsBuildNumber                                   
OsName                   OSDisplayVersion OsBuildNumber
------                   ---------------- -------------
Microsoft Windows 11 Pro 22H2             22621

osname is taken from the registry HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\ProductName value, but this implementation may not be sufficient. On a Windows 11 PC, this registry value is still "Windows 10".

I received this error twice.
Is there a way to avoid this?

I instlled mackerel-agent from binary.
Ver: 042.3
CPU: 1 x ARM926EJ-S rev 4 (v5l)
OS: GNU/Linux 3.10.37-3.10.37-klk4

Hi,
I discarded the old organization and purged the mackerel-agent package 0.73.3-1.systemd from my Debian environment. Later, when I installed for the new organization, I got an error that the id file is different.

I noticed mackerel-agent Debian package wouldn't remove /var/lib/mackerel-agent/id file even if I ran apt-get purge mackerel-agent.

Of course remove should keep configuration and associtated files, but as far as I know, purge is required to remove files automatically generated by the package, as well as the package's own files .

I think it is needed to add rm -f /var/lib/mackerel/id inpurge condition of postrm.

I'm not sure about RPM package.

In OpenWrt, mackerel-agent fails to execute df command and prints a lot of warnings.

log:

2019/06/14 10:56:01 WARNING <util.filesystem> 'df -Pkl' command exited with a non-zero status: 1: "df: unrecognized option: l\nBusyBox v1.30.0 () multi-call binary.\n\nUsage: df [-PkmhT] [FILESYSTEM]...\n\nPrint filesystem usage statistics\n\n\t-P\tPOSIX output format\n\t-k\t1024-byte blocks (default)\n\t-m\t1M-byte blocks\n\t-h\tHuman readable (e.g. 1K 243M 2G)\n\t-T\tPrint filesystem type\n"
...

BusyBox df in OpenWrt outputs df: unrecognized option: l as a warning, so it seems that it has slipped through the check in func init().

For example, metrics/darwin/filesystem.go and metrics/linux/filesystem.go are almost same.

Similar situation will be occur, so we need refactoring.

Hi, thanks great service && ecosystems.

We have used mackerel agents for many hosts, and found that some drives always raises alerts of disk full.

$ LANG=C df -P
Filesystem              1024-blocks    Used Available Capacity Mounted on
/dev/mapper/centos-root    82929664 3794152  79135512       5% /
devtmpfs                    8125124       0   8125124       0% /dev
tmpfs                       8134412       0   8134412       0% /dev/shm
tmpfs                       8134412  164124   7970288       3% /run
tmpfs                       8134412       0   8134412       0% /sys/fs/cgroup
/dev/sr0                        422     422         0     100% /mnt/config-drive
/dev/vda1                    508588  165184    343404      33% /boot

/dev/sr0 is a CD-ROM drive, so it seems to be unreasonable to warn us of disk usage.

(Especially, OpenStack distributes its metadata to instances via CD-ROM drive according to its construction settings, then in some cases, there always exists /dev/sr0 and uses 100% drive.)

So we may feel happy if some of drives are able to be ignored by its filesystem type(e.g. ign iso9660), or by its name.

Thanks!

（もし詳細がよくわからなければ、twitter等で日本語で説明させていただきます...）

実行環境

Ubuntu 16.04 64bit

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"

$ uname -a
Linux XXXXXXXXX 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

事象

aptでUpdateすると、以下のエラーが出るようになっています。

N: Skipping acquire of configured file 'contrib/binary-i386/Packages' as repository 'http://apt.mackerel.io/v2 mackerel InRelease' doesn't support architecture 'i386'

現状は放っておいても、大丈夫なのですが、気持ち悪いのと
他のパッケージでは、エラーで止まった経緯があったので修正を提案したいです。

$ sudo apt update && sudo apt upgrade -y
Hit:1 http://jp.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://jp.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:3 http://jp.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Hit:4 http://apt.mackerel.io/v2 mackerel InRelease
Get:5 http://repo.percona.com/apt xenial InRelease [16.0 kB]
Get:6 http://jp.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [856 kB]
Get:7 http://jp.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [767 kB]
Hit:8 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial InRelease
Get:9 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Get:10 http://jp.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [349 kB]
Get:11 http://jp.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [690 kB]
Get:12 http://jp.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [632 kB]
Hit:13 http://ppa.launchpad.net/ondrej/nginx-mainline/ubuntu xenial InRelease
Hit:14 http://ppa.launchpad.net/ondrej/php/ubuntu xenial InRelease
Hit:15 https://download.docker.com/linux/ubuntu xenial InRelease
Get:16 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [563 kB]
Get:17 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages [485 kB]
Get:18 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [237 kB]
Fetched 4,919 kB in 4s (1,076 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
N: Skipping acquire of configured file 'contrib/binary-i386/Packages' as repository 'http://apt.mackerel.io/v2 mackerel InRelease' doesn't support architecture 'i386'
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

解決案

case 1.

[mackerel-agent for Ubuntu]が、64bitしかサポートしていないなら

• sources.list 書き換えてしまう

--- /etc/apt/sources.list.d/mackerel.list.save  2018-07-08 18:31:30.896010812 +0900
+++ /etc/apt/sources.list.d/mackerel.list       2018-08-10 07:25:00.363902801 +0900
@@ -1 +1 @@
-deb http://apt.mackerel.io/v2/ mackerel contrib
+deb [arch=amd64] http://apt.mackerel.io/v2/ mackerel contrib

case 2.

• 32bit or 64bit で分岐してもらう

setup-all-apt-v2.sh の場合

• setup-all-apt-v2.sh

テスト済み

--- setup-all-apt-v2.sh.orig    2018-10-09 16:31:57.457751555 +0900
+++ setup-all-apt-v2.sh 2018-10-09 16:40:52.229780129 +0900
@@ -22,7 +22,11 @@
     exit 1
   fi

-  echo "deb http://apt.mackerel.io/v2/ mackerel contrib" > /etc/apt/sources.list.d/mackerel.list
+  if uname -a |grep -q x86_64 ;then
+    echo "deb [arch=amd64] http://apt.mackerel.io/v2/ mackerel contrib" > /etc/apt/sources.list.d/mackerel.list
+  else
+    echo "deb http://apt.mackerel.io/v2/ mackerel contrib" > /etc/apt/sources.list.d/mackerel.list
+  fi
   $http_get https://mackerel.io/file/cert/GPG-KEY-mackerel-v2 | apt-key add -
   apt-get update -qq

[mackerelio/ansible-mackerel-agent] の場合

• debian.yml

テストしていません

--- debian.yml.orig     2018-10-09 16:43:03.781786840 +0900
+++ debian.yml  2018-10-17 17:58:54.833730905 +0900
@@ -14,10 +14,15 @@
   when: (ansible_distribution == "Debian" and ansible_distribution_major_version|int < 8) or
         (ansible_distribution == "Ubuntu" and ansible_distribution_version|float < 16.04)

+- name: add repository 'mackerel' v2 for x86_64
+  apt_repository: repo='deb [arch=amd64] http://apt.mackerel.io/v2/ mackerel contrib' state=present update_cache=yes
+  when: (ansible_distribution == "Debian" and ansible_distribution_major_version|int >= 8 and ansible_architecture == "x86_64") or
+        (ansible_distribution == "Ubuntu" and ansible_distribution_version|float >= 16.04 and ansible_architecture == "x86_64")
+
 - name: add repository 'mackerel' v2
   apt_repository: repo='deb http://apt.mackerel.io/v2/ mackerel contrib' state=present update_cache=yes
-  when: (ansible_distribution == "Debian" and ansible_distribution_major_version|int >= 8) or
-        (ansible_distribution == "Ubuntu" and ansible_distribution_version|float >= 16.04)
+  when: (ansible_distribution == "Debian" and ansible_distribution_major_version|int >= 8 and ansible_architecture != "x86_64") or
+        (ansible_distribution == "Ubuntu" and ansible_distribution_version|float >= 16.04 and ansible_architecture != "x86_64")

 - name: install mackerel-agent
   apt:

case 3.

• apt のリポジトリ側で対応する

これができるのか、詳細なことがわかりませんが出来るのであれば。

aptのリポジトリ側で対応をお願いしたいです。

"roles" written in end of file(or after plugins) can't configure.
Only when written at the beginning(or before plugins) is valid.

$ cat /etc/mackerel-agent/mackerel-agent.conf

apikey = "apikey"
include = "/etc/mackerel-agent/conf.d/*.conf"

[plugin.metrics.linux]
command = "mackerel-plugin-linux"
[plugin.metrics.nginx]
command = "mackerel-plugin-nginx"
[plugin.metrics.multicore]
command = "mackerel-plugin-multicore"
[plugin.metrics.uwsgi-vassal]
command = "mackerel-plugin-uwsgi-vassal --socket unix:///var/run/uwsgi/stats.sock"
[plugin.metrics.accesslog]
command = "mackerel-plugin-accesslog /var/log/nginx/access.log"

roles = [ "My-Service:app", "Another-Service:db" ]

rolesがpluginの後に書かれた場合、設定が反映されません。
apikeyの下など、先頭に書かれたときのみ有効となりました。

To change username, sudo must be used as work-around like:

command="sudo -u xxxx ..."

Specifying username is as follows.

[plugin.metrics.vmstat]
command = "ruby /etc/sensu/plugins/system/vmstat-metrics.rb"
username = nobody

Tried to run mackerel-agent that installed by latest Homebrew tap on my refreshed OS X and caught these errors:

$ cd /usr/local/var/mackerel-agent; mackerel-agent -conf /usr/local/etc/mackerel-agent.conf
2015/10/05 11:10:34 main.go:84: INFO <main> Starting mackerel-agent version:0.23.0, rev:5dd63a4, apibase:https://mackerel.io
2015/10/05 11:10:34 main.go:272: CRITICAL <main> Failed to create a pidfile: open /Users/aereal/Library/mackerel-agent/pid: no such file or directory

I confirmed that the directory (~/Library/mackerel-agent) does not exist.

(Missing preparation? (such as mkdir -p conf.Root))

mackerel-agent.conf:

# pidfile = "/var/run/mackerel-agent.pid"
# root = "/var/lib/mackerel-agent"
verbose = true
apikey = "<censored>"

# Configuration for connection
# [connection]
# post_metrics_dequeue_delay_seconds = 30 # delay for dequeuing from buffer queue
# post_metrics_retry_delay_seconds = 60   # delay for retring a request that causes errors
# post_metrics_retry_max = 10             # max numbers of retries for a request that causes errors
# post_metrics_buffer_size = 30           # max numbers of requests stored in buffer queue.

# Configuration for Custm Metrics Plugins
# see also: http://help-ja.mackerel.io/entry/advanced/custom-metrics

# followings are mackerel-agent-plugins https://github.com/mackerelio/mackerel-agent-plugins

# Plugin for Apache2 mod_status
#   By default, the plugin accesses to http://127.0.0.1/server-status?auto
# [plugin.metrics.apache2]
# command = "/usr/local/bin/mackerel-plugin-apache2"

# Plugin for EC2 CPU Credit
# [plugin.metrics.aws-ec2_cpucredit]
# command = "/usr/local/bin/mackerel-plugin-aws-ec2-cpucredit"

# Plugin for AWS ELB
# [plugin.metrics.aws-elb]
# command = "/usr/local/bin/mackerel-plugin-aws-elb"

# Plugin for Amazon RDS
# [plugin.metrics.aws-rds]
# command = "/usr/local/bin/mackerel-plugin-aws-rds -identifier=<required>"

# Plugin for Elasticsearch
#   By default, the plugin accesses Elasticsearch on localhost.
# [plugin.metrics.elasticsearch]
# command = "/usr/local/bin/mackerel-plugin-elasticsearch"

# Plugin for HAProxy
#   By default, the plugin accesses Elasticsearch on localhost.
# [plugin.metrics.haproxy]
# command = "/usr/local/bin/mackerel-plugin-haproxy"

# Plugin for JVM
#   Required javaname
# [plugin.metrics.jvm]
# command = "/usr/local/bin/mackerel-plugin-jvm -javaname=<required>"

# Plugin for Linux
# [plugin.metrics.linux]
# command = "/usr/local/bin/mackerel-plugin-linux"

# Plugin for Memcached
# [plugin.metrics.memcached]
# command = "/usr/local/bin/mackerel-plugin-memcached"

# Plugin for MongoDB
# [plugin.metrics.mongodb]
# command = "/usr/local/bin/mackerel-plugin-mongodb"

# Plugin for MySQL
#   Appropriate previlege settings required.
#   By default, the plugin accesses MySQL on localhost by 'root' with no password.
# [plugin.metrics.mysql]
# command = "/usr/local/bin/mackerel-plugin-mysql"

# Plugin for Nginx
#   By default, the plugin accesses to http://localhost:8080/nginx_status
# [plugin.metrics.nginx]
# command = "/usr/local/bin/mackerel-plugin-nginx"

# Plugin for PHP APC
# [plugin.metrics.php-apc]
# command = "/usr/local/bin/mackerel-plugin-php-apc"

# Plugin for Plack
#   By default, the plugin accesses to http://localhost:5000/server-status?json
# [plugin.metrics.plack]
# command = "/usr/local/bin/mackerel-plugin-plack"

# Plugin for PostgreSQL
#   Appropriate previlege settings required.
#   By default, the plugin accesses PostgreSQL on localhost.
# [plugin.metrics.postgres]
# command = "/usr/local/bin/mackerel-plugin-postgres"

# Plugin for Redis
#   By default, the plugin accesses Redis on localhost.
#   Currently AUTH password has not been supported yet.
# [plugin.metrics.redis]
# command = "/usr/local/bin/mackerel-plugin-redis"

# Plugin for SNMP
# [plugin.metrics.pps]
# command = "/usr/local/bin/mackerel-plugin-snmp -name='pps' -community='private' '.1.3.6.1.2.1.31.1.1.1.7.2:eth01in:1:0' '.1.3.6.1.2.1.31.1.1.1.11.2:eth01out:1:0'"

# Plugin for Squid
# [plugin.metrics.squid]
# command = "/usr/local/bin/mackerel-plugin-squid"

# Plugin for Varnish
# [plugin.metrics.varnish]
# command = "/usr/local/bin/mackerel-plugin-varnish"

# Plugin for munin (wrapper)
# [plugin.metrics.nfsd]
# command = "/usr/local/bin/mackerel-plugin-munin -plugin=/usr/share/munin/plugins/nfsd"
# [plugin.metrics.bind9]
# command = "/usr/local/bin/mackerel-plugin-munin -plugin=/etc/munin/plugins/bind9 -plugin-conf-d=/etc/munin/plugin-conf.d"
# [plugin.metrics.postfix]
# command = "MUNIN_LIBDIR=/usr/share/munin /usr/local/bin/mackerel-plugin-munin -plugin=/usr/share/munin/plugins/postfix_mailqueue -name=postfix.mailqueue"

# followings are other samples
# [plugin.metrics.vmstat]
# command = "ruby /etc/sensu/plugins/system/vmstat-metrics.rb"
# [plugin.metrics.curl]
# command = "ruby /etc/sensu/plugins/http/metrics-curl.rb"

Versino info.:

$ mackerel-agent --version
mackerel-agent version 0.23.0 (rev 5dd63a4) [darwin amd64 go1.4]
$ uname -a
Darwin Hortensia.local 15.0.0 Darwin Kernel Version 15.0.0: Wed Aug 26 16:57:32 PDT 2015; root:xnu-3247.1.106~1/RELEASE_X86_64 x86_64 i386 MacBookPro10,2 Darwin
$ go version
go version go1.5.1 darwin/amd64
$ brew --version
0.9.5 (git revision 648a; last commit 2015-10-05)

Now, we run resolveConfig to do configtest and detect Invalid syntax (config_test.go shows that).
But, we can't detect unexpected key, so I want to improve by using Undecoded().
It would be better if it could suggest intended key.

This is sample of mackerel-agent.conf which includes unexpected key.

1  apikey = "abcdefg"
2  podfile = "/path/to/pidfile"
3 
4  [foo]
5  command = "bar"
6 
7  [plugins.checks.foo]
8  command = "bar"
9 
10 [plugin.check.foo]
11 command = "bar"

Actual behavior

$ mackerel-agent configtest
/path/to/mackerel-agent.conf Syntax OK

Expected behavior

$ mackerel-agent configtest
warn: podfile is invalid key. Do you mean pidfile ?(line 2)
warn: foo is invalid key (line 4)
warn: plugins.checks.foo is invalid key. Do you mean plugin.checks.foo ? (line 7)
warn: plugin.check.foo is invalid key. Do you mean plugin.checks.foo ? (line 10)

Implementation image

Create config/validate.go#ValidateConfigFile and use it in commands.go#doConfigtest

func doConfigtest(fs *flag.FlagSet, argv []string) error {
	conf, err := resolveConfig(fs, argv)
	if err != nil {
		return fmt.Errorf("failed to test config: %s", err)
	}
	warnings := config.ValidateConfigFile(conf.Conffile)
	if warnings != nil {
                 return <display warnings>
        }
        fmt.Fprintf(os.Stderr, "%s Syntax OK\n", conf.Conffile)
	return nil
}

Implementation stage

• detect unexpected key #813
• show suggestion #818

Overview

The metric host.meta.interfaces are always null on the particular host.

This problem seems to occur on the hosts satisfy the following conditions.

• not installed ip command
• installed ifconfig command
• a newer version of Linux (maybe)

Output of mackerel-agent

The following log shows mackerel-agent outputs the metrics without proper .host.meta.interfaces.
It is expected .host.meta.interfaces have valid information of the NICs of host, but it always be null.

ruins@vm-ubnt1804-Frigg:~$ cat /etc/issue
Ubuntu 18.04.3 LTS \n \l
ruins@vm-ubnt1804-Frigg:~$ uname -r
5.0.0-25-generic
ruins@vm-ubnt1804-Frigg:~$ docker run --rm -it mackerel/mackerel-agent mackerel-agent once -apikey=dummy | jq .host.meta.interfaces
null

Consideration

It seems caused by mismatched Regexp on here (https://github.com/mackerelio/mackerel-agent/blob/master/spec/linux/interface.go#L97-L100).

This Regexp expects the section of IP addresses begins with inet addr but it seems to begin with inet in the newer version of Linux.

On Debian Jessie, the section of IP address begins with inet addr.
But on Debian Stretch, it begins with inet.

ruins@thor:~$ docker run --rm -it debian:jessie-slim sh -c "apt update;apt install net-tools;ifconfig -a"
(omit apt logs)
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:817 errors:0 dropped:0 overruns:0 frame:0
          TX packets:364 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10517296 (10.0 MiB)  TX bytes:25479 (24.8 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ruins@thor:~$ docker run --rm -it debian:stretch-slim sh -c "apt update;apt install net-tools;ifconfig -a"
(omit apt logs)
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 508  bytes 8413071 (8.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 256  bytes 18348 (17.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

makeにてmackerel-agentのソースコンパイルを試みたところ、

_tools/go-linter: golint: not found
*** Error code 127

にてコケてビルドできません。golintはインストール済みの状態でこのエラーが出ます。

環境はmacOS、gopathは$HOME/goに、バージョンは1.13.4（goenv経由）を使用しています。

make build、make runじゃないとできないのですかね？

I think at least 2048R key is required in 2016.

$ gpg GPG-KEY-mackerel
pub  1024D/C2B48821 2014-05-08 Hatena Inc. <[email protected]>

ret[name+".percentage"] = (currValues[i] - prevValues[i]) * 100.0 * float64(cpuCount) / (currTotal - prevTotal)

no need cpucount
right is

ret[name+".percentage"] = (currValues[i] - prevValues[i]) * 100.0 / (currTotal - prevTotal)

The permission of root directory (default: /var/lib/mackerel-agent) should be 755, because some users or tools who has no root permissions may want to know host id which is saved in file id under root directory.

I'm sorry if this is not the right place to report issues related to setup-all-apt-v2.sh, which is not part of this repository, but I was not aware of anywhere else better.

The installer script setup-all-apt-v2.sh currently uses the apt-key command to add the repository's key to APT's trusted keyring, but this method has been deprecated due to security reasons.
According to the Debian's document the recommend method is to create a new GPG key ring in some location (such as /usr/share/keyrings/mackerel.gpg) and then pin the repository signature to it in the APT source file

deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/mackerel.gpg] http://apt.mackerel.io/v2/ mackerel contrib

PS: I am not a security expert nor a Debian committer so I can't take responsibility for the correctness of the above method.

owatan@7a85fb4

Hi.

I read Installing mackerel-agent in binary - Mackerel Docs, so
I wrote rc script for FreeBSD , but I don't know where I would put this file when I send pull-request.

I checked operation verification on FreeBSD 11.2-RELEASE, and FreeBSD 12.0-RELEASE.

日本語

バイナリファイルで直接mackerel-agentをインストールする - Mackerel ヘルプ を読むと

ご利用の環境にあわせてサービス化してください。

とあったので FreeBSD で mackerel-agent をサービス化するスクリプトを書きましたが、
これが (一般に) 提供されていればより使いやすくなるのではないか、と思い PR を送ろうとした次第です。

mackerelio/mackerel-agent のレポジトリを見ても、このファイルを追加する PR を出す際に、どこに置けば良いのか分からなかったため、ひとまずこの Issue を立てました。

FreeBSD 11.2-RELEASE および FreeBSD 12.0-RELEASE で実際に動作することを確認しています。

maybe, it is caused by kernel version. (v4.4 kernel series)

mackerel-agent Version

$ rpm -qa | grep mackerel
mackerel-check-plugins-0.5.0-1.noarch
mackerel-agent-0.30.0-1.noarch
mackerel-agent-plugins-0.19.0-1.noarch

Steps

kernel v4.4

$ uname -r
4.4.5-15.26.amzn1.x86_64
$ /etc/init.d/mackerel-agent configtest
Testing configuration of mackerel-agent: /etc/init.d/mackerel-agent: 57 行:  2834 Trace/breakpoint trap   $BIN configtest ${APIBASE:+--apibase=$APIBASE} ${APIKEY:+--apikey=$APIKEY} --pidfile=$PIDFILE --root=$ROOT $OTHER_OPTS >> $LOGFILE 2>&1
                                                           [失敗]

downgrade kernel version

Change default=0 to default=1, and reboot.

$ cat /etc/grub.conf
# created by imagebuilder
default=1
timeout=1
hiddenmenu

title Amazon Linux 2016.03 (4.4.5-15.26.amzn1.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-4.4.5-15.26.amzn1.x86_64 root=LABEL=/ console=ttyS0 LANG=ja_JP.UTF-8 KEYTABLE=us
initrd /boot/initramfs-4.4.5-15.26.amzn1.x86_64.img

title Amazon Linux 2015.09 (4.1.17-22.30.amzn1.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-4.1.17-22.30.amzn1.x86_64 root=LABEL=/ console=ttyS0 LANG=en_US.UTF-8 KEYTABLE=us
initrd /boot/initramfs-4.1.17-22.30.amzn1.x86_64.img

title Amazon Linux 2015.03 (3.14.48-33.39.amzn1.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-3.14.48-33.39.amzn1.x86_64 root=LABEL=/ console=ttyS0 LANG=en_US.UTF-8 KEYTABLE=us
initrd /boot/initramfs-3.14.48-33.39.amzn1.x86_64.img

$ reboot

kernel v4.1

$ uname -r
4.1.17-22.30.amzn1.x86_64
$ /etc/init.d/mackerel-agent configtest
Testing configuration of mackerel-agent:                   [  OK  ]

Ref.

Amazon Linux AMI 2016.03

If you create a config file like the one below and run configtest, a suggestion will be output to warnings. However, I could not understand the meaning of this content. Is this correct behavior?

apikey = "aaa"

[plugin.metrics.aaa]
command = ["exit 1"]
[plugin.metrics.aaa.bbb] ← Target of warning
command = ["exit 2"]

$ mackerel-agent version
mackerel-agent version 0.73.2 (rev 3fd96af) [darwin amd64 go1.19]

$ mackerel-agent configtest -conf a.conf
[WARNING] plugin.metrics.aaa.bbb is unexpected key.
[WARNING] plugin.metrics.aaa.bbb.command is unexpected key. did you mean plugin.metrics.aaa.bbb.command ?

According to the Mackerel document, custom metrics plugins should output tab-separated values as follows;

{metric name}\t{metric value}\t{epoch seconds}

but Sensu plugins using Sensu::Plugin::Metric::CLI::Graphite output space-separated values;

https://github.com/sensu-plugins/sensu-plugin/blob/master/lib/sensu-plugin/metric/cli.rb#L26

so if you say the format is "compatible with sensu", the parser should split values with spaces not tabs or both of them.

https://github.com/mackerelio/mackerel-agent/blob/master/metrics/plugin.go#L231

Hi.

I've found that metrics/linux/cpuusage.go calucurates user inaccurately on Linux hosts running KVM vms.

What's inaccurate?

As you can see man 5 proc, 1st column of /proc/stat means user (1) Time spent in user mode, and 9th column means guest (9) Time spent running a virtual CPU for guest operating systems under the control of the Linux kernel.

Acutually, user includes guest, so cputime is duplicated, if host running KVM vms. ( ... it is undocumented 😭 )

Evidence

In Linux kernel sources, user is accounted as cputime[CPUTIME_USER] and guest is as cputime[CPUTIME_GUEST], by account_guest_time

Here is a source of 4.14-rc3

/*
 * Account guest cpu time to a process.
 * @p: the process that the cpu time gets accounted to
 * @cputime: the cpu time spent in virtual machine since the last update
 */
void account_guest_time(struct task_struct *p, u64 cputime)
{
	u64 *cpustat = kcpustat_this_cpu->cpustat;

	/* Add guest time to process. */
	p->utime += cputime;
	account_group_user_time(p, cputime);
	p->gtime += cputime;

	/* Add guest time to cpustat. */
	if (task_nice(p) > 0) {
		cpustat[CPUTIME_NICE] += cputime;
		cpustat[CPUTIME_GUEST_NICE] += cputime;
	} else {
		cpustat[CPUTIME_USER] += cputime; ⭐
		cpustat[CPUTIME_GUEST] += cputime; ⭐
	}
}

In case task_struct is virtual machine (e.g. KVM), account_guest_time accounts cputime to both cpustat[CPUTIME_USER] and cpustat[CPUTIME_GUEST].

account_guest_time has been introduced in v2.6.24-rc1. ( torvalds/linux@94886b8 )

/*
 * Account guest cpu time to a process.
 * @p: the process that the cpu time gets accounted to
 * @cputime: the cpu time spent in virtual machine since the last update
 */
void account_guest_time(struct task_struct *p, cputime_t cputime)
{
	cputime64_t tmp;
	struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;

	tmp = cputime_to_cputime64(cputime);

	p->utime = cputime_add(p->utime, cputime);
	p->gtime = cputime_add(p->gtime, cputime);

	cpustat->user = cputime64_add(cpustat->user, tmp);
	cpustat->guest = cputime64_add(cpustat->guest, tmp);
}

How to correct ?

Subtracting guest time from user time.

user = user - guest

Example

Here is mpstat implementation.

https://github.com/sysstat/sysstat/blob/master/mpstat.c#L529-L568 (d7321c1)

/*
 ***************************************************************************
 * Display CPU statistics in JSON format.
 *
 * IN:
 * @tab		Number of tabs to print.
 * @g_itv	Interval value in jiffies multiplied by the number of CPU.
 * @prev	Position in array where statistics used	as reference are.
 *		Stats used as reference may be the previous ones read, or
 *		the very first ones when calculating the average.
 * @curr	Position in array where current statistics will be saved.
 ***************************************************************************
 */
void write_json_cpu_stats(int tab, unsigned long long g_itv, int prev, int curr)
{
	struct stats_cpu *scc, *scp;
	unsigned long long pc_itv;
	int cpu, next = FALSE;

	xprintf(tab++, "\"cpu-load\": [");

	/* Check if we want global stats among all proc */
	if (*cpu_bitmap & 1) {

		next = TRUE;
		xprintf0(tab, "{\"cpu\": \"all\", \"usr\": %.2f, \"nice\": %.2f, \"sys\": %.2f, "
			      "\"iowait\": %.2f, \"irq\": %.2f, \"soft\": %.2f, \"steal\": %.2f, "
			      "\"guest\": %.2f, \"gnice\": %.2f, \"idle\": %.2f}",
			 (st_cpu[curr]->cpu_user - st_cpu[curr]->cpu_guest) < ⭐
			 (st_cpu[prev]->cpu_user - st_cpu[prev]->cpu_guest) ? ⭐
			 0.0 :
			 ll_sp_value(st_cpu[prev]->cpu_user - st_cpu[prev]->cpu_guest, ⭐
				     st_cpu[curr]->cpu_user - st_cpu[curr]->cpu_guest, ⭐
				     g_itv),

⭐ are the points I mentioned above.

Also torvalds/linux@94886b8 said

A modified "top(1)" is able to display good cpu user time and cpu guest
time by subtracting cpu guest time from cpu user time. Update "gtime" in
task_struct accordingly.

日本語

仮想マシン KVM で動かしている Linux ホストで metrics/linux/cpuusage.go の user と guest 時間が正確に表示できないので issue をたてました

• Linux をホストとして KVM でゲストを実行していると、 /proc/stat が返す user 時間に guest 時間が余分に加算されます (カーネルのソースを参照してください)
• 解決するには user 時間から guest 時間を減算するよいです (mpstat のソースや　account_guest_time が追加された際のコミットログを参考にしてください)

Hi,

I installed mackerel-agent with this article on Heroku,
then there are some errors like below every hour:

block_device.go:29: ERROR <spec.block_device> Failed (skip this spec): open /sys/block: no such file or directory

I use only 1 dyno, and Django is working on Heroku.
How do I do to stop this error ?

Thanks.

Since recent update Custom Identifier (eg. AWS instance id) check is introduced( #405 ). This check ensures that the Custom Identifier of the current instance matches the one previously stored on the mackerel.io server for the HostID. In our setup, this check prevents mackerel-agent to start up.

The Custom Identifier of the current instance is fetched from the cloud meta-data service (i.e. http://169.25.169.254 in case of AWS). The meta-data service is expected to return response within 100ms:

When the meta-data service is unable to respond within 100ms, an empty string is used as the Custom Identifier of the current instance, which does not match the one successfully stored on the server in the previous agent run.

We use OpenStack, which provides AWS-compatible meta-data service at 169.254.169.254, but in our setup it sometimes take a bit longer than 100ms to respond. Due to the reason described above, we are having trouble that mackerel-agent often fails to restart.

I have some idea to deal with this problem,

• Add an option to extend the HTTP timeout for the meta-data service,
• Add an option to disable cloud meta-data retrieval, or
• Skip the check when the Custom Identifier cannot be fetched for some reason.

Which one is preferred? I can provide a patch for the solutions if necessary.

• care server load
• up to 30 seconds

Discussion Issue about mult platform support.