ciscocloud / distributive Goto Github PK

TableToString is almost RST format, and there's no reason why it shouldn't be. It is a pretty format for printing, and useful in other scenarios. WIll make tabular useful in more cases.

This is a bug that has been floating around for a while. The source is definitely probabalisticSplit.

Proposed solutions:

• generalize the process of throwing out outliers, create a loop that throws out up to three.
• create a special case for first and last rows.

It seems if the file isn't there the error it throughs isn't a very had one. @noahjl discovered this in the haproxy.json health checks.

This was using version 0.1.1 on the mi vagrant box.

# The way to cause it - 

bash-4.2# distributive -v=2 -f haproxy.json
Running with verbosity level 2
Creating checklist(s)...
Running checklist: haproxy health check
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x30 pc=0x418a70]

goroutine 1 [running]:
main.func·001(0xc20801c5a0, 0x14, 0x4a41b5, 0x0, 0x0)
        /home/vagrant/rpmbuild/BUILD/distributive-0.1.1/filesystem.go:38 +0x70
main.isType(0x73d0b0, 0x4, 0x7cead0, 0xc20801c5a0, 0x14, 0x10, 0x0, 0x0)
        /home/vagrant/rpmbuild/BUILD/distributive-0.1.1/filesystem.go:20 +0x58
main.file(0xc20800a240, 0x1, 0x4, 0x1, 0x0, 0x0)
        /home/vagrant/rpmbuild/BUILD/distributive-0.1.1/filesystem.go:43 +0x6e
main.runChecks(0xc20801c4e0, 0x14, 0xc20803ad80, 0x28, 0xc20803c320, 0x2, 0x2, 0x0, 0x0, 0x0, ...)
        /home/vagrant/rpmbuild/BUILD/distributive-0.1.1/main.go:288 +0x3b2
main.main()
        /home/vagrant/rpmbuild/BUILD/distributive-0.1.1/main.go:326 +0x31f

# The particular file used to get here
-bash-4.2# cat haproxy.json
{
    "Name": "haproxy health check",
    "Notes": "For use with Microservice Infrastructure",
    "Checklist" : [
        {
            "Name": "haproxy environment file",
            "Check": "file",
            "Parameters": ["/etc/default/haproxy"]
        },
        {
            "Name": "haproxy configuration file",
            "Check": "file",
            "Parameters": ["/usr/lib/systemd/system/haproxy.service"]
        }
    ]
}

I am implementing the ability to input multiple health checks in a single JSON file. Every JSON file will have the following structure:

{
    "Name": "Name",
    "Notes": "Notes",
    "Checklist": [
        {
            "Port": 80
        },
        {
            "Exists": "/foo/bar"
        }
    ]
}

We should have an option to disable certificate validation when connection to a remote server to get a response. The solution is outlined here.

Repo checks are currently known to be buggy and don't work for any package manager.

Right now, it ain't much. Guidelines here, page here.

Also, get the little Github sticker thingy for the README!

I'm currently implementing these network interface checks:

• exist
• up
• speed
• have_ipv4_address
• have_ipv6_address

A command line flag, maybe -u for url, which then downloads the file to /var/run/distributive/checks.

We have to consider security.

Add a function to print only if a certain verbosity requirement is met.
The main logic should be this:

1. Parse flags (can they be parsed once for each flag as separate methods?)
2. Check validity of flag options
3. Create checklist
4. Run checks
5. Make and print report

So testers can verify which version of distributive is installed

Given an ocatal permissions mode (0[0-7][0-7][0-7]), ensure that the given file has those permissions.

Add a check for:

• if a kernel module is loaded (serverspec).
• if a kernel parameter is set (serverspec)

I'd like to add considerably more documentation. It will take the form of a Github wiki.

Things to be documented:

• General JSON structure
• Detail on each check
  • Inputs
  • Outputs/description (when would this check fail?)
  • Dependencies
• Creating a Consul check configuration
• Architecture of the software (and how to get started writing checks)

The README will then consist of the following sections:

• Installation and usage
• Dependencies (outline of how this works)
• the rest

Implement a check that takes three arguments:

• The hash algorithm name: SHA1 | SHA224 | SHA256 | SHA384 | SHA512 | MD5
• The hash to check against
• The file to check the hash of

In these spots

• main.go, circa 215, where the thunks are constructed
• main.go, circa 262, where the checks are run

Given a regex, does this file match said regex?

Static checks should go in /etc/distributive.d, dynamic in /var/run/distributive/checks/

Even if the command fails.

Noah had difficulties installing Distributive with the current README instructions. Add more details, especially about $GOPATH and go get.

Currently fails if the table can't be split. instead, it should log a warning and then return the rows, unsplit.

• Is systemctl available on the machine?
• What is the status of a service? (systemctl list-units)
• Is a sock listening at this path? (systemctl list-sockets)
• Is a timer running (systemctl list-timers)
• Is a timer loaded? (systemctl list-timers --all)
• Is a unit file loaded? (systemctl list-unit-files)

So that the response from something like Consul can be parsed.

A few things:

• When no tests fail, no message should be printed
• When a test prints a given/actual pair, it should be indented one tab
• Tests with given/expected pairs can be generalized - a function in thunk-utils.go should do the trick
• All tests should include given/actual pairs when appropriate
• Replace instances of fatal with a descriptive log.Fatal.
• The sample dir should include two files for each round of tests - one that will succeed for the most part, and one that will intentionally fail (instead of having them mixed together)
• Make error reporting depend on verbosity (command line flag)
• Ensure that all checks use a unified error report creating method that depends on verbosity level (mostly done)

For example:

docker ps image NAME:
ciscocloud/mesos-consul

distributive query required to match that image:
docker.io/ciscocloud/mesos-consul

Add a new command line flag, --stdin

The dict/switch scenario in main.go is unsustainable. We need something like

registerCheck(name string, thunk Thunk, numParams int)

Every single check will now just take a list of strings, Parameters. They will simply return an exit code and exit message. This is a complete overhaul of the internal architecture, but I've already got a working version (decomposition FTW).

This should ultimately lead to more sustainability and simpler code.

Likely, none of the methods need to be public, as they are all called within the same namespace. Just lower the caps.

All workers should be lower case.

It seems that the systemctl checks are not quite parsing the unit status correctly. Example:

        {
            "Name": "zookeeper service active",
            "Check": "systemctlActive",
            "Parameters": ["zookeeper.service"]
        },
        {
            "Name": "zookeeper service loaded",
            "Check": "systemctlLoaded",
            "Parameters": ["zookeeper.service"]
        },

Which returns -

Service did not have state:
        Specified: active
        Actual: []
Service did not have state:
        Specified: loaded
        Actual: []

From looking at systemctl.go (which I may completely be mis-interpreting) I'm wondering if it would be simpler running something like -

Active -

systemctl show -p ActiveState $SERVICE

Loaded -

systemctl show -p LoadState $SERVICE

This returns something like -

-bash-4.2# systemctl show -p ActiveState zookeeper.service
ActiveState=active
-bash-4.2# systemctl show -p LoadState zookeeper.service
LoadState=loaded

Some headers have changed, there are new ones as well

This function will have a signature of:
func SplitByAlignment(str string) Table
It will take a string and split it into rows, then find the index of name in the first row, and separate the column that is aligned by index with that name.

Distributive deals with a lot of tabular data. With methods like strIn, strContainedIn, reIn, commandColumnNoHeader, it's clear that this could be abstracted even further. My proposal is as follows:

We need one, unified method for splitting a tabular string into a 2D slice. It should detect which regexp to use, judging by the consistency of the row widths (it would assume even rows). It might do this using standard deviation or something. It would abstract further and eliminate the need for separateString, stringToSlice, and stringToSliceMultispace. Possible regexp's: for rows: "\n+", for columns: "\\s{2,}", "\\s+", "\t+".

In conjunction, we need a method that fetches the a column (sans header) by the header title. This should be super simple and will totally prettify the code.

For organization, this will all go into another go package: tabular.go

From @LeeLammert:

It would be very nice to have a check that would match a string via http/https - e.g. "Running" for the Marathon UI, to simple status checks via the a webUI.

The basic process is curl -> response -> regex match

Questions: how do we deal with HTTP/HTTPS?

Generalize yum repo to other package managers

• apt
• pacman

Then, the following functions will cover the functionality of PPA, yumrepo, etc.:

• repoExists(manager string, name string) : works for Pacman, Yum
• repoExistsURI(manager string, urlstr string) : works for any package manager

Concurrency stands to be implemented in a few more spots.

I've learned a lot more about concurrecy recently (2015-08-05) and am starting work on this.

Two phases:

• Max percent used (memory)
• Max percent used (disk)
• Max percent used (swap)

And later,

• min free/max used kb, mb, gb, tb (memory)
• min free/max used kb, mb, gb, tb (disk)
• min free/max used kb, mb, gb, tb (swap)

Enable sending logs to disk:

• Command line option to send all output to one file
• JSON checklist option for where to log that checklist
• JSON check option for where to log the output of that check

It must be

• Logstash compatible
• Permissions concious

Taken from the official documentation. Need to figure out which ones we want to do, as well as which ones we've already done.

• check_apt
• check_breeze
• check_by_ssh
• check_clamd
• check_cluster
• check_dhcp
• check_dig
• check_disk
• check_disk_smb
• check_dns
• check_dummy
• check_file_age
• check_flexlm
• check_ftp
• check_hpjd
• check_http
• check_icmp
• check_ide_smart
• check_ifoperstatus
• check_ifstatus
• check_imap
• check_ircd
• check_jabber
• check_load
• check_log
• check_mailq
• check_mrtg
• check_mrtgtraf
• check_nagios
• check_nntp
• check_nntps
• check_nt
• check_ntp
• check_ntp_peer
• check_ntp_time
• check_nwstat
• check_oracle
• check_overcr
• check_ping
• check_pop
• check_procs
• check_real
• check_rpc
• check_sensors
• check_simap
• check_smtp
• check_snmp
• check_spop
• check_ssh
• check_ssmtp
• check_swap
• check_tcp
• check_time
• check_udp
• check_ups
• check_uptime
• check_users
• check_wave
• negate
• urlize

-a should run everything in a given directory. It should be non-blank by default, in particular -a=/etc/distributive.d/.

https://docs.docker.com/reference/api/docker_remote_api/
https://github.com/fsouza/go-dockerclient

Serverspec currently defines these kinds of checks for Linux systems:

• bond
• bridge
• cgroup
• command
• cron
• default_gateway
• docker_container
• docker_image
• file
• group
• host
• interface
• ipfilter
• ipnat
• iptables
• kernel_module
• linux_kernel_parameter
• lxc
• mail_alias
• package
• php_config
• port
• ppa
• process
• routing_table
• selinux
• service
• user
• x509_certificate
• x509_private_key
• yumrepo
• zfs

I'd like to implement more of them.

Several package managers can output a list of active repos. Try parsing this. For example, yum repolist.

Repo name matches regex

Update docs

Golang's flag package just isn't great. Look into libraries for this.

All *.go files in the root dir should be autonomous. They should register their own checks when imported, so that they can be left out at compile time.

Improvement for genericError: Don't just fmt.Sprint whole tables depending on length. Join them with ", " and then truncate that string.

Validating some of the checks and noticing a consistent fail for repoExists checks -

{
    "Name": "chronos health check",
    "Notes": "For use with Microservice Infrastructure",
    "Checklist" : [
        {
            "Name": "mesosphere repo present in yum",
            "Check": "repoExistsURI",
            "Parameters": ["yum","http://repos.mesosphere.io/el/7/noarch/"]
        },
....

based on packages.go it seems that the baseurl or name in /etc/yum.repos.d/mesosphere.repo should fit the parameters. But they return errors:

Repo with given Url not found:
        Specified: http://repos.mesosphere.io/el/7/$basearch/
        Actual: []

Using repoExists and the "name=" in mesosphere.repo

Repo with given Name not found:
        Specified: Mesosphere Packages for EL 7 - noarch
        Actual: []

Using the microservices-infrastructure vagrant box for testing.

I've also installed the RPM of distributive 0.1.

Make the Check struct have two fields: check type (string) and options ([]string).

I am implementing more granular filesystem checks, specifically File, Directory, and Symlink. These will all properly handle not found and permissions errors.