This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.

This plugin collects metrics from Yarn hadoop scheduler.

It's used in the Snap framework.

1. Getting Started

• System Requirements
• Operating systems
• Installation
• Configuration and Usage

2. Documentation

• Global Config
• Collected Metrics
• Examples
• Roadmap

3. Community Support
4. Contributing
5. License
6. Acknowledgements

• golang 1.7+ - needed only for building
• hadoop 3.2.x

All OSs currently supported by snap:

• Linux/amd64
• Darwin/amd64

You can get the pre-built binaries for your OS and architecture from the plugin's GitHub Releases page. Download the plugin from the latest release and load it into snapteld (/opt/snap/plugins is the default location for snap packages).

Fork https://github.com/intelsdi-x/snap-plugin-collector-yarn Clone repo into $GOPATH/src/github.com/intelsdi-x/:

$ git clone https://github.com/<yourGithubID>/snap-plugin-collector-yarn.git

Build the snap yarn plugin by running make within the cloned repo:

$ make

This builds the plugin in ./build/

• Set up the snap framework.

• Configure connection strings to Hadoop resource manager which includes hostname and port.

• Load the plugin and create a task, see example in Examples.

This collector gathers metrics from yarn server status command.

Global configuration files are described in snap's documentation. You have to add "yarn" section with following entries:

• "hostname" - hostname of the hadoop cluster
• "port" - resource manager port number

See exemplary Global configuration files in [examplary config files] (examples/configs/).

List of collected metrics is described in METRICS.md.

Example of running snap yarn collector and writing data to file.

Ensure snap daemon is running:

• initd: service snap-telemetry start
• systemd: systemctl start snap-telemetry
• command line: snapteld -l 1 -t 0 &

Download and load snap plugins:

$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-yarn/latest/linux/x86_64/snap-plugin-collector-yarn
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-file/latest/linux/x86_64/snap-plugin-publisher-file
$ chmod 755 snap-plugin-*
$ snaptel plugin load snap-plugin-collector-yarn
$ snaptel plugin load snap-plugin-publisher-file

Create a task manifest file  (exemplary files in [examples/tasks/] (examples/tasks/)):
```yaml
---
  version: 1
  schedule:
    type: "simple"
    interval: "1s"
  max-failures: 10
  workflow:
    collect:
      metrics:
        /intel/yarn/queue/*/absolutecapacity: {}
        /intel/yarn/queue/*/absolutemaxcapacity: {}
        /intel/yarn/queue/*/absoluteusedcapacity: {}
        /intel/yarn/queue/*/capacity: {}
        /intel/yarn/queue/*/maxactiveapplications: {}
        /intel/yarn/queue/*/maxactiveapplicationsperuser: {}
        /intel/yarn/queue/*/maxapplications: {}
        /intel/yarn/queue/*/maxapplicationsperuser: {}
        /intel/yarn/queue/*/maxcapacity: {}
        /intel/yarn/queue/*/numactiveapplications: {}
        /intel/yarn/queue/*/numapplications: {}
        /intel/yarn/queue/*/numcontainers: {}
        /intel/yarn/queue/*/numpendingapplications: {}
        /intel/yarn/queue/*/usedcapacity: {}
        /intel/yarn/queue/*/usedresources: {}
        /intel/yarn/queue/*/userlimit: {}
        /intel/yarn/queue/*/userlimitfactor: {}
        /intel/yarn/queue/*/resources_memory: {}
        /intel/yarn/queue/*/resources_vcores: {}
        /intel/yarn/schedulerinfo/capacity: {}
        /intel/yarn/schedulerinfo/maxcapacity: {}
      publish:
        - plugin_name: "file"
          config:
            file: "/tmp/yarn_metrics"

Download an example task file and load it:

$ curl -sfLO https://raw.githubusercontent.com/intelsdi-x/snap-plugin-collector-yarn/master/examples/tasks/yarn-file.yml
$ snaptel task create -t yarn-file.json
Using task manifest to create task
Task created
ID: 480323af-15b0-4af8-a526-eb2ca6d8ae67
Name: Task-480323af-15b0-4af8-a526-eb2ca6d8ae67
State: Running

See realtime output from snaptel task watch <task_id> (CTRL+C to exit)

$ snaptel task watch 480323af-15b0-4af8-a526-eb2ca6d8ae67

This data is published to a file /tmp/yarn_metrics per task specification

Stop task:

$ snaptel task stop 480323af-15b0-4af8-a526-eb2ca6d8ae67
Task stopped:
ID: 480323af-15b0-4af8-a526-eb2ca6d8ae67

There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release.

If you have a feature request, please add it as an issue and/or submit a pull request.

This repository is one of many plugins in snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap.

To reach out to other users, head to the main framework or visit Slack.

We love contributions!

There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.

Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.

• Author: @Marcin Spoczynski

This software has been contributed by MIKELANGELO, a Horizon 2020 project co-funded by the European Union. https://www.mikelangelo-project.eu/

And thank you! Your contribution, through code and participation, is incredibly important to us.