This project is no longer maintained.
If you're interested in taking it over, we'd love to hear from you. Email [email protected], or check out one of its maintained forks.
A complete Logstash stack on AWS OpsWorks
UPDATE: I wrote a blog post that will help you set this up.
This is a bunch of cookbooks that will allow you to run a complete Logstash setup on a scalable AWS OpsWorks stack. At Springest we use it to ship from 250 to about 1k log entries per second, depending on the RPM on our 18 servers.
- A RabbitMQ cluster layer (Ubuntu) – Agents wil ship their logs to this cluster and the Logstash server cluster uses it as an input source.
- An ElasticSearch cluster layer (Amazon Linux) – All log messages are stored and indexed here.
- A LogStash cluster layer (Ubuntu) – Takes the messages from the RabbitMQ fanout and puts them into ElasticSearch.
- A Kibana server (Ubuntu) - An Angular.js interface on top of ElasticSearch to search, graph etc.
- Set
git://github.com/Springest/opsworks-logstash.gitas a repository URL for your custom cookbooks. - Use the following Chef custom JSON:
{
"chef_environment": "production",
"elasticsearch": {
"cluster": {
"name": "logstash"
},
"basic_auth": {
"user": "<es username here>",
"password": "<es password here>"
}
},
"rabbitmq": {
"cluster": true,
"erlang_cookie": "<any random alphanumeric string will do>"
},
"rabbitmq_cluster": {
"user": "<rabbitmq username here>",
"password": "<rabbitmq password here>"
},
"kibana": {
"webserver": "nginx",
"webserver_hostname": "<your kibana domain, eg: kibana.example.com>",
"web_user": "<kibana username here>",
"web_password": "<kibana password here>",
"es_port": "9200",
"es_role": "elasticsearch",
"es_server": "<es loadbalancer address here>",
"es_user": "<es username here>",
"es_password": "<es password here>"
},
"logstash": {
"elasticsearch_cluster": "logstash",
"server": {
"install_rabbitmq": false,
"enable_embedded_es": false,
"elasticsearch_role": "elasticsearch",
"inputs": [
{
"rabbitmq": {
"exchange": "rawlogs",
"user": "<rabbitmq username here>",
"password": "<rabbitmq password here>",
"host": "<IP or LB address of your rabbitmq server/cluster>",
"type": "all"
}
}
]
}
}
}
- Fork the project
- Create a feature branch
- Fix the code
- Create a pull request
All of these people have made our lives a little better by writing great software.
The contributors at these projects:
- ElasticSearch for their awesome search engine.
- Kibana for a beautiful graphing tool.
- LogStash for bringing our logs together.
- RabbitMQ for a robust message broker.
And the people who wrote these cookbooks:
- @lusis for his work on the Kibana cookbook and the LogStash cookbook
- @karmi for the original, and @ehlertij, @tkawa, and @interu for their efforts on the ElasticSearch cookbook for OpsWorks.
- @brianbianco for his Redis cookbook
- @sethvargo for the hostsfile cookbook
- @ddux for the rbenv cookbook
- @bryanwb for the Yumrepo cookbook
- @guilhem for his efforts on the Htpaswd cookbook
And the guys at Opscode and their contributors for Chef and cookbooks like:
- RabbitMQ
- Java
- Apt
- Erlang
- Nginx
- NTP
- Runit
- Ant
- Apache2
- Ark
- AWS
- Build-essential
- Chef handler
- Git
- Ohai
- OpenSSL
- Perl
- Python
- Windows
- xfs
- xml
- yum
- rsyslog
And last but not least:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent