logstash-plugins / logstash-input-github Goto Github PK

Accept Github Webhooks into Logstash

License: Apache License 2.0

Ruby 100.00%

logstash-input-github's Introduction

Logstash Plugin

This is a plugin for Logstash.

It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

Documentation

Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one central location.

For formatting code or config example, you can use the asciidoc [source,ruby] directive
For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide

Need Help?

Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.

Developing

1. Plugin Developement and Testing

Code

To get started, you'll need JRuby with the Bundler gem installed.
Create a new plugin or clone and existing from the GitHub logstash-plugins organization. We also provide example plugins.
Install dependencies

bundle install

Test

Update your dependencies

bundle install

Run tests

bundle exec rspec

2. Running your unpublished Plugin in Logstash

2.1 Run in a local Logstash clone

Edit Logstash Gemfile and add the local plugin path, for example:

gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"

Install plugin

# Logstash 2.3 and higher
bin/logstash-plugin install --no-verify

# Prior to Logstash 2.3
bin/plugin install --no-verify

Run Logstash with your plugin

bin/logstash -e 'filter {awesome {}}'

At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.

2.2 Run in an installed Logstash

You can use the same 2.1 method to run your plugin in an installed Logstash by editing its Gemfile and pointing the :path to your local plugin development directory or you can build the gem and install it using:

Build your plugin gem

gem build logstash-filter-awesome.gemspec

Install the plugin from the Logstash home

# Logstash 2.3 and higher
bin/logstash-plugin install --no-verify

# Prior to Logstash 2.3
bin/plugin install --no-verify

Start Logstash and proceed to test the plugin

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important to the community that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.

logstash-input-github's People

Contributors

Stargazers

Watchers

Forkers

ph wiibaa yudupi xyzhaoyx alexlance jonespm timani nnordrum yaauie denzuko-forked mindbat lstanwan chrisneff isabella232 mashhurs edmocosta robbavey andsel

logstash-input-github's Issues

Need to parse out the headers

More underderstanding the FTW lib to parse the headers.

[Function] Add x-hub-signature support

Github can sign events using a shared secret - Need to implement this and provide to add a tag or discard events if they don't match

uninitialized constant LogStash::Inputs::GitHub::Rack

Hi the plugin seems to be crashing logstash when github POSTs to it.

Logstash 2.3.4
logstash-input-github 2.0.5
debian stretch/sid (docker container sebp/elk)

config:

# logstash.conf
input {
  github {
    codec => "json"
    port => 8080
    secret_token => "secret"
  } 
} 

filter {
}

output {
  elasticsearch {
    hosts => "localhost:9200"
  }          
}

Steps to Reproduce:

Post some data from github and see the stack trace in the logstash errors file:

NameError: uninitialized constant LogStash::Inputs::GitHub::Rack
      const_missing at org/jruby/RubyModule.java:2745
                run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-github-2.0.5/lib/logstash/inputs/github.rb:41
               call at org/jruby/RubyProc.java:281
     handle_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.44/lib/ftw/webserver.rb:77
  handle_connection at /opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.44/lib/ftw/webserver.rb:59
                run at /opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.44/lib/ftw/webserver.rb:29

Github Expects a response

Github servers wait for a response from the system - like any good web server.

So, do I write an input to handle this, or somehow modify the TCP input to allow codec's some way of sending a response.

NoMethodError Exception

I'm trying to use the git plugin but I get:

C:\work\jiras\logstash\ls522>bin/logstash -f gitlogstash3.conf
C:\work\development-environment\Tools\GnuWin32\bin\stty.exe: standard input: Function not implemented
Could not find log4j2 configuration at path /work/jiras/logstash/ls522/config/log4j2.properties. Using default config which logs to console
16:55:24.136 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>24, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>3000}
16:55:24.571 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
16:55:24.650 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}

<NoMethodError: undefined method headers_' for #> C:/work/jiras/logstash/ls522/vendor/bundle/jruby/1.9/gems/logstash-input-github-3.0.1/lib/logstash/inputs/github.rb:32:in run' org/jruby/RubyProc.java:281:incall' C:/work/jiras/logstash/ls522/vendor/bundle/jruby/1.9/gems/ftw-0.0.45/lib/ftw/webserver.rb:77:in handle_request' C:/work/jiras/logstash/ls522/vendor/bundle/jruby/1.9/gems/ftw-0.0.45/lib/ftw/webserver.rb:59:inhandle_connection' C:/work/jiras/logstash/ls522/vendor/bundle/jruby/1.9/gems/ftw-0.0.45/lib/ftw/webserver.rb:29:in run' 16:58:31.249 [LogStash::Runner] FATAL logstash.runner - An unexpected error occurred! {:error=>#<NoMethodError: undefined methodheaders_' for #<FTW::Request:0x322b1385>>, :backtrace=>["C:/work/jiras/logstash/ls522/vendor/bundle/jruby/1.9/gems/logstash-input-github-3.0.1/lib/logstash/inputs/github.rb:32:inrun'", "org/jruby/RubyProc.java:281:incall'", "C:/work/jiras/logstash/ls522/vendor/bundle/jruby/1.9/gems/ftw-0.0.45/lib/ftw/webserver.rb:77:inhandle_request'", "C:/work/jiras/logstash/ls522/vendor/bundle/jruby/1.9/gems/ftw-0.0.45/lib/ftw/webserver.rb:59:inhandle_connection'", "C:/work/jiras/logstash/ls522/vendor/bundle/jruby/1.9/gems/ftw-0.0.45/lib/ftw/webserver.rb:29:inrun'"]}
16:58:31.344 [[main]<github] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Plugin: 8083, id=>"04388f32709d653b85caa29a820e09609393f127-1", enable_metric=>true, codec=>"plain_e1b9d3c3-c580-47f0-9663-ea4eb1f21bfa", enable_metric=>true, charset=>"UTF-8">, ip=>"0.0.0.0", drop_invalid=>false>
Error: Bad file descriptor - Bad file descriptor

my config is:
input {
github {
port => 8083
}
}

output {
stdout {
codec => rubydebug
}
}

It's obvious that the plugin get something but it does not seem to be able to parse it. The hook is just listenting for push events at and posts to http://10.10.12.86:8083/

My webhooks are configured properly to handle events and I can see that working when substituting the logstash git plugin with a web server and checking the logs for the webhooks Posts.

Did anybody get this plugin to listen to git hooks?

The C:\work\development-environment\Tools\GnuWin32\bin\stty.exe: standard input: Function not implemented is a not a problem as it appears everywhere with all my working plugins.

Run on Windows as you might have guessed from the path :)

Cheers

drop_invalid contains security issue if no header x-hub-signature

I found a security issue with the plugin. If the header "x-hub-signature" isn't in the request and drop_invalid is setted to "true" then the message isn't dropped...

Version: latest

Operating System: MACOS

Config File :
input {
github {
drop_invalid => true
ip => "0.0.0.0"
port => "8080"
secret_token => "test"
}
}

Steps to Reproduce:
run
curl -XPOST -d'test' localhost:8080
The request won't be dropped by the plugin because the header "x-hub-signature" isn't present, so it pass the control...

refactor shutdown sequence to use plugin.stop

Change shutdown sequence to be triggered by plugin.stop instead of ShutdownSignal exception.
Also remove any calls to: shutdown, finished, finished?, running? or terminating?
This depends on elastic/logstash#3895

Change secret_token config to `Password` type.

Issue description

When --debug is enabled, there might be a key leak in the debug log in Logstash-code.

How to reproduce?

# config
input {
  github {
      codec => "json"
      secret_token  => "super-secret"
      port => 5010
      drop_invalid => true
  }
}
output {
    stdout {
        codec => rubydebug
    }
}

# Log before change
[2022-12-05T13:24:04,008][DEBUG][logstash.inputs.github   ] config LogStash::Inputs::GitHub/@port = 5010
[2022-12-05T13:24:04,008][DEBUG][logstash.inputs.github   ] config LogStash::Inputs::GitHub/@secret_token = "super-secret"

Logstash Crashing Due to Error from GitHub Input

Logstash information:

Please include the following information:

Logstash version (e.g. bin/logstash --version): 8.4.1
Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker): Official Logstash Docker image
How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes): docker
How was the Logstash Plugin installed: /usr/share/logstash/bin/logstash-plugin install logstash-input-github

JVM (e.g. java -version): bundled (Temurin-17.0.4+8)

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:
Several errors are logged by the GitHub input plugin before Logstash eventually crashes and restarts.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

    input {
      github {
        codec => "json"
        secret_token  => "${AUTH_SECRET}"
        port => 5010
        drop_invalid => true
      }
    }

    filter {
      mutate {
        add_field => { "[@metadata][index]" => "github_audit" }
      }
      date {
        match => [ "[repository][pushed_at]", "UNIX", "ISO8601" ]
        target => "[repository][pushed_at]"
      }
      date {
        match => [ "[repository][created_at]", "UNIX", "ISO8601" ]
        target => "[repository][created_at]"
      }
    }

    output {
      elasticsearch {
        hosts => "https://es-http:9200"
        user => "elastic"
        password => "${ELASTICSEARCH_PASSWORD}"
        index => "%{[@metadata][index]}"
        pipeline => "github-audit-pipeline"
        ssl => true
        cacert => "/usr/share/logstash/config/certs/elastic.crt"
        sniffing => false
      }
    }

Provide logs (if relevant):

[2022-09-13T12:42:51,300][ERROR][org.logstash.Logstash    ][github-audit][90769f607768f706b542e8d25df67351c6484b097a89b3b61228cdb7c577176e] uncaught exception (in thread Ruby-0-Thread-53533: :1)
java.lang.NullPointerException: Cannot invoke "org.jruby.runtime.ThreadContext.getThread()" because "this.mainContext" is null
	at org.jruby.internal.runtime.ThreadService.getMainThread(ThreadService.java:245) ~[jruby.jar:?]
	at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1995) ~[jruby.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:116) ~[jruby.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-13T12:42:51,300][ERROR][org.logstash.Logstash    ][github-audit][90769f607768f706b542e8d25df67351c6484b097a89b3b61228cdb7c577176e] uncaught exception (in thread Ruby-0-Thread-53462: :1)
java.lang.NullPointerException: Cannot invoke "org.jruby.runtime.ThreadContext.getThread()" because "this.mainContext" is null
	at org.jruby.internal.runtime.ThreadService.getMainThread(ThreadService.java:245) ~[jruby.jar:?]
	at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1995) ~[jruby.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:116) ~[jruby.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-13T12:42:51,298][ERROR][org.logstash.Logstash    ][github-audit][90769f607768f706b542e8d25df67351c6484b097a89b3b61228cdb7c577176e] uncaught exception (in thread Ruby-0-Thread-53700: :1)
java.lang.NullPointerException: Cannot invoke "org.jruby.runtime.ThreadContext.getThread()" because "this.mainContext" is null
	at org.jruby.internal.runtime.ThreadService.getMainThread(ThreadService.java:245) ~[jruby.jar:?]
	at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1995) ~[jruby.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:116) ~[jruby.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-13T12:42:51,300][ERROR][org.logstash.Logstash    ][github-audit][90769f607768f706b542e8d25df67351c6484b097a89b3b61228cdb7c577176e] uncaught exception (in thread Ruby-0-Thread-52950: :1)
java.lang.NullPointerException: Cannot invoke "org.jruby.runtime.ThreadContext.getThread()" because "this.mainContext" is null
	at org.jruby.internal.runtime.ThreadService.getMainThread(ThreadService.java:245) ~[jruby.jar:?]
	at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1995) ~[jruby.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:116) ~[jruby.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-13T12:42:51,304][ERROR][logstash.inputs.github   ][github-audit][90769f607768f706b542e8d25df67351c6484b097a89b3b61228cdb7c577176e] Error while stopping FTW::WebServer {:exception=>"closed stream", :backtrace=>["org/jruby/RubyIO.java:1326:in `syswrite'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/ftw-0.0.49/lib/ftw/server.rb:94:in `stop'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/ftw-0.0.49/lib/ftw/webserver.rb:35:in `stop'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-github-3.0.9/lib/logstash/inputs/github.rb:49:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:410:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:in `block in start_input'"]}
[2022-09-13T12:42:51,332][ERROR][logstash.javapipeline    ][github-audit][90769f607768f706b542e8d25df67351c6484b097a89b3b61228cdb7c577176e] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:github-audit
  Plugin: <LogStash::Inputs::GitHub codec=><LogStash::Codecs::JSON id=>"json_606d282c-fd52-4070-9855-959cf7fc542f", enable_metric=>true, charset=>"UTF-8">, drop_invalid=>true, id=>"90769f607768f706b542e8d25df67351c6484b097a89b3b61228cdb7c577176e", port=>5010, secret_token=>"", enable_metric=>true, ip=>"0.0.0.0">
  Error: closed stream
  Exception: IOError
  Stack: org/jruby/RubyIO.java:3755:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/ftw-0.0.49/lib/ftw/server.rb:112:in `block in each_connection'
org/jruby/ext/thread/Mutex.java:171:in `synchronize'
/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/ftw-0.0.49/lib/ftw/server.rb:110:in `each_connection'
/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/ftw-0.0.49/lib/ftw/webserver.rb:27:in `run'
/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-github-3.0.9/lib/logstash/inputs/github.rb:44:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:410:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:in `block in start_input'

Doesn't respect shutdown

Need to figure out how to kill the FTW server on shutdown.

Conflicting gem dependencies on vanilla logstash 2.1.1 install

When trying to install this plugin onto logstash 2.1.1 (other versions may be affected, haven't checked) bin/plugin install logstash-input-github fails with:

Validating logstash-input-github
Installing logstash-input-github
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-codec-plain":
  In snapshot (Gemfile.lock):
    logstash-codec-plain (= 2.0.2)

  In Gemfile:
    logstash-output-http (>= 0) java depends on
      logstash-mixin-http_client (< 3.0.0, >= 2.0.2) java depends on
        logstash-codec-plain (>= 0) java

... lots of lines truncated for readability ...

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "logstash":
  In Gemfile:
    logstash-input-github (>= 0) java depends on
      logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-input-github (>= 0) java', in any of the sources.

To reproduce this, just download the 2.1.1 tarball and try the plugin install command, similar failures should follow.

Can we have steps for using this plugin in logstash configuration and in github webhook

Hi
I want to get the details of how we can add the details of our GitHub repository in logstash configuration file and for creating a webhook in GitHub what to give the payload URL for logstash?

In my case, logstash is running on an AWS ec2 instance and I am using AWS OpenSearch service.

Please help me.

How to Use Logstash for GitHub Enterprise apliance

Hi Experts, Looking for your suggestion on below requirement.

We Have our own GitHub Enterprise which is installed as an appliance.

We need to have a central log system. which will analyze the logs. GitHub appliance will supports syslog-style log streams is supported.

How to configure log-stash for forwarding the logs ?.. Whether the log-stash is inbuilt with GHE, any other separate configuration needed and from log-stash how the logs can be monitored.

Please help me with some more details on the above part....

Versions greater than 3.0.5 cannot be installed on Logstash 7.x

Version: >=3.0.6
Operating System: Centos (logstash-docker image)
Steps to Reproduce:

Of the three most recent versions of this plugin (3.0.6, 3.0.7, 3.0.8), it appears none can be installed on the most recent Logstash. The following is from the official logstash-docker image (7.1.1):

bash-4.2$ ./bin/logstash-plugin install --version 3.0.8 logstash-input-github
Validating logstash-input-github-3.0.8
Installing logstash-input-github
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "addressable":
  In snapshot (Gemfile.lock):
    addressable (= 2.6.0)

  In Gemfile:
    logstash-input-github (= 3.0.8) java was resolved to 3.0.8, which depends on
      ftw (~> 0.0.48) java was resolved to 0.0.48, which depends on
        addressable (~> 2.3.8) java

    json-schema (~> 2) java was resolved to 2.8.1, which depends on
      addressable (>= 2.4) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
bash-4.2$ vi Gemfile
bash-4.2$ ./bin/ruby --version
jruby 9.2.7.0 (2.5.3) 2019-04-09 8a269e3 OpenJDK 64-Bit Server VM 25.212-b04 on 1.8.0_212-b04 +indy +jit [linux-x86_64]

bash-4.2$ ./bin/logstash-plugin install --version 3.0.7 logstash-input-github
Validating logstash-input-github-3.0.7
Installing logstash-input-github
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "addressable":
  In snapshot (Gemfile.lock):
    addressable (= 2.6.0)

  In Gemfile:
    logstash-input-github (= 3.0.7) java was resolved to 3.0.7, which depends on
      ftw (~> 0.0.48) java was resolved to 0.0.48, which depends on
        addressable (~> 2.3.8) java

    json-schema (~> 2) java was resolved to 2.8.1, which depends on
      addressable (>= 2.4) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

bash-4.2$ ./bin/logstash-plugin install --version 3.0.6 logstash-input-github
Validating logstash-input-github-3.0.6
Installing logstash-input-github
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "addressable":
  In snapshot (Gemfile.lock):
    addressable (= 2.6.0)

  In Gemfile:
    logstash-input-github (= 3.0.6) java was resolved to 3.0.6, which depends on
      ftw (~> 0.0.48) java was resolved to 0.0.48, which depends on
        addressable (~> 2.3.8) java

    json-schema (~> 2) java was resolved to 2.8.1, which depends on
      addressable (>= 2.4) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

bash-4.2$ ./bin/logstash-plugin install --version 3.0.5 logstash-input-github
Validating logstash-input-github-3.0.5
Installing logstash-input-github
Installation successful

The problem appears to be the ftw plugin, whose addressable dependency has also caused problems for other logstash plugins.

A proper fix will probably mean updating the ftw plugin itself, but in the meantime I wanted to record this issue, in the hopes that there's an easy workaround that I'm missing?

On shutdown the plugin raises IOError

First found in #2, and is caused by jordansissel/ruby-ftw#25

bin/logstash -e "input { github { port => 3333 } }"
Logstash startup completed
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed
IOError: closed stream
     syswrite at org/jruby/RubyIO.java:1303
         stop at /tmp/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/ftw-0.0.42/lib/ftw/server.rb:92
         stop at /tmp/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/ftw-0.0.42/lib/ftw/webserver.rb:35
     teardown at /tmp/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-input-github-0.1.5/lib/logstash/inputs/github.rb:61
  inputworker at /tmp/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:203
  synchronize at org/jruby/ext/thread/Mutex.java:149
  inputworker at /tmp/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:203
  start_input at /tmp/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:171