papertrail / papertrail-cli Goto Github PK

It seems like there is no activity although there are issues and PRs that are (were) easy to merge.

Should this gem be considered abandoned? I hope not...

currently I'm using

papertrail-systems() {
    curl -s -H "X-Papertrail-Token: `cat ~/.papertrail.yml | sed -r 's/token://'`" https://papertrailapp.com/api/v1/systems.json | python -m json.tool | grep '"name":' | sed -r 's/ +"name": "(.*)",/\1/'
}

in my .bashrc file.

But it was a faff to setup, would be greate if there was a simpler way of getting this.

papertrail --min-time '12:30' --max-time '14:30' will return results after 14:30 today.

A workaround is to explicitly specify today:
papertrail --min-time '12:30' --max-time 'today 14:30'

Maybe this is intended, but say I run papertrail blah | head -n 10 and the output has more than 10 lines, I get:

/Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/cli.rb:176:in `write': Broken pipe @ io_write - <STDOUT> (Errno::EPIPE)
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/cli.rb:176:in `puts'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/cli.rb:176:in `display_result'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/cli.rb:146:in `block in query_time_range'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/connection.rb:207:in `block (2 levels) in each_event'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/connection.rb:197:in `each'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/connection.rb:197:in `block in each_event'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/connection.rb:196:in `loop'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/connection.rb:196:in `each_event'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/cli.rb:142:in `query_time_range'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/lib/papertrail/cli.rb:127:in `run'
    from /Users/burke/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/papertrail-0.9.14/bin/papertrail:5:in `<top (required)>'
    from /Users/burke/.rbenv/versions/2.3.0/bin/papertrail:23:in `load'
    from /Users/burke/.rbenv/versions/2.3.0/bin/papertrail:23:in `<main>'

The -j output includes a time offset for generated_at, even though the sending device's time zone is not reliably known.

Thanks to @keturn for discovering this.

After adding gem 'papertrail' v 0.9.2 to a rails 3 application, I get the following error on startup:

cannot load such file -- papertrail/search_client (LoadError)

This file seems to be required from papertrail.rb in the gem's root.

NOTE: this issue blocks Papertrail addon on Heroku.

$ papertrail 
/usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/always_verify_ssl_certificates-0.3.0/lib/always_verify_ssl_certificates.rb:17:in `initialize': wrong argument (NilClass)! (Expected kind of OpenSSL::SSL::SSLContext) (TypeError)
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/always_verify_ssl_certificates-0.3.0/lib/always_verify_ssl_certificates.rb:17:in `new'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/always_verify_ssl_certificates-0.3.0/lib/always_verify_ssl_certificates.rb:17:in `connect'
    from /usr/local/rvm/rubies/ruby-1.9.2-p180-good/lib/ruby/1.9.1/net/http.rb:637:in `do_start'
    from /usr/local/rvm/rubies/ruby-1.9.2-p180-good/lib/ruby/1.9.1/net/http.rb:626:in `start'
    from /usr/local/rvm/rubies/ruby-1.9.2-p180-good/lib/ruby/1.9.1/net/http.rb:1168:in `request'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/faraday-0.5.7/lib/faraday/adapter/net_http.rb:45:in `call'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/faraday-0.5.7/lib/faraday/request.rb:85:in `run'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/faraday-0.5.7/lib/faraday/request.rb:27:in `run'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/faraday-0.5.7/lib/faraday/connection.rb:177:in `run_request'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/faraday-0.5.7/lib/faraday/connection.rb:66:in `get'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/papertrail-cli-0.6.1/lib/papertrail/search_client.rb:36:in `search'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/papertrail-cli-0.6.1/bin/papertrail:53:in `search_and_print'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/papertrail-cli-0.6.1/bin/papertrail:42:in `run'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/gems/papertrail-cli-0.6.1/bin/papertrail:80:in `'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/bin/papertrail:19:in `load'
    from /usr/local/rvm/gems/ruby-1.9.2-p180-good/bin/papertrail:19:in `

On a 32-bit Amazon Linux machine, I'm getting the following error with papertrail 0.9.3:

/usr/lib/ruby/gems/1.8/gems/faraday-0.8.2/lib/faraday/adapter/net_http.rb:21: uninitialized constant Faraday::Adapter::NetHttp::Zlib (NameError)
    from /usr/lib/ruby/gems/1.8/gems/faraday-0.8.2/lib/faraday.rb:67:in `const_get'
    from /usr/lib/ruby/gems/1.8/gems/faraday-0.8.2/lib/faraday.rb:67:in `lookup_middleware'
    from /usr/lib/ruby/gems/1.8/gems/faraday-0.8.2/lib/faraday/builder.rb:146:in `use_symbol'
    from /usr/lib/ruby/gems/1.8/gems/faraday-0.8.2/lib/faraday/builder.rb:108:in `adapter'
    from /usr/lib/ruby/gems/1.8/gems/papertrail-0.9.3/lib/papertrail/connection.rb:32:in `initialize'
    from /usr/lib/ruby/gems/1.8/gems/faraday-0.8.2/lib/faraday/connection.rb:48:in `initialize'
    from /usr/lib/ruby/gems/1.8/gems/papertrail-0.9.3/lib/papertrail/connection.rb:30:in `new'
    from /usr/lib/ruby/gems/1.8/gems/papertrail-0.9.3/lib/papertrail/connection.rb:30:in `initialize'
    from /usr/lib/ruby/gems/1.8/gems/papertrail-0.9.3/lib/papertrail/cli_add_system.rb:54:in `new'
    from /usr/lib/ruby/gems/1.8/gems/papertrail-0.9.3/lib/papertrail/cli_add_system.rb:54:in `run'
    from /usr/lib/ruby/gems/1.8/gems/papertrail-0.9.3/bin/papertrail-add-system:5
    from /usr/bin/papertrail-add-system:19:in `load'
    from /usr/bin/papertrail-add-system:19

Adding require 'zlib' to lib/papertrail/connection.rb appears to fix the issue. The 64-bit machines that I have tried this on don't seem to be hitting this problem.

1. I made sure the runtime dependencies are installed.
2. I have the correct user/pass in ~/.papertrail

$ papertrail
/var/lib/gems/1.8/gems/always_verify_ssl_certificates-0.3.0/lib/always_verify_ssl_certificates.rb:36:in connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError) from /var/lib/gems/1.8/gems/always_verify_ssl_certificates-0.3.0/lib/always_verify_ssl_certificates.rb:36:inconnect'
...

When a switch or value isn't recognized, OptionParser::InvalidOption exception is raised. It'd be nice to rescue those and print a more helpful error message.

$ papertrail --hi
/Users/larry/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/papertrail-0.9.12/lib/papertrail/cli.rb:69:in `run': invalid option: --hi (OptionParser::InvalidOption)
    from /Users/larry/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/papertrail-0.9.12/bin/papertrail:5:in `<top (required)>'
    from /Users/larry/.rbenv/versions/2.2.3/bin/papertrail:23:in `load'
    from /Users/larry/.rbenv/versions/2.2.3/bin/papertrail:23:in `<main>'

Hi,

Perhaps an easy to implement and non breaking feature could be to first look for a .papertrail file in the current directory, and if found, use it as the config source rather than the one in the home directory?

I know of the --configfile PATH argument, but it is less convenient for day to day use.

Having the dotfile config would allow to easily have multiple accounts connected, and have a config file per project.

EDIT:

It seems like you already support it...

It should be mentioned in the README :)

I am behind proxy, an i am not able to connect to papertrail from papertrail-cli due to this. How can i configure proxy? I am using cli in wondows. Ruby gem have proxy configuration and is working fine (as i was able to install the cli-plugin).

Hi, would it be possible for you to add a license file to specify under what license we can use this software?

Thanks a lot in advance

I've recently install papertrail-cli, I have placed my token in my ~/.papertrail.yml file and when I execute papertrail from the command line I get the following error:

/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/net/http/response.rb:120:in error!': 403 "Forbidden" (Net::HTTPServerException)
from /Library/Ruby/Gems/2.3.0/gems/papertrail-0.11.0/lib/papertrail/http_client.rb:143:in on_complete' from /Library/Ruby/Gems/2.3.0/gems/papertrail-0.11.0/lib/papertrail/http_client.rb:84:in request'
from /Library/Ruby/Gems/2.3.0/gems/papertrail-0.11.0/lib/papertrail/http_client.rb:62:in get' from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/forwardable.rb:202:in get'
from /Library/Ruby/Gems/2.3.0/gems/papertrail-0.11.0/lib/papertrail/search_query.rb:34:in next_results_page' from /Library/Ruby/Gems/2.3.0/gems/papertrail-0.11.0/lib/papertrail/cli.rb:139:in run'
from /Library/Ruby/Gems/2.3.0/gems/papertrail-0.11.0/bin/papertrail:5:in <top (required)>' from /usr/local/bin/papertrail:22:in load'
from /usr/local/bin/papertrail:22:in <main>'

Just FYI, I'm running a Mac, version 10.14.5

the colortail command and sample file doesn't color the log the same way heroku papertrail does it.

you really should just build the color functionality into papertrail-cli

I'm trying to implement logging in my python (FastAPI) web app with severity levels and deploy in Heroku and Papertrail's log drain add-on. However, with python's logging module as configured below:

import logging
import sys

logging.basicConfig( 
    format="[%(asctime)s] %(filename)s:%(lineno)-5d %(levelname)-8s: %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S", 
    level=logging.INFO,
    handlers=[
        logging.StreamHandler(sys.stdout)
    ]
)

logger = logging.getLogger("plant-pot-backend")

@router.post('/add')
async def create(new_pot: PotHttpReq):
    try:
        pot_id = new_pot.id
        new_pot = new_pot_registration(pot_id) 
        pots_collection.document(pot_id).set(new_pot.dict())
        logger.warning("New pot added: {}".format(pot_id))
        return {"success": True}
    except Exception as e:
        print(e)
        return f"An Error Occured: {e}"

Specifically, I wish to get the same type of logging format, as shown in the output line with the green INFO. This allows me to filter these severity levels through Papertrail as well.

However, even with the python's logging config above, in Papertrail, the std output is still categorized as under INFO even though it is a WARNING log, as seen in the picture below.

If everything is under INFO type log in the Heroku console and logs, I can't filter them in log drains.

Any help on this?

We should already do this. Credit to @jpirkey for this idea.

If I want to remove logs with the keyword Status I would do:

papertrail "-Status"

But instead I get the following error:

Search "tatus" not found

My guess is that is parsing the -S as interpreting it as the -S flag or --search

Is that correct?

My API token has access to multiple organizations. I can't figure out how to control which one papertrail-cli accesses.

Hi there

we have a quite severe issue with your API. We're using the "papertrail-cli" gem in the latest version (0.11.0) to register new systems.

This worked perfectly fine until recently. I was able to track this down to this part: https://github.com/papertrail/papertrail-cli/blob/master/lib/papertrail/connection.rb#L54-L58

For a system name like foo.bar.com this returns an internal server error. But it should not :)

Any help on this is highly appreciated.

Kind regards
Dirk

The Papertrail::Connection class uses faraday and faraday_middleware to make requests against Papertrail's HTTP API. Here's an example.

While this functions fine, there's a challenge. This papertrail-cli gem (or at least its Connection class) needs to be vendored in papertrail-heroku-plugin.

In this case, vendoring means not using native code (reason). That means parsing the API responses with okjson, which faraday_middleware won't support (reason).

This issue is to:

• remove the dependencies on faraday, faraday_middleware, and yajl-ruby
• replace them with Net::HTTP to make requests and okjson to parse the JSON

Also, it needs to continue working on MRI 1.8.7.

Any chance of a Windows port, either a single 32-bit that works WinXP+ or both Win32 and Win64 if you like?

If --follow and --{min,max}-time flags are given, it will run a follow query and ignore the time flags. A warning message should be printed that the given times will be ignored and that a follow query is running.

It would be very handy if you could limit the number of rows returned.

e.g: papertrail -n 1 -s systemname

To return just the latest log line for a system.

Hey,

I have an issue starting the client:

➜  ~  papertrail
/Users/philmod/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:799:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (Faraday::Error::ConnectionFailed)

How can I fix it?

Thanks,
Philmod

Please switch to a password encryption model for your yml file.

Not sure if this is a complicated request or not (maybe here?), but I would love to be able to set the timezone of the output in the config file.

My use case is this - I try to keep all linux machines I am working on as UTC - to maintain a common standard, and to be identical to production. But naturally, I am not at UTC timezone, and always need to do the "this minus that" to get the actual time the event happend (and dont get me started on DST....)

If actual (DST-sensitive) timezone is a problem, then the second best thing would be a manual time offset setting.

Howdy, the latest update is causing all runs of the papertrail client to fail with the error in the subject. We're locking our version to 0.10.1 to prevent the issue for now, but thought you all might want to know.

This is our cookbook line for running the client

execute "PAPERTRAIL_API_TOKEN=#{node['remote_syslog2']['papertrail_api_token']} papertrail-add-system --system #{node['hostname']} --destination-port #{node['remote_syslog2']['remoteport']}"

I tried for example papertrail --min-time='16:06:53' (9am Pacific, but the log timestamps showed up as UTC). Papertrail prints no results and exits 0.

I know times are hard but it'd be nice if there was a warning that you were unlikely to get any results, as the timestamp is in the future, and/or an error.

Hello,

Would be great if I could install the papertrail CLI via Homebrew on OSX! Think we could add this?

Thanks.

When I invoke papertrail CLI utility (latest release) and I use more than one system by adding more than one -s switches and one of these systems is just yet in preparation, papertrail complains that such a system is (yet) not found.

$ papertrail -s ip-10-0-1-204 -s ip-10-0-1-54 -f --
System "ip-10-0-1-54" not found

Here the first system has already send to papertrail, but the second not yet.

If I switch both parameters and put the yet to be provisioned one in front, this works w/o complains.

In any case it would be good if not yet existing systems are considered into coming online especially if there are any other systems online.

Hey guys,

I'm trying to run the papertrail-cli to tail some logs programatically and am getting the following error:

/usr/lib/ruby/2.2.0/net/http.rb:923:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
    from /usr/lib/ruby/2.2.0/net/http.rb:923:in `block in connect'
    from /usr/lib/ruby/2.2.0/timeout.rb:73:in `timeout'
    from /usr/lib/ruby/2.2.0/net/http.rb:923:in `connect'
    from /usr/lib/ruby/2.2.0/net/http.rb:863:in `do_start'
    from /usr/lib/ruby/2.2.0/net/http.rb:852:in `start'
    from /usr/lib/ruby/2.2.0/net/http.rb:1375:in `request'
    from /usr/lib/ruby/2.2.0/net/http.rb:1133:in `get'
    from /usr/lib/ruby/gems/2.2.0/gems/papertrail-0.9.15/lib/papertrail/http_client.rb:47:in `get'
    from /usr/lib/ruby/gems/2.2.0/gems/papertrail-0.9.15/lib/papertrail/search_query.rb:16:in `search'
    from /usr/lib/ruby/gems/2.2.0/gems/papertrail-0.9.15/lib/papertrail/cli.rb:129:in `block in run'
    from /usr/lib/ruby/gems/2.2.0/gems/papertrail-0.9.15/lib/papertrail/cli.rb:128:in `loop'
    from /usr/lib/ruby/gems/2.2.0/gems/papertrail-0.9.15/lib/papertrail/cli.rb:128:in `run'
    from /usr/lib/ruby/gems/2.2.0/gems/papertrail-0.9.15/bin/papertrail:5:in `<top (required)>'
    from /usr/bin/papertrail:23:in `load'
    from /usr/bin/papertrail:23:in `<main>'

any idea?

With any query that would return more than 100 results the number of results is capped at 100. Is there a way to configure this via the CLI to increase the cap?

Just install on Ruby 2.0. After setting my token and running papertrail this is what i get:

/Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/net/http.rb:918:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (Faraday::Error::ConnectionFailed)
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/net/http.rb:918:in `block in connect'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/net/http.rb:918:in `connect'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/net/http.rb:851:in `start'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/net/http.rb:1367:in `request'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/net/http.rb:1126:in `get'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/faraday-0.8.8/lib/faraday/adapter/net_http.rb:73:in `perform_request'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/faraday-0.8.8/lib/faraday/adapter/net_http.rb:38:in `call'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/faraday-0.8.8/lib/faraday/request/url_encoded.rb:14:in `call'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/faraday-0.8.8/lib/faraday/connection.rb:253:in `run_request'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/faraday-0.8.8/lib/faraday/connection.rb:106:in `get'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/papertrail-0.9.7/lib/papertrail/search_query.rb:12:in `search'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/papertrail-0.9.7/lib/papertrail/cli.rb:103:in `run'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/papertrail-0.9.7/bin/papertrail:5:in `<top (required)>'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/bin/papertrail:23:in `load'
    from /Users/bhilkert/.rbenv/versions/2.0.0-p195/bin/papertrail:23:in `<main>'

I've run into this issue a few times now, using the latest version of the gem - Below is the exception in question:

/usr/lib/ruby/1.9.1/net/http.rb:2632:in `error!': 503 "Service Unavailable" (Net::HTTPFatalError)
        from /var/lib/gems/1.9.1/gems/papertrail-0.9.12/lib/papertrail/http_client.rb:105:in `on_complete'
        from /var/lib/gems/1.9.1/gems/papertrail-0.9.12/lib/papertrail/http_client.rb:45:in `get'
        from /var/lib/gems/1.9.1/gems/papertrail-0.9.12/lib/papertrail/search_query.rb:16:in `search'
        from /var/lib/gems/1.9.1/gems/papertrail-0.9.12/lib/papertrail/cli.rb:112:in `block in run'
        from /var/lib/gems/1.9.1/gems/papertrail-0.9.12/lib/papertrail/cli.rb:111:in `loop'
        from /var/lib/gems/1.9.1/gems/papertrail-0.9.12/lib/papertrail/cli.rb:111:in `run'
        from /var/lib/gems/1.9.1/gems/papertrail-0.9.12/bin/papertrail:5:in `<top (required)>'
        from /usr/local/bin/papertrail:23:in `load'
        from /usr/local/bin/papertrail:23:in `<main>'

This should just be a matter of a try/catch block around HTTPFatalError, with a retry and retry counter (say, 3 failures in a row = halt application?). I can submit a PR for this.

I've spent 5 minutes going through the Papertrail UI to find a place where I can get my API key or generate one, and I have no clue.

% papertrail                
Authentication token not found. Set config file "token" attribute or PAPERTRAIL_API_TOKEN.

Because apparently Papertrail's UI is convoluted, having a direct link in this error message would be a big help.

The tool requires Ruby to be installed. This immediately disqualifies it from being described as a 'standalone binary'.

Unless I am missing something?

The documentation should be updated to remove this.

Every time we start up the application, I get this error. We're running on the Cedar stack, and using the Papertrail "Forsta" plan.

$ heroku run console --app primaryapp-production
! Unable to load plugin papertrail-heroku-plugin.
! Search for help at: https://help.heroku.com
! Or report a bug at: https://github.com/heroku/heroku/issues/new

Error:       papertrail gem is missing. Please install papertrail: gem install papertrail (RuntimeError)
Backtrace:   /Users/sparky/.heroku/plugins/papertrail-heroku-plugin/init.rb:4
             /usr/local/heroku/lib/heroku/plugin.rb:64:in `load'
             /usr/local/heroku/lib/heroku/plugin.rb:64:in `load_plugin'
             /usr/local/heroku/lib/heroku/plugin.rb:42:in `load!'
             /usr/local/heroku/lib/heroku/plugin.rb:39:in `each'
             /usr/local/heroku/lib/heroku/plugin.rb:39:in `load!'
             /usr/local/heroku/lib/heroku/command.rb:17:in `load'
             /usr/local/heroku/lib/heroku/cli.rb:24:in `start'
             /usr/bin/heroku:25

Command:     heroku run console --app primaryapp-production
Plugins:     papertrail-heroku-plugin
Version:     heroku-toolbelt/2.30.2 (universal-darwin11.0) ruby/1.8.7

I did gem install papertrail but that doesn't fix the issue.

Right now it doesn't appear that we are setting any timeouts on the HTTP client, but we really should.

Steps to reproduce:

1. Have many log messages. We generate 10GB of messages per day
2. Run papertrail -f
3. Observe that there are occasional gaps in the messages of 1-2 seconds. For example, we'll see a message from 12:01:01, followed by a message from 12:01:03 (without any of the messages from 12:01:02).

I assume this is by design! I'm guessing that if there are a ton of messages, you didn't want to overwhelm the servers or delay the CLI with too much data.

Regardless, I'd like a realtime (or near realtime) firehose to parse. What is the best way to get that data? My ideas:

1. Use the "archive to s3" function, but that forces a delay of 1-2 hours and is unusable for this project
2. Manually "chunk" the data on my side, by requesting 5 minutes of data at a time (so at 12:05, I request the data for 12:00 til 12:05, etc)
3. ...?

Is there any way to get papertrail -f to stop dropping messages? If not, how would you develop a realtime-ish system?

I would like to be able to delete groups via papertrail-cli.

Example usage: papertrail-remove-group -g mygroup

That would allow us to test new changes on multiple versions of Ruby and avoid situations where a method is called that doesn't exist. For example, here force_encoding doesn't exist on 1.8.7.

I would like to suggest refactoring the code to improve coding standards, and hopefully reach a 4.0 CodeClimate GPA (or at least > 3.0).

Right now, it is at GPA 1.15, and has many issues as can be seen on the Issues page at CodeClimate

I might be able to participate, if there is interest - but it looks like more than a one man job.

Right now searches are done across all systems.

papertrail --min-time "20 minutes ago" - works
papertrail -f --min-time "20 minutes ago" - doesn't work

How about providing short flags for these --min-time and --max-time?

Possible options:

• -n for min, -x for max
• -m for min, -x for max
• -m for min, -M for max

Also - would be nice to have a flag that shows just the last N lines. Perhaps a -l (lowercase L) or -n, so we can simply $ pt -l5

Running papertrail --min-time today just comes up empty. Could it be that it is translated to today at the end of the day?

Leaving papertrail -f running for a long period of time appears to accumulate memory proportionate to the amount of log messages seen.

For instance, I had a couple of papertrail instances running this morning, and saw them each using over 300MB of RPRVT. Even tailing a fairly low traffic host for about an hour this afternoon I see ruby at ~95MB of RPRVT.

~$ papertrail -v
papertrail version 0.9.17

This would make this tool a LOT more useful! Examples include when finding error rows I basically always want to see the surrounding context, which is not possible at the moment.

Right now, there's no way to bulk-download many archive files at once. The current method is a shell script. This is to create a new papertrail-archives CLI program which downloads either the newest N archive files or all archives before/after/between date(s)/time(s).

Papertrail has an API endpoint at /api/v1/archives.json which returns this response. It uses the same authentication as the other API endpoint.

See bin/papertrail-add-group and lib/papertrail/cli_add_group.rb for examples how to structure a new standalone command and supporting library. This will be bin/papertrail-archives and something like lib/papertrail/cli_archives.rb.

Arguments

It requires 1 of these 3 arguments:

--newest N
--min-time MIN
--max-time MAX

Examples:

papertrail-archives --newest 15
papertrail-archives --min-time '2015-01-15 00:00:00' 
papertrail-archives --min-time '2015-01-15 00:00:00' --max-time '2015-01-19'

Date parsing

Use the same behavior as the min-time and max-time arguments to the main papertrail command. That is, call parse_time on the CLI input and then pass those return values as API query params.

The client doesn't need to do any time comparisons nor anything beyond what's already in the codebase.

API client

This will entail a new lib/archives.rb for doing the API query (very similar to lib/search_query.rb), and probably a new lib/archive.rb model (very similar to lib/event.rb).

Note: The server doesn't currently honor min_time and max_time query parameters. Those will be added before this is released.

Behavior

• If newest is given, hit the API target with no params and download the first n elements/files in the response array
• If min-time and/or max-time are given, pass min_time and/or max_time to server and download all resulting elements/files. The the server will handle the time constraints.
• If newest and either of the other args are given, or no arguments are given, refuse to run

In the API response, the URL to download a given archive file is in the download href hypermedia URL - example.

Downloads

Files may be large. I propose adding something like this as a new HttpClient.download method, since it only uses Net::HTTP but writes the files gradually.

Writing files

• Write to current directory using the filename API response attribute. The filename values are unique.
• Output a line each time a file completes. We'll figure out what it says later, but it'll probably just be the datestamp.

Notes

• All of the download URLs use at least 1 redirect