zkillboard / redisq Goto Github PK

View Code? Open in Web Editor NEW

36.0 36.0 4.0 184 KB

A simple queue service for "pushing" killmails from zKillboard.

Home Page: https://redisq.zkillboard.com/listen.php

License: Other

JavaScript 100.00%

redisq's Introduction

zKillboard

zKillboard is a killboard created for EVE-Online, for use on zkillboard.com, but can also be used for single entities.

Fun fact: zKillboard.com was originally called killwhore.com until it was discovered that the EVE Online forums censored the word whore.

Installation

This is a set of code that is beta and is constantly in flux. Which means it is a work in progress. It lacks complete documentation and is currently not meant for use by those who do not have a lot of experience in setting up PHP, TokuDB (a derivative of MongoDB), and Redis. Please feel free to attempt to install zKillboard on your own server, however, we are not responsible for any difficulties you come across during installation and continuing execution.

Contact

Via Twitter at @zkillboard, via the ticket system itself on zkillboard.com (you have to log in), send an email to [email protected], or you can talk to Squizz on TweetFleet.

Minimum requirements

To be updated.

Cronjobs

zKillboard comes with a script that automates the cron execution. It keeps track of when each job has been run and how frequently it needs to be executed. Just run it every minute via cron or a similar system:

* * * * * /var/killboard/zkillboard.com/cron/cron.sh

The cron.sh file handles the output as well as rotating of the logfiles in /cron/logs/

Credits

zKillboard is released under the GNU Affero General Public License, version 3. The full license is available in the AGPL.md file. zKillboard also uses data and images from EVE-Online, which is covered by a separate license from CCP. You can see the full license in the CCP.md file. It also uses various 3rd party libraries, which all carry their own licensing. Please refer to them for more info.

License and Copyright

Licensing for all files in this repository can be found in AGPL.md

History and previous versions

zKillboard.com came as the brainchild of Squizz Caphinator who wanted to improve upon Eve-Dev Killboard. Squizz decided to write a new killboard completely from scratch and began the zKillboard project. Karbowiak of eve-kill.net eventually joined into the project, contributed much code, created a repository on Github, and announced zKillboard as the new Beta killboard for eve-kill.net. zKillboard matured and gained a fanbase, and of course, haters. As time went on Squizz and Karbowiak had some differences and Squizz forked his code into this repository and made this new repository the primary code base for zkillboard.com. After about a year Squizz then began dabbling in NoSQL, as it seemed the perfect database for the type of data consumed by zKillboard. Two months of heavy coding and extreme database changes, the repository zKillboard/zKillboard was updated to make the code public to the masses with the various NoSQL changes.

redisq's People

Contributors

Stargazers

Watchers

Forkers

vivace-io orb staropera mr00anderson

redisq's Issues

Service does not collects KMs between calls

I am expecting that service will collect all killmails between API invocations.

I've used '?queueID=54689e7ff0b3cebfa1' id for each call but have not received more than one record in response.

This looks like a defect.

And the second question.
Does there are possible to configure redisq.zkillboard.com api to send response in short format (id, hash), like the history API (zkillboard.com/api/history/)

Rate Limit

I've encountered the 429 rate limit. There was a bug in the code that didn't close the connections it seems. I've read through the issues, and you seem to talk about documentation. The only documentation regarding redisq requests i've seen is the README in this project. How do i go about unblocking my bot? Is there some ban timeout i have to wait out?

Getting blocked by CloudFalre

Until some recent changes it worked correclty for me. I were able to query the RedisQ with https://redisq.zkillboard.com/listen.php?ttw=1 and now I'm getting blocked. I even added a 3 seconds delay on getting null data from the request but it won't work. I'm still getting banned.

Please advice on possible solution for this issue. I want to suppliy my bot with the ZKB killmail livestream.

PS: bot uses different a bit randomized http header for request for each bot instance to distinct them

Rate Limit

in the FAQ it states that no rate limit is applied and to hit it as hard as you like; however even with a 100ms pause when pulling kills i occasionally get a 429 (Too Many Requests) response.

What is the maximum limit ? I dont want to overload the server however also dont want to get behind in the tool.

Thanks,

522 - origin connection timeout

Hey guys,

I'm using redisq and mostly it works just fine. But the last few days ive seen some 522s. I saw one this morning, and one today at 18:30 GMT.

Server error: GET http://redisq.zkillboard.com/listen.php?ttw=2&queueID=my-queue
resulted in a 522 Origin Connection Time-out response:

My ttw is 2, but since 522 seems like its a "we cant find the server at all" issue. Are you guys seeing any logs on your end?

Many timeouts after API rewrite

AA-Killtracker has been updated and is now also using the QueueID in all request.

However, we are seeing a lot of read timeouts with the new implementation of the RedisQ API. Some request are going through, but many run into the timeout. Some of my users are reporting they do not get any successful requests, just timeouts.

Example:

Traceback (most recent call last):
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/killtracker/tasks.py", line 53, in run_killtracker
killmail = Killmail.create_from_zkb_redisq()
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/killtracker/core/killmails.py", line 411, in create_from_zkb_redisq
response = requests.get(
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='redisq.zkillboard.com', port=443): Read timed out. (read timeout=30)

Is RedisQ working?

It's always { package: null }

url slashes are escaped

The JSON has escaped slashes when viewing https://redisq.zkillboard.com/listen.php

Not sure if this was intentional but it adds extra processing steps to a consumer.

Inconsistent package format?

So one of my apps which uses RedisQ started throwing a bunch of errors. I checked it out and it seemed like the package format had changed. No problem, I rewrote part of it to compensate.

I kept on getting the odd error. It seemed like some of the packages arriving were still coming in the old format. Okay no problem, I wrote a bit of logic to handle both formats.

Now it seems that my app is getting all packages in the "old" format.

However, when I point my browser at RedisQ, the packages I see are in the 'new' format. I captured some examples:

What my app sees: https://hastebin.com/eduyawuhol.json
What I see in my browser: https://hastebin.com/honineyepa.json

Is there any reason why this might be happening?

Support for ESI/alternative responses

Currently, RedisQ sends Killmails in the CREST format (which is awesome, by the way), however with it getting phased out in the long term, it would be nice to consider a way of providing ESI responses instead of CREST in a way that would not break applications in the mean time.

I'm wondering if it would be feasable to do one of two things:

(A) Support a URL parameter format=esi which would instruct RedisQ to reply with ESI (and the appended zkb JSON field) or...
(B) Support a URL parameter format=zkb which simply returns the zkb JSON field, with the addition of the kill ID in that field.

The first seems like it would add a bit of extra overhead on Redis if you stored the ESI format along side CREST's, or converted CREST to ESI. The second idea seems slightly more doable, especially considering there exists no rate limit for ESI so the leg work could easily be left to the client in retrieving the data.

What are your thoughts?

Package contains invalid urls for item/ship icons

For example,

"shipType":{
    "id_str":"29336",
    "href":"https://crest-tq.eveonline.com/inventory/types/29336/",
    "id":29336,
    "name":"Scythe Fleet Issue",
    "icon":{
        "href":"http://imageserver.eveonline.com/Type/29336_128.png"
    }
}

The href given for the ship icon leads to a 404 since the EVE Online Image Server inventory type icons are only available in 64x64 and in 32x32.

The icon should lead to "http://imageserver.eveonline.com/Type/29336_64.png"

Can I subscribe to just my pilot's / character's / alliance's killmails?

That FAQ.. is this in the works yet?

Faction Information

Is it possible for you to include faction information as well?

Using it here: https://shadowcodex.github.io/fw-eve-map/

Feature Request - Another way to keep track of subsequent requests (other than IP)

I want to use this with AWS Lambda on a schedule, but I don't think the same IP will be used with Lambda.

Would it be a good idea to add another way to track this, maybe with a query parameter or something?

Alternatively, I may try to manually set one of the headers that are tracked in https://github.com/zKillboard/RedisQ/blob/master/src/IP.php.

CORS

I am hitting the endpoint once every second, and wanted to do it that quick to grab as many kill mails as I can in a timely fashion.

However I noticed that when there is no killmail to report I get a CORS error... So the console starts filling up. I've tried setting crossDomain to true in the jQuery ajax call, but that isn't working.

Is there something configured incorrectly on the redisq server that is causing the null response to have a limiting Access-Control-Allow-Origin header?

Service does not collect all killmails

I make discord bot and service lost some killmails

Kill order doesn't make any sense

figured it out

fitted isk doesn't match value on website

I hit the redisQ endpoint and saw:

{
    "package":
        {
            "killID": 93719959,
            "killmail":
                {...},
            "zkb":
                {
                    "awox": false,
                    "destroyedValue": 380199332.7,
                    "droppedValue": 18083376.8,
                    "fittedValue": 371963885.27,
                    "hash": "551c7df74dbfdad8841b0f0947766f25ab149bb7",
                    "href":
                        "https://esi.evetech.net/v1/killmails/93719959/551c7df74dbfdad8841b0f0947766f25ab149bb7/",
                    "labels": ["cat:6", "2+", "npc", "highsec"],
                    "locationID": 50006549,
                    "npc": true,
                    "points": 1,
                    "solo": false,
                    "totalValue": 398282709.5
                }
        }
}

The website lists the fitted value as 63,479,570.62 ISK, much lower than the 371M isk listed in the redisq result. I've looked around but have not been able to determine the cause of this difference. Can you explain it? Do I need to do another api call to zkillboard to get the corrected value?

RedisQ down?

Hi
I have been using RedisQ for my services, but since 2017-01-05 19:10:10 he is no providing killmails, but returning NULL all the time.

I tried from different machines. Always the same result.

The result of any request is:

{
      package : null
}

But we great if this issue could be investigates.
Cheers,
Erik

Stability issues with redisq API

Our app is using the RedisQ API to fetch killmails. It is called killtracker and used by many alliances incl. my own.

Since a few days we are seeing re-occurring failures when accessing the API. These are mainly HTTP 500s and read timeouts.
Those issues are reported by several alliances and I can see them in my logs too. My logs shows ~ 650 HTTP 500s and ~150 read timeouts over 24 hrs.

The app has been working without these kind of issues for 3+ years, so we currently assume the cause of the failures lies with the API.

We would very appreciate if you could investigate these failures or maybe let us now if there is anything we can do on our side to mitigate them.

Here is a snapshot of those errors over the last 24 hrs:

Here are some examples from the logs (this is Django / Python).

HTTP Error:

Traceback (most recent call last):
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/killtracker/tasks.py", line 53, in run_killtracker
killmail = Killmail.create_from_zkb_redisq()
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/killtracker/core/killmails.py", line 410, in create_from_zkb_redisq
response.raise_for_status()
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://redisq.zkillboard.com/listen.php?ttw=5

Timeout:

Traceback (most recent call last):
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.10/ssl.py", line 1274, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.10/ssl.py", line 1130, in read
return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/urllib3/connectionpool.py", line 538, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='redisq.zkillboard.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/killtracker/tasks.py", line 53, in run_killtracker
killmail = Killmail.create_from_zkb_redisq()
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/killtracker/core/killmails.py", line 394, in create_from_zkb_redisq
response = requests.get(
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/allianceserver/venv/auth/lib/python3.10/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='redisq.zkillboard.com', port=443): Read timed out. (read timeout=30)

The Killtracker app is open source and can be found here: https://gitlab.com/ErikKalkoken/aa-killtracker