urbanriskmap / cognicity-reports-powertrack Goto Github PK

Current gnip rules are

(contains:flood OR contains:banjir OR contains:jakartabanjir) @petabencana OR (contains:flood OR contains:banjir OR contains:jakartabanjir) @petajkt
( contains:flood OR contains:banjir OR contains:jakartabanjir ) ( bounding_box:[106.471 -6.19140 106.79381 -5.880] OR bounding_box:[106.79381 -6.19140 107.10880 -5.880] OR bounding_box:[107.10880 -6.19140 107.175 -5.880] OR bounding_box:[106.471 -6.50213 106.79381 -6.19140] OR bounding_box:[106.79381 -6.50213 107.10880 -6.19140] OR bounding_box:[107.10880 -6.50213 107.175 -6.19140] OR bounding_box:[106.471 -6.733 106.79381 -6.50213] OR bounding_box:[106.79381 -6.733 107.10880 -6.50213] OR bounding_box:[107.10880 -6.733 107.175 -6.50213] OR bio_location:jakarta OR place:jakarta)
( contains:flood OR contains:banjir OR contains:jakartabanjir ) ( bounding_box:[107.369 -6.97964 107.68291 -6.668] OR bounding_box:[107.68921 -6.97964 107.931 -6.668] OR bounding_box:[107.369 -7.165 107.68921 -6.97964] OR bounding_box:[107.68291 -7.165 107.931 -6.97962] OR bio_location:bandung OR place:bandung)
( contains:flood OR contains:banjir OR contains:jakartabanjir ) ( bounding_box:[112.3975 -7.32570 112.71169 -7.0143] OR bounding_box:[112.71169 -7.32570 113.0318 -7.0143] OR bounding_box:[112.3975 -7.5499 112.71169 -7.32570] OR bounding_box:[112.71169 -7.5499 113.0318 -7.32570] OR bio_location:surabaya OR place:surabaya)

I would have thought this tweet reported by @prittandrani would match the following rule:

( contains:flood OR contains:banjir OR contains:jakartabanjir ) ( bounding_box:[107.369 -6.97964 107.68291 -6.668] OR bounding_box:[107.68921 -6.97964 107.931 -6.668] OR bounding_box:[107.369 -7.165 107.68921 -6.97964] OR bounding_box:[107.68291 -7.165 107.931 -6.97962] OR bio_location:bandung OR place:bandung)

as

1. contains:banjir
2. bio_location:bandung should match per docs as
   a. matches are insensitive,
   b. looks for that keyword excl. whitespace and punctuation

but

• No record in the logs.
• That user not seen before, ran

select * from twitter.invitees where user_hash=md5('infobdg');

and

select * from twitter.invitees where user_hash=md5('@infobdg');

for good measure to check.

From the log file:
2016-12-21T21:58:32.219Z - info: Error setting last contribution ID from database (is the reports table empty?)

I am out and about on iPad right now (back this arvo AEDT) so it's hard for me to check if the issue is with our copy of the database or the code.

E.g. https://twitter.com/riskmapindia/status/912779836849979393

Filter by PowerTrack activity verb:

• "post" = tweet
• "share" = retweet

see: http://support.gnip.com/sources/twitter/data_format.html

• need to remove dependency on GRASP module, and ask API for unique CARD

Default language needs to be different for Indonesia and India respectively, but is currently hard coded to EN in config, see:

This should be made a .env variable in the parent reports repo, but would require changes for both Indonesia and Indian instances.

Also note, that the names of the rules are hard coded in the powertrack filter, as per here:

https://github.com/urbanriskmap/cognicity-reports-powertrack/blob/master/PowertrackDataSource.js#L215

no error handling on gnip API errors
e.g.

2017-04-30T00:16:31.730Z - error: uncaughtException: Unable to fetch rules. Requ
est failed with status code: 403, Error: Unable to fetch rules. Request failed with status code: 403
    at Request._callback (/home/ubuntu/cognicity-reports/gnip-powertrack/node_modules/gnip/lib/rules.js:93:7)
    at Request.self.callback (/home/ubuntu/cognicity-reports/gnip-powertrack/node_modules/request/main.js:120:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/home/ubuntu/cognicity-reports/gnip-powertrack/node_modules/request/main.js:555:16)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/home/ubuntu/cognicity-reports/gnip-powertrack/node_modules/request/main.js:517:14)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
2017-04-30T00:16:31.730Z - error: Fatal error: Application shutting down
2017-04-30T00:16:31.730Z - info: Exiting with status 1

Note that "monsoon prep" is in the gnip rules but that doesn't match the telegram key word nor the internal cognicity-reports-powertrack key word (simply "prep")

https://twitter.com/petabencana/status/931485559284838400

Hi @MaanasaPriyaa you mentioned a bug in this but I can't see what the issue is?
I did a bit of testing and I can get a duplicate reply however that's only when I close down the database shortly after sending the message to the bot, due to a known timing issue where the tweet is processed but its ID not yet stored in the database. Tweets sent to the bot long enough before process termination (i.e. they are stored in the database) are ignored, and tweets sent while the bot is down only receive one reply as expected.

Note that the variable in the code is never updated after initially being set, as

• Tweet IDs are monotonically increasing, so it's sufficient to check > last seen tweet ID on load, and just keep the database UPDATEd.
• Updating the seen tweet ID is very problematic to do due to the asynchronous nature of JavaScript, i.e. you can end up with a race condition with one reply started before another has finished.

The small race condition identified above is very low priority as it's not often that we end up taking the bot down (typically once a month for kernel patches on the underlying instance).

Is there anything else you've noticed that seems odd?

2017-02-08T02:50:11.896Z - error: Tweeting "@tt221101 null" with params "{"in_reply_to_status_id":"829160325911961601","media_ids":"Hi! Laporan menggunakan link ini, terima kasih. https://petabencana.id/cards/S12DPZ_dg"}" failed: Error: HTTP Error 400: Bad Request, API message: {"errors":[{"code":44,"message":"media_ids parameter is invalid."}]}

@talltom I've rolled back on production while you work on a fix since we can't test on dev (#10)

On restart following messages get sent:

2017-09-28T17:23:10.337Z - info: connectStream: Updating rules...

2017-09-28T17:23:10.390Z - info: DB connection successful

2017-09-28T17:23:10.397Z - info: Error setting last contribution ID from database (is the reports table empty?)

At the moment we can't use our dev environment for testing replies from the bot, because in that case we'd need to have code and config for whitelisting which accounts to send replies to.

https://github.com/urbanriskmap/cognicity-reports-powertrack/blob/chennaiChanges/PowertrackDataSource.js#L208
Note the comment to use "within_location" is out of date with respect to the key word used in the GNIP rules https://github.com/urbanriskmap/cognicity-reports-powertrack/blob/chennaiChanges/powertrack-config.js#L54

Note: do not deploy before launch

Update PowerTrack rules to watch for @petabencana or @petajkt

case insensitive

Remove /location from the following:

var cardLink = self.config.front_end.card_url_prefix + disasterType + "/" + body.cardId + '/location';

the key words have been hard coded in in various places; we should make these variables, so it will be easy to deploy ( / redeploy with new changes).