Persistently store scraped tweets about liveactionmap HOT 14 OPEN

I am working on a fix for duplicate tweets.

from liveactionmap.

We should also store the time the tweet was created and discard tweets after a certain time or allow the user to select a time-range. The latter would probably require the map to be generated client-side.

from liveactionmap.

I agree a json-file is probably the best option. I don't think we should generate things client side, especially because that might add unnecessary lag, especially in places where there might be very slow internet because of the current circumstances. I want to serve a static html to keep the load times as low as possible. Lets just keep set discard tweet time as a parameter server side.

from liveactionmap.

We can consider SQLite here too, since it's simple and file-based. It sounds like we're performing some conditional manipulation, and this will help us cut down on time complexity.

from liveactionmap.

@DomiiBunn mentioned firebase, would work here.

from liveactionmap.

@DomiiBunn mentioned firebase, which would work here.

It depends on the complexity you'd look for. Firebase is a nice balance between file storage(JSON files, SQLite, etc) and standalone databases as it's almost as flexible as and handles security, hosting, high availability and at the usage, we'd be expecting it should be fully free. As long as DB reads are cached that is.

from liveactionmap.

The reason I'm a little hesitant about firebase is that it adds another steps for developed looking to reproduce the repo and contribute. The simpler the project, the easier it is to contribute (as long as it doesn't impact performance or features).

from liveactionmap.

Use a config file and specify

useDatabaseCache: false

That way for a larger deployment it's worth caching and for personal deployment it's still working fine without added complexity

from liveactionmap.

Or using redis but idk how painful it is to implement with python

And i think it would be a bit of an over kill.

from liveactionmap.

Let's just go with a json file.

from liveactionmap.

Sounds good to me

from liveactionmap.

Nvm, I failed miserably at it.

from liveactionmap.

I'd love to help but python ain't my coup of tea

from liveactionmap.

Sure-a-mundo

from liveactionmap.