kipukun / sanic_highway Goto Github PK

Currently, authentication is done through a function inside each Handler. This is ugly, and does not correspond to the way Handlers are currently utilized throughout the rest of the (local) http package. The authentication function's type signature is exactly the same as the HandlerFunc, so this may be implemented as some sort of Middleware field inside a Handler.

The database will be frequently edited by either the admins or the automated scraper. For each database action upon anything but users, there should be a log for posterity. How to implement this is TBD.

The scraper package currently requires a concrete design to be based off of. The scraper is responsible for scanning the database for entries that have not been scrapped yet, and using their id_* entry, go to their respective sites and use the page for data ingesting. The scraper would then set a scraped = true entry inside the table row.

A lot of thoughts have been swirling in my head on how best to implement the scraper concurrently. Naturally, the scraper would have to be run in a different goroutine to be run in the background. The scraper would send IDs down a channel to another goroutine which would be doing the actual processing. This goroutine would be waiting in an infinite select loop for values to come down the channel. TBC

As of now, the database code works but is not, at least in my mind, fully realized.
PostgreSQL performs well for this case, and has a solid Go driver, but the rigidity of the schema might be a hindrance as our use case develops. Alternatively, MongoDB as a backend has been discussed. The flexibility of MongoDB comes as a direct contrast to PSQL, but may be too flexible and require code changes in the future to accommodate for changes in object schema. Whatever the case, choosing one would mean a certain way of thinking for the database code.

As it stands right now, PSQL and a rigid modeling structure is being used. This matches well with Go's structs, allowing for a mirror of the database schema reflected in the code. MongoDB may present an issue with the static nature of Go's types, but this will require further investigation.

this issue will be updated with QoL requests as they are requested/added

• make ID'd entries a different color for easier visual distinction
• remove minus signs from search queries
• Titles column should be on the right side and the MetaIDs should be placed on the left: https://i.imgur.com/uBd8ZTF.jpg Title text formatting can be adjusted to the left of their field with the Search button placed on the left of them.
• pressing the search button on a particular line temporarily changes that lines BG colour to green. Helps speed up where I need to put the ID when I tab back into the editing screen.

As the title implies, this would require storing some user authentication in a table. This would also require the ability to edit fields directly inside entries for convenience. We need to make sure to avoid possible data races with the scraper and manual entry of data. This would most likely be prevented by a flag on the entry.