kipukun / sanic_highway Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Currently, authentication is done through a function inside each Handler
. This is ugly, and does not correspond to the way Handler
s are currently utilized throughout the rest of the (local) http package. The authentication function's type signature is exactly the same as the HandlerFunc
, so this may be implemented as some sort of Middleware
field inside a Handler.
The database will be frequently edited by either the admins or the automated scraper. For each database action upon anything but users, there should be a log for posterity. How to implement this is TBD.
The scraper
package currently requires a concrete design to be based off of. The scraper is responsible for scanning the database for entries that have not been scrapped yet, and using their id_*
entry, go to their respective sites and use the page for data ingesting. The scraper would then set a scraped = true
entry inside the table row.
A lot of thoughts have been swirling in my head on how best to implement the scraper concurrently. Naturally, the scraper would have to be run in a different goroutine to be run in the background. The scraper would send IDs down a channel to another goroutine which would be doing the actual processing. This goroutine would be waiting in an infinite select loop for values to come down the channel. TBC
As of now, the database code works but is not, at least in my mind, fully realized.
PostgreSQL performs well for this case, and has a solid Go driver, but the rigidity of the schema might be a hindrance as our use case develops. Alternatively, MongoDB as a backend has been discussed. The flexibility of MongoDB comes as a direct contrast to PSQL, but may be too flexible and require code changes in the future to accommodate for changes in object schema. Whatever the case, choosing one would mean a certain way of thinking for the database code.
As it stands right now, PSQL and a rigid modeling structure is being used. This matches well with Go's structs, allowing for a mirror of the database schema reflected in the code. MongoDB may present an issue with the static nature of Go's types, but this will require further investigation.
this issue will be updated with QoL requests as they are requested/added
As the title implies, this would require storing some user authentication in a table. This would also require the ability to edit fields directly inside entries for convenience. We need to make sure to avoid possible data races with the scraper and manual entry of data. This would most likely be prevented by a flag on the entry.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.