Giter Club home page Giter Club logo

lacuscore's Introduction

Documentation Status

Modulable Lacus

Lacus, but as a simple module.

Installation

pip install lacuscore

Design

LacusCore is the part taking care of enqueuing and capturing URLs or web enabled documents. It can be used as a module in your own project, see below for the usage

Lacus is the webservice that uses LacusCore, and you can use Pylacus to query it.

The enqueue, get_capture_status, and get_capture methods if LacusCore and PyLacus have the same parameters which means you can easily use them interchangeably in your project.

For more information regarding the usage of the module and a few examples, please refer to the documentation

lacuscore's People

Contributors

adulau avatar dependabot[bot] avatar rafiot avatar terrtia avatar

Stargazers

avatar avatar avatar avatar avatar

Watchers

avatar avatar

lacuscore's Issues

[Feature] Add monitoring

  • Store keys in lacus:ongoing as a ranked set with the start time as score
  • Add a way to get the settings related to a capture, especially the URL and depth
  • Get the size stored in redis for a specific capture
  • Get the length of the lacus:to_capture key, and the content of each of them
  • Add a monitoring script

(optional) Bypass of GDPR / cookies popups

The Playwright v1.42.0 release adds a handy endpoint to automatically click on it when it pops up. Problem is that it will depends based on the website.

But we can probably gradually add an automation for all the common GDPR frameworks, as each of them will be identical across websites.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.