To query scraped docs
curl http://localhost:8000/api/v1/articles/search/?q=test
To kick off a scrape job
curl http://localhost:8000/api/v1/articles/search/scrape/
API documentation is automatically generated using Swagger. You can view documention by visiting this link.
If you are familiar with Docker, then you just need Docker. If you don't want to use Docker, then you just need Python3 and Postgres installed.
Start the dev server for local development:
cp .env.dist .env
docker-compose up -d --remove-orphans --buildRun a command inside the docker container:
docker-compose run --rm web [command]pip install -U poetry & poetry install & poertry shell # activate venv
cp .env.dist .env # create .env file and fill-in DB info
./manage.py migrate # run migrations
./manage.py collectstatic --noinput # collect static files
redis-server # run redis locally for celery
celery -A src.config worker --beat --loglevel=debug
--pidfile="./celerybeat.pid"
--scheduler django_celery_beat.schedulers:DatabaseScheduler # run celery beat and workerThis will run server on http://localhost:8000
./manage.py runserverIf you want, you can create initial super-user with next commad:
./manage.py createsuperuserTo run all tests with code-coverate report, simple run:
pytest -s src/scraper/
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent