NewsMiner is a web crawler to extract data from Persian news websites.
use init-db
to initialize the database
to start extracting from all websites run the command below:
go run ./main.go -d "NewsMiner" -c "mongodb://localhost:27017/?compressors=disabled&gssapiServiceName=mongodb" -p 10 -t 10 --farsnews --isna --tabnak --tasnim --yjc