Download lightnovels from various online sources and generate output in different formats, e.g. epub, mobi, json, html, text, docx and pdf.
๐ฆ lightnovel-crawler v2.9.10 for windows ~ 14.2MB
In Windows 8, 10 or later versions, it might say that
lncrawl.exe
is not safe to dowload or execute. You should bypass/ignore this security check to execute this program. Actually, I am too lazy to add proper configuration files to solve this issue. Excuse me please ๐.
PDF and DOCX generation is disabled for EXE build. It only works with (pip)[#a3-pip-for-windows-mac-and-linux].
๐ฆ A python package named lightnovel-crawler
is available in pypi.
Make sure you have installed
python 3.5
or above and havepip
enabled. Visit these links for installating python and pip in Windows, Linux and Mac. Feel free ask me if you are stuck.
To install this app or to update installed one via pip
, just run:
$ pip install --user -U lightnovel-crawler
Remember, in some cases you have to use python3 -m pip
or pip3
or python -m pip
. And you do not need --user
option, if you are running from root.
To Windows users: Download and install the GTK3-Runtime from https://github.com/tschoonj/GTK-for-Windows-Runtime-Environment-Installer/releases
Next, open your terminal and enter:
$ lightnovel-crawler
# Or, a shortcut:
$ lncrawl
To view extra logs, use:
lncrawl -lll
Visit this link to get started with the telegram bot: https://t.me/epub_smelter_bot
Visit this link to install discord bot to your server: https://discordapp.com/oauth2/authorize?client_id=537526751170002946&permissions=51264&scope=bot
Send !help
to open the bot help message.
- First clone the repository:
$ git clone https://github.com/dipu-bd/lightnovel-crawler
- Open command prompt inside of the project folder and install requirements:
$ pip3 install --user -r requirements.txt
- Run the program:
$ python3 __main__.py
# Or, in short,
$ python3 .
To view list of available options:
$ lncrawl -h
================================================================================
๐ Lightnovel Crawler ๐2.9.10
Download lightnovels into html, text, epub, mobi and json
--------------------------------------------------------------------------------
usage: lncrawl [options...]
lightnovel-crawler [options...]
positional arguments:
EXTRA To pass a query string to use as extra arguments
optional arguments:
-h, --help show this help message and exit
-l Set log levels (1 = warn, 2 = info, 3 = debug)
-v, --version show program's version number and exit
-s URL, --source URL Profile page url of the novel
-q STR, --query STR Novel query followed by list of source sites.
--sources Display the source selection menu while searching
-o PATH, --output PATH
Path where the downloads to be stored
--format E [E ...] Define which formats to output. Default: all
-f, --force Force replace any existing folder
-i, --ignore Ignore any existing folder (do not replace)
--single Put everything in a single book
--multi Build separate books by volumes
--login USER PASSWD User name/email address and password for login
--all Download all chapters
--first [COUNT] Download first few chapters (default: 10)
--last [COUNT] Download last few chapters (default: 10)
--page START STOP The start and final chapter urls
--range FROM TO The start and final chapter indexes
--volumes [N [N ...]]
The list of volume numbers to download
--chapters [URL [URL ...]]
A list of specific chapter urls
--suppress Suppress input prompts (use defaults instead)
--bot {console,telegram,discord,test}
Select a bot. Default: console
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are two chatbots available at this moment: Telegram and Discord. To run your own bot server, follow these instructions:
# Clone this repository
$ git clone https://github.com/dipu-bd/lightnovel-crawler
# Install requirements
$ pip3 install --user -r requirements.txt
$ pip3 install --user -r bot_requirements.txt
# Edit the environment variables
# You should give your API keys and log info here
# Also specify which bot server you want to run
$ cp .env.example .env
$ vim .env
# Run the server using:
$ python3 .
There is a server.sh
script to run a bot in ubuntu servers. It will basically execute the python __main__.py
and send the task to run in background. I use it to run my discord bot in the server.
You are very welcome to contribute in this project. You can:
- create new issues pointing out the bugs.
- solve existing issues.
- add your own sources.
- add new output formats.
- create new bots.
- Create new crawler using the
spiders/_sample.py
as blueprint. You can check out existing bots for idea. - Import your crawler to
spiders/__init__.py
file.
- Create a new bot file using
bots/_sample.py
as a standard. You can check out existing bots for idea. - Import your bot to
bots/__init__.py
file.
Request new one by creating a new issue. Or, make a pull request by adding a new source.
The list of currently available sources and the future plans are given below:
Rejected:
Rejected Link | Reason |
---|---|
403 - Forbidden: Access is denied |
|
Does not host any novels |
|
Does not follow uniform format |
|
503 Service Unavailable |
|
Site is down |
|
Site is down |
|
Does not like being crawled |
When download is done, the following files can be generated:
- JSON (default)
- HTML
- TEXT
- EPUB
- MOBI
- DOCX
- Console Bot
- Telegram Bot
- Discord Bot
Click here to view details.