Sorry my ignorance can you tell me how i can fix it ?
python3 -m onioningestor --config onioningestor.yml
[05 Jun 2023 20:05:26] - INFO - Starting OnionScraper
[05 Jun 2023 20:05:26] - INFO - Loading config file
[05 Jun 2023 20:05:27] - INFO - Onions will be saved synchronously
[05 Jun 2023 20:05:28] - INFO - Initializing simple-html
[05 Jun 2023 20:05:28] - INFO - Running forever, in a loop
Traceback (most recent call last):
File "/home/OnionIngestor/onioningestor/config.py", line 39, in _load_plugin
module = importlib.import_module(".".join([plugin_type, plugin]))
File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 879, in exec_module
File "", line 1017, in get_code
File "", line 947, in source_to_code
File "", line 241, in _call_with_frames_removed
File "/home/OnionIngestor/onioningestor/sources/hunchly.py", line 33
call(['wget', self.domain]) # Downloading Hunchly spreadsheet file
TabError: inconsistent use of tabs and spaces in indentation

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/OnionIngestor/onioningestor/main.py", line 49, in
app.run()
File "/home/OnionIngestor/onioningestor/init.py", line 104, in run
self.run_forever()
File "/home/OnionIngestor/onioningestor/init.py", line 144, in run_forever
self.run_once()
File "/home/OnionIngestor/onioningestor/init.py", line 113, in run_once
self.collect_sources()
File "/home/OnionIngestor/onioningestor/init.py", line 76, in collect_sources
for name, collect, kwargs in self.config.sources():
File "/home/OnionIngestor/onioningestor/config.py", line 126, in sources
self._load_plugin(SOURCE, source["module"]),
File "/home/OnionIngestor/onioningestor/config.py", line 42, in _load_plugin
self.logger.error(e)
NameError: name 'self' is not defined

Crawled onion links aren't getting scanned by operators. Probably it's an issue with the queue! More testing and help needed!

Issue Template

Please use this template!

Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'onioningestor' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'onioningestor'. Thanks.

Make sure you've checked the following:

• [] Python version is 3.10.6

OnionIngestor Output

Please provide the output of OnionIngestor

OnionIngestor Yaml file

This is an example ThreatIngestor config file with some preconfigured RSS

sources, feeding extracted artifacts into a CSV file.

general:
# Run forever, check feeds once an hour.
daemon: True
sleep: 10
onion_validation: ([a-z2-7]{16,56}.onion)
blacklist: blacklist,keywords,go,here
interestingKeywords: Interesting,Keywords,Go,Here
save-thread: no # Use a separate thread to save onions
TorController:
port: 9051
password: mine password

monitor:
filename: monitoring.txt

#sources:
# A few threat intel blogs to get you started!
- name: simple-text-file
module: simplefile
filename: onion_master_list.txt

- name: hunchly
  module: hunchly
  domain: https://www.dropbox.com/sh/wdleu9o7jj1kk7v/AADq2sapbxm7rVtoLOnFJ7HHa/HiddenServices.xlsx

- name: pystemon
  module: pystemon
  dirname: pystemon/alerts/

- name: dark.fail
  module: dark.fail
  domain: https://dark.fail/

#  - name: source-gist
#    module: gist
#    url: https://gist.github.com/search?l=Text&q=.onion

#  - name: source-reddit
#    module: reddit
#    url: https://api.pushshift.io/reddit/search/comment/?subreddit=onions&limit=1000000
#    feed_type: messy
#
#  - name: pastebin
#    module: pastebin-account
#    url: https://gist.github.com/search?l=Text&q=.onion
#    feed_type: messy
#
#  - name: hunchly-report
#    module: gmail-hunchly
#    url: https://gist.github.com/search?l=Text&q=.onion
#    feed_type: messy
#
#  - name: onionland-search
#    module: collect-onions
#    url: http://3bbaaaccczcbdddz.onion/discover
#    feed_type: messy
#
#  - name: torch
#    module: collect-onions
#    url: http://xmh57jrzrnw6insl.onion
#    feed_type: messy

#operators:

• name: simple-html
  module: html
  timeout: 300
  retries: 2
  interestingKeywords: YOUR,INTERESTING,KEYWORDS,GO,HERE
  socks5:
  http: 'socks5h://127.0.0.1:9050'
  https: 'socks5h://127.0.0.1:9050'

- name: onionscan-go

module: onionscan

binpath: /PATH/TO/YOUR/ONIONSCAN/GO/BINARY

- name: simple-screenshot

module: screenshot

screenshots_path: null

- name: yara-rule

module: yara

filename: categories.yar

base_score: 50

#database_Engines:
# Simple telegram notifier

• name: telegram-notifer # see https://core.telegram.org/bots/api#authorizing-your-bot
  module: telegram
  chat_id: YOUR-TELEGRAM-CHAT
  token: YOUR-TELEGRAM-TOKEN

- name: elasticsearch

module: elasticsearch

index: YOUR-ELASTICSEARCH-INDEX_NAME

port : 9200

host : 127.0.0.1

- name: email

module: send_email

alert: no # Enable/disable email alerts

from: [email protected]

to: [email protected]

server: 127.0.0.1 # Address of the server (hostname or IP)

port: 25 # Outgoing SMTP port: 25, 587, ...

tls: no # Enable/disable tls support

username: '' # (optional) Username for authentication. Leave blank for no authentication.

password: '' # (optional) Password for authentication. Leave blank for no authentication.

subject: '[onioningestor] - {subject}'

size-limit: 1048576 # Size limit for pastie, above it's sent as attachement

Description of Issue

sudo python3 -m onioningestor -c config.yml
[30 Mar 2023 21:48:00] - INFO - Starting OnionScraper
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/Desktop/tools/OnionIngestor/onioningestor/main.py", line 47, in
app = Ingestor(args)
File "/home/Desktop/tools/OnionIngestor/onioningestor/init.py", line 27, in init
self.config = config.Config(args.configFile, self.logger)
File "/home/Desktop/tools/OnionIngestor/onioningestor/config.py", line 26, in init
with io.open(self.filename, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'config.yml'

Environment Details

Linux Ubuntu 22.04
Additional context
Add any other context about the problem here.

Kibana's maximum allowed to be analyzed for highlighting is 1000000. Can't update kibana

I have installed the fresh onioningestor from git and on first run it gave following error:
(it also gave error about monitoring.txt not there but I created a file with single onion link)

$ python -m onioningestor --config onioningestor.yml
[23 Jul 2020 18:38:05] - INFO - Starting OnionScraper
[23 Jul 2020 18:38:05] - INFO - Loading config file
[23 Jul 2020 18:38:05] - INFO - Creating Elasticsearch mapping
[23 Jul 2020 18:38:05] - INFO - Onions will be saved synchronously
[23 Jul 2020 18:38:05] - INFO - Initializing simple-html
[23 Jul 2020 18:38:05] - INFO - Initializing onionscan-go
[23 Jul 2020 18:38:05] - ERROR - __init__() takes 2 positional arguments but 4 were given
Traceback (most recent call last):
  File "/home/user1/OnionIngestor/onioningestor/__init__.py", line 66, in __init__
    for name, operator, kwargs in self.config.operators()}
  File "/home/user1/OnionIngestor/onioningestor/__init__.py", line 66, in <dictcomp>
    for name, operator, kwargs in self.config.operators()}
TypeError: __init__() takes 2 positional arguments but 4 were given

Current onioningestor.yml:

# This is an example ThreatIngestor config file with some preconfigured RSS
# sources, feeding extracted artifacts into a CSV file.

general:
    # Run forever, check feeds once an hour.
    daemon: True
    sleep: 10
    onion_validation: ([a-z2-7]{16,56}\.onion)
    blacklist: porn,cvv
    interestingKeywords: leak,deface,hack,ddos,exploit
    save-thread: no         # Use a separate thread to save onions
    TorController:
        port: 9051
        password: mysecreatetorpass

monitor:
    filename: monitoring.txt

sources:
    # A few threat intel blogs to get you started!
    - name: simple-text-file
      module: simplefile
      filename: onion_master_list.txt

    - name: source-gist
      module: gist
      url: https://gist.github.com/search?l=Text&q=.onion

    - name: source-reddit
      module: reddit
      url: https://api.pushshift.io/reddit/search/comment/?subreddit=onions&limit=1000000
      feed_type: messy

    - name: pastebin
      module: pastebin-account
      url: https://gist.github.com/search?l=Text&q=.onion
      feed_type: messy

    - name: hunchly-report
      module: gmail-hunchly
      url: https://gist.github.com/search?l=Text&q=.onion
      feed_type: messy

    - name: onionland-search
      module: collect-onions
      url: http://3bbaaaccczcbdddz.onion/discover
      feed_type: messy
 
    - name: torch
      module: collect-onions
      url: http://xmh57jrzrnw6insl.onion
      feed_type: messy
	  
operators:
   - name: simple-html
     module: html
     timeout: 300
     retries: 2
     interestingKeywords: leak,deface,exploit,hack
     socks5:
         http: 'socks5h://127.0.0.1:9050'
         https: 'socks5h://127.0.0.1:9050'

   - name: onionscan-go
     module: onionscan
     binpath: /home/user1/go/bin/onionscan


   - name: simple-screenshot
     module: screenshot
     screenshots_path: /home/user1/screenshots/


#  - name: yara-rule
#    module: yara
#    filename: categories.yar
#    base_score: 50


database_Engines:
    # Simple telegram notifier
    - name: telegram-notifer
      module: telegram
      chat_id: YOUR-TELEGRAM-CHAT
      token: YOUR-TELEGRAM-TOKEN

    - name: elasticsearch
      module: elasticsearch
      index: onioningest
      port : 9200
      host : 127.1.1.1

#  - name: email
#    module: send_email
#    alert: no             # Enable/disable email alerts
#    from: [email protected]
#    to: [email protected]
#    server: 127.0.0.1     # Address of the server (hostname or IP)
#    port: 25              # Outgoing SMTP port: 25, 587, ...
#    tls: no               # Enable/disable tls support
#    username: ''          # (optional) Username for authentication. Leave blank for no authentication.
#    password: ''          # (optional) Password for authentication. Leave blank for no authentication.
#    subject: '[onioningestor] - {subject}'
#    size-limit: 1048576   # Size limit for pastie, above it's sent as attachement

After command python3 -m onioningestor -c config.yml the application are thrworing "Monitoring.txt missing file"

Enviroment:
ParrotOS 4.9
Python 3.6