certat / intelmq-docker Goto Github PK

View Code? Open in Web Editor NEW

6.0 6.0 8.0 243 KB

Tools 4 IntelMQ w/ Docker. Make IntelMQ great again!

Dockerfile 33.08% Shell 66.92%

intelmq-docker's People

Contributors

Stargazers

Watchers

Forkers

certunlp a35924 sebix rkokkelk like-a-freedom agustinparmisano dysnome

intelmq-docker's Issues

example config in the docker images is broken

after a docker compose up -d, I get:

intelmq-docker-redis-1  | 1:M 27 Jul 2023 14:19:18.258 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
intelmq-docker-redis-1  | 1:M 27 Jul 2023 14:19:18.258 * Ready to accept connections
intelmq-docker-intelmq-1  | Found previous version 3.2.0 in state file.
intelmq-docker-nginx-1    | 2023/07/27 14:19:19 [error] 7#7: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.72.0.2, server: localhost, request: "GET /intelmq/v1/api/queues-and-status HTTP/1.1", upstream: "http://172.27.0.3:8080/v1/api/queues-and-status", host: "nanu:1337", referrer: "http://nanu:1337/configs.html"
intelmq-docker-intelmq-1  | Checks if all harmonization fields and types are correct: Nothing to do.
intelmq-docker-intelmq-1  | Configuration upgrade successful!
intelmq-docker-intelmq-1  | Incomplete installation: Bot 'malc0de-parser' not importable: ModuleNotFoundError("No module named 'intelmq.bots.parsers.malc0de'").
intelmq-docker-intelmq-1  | Bot 'url2fqdn-expert': This bot is deprecated and will be removed in version 4.0. Use 'URL Expert' bot instead.
intelmq-docker-intelmq-1  | Reading configuration files.
intelmq-docker-intelmq-1  | Checking runtime and pipeline configuration.
intelmq-docker-intelmq-1  | Checking harmonization configuration.
intelmq-docker-intelmq-1  | Checking for bots.
intelmq-docker-intelmq-1  | Some issues have been found, please check the above output.
intelmq-docker-intelmq-1  | Loading config from /etc/intelmq/api-config.json
intelmq-docker-intelmq-1  | Added user intelmq to intelmq session file.
intelmq-docker-intelmq-1  | Usage: uvicorn [OPTIONS] APP
intelmq-docker-nginx-1    | 10.72.0.2 - - [27/Jul/2023:14:19:19 +0000] "GET /intelmq/v1/api/queues-and-status HTTP/1.1" 502 576 "http://nanu:1337/configs.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-"
intelmq-docker-nginx-1    | 2023/07/27 14:19:21 [error] 7#7: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.72.0.2, server: localhost, request: "GET /intelmq/v1/api/queues-and-status HTTP/1.1", upstream: "http://172.27.0.3:8080/v1/api/queues-and-status", host: "nanu:1337", referrer: "http://nanu:1337/configs.html"
intelmq-docker-nginx-1    | 10.72.0.2 - - [27/Jul/2023:14:19:21 +0000] "GET /intelmq/v1/api/queues-and-status HTTP/1.1" 502 576 "http://nanu:1337/configs.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-"
intelmq-docker-intelmq-1  | Try 'uvicorn --help' for help.
intelmq-docker-intelmq-1  | 
intelmq-docker-intelmq-1  | 
intelmq-docker-intelmq-1  | Error: no such option: -p

password is not correct

There is a mismatch with (whatever) the password is in the dockerhub images (which you get if you do a docker compose pull ) and the documentation at https://intelmq.readthedocs.io/en/latest/user/installation.html#installation-docker

You might want to make sure that the docker images on dockerhub follow the installation instructions 1-1.

Add python3-pymongo as dependecy

Related to certtools/intelmq#2261

Please update docker images to IntelMQ 3.1

We should quickly also provide an intelMQ 3.1 docker image. Thank you!

Add python3-elasticsearch as dependency

Related to certtools/intelmq#2261

Default password

What is the default password used in this docker image? intelmq : intelmq does not seem to work.

dockerhub docker pull image -> permission denied

(base) ➜  intelmq-docker git:(main) ✗ docker compose pull
[+] Pulling 3/3
 ✘ nginx Error                                                                                                                                                                        1.5s 
 ✘ intelmq Error                                                                                                                                                                      1.5s 
 ✘ redis Error                                                                                                                                                                        1.5s 
Error response from daemon: pull access denied for intelmq-full, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

MISP API output bot duplicating events in MISP

Hi,

I'm trying to get intelmq (docker version) to parse viriback (https://tracker.viriback.com/) data into a MISP instance using the guide at: https://intelmq.readthedocs.io/en/latest/user/feeds.html#viriback
The issues I'm having is that the bot that supposed to insert the events into my MISP instance (intelmq.bots.outputs.misp.output_api) does not seem to be checking for exising events in MISP and just feeds it with new events every time I restart the botnet and thus flooding the MISP instance with duplicated events.
I'm running MISP 2.4.143 with intelmq-docker 2.3.3 container (pulled with: git clone --recursive https://github.com/certat/intelmq-docker.git at 04.06.2021)

I did recently update my MISP instance from .133 to .143 hoping this'd resolve the problem, but it's still the same.

Are there any details I'm missing here? Any logs I could look into?
Thanks.

Error sqlite3.OperationalError docker-compose-dev.yml

after following instruction https://github.com/certat/intelmq-docker/blob/main/DEVELOP-GUIDE.md, dev containers doesn't load.

I got error:

intelmq-docker-intelmq-1  | Loading config from /etc/intelmq/api-config.json
intelmq-docker-intelmq-1  | Traceback (most recent call last):
intelmq-docker-intelmq-1  |   File "/usr/local/bin/intelmq-api-adduser", line 4, in <module>
intelmq-docker-intelmq-1  |     __import__('pkg_resources').run_script('intelmq-api==3.0.1', 'intelmq-api-adduser')
intelmq-docker-intelmq-1  |   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 651, in run_script
intelmq-docker-intelmq-1  |     self.require(requires)[0].run_script(script_name, ns)
intelmq-docker-intelmq-1  |   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1448, in run_script
intelmq-docker-intelmq-1  |     exec(code, namespace, namespace)
intelmq-docker-intelmq-1  |   File "/usr/local/lib/python3.9/dist-packages/intelmq_api-3.0.1-py3.9.egg/EGG-INFO/scripts/intelmq-api-adduser", line 27, in <module>
intelmq-docker-intelmq-1  |     session_store = intelmq_api.session.SessionStore(str(api_config.session_store), api_config.session_duration)
intelmq-docker-intelmq-1  |   File "/usr/local/lib/python3.9/dist-packages/intelmq_api-3.0.1-py3.9.egg/intelmq_api/session.py", line 86, in __init__
intelmq-docker-intelmq-1  |     self.init_sqlite_db()
intelmq-docker-intelmq-1  |   File "/usr/local/lib/python3.9/dist-packages/intelmq_api-3.0.1-py3.9.egg/intelmq_api/session.py", line 100, in init_sqlite_db
intelmq-docker-intelmq-1  |     with self.connect() as con:
intelmq-docker-intelmq-1  |   File "/usr/local/lib/python3.9/dist-packages/intelmq_api-3.0.1-py3.9.egg/intelmq_api/session.py", line 91, in connect
intelmq-docker-intelmq-1  |     return sqlite3.connect(self.dbname, check_same_thread=False,
intelmq-docker-intelmq-1  | sqlite3.OperationalError: unable to open database file
intelmq-docker-intelmq-1  | /opt/entrypoint-dev.sh: line 29: cd: /etc/intelmq-api: No such file or directory

Getting a 404 error when trying to access intelmq-manager

Hi!

I just tried this docker-setup as described in https://github.com/certat/intelmq-docker#fastest-way-to-run--deploy, but when I try to open http://127.0.0.1:1337/ I'm getting a 404 nginx error.

This is because docker-compose.yml maps the folder ./intelmq-manager/html (https://github.com/certat/intelmq-docker/blob/main/docker-compose.yml#L19), which is empty when I check out this repository... am I missing something?

intelmq-full-dev is outdated

With the release of intelmq version 3.0, the intelmq-full-dev image needs an update, as already discussed in #3. @puckk @einar-lanfranco we leave this to you, as we agreed you are maintaining this.

Thanks in advance.

intelmq-dev needs to add intelmq to sudoers file

@einar-lanfranco As stated here certtools/intelmq#2261, intelmq-full-dev is not able to install any packages during the run. You need to add intelmq to sudoers. To achieve this, see here

RUN useradd -d /opt/intelmq -U -s /bin/bash intelmq \
    && adduser intelmq sudo \
    && echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /opt/sudoers \

Documentation

Add documentation

How to install
How to use
Configuration
How to update

Suggestion: use init for intelmq container

In case a bot crashes, the leftover process is usually reaped by the init system. However because there is no init system inside a container by default, there is a zombie process left behind which can cause trouble to intelmqctl (particularly the error "Commandline of the process N with commandline X could not be interpreted").

There is an option to have a simple init system inside the container which solves this issue:
https://docs.docker.com/engine/reference/run/#specify-an-init-process
https://docs.docker.com/compose/compose-file/#init

Docker versioning more detailed

Add multiple docker version tags like

2.x
2.x.x
3.x
3.x.x

Ownership of example_config/intelmq is set to user id 1000

After starting containers as documented the ownership and group of example_config is set to user 1000:

~/intelmq-docker$ ls -l example_config/intelmq/
total 4
drwxr-xr-x 3 1000 1000 4096 Jun 27 18:56 etc
~/intelmq-docker$ ls -l example_config/intelmq/etc/
total 120
-rw-r--r-- 1 1000 1000 75147 Jun 27 18:56 feeds.yaml
-rw-r--r-- 1 1000 1000 21097 Jun 27 18:56 harmonization.conf
-rw-r--r-- 1 1000 1000    89 Jun 27 18:56 harmonization.conf.license
drwxr-xr-x 2 1000 1000  4096 Jun 27 18:56 manager
-rw-r--r-- 1 1000 1000  6530 Jun 27 18:56 runtime.yaml
-rw-r--r-- 1 1000 1000    87 Jun 27 18:56 runtime.yaml.license

If the current user is not UID 1000, these files become unwritable and need to be corrected manually.

Dev instructions missing dependency

For running the the build.sh script it is necessary to have mako python package installed.

Quoting for bash variables

intelmq-docker/entrypoint.sh

Lines 7 to 9 in 3df283b

 intelmq_user=${INTELMQ_API_USER:-intelmq} 

 intelmq_pass=${INTELMQ_API_PASS:-intelmq} 

 intelmq-api-adduser --user $intelmq_user --pass $intelmq_pass

Please use quoting:

 intelmq_user="${INTELMQ_API_USER:-intelmq}"
 intelmq_pass="${INTELMQ_API_PASS:-intelmq}"
 intelmq-api-adduser --user "$intelmq_user" --pass "$intelmq_pass"

Otherwise the command fails if someone uses are username or password with spaces or other weird characters

Install in persistent mode the Stomp library in a intelMQ-Manager docker compose instance

Hello,

I have a stupid question. My IntelMQ instance is running from following docker-compose.yml file:

version: "3"
services:
    redis:
        container_name: intelmq-redis
        image: redis:latest
        volumes:
            - ./config/redis/redis.conf:/usr/local/etc/redis/redis.conf
        command:
            - redis-server
            - /usr/local/etc/redis/redis.conf
        restart: always
        networks:
            - internal
    nginx:
        container_name: intelmq-nginx
        image: certat/intelmq-nginx:latest
        restart: always
        ports:
            - 1337:80
        depends_on:
            - intelmq
        networks:
            - internal
    intelmq:
        container_name: intelmq
        image: certat/intelmq-full:latest
        volumes:
            - ./config/intelmq/etc/:/opt/intelmq/etc/
            - ./config/intelmq-api/config.json:/etc/intelmq/api-config.json
            - ./logs:/opt/intelmq/var/log
            - ./output:/opt/intelmq/var/lib/bots
        depends_on:
            - redis
        environment:
            INTELMQ_SOURCE_PIPELINE_BROKER: "redis"
            INTELMQ_PIPELINE_BROKER: "redis"
            INTELMQ_DESTIONATION_PIPELINE_BROKER: "redis"
            INTELMQ_PIPELINE_HOST: redis
            INTELMQ_SOURCE_PIPELINE_HOST: redis
            INTELMQ_DESTINATION_PIPELINE_HOST: redis
            INTELMQ_REDIS_CACHE_HOST: redis
        networks:
            - internal

networks:
    internal:
        driver: bridge

How to apply persistent pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt ? Rebuild "intelmq" container in docker-compose.yml adding build parameter to launch a Dockerfile with pip command before to launch intelmq ?
Some bot collectors requires to install IntelMQ as package in Linux system instead of docker compose ?

Regards,

	intelmq_user=${INTELMQ_API_USER:-intelmq}
	intelmq_pass=${INTELMQ_API_PASS:-intelmq}
	intelmq-api-adduser --user $intelmq_user --pass $intelmq_pass