salamek / gitlab-tools Goto Github PK

View Code? Open in Web Editor NEW

99.0 99.0 15.0 795 KB

Web application providing tools not avaiable in GitLab CE (Gitlab push/pull mirror and more)

License: GNU General Public License v3.0

Shell 0.59% Python 72.71% Mako 0.18% JavaScript 3.80% CSS 0.73% HTML 21.99%

git gitlab gitlab-mirror gitlab-mirror-webhook mirrored-repositories mirroring replication

gitlab-tools's People

Contributors

Stargazers

Watchers

Forkers

lichnak dontfight minstrelsy androvideo-dev mattv8 helpless1 vsperson rhinoceros goldendeng bellyfat joaomarques90 rhewage chris3081 preussal ericxliu1990

gitlab-tools's Issues

500 Internal Server Error at pull mirror

with Gitlab v16.6.1

When I click on Pull Mirror
500 Internal Server Error
Uh-oh you found a bug! On Email Has Been Sent to the Admins List.

With Home, Push Mirror, fingerprint everything is ok.

E1211 11:07:36.859 3122222 app.py:1891] Exception on /pull-mirror/ [GET]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib/python3/dist-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/gitlab_tools/views/pull_mirror/index.py", line 116, in get_mirror
    return flask.render_template('pull_mirror.index.pull_mirror.html', pagination=pagination), 200
  File "/usr/lib/python3/dist-packages/flask/templating.py", line 137, in render_template
    return _render(
  File "/usr/lib/python3/dist-packages/flask/templating.py", line 120, in _render
    rv = template.render(context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/gitlab_tools/views/pull_mirror/templates/pull_mirror.index.pull_mirror.html", line 1, in <module>
    {% extends "base.html" %}
  File "/usr/lib/python3/dist-packages/gitlab_tools/templates/base.html", line 72, in <module>
    {% block body %}{% endblock %}
  File "/usr/lib/python3/dist-packages/gitlab_tools/views/pull_mirror/templates/pull_mirror.index.pull_mirror.html", line 26, in <module>
    <tr class="bg-{{item.task_results.first()|format_task_status_class}}" title="{{item.note}}">
  File "/usr/lib/python3/dist-packages/gitlab_tools/middleware.py", line 134, in format_task_status_class_filter
    }.get(task_result.taskmeta.status, 'warning')
AttributeError: 'NoneType' object has no attribute 'status'
I1211 11:07:36.861 3122222 _internal.py:122] 10.83.10.105 - - [11/Dec/2023 11:07:36] "GET /pull-mirror/ HTTP/1.1" 500 -

Allow . in project name

I can't include a . in a project name even though it's valid in GitLab without receiving an error:

Project name cannot contain special characters

Periodic sync not triggered

I can manually trigger the sync using the web UI or webhook. But the period sync never triggered. Did I missed something trivial?

Periodic sync: Every minute (*/1 * * * *)

94d5e2f-9610-4957-9089-6c5be5e8f57e	sync_pull_mirror	Web hook	SUCCESS	None	Jun 30, 2022, 6:16:40 AM	
b5d3d387-e7c2-492b-a688-57b2e0753ed3	sync_pull_mirror	Manual	SUCCESS	None	Jun 30, 2022, 6:04:41 AM	
06af6586-90e3-4173-9ee0-ac276dc2b6b4	create_ssh_config	Manual	SUCCESS	None	Jun 30, 2022, 5:52:15 AM	
 29ca70fa-52e5-4edb-ad8b-561e3b8c279b	save_pull_mirror	Manual	SUCCESS	None	Jun 30, 2022, 5:52:19 AM	
ac0bb692-042a-4b9a-841e-64b8b9ed8e31	sync_pull_mirror	Manual	SUCCESS	None	Jun 30, 2022, 5:43:10 AM	
6fe154ee-956f-4bf2-92ee-8396677a95cf	create_ssh_config	Manual	SUCCESS	None	Jun 30, 2022, 5:41:41 AM	
 4bf3c0ec-b0c5-4024-b1b5-6aa8e3dd2448	save_pull_mirror	Manual	SUCCESS	None	Jun 30, 2022, 5:41:47 AM	
be48cddf-a95b-4348-866f-c211d8f191e4	sync_pull_mirror	Manual	SUCCESS	None	Jun 29, 2022, 6:00:21 AM	
2ffc1628-9753-402a-93bf-0656f750ef5e	create_ssh_config	Manual	SUCCESS	None	Jun 21, 2022, 10:38:10 PM	
 4952c8b1-b81a-49df-97bf-02a3f7cbde64	save_pull_mirror	Manual	SUCCESS	None	Jun 21, 2022, 10:38:19 PM

gitlab-tools git:(main) kubectl -n gitlab-tools logs -f gitlab-tools-649f54b688-hvl48
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 00-user-directory.sh: applying...
[fix-attrs.d] 00-user-directory.sh: exited 0.
[fix-attrs.d] 10-ssh-keys.sh: applying...
[fix-attrs.d] 10-ssh-keys.sh: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-generate-configuration.sh: executing...
[START] Checking for required variables...
[START] Testing required environment variables...
[FINISH] All required environment variables are in place.
[START] Setting defaults for optional environment variables...
[FINISH] Set some sane-defaults for environment variables.
[START] Generating configuration file...
[cont-init.d] 00-generate-configuration.sh: exited 0.
[cont-init.d] 10-migrate-database.sh: executing...
[START] Migrating the database...
[START] Creating database tables...
D0630 11:16:36.484 3047 cmd.py:870] Popen(['git', 'version'], cwd=/opt/gitlab-tools, universal_newlines=False, shell=None, istream=None)
D0630 11:16:36.490 3047 cmd.py:870] Popen(['git', 'version'], cwd=/opt/gitlab-tools, universal_newlines=False, shell=None, istream=None)
D0630 11:16:36.541 3047 base.py:272] Configuring Raven for host: None
I0630 11:16:36.542 3047 base.py:219] Raven is not configured (logging is disabled). Please see the documentation for more information.
/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py:220: SADeprecationWarning: The Engine.table_names() method is deprecated and will be removed in a future release.  Please refer to Inspector.get_table_names(). (deprecated since: 1.4)
  tables_before = set(db.engine.table_names())
/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py:222: SADeprecationWarning: The Engine.table_names() method is deprecated and will be removed in a future release.  Please refer to Inspector.get_table_names(). (deprecated since: 1.4)
  tables_after = set(db.engine.table_names())
[START] Migrating the database...
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 56189bfb2c5f, empty message
Traceback (most recent call last):
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.DuplicateTable: relation "fingerprint" already exists


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/gitlab-tools/manage.py", line 7, in <module>
    main()
  File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 558, in main
    getattr(command, 'chosen')()  # Execute the function specified by the user.
  File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 553, in _db
    cli.main(args=sys.argv[1:])
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/cli.py", line 136, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/__init__.py", line 96, in wrapped
    f(*args, **kwargs)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/__init__.py", line 271, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/gitlab-tools/gitlab_tools/migrations/env.py", line 89, in <module>
    run_migrations_online()
  File "/opt/gitlab-tools/gitlab_tools/migrations/env.py", line 81, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
  File "/opt/gitlab-tools/gitlab_tools/migrations/versions/56189bfb2c5f_.py", line 21, in upgrade
    op.create_table('fingerprint',
  File "<string>", line 8, in create_table
  File "<string>", line 3, in create_table
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/ops.py", line 1254, in create_table
    return operations.invoke(op)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/base.py", line 392, in invoke
    return fn(self, operation)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/toimpl.py", line 114, in create_table
    operations.impl.create_table(table)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/ddl/impl.py", line 352, in create_table
    self._exec(schema.CreateTable(table))
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/ddl/impl.py", line 193, in _exec
    return conn.execute(construct, multiparams)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1306, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
    return connection._execute_ddl(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1398, in _execute_ddl
    ret = self._execute_context(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
    self._handle_dbapi_exception(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
    util.raise_(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.DuplicateTable) relation "fingerprint" already exists

[SQL:
CREATE TABLE fingerprint (
	updated TIMESTAMP WITHOUT TIME ZONE,
	created TIMESTAMP WITHOUT TIME ZONE,
	id SERIAL NOT NULL,
	user_id INTEGER,
	hostname VARCHAR(255),
	sha256_fingerprint VARCHAR(255),
	hashed_hostname VARCHAR(255),
	PRIMARY KEY (id),
	FOREIGN KEY(user_id) REFERENCES "user" (id),
	CONSTRAINT _user_id_hashed_hostname_uc UNIQUE (user_id, hashed_hostname)
)

]
(Background on this error at: https://sqlalche.me/e/14/f405)
[INFO] Migrations are failed or already applied.
[cont-init.d] 10-migrate-database.sh: exited 0.
[cont-init.d] 20-start-services.sh: executing...
-------------------------
[START] Starting Nginx server...
Starting nginx: nginx.
-------------------------
[START] Starting RabbitMQ server...
Starting RabbitMQ Messaging Server: rabbitmq-server.
-------------------------
[START] Starting Redis server...
Starting redis-server: redis-server.
[cont-init.d] 20-start-services.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[START] Starting the celerybeat...
[START] Starting uwsgi...
[START] Starting the celeryworker...
[services.d] done.
[app] [uWSGI] getting INI configuration from /etc/uwsgi/gitlab-tools.ini
[app] *** Starting uWSGI 2.0.19.1-debian (64bit) on [Thu Jun 30 11:16:44 2022] ***
[app] compiled with version: 10.2.1 20210110 on 11 June 2021 09:08:33
[app] os: Linux-5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09)
[app] nodename: gitlab-tools-649f54b688-hvl48
[app] machine: x86_64
[app] clock source: unix
[app] pcre jit disabled
[app] detected number of CPU cores: 2
[app] current working directory: /opt/gitlab-tools
[app] detected binary path: /usr/bin/uwsgi-core
[app] chdir() to /opt/gitlab-tools
[app] your memory page size is 4096 bytes
[app] detected max file descriptor number: 1048576
[app] lock engine: pthread robust mutexes
[app] thunder lock: enabled
[app] uwsgi socket 0 bound to UNIX address /tmp/gitlab-tools.sock fd 3
[app] Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110]
[app] PEP 405 virtualenv detected: /opt/gitlab-tools/venv
[app] Set PythonHome to /opt/gitlab-tools/venv
[app] Python main interpreter initialized at 0x56251c670780
[app] python threads support enabled
[app] your server socket listen backlog is limited to 100 connections
[app] your mercy for graceful operations on workers is 60 seconds
[app] mapped 203216 bytes (198 KB) for 1 cores
[app] *** Operational MODE: single process ***
[app] WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x56251c670780 pid: 3438 (default app)
[app] spawned uWSGI master process (pid: 3438)
[app] spawned uWSGI worker 1 (pid: 3455, cores: 1)
[bgtask] celery beat v5.0.2 (singularity) is starting.
[bgtask] __    -    ... __   -        _
[bgtask] LocalTime -> 2022-06-30 11:16:46
[bgtask] Configuration ->
[bgtask]     . broker -> amqp://guest:**@127.0.0.1:5672//
[bgtask]     . loader -> celery.loaders.app.AppLoader
[bgtask]     . scheduler -> celery.beat.PersistentScheduler
[bgtask]     . db -> /home/service/celery_beat.db
[bgtask]     . logfile -> [stderr]@%WARNING
[bgtask]     . maxinterval -> 5.00 minutes (300s)
[bgtask] W0630 11:16:46.060 3442 beat.py:564] DB Reset: Account for new __version__ field
[worker]
[worker]  -------------- celery@1 v5.0.2 (singularity)
[worker] --- ***** -----
[worker] -- ******* ---- Linux-5.10.0-15-amd64-x86_64-with-glibc2.31 2022-06-30 11:16:46
[worker] - *** --- * ---
[worker] - ** ---------- [config]
[worker] - ** ---------- .> app:         gitlab_tools.application:0x7f3970dbc7f0
[worker] - ** ---------- .> transport:   amqp://guest:**@127.0.0.1:5672//
[worker] - ** ---------- .> results:     postgresql://gitlab-tools:**@postgresql13.postgresql13.svc.cluster.local/gitlab-tools
[worker] - *** --- * --- .> concurrency: {min=1, max=10} (prefork)
[worker] -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
[worker] --- ***** -----
[worker]  -------------- [queues]
[worker]                 .> celery           exchange=celery(direct) key=celery
[worker]
[worker]
[worker] W0630 11:16:46.194 3446 warnings.py:109] /opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/app/utils.py:204: CDeprecationWarning:
[worker]     The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
[worker]     version 6.0.0. Use the result_backend instead
[worker]
[worker]   deprecated.warn(description=f'The {setting!r} setting',
[worker]
[worker] W0630 11:16:46.194 3446 worker.py:156] Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.

500 Internal Server Error after login / Redirect

I use Ubuntu 18.
Install the package from your mirror.
And do the steps of configuration from your doku

I use the Internal Webserver from your gitlab-tools

After Login in Gitlab i get this

500 Internal Server Error
Uh-oh you found a bug! An email has been sent to the ADMINs list.
URL: http://gitlab-tools.domain.tld:8081/sign/in/do?code=29f6d9adbdcb.....

The config /etc/gitlab-tools/config.yml

GITLAB_APP_ID: 7b4f7f......
GITLAB_APP_SECRET: 21b0f3fe4......
GITLAB_URL: https://gitlab.fti-group.com
HOST: 0.0.0.0
PORT: '8081'
SECRET_KEY: 9b51054c2d.......
SERVER_NAME: gitlab-tools.domain.tld:8081
SQLALCHEMY_DATABASE_URI: sqlite:////home/gitlab-tools/gitlab-tools.db
USER: gitlab-tools

What is wrong ?

gitlab-ce:amd64/bionic 11.9.8-ce.0

pycrypto is unmaintained and makes it impossible to install gitlab-tools on Debian Bullseye

> sudo apt depends python3-gitlab-tools 
python3-gitlab-tools
  Depends: python3-celery
  Depends: python3-cron-descriptor
  Depends: python3-crypto (>= 2.6.1)
[...]

python3-crypto was removed from Debian in the upcoming Bullseye release: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=972184

python3-cryptography seems to be a successor, but for whatever reason even after installing python3-pycryptodome import Crypto still doesn't work.

sqlite3 migration to MySQL or Postgress

Do you have instructions on how to migrate the SQLight Database to MySQL or Postgress?

I have now the problem with ~ 60 mirrors that I keep getting
“sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked”

only a restart of all 3 service help to resolve this for a short time

Option to configure periodical syncs for each mirror

When we cant set webhook to project... we want to mirror.

Current workaround for this issue:

Enable celerybeat in config.py
Click on "Trigger sync" manually
Call webhook from cron using curl

Celeryworker crash

Traceback (most recent call last):
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:   File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 382, in trace_task
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:     R = retval = fun(*args, **kwargs)
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:   File "/usr/lib/python3/dist-packages/flask_celery/__init__.py", line 89, in __call__
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:     return task_base.__call__(self, *_args, **_kwargs)
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:   File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 641, in __protected_call__
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:     return self.run(*args, **kwargs)
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:   File "/usr/lib/python3/dist-packages/celery/app/builtins.py", line 25, in backend_cleanup
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:     app.backend.cleanup()
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:   File "/usr/lib/python3/dist-packages/celery/backends/database/__init__.py", line 178, in cleanup
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]:     Task.date_done < (now - expires)).delete()
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'

gitlab-tools-bgtask has crashed

I run into the following failure with the "cenk1cenk2/gitlab-tools" docker image. Please let me know what you need to debug this issue further.

[START] Starting the celerybeat...
[bgtask] celery beat v5.0.2 (singularity) is starting.
[bgtask] __ - ... __ - _
[bgtask] LocalTime -> 2023-01-02 19:15:58
[bgtask] Configuration ->
[bgtask] . broker -> amqp://guest:**@127.0.0.1:5672//
[bgtask] . loader -> celery.loaders.app.AppLoader
[bgtask] . scheduler -> gitlab_tools.celery_beat.schedulers.DatabaseScheduler
[bgtask]
[bgtask] . logfile -> [stderr]@%WARNING
[bgtask] . maxinterval -> 5.00 seconds (5s)
[bgtask] F0102 19:15:59.032 195392 beat.py:107] beat raised exception <class 'AttributeError'>: AttributeError("'NoneType' object has no attribute 'now'")
[bgtask] Traceback (most recent call last):
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/apps/beat.py", line 105, in start_scheduler
[bgtask] service.start()
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 618, in start
[bgtask] humanize_seconds(self.scheduler.max_interval))
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/kombu/utils/objects.py", line 30, in get
[bgtask] return super().get(instance, owner)
[bgtask] File "/usr/lib/python3.9/functools.py", line 969, in get
[bgtask] val = self.func(instance)
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 662, in scheduler
[bgtask] return self.get_scheduler()
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 653, in get_scheduler
[bgtask] return symbol_by_name(self.scheduler_cls, aliases=aliases)(
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 144, in init
[bgtask] super().init(*args, **kwargs)
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 253, in init
[bgtask] self.setup_schedule()
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 150, in setup_schedule
[bgtask] self.install_default_entries(self.schedule)
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 225, in schedule
[bgtask] self._schedule = self.all_as_schedule()
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 158, in all_as_schedule
[bgtask] s[model.name] = self.Entry(model)
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 34, in init
[bgtask] super().init()
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 124, in init
[bgtask] self.last_run_at = last_run_at or self.default_now()
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 128, in default_now
[bgtask] return self.schedule.now() if self.schedule else self.app.now()
[bgtask] AttributeError: 'NoneType' object has no attribute 'now'
[bgtask] W0102 19:15:59.034 195392 log.py:227] Traceback (most recent call last):
[bgtask] W0102 19:15:59.034 195392 log.py:227] File "/opt/gitlab-tools/manage.py", line 7, in
[bgtask] W0102 19:15:59.034 195392 log.py:227] main()
[bgtask] W0102 19:15:59.034 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 572, in main
[bgtask] W0102 19:15:59.034 195392 log.py:227] getattr(command, 'chosen')() # Execute the function specified by the user.
[bgtask] W0102 19:15:59.034 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 534, in celerybeat
[bgtask] W0102 19:15:59.035 195392 log.py:227] return celery.Beat(
[bgtask] W0102 19:15:59.035 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/apps/beat.py", line 77, in run
[bgtask] W0102 19:15:59.035 195392 log.py:227] self.start_scheduler()
[bgtask] W0102 19:15:59.035 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/apps/beat.py", line 105, in start_scheduler
[bgtask] W0102 19:15:59.035 195392 log.py:227] service.start()
[bgtask] W0102 19:15:59.035 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 618, in start
[bgtask] W0102 19:15:59.036 195392 log.py:227] humanize_seconds(self.scheduler.max_interval))
[bgtask] W0102 19:15:59.036 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/kombu/utils/objects.py", line 30, in get
[bgtask] W0102 19:15:59.036 195392 log.py:227] return super().get(instance, owner)
[bgtask] W0102 19:15:59.036 195392 log.py:227] File "/usr/lib/python3.9/functools.py", line 969, in get
[bgtask] W0102 19:15:59.036 195392 log.py:227] val = self.func(instance)
[bgtask] W0102 19:15:59.036 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 662, in scheduler
[bgtask] W0102 19:15:59.037 195392 log.py:227] return self.get_scheduler()
[bgtask] W0102 19:15:59.037 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 653, in get_scheduler
[bgtask] W0102 19:15:59.037 195392 log.py:227] return symbol_by_name(self.scheduler_cls, aliases=aliases)(
[bgtask] W0102 19:15:59.037 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 144, in init
[bgtask] W0102 19:15:59.037 195392 log.py:227] super().init(*args, **kwargs)
[bgtask] W0102 19:15:59.037 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 253, in init
[bgtask] W0102 19:15:59.037 195392 log.py:227] self.setup_schedule()
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 150, in setup_schedule
[bgtask] W0102 19:15:59.038 195392 log.py:227] self.install_default_entries(self.schedule)
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 225, in schedule
[bgtask] W0102 19:15:59.038 195392 log.py:227] self._schedule = self.all_as_schedule()
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 158, in all_as_schedule
[bgtask] W0102 19:15:59.038 195392 log.py:227] s[model.name] = self.Entry(model)
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 34, in init
[bgtask] W0102 19:15:59.038 195392 log.py:227] super().init()
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 124, in init
[bgtask] W0102 19:15:59.038 195392 log.py:227] self.last_run_at = last_run_at or self.default_now()
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 128, in default_now
[bgtask] W0102 19:15:59.038 195392 log.py:227] return self.schedule.now() if self.schedule else self.app.now()
[bgtask] W0102 19:15:59.038 195392 log.py:227] AttributeError
[bgtask] W0102 19:15:59.038 195392 log.py:227] :
[bgtask] W0102 19:15:59.038 195392 log.py:227] 'NoneType' object has no attribute 'now'

How to build this package from source code OR how to prepare an development enviorment?

Celery worker error

Hello,

Thank you for this great contribution.

I'm having error with the celery worker service (see the attached image), and the encryption keys could not been created.

Could you please help ?

Request failed error while adding a fingerprint

I started to use this tool for using the gitlab pull mirrors.
I installed the tool using the suggested apt-get installation as given in the readme. GitLab and the tools are installed in the same linux (ubuntu 16.4) machine.
The logging in to GitLab Tools page is working fine and login gitlab authorization is working as expected.

Here is my fingerprints screen

I have configured an SSH key for my GitLab account and it is working.

Issue :

While trying to create a pull mirror following are my settings.

And upon saving following error occures.

Attched the server log

gitlab-tools_server.log

Fix missing user checks

Some parts are missing user specific checks like Forms...

Fingerprint error

Hello
Using RHEL7
Install manualy, take /static/node_modules from deb package - missing here
Take error on adding fingerprint or mirror:

[pid: 29400|app: 0|req: 8/8] 192.168.38.138 () {50 vars in 1430 bytes} [Tue Jul 23 07:59:43 2019] POST /fingerprint/fingerprint-check => generated 918 bytes in 253 msecs (HTTP/1.1 200) 2 headers in 72 bytes (2 switches on core 0)
[2019-07-23 07:59:44,611] ERROR in app: Exception on /fingerprint/fingerprint-add [POST]
Traceback (most recent call last):
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/gitlab_tools/views/fingerprint/index.py", line 207, in add_hostname_fingerprint
public_key = import_key(get_user_public_key_path(current_user, flask.current_app.config['USER']))
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/gitlab_tools/tools/crypto.py", line 43, in import_key
return RSA.importKey(f.read())
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/Crypto/PublicKey/RSA.py", line 678, in importKey
if bord(externKey[0])==0x30:
IndexError: index out of range
[pid: 29400|app: 0|req: 9/9] 192.168.38.138 () {50 vars in 1428 bytes} [Tue Jul 23 07:59:44 2019] POST /fingerprint/fingerprint-add => generated 4219 bytes in 68 msecs (HTTP/1.1 500) 2 headers in 100 bytes (1 switches on core 0)

Can you help?

Documetation and Support for Building & Setting Up from Scratch

Thanks for this nice project. I found it via your issue on the gitlab-mirrors project. I like your improvements, but I have to say that the additional complexity is a draw back for deployment.

It's convenient that you supply pre-built packages, but I need to be able to run gitlab-tools from the source provided in the repo. Your build instructions are useful, but they also expect me to install your prebuilt dependencies. I have been able to work around most of this on my own, but I'm still running into issues. Would you be willing to provide some additional support/docs on setting this up from source using only either system packages (any distro) or upstream source?

Option to show background job logs

fingerprint/new : We are sorry but request failed. Check console for more information

I run development env using docker container.
base image is: python:3.5-stretch

gitlab_tools/tools/crypto.py
I add my_transport.banner_timeout = 300 to fix request-new-key error

def get_remote_server_key(ip: str, port: int=22) -> paramiko.pkey.PKey:
    """
    Returns PKey for given server
    :param ip: IP or Hostname
    :param port: Port
    :return: Returns PKey
    """
    print(ip)
    print(port)
    my_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    my_socket.settimeout(10)
    my_socket.connect((ip, port))

    my_transport = paramiko.Transport(my_socket)
    print(my_transport.banner_timeout)
    my_transport.banner_timeout = 300
    print(my_transport.banner_timeout)
    my_transport.start_client()
    ssh_key = my_transport.get_remote_server_key()

    my_transport.close()
    my_socket.close()
    return ssh_key

console output:

I0325 08:52:33.644 2606 gitlab_tools.py:136] Server is running at http://0.0.0.0:5000/
I0325 08:52:33.645 2606 gitlab_tools.py:137] Flask version: 1.0.2
I0325 08:52:33.645 2606 gitlab_tools.py:138] DEBUG: True
I0325 08:52:33.645 2606 gitlab_tools.py:139] STATIC_FOLDER: /var/app/gitlab-tools/gitlab_tools/static
W0325 08:52:33.692 2606 _internal.py:97]  * Debugger is active!
I0325 08:52:33.693 2606 _internal.py:97]  * Debugger PIN: 266-783-398
D0325 08:52:38.310 2606 transport.py:1746] starting thread (client mode): 0x44dacd68
D0325 08:52:38.311 2606 transport.py:1746] Local version/idstring: SSH-2.0-paramiko_2.4.2
D0325 08:52:38.315 2606 transport.py:1746] Banner: HTTP/1.1 400 Bad Request
D0325 08:52:38.315 2606 transport.py:1746] Banner: Server: nginx
D0325 08:52:38.316 2606 transport.py:1746] Banner: Date: Mon, 25 Mar 2019 08:52:38 GMT
D0325 08:52:38.316 2606 transport.py:1746] Banner: Content-Type: text/html
D0325 08:52:38.316 2606 transport.py:1746] Banner: Content-Length: 166
D0325 08:52:38.317 2606 transport.py:1746] Banner: Connection: close
D0325 08:52:38.317 2606 transport.py:1746] Banner: 
D0325 08:52:38.317 2606 transport.py:1746] Banner: <html>
D0325 08:52:38.318 2606 transport.py:1746] Banner: <head><title>400 Bad Request</title></head>
D0325 08:52:38.318 2606 transport.py:1746] Banner: <body bgcolor="white">
D0325 08:52:38.318 2606 transport.py:1746] Banner: <center><h1>400 Bad Request</h1></center>
D0325 08:52:38.319 2606 transport.py:1746] Banner: <hr><center>nginx</center>
D0325 08:52:38.319 2606 transport.py:1746] Banner: </body>
D0325 08:52:38.320 2606 transport.py:1746] Banner: </html>
E0325 08:52:38.321 2606 transport.py:1746] Exception: Error reading SSH protocol banner
E0325 08:52:38.322 2606 transport.py:1744] Traceback (most recent call last):
E0325 08:52:38.322 2606 transport.py:1744]   File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 2138, in _check_banner
E0325 08:52:38.323 2606 transport.py:1744]     buf = self.packetizer.readline(timeout)
E0325 08:52:38.323 2606 transport.py:1744]   File "/usr/local/lib/python3.5/site-packages/paramiko/packet.py", line 367, in readline
E0325 08:52:38.323 2606 transport.py:1744]     buf += self._read_timeout(timeout)
E0325 08:52:38.324 2606 transport.py:1744]   File "/usr/local/lib/python3.5/site-packages/paramiko/packet.py", line 563, in _read_timeout
E0325 08:52:38.324 2606 transport.py:1744]     raise EOFError()
E0325 08:52:38.324 2606 transport.py:1744] EOFError
E0325 08:52:38.325 2606 transport.py:1744] 
E0325 08:52:38.325 2606 transport.py:1744] During handling of the above exception, another exception occurred:
E0325 08:52:38.325 2606 transport.py:1744] 
E0325 08:52:38.326 2606 transport.py:1744] Traceback (most recent call last):
E0325 08:52:38.326 2606 transport.py:1744]   File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 1966, in run
E0325 08:52:38.326 2606 transport.py:1744]     self._check_banner()
E0325 08:52:38.326 2606 transport.py:1744]   File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 2143, in _check_banner
E0325 08:52:38.326 2606 transport.py:1744]     "Error reading SSH protocol banner" + str(e)
E0325 08:52:38.327 2606 transport.py:1744] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
E0325 08:52:38.327 2606 transport.py:1744] 
I0325 08:52:38.341 2606 _internal.py:97] 172.21.0.1 - - [25/Mar/2019 08:52:38] "POST /fingerprint/fingerprint-check HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 2138, in _check_banner
    buf = self.packetizer.readline(timeout)
  File "/usr/local/lib/python3.5/site-packages/paramiko/packet.py", line 367, in readline
    buf += self._read_timeout(timeout)
  File "/usr/local/lib/python3.5/site-packages/paramiko/packet.py", line 563, in _read_timeout
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/var/app/gitlab-tools/gitlab_tools/views/fingerprint/index.py", line 176, in check_hostname_fingerprint
    return check_fingerprint_hostname(hostname_string)
  File "/var/app/gitlab-tools/gitlab_tools/views/fingerprint/index.py", line 40, in check_fingerprint_hostname
    found, remote_server_key = check_hostname(hostname, known_hosts_path)
  File "/var/app/gitlab-tools/gitlab_tools/tools/fingerprint.py", line 22, in check_hostname
    remote_server_key = get_remote_server_key(hostname)
  File "/var/app/gitlab-tools/gitlab_tools/tools/crypto.py", line 75, in get_remote_server_key
    my_transport.start_client()
  File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 587, in start_client
    raise e
  File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 1966, in run
    self._check_banner()
  File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 2143, in _check_banner
    "Error reading SSH protocol banner" + str(e)
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner

Deploy

Remove is_public and use visibility level for projects

Refactor fingerprint check to use ssh info from gitlab project instead of hardcoded one in config

Issue

Currently when checking fingerprint, we use GITLAB_SSH config option with hostname:port of gitlab SSH server, but gitlab API actually returns a ssh URI for created/existing project. Issue is that project creation is handled in async task (celery) and no user interaction is possible, that means that mirror has to already exist in database when task is triggered...

Repository names are not correct

Hey again @Salamek ,

I just wonder now with the latest additions I have encountered two issues:

One of them is very fatal and it modifies the git URLs with the user id, I am not sure if this was the behavior before, so I am not a hundred percent confident that this is the actual error.

The error message is: Error lines received while fetching: error: failed to push some refs to 'gitlab.kilic.dev_1:libraries/cz-cc.git'

My debugging output:

root@bb31e0793154:/home/service/repositories/1/pull/2/7# git pull --mirror
fatal: this operation must be run in a work tree
root@bb31e0793154:/home/service/repositories/1/pull/2/7# cd ..
root@bb31e0793154:/home/service/repositories/1/pull/2# ls
2  7
root@bb31e0793154:/home/service/repositories/1/pull/2# git ^C
root@bb31e0793154:/home/service/repositories/1/pull/2# cd 7
root@bb31e0793154:/home/service/repositories/1/pull/2/7# git remote show origin
ssh: Could not resolve hostname github.com_1: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
root@bb31e0793154:/home/service/repositories/1/pull/2/7# git remote show gitlab
ssh: Could not resolve hostname gitlab.kilic.dev_1: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
root@bb31e0793154:/home/service/repositories/1/pull/2/7#

The other one is that it does not auto check the form options from the Advanced Options anymore, therefore it changes the options every time the repository is edited unless it is been done manually.

What can I do to help with these issues?

These do not show inside the frontend task logs as errors but on the stdoutput of the application.

Support provisioning via docker-compose

It would be nice if an example of running this service was available for running via docker-compose. Mainly because this seems to have a lot of moving parts and if one wanted to get set up quickly, docker-compose would be easiest from a systems perspective.

Use new post_install, create and configure postgresql

Pull mirrors, SSH, Fingerprint

Hello.

gitlab and gitlab-tools [1.2.9] will be deployed in my docker.

After adding to "Pull Mirrors" repository.

In "FingerPrints" I get two entries with the type of key SSH-ED25519:

gitlab.mydomain.com
github.com

After I click "Trigger Sync" in "Pull Mirrors" in the log I get:

Traceback (most recent call last):
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/app/trace.py", line 409, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_celery/init.py", line 96, in call
return task_base.call(self, *_args, **_kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/app/trace.py", line 701, in protected_call
return self.run(*args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_celery/init.py", line 150, in wrapped
ret_value = func(*args, **kwargs)
File "/opt/gitlab-tools/gitlab_tools/tasks/gitlab_tools.py", line 371, in sync_pull_mirror
Git.sync_mirror(namespace_path, str(mirror.id), git_remote_source, git_remote_target)
File "/opt/gitlab-tools/gitlab_tools/tools/Git.py", line 39, in sync_mirror
repo.remotes.gitlab.push(
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/git/remote.py", line 1045, in push
return self._get_push_info(proc, progress, kill_after_timeout=kill_after_timeout)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/git/remote.py", line 893, in _get_push_info
proc.wait(stderr=stderr_text)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/git/cmd.py", line 559, in wait
raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git push --porcelain --mirror --force --prune gitlab
stderr: 'fatal: Could not read from remote repository.'

I watched the file "repositories/1/pull/1/1/config"

[remote "gitlab"]
url = ssh://[email protected]_1:10022/thirdparty/date.git

If I make from the container:
ssh [email protected]_1 -p 10022

The authenticity of host '[gitlab.mydomain.com]:10022 ([192.168.xxx.xxx]:10022)' can't be established.
ECDSA key fingerprint is SHA256:oc1Buil1L3tCDGRTh1QvQdaK6R9nfGE2JvjaD+W3hZI.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

ECDSA

I introduced "yes"
A record was made in .ssh/known_host_1 with key type ecdsa-sha2-nistp256

After this, the updates were earned from gitlab-tools.
But I don't like to do this for every user.

I made an appointment with a file /etc/ssh/ssh_config

Host *
HostKeyAlgorithms ^ssh-ed25519

And repeated
ssh [email protected]_1 -p 10022

The authenticity of host '[gitlab.mydomain.com]:10022 ([192.168.xxx.xxx]:10022)' can't be established.
ED25519 key fingerprint is SHA256:oc1Buil1L3tCDGRTh1QvQdaK6R9nfGE2JvjaD+W3hZI.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

He asked for "fingerprint"

How to fix it?
In order not to make ssh [email protected]_X -p 10022 for each user.
Thank you.

Support of custom SSH ports

Hello,

We can't use pull mirrors with custom ssh server port, could we help with that ? Any advice ?

Thank you.

Gitlab Tools on Fresh Ubuntu Jammy 22.04

I installed a new Gitlab with Ubuntu 22.04 Jammy, and when installing Gitlab tools I noticed that the python3-flask package was not available in the required version.

And in the end there are too many Python packages that are not compatible and require a downgrade.
so it must be an Ubuntu Focal 20.04 after all.

Here is my attempt to resolve the dependencies.

Try to install Gitlab-Tool

apt-get install gitlab-tools

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 gitlab-tools : Depends: python3-flask (< 1.2) but 2.0.1-2ubuntu1.1 is to be installed
E: Unable to correct problems, you have held broken packages.

Check Version of python3-flask

apt-cache policy python3-flask

python3-flask:
  Installed: (none)
  Candidate: 2.0.1-2ubuntu1.1
  Version table:
     2.0.1-2ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     2.0.1-2ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Check Version of gitlab-tools

apt-cache policy gitlab-tools

gitlab-tools:
  Installed: (none)
  Candidate: 1.4.7-1
  Version table:
     1.4.7-1 500
        500 https://repository.salamek.cz/deb/pub all/main amd64 Packages

Download vom PPA
https://answers.launchpad.net/ubuntu/jammy/amd64/python3-flask/1.1.2-2

Install python3-flask via dpkg to get dependency

wget http://launchpadlibrarian.net/504048010/python3-flask_1.1.2-2_all.deb

dpkg -i python3-flask_1.1.2-2_all.deb

Selecting previously unselected package python3-flask.
(Reading database ... 110161 files and directories currently installed.)
Preparing to unpack python3-flask_1.1.2-2_all.deb ...
Unpacking python3-flask (1.1.2-2) ...
dpkg: dependency problems prevent configuration of python3-flask:
 python3-flask depends on python3-click (>= 5.1); however:
  Package python3-click is not installed.
 python3-flask depends on python3-itsdangerous (>= 0.24); however:
  Package python3-itsdangerous is not installed.
 python3-flask depends on python3-jinja2 (>= 2.10); however:
  Package python3-jinja2 is not installed.
 python3-flask depends on python3-werkzeug (>= 0.15); however:
  Package python3-werkzeug is not installed.

dpkg: error processing package python3-flask (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 python3-flask

Install dependency and python3-flask

apt-get install  -f

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  javascript-common libjs-jquery python-babel-localedata python3-babel python3-cffi-backend python3-click python3-colorama python3-cryptography
  python3-itsdangerous python3-jinja2 python3-markupsafe python3-openssl python3-pyinotify python3-simplejson python3-six python3-tz python3-werkzeug
Suggested packages:
  apache2 | lighttpd | httpd python-cryptography-doc python3-cryptography-vectors python-jinja2-doc python-openssl-doc python3-openssl-dbg python-pyinotify-doc
  ipython3 python-werkzeug-doc python3-lxml python3-watchdog
The following NEW packages will be installed:
  javascript-common libjs-jquery python-babel-localedata python3-babel python3-cffi-backend python3-click python3-colorama python3-cryptography
  python3-itsdangerous python3-jinja2 python3-markupsafe python3-openssl python3-pyinotify python3-simplejson python3-six python3-tz python3-werkzeug
0 upgraded, 17 newly installed, 0 to remove and 1 not upgraded.
1 not fully installed or removed.
Need to get 0 B/6294 kB of archives.
After this operation, 32.9 MB of additional disk space will be used.

Set on Hold

apt-mark hold python3-flask

python3-flask set on hold.

Install Gitlab-tools

apt-get install gitlab-tools

Reading package lists... Done
Building dependency tree... Done

Reading state information... Done
The following additional packages will be installed:
  alembic erlang-asn1 erlang-base erlang-crypto erlang-eldap erlang-ftp erlang-inets erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key
  erlang-runtime-tools erlang-snmp erlang-ssl erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl git git-man libatomic1 libcurl3-gnutls liberror-perl
  libjemalloc2 libjs-sphinxdoc libjs-underscore liblua5.1-0 liblzf1 libpq5 libsctp1 libsodium23 lua-bitop lua-cjson patch python3-alembic python3-amqp
  python3-anyjson python3-bcrypt python3-billiard python3-blinker python3-celery python3-certifi python3-chardet python3-click-didyoumean python3-click-plugins
  python3-click-repl python3-cron-descriptor python3-dateutil python3-docopt python3-ephem python3-flask-babel python3-flask-celery-tools python3-flask-login
  python3-flask-migrate python3-flask-sqlalchemy python3-git python3-gitdb python3-gitlab python3-greenlet python3-idna python3-kombu python3-mako
  python3-memcache python3-nacl python3-paramiko python3-prompt-toolkit python3-psycopg2 python3-pycryptodome python3-pygments python3-pyparsing python3-raven
  python3-redis python3-requests python3-requests-toolbelt python3-smmap python3-sqlalchemy python3-sqlalchemy-ext python3-urllib3 python3-vine python3-wcwidth
  python3-wtforms rabbitmq-server redis-server redis-tools socat
Suggested packages:
  erlang erlang-manpages erlang-doc git-daemon-run | git-daemon-sysvinit git-doc git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn lksctp-tools
  diffutils-doc python-amqp-doc python-billiard-doc python-blinker-doc python-celery-doc python-flask-babel-doc python-flask-login-doc python-git-doc
  python-gitlab-doc python-greenlet-dev python-greenlet-doc python-kombu-doc python3-boto python3-django python3-pymongo python3-beaker python-mako-doc memcached
  python-nacl-doc python3-gssapi python3-invoke python-psycopg2-doc python-pygments-doc ttf-bitstream-vera python-pyparsing-doc python3-hiredis python3-socks
  python-requests-doc python3-nose python-sqlalchemy-doc python3-fdb python3-pymssql python3-mysqldb python3-asyncpg python3-aiosqlite python3-django-localflavor
  ruby-redis
The following NEW packages will be installed:
  alembic erlang-asn1 erlang-base erlang-crypto erlang-eldap erlang-ftp erlang-inets erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key
  erlang-runtime-tools erlang-snmp erlang-ssl erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl git git-man gitlab-tools libatomic1 libcurl3-gnutls
  liberror-perl libjemalloc2 libjs-sphinxdoc libjs-underscore liblua5.1-0 liblzf1 libpq5 libsctp1 libsodium23 lua-bitop lua-cjson patch python3-alembic
  python3-amqp python3-anyjson python3-bcrypt python3-billiard python3-blinker python3-celery python3-certifi python3-chardet python3-click-didyoumean
  python3-click-plugins python3-click-repl python3-cron-descriptor python3-dateutil python3-docopt python3-ephem python3-flask-babel python3-flask-celery-tools
  python3-flask-login python3-flask-migrate python3-flask-sqlalchemy python3-git python3-gitdb python3-gitlab python3-greenlet python3-idna python3-kombu
  python3-mako python3-memcache python3-nacl python3-paramiko python3-prompt-toolkit python3-psycopg2 python3-pycryptodome python3-pygments python3-pyparsing
  python3-raven python3-redis python3-requests python3-requests-toolbelt python3-smmap python3-sqlalchemy python3-sqlalchemy-ext python3-urllib3 python3-vine
  python3-wcwidth python3-wtforms rabbitmq-server redis-server redis-tools socat
0 upgraded, 86 newly installed, 0 to remove and 1 not upgraded.
Need to get 50.0 MB/50.0 MB of archives.
After this operation, 127 MB of additional disk space will be used.

On this Point you need to add a ubuntu Focal Mirror and Downgrade some Packages

apt-get install -t focal python3-flask-babel=0.11.2-3 python3-flask-login=0.4.1-3 python3-flask-migrate=2.5.2-1 python3-flask-sqlalchemy=2.1-4 python3-itsdangerous=1.1.0-1 python3-werkzeug=0.16.1+dfsg1-2ubuntu0.1
apt-mark hold python3-flask-babel python3-flask-login python3-flask-migrate python3-flask-sqlalchemy python3-itsdangerous python3-werkzeug

but no luck here either because of Python 3.7 vs 3.10

Add new mirror or change the Cron got an error

when i add a now mirror with a Cron or edit a old mirror and change the Cron then i get this.

E0809 10:33:39.827 483829 app.py:1891] Exception on /pull-mirror/edit/40 [POST]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib/python3/dist-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/gitlab_tools/views/pull_mirror/index.py", line 230, in edit_mirror
    if flask.request.method == 'POST' and form.validate():
  File "/usr/lib/python3/dist-packages/gitlab_tools/forms/pull_mirror.py", line 170, in validate
    ExpressionDescriptor(self.periodic_sync.data, throw_exception_on_parse_error=True)
  File "/usr/lib/python3/dist-packages/cron_descriptor/ExpressionDescriptor.py", line 69, in __init__
    raise WrongArgumentException("Unknown {} configuration argument".format(kwarg))
cron_descriptor.Exception.WrongArgumentException: Unknown throw_exception_on_parse_error configuration argument

requirements.txt does not provide a repeatable experience

It would be great if you replaced your requirements.txt file with a pip freeze version. This way the exact version of all dependencies used is known. Imagine 1 year from now when all of the dependencies have changed (likely). There’s no guarantee your project would work with those latest versions.

By pinning your dependencies you ensure users install the exact version of the dependencies your application needs rather than relying on a floating version.

repository.salamek.cz is not available

Hi,

Thanks for your great tool.
Maybe I just have a bad timing, but https://repository.salamek.cz/ doesn't seem to be reachable for me.

➜  ~ curl -v https://repository.salamek.cz/
*   Trying 78.80.161.207:443...
* connect to 78.80.161.207 port 443 failed: No route to host
* Failed to connect to repository.salamek.cz port 443 after 895 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to repository.salamek.cz port 443 after 895 ms: Couldn't connect to server

Static files not found

Any file from directory is not found (Error 404).

Try to add to NGINX

location /static {
    root /usr/lib/python3/dist-packages/gitlab_tools/static;
    gzip_static on;
    expires max;
    add_header Cache-Control public;
}

But still not found

Status code 500 on /pull-mirror/new

Hi,

I just tried setting up gitlab-tools and it seems like everything worked correctly.

But when I now try to add a Pull Mirror I only get an empty page with 500 after clicking on save.

Is there any way to debug what error is occurring?

Thanks for your work

No module named 'celery.utils.time' in celerybeat

support celery 5

related to #31

The way celery is started, doesn't seem to be supported any more in celery 5 (shipped with Debian Bullseye)

gitlab-tools/gitlab_tools/bin/gitlab_tools.py

Lines 513 to 521 in 2b66194

 @command 

 def celeryworker(): 

 options = parse_options() 

 setup_logging('celeryworker{}'.format(OPTIONS['--name']), logging.DEBUG if options.DEBUG else logging.WARNING) 

 app = create_app(options, no_sql=True) 

 Logging._setup = True 

 celery_args = ['celery', 'worker', '-n', OPTIONS['--name'], '-C', '--autoscale=10,1', '--without-gossip'] 

 with app.app_context(): 

 return celery_main(celery_args)

https://github.com/celery/celery/blob/552e067b40198429cd7c866a397069366ac8e530/celery/bin/celery.py#L206-L213

Pull mirroring does not work for existing repository

If the mirror and the mirrored both repos existing in the gitlab there is no way to setup mirroring between them.

Use case:
For syncing fork repositories and their upstream repo, the tool is not supporting the sync.

Expected behavior:

The tool should be able to setup sync between the two existing repos (fork and original)
And the pull sync should have the merging capabilities for the fork repo

ImportError when Setting up development environment

Since I use CentOS I was unable to install it using the repo. I'm trying to use from source following your wiki

But when I reach the python manage.py to test the installation I get an error the error: ImportError: cannot import name 'ImmutableDict'

Full Traceback

Traceback (most recent call last):
  File "manage.py", line 4, in <module>
    from gitlab_tools.bin.gitlab_tools import main
  File "/root/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 75, in <module>
    from gitlab_tools.application import create_app, get_config
  File "/root/gitlab-tools/gitlab_tools/application.py", line 9, in <module>
    from flask_babel import gettext
  File "/root/gitlab-tools/venv/lib/python3.6/site-packages/flask_babel/__init__.py", line 19, in <module>
    from werkzeug import ImmutableDict
ImportError: cannot import name 'ImmutableDict'

Accept public key of gitlab server

This took me a while to debug and I am still not completely sure what the issue is.

On every mirror I got the following error:

Jul 22 18:59:58 vServer gitlab-tools[8600]: git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)                                                                 
Jul 22 18:59:58 vServer gitlab-tools[8600]:   cmdline: git push --mirror --porcelain gitlab                                                                                   
Jul 22 18:59:58 vServer gitlab-tools[8600]:   stderr: 'fatal: Could not read from remote repository.                                                                          
Jul 22 18:59:58 vServer gitlab-tools[8600]: Please make sure you have the correct access rights                                                                               
Jul 22 18:59:58 vServer gitlab-tools[8600]: and the repository exists.'

Then I had the idea to login as and then cd repositories/1/pull/2/9/ and run git push --mirror gitlab manually.
After accepting the Are you sure you want to continue connecting (yes/no)? yes the push worked.

And afterwards all pushes from gitlab-tools also worked.
I guess the known_hosts entry was missing.

Pull mirror: cannot create mirror inside user namespace, only groups work

Ajax load groups and projects

The redirect URI included is not valid

I have github-tools installed and running, but when I sign in I get the following error in GitLab:

The redirect URI included is not valid

My callback URL in the GitLab application is set to https://gitlab-tools.domain/sign/in/do, so I'm not sure what else I can check or do to troubleshoot this.

EOF in transport thread

Hi.
Great work here, thank you for this tool!
I am having an issue trying to pull mirror a github repo.
I have successfully created a pull mirror to a gitlab repo, but github doesn't seem to work.
gitlab-tools doesn't want to accept the github.com fingerprint.

Relevant log section:

I0824 19:02:08.646 26155 log.py:282] Connected (version 2.0, client babeld-216c4091)
D0824 19:02:08.647 26155 log.py:282] kex algos:['curve25519-sha256', '[email protected]', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256'] server key:['ssh-dss', 'rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa'] client encrypt:['[email protected]', '[email protected]', '[email protected]', 'aes256-ctr', 'aes192-ctr', 'aes128-ctr', 'aes256-cbc', 'aes192-cbc', 'aes128-cbc'] server encrypt:['[email protected]', '[email protected]', '[email protected]', 'aes256-ctr', 'aes192-ctr', 'aes128-ctr', 'aes256-cbc', 'aes192-cbc', 'aes128-cbc'] client mac:['[email protected]', '[email protected]', '[email protected]', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['[email protected]', '[email protected]', '[email protected]', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] client compress:['none', 'zlib', '[email protected]'] server compress:['none', 'zlib', '[email protected]'] client lang:[''] server lang:[''] kex follows?False
D0824 19:02:08.647 26155 log.py:282] Kex agreed: diffie-hellman-group-exchange-sha256
D0824 19:02:08.647 26155 log.py:282] Cipher agreed: aes128-ctr
D0824 19:02:08.647 26155 log.py:282] MAC agreed: hmac-sha2-256
D0824 19:02:08.647 26155 log.py:282] Compression agreed: none
D0824 19:02:08.879 26155 log.py:282] Got server p (2048 bits)
D0824 19:02:09.029 26155 log.py:282] kex engine KexGexSHA256 specified hash_algo <built-in function openssl_sha256>
D0824 19:02:09.030 26155 log.py:282] Switch to new keys ...
D0824 19:02:09.031 26155 log.py:282] EOF in transport thread
I0824 19:02:09.040 26155 _internal.py:87] 10.7.151.242 - - [24/Aug/2019 19:02:09] "POST /fingerprint/fingerprint-add HTTP/1.1" 500 -

Is there a way to manually add a fingerprint?
Am I just being stupid? :)

	@command
	def celeryworker():
	options = parse_options()
	setup_logging('celeryworker{}'.format(OPTIONS['--name']), logging.DEBUG if options.DEBUG else logging.WARNING)
	app = create_app(options, no_sql=True)
	Logging._setup = True
	celery_args = ['celery', 'worker', '-n', OPTIONS['--name'], '-C', '--autoscale=10,1', '--without-gossip']
	with app.app_context():
	return celery_main(celery_args)