salamek / gitlab-tools Goto Github PK
View Code? Open in Web Editor NEWWeb application providing tools not avaiable in GitLab CE (Gitlab push/pull mirror and more)
License: GNU General Public License v3.0
Web application providing tools not avaiable in GitLab CE (Gitlab push/pull mirror and more)
License: GNU General Public License v3.0
with Gitlab v16.6.1
When I click on Pull Mirror
500 Internal Server Error
Uh-oh you found a bug! On Email Has Been Sent to the Admins List.
With Home, Push Mirror, fingerprint everything is ok.
E1211 11:07:36.859 3122222 app.py:1891] Exception on /pull-mirror/ [GET]
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3/dist-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3/dist-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/lib/python3/dist-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3/dist-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/lib/python3/dist-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/gitlab_tools/views/pull_mirror/index.py", line 116, in get_mirror
return flask.render_template('pull_mirror.index.pull_mirror.html', pagination=pagination), 200
File "/usr/lib/python3/dist-packages/flask/templating.py", line 137, in render_template
return _render(
File "/usr/lib/python3/dist-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/gitlab_tools/views/pull_mirror/templates/pull_mirror.index.pull_mirror.html", line 1, in <module>
{% extends "base.html" %}
File "/usr/lib/python3/dist-packages/gitlab_tools/templates/base.html", line 72, in <module>
{% block body %}{% endblock %}
File "/usr/lib/python3/dist-packages/gitlab_tools/views/pull_mirror/templates/pull_mirror.index.pull_mirror.html", line 26, in <module>
<tr class="bg-{{item.task_results.first()|format_task_status_class}}" title="{{item.note}}">
File "/usr/lib/python3/dist-packages/gitlab_tools/middleware.py", line 134, in format_task_status_class_filter
}.get(task_result.taskmeta.status, 'warning')
AttributeError: 'NoneType' object has no attribute 'status'
I1211 11:07:36.861 3122222 _internal.py:122] 10.83.10.105 - - [11/Dec/2023 11:07:36] "GET /pull-mirror/ HTTP/1.1" 500 -
I can't include a .
in a project name even though it's valid in GitLab without receiving an error:
Project name cannot contain special characters
I can manually trigger the sync using the web UI or webhook. But the period sync never triggered. Did I missed something trivial?
Periodic sync: Every minute (*/1 * * * *)
94d5e2f-9610-4957-9089-6c5be5e8f57e sync_pull_mirror Web hook SUCCESS None Jun 30, 2022, 6:16:40 AM
b5d3d387-e7c2-492b-a688-57b2e0753ed3 sync_pull_mirror Manual SUCCESS None Jun 30, 2022, 6:04:41 AM
06af6586-90e3-4173-9ee0-ac276dc2b6b4 create_ssh_config Manual SUCCESS None Jun 30, 2022, 5:52:15 AM
29ca70fa-52e5-4edb-ad8b-561e3b8c279b save_pull_mirror Manual SUCCESS None Jun 30, 2022, 5:52:19 AM
ac0bb692-042a-4b9a-841e-64b8b9ed8e31 sync_pull_mirror Manual SUCCESS None Jun 30, 2022, 5:43:10 AM
6fe154ee-956f-4bf2-92ee-8396677a95cf create_ssh_config Manual SUCCESS None Jun 30, 2022, 5:41:41 AM
4bf3c0ec-b0c5-4024-b1b5-6aa8e3dd2448 save_pull_mirror Manual SUCCESS None Jun 30, 2022, 5:41:47 AM
be48cddf-a95b-4348-866f-c211d8f191e4 sync_pull_mirror Manual SUCCESS None Jun 29, 2022, 6:00:21 AM
2ffc1628-9753-402a-93bf-0656f750ef5e create_ssh_config Manual SUCCESS None Jun 21, 2022, 10:38:10 PM
4952c8b1-b81a-49df-97bf-02a3f7cbde64 save_pull_mirror Manual SUCCESS None Jun 21, 2022, 10:38:19 PM
gitlab-tools git:(main) kubectl -n gitlab-tools logs -f gitlab-tools-649f54b688-hvl48
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 00-user-directory.sh: applying...
[fix-attrs.d] 00-user-directory.sh: exited 0.
[fix-attrs.d] 10-ssh-keys.sh: applying...
[fix-attrs.d] 10-ssh-keys.sh: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-generate-configuration.sh: executing...
[START] Checking for required variables...
[START] Testing required environment variables...
[FINISH] All required environment variables are in place.
[START] Setting defaults for optional environment variables...
[FINISH] Set some sane-defaults for environment variables.
[START] Generating configuration file...
[cont-init.d] 00-generate-configuration.sh: exited 0.
[cont-init.d] 10-migrate-database.sh: executing...
[START] Migrating the database...
[START] Creating database tables...
D0630 11:16:36.484 3047 cmd.py:870] Popen(['git', 'version'], cwd=/opt/gitlab-tools, universal_newlines=False, shell=None, istream=None)
D0630 11:16:36.490 3047 cmd.py:870] Popen(['git', 'version'], cwd=/opt/gitlab-tools, universal_newlines=False, shell=None, istream=None)
D0630 11:16:36.541 3047 base.py:272] Configuring Raven for host: None
I0630 11:16:36.542 3047 base.py:219] Raven is not configured (logging is disabled). Please see the documentation for more information.
/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py:220: SADeprecationWarning: The Engine.table_names() method is deprecated and will be removed in a future release. Please refer to Inspector.get_table_names(). (deprecated since: 1.4)
tables_before = set(db.engine.table_names())
/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py:222: SADeprecationWarning: The Engine.table_names() method is deprecated and will be removed in a future release. Please refer to Inspector.get_table_names(). (deprecated since: 1.4)
tables_after = set(db.engine.table_names())
[START] Migrating the database...
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> 56189bfb2c5f, empty message
Traceback (most recent call last):
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.DuplicateTable: relation "fingerprint" already exists
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/gitlab-tools/manage.py", line 7, in <module>
main()
File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 558, in main
getattr(command, 'chosen')() # Execute the function specified by the user.
File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 553, in _db
cli.main(args=sys.argv[1:])
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask/cli.py", line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/cli.py", line 136, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/__init__.py", line 96, in wrapped
f(*args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_migrate/__init__.py", line 271, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/command.py", line 320, in upgrade
script.run_env()
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/script/base.py", line 563, in run_env
util.load_python_file(self.dir, "env.py")
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 790, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/opt/gitlab-tools/gitlab_tools/migrations/env.py", line 89, in <module>
run_migrations_online()
File "/opt/gitlab-tools/gitlab_tools/migrations/env.py", line 81, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
self.get_context().run_migrations(**kw)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
step.migration_fn(**kw)
File "/opt/gitlab-tools/gitlab_tools/migrations/versions/56189bfb2c5f_.py", line 21, in upgrade
op.create_table('fingerprint',
File "<string>", line 8, in create_table
File "<string>", line 3, in create_table
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/ops.py", line 1254, in create_table
return operations.invoke(op)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/base.py", line 392, in invoke
return fn(self, operation)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/operations/toimpl.py", line 114, in create_table
operations.impl.create_table(table)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/ddl/impl.py", line 352, in create_table
self._exec(schema.CreateTable(table))
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/alembic/ddl/impl.py", line 193, in _exec
return conn.execute(construct, multiparams)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1306, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
return connection._execute_ddl(
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1398, in _execute_ddl
ret = self._execute_context(
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
self._handle_dbapi_exception(
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
util.raise_(
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.DuplicateTable) relation "fingerprint" already exists
[SQL:
CREATE TABLE fingerprint (
updated TIMESTAMP WITHOUT TIME ZONE,
created TIMESTAMP WITHOUT TIME ZONE,
id SERIAL NOT NULL,
user_id INTEGER,
hostname VARCHAR(255),
sha256_fingerprint VARCHAR(255),
hashed_hostname VARCHAR(255),
PRIMARY KEY (id),
FOREIGN KEY(user_id) REFERENCES "user" (id),
CONSTRAINT _user_id_hashed_hostname_uc UNIQUE (user_id, hashed_hostname)
)
]
(Background on this error at: https://sqlalche.me/e/14/f405)
[INFO] Migrations are failed or already applied.
[cont-init.d] 10-migrate-database.sh: exited 0.
[cont-init.d] 20-start-services.sh: executing...
-------------------------
[START] Starting Nginx server...
Starting nginx: nginx.
-------------------------
[START] Starting RabbitMQ server...
Starting RabbitMQ Messaging Server: rabbitmq-server.
-------------------------
[START] Starting Redis server...
Starting redis-server: redis-server.
[cont-init.d] 20-start-services.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[START] Starting the celerybeat...
[START] Starting uwsgi...
[START] Starting the celeryworker...
[services.d] done.
[app] [uWSGI] getting INI configuration from /etc/uwsgi/gitlab-tools.ini
[app] *** Starting uWSGI 2.0.19.1-debian (64bit) on [Thu Jun 30 11:16:44 2022] ***
[app] compiled with version: 10.2.1 20210110 on 11 June 2021 09:08:33
[app] os: Linux-5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09)
[app] nodename: gitlab-tools-649f54b688-hvl48
[app] machine: x86_64
[app] clock source: unix
[app] pcre jit disabled
[app] detected number of CPU cores: 2
[app] current working directory: /opt/gitlab-tools
[app] detected binary path: /usr/bin/uwsgi-core
[app] chdir() to /opt/gitlab-tools
[app] your memory page size is 4096 bytes
[app] detected max file descriptor number: 1048576
[app] lock engine: pthread robust mutexes
[app] thunder lock: enabled
[app] uwsgi socket 0 bound to UNIX address /tmp/gitlab-tools.sock fd 3
[app] Python version: 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
[app] PEP 405 virtualenv detected: /opt/gitlab-tools/venv
[app] Set PythonHome to /opt/gitlab-tools/venv
[app] Python main interpreter initialized at 0x56251c670780
[app] python threads support enabled
[app] your server socket listen backlog is limited to 100 connections
[app] your mercy for graceful operations on workers is 60 seconds
[app] mapped 203216 bytes (198 KB) for 1 cores
[app] *** Operational MODE: single process ***
[app] WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x56251c670780 pid: 3438 (default app)
[app] spawned uWSGI master process (pid: 3438)
[app] spawned uWSGI worker 1 (pid: 3455, cores: 1)
[bgtask] celery beat v5.0.2 (singularity) is starting.
[bgtask] __ - ... __ - _
[bgtask] LocalTime -> 2022-06-30 11:16:46
[bgtask] Configuration ->
[bgtask] . broker -> amqp://guest:**@127.0.0.1:5672//
[bgtask] . loader -> celery.loaders.app.AppLoader
[bgtask] . scheduler -> celery.beat.PersistentScheduler
[bgtask] . db -> /home/service/celery_beat.db
[bgtask] . logfile -> [stderr]@%WARNING
[bgtask] . maxinterval -> 5.00 minutes (300s)
[bgtask] W0630 11:16:46.060 3442 beat.py:564] DB Reset: Account for new __version__ field
[worker]
[worker] -------------- celery@1 v5.0.2 (singularity)
[worker] --- ***** -----
[worker] -- ******* ---- Linux-5.10.0-15-amd64-x86_64-with-glibc2.31 2022-06-30 11:16:46
[worker] - *** --- * ---
[worker] - ** ---------- [config]
[worker] - ** ---------- .> app: gitlab_tools.application:0x7f3970dbc7f0
[worker] - ** ---------- .> transport: amqp://guest:**@127.0.0.1:5672//
[worker] - ** ---------- .> results: postgresql://gitlab-tools:**@postgresql13.postgresql13.svc.cluster.local/gitlab-tools
[worker] - *** --- * --- .> concurrency: {min=1, max=10} (prefork)
[worker] -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
[worker] --- ***** -----
[worker] -------------- [queues]
[worker] .> celery exchange=celery(direct) key=celery
[worker]
[worker]
[worker] W0630 11:16:46.194 3446 warnings.py:109] /opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/app/utils.py:204: CDeprecationWarning:
[worker] The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
[worker] version 6.0.0. Use the result_backend instead
[worker]
[worker] deprecated.warn(description=f'The {setting!r} setting',
[worker]
[worker] W0630 11:16:46.194 3446 worker.py:156] Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.
I use Ubuntu 18.
Install the package from your mirror.
And do the steps of configuration from your doku
I use the Internal Webserver from your gitlab-tools
After Login in Gitlab i get this
500 Internal Server Error
Uh-oh you found a bug! An email has been sent to the ADMINs list.
URL: http://gitlab-tools.domain.tld:8081/sign/in/do?code=29f6d9adbdcb.....
The config /etc/gitlab-tools/config.yml
GITLAB_APP_ID: 7b4f7f......
GITLAB_APP_SECRET: 21b0f3fe4......
GITLAB_URL: https://gitlab.fti-group.com
HOST: 0.0.0.0
PORT: '8081'
SECRET_KEY: 9b51054c2d.......
SERVER_NAME: gitlab-tools.domain.tld:8081
SQLALCHEMY_DATABASE_URI: sqlite:////home/gitlab-tools/gitlab-tools.db
USER: gitlab-tools
What is wrong ?
gitlab-ce:amd64/bionic 11.9.8-ce.0
> sudo apt depends python3-gitlab-tools
python3-gitlab-tools
Depends: python3-celery
Depends: python3-cron-descriptor
Depends: python3-crypto (>= 2.6.1)
[...]
python3-crypto
was removed from Debian in the upcoming Bullseye release: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=972184
python3-cryptography
seems to be a successor, but for whatever reason even after installing python3-pycryptodome
import Crypto
still doesn't work.
Do you have instructions on how to migrate the SQLight Database to MySQL or Postgress?
I have now the problem with ~ 60 mirrors that I keep getting
“sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked”
only a restart of all 3 service help to resolve this for a short time
When we cant set webhook to project... we want to mirror.
Current workaround for this issue:
Traceback (most recent call last):
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 382, in trace_task
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: R = retval = fun(*args, **kwargs)
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: File "/usr/lib/python3/dist-packages/flask_celery/__init__.py", line 89, in __call__
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: return task_base.__call__(self, *_args, **_kwargs)
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 641, in __protected_call__
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: return self.run(*args, **kwargs)
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: File "/usr/lib/python3/dist-packages/celery/app/builtins.py", line 25, in backend_cleanup
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: app.backend.cleanup()
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: File "/usr/lib/python3/dist-packages/celery/backends/database/__init__.py", line 178, in cleanup
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: Task.date_done < (now - expires)).delete()
Nov 03 16:45:00 gitlab-tools gitlab-tools[343]: TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
I run into the following failure with the "cenk1cenk2/gitlab-tools" docker image. Please let me know what you need to debug this issue further.
[START] Starting the celerybeat...
[bgtask] celery beat v5.0.2 (singularity) is starting.
[bgtask] __ - ... __ - _
[bgtask] LocalTime -> 2023-01-02 19:15:58
[bgtask] Configuration ->
[bgtask] . broker -> amqp://guest:**@127.0.0.1:5672//
[bgtask] . loader -> celery.loaders.app.AppLoader
[bgtask] . scheduler -> gitlab_tools.celery_beat.schedulers.DatabaseScheduler
[bgtask]
[bgtask] . logfile -> [stderr]@%WARNING
[bgtask] . maxinterval -> 5.00 seconds (5s)
[bgtask] F0102 19:15:59.032 195392 beat.py:107] beat raised exception <class 'AttributeError'>: AttributeError("'NoneType' object has no attribute 'now'")
[bgtask] Traceback (most recent call last):
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/apps/beat.py", line 105, in start_scheduler
[bgtask] service.start()
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 618, in start
[bgtask] humanize_seconds(self.scheduler.max_interval))
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/kombu/utils/objects.py", line 30, in get
[bgtask] return super().get(instance, owner)
[bgtask] File "/usr/lib/python3.9/functools.py", line 969, in get
[bgtask] val = self.func(instance)
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 662, in scheduler
[bgtask] return self.get_scheduler()
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 653, in get_scheduler
[bgtask] return symbol_by_name(self.scheduler_cls, aliases=aliases)(
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 144, in init
[bgtask] super().init(*args, **kwargs)
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 253, in init
[bgtask] self.setup_schedule()
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 150, in setup_schedule
[bgtask] self.install_default_entries(self.schedule)
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 225, in schedule
[bgtask] self._schedule = self.all_as_schedule()
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 158, in all_as_schedule
[bgtask] s[model.name] = self.Entry(model)
[bgtask] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 34, in init
[bgtask] super().init()
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 124, in init
[bgtask] self.last_run_at = last_run_at or self.default_now()
[bgtask] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 128, in default_now
[bgtask] return self.schedule.now() if self.schedule else self.app.now()
[bgtask] AttributeError: 'NoneType' object has no attribute 'now'
[bgtask] W0102 19:15:59.034 195392 log.py:227] Traceback (most recent call last):
[bgtask] W0102 19:15:59.034 195392 log.py:227] File "/opt/gitlab-tools/manage.py", line 7, in
[bgtask] W0102 19:15:59.034 195392 log.py:227] main()
[bgtask] W0102 19:15:59.034 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 572, in main
[bgtask] W0102 19:15:59.034 195392 log.py:227] getattr(command, 'chosen')() # Execute the function specified by the user.
[bgtask] W0102 19:15:59.034 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 534, in celerybeat
[bgtask] W0102 19:15:59.035 195392 log.py:227] return celery.Beat(
[bgtask] W0102 19:15:59.035 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/apps/beat.py", line 77, in run
[bgtask] W0102 19:15:59.035 195392 log.py:227] self.start_scheduler()
[bgtask] W0102 19:15:59.035 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/apps/beat.py", line 105, in start_scheduler
[bgtask] W0102 19:15:59.035 195392 log.py:227] service.start()
[bgtask] W0102 19:15:59.035 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 618, in start
[bgtask] W0102 19:15:59.036 195392 log.py:227] humanize_seconds(self.scheduler.max_interval))
[bgtask] W0102 19:15:59.036 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/kombu/utils/objects.py", line 30, in get
[bgtask] W0102 19:15:59.036 195392 log.py:227] return super().get(instance, owner)
[bgtask] W0102 19:15:59.036 195392 log.py:227] File "/usr/lib/python3.9/functools.py", line 969, in get
[bgtask] W0102 19:15:59.036 195392 log.py:227] val = self.func(instance)
[bgtask] W0102 19:15:59.036 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 662, in scheduler
[bgtask] W0102 19:15:59.037 195392 log.py:227] return self.get_scheduler()
[bgtask] W0102 19:15:59.037 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 653, in get_scheduler
[bgtask] W0102 19:15:59.037 195392 log.py:227] return symbol_by_name(self.scheduler_cls, aliases=aliases)(
[bgtask] W0102 19:15:59.037 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 144, in init
[bgtask] W0102 19:15:59.037 195392 log.py:227] super().init(*args, **kwargs)
[bgtask] W0102 19:15:59.037 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 253, in init
[bgtask] W0102 19:15:59.037 195392 log.py:227] self.setup_schedule()
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 150, in setup_schedule
[bgtask] W0102 19:15:59.038 195392 log.py:227] self.install_default_entries(self.schedule)
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 225, in schedule
[bgtask] W0102 19:15:59.038 195392 log.py:227] self._schedule = self.all_as_schedule()
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 158, in all_as_schedule
[bgtask] W0102 19:15:59.038 195392 log.py:227] s[model.name] = self.Entry(model)
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/gitlab_tools/celery_beat/schedulers.py", line 34, in init
[bgtask] W0102 19:15:59.038 195392 log.py:227] super().init()
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 124, in init
[bgtask] W0102 19:15:59.038 195392 log.py:227] self.last_run_at = last_run_at or self.default_now()
[bgtask] W0102 19:15:59.038 195392 log.py:227] File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/beat.py", line 128, in default_now
[bgtask] W0102 19:15:59.038 195392 log.py:227] return self.schedule.now() if self.schedule else self.app.now()
[bgtask] W0102 19:15:59.038 195392 log.py:227] AttributeError
[bgtask] W0102 19:15:59.038 195392 log.py:227] :
[bgtask] W0102 19:15:59.038 195392 log.py:227] 'NoneType' object has no attribute 'now'
I started to use this tool for using the gitlab pull mirrors.
I installed the tool using the suggested apt-get installation as given in the readme. GitLab and the tools are installed in the same linux (ubuntu 16.4) machine.
The logging in to GitLab Tools page is working fine and login gitlab authorization is working as expected.
Here is my fingerprints screen
I have configured an SSH key for my GitLab account and it is working.
Issue :
While trying to create a pull mirror following are my settings.
And upon saving following error occures.
Attched the server log
Some parts are missing user specific checks like Forms...
Hello
Using RHEL7
Install manualy, take /static/node_modules from deb package - missing here
Take error on adding fingerprint or mirror:
[pid: 29400|app: 0|req: 8/8] 192.168.38.138 () {50 vars in 1430 bytes} [Tue Jul 23 07:59:43 2019] POST /fingerprint/fingerprint-check => generated 918 bytes in 253 msecs (HTTP/1.1 200) 2 headers in 72 bytes (2 switches on core 0)
[2019-07-23 07:59:44,611] ERROR in app: Exception on /fingerprint/fingerprint-add [POST]
Traceback (most recent call last):
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/gitlab_tools/views/fingerprint/index.py", line 207, in add_hostname_fingerprint
public_key = import_key(get_user_public_key_path(current_user, flask.current_app.config['USER']))
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/gitlab_tools/tools/crypto.py", line 43, in import_key
return RSA.importKey(f.read())
File "/home/gitlab-tools/venv-gitlab-tools/env/lib/python3.6/site-packages/Crypto/PublicKey/RSA.py", line 678, in importKey
if bord(externKey[0])==0x30:
IndexError: index out of range
[pid: 29400|app: 0|req: 9/9] 192.168.38.138 () {50 vars in 1428 bytes} [Tue Jul 23 07:59:44 2019] POST /fingerprint/fingerprint-add => generated 4219 bytes in 68 msecs (HTTP/1.1 500) 2 headers in 100 bytes (1 switches on core 0)
Can you help?
Thanks for this nice project. I found it via your issue on the gitlab-mirrors
project. I like your improvements, but I have to say that the additional complexity is a draw back for deployment.
It's convenient that you supply pre-built packages, but I need to be able to run gitlab-tools
from the source provided in the repo. Your build instructions are useful, but they also expect me to install your prebuilt dependencies. I have been able to work around most of this on my own, but I'm still running into issues. Would you be willing to provide some additional support/docs on setting this up from source using only either system packages (any distro) or upstream source?
I run development env using docker container.
base image is: python:3.5-stretch
gitlab_tools/tools/crypto.py
I add my_transport.banner_timeout = 300 to fix request-new-key error
def get_remote_server_key(ip: str, port: int=22) -> paramiko.pkey.PKey:
"""
Returns PKey for given server
:param ip: IP or Hostname
:param port: Port
:return: Returns PKey
"""
print(ip)
print(port)
my_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
my_socket.settimeout(10)
my_socket.connect((ip, port))
my_transport = paramiko.Transport(my_socket)
print(my_transport.banner_timeout)
my_transport.banner_timeout = 300
print(my_transport.banner_timeout)
my_transport.start_client()
ssh_key = my_transport.get_remote_server_key()
my_transport.close()
my_socket.close()
return ssh_key
console output:
I0325 08:52:33.644 2606 gitlab_tools.py:136] Server is running at http://0.0.0.0:5000/
I0325 08:52:33.645 2606 gitlab_tools.py:137] Flask version: 1.0.2
I0325 08:52:33.645 2606 gitlab_tools.py:138] DEBUG: True
I0325 08:52:33.645 2606 gitlab_tools.py:139] STATIC_FOLDER: /var/app/gitlab-tools/gitlab_tools/static
W0325 08:52:33.692 2606 _internal.py:97] * Debugger is active!
I0325 08:52:33.693 2606 _internal.py:97] * Debugger PIN: 266-783-398
D0325 08:52:38.310 2606 transport.py:1746] starting thread (client mode): 0x44dacd68
D0325 08:52:38.311 2606 transport.py:1746] Local version/idstring: SSH-2.0-paramiko_2.4.2
D0325 08:52:38.315 2606 transport.py:1746] Banner: HTTP/1.1 400 Bad Request
D0325 08:52:38.315 2606 transport.py:1746] Banner: Server: nginx
D0325 08:52:38.316 2606 transport.py:1746] Banner: Date: Mon, 25 Mar 2019 08:52:38 GMT
D0325 08:52:38.316 2606 transport.py:1746] Banner: Content-Type: text/html
D0325 08:52:38.316 2606 transport.py:1746] Banner: Content-Length: 166
D0325 08:52:38.317 2606 transport.py:1746] Banner: Connection: close
D0325 08:52:38.317 2606 transport.py:1746] Banner:
D0325 08:52:38.317 2606 transport.py:1746] Banner: <html>
D0325 08:52:38.318 2606 transport.py:1746] Banner: <head><title>400 Bad Request</title></head>
D0325 08:52:38.318 2606 transport.py:1746] Banner: <body bgcolor="white">
D0325 08:52:38.318 2606 transport.py:1746] Banner: <center><h1>400 Bad Request</h1></center>
D0325 08:52:38.319 2606 transport.py:1746] Banner: <hr><center>nginx</center>
D0325 08:52:38.319 2606 transport.py:1746] Banner: </body>
D0325 08:52:38.320 2606 transport.py:1746] Banner: </html>
E0325 08:52:38.321 2606 transport.py:1746] Exception: Error reading SSH protocol banner
E0325 08:52:38.322 2606 transport.py:1744] Traceback (most recent call last):
E0325 08:52:38.322 2606 transport.py:1744] File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 2138, in _check_banner
E0325 08:52:38.323 2606 transport.py:1744] buf = self.packetizer.readline(timeout)
E0325 08:52:38.323 2606 transport.py:1744] File "/usr/local/lib/python3.5/site-packages/paramiko/packet.py", line 367, in readline
E0325 08:52:38.323 2606 transport.py:1744] buf += self._read_timeout(timeout)
E0325 08:52:38.324 2606 transport.py:1744] File "/usr/local/lib/python3.5/site-packages/paramiko/packet.py", line 563, in _read_timeout
E0325 08:52:38.324 2606 transport.py:1744] raise EOFError()
E0325 08:52:38.324 2606 transport.py:1744] EOFError
E0325 08:52:38.325 2606 transport.py:1744]
E0325 08:52:38.325 2606 transport.py:1744] During handling of the above exception, another exception occurred:
E0325 08:52:38.325 2606 transport.py:1744]
E0325 08:52:38.326 2606 transport.py:1744] Traceback (most recent call last):
E0325 08:52:38.326 2606 transport.py:1744] File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 1966, in run
E0325 08:52:38.326 2606 transport.py:1744] self._check_banner()
E0325 08:52:38.326 2606 transport.py:1744] File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 2143, in _check_banner
E0325 08:52:38.326 2606 transport.py:1744] "Error reading SSH protocol banner" + str(e)
E0325 08:52:38.327 2606 transport.py:1744] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
E0325 08:52:38.327 2606 transport.py:1744]
I0325 08:52:38.341 2606 _internal.py:97] 172.21.0.1 - - [25/Mar/2019 08:52:38] "POST /fingerprint/fingerprint-check HTTP/1.1" 500 -
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 2138, in _check_banner
buf = self.packetizer.readline(timeout)
File "/usr/local/lib/python3.5/site-packages/paramiko/packet.py", line 367, in readline
buf += self._read_timeout(timeout)
File "/usr/local/lib/python3.5/site-packages/paramiko/packet.py", line 563, in _read_timeout
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/var/app/gitlab-tools/gitlab_tools/views/fingerprint/index.py", line 176, in check_hostname_fingerprint
return check_fingerprint_hostname(hostname_string)
File "/var/app/gitlab-tools/gitlab_tools/views/fingerprint/index.py", line 40, in check_fingerprint_hostname
found, remote_server_key = check_hostname(hostname, known_hosts_path)
File "/var/app/gitlab-tools/gitlab_tools/tools/fingerprint.py", line 22, in check_hostname
remote_server_key = get_remote_server_key(hostname)
File "/var/app/gitlab-tools/gitlab_tools/tools/crypto.py", line 75, in get_remote_server_key
my_transport.start_client()
File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 587, in start_client
raise e
File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 1966, in run
self._check_banner()
File "/usr/local/lib/python3.5/site-packages/paramiko/transport.py", line 2143, in _check_banner
"Error reading SSH protocol banner" + str(e)
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
Currently when checking fingerprint, we use GITLAB_SSH
config option with hostname:port
of gitlab SSH server, but gitlab API actually returns a ssh URI for created/existing project. Issue is that project creation is handled in async task (celery) and no user interaction is possible, that means that mirror has to already exist in database when task is triggered...
Hey again @Salamek ,
I just wonder now with the latest additions I have encountered two issues:
The error message is: Error lines received while fetching: error: failed to push some refs to 'gitlab.kilic.dev_1:libraries/cz-cc.git'
My debugging output:
root@bb31e0793154:/home/service/repositories/1/pull/2/7# git pull --mirror
fatal: this operation must be run in a work tree
root@bb31e0793154:/home/service/repositories/1/pull/2/7# cd ..
root@bb31e0793154:/home/service/repositories/1/pull/2# ls
2 7
root@bb31e0793154:/home/service/repositories/1/pull/2# git ^C
root@bb31e0793154:/home/service/repositories/1/pull/2# cd 7
root@bb31e0793154:/home/service/repositories/1/pull/2/7# git remote show origin
ssh: Could not resolve hostname github.com_1: Name or service not known
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
root@bb31e0793154:/home/service/repositories/1/pull/2/7# git remote show gitlab
ssh: Could not resolve hostname gitlab.kilic.dev_1: Name or service not known
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
root@bb31e0793154:/home/service/repositories/1/pull/2/7#
What can I do to help with these issues?
These do not show inside the frontend task logs as errors but on the stdoutput of the application.
It would be nice if an example of running this service was available for running via docker-compose
. Mainly because this seems to have a lot of moving parts and if one wanted to get set up quickly, docker-compose would be easiest from a systems perspective.
Hello.
gitlab and gitlab-tools [1.2.9] will be deployed in my docker.
After adding to "Pull Mirrors" repository.
In "FingerPrints" I get two entries with the type of key SSH-ED25519:
After I click "Trigger Sync" in "Pull Mirrors" in the log I get:
Traceback (most recent call last):
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/app/trace.py", line 409, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_celery/init.py", line 96, in call
return task_base.call(self, *_args, **_kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/celery/app/trace.py", line 701, in protected_call
return self.run(*args, **kwargs)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/flask_celery/init.py", line 150, in wrapped
ret_value = func(*args, **kwargs)
File "/opt/gitlab-tools/gitlab_tools/tasks/gitlab_tools.py", line 371, in sync_pull_mirror
Git.sync_mirror(namespace_path, str(mirror.id), git_remote_source, git_remote_target)
File "/opt/gitlab-tools/gitlab_tools/tools/Git.py", line 39, in sync_mirror
repo.remotes.gitlab.push(
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/git/remote.py", line 1045, in push
return self._get_push_info(proc, progress, kill_after_timeout=kill_after_timeout)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/git/remote.py", line 893, in _get_push_info
proc.wait(stderr=stderr_text)
File "/opt/gitlab-tools/venv/lib/python3.9/site-packages/git/cmd.py", line 559, in wait
raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git push --porcelain --mirror --force --prune gitlab
stderr: 'fatal: Could not read from remote repository.'
I watched the file "repositories/1/pull/1/1/config"
[remote "gitlab"]
url = ssh://[email protected]_1:10022/thirdparty/date.git
If I make from the container:
ssh [email protected]_1 -p 10022
The authenticity of host '[gitlab.mydomain.com]:10022 ([192.168.xxx.xxx]:10022)' can't be established.
ECDSA key fingerprint is SHA256:oc1Buil1L3tCDGRTh1QvQdaK6R9nfGE2JvjaD+W3hZI.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
ECDSA
I introduced "yes"
A record was made in .ssh/known_host_1 with key type ecdsa-sha2-nistp256
After this, the updates were earned from gitlab-tools.
But I don't like to do this for every user.
I made an appointment with a file /etc/ssh/ssh_config
Host *
HostKeyAlgorithms ^ssh-ed25519
And repeated
ssh [email protected]_1 -p 10022
The authenticity of host '[gitlab.mydomain.com]:10022 ([192.168.xxx.xxx]:10022)' can't be established.
ED25519 key fingerprint is SHA256:oc1Buil1L3tCDGRTh1QvQdaK6R9nfGE2JvjaD+W3hZI.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
He asked for "fingerprint"
How to fix it?
In order not to make ssh [email protected]_X -p 10022
for each user.
Thank you.
Hello,
We can't use pull mirrors with custom ssh server port, could we help with that ? Any advice ?
Thank you.
I installed a new Gitlab with Ubuntu 22.04 Jammy, and when installing Gitlab tools I noticed that the python3-flask package was not available in the required version.
And in the end there are too many Python packages that are not compatible and require a downgrade.
so it must be an Ubuntu Focal 20.04 after all.
Here is my attempt to resolve the dependencies.
Try to install Gitlab-Tool
apt-get install gitlab-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
gitlab-tools : Depends: python3-flask (< 1.2) but 2.0.1-2ubuntu1.1 is to be installed
E: Unable to correct problems, you have held broken packages.
Check Version of python3-flask
apt-cache policy python3-flask
python3-flask:
Installed: (none)
Candidate: 2.0.1-2ubuntu1.1
Version table:
2.0.1-2ubuntu1.1 500
500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
500 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages
2.0.1-2ubuntu1 500
500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
Check Version of gitlab-tools
apt-cache policy gitlab-tools
gitlab-tools:
Installed: (none)
Candidate: 1.4.7-1
Version table:
1.4.7-1 500
500 https://repository.salamek.cz/deb/pub all/main amd64 Packages
Download vom PPA
https://answers.launchpad.net/ubuntu/jammy/amd64/python3-flask/1.1.2-2
Install python3-flask via dpkg to get dependency
wget http://launchpadlibrarian.net/504048010/python3-flask_1.1.2-2_all.deb
dpkg -i python3-flask_1.1.2-2_all.deb
Selecting previously unselected package python3-flask.
(Reading database ... 110161 files and directories currently installed.)
Preparing to unpack python3-flask_1.1.2-2_all.deb ...
Unpacking python3-flask (1.1.2-2) ...
dpkg: dependency problems prevent configuration of python3-flask:
python3-flask depends on python3-click (>= 5.1); however:
Package python3-click is not installed.
python3-flask depends on python3-itsdangerous (>= 0.24); however:
Package python3-itsdangerous is not installed.
python3-flask depends on python3-jinja2 (>= 2.10); however:
Package python3-jinja2 is not installed.
python3-flask depends on python3-werkzeug (>= 0.15); however:
Package python3-werkzeug is not installed.
dpkg: error processing package python3-flask (--install):
dependency problems - leaving unconfigured
Errors were encountered while processing:
python3-flask
Install dependency and python3-flask
apt-get install -f
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
javascript-common libjs-jquery python-babel-localedata python3-babel python3-cffi-backend python3-click python3-colorama python3-cryptography
python3-itsdangerous python3-jinja2 python3-markupsafe python3-openssl python3-pyinotify python3-simplejson python3-six python3-tz python3-werkzeug
Suggested packages:
apache2 | lighttpd | httpd python-cryptography-doc python3-cryptography-vectors python-jinja2-doc python-openssl-doc python3-openssl-dbg python-pyinotify-doc
ipython3 python-werkzeug-doc python3-lxml python3-watchdog
The following NEW packages will be installed:
javascript-common libjs-jquery python-babel-localedata python3-babel python3-cffi-backend python3-click python3-colorama python3-cryptography
python3-itsdangerous python3-jinja2 python3-markupsafe python3-openssl python3-pyinotify python3-simplejson python3-six python3-tz python3-werkzeug
0 upgraded, 17 newly installed, 0 to remove and 1 not upgraded.
1 not fully installed or removed.
Need to get 0 B/6294 kB of archives.
After this operation, 32.9 MB of additional disk space will be used.
Set on Hold
apt-mark hold python3-flask
python3-flask set on hold.
Install Gitlab-tools
apt-get install gitlab-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
alembic erlang-asn1 erlang-base erlang-crypto erlang-eldap erlang-ftp erlang-inets erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key
erlang-runtime-tools erlang-snmp erlang-ssl erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl git git-man libatomic1 libcurl3-gnutls liberror-perl
libjemalloc2 libjs-sphinxdoc libjs-underscore liblua5.1-0 liblzf1 libpq5 libsctp1 libsodium23 lua-bitop lua-cjson patch python3-alembic python3-amqp
python3-anyjson python3-bcrypt python3-billiard python3-blinker python3-celery python3-certifi python3-chardet python3-click-didyoumean python3-click-plugins
python3-click-repl python3-cron-descriptor python3-dateutil python3-docopt python3-ephem python3-flask-babel python3-flask-celery-tools python3-flask-login
python3-flask-migrate python3-flask-sqlalchemy python3-git python3-gitdb python3-gitlab python3-greenlet python3-idna python3-kombu python3-mako
python3-memcache python3-nacl python3-paramiko python3-prompt-toolkit python3-psycopg2 python3-pycryptodome python3-pygments python3-pyparsing python3-raven
python3-redis python3-requests python3-requests-toolbelt python3-smmap python3-sqlalchemy python3-sqlalchemy-ext python3-urllib3 python3-vine python3-wcwidth
python3-wtforms rabbitmq-server redis-server redis-tools socat
Suggested packages:
erlang erlang-manpages erlang-doc git-daemon-run | git-daemon-sysvinit git-doc git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn lksctp-tools
diffutils-doc python-amqp-doc python-billiard-doc python-blinker-doc python-celery-doc python-flask-babel-doc python-flask-login-doc python-git-doc
python-gitlab-doc python-greenlet-dev python-greenlet-doc python-kombu-doc python3-boto python3-django python3-pymongo python3-beaker python-mako-doc memcached
python-nacl-doc python3-gssapi python3-invoke python-psycopg2-doc python-pygments-doc ttf-bitstream-vera python-pyparsing-doc python3-hiredis python3-socks
python-requests-doc python3-nose python-sqlalchemy-doc python3-fdb python3-pymssql python3-mysqldb python3-asyncpg python3-aiosqlite python3-django-localflavor
ruby-redis
The following NEW packages will be installed:
alembic erlang-asn1 erlang-base erlang-crypto erlang-eldap erlang-ftp erlang-inets erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key
erlang-runtime-tools erlang-snmp erlang-ssl erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl git git-man gitlab-tools libatomic1 libcurl3-gnutls
liberror-perl libjemalloc2 libjs-sphinxdoc libjs-underscore liblua5.1-0 liblzf1 libpq5 libsctp1 libsodium23 lua-bitop lua-cjson patch python3-alembic
python3-amqp python3-anyjson python3-bcrypt python3-billiard python3-blinker python3-celery python3-certifi python3-chardet python3-click-didyoumean
python3-click-plugins python3-click-repl python3-cron-descriptor python3-dateutil python3-docopt python3-ephem python3-flask-babel python3-flask-celery-tools
python3-flask-login python3-flask-migrate python3-flask-sqlalchemy python3-git python3-gitdb python3-gitlab python3-greenlet python3-idna python3-kombu
python3-mako python3-memcache python3-nacl python3-paramiko python3-prompt-toolkit python3-psycopg2 python3-pycryptodome python3-pygments python3-pyparsing
python3-raven python3-redis python3-requests python3-requests-toolbelt python3-smmap python3-sqlalchemy python3-sqlalchemy-ext python3-urllib3 python3-vine
python3-wcwidth python3-wtforms rabbitmq-server redis-server redis-tools socat
0 upgraded, 86 newly installed, 0 to remove and 1 not upgraded.
Need to get 50.0 MB/50.0 MB of archives.
After this operation, 127 MB of additional disk space will be used.
On this Point you need to add a ubuntu Focal Mirror and Downgrade some Packages
apt-get install -t focal python3-flask-babel=0.11.2-3 python3-flask-login=0.4.1-3 python3-flask-migrate=2.5.2-1 python3-flask-sqlalchemy=2.1-4 python3-itsdangerous=1.1.0-1 python3-werkzeug=0.16.1+dfsg1-2ubuntu0.1
apt-mark hold python3-flask-babel python3-flask-login python3-flask-migrate python3-flask-sqlalchemy python3-itsdangerous python3-werkzeug
but no luck here either because of Python 3.7 vs 3.10
Hi
when i add a now mirror with a Cron or edit a old mirror and change the Cron then i get this.
E0809 10:33:39.827 483829 app.py:1891] Exception on /pull-mirror/edit/40 [POST]
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3/dist-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3/dist-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/lib/python3/dist-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3/dist-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/lib/python3/dist-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/gitlab_tools/views/pull_mirror/index.py", line 230, in edit_mirror
if flask.request.method == 'POST' and form.validate():
File "/usr/lib/python3/dist-packages/gitlab_tools/forms/pull_mirror.py", line 170, in validate
ExpressionDescriptor(self.periodic_sync.data, throw_exception_on_parse_error=True)
File "/usr/lib/python3/dist-packages/cron_descriptor/ExpressionDescriptor.py", line 69, in __init__
raise WrongArgumentException("Unknown {} configuration argument".format(kwarg))
cron_descriptor.Exception.WrongArgumentException: Unknown throw_exception_on_parse_error configuration argument
It would be great if you replaced your requirements.txt file with a pip freeze
version. This way the exact version of all dependencies used is known. Imagine 1 year from now when all of the dependencies have changed (likely). There’s no guarantee your project would work with those latest versions.
By pinning your dependencies you ensure users install the exact version of the dependencies your application needs rather than relying on a floating version.
Hi,
Thanks for your great tool.
Maybe I just have a bad timing, but https://repository.salamek.cz/
doesn't seem to be reachable for me.
➜ ~ curl -v https://repository.salamek.cz/
* Trying 78.80.161.207:443...
* connect to 78.80.161.207 port 443 failed: No route to host
* Failed to connect to repository.salamek.cz port 443 after 895 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to repository.salamek.cz port 443 after 895 ms: Couldn't connect to server
Any file from directory is not found (Error 404).
Try to add to NGINX
location /static {
root /usr/lib/python3/dist-packages/gitlab_tools/static;
gzip_static on;
expires max;
add_header Cache-Control public;
}
But still not found
Hi,
I just tried setting up gitlab-tools and it seems like everything worked correctly.
But when I now try to add a Pull Mirror I only get an empty page with 500 after clicking on save.
Is there any way to debug what error is occurring?
Thanks for your work
related to #31
The way celery is started, doesn't seem to be supported any more in celery 5 (shipped with Debian Bullseye)
gitlab-tools/gitlab_tools/bin/gitlab_tools.py
Lines 513 to 521 in 2b66194
If the mirror and the mirrored both repos existing in the gitlab there is no way to setup mirroring between them.
Use case:
For syncing fork repositories and their upstream repo, the tool is not supporting the sync.
Expected behavior:
The tool should be able to setup sync between the two existing repos (fork and original)
And the pull sync should have the merging capabilities for the fork repo
Since I use CentOS I was unable to install it using the repo. I'm trying to use from source following your wiki
But when I reach the python manage.py
to test the installation I get an error the error: ImportError: cannot import name 'ImmutableDict'
Full Traceback
Traceback (most recent call last):
File "manage.py", line 4, in <module>
from gitlab_tools.bin.gitlab_tools import main
File "/root/gitlab-tools/gitlab_tools/bin/gitlab_tools.py", line 75, in <module>
from gitlab_tools.application import create_app, get_config
File "/root/gitlab-tools/gitlab_tools/application.py", line 9, in <module>
from flask_babel import gettext
File "/root/gitlab-tools/venv/lib/python3.6/site-packages/flask_babel/__init__.py", line 19, in <module>
from werkzeug import ImmutableDict
ImportError: cannot import name 'ImmutableDict'
This took me a while to debug and I am still not completely sure what the issue is.
On every mirror I got the following error:
Jul 22 18:59:58 vServer gitlab-tools[8600]: git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
Jul 22 18:59:58 vServer gitlab-tools[8600]: cmdline: git push --mirror --porcelain gitlab
Jul 22 18:59:58 vServer gitlab-tools[8600]: stderr: 'fatal: Could not read from remote repository.
Jul 22 18:59:58 vServer gitlab-tools[8600]: Please make sure you have the correct access rights
Jul 22 18:59:58 vServer gitlab-tools[8600]: and the repository exists.'
Then I had the idea to login as and then cd repositories/1/pull/2/9/
and run git push --mirror gitlab
manually.
After accepting the Are you sure you want to continue connecting (yes/no)? yes
the push worked.
And afterwards all pushes from gitlab-tools also worked.
I guess the known_hosts
entry was missing.
I have github-tools
installed and running, but when I sign in I get the following error in GitLab:
The redirect URI included is not valid
My callback URL in the GitLab application is set to https://gitlab-tools.domain/sign/in/do
, so I'm not sure what else I can check or do to troubleshoot this.
Hi.
Great work here, thank you for this tool!
I am having an issue trying to pull mirror a github repo.
I have successfully created a pull mirror to a gitlab repo, but github doesn't seem to work.
gitlab-tools doesn't want to accept the github.com fingerprint.
Relevant log section:
I0824 19:02:08.646 26155 log.py:282] Connected (version 2.0, client babeld-216c4091)
D0824 19:02:08.647 26155 log.py:282] kex algos:['curve25519-sha256', '[email protected]', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256'] server key:['ssh-dss', 'rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa'] client encrypt:['[email protected]', '[email protected]', '[email protected]', 'aes256-ctr', 'aes192-ctr', 'aes128-ctr', 'aes256-cbc', 'aes192-cbc', 'aes128-cbc'] server encrypt:['[email protected]', '[email protected]', '[email protected]', 'aes256-ctr', 'aes192-ctr', 'aes128-ctr', 'aes256-cbc', 'aes192-cbc', 'aes128-cbc'] client mac:['[email protected]', '[email protected]', '[email protected]', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['[email protected]', '[email protected]', '[email protected]', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] client compress:['none', 'zlib', '[email protected]'] server compress:['none', 'zlib', '[email protected]'] client lang:[''] server lang:[''] kex follows?False
D0824 19:02:08.647 26155 log.py:282] Kex agreed: diffie-hellman-group-exchange-sha256
D0824 19:02:08.647 26155 log.py:282] Cipher agreed: aes128-ctr
D0824 19:02:08.647 26155 log.py:282] MAC agreed: hmac-sha2-256
D0824 19:02:08.647 26155 log.py:282] Compression agreed: none
D0824 19:02:08.879 26155 log.py:282] Got server p (2048 bits)
D0824 19:02:09.029 26155 log.py:282] kex engine KexGexSHA256 specified hash_algo <built-in function openssl_sha256>
D0824 19:02:09.030 26155 log.py:282] Switch to new keys ...
D0824 19:02:09.031 26155 log.py:282] EOF in transport thread
I0824 19:02:09.040 26155 _internal.py:87] 10.7.151.242 - - [24/Aug/2019 19:02:09] "POST /fingerprint/fingerprint-add HTTP/1.1" 500 -
Is there a way to manually add a fingerprint?
Am I just being stupid? :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.