tracboat / tracboat Goto Github PK

View Code? Open in Web Editor NEW

35.0 5.0 27.0 345 KB

Trac to GitLab migration toolbox

License: GNU General Public License v3.0

Python 99.84% Shell 0.16%

trac gitlab migration exporter

tracboat's Introduction

TracBoat

A life boat to escape the Trac ocean liner.

TracBoat is a toolbox for exporting whole Trac instances, saving them and migrating them to other platforms.

Features

Export Trac projects along with issues, issue changelogs (with attachments) and wiki pages (with attachments)
Create GitLab users, projects and namespaces involved in the migration process
Export Trac projects to local files (in json, toml, Python literal or Python pickle formats)
Migrate Trac projects directly from remote instances as well as from previously created export files
Migrate Trac projects to a live GitLab instance
Migrate Trac projects to a mocked GitLab on the file system to check for correctness

Installation

If you want to install from source, doing it in a virtualenv is highly recommended. After having this repo cloned, just type:

$ cd tracboat
$ virtualenv -p python2.7 VENV
$ source VENV/bin/activate
$ pip install -r requirements/dist.txt
$ pip install -e .

Alternatively with pipenv

$ cd tracboat
$ pip install pipenv
$ pipenv install -e .
$ pipenv run tracboat

Dependencies

Tested on Python 2.7 and Python >= 3.4
peewee
psycopg2
six
click
toml
pymongo

Getting started

Every command line option can be specified as an environment variable, so the following command:

$ tracboat users --trac-uri=https://mytrac/xmlrpc --no-ssl-verify

...is exactly the same as the following:

$ export TRACBOAT_TRAC_URI=https://mytrac/xmlrpc
$ export TRACBOAT_SSL_VERIFY=0
$ tracboat users

Another way to conveniently configure tracboat is with a configuration file in TOML format. Providing a file

$ cat mytrac.toml
[tracboat]
trac_uri = "https://mytrac/xmlrpc"
ssl_verify = false
$ tracboat --config-file=mytrac.toml users

Please note that when a value is specified more that once, the priority order considered is the following:

command line option;
environment variable;
configuration file;
default value.

If you are very curious about how to play with command line options, have a look to the click documentation.

Collecting Trac users

$ tracboat users --trac-uri=http://localhost/xmlrpc

Export a Trac instance

$ tracboat export --trac-uri=http://localhost/xmlrpc --format=json --out-file=myproject.json

Migrate to GitLab

$ cat awesomemigration.toml
[tracboat]
from_export_file = "myexportedtracproject.json"
gitlab_project_name = "migrated/myproject"
# see below to choose the right version
gitlab_version = "10.5"
gitlab_db_password = "Բարեւ աշխարհ"
$ tracboat --config-file=awesomemigration.toml migrate

GitLab version compatibility

Not all versions of GitLab need a specific model. You can refer to the table below to choose the correct model for your GitLab version.

Tracboat model	Compatible GitLab version
11.0	11.0, 11.1, 11.2
10.5	10.5, 10.6, 10.7, 10.8
10.4	10.4
10.3	10.3
10.2	10.2
9.5	9.5
9.4	9.4
9.3	9.3
9.0	9.0
8.17	8.17
8.16	8.16
8.15	8.15
8.13	8.13
8.7	8.7
8.5	8.5
8.4	8.4

If your GitLab version is not in the table, it has not been tested yet. Try with the first model lower than your version, and report any issue you encounter.

e.g. for GitLab 9.2, try 9.0 model

Migrating users

During a migration we need to map Trac usernames to GitLab user accounts to keep all associations between issues, changelog entries and wiki pages and their authors. By default, all Trac usernames are mapped to a single GitLab user, the so called fallback user. This way you'll end up with a migrated project where all activity looks like it come from a single user. Not so fancy, but definitely handy if you just care about content. You can specify a custom fallback username with the proper option:

$ cat config.toml
[tracboat]
fallback_user = "[email protected]"

As usual, the same behaviour can be obtained via command line option or environment variable. So doing this:

$ export [email protected]
$ tracboat migrate

...is the same as doing this:

$ tracboat migrate [email protected]

Mapping users

When you want your Trac users mapped to a GitLab user (and to the corresponding account) you need to specify a custom user mapping, or an association between a Trac username and a GitLab account. You can use a key-value section in the configuration file:

$ cat config.toml
[tracboat.usermap]
    tracuser1 = "[email protected]"
    tracuser2 = "[email protected]"
    tracuser3 = "[email protected]"

In this case, every action that in the Trac project belongs to tracuser1, in the migrated GitLab project will end up as being authored by [email protected].

You can add extra mappings using the --umap command line option, so doing like this:

$ tracboat migrate --umap tracuser1 [email protected] --umap tracuser2 [email protected] ...

...obtains exactly the same behaviour as with the configuration file above. Remember that for repeated values, command line takes precedence over configuration file.

Custom user attributes

If a user doesn't exist in GitLab yet, he will be created during the migration process. However, creating a new GitLab account is a fairly complex affair: you can specify social accounts, biography, links and a lot of other stuff. If you don't say anything about how an user should be created, Tracboat uses some defaults. However you can throw a proper section in the configuration file to tweak those user creation attributes:

$ cat config.toml
[tracboat.users.default]
    admin = false
    external = true
    website_url = "http://www.foo.gov"

Those values will be applied to all new accounts created during the migration process. However, you can specify additional user subsections to precisely control which values would be used for a particular account:

$ cat config.toml
[tracboat.users.default]
    admin = false
    external = true
    website_url = "http://www.foo.gov"

[tracboat.users."[email protected]"]
    username = "theboss"
    bio = "Hi. I am the boss here."
    admin = true
    twitter = "@theboss"
    external = false

In this case, all users are going to be created with the attributes contained in the [tracboat.users.default] section except for the boss that asked explicitly for some extra goodies.

Example

This is a fairly complete configuration example with a usermap and custom user attributes. You can find additional examples in the examples/ directory.

# Tracboat will look for values in the [tracboat] section only, so
# you can merge in a single file values for other applications.

[tracboat]

# The Trac instance to be crawled.
# If you have any secrets in the URL (just like in this case,
# our password is in plain text), consider using the corresponding
# environment variable TRACBOAT_TRAC_URI to avoid having secrets in
# the configuration file.
trac_uri = "https://myuser:MYPASSWORD@localhost/login/xmlrpc"

# Disable ssl certificate verification (e.g.: needed with self signed certs).
ssl_verify = false

# The GitLab project name.
# Can be specified as a path, subdirectories will be treated as GitLab
# namespaces.
gitlab_project_name = "migrated/myproject"

# The fallback user, used when a Trac username has no entry in the
# [tracboat.usermap] section.
fallback_user = "[email protected]"

# Users configuration.
# Every section beyond this point can be passed in separate TOML files
# with repeated --umap-file command line options or directly here:
#
# umap_file = ['users1.toml', 'users2.toml']

# The Trac -> GitLab user conversion mapping.
# It is *highly* recommended to use a valid email address for the GitLab part
# since by default each account will be created with a random password
# (you need a valid address for the password reset procedure to work properly).
[tracboat.usermap]
    tracuser1 = "[email protected]"
    tracuser2 = "[email protected]"
    tracuser3 = "[email protected]"
    tracuser4 = "[email protected]"

[tracboat.users]
# GitLab users attributes.
# This section allows to specify custom attributes
# to be used during GitLab user creation. Accepted values are
# listed here:
# https://docs.gitlab.com/ce/api/users.html#user-creation

[tracboat.users.default]
    # This 'default' section specifies attributes applied
    # to all new GitLab users.
    external = true

[tracboat.users."[email protected]"]
    # This section affects a specific user (in this case "[email protected]").
    # These key-value entries will be merged with those in the
    # [tracboat.users.default] section. For repeated values, those specified
    # here will prevail.
    #
    # There are some mandatory values that must be specified
    # for each user, otherwise the following default values
    # will be used:
    #
    # username = ...
    #     Defaults to the user part of the GitLab email address
    #     (e.g. "gitlabuser4" for "[email protected]").
    #
    # encrypted_password = ...
    #     Defaults to a random password (at the first login the user must carry
    #     out a password reset procedure). Anyway, you are *highly* discouraged
    #     to specify secrets here, please stick to the default behaviour.
    username = "theboss"
    bio = "Hi. I am the boss here."
    admin = true
    twitter = "@theboss"
    external = false  # this value overrides tracboat.users.default.external

[tracboat.users."[email protected]"]
    # This section affects the fallback user, used when a Trac
    # username has no entry in the [tracboat.usermap] section.
    username = "migration-bot"
    bio = "Hi. I am the robot that migrated all your stuff."
    admin = true
    external = false

Credits

Tracboat was initially created by Federico Ficarelli and is now maintained by a pack of great contributors (refer to AUTHORS file for details). Initial inspiration and core migration logic comes from the trac-to-gitlab project by Maël Lavault: this project was born from heavy cleanup and refactoring of that original code, so this is why this spinoff inherited its GPLv3 license.

tracboat's People

Contributors

Stargazers

Watchers

tracboat's Issues

support for GitLab nested groups

It seems that GitLab 9.0.0 introduced nested groups. We need to figure out how nested groups are represented in the database schema and add some tests.

fix user creation

Allow the user to specify username+email for each GitLab user to be created. Both fields are required even if username is not strictly required by the GitLab database schema.

release on PyPI

cleanup migration interface

Now, a lot of methods in the tracboat.gitlab.direct interface require the user to pass an already constructed database model object, for example:

def create_issue(self, new_issue):
        M = self.model
        new_issue.save()
        # ...

Refactor all methods to accept entities construction options in kwargs, model entities should be handled only inside tracboat.gitlab.direct just like we do here:

def create_user(self, email, **kwargs):
        M = self.model
        try:
            user = M.Users.get(M.Users.email == email)
        except M.Users.DoesNotExist:
            parms = dict(USER_DEFAULTS)
            parms.update(kwargs)
            parms['email'] = email
            parms.setdefault('public_email', email)
            user = M.Users.create(**parms)
            user.save()
            LOG.debug("user %r created", email)
        return user.id

user creation killed gitlab admin

https://gitlab.example.net/admin

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/admin" for 10.1.2.66 at 2017-04-21 23:56:29 +0300
Processing by Admin::DashboardController#index as HTML
Completed 500 Internal Server Error in 127ms (ActiveRecord: 11.6ms)

ActionView::Template::Error (undefined method `strftime' for nil:NilClass):
    155:             = link_to [:admin, user], class: 'str-truncated-60' do
    156:               = user.name
    157:             %span.light.pull-right
    158:               #{time_ago_with_tooltip(user.created_at)}
    159:
    160:       .col-md-4   
    161:         %h4 Latest groups
  app/helpers/application_helper.rb:170:in `time_ago_with_tooltip'
  app/views/admin/dashboard/index.html.haml:158:in `block in _app_views_admin_dashboard_index_html_haml__3452472750118434001_69913778079960' 
  app/views/admin/dashboard/index.html.haml:153:in `_app_views_admin_dashboard_index_html_haml__3452472750118434001_69913778079960' 
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/etag_caching/middleware.rb:14:in `call'
  lib/gitlab/request_context.rb:18:in `call'

https://gitlab.example.net/admin/users?filter=wop

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/admin/users?filter=wop" for 10.1.2.66 at 2017-04-21 23:55:00 +0300
Processing by Admin::UsersController#index as HTML
  Parameters: {"filter"=>"wop"}
Completed 500 Internal Server Error in 179ms (ActiveRecord: 11.0ms)

ActionView::Template::Error (nil is not a known state value):
    4:   .row-main-content
    5:     .user-name.row-title.str-truncated-100
    6:       = link_to user.name, [:admin, user]
    7:     - if user.blocked?
    8:       %span.label.label-danger blocked
    9:     - if user.admin?
   10:       %span.label.label-success Admin
  app/views/admin/users/_user.html.haml:7:in `_app_views_admin_users__user_html_haml__2653120606476374577_69913714381860'
  app/views/admin/users/index.html.haml:76:in `_app_views_admin_users_index_html_haml___3881772783949202085_69913709853260'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/etag_caching/middleware.rb:14:in `call'
  lib/gitlab/request_context.rb:18:in `call'

this happened after i finally managed to run this tool for the first time into actual database (using gitlab direct):

# time sudo -u gitlab-psql ./VENV/bin/tracboat --config-file=comment.toml migrate

i think the problem was that it created user with invalid email, as my user map was:

[tracboat.usermap]
    glen = "glen"

why the mapping is login -> email, not login -> login? can it be both?

Migrating Trac 1.2 to Gitlab 11.5.0

I am currently trying to migrate Trac 1.2 to GitLab 11.5.0. GitLab is running on Ubuntu 18.04.1 LTS x86_64. Per the README notes, I've tried migrating with gitlab_version set to 11.0. The README also mentioned that any issues encountered with the migration process should be opened here.

My 'migrate.toml' file:

[tracboat]
trac_uri = "https://user:[email protected]/rpc"
ssl_verify = false
gitlab_project_name = "migrated/tracwiki"
fallback_user = "[email protected]"
gitlab_version = "11.0"

I am running this in my terminal:

tracboat --config-file migrate.toml migrate

I am getting the following error:

Traceback (most recent call last):
  File "/home/ben/code/tracboat/VENV/bin/tracboat", line 11, in <module>
    load_entry_point('tracboat', 'console_scripts', 'tracboat')()
  File "/home/ben/code/tracboat/src/tracboat/cli.py", line 428, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/home/ben/code/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/ben/code/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/ben/code/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ben/code/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ben/code/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/ben/code/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/home/ben/code/tracboat/src/tracboat/cli.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/home/ben/code/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ben/code/tracboat/src/tracboat/cli.py", line 418, in migrate
    svn2git_revisions=svn2git_revisions,
  File "/home/ben/code/tracboat/src/tracboat/migrate.py", line 449, in migrate
    gitlab.clear_issues()
  File "/home/ben/code/tracboat/src/tracboat/gitlab/direct.py", line 140, in clear_issues
    for issue in M.Issues.select().where(M.Issues.project == self.project_id):
  File "/home/ben/code/tracboat/src/tracboat/gitlab/__init__.py", line 53, in project_id
    self._project_id = self._get_project_id()
  File "/home/ben/code/tracboat/src/tracboat/gitlab/direct.py", line 109, in _get_project_id
    raise ValueError("Project {!r} not found".format(self.project_qualname))
ValueError: Project u'migrated/tracwiki' not found

Any advice is greatly appreciated. Cheers!

how does mock migration works?

The README says:

Migrate Trac projects to a mocked GitLab on the file system to check for correctness

Yet it doesn't explain anywhere how to use that feature, how it works or what it actually does.

I understand there's a --mock commandline option and it seems it will create stuff in a sqlite database, is that right? What can we expect of that feature?

Thanks!

xml.parsers.expat.ExpatError: not well-formed (invalid token)

I ran the users.sh and I'm only getting to ticket #3735 out of 18,000+ tickets. I assume there is a character in the changelog that the xml parser doesn't like. When I Googled the error, I see that many times it's a '&' that xml parsers dislike, but I have none of those. I do see a ' < ' in one comment (surrounded by a single whitespace on both sides). I scoured documentation here and with trac_to_git and see no mention of xml parsing errors or how to handle them.

........
2019-02-27 17:59:26,121 DEBUG tracboat.trac: ticket_get_changelog of ticket #3735
Traceback (most recent call last):
  File "/home/user1/tracboat/VENV/bin/tracboat", line 11, in <module>
    load_entry_point('tracboat', 'console_scripts', 'tracboat')()
  File "/home/user1/tracboat/src/tracboat/cli.py", line 428, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/user1/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user1/tracboat/src/tracboat/cli.py", line 240, in users
    authors = trac.authors_get(source)
  File "/home/user1/tracboat/src/tracboat/trac.py", line 154, in authors_get
    tickets = ticket_get_all(source, attachments=False) if from_tickets else None
  File "/home/user1/tracboat/src/tracboat/trac.py", line 81, in ticket_get_all
    for ticket_id in source.ticket.query("max=0&order=id")
  File "/home/user1/tracboat/src/tracboat/trac.py", line 81, in <dictcomp>
    for ticket_id in source.ticket.query("max=0&order=id")
  File "/home/user1/tracboat/src/tracboat/trac.py", line 52, in ticket_get_changelog
    for c in source.ticket.changeLog(ticket_id)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1283, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1316, in single_request
    return self.parse_response(response)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1487, in parse_response
    p.feed(data)
  File "/usr/lib/python2.7/xmlrpclib.py", line 558, in feed
    self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 152, column 97

Additional question: does tracboat look only at the current state of changelog entries or does it also look at previous versions of modified changelog comments? If it's the latter, that's a bit more painful for debugging.

use XmlRpc multicalls during export

See #28

migrate from exported file

Hi,

the file is exported from trac instance as python format, while json format has utf8 issue like #6

my migration configuration file

[tracboat]
from_export_file = "export_file.py"
fallback_user = "[email protected]"
gitlab_project_name = "testGroup/testProject"
gitlab_version = "8.9.11"
gitlab_db_user = "sagsdga"
gitlab_db_password = "gsagasgsadgas"
gitlab_db_path = "gitlab.example.com"
gitlab_uploads_path = "/tmp/gitlab/uploads/"

tracboat --config-file=migration.toml migrate

Traceback (most recent call last):
  File "/tmp/tracboat/tracboat/VENV/bin/tracboat", line 9, in <module>
    load_entry_point('tracboat==0.2.0a0', 'console_scripts', 'tracboat')()
  File "/tmp/tracboat/tracboat/src/tracboat/cli.py", line 423, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/tmp/tracboat/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/tracboat/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/tmp/tracboat/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/tracboat/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/tracboat/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/tmp/tracboat/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/tmp/tracboat/tracboat/src/tracboat/cli.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/tmp/tracboat/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/tracboat/tracboat/src/tracboat/cli.py", line 374, in migrate
    project = _loads(content, fmt=fmt)
  File "/tmp/tracboat/tracboat/src/tracboat/cli.py", line 72, in _loads
    return ast.literal_eval(content)
  File "/usr/lib/python2.7/ast.py", line 80, in literal_eval
    return _convert(node_or_string)
  File "/usr/lib/python2.7/ast.py", line 63, in _convert
    in zip(node.keys, node.values))
  File "/usr/lib/python2.7/ast.py", line 62, in <genexpr>
    return dict((_convert(k), _convert(v)) for k, v
  File "/usr/lib/python2.7/ast.py", line 63, in _convert
    in zip(node.keys, node.values))
  File "/usr/lib/python2.7/ast.py", line 62, in <genexpr>
    return dict((_convert(k), _convert(v)) for k, v
  File "/usr/lib/python2.7/ast.py", line 63, in _convert
    in zip(node.keys, node.values))
  File "/usr/lib/python2.7/ast.py", line 62, in <genexpr>
    return dict((_convert(k), _convert(v)) for k, v
  File "/usr/lib/python2.7/ast.py", line 79, in _convert
    raise ValueError('malformed string')
ValueError: malformed string

Any suggestion is appreciated.

fix GitLab database model versioning

As pointed out by @glensc , the model filename for GitLab ver. 9.0.0 should be model90.py. This implies that GitLab doesn't change the database schema between patch releases (I don't think it is somehow guaranteed since the schema itself is not part of their public API).

It could be useful to make the separation between major and minor in model file name explicit, e.g. rename model817.py to model8.17.py or model_8_17.py, something like that, bare 817 is a bit ambiguous imho.

400 bad request

Trying to run
pipenv run tracboat users --trac-uri=http://user.name:[email protected]/projects/projectname/
on windows and I keep getting a 400 bad request.

Trac version: 1.0.1

I tried the same request in postman and it returned the expected html page.
Unless with tracboat I have to login to /projects and then create another request for the individual project?

Here is the full log from windows cmd:

  File "C:\Users\username\.virtualenvs\tracboat-master-ZsaC0Ru0\Scripts\tracboat-script.py", line 11, in <module>
    load_entry_point('tracboat', 'console_scripts', 'tracboat')()
  File "c:\wamp64\www\000_tracboat-master\tracboat-master\src\tracboat\cli.py", line 428, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "c:\users\username\.virtualenvs\tracboat-master-zsac0ru0\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\username\.virtualenvs\tracboat-master-zsac0ru0\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\users\username\.virtualenvs\tracboat-master-zsac0ru0\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\username\.virtualenvs\tracboat-master-zsac0ru0\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\username\.virtualenvs\tracboat-master-zsac0ru0\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\wamp64\www\000_tracboat-master\tracboat-master\src\tracboat\cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "c:\users\username\.virtualenvs\tracboat-master-zsac0ru0\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\wamp64\www\000_tracboat-master\tracboat-master\src\tracboat\cli.py", line 240, in users
    authors = trac.authors_get(source)
  File "c:\wamp64\www\000_tracboat-master\tracboat-master\src\tracboat\trac.py", line 153, in authors_get
    wiki = wiki_get_all_pages(source, contents=False, attachments=False) if from_wiki else None
  File "c:\wamp64\www\000_tracboat-master\tracboat-master\src\tracboat\trac.py", line 116, in wiki_get_all_pages
    for name in source.wiki.getAllPages()
  File "c:\python27\Lib\xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "c:\python27\Lib\xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "c:\python27\Lib\xmlrpclib.py", line 1283, in request
    return self.single_request(host, handler, request_body, verbose)
  File "c:\python27\Lib\xmlrpclib.py", line 1331, in single_request
    response.msg,
xmlrpclib.ProtocolError: <ProtocolError for user.name:[email protected]/projects/projectname: 400 Bad Request>```

Error in gitlab 9.3.5 version data model

Hi,
TracebacDoes the current version support the gitlab version 9.3.5 data model?

It looks like an error in the model:

Traceback (most recent call last):
File "/home/ivan/tracboat/VENV/bin/tracboat", line 11, in
load_entry_point('tracboat', 'console_scripts', 'tracboat')()
File "/home/ivan/tracboat/src/tracboat/cli.py", line 423, in main
cli(obj={}) # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/ivan/tracboat/src/tracboat/cli.py", line 118, in wrapper
return func(*args, **kwargs)
File "/home/ivan/tracboat/src/tracboat/cli.py", line 168, in wrapper
return func(*args, **kwargs)
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/ivan/tracboat/src/tracboat/cli.py", line 413, in migrate
userattrs=userattrs,
File "/home/ivan/tracboat/src/tracboat/migrate.py", line 280, in migrate
gitlab.create_user(**attrs)
File "/home/ivan/tracboat/src/tracboat/gitlab/direct.py", line 182, in create_user
user = M.Users.get(M.Users.email == email)
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 4916, in get
return sq.get()
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3171, in get
return next(clone.execute())
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3223, in execute
self._qr = ResultWrapper(model_class, self._execute(), query_meta)
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 2902, in _execute
return self.database.execute_sql(sql, params, self.require_commit)
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3765, in execute_sql
self.commit()
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3588, in exit
reraise(new_type, new_type(*exc_args), traceback)
File "/home/ivan/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3758, in execute_sql
cursor.execute(sql, params or ())
peewee.ProgrammingError: column t1.authorized_projects_populated does not exist
LINE 1: ...."id", "t1"."admin", "t1"."authentication_token", "t1"."auth..

issues creation missing iid

seems required :iid is not filled when creating issues in direct mode. causing issue listing to crash.

https://gitlab.example.net/.../issues

==> /var/log/gitlab/gitlab-rails/production.log <==
Completed 500 Internal Server Error in 156ms (ActiveRecord: 11.8ms)  

ActionView::Template::Error (No route matches {:action=>"show", :controller=>"projects/issues", :id=>#<Issue id: 7956, title: "
miniprofile: remove \"view profile\" link for DID us...", assignee_id: 1, author_id: 1, project_id: 166, created_at: "2015-03-1
0 08:38:57", updated_at: "2015-03-10 08:38:57", position: 0, branch_name: nil, description: "Screenshot: \n\nIn co...", milestone_id: 2419, state: "opened", iid: nil, updated_by_id: nil, confidential: false, deleted_at: nil
, moved_to_id: nil, due_date: nil, lock_version: nil, title_html: nil, description_html: nil, time_estimate: 0, relative_positi
on: nil>, :namespace_id=>#<Group id: 39, name: "comment", path: "comment", owner_id: nil, created_at: "2017-04-06 11:09:12", up
dated_at: "2017-04-06 11:09:12", type: "Group", description: "", avatar: nil, share_with_group_lock: false, visibility_level: 1
0, request_access_enabled: false, deleted_at: nil, lfs_enabled: nil, description_html: nil, parent_id: nil>, :project_id=>#<Pro
ject id: 166, name: "comment", path: "comment", description: "", created_at: "2017-03-31 08:40:49", updated_at: "2017-04-06 11:
09:40", creator_id: 2, namespace_id: 39, last_activity_at: "2017-04-21 20:45:09", import_url: nil, visibility_level: 10, archiv
ed: false, avatar: nil, import_status: "none", star_count: 0, import_type: nil, import_source: nil, import_error: nil, ci_id: n
il, shared_runners_enabled: true, runners_token: "", build_coverage_regex: nil, build_allow_git_fetch: true
, build_timeout: 3600, pending_delete: false, public_builds: true, last_repository_check_failed: false, last_repository_check_a
t: "2017-04-01 09:20:14", container_registry_enabled: true, only_allow_merge_if_pipeline_succeeds: false, has_external_issue_tr
acker: false, repository_storage: "default", request_access_enabled: false, has_external_wiki: false, lfs_enabled: nil, descrip
tion_html: "", only_allow_merge_if_all_discussions_are_resolved: false>} missing required keys: [:id]):
    1: %li{ id: dom_id(issue), class: issue_css_classes(issue), url: issue_path(issue), data: { labels: issue.label_ids, id: is
sue.id } }
    2:   .issue-box
    3:     - if @bulk_edit
    4:       .issue-check
  app/helpers/gitlab_routing_helper.rb:82:in `issue_path'
  app/views/projects/issues/_issue.html.haml:1:in `_app_views_projects_issues__issue_html_haml___1240390849716462797_6991371870
2360'
  app/views/projects/issues/_issues.html.haml:2:in `_app_views_projects_issues__issues_html_haml___2424223834673794016_69913717
256020'
  app/views/projects/issues/index.html.haml:34:in `_app_views_projects_issues_index_html_haml___430271349031898150_699137755102
80'
  app/controllers/projects/issues_controller.rb:51:in `index'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:i

# gitlab-rails console

irb(main):033:0* Issue.find_by(id: 7956)
=> #<Issue id: 7956, title: "miniprofile: remove \"view profile\" link for DID us...", assignee_id: 1, author_id: 1, project_id: 166, created_at: "2015-03-10 08:38:57", updated_at: "2015-03-10 08:38:57", position: 0, branch_name: nil, description: "Screenshot: \n\nIn co...", milestone_id: 2419, state: "opened", iid: nil, updated_by_id: nil, confidential: false, deleted_at: nil, moved_to_id: nil, due_date: nil, lock_version: nil, title_html: nil, description_html: nil, time_estimate: 0, relative_position: nil>
irb(main):034:0>

json output utf8 error

invoking with json output --format=json i get error:

UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 454: invalid start byte

traceback is not useful, so i'm not posting it here.

as you may already know json is utf8 strict, no single byte encodings allowed. perhaps allow specify encoding or just try to convert from latin1?

wiki migration "file already exists"

It took me a bit, but I did finally figure out that the wiki step is actually commented out in migration.py. Maybe this could be a command line option instead of just commenting it out? Or at least perhaps a comment on README.rst would help.

I did get pretty far in my wiki migration step, but failed because of a duplicate filename:

....
2019-03-08 19:29:33,887 DEBUG tracboat.migrate: migrated attachment TestResults-s7dvr-7.5.0.35184 @ TestResults-s7dvr-7.5.0.35184/EncoderResults.htm
......
Traceback (most recent call last):
  File "/var/opt/gitlab/temp/tracboat/VENV/bin/tracboat", line 11, in <module>
    load_entry_point('tracboat', 'console_scripts', 'tracboat')()
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/cli.py", line 429, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/cli.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/cli.py", line 419, in migrate
    svn2git_revisions=svn2git_revisions,
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/migrate.py", line 454, in migrate
    migrate_wiki(trac['wiki'], gitlab, output_wiki_path)
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/migrate.py", line 406, in migrate_wiki
    gitlab.save_attachment(name, data)
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/gitlab/direct.py", line 369, in save_attachment
    raise Exception("file already exists: %r" % filename)
Exception: file already exists: u'/var/opt/gitlab/gitlab-rails/uploads/exacq/DVRtemp/EncoderResults.htm'

So there was an import of EncoderResults.htm directly to /var/opt/gitlab/gitlab-rails/uploads/exacq/DVRtemp, but that was from the wiki page TestResults-s7dvr-7.5.0.35184. There is another wiki page named TestResults-s7dvr-7.5.0.33815 that also has a EncoderResults.htm.

How should this be handled?

Export fails if attachments can't be found

Export fails if attachments can't be found, I would instead expect a warning?

(VENV) josm@josm:~$ tracboat export --trac-uri=$TRACBOAT_TRAC_URI --format=json --out-file=/home/josm/team/Vincent/josm_trac_export_20190504_1924.json
/home/josm/tracboat/VENV/local/lib/python2.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/home/josm/tracboat/VENV/bin/tracboat", line 11, in <module>
    load_entry_point('tracboat', 'console_scripts', 'tracboat')()
  File "/home/josm/tracboat/src/tracboat/cli.py", line 428, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/home/josm/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/josm/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/josm/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/josm/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/josm/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/josm/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/home/josm/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/josm/tracboat/src/tracboat/cli.py", line 267, in export
    project = trac.project_get(source, collect_authors=True)
  File "/home/josm/tracboat/src/tracboat/trac.py", line 142, in project_get
    'tickets': ticket_get_all(source),
  File "/home/josm/tracboat/src/tracboat/trac.py", line 81, in ticket_get_all
    for ticket_id in source.ticket.query("max=0&order=id")
  File "/home/josm/tracboat/src/tracboat/trac.py", line 81, in <dictcomp>
    for ticket_id in source.ticket.query("max=0&order=id")
  File "/home/josm/tracboat/src/tracboat/trac.py", line 69, in ticket_get_attachments
    for meta in source.ticket.listAttachments(ticket_id)
  File "/home/josm/tracboat/src/tracboat/trac.py", line 69, in <dictcomp>
    for meta in source.ticket.listAttachments(ticket_id)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1283, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1316, in single_request
    return self.parse_response(response)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1493, in parse_response
    return u.close()
  File "/usr/lib/python2.7/xmlrpclib.py", line 800, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 404: "Attachment '6964_validator_init_quickfix.patch' not found">

In this case it seems we have indeed a database corruption:

Plan to support GitLab 11?

Do you have any plan to support migration towards an up-to-date GitLab instance? (currently 11.0 or 11.1)

port model introspector/generator

Trouble using migrate

I was trying to migrate a bunch of old class Trac sites that were hosted on an old Mac runing 10.9. When I try to run the migrate option, I run into issue. Here's the output:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/tracboat", line 11, in <module>
    load_entry_point('tracboat==0.2.0a0', 'console_scripts', 'tracboat')()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tracboat-0.2.0a0-py2.7.egg/tracboat/cli.py", line 423, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tracboat-0.2.0a0-py2.7.egg/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tracboat-0.2.0a0-py2.7.egg/tracboat/cli.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tracboat-0.2.0a0-py2.7.egg/tracboat/cli.py", line 413, in migrate
    userattrs=userattrs,
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tracboat-0.2.0a0-py2.7.egg/tracboat/migrate.py", line 309, in migrate
    output_uploads_path, create_missing=True)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tracboat-0.2.0a0-py2.7.egg/tracboat/gitlab/direct.py", line 75, in __init__
    self.model.Projects.create_table(fail_silently=True)
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 4966, in create_table
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 4962, in table_exists
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 4137, in get_tables
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3775, in execute_sql
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3598, in __exit__
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3766, in execute_sql
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3714, in get_cursor
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3703, in get_conn
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3680, in connect
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3708, in _create_connection
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3598, in __exit__
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 3708, in _create_connection
  File "build/bdist.macosx-10.6-intel/egg/peewee.py", line 4102, in _connect
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/psycopg2-2.7.3-py2.7-macosx-10.6-intel.egg/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
peewee.OperationalError: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql//.s.PGSQL.5432"?

I've never used PostgreSQL so please forgive my ignorance. It seems like defaults for Postgres seem to favor /usr/local/var/postgres on Mac. I started up postgres, made a user "gitlab" and a database "gitlabhq_production" like this:

pg_ctl -D /usr/local/var/postgres/ start

psql postgres
[postgres=# CREATE ROLE gitlab WITH LOGIN PASSWORD 'gitlab1234';
[postgres=# ALTER ROLE gitlab CREATEDB
[postgres=# \q

psql postgres -U gitlab
[postgres=> CREATE DATABASE gitlabhq_production;
[postgres=> GRANT ALL PRIVILEGES ON DATABASE gitlabhq_production TO gitlab;
[postgres=> \list;

I added the following to the config file:

gitlab_db_path="usr/local/var/postgres"
gitlab_db_password = "gitlab1234"

But I still end up with the error like:

    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
peewee.OperationalError: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/usr/local/var/postgres/.s.PGSQL.5432"?

Is there something I am missing or not configuring right?

missing git tag

https://pypi.python.org/pypi/tracboat has 0.0.1 released
it should be tagged in git with same version 0.0.1
but none published as of now: https://github.com/nazavode/tracboat/releases

problems with attachments and encoding

Thanks for tracboat - it gives me hope to migrate sucessfully from trac to gitlab.

I have problems to read the attachments from the trac instance.

with python:3.4 I get several messages like this. I assume it is one per attachment.

2019-03-08 13:47:49,730 ERROR tracboat.trac: error while decoding data from base64
Traceback (most recent call last):
  File "/src/tracboat/trac.py", line 15, in _safe_retrieve_data
    return six.b(data)
  File "/usr/local/lib/python3.4/site-packages/six.py", line 620, in b
    return s.encode("latin-1")
AttributeError: 'bytes' object has no attribute 'encode'

when I use python:2.7 i get

Traceback (most recent call last):
  File "/usr/local/bin/tracboat", line 11, in <module>
    load_entry_point('tracboat', 'console_scripts', 'tracboat')()
  File "/src/tracboat/cli.py", line 428, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/src/tracboat/cli.py", line 268, in export
    project = _dumps(project, fmt=format)
  File "/src/tracboat/cli.py", line 44, in _dumps
    return json.dumps(obj, sort_keys=True, indent=2, default=json_util.default)
  File "/usr/local/lib/python2.7/json/__init__.py", line 251, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/local/lib/python2.7/json/encoder.py", line 209, in encode
    chunks = list(chunks)
  File "/usr/local/lib/python2.7/json/encoder.py", line 434, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/usr/local/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/local/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/local/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/local/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/local/lib/python2.7/json/encoder.py", line 390, in _iterencode_dict
    yield _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte

Extremely slow

This process via RPC is extremely slow! Or am I doing something wrong? My base has more than 14,000 tickets. The migration process is running for more than 8 hours and does not end.

custom parameters for user creation

As requested in #7 it would be useful to allow the user to specify custom values for kwargs passed to __init__ of Users class.

UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

2019-03-01 20:37:26,687 DEBUG tracboat.trac: milestone_get_all
2019-03-01 20:37:26,687 DEBUG tracboat.trac: milestone_get_all_names
2019-03-01 20:37:58,756 DEBUG tracboat.trac: project_get is collecting authors from project
Traceback (most recent call last):
  File "/home/user1/tracboat/VENV/bin/tracboat", line 11, in <module>
    load_entry_point('tracboat', 'console_scripts', 'tracboat')()
  File "/home/user1/tracboat/src/tracboat/cli.py", line 428, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/user1/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/home/user1/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user1/tracboat/src/tracboat/cli.py", line 268, in export
    project = _dumps(project, fmt=format)
  File "/home/user1/tracboat/src/tracboat/cli.py", line 44, in _dumps
    return json.dumps(obj, sort_keys=True, indent=2, default=json_util.default)
  File "/usr/lib/python2.7/json/__init__.py", line 251, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 209, in encode
    chunks = list(chunks)
  File "/usr/lib/python2.7/json/encoder.py", line 434, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/usr/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 390, in _iterencode_dict
    yield _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

how to use usermap

need example how to use usermap from a config file. might be simple error here due not knowing toml syntax (or it's glitches)

# cat comment.toml
[tracboat]
trac_uri = "https://trac.example.net/trac/comment/login/xmlrpc"
usermap_file = comment-users.toml

# cat comment-users.toml
[usermap]
glen = "glen"

# tracboat --config-file=comment.toml migrate --mock
Traceback (most recent call last):
  File "/home/glen/tracboat/VENV/bin/tracboat", line 9, in <module>
    load_entry_point('tracboat==0.2.0-alpha', 'console_scripts', 'tracboat')()
  File "/home/glen/tracboat/tracboat/cli.py", line 412, in main
    cli(obj={})
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1063, in invoke
    Command.invoke(self, ctx)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/glen/tracboat/tracboat/cli.py", line 198, in cli
    conf = toml.load(config_file).get('tracboat', {})
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/toml.py", line 40, in load
    return loads(ffile.read(), _dict)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/toml.py", line 286, in loads
    ret = _load_line(line, currentlevel, multikey, multibackslash)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/toml.py", line 353, in _load_line
    value, vtype = _load_value(pair[1])
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/toml.py", line 508, in _load_value
    raise TomlDecodeError("This float doesn't have a leading digit")
toml.TomlDecodeError: This float doesn't have a leading digit

use gitlab-export format

perhaps it's best just to output json which can be imported by gitlab:
https://gitlab.com/help/user/project/settings/import_export.md

this way don't have to know about internals, and keep updating schema files.
and it's actually stable api, given you support specific version of export.

project export creates simple tar archive:

VERSION - contains export archive version
project.json - json structure of the project, settings, users, milestones, ...
project.bundle - git bundle of the repository

here's overview of the project.json:

unit tests and CI

Environment variable TRACBOAT_TRAC_URI not taken into account

Using current master (VENV), it seems the environment variables are not taken into account:

$ export TRACBOAT_TRAC_URI="https://myserver.de/xmlrpc"
$ tracboat -v users
2019-05-04 16:09:12,122 INFO  users: crawling Trac instance: http://localhost/xmlrpc
...

On the command line it works:

$ tracboat -v users --trac-uri=$TRACBOAT_TRAC_URI
2019-05-04 16:16:06,407 INFO  users: crawling Trac instance: https://myserver.de/xmlrpc
...

look-behind requires fixed-width pattern

Using python2.7 with the head from tracboat I get the following error.

  File "/home/bbn-admin/tracboat/src/tracboat/migrate.py", line 29, in _wikiconvert
    return trac2down.convert(text, basepath, multiline, note_map=note_map, attachments_path=attachments_path, svn2git_revisions=svn2git_revisions)
  File "/home/bbn-admin/tracboat/src/tracboat/trac2down.py", line 219, in convert
    line = re.sub(r'\[(?<!!|wiki:)([A-Z][a-z]+[a-z0-9]*[A-Z][^\ \]]*)\]', r'[\1](\1)', line) # [WikiName] format
  File "/home/bbn-admin/tracboat/VENV/lib64/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/home/bbn-admin/tracboat/VENV/lib64/python2.7/re.py", line 242, in _compile
    raise error, v # invalid expression
sre_constants.error: look-behind requires fixed-width pattern

unable to match users by aliases

seems usermap is not able to find users via user aliases (https://gitlab.com/profile/emails)

[tracboat.usermap]
    glen = "[email protected]"

the [email protected] is specified as secondary email for user having [email protected] as primary address

12.7.5 : AttributeError: 'module' object has no attribute 'IssueAssignees'

Trying to migrate a Trac 1.2 to a GitLab 12.7.5 using 11.0 model, i get this error

Traceback (most recent call last):
File "/usr/bin/tracboat", line 11, in
load_entry_point('tracboat', 'console_scripts', 'tracboat')()
File "/var/opt/gitlab/tracboat/src/tracboat/cli.py", line 443, in main
cli(obj={}) # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/var/opt/gitlab/tracboat/src/tracboat/cli.py", line 119, in wrapper
return func(*args, **kwargs)
File "/var/opt/gitlab/tracboat/src/tracboat/cli.py", line 169, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/var/opt/gitlab/tracboat/src/tracboat/cli.py", line 420, in migrate
svn2git_revisions=svn2git_revisions,
File "/var/opt/gitlab/tracboat/src/tracboat/migrate.py", line 431, in migrate
output_uploads_path, create_missing=False)
File "/var/opt/gitlab/tracboat/src/tracboat/gitlab/direct.py", line 82, in init
self.model.IssueAssignees.create_table(fail_silently=True)
AttributeError: 'module' object has no attribute 'IssueAssignees'

Any plans to officially support the 12.x branch ?

Thanks

12.x support?

are 12.x models supported?

add proper unit tests

11.9.7-ce.0: missing index_issues_on_project_id_and_iid update

at least with 11.9 must update internal_ids table:

https://gitlab.com/gitlab-org/gitlab-ce/issues/61111

or creating new issues yields error:

web_1  | 2019-04-29T11:05:23.962966900Z ActiveRecord::RecordNotUnique (PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_issues_on_project_id_and_iid"
web_1  | 2019-04-29T11:05:23.962999800Z DETAIL:  Key (project_id, iid)=(256, 3) already exists.

13.x support

I'm trying to migrate on-premise to gitlab-ee version 13.3. I've generated a new model for this version but have some difficulties to track the issue.

When is run I get the following output:
(VENV) [gitlab-psql@olnxgitl01 tracboat-master]$ tracboat --verbose --config-file=config.toml migrate 2020-10-01 15:30:45,905 INFO migrate: updating usermap with mappings from 'config.toml' 2020-10-01 15:30:45,905 INFO migrate: updating user attributes with info from 'config.toml' 2020-10-01 15:30:45,906 INFO migrate: loading Trac instance from export file: satellite.json 2020-10-01 15:30:46,050 INFO migrate: migrating Trac project to GitLab 2020-10-01 15:30:46,050 INFO migrate: Trac: https://plnxsccl02.ont.belastingdienst.nl/trac/oes/login/xmlrpc 2020-10-01 15:30:46,050 INFO migrate: GitLab project: lph/satellite 2020-10-01 15:30:46,050 INFO migrate: GitLab version: 13.3 2020-10-01 15:30:46,050 INFO migrate: GitLab db path: /var/opt/gitlab/postgresql/ 2020-10-01 15:30:46,050 INFO migrate: GitLab db name: gitlabhq_production 2020-10-01 15:30:46,051 INFO migrate: GitLab uploads: /var/opt/gitlab/gitlab-rails/uploads 2020-10-01 15:30:46,051 INFO migrate: GitLab fallback user: hendr00 2020-10-01 15:30:46,051 INFO tracboat.migrate: migrating project u'lph/satellite' to GitLab ver. 13.3 2020-10-01 15:30:46,051 INFO tracboat.migrate: uploads repository path is: '/var/opt/gitlab/gitlab-rails/uploads' Traceback (most recent call last): File "/srv/migrate/tracboat-master/VENV/bin/tracboat", line 11, in <module> load_entry_point('tracboat', 'console_scripts', 'tracboat')() File "/srv/migrate/tracboat-master/src/tracboat/cli.py", line 442, in main cli(obj={}) # pylint: disable=unexpected-keyword-arg,no-value-for-parameter File "/srv/migrate/tracboat-master/VENV/lib/python2.7/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/srv/migrate/tracboat-master/VENV/lib/python2.7/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/srv/migrate/tracboat-master/VENV/lib/python2.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/srv/migrate/tracboat-master/VENV/lib/python2.7/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/srv/migrate/tracboat-master/VENV/lib/python2.7/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/srv/migrate/tracboat-master/src/tracboat/cli.py", line 132, in wrapper return func(*args, **kwargs) File "/srv/migrate/tracboat-master/src/tracboat/cli.py", line 182, in wrapper return func(*args, **kwargs) File "/srv/migrate/tracboat-master/VENV/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/srv/migrate/tracboat-master/src/tracboat/cli.py", line 432, in migrate svn2git_revisions=svn2git_revisions, File "/srv/migrate/tracboat-master/src/tracboat/migrate.py", line 428, in migrate db_model = model.get_model(gitlab_version) File "/srv/migrate/tracboat-master/src/tracboat/gitlab/model/__init__.py", line 16, in get_model return importlib.import_module(module_path) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/srv/migrate/tracboat-master/src/tracboat/gitlab/model/model133.py", line 42, in <module> class PoolRepositories(BaseModel): File "/srv/migrate/tracboat-master/src/tracboat/gitlab/model/model133.py", line 46, in PoolRepositories source_project = ForeignKeyField(db_column='source_project_id', null=True, rel_model=Projects, to_field='id') NameError: name 'Projects' is not defined

model133.py.txt

milestones created with missing :iid crash issue view

similar to #22 milestones created need also :iid field. for issue i thought it's issue id, but no clue what is it for milestones, if i created from admin, then the value was 1

irb(main):020:0* Milestone.find_by(id:2468)
=> #<Milestone id: 2468, title: "V 2.3", project_id: 166, description: "", due_date: "2014-01-29", created_at: "2017-04-22 01:58:19", updated_at: "2017-04-22 01:58:19", state: "active", iid: nil, title_html: nil, description_html: nil, start_date: nil>
irb(main):017:0> Milestone.find_by(iid:nil).delete

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/comment/comment/issues/1" for 10.1.2.66 at 2017-04-22 01:58:48 +0300
Processing by Projects::IssuesController#show as HTML
  Parameters: {"namespace_id"=>"comment", "project_id"=>"comment", "id"=>"1"}
Completed 500 Internal Server Error in 225ms (ActiveRecord: 13.8ms)

ActionView::Template::Error (No route matches {:action=>"show", :controller=>"projects/milestones", :id=>#<Milestone id: 2475, 
title: "Launch candidate release", project_id: 166, description: "", due_date: "2010-10-26", created_at: "2017-04-22 01:58:19",
 updated_at: "2017-04-22 01:58:19", state: "closed", iid: nil, title_html: nil, description_html: nil, start_date: nil>, :names
pace_id=>#<Group id: 39, name: "comment", path: "comment", owner_id: nil, created_at: "2017-04-06 11:09:12", updated_at: "2017-
04-06 11:09:12", type: "Group", description: "", avatar: nil, share_with_group_lock: false, visibility_level: 10, request_acces
s_enabled: false, deleted_at: nil, lfs_enabled: nil, description_html: nil, parent_id: nil>, :project_id=>#<Project id: 166, na
me: "comment", path: "comment", description: "", created_at: "2017-03-31 08:40:49", updated_at: "2017-04-06 11:09:40", creator_
id: 2, namespace_id: 39, last_activity_at: "2017-04-21 22:40:18", import_url: nil, visibility_level: 10, archived: false, avata
r: nil, import_status: "none", star_count: 0, import_type: nil, import_source: nil, import_error: nil, ci_id: nil, shared_runne
rs_enabled: true, runners_token: "", build_coverage_regex: nil, build_allow_git_fetch: true, build_timeout:
 3600, pending_delete: false, public_builds: true, last_repository_check_failed: false, last_repository_check_at: "2017-04-01 0
9:20:14", container_registry_enabled: true, only_allow_merge_if_pipeline_succeeds: false, has_external_issue_tracker: false, re
pository_storage: "default", request_access_enabled: false, has_external_wiki: false, lfs_enabled: nil, description_html: "", only_allow_merge_if_all_discussions_are_resolved: false>} missing required keys: [:id]):
    69:             = link_to 'Edit', '#', class: 'edit-link pull-right'
    70:         .value.hide-collapsed
    71:           - if issuable.milestone
    72:             = link_to issuable.milestone.title, namespace_project_milestone_path(@project.namespace, @project, issuable.milestone), class: "bold has-tooltip", title: milestone_remaining_days(issuable.milestone), data: { container: "body", html: 1 }
    73:           - else
    74:             %span.no-value None
    75:
  app/views/shared/issuable/_sidebar.html.haml:72:in `block in _app_views_shared_issuable__sidebar_html_haml__3610394230630219408_69913765138340'
  app/views/shared/issuable/_sidebar.html.haml:24:in `_app_views_shared_issuable__sidebar_html_haml__3610394230630219408_69913765138340'
  app/views/projects/issues/show.html.haml:82:in `_app_views_projects_issues_show_html_haml___972156526271696380_69913708880840'
  app/controllers/projects/issues_controller.rb:88:in `show'
  lib/gitlab/middleware/multipart.rb:93:in `call' 
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/etag_caching/middleware.rb:14:in `call'
  lib/gitlab/request_context.rb:18:in `call'

fix wiki conversion

As of today, Trac to Markdown conversion code was pulled in with no (or just minor) modifications. At a quick look, converted files are full of crap, we need to check for:

Inline code (e.g.: stuff converted to _code_ (italic) instead of `code` (code)
Code blocks with highlighting aren't converted (e.g.: Trac sections starting with {{{ #!C++)
Some code blocks ({{{ ... }}}) are not converted at all
Check cross-references between pages
Check references to attachments

migrating other changelog entries than field=comment

there are other events in changelog, like status and milestone changes. those could be imported as well.

but trac structure is weird:

INFO:tracboat.migrate:changelog: [ 
    {'author': 'indrek', 'field': 'comment', 'permanent': True, 'oldvalue': '1', 'time': datetime.datetime(2010, 4, 26, 13, 7, 39), 'newvalue': "1. time"}, 
    {'author': 'indrek', 'field': 'comment', 'permanent': True, 'oldvalue': '2', 'time': datetime.datetime(2010, 4, 27, 6, 40, 14), 'newvalue': '2. there'}, 
    {'author': 'toomas', 'field': 'comment', 'permanent': True, 'oldvalue': '3', 'time': datetime.datetime(2010, 4, 30, 14, 15, 39), 'newvalue': ''},  
    {'author': 'toomas', 'field': 'resolution', 'permanent': True, 'oldvalue': '', 'time': datetime.datetime(2010, 4, 30, 14, 15, 39), 'newvalue': 'fixed'},  
    {'author': 'toomas', 'field': 'status', 'permanent': True, 'oldvalue': 'new', 'time': datetime.datetime(2010, 4, 30, 14, 15, 39), 'newvalue': 'closed'}  
] ```

basically this ticket has two visible comments and third comment saying

Resolution set to fixed
Status changed from new to closed


and the third field=comment body is empty. as the last 3 changelog entries have same date trac shows them all in same "comment" visually.

oldvalue for field=comment is comment sequence number?

11.4 Migration Issue.

GitLab 11.4.5 (f5536c61210)

Properties file entires:

[tracboat]
trac_uri = "https://url/project/login/xmlrpc"
gitlab_project_name = "user/project"
gitlab_version = "11.0"
fallback_user = "[email protected]"

Log Snippet

2018-11-14 17:26:40,318 DEBUG tracboat.trac: ticket_get_attachments of ticket #242
2018-11-14 17:26:41,415 DEBUG tracboat.trac: milestone_get_all
2018-11-14 17:26:41,416 DEBUG tracboat.trac: milestone_get_all_names
2018-11-14 17:26:52,860 DEBUG tracboat.trac: project_get is collecting authors from project
2018-11-14 17:26:52,862 INFO migrate: migrating Trac project to GitLab
2018-11-14 17:26:52,862 INFO migrate: Trac: https://url/project/login/xmlrpc
2018-11-14 17:26:52,862 INFO migrate: GitLab project: user/project
2018-11-14 17:26:52,862 INFO migrate: GitLab version: 11.0
2018-11-14 17:26:52,862 INFO migrate: GitLab db path: /var/opt/gitlab/postgresql/
2018-11-14 17:26:52,862 INFO migrate: GitLab db name: gitlabhq_production
2018-11-14 17:26:52,862 INFO migrate: GitLab uploads: /var/opt/gitlab/gitlab-rails/uploads
2018-11-14 17:26:52,862 INFO migrate: GitLab fallback user: [email protected]
2018-11-14 17:26:52,862 INFO tracboat.migrate: migrating project u'user/project' to GitLab ver. 11.0
2018-11-14 17:26:52,862 INFO tracboat.migrate: uploads repository path is: '/var/opt/gitlab/gitlab-rails/uploads'
2018-11-14 17:26:52,920 INFO tracboat.migrate: retrieved database model for GitLab ver. 11.0: '/home/ubuntu/tracboat/src/tracboat/gitlab/model/model110.pyc'
Traceback (most recent call last):
File "/home/ubuntu/tracboat/VENV/bin/tracboat", line 11, in
load_entry_point('tracboat', 'console_scripts', 'tracboat')()
File "/home/ubuntu/tracboat/src/tracboat/cli.py", line 444, in main
cli(obj={}) # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/ubuntu/tracboat/src/tracboat/cli.py", line 134, in wrapper
return func(*args, **kwargs)
File "/home/ubuntu/tracboat/src/tracboat/cli.py", line 184, in wrapper
return func(*args, **kwargs)
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/ubuntu/tracboat/src/tracboat/cli.py", line 434, in migrate
svn2git_revisions=svn2git_revisions,
File "/home/ubuntu/tracboat/src/tracboat/migrate.py", line 438, in migrate
output_uploads_path, create_missing=False)
File "/home/ubuntu/tracboat/src/tracboat/gitlab/direct.py", line 72, in init
self.model.database_proxy.initialize(db_connector)
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 420, in initialize
callback(obj)
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 1186, in _set_constructor
self._constructor = database.get_binary_type()
File "/home/ubuntu/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 4209, in get_binary_type
return psycopg2.Binary
AttributeError: 'NoneType' object has no attribute 'Binary'

migrate binary attachments associated to ticket changelog entries

port GitLab remote api-based migration

improve documentation

No proper documentation is planned, everything should be in the README with a fairly complete coverage of usage scenarios.

create projects using rest-api not direct-api

(as pointed out earlier), creating users and now projects using directapi can cause gitlab to malfunction.

currently i got it broken because project was created so that some field is missing causing admin to crash:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/admin" for 10.1.2.66 at 2017-08-07 16:01:05 +0300
Processing by Admin::DashboardController#index as HTML
Completed 500 Internal Server Error in 60ms (ActiveRecord: 6.8ms)

ActionView::Template::Error (undefined method `+' for nil:NilClass):
    161:             %h4 Latest projects
    162:             - @projects.each do |project|
    163:               %p
    164:                 = link_to project.name_with_namespace, [:admin, project.namespace.becomes(Namespace), project], class: 'str-truncated-60'
    165:                 %span.light.pull-right
    166:                   #{time_ago_with_tooltip(project.created_at)}
    167:       .col-md-4
  app/models/concerns/routable.rb:148:in `build_full_name'
  app/models/concerns/routable.rb:162:in `prepare_route'
  app/models/concerns/routable.rb:155:in `update_route'
  app/models/concerns/routable.rb:93:in `full_name'
  app/views/admin/dashboard/index.html.haml:164:in `block in _app_views_admin_dashboard_index_html_haml___3343335019850492670_69929727036400'
  app/views/admin/dashboard/index.html.haml:162:in `_app_views_admin_dashboard_index_html_haml___3343335019850492670_69929727036400'
  lib/gitlab/i18n.rb:45:in `with_locale'
  lib/gitlab/i18n.rb:51:in `with_user_locale'
  app/controllers/application_controller.rb:294:in `set_locale'
  lib/gitlab/performance_bar/peek_performance_bar_with_rack_body.rb:16:in `call'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'

Exclude ticket states from export

As show in #28 it would be useful to be able to exclude (or include only a subset) some ticket states from the export process.

migrate wiki binary attachments

OOM during migrate

I'm running out of system memory during tracboat migrate (when migrating with attachments). Anyone else ever experience this? Is there a way to break up the import so I can complete it with the memory I have? Other suggestions?

root@gitlab:/var/opt/gitlab/temp/tracboat# ./tracboat.sh -vv --config-file=mytrac.toml migrate 2>&1 | tee log_newJ
2019-03-07 22:27:07,656 DEBUG migrate: usermap is: {}
2019-03-07 22:27:07,656 DEBUG migrate: default user attributes are: {}
2019-03-07 22:27:07,656 DEBUG migrate: user attributes is: defaultdict(<function <lambda> at 0x7fcf7ea10848>, {})
2019-03-07 22:27:07,656 INFO  migrate: loading Trac instance from export file: new_repo.json
2019-03-07 22:27:07,656 DEBUG migrate: detected file format: json
Traceback (most recent call last):
  File "/var/opt/gitlab/temp/tracboat/VENV/bin/tracboat", line 11, in <module>
    load_entry_point('tracboat', 'console_scripts', 'tracboat')()
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/cli.py", line 428, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/cli.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/var/opt/gitlab/temp/tracboat/src/tracboat/cli.py", line 377, in migrate
    content = export_f.read()
  File "/var/opt/gitlab/temp/tracboat/VENV/lib/python2.7/codecs.py", line 686, in read
    return self.reader.read(size)
  File "/var/opt/gitlab/temp/tracboat/VENV/lib/python2.7/codecs.py", line 492, in read
    newchars, decodedbytes = self.decode(data, self.errors)
MemoryError

minimum python2.7 requirement

i encountered this problem when running on debian7:

return xmlrpc.ServerProxy(url, encoding=encoding, use_datetime=use_datetime, context=context)
TypeError: __init__() got an unexpected keyword argument 'context'

full trace:

Traceback (most recent call last):
  File "/home/glen/tracboat/VENV/bin/tracboat", line 9, in <module>
    load_entry_point('tracboat==0.2.0a0', 'console_scripts', 'tracboat')()
  File "/home/glen/tracboat/src/tracboat/cli.py", line 428, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/glen/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/home/glen/tracboat/src/tracboat/cli.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/glen/tracboat/src/tracboat/cli.py", line 381, in migrate
    source = trac.connect(trac_uri, encoding='UTF-8', use_datetime=True, ssl_verify=ssl_verify)
  File "/home/glen/tracboat/src/tracboat/trac.py", line 160, in connect
    return xmlrpc.ServerProxy(url, encoding=encoding, use_datetime=use_datetime, context=context)
TypeError: __init__() got an unexpected keyword argument 'context'

seems that xmlrpc.ServerProxy does not have context param in 2.7.3, while does have in 2.7.9

$ pydoc xmlrpclib.ServerProxy|grep __init__
 |  __init__(self, uri, transport=None, encoding=None, verbose=0, allow_none=0, use_datetime=0)
(VENV)
$ python --version
Python 2.7.3

$ pydoc xmlrpclib.ServerProxy|grep __init__
 |  __init__(self, uri, transport=None, encoding=None, verbose=0, allow_none=0, use_datetime=0, context=None)
$ python --version
Python 2.7.9

probably should define the min version somewhere (or add support for older python?)

user creation broken with 9.3 models

with 9.3 model (#33) i get error:

root@soa ~/tracboat# sudo -H -u git `which tracboat` --config-file=zave.toml migrate  --mock --mock-path=export/zave
Traceback (most recent call last):
  File "/home/glen/tracboat/VENV/bin/tracboat", line 9, in <module>
    load_entry_point('tracboat==0.2.0a0', 'console_scripts', 'tracboat')()
  File "/home/glen/tracboat/src/tracboat/cli.py", line 423, in main
    cli(obj={})  # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/glen/tracboat/src/tracboat/cli.py", line 118, in wrapper
    return func(*args, **kwargs)
  File "/home/glen/tracboat/src/tracboat/cli.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/glen/tracboat/src/tracboat/cli.py", line 413, in migrate
    userattrs=userattrs,
  File "/home/glen/tracboat/src/tracboat/migrate.py", line 344, in migrate
    output_uploads_path, create_missing=True)
  File "/home/glen/tracboat/src/tracboat/gitlab/direct.py", line 92, in __init__
    **NAMESPACE_DEFAULTS)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 4905, in create
    inst.save(force_insert=True)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 5098, in save
    pk_from_cursor = self.insert(**field_dict).execute()
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3516, in execute
    cursor = self._execute()
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 2902, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3765, in execute_sql
    self.commit()
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3588, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/home/glen/tracboat/VENV/local/lib/python2.7/site-packages/peewee.py", line 3758, in execute_sql
    cursor.execute(sql, params or ())
peewee.IntegrityError: NOT NULL constraint failed: namespaces.require_two_factor_authentication
(VENV)

i tried to patch this based on documentation, but seems the values are different.

root@soa ~/tracboat# git diff
diff --git a/src/tracboat/migrate.py b/src/tracboat/migrate.py
index 28042b1..8b13316 100644
--- a/src/tracboat/migrate.py
+++ b/src/tracboat/migrate.py
@@ -349,7 +349,9 @@ def migrate(trac, gitlab_project_name, gitlab_version, gitlab_db_connector,
             'email': email,
             'username': email.split('@')[0],
             'encrypted_password': generate_password(),
+            'two_factor_enabled' : False,
         }
+
         attrs.update(userattrs.get(email, {}))
         gitlab.create_user(**attrs)
         LOG.info('created GitLab user %r', email)
(VENV)

imho creating users should be always done over rest API, never via direct models. as it's better maintained this way and doesn't need to know internals.

for creating tickets i understand the need for direct-api because rest api doesn't support "faking" date fields, etc. already suggested this: #21 (comment)