tonyduckles / svn2svn Goto Github PK

View Code? Open in Web Editor NEW

32.0 32.0 10.0 271 KB

Replicate/replay Subversion revision history

License: GNU General Public License v3.0

Shell 22.70% Python 76.05% Batchfile 0.32% Makefile 0.93%

svn2svn's People

Contributors

Stargazers

Watchers

Forkers

abalkin pombredanne davidcie xman1212 quinot a-wilhelm freddieho lithiumtech smithgb gainsury

svn2svn's Issues

Create automated-tests which cover all code-paths

I don't think the current tests/make-ref-repo.sh script covers all the code-paths in svn2svn/run/svn2svn.py. The automated tests should have full code-coverage, to make sure we're testing all the nooks and crannies.

Add verify-mode

Add optional command-line arg(s) for verifying the replayed target history matches the (logical) source history.

Need a verify-all (post-commit) arg, for verifying a fully replayed target.
Maybe want a verify-changed (ideally pre-commit, but post-commit would work too), to verify individual replayed commits as we go. Might be useful for debug/testing, but likely not for real-world uses.

Better KeyboardInterrupt handling in real_main()

Do better KeyboardInterrupt try/except handling in real_main(). It's fine for a process_svn_log_entry call to be interrupted, but other calls here should be considered atomic, e.g. commit_from_svn_log_entry.

Windows: terminal color coding doesn't work under cmd.exe

When running under cmd.exe (CPython 2.7.2 on Windows XP 64), the color escape codes aren't interpreted and make debug output rather messy. I ended up just commenting out:

#if color in _colors:
#    msg = '%s%s%s' % ("\x1b["+_colors[color]+(";1" if bold else "")+"m", msg, "\x1b[0m")

in ui.py to get going. I'm not aware of the coding necessary for cmd.exe, so perhaps it's easiest to just check the SHELL environment variable or os.name and disable it.

For options.keep_date, check if pre-revprop-change hook setup in target repo

If options.keep_date is True, should we try doing a "svn propset" on an existing revision as a sanity check, so we check if the pre-revprop-change hook script is correctly setup before doing first replay-commit?

Incorrect non-ASCII paths handling

When working with Subversion repo with non-ASCII characters in paths one gets KeyError in urllib.quote:

Traceback (most recent call last):
  File ".../svnreplay.py", line 5, in <module>
    sys.exit(main() or 0)
  File ".../svn2svn/run/svnreplay.py", line 1108, in main
    return real_main(args)
  File ".../svn2svn/run/svnreplay.py", line 896, in real_main
    svnclient.export(join_path(source_start_url, path_offset), source_start_rev, path_offset, force=True)
  File ".../svn2svn/svnclient.py", line 506, in export
    args += [safe_path(svn_url, rev_number), safe_path(path)]
  File ".../svn2svn/svnclient.py", line 43, in safe_path
    path = urllib.quote(path, ":/+")
  File ".../python2.7/urllib.py", line 1303, in quote
    return ''.join(map(quoter, s))
KeyError: u'\u0430'

This can be fixed with explicit encoding unicode paths with user-provided encoding when forming safe path.

Incorrect handling of dir -> file replaces

When replacing dir, working copy must be updated before committing. Svn2svn does this correctly for dir -> dir replaces, but not for cases when dir is replaced by file:

Traceback (most recent call last):
  File ".../svn2svn/run/svnreplay.py", line 960, in real_main
    target_rev = commit_from_svn_log_entry(log_entry, commit_paths, target_revprops=target_revprops)
  File ".../svn2svn/run/svnreplay.py", line 102, in commit_from_svn_log_entry
    output = run_svn(args)
  File ".../svn2svn/shell.py", line 208, in run_svn
    args=args, bulk_args=bulk_args, fail_if_stderr=fail_if_stderr, no_fail=no_fail)
  File ".../svn2svn/shell.py", line 141, in run_command
    return _run_raw_command(cmd, map(_transform_arg, args), fail_if_stderr, no_fail)
  File ".../svn2svn/shell.py", line 114, in _run_raw_command
    % (pipe.returncode, cmd_string, err, out))
ExternalCommandFailed: External program failed (return code 1): svn commit --force-log -m 'Replace dir with file' --with-revprop 'svn2svn:source_uuid=46727182-e339-4f52-8b88-d18f8a30a573' --with-revprop 'svn2svn:source_rev=3' --with-revprop 'svn2svn:source_url=file://...' dir
svn: E155011: Commit failed (details follow):
svn: E155011: File '.../_wc_target/dir' is out of date
svn: E160028: Directory '/dir' is out of date

Multiple source roots?

I would like to extract multiple directories, e.g. trunk/foo and trunk/bar, keeping revision numbers, into a new repo. Is this possible with svn2svn?

Inital padding revisions error

An excellent piece of work and is just what I needed for dis-tangling some projects out of a massive repo!

It all seems to work well under Cywin on my Windows machine.

One small error I think I've found is that the when preserving revision numbers, the first lot of padding if skipped, as you've used source_rev instead of source_start_rev and this meant the first commit from the source always went in as rev #1 in the destination and then there was a bunch of padding to get it back in step for the next commit. Diff file for the fix is:

--- C:/Temp/SVN Migration/svn2svn-master/svn2svn-master/svn2svn/run/svnreplay.py    Sat Sep 19 23:18:50 2015
+++ C:/Temp/SVN Migration/svn2svn-master/svn2svn-master/svn2svn/run/svnreplay - Copy.py Fri Jan 22 10:33:35 2016
@@ -876,8 +876,8 @@ def real_main(args):
         source_start_rev = int(source_start_log['revision'])
         ui.status("Starting at source revision %s.", source_start_rev, level=ui.VERBOSE)
         ui.status("", level=ui.VERBOSE)
-        if options.keep_revnum and source_rev > target_rev_last:
-            target_rev_last = keep_revnum(source_rev, target_rev_last, wc_target_tmp)
+        if options.keep_revnum and source_start_rev > target_rev_last:
+            target_rev_last = keep_revnum(source_start_rev, target_rev_last, wc_target_tmp)

         # For the initial commit to the target URL, export all the contents from
         # the source URL at the start-revision

Verify do_svn_add_dir() handling

There are some "TODO" questions in do_svn_add_dir(). Need to follow-up on those.

svnreplay doesn't seem to handle comments with carriage returns (running on Windows)

Traceback (most recent call last):
File "C:\stephen\working\svnreplay.py", line 5, in
sys.exit(main() or 0)
File "C:\stephen\working\svn2svn\run\svnreplay.py", line 1106, in main
return real_main(args)
File "C:\stephen\working\svn2svn\run\svnreplay.py", line 910, in real_main
target_rev = commit_from_svn_log_entry(source_start_log, target_revprops=target_revprops)
File "C:\stephen\working\svn2svn\run\svnreplay.py", line 103, in commit_from_svn_log_entry
rev_num = parse_svn_commit_rev(output) if output else None
File "C:\stephen\working\svn2svn\run\svnreplay.py", line 46, in parse_svn_commit_rev
return int(rev_num)
ValueError: invalid literal for int() with base 10: '2.\r'

Windows: extracting paths from svn logs misses CRs

In svn2svn.py, I had to change any calls to

top_paths.strip("\n").split("\n")

top_paths.strip("\r\n").split("\r\n").

Otherwise, the carriage returns stay in and mess up subsequent svn commands. I'm not certain about the .split() portion, but it worked for me.

Allow optional before-commit command

Add support for running an optional external shell command before each commit, for doing any pre-commit filtering.

Symoblic link treatment

In my SVN history I have a symbolic link that points to itself. Chashes when replaying SVN history.

While script is performing svn export, this error message occurs:

svn: E000040: Can't check path '...': Too many levels of symbolic links

Windows: rmtree fails

In svn2svn.py, rmtree should be imported from ..shell and all instances of shutil.rmtree changed to the rmtree override defined in shell.py.

invalid syntax on <>

I'm getting a python error. my-project/trunk does exist, but my-project/trunk/my-lib does not.

$ ./svnreplay.py --verbose --keep-author --keep-date --keep-prop file:///var/svn/my-lib/trunk file:///var/svn/my-project/trunk/my-lib
Traceback (most recent call last):
  File "./svnreplay.py", line 3, in <module>
    from svn2svn.run.svnreplay import main
  File "/var/svn/svn2svn/svn2svn/run/svnreplay.py", line 251
    is_diff = True if sum1 <> sum2 else False
                            ^
SyntaxError: invalid syntax

$ python --version
Python 3.4.6

Incorrect symbolic links transferring

Come across this issue when replaying SVN repo with symbolic links.

Code in in_svn function uses svn status to check is file is under Subversion control, but svn status, svn info and possibly other commands seem to follow symbolic links when called directly on them.

Traceback (most recent call last):
  File ".../svn2svn/run/svnreplay.py", line 957, in real_main
    process_svn_log_entry(log_entry, source_ancestors, commit_paths)
  File ".../svn2svn/run/svnreplay.py", line 718, in process_svn_log_entry
    sync_svn_props(source_url, source_rev, path_offset)
  File ".../svn2svn/run/svnreplay.py", line 391, in sync_svn_props
    run_svn(["propset", prop, source_props[prop], svnclient.safe_path(path_offset)])
  File ".../svn2svn/shell.py", line 208, in run_svn
    args=args, bulk_args=bulk_args, fail_if_stderr=fail_if_stderr, no_fail=no_fail)
  File ".../svn2svn/shell.py", line 141, in run_command
    return _run_raw_command(cmd, map(_transform_arg, args), fail_if_stderr, no_fail)
  File ".../svn2svn/shell.py", line 114, in _run_raw_command
    % (pipe.returncode, cmd_string, err, out))
ExternalCommandFailed: External program failed (return code 1): svn propset 'svn:special' '*' '.../symlink'
svn: E200009: Cannot set 'svn:special' on a directory ('.../_wc_target/.../symlink')

Furthermore, cleanup code is not designed to work with symbolic links to directories.

Traceback (most recent call last):
  File ".../svn2svn/svnreplay.py", line 5, in <module>
    sys.exit(main() or 0)
  File ".../svn2svn/run/svnreplay.py", line 1110, in main
    return real_main(args)
  File ".../svn2svn/run/svnreplay.py", line 994, in real_main
    full_svn_revert()
  File ".../svn2svn/run/svnreplay.py", line 361, in full_svn_revert
    shell.rmtree(path)
  File ".../svn2svn/shell.py", line 72, in rmtree
    return shutil.rmtree(path, False, _rmtree_error_handler)
  File "/usr/lib64/python2.6/shutil.py", line 197, in rmtree
    onerror(os.path.islink, path, sys.exc_info())
  File "/usr/lib64/python2.6/shutil.py", line 195, in rmtree
    raise OSError("Cannot call rmtree on a symbolic link")
OSError: Cannot call rmtree on a symbolic link

Need to URL-escape user-supplied source_url/target_url params

If the user-supplied source_url or target_url URLs aren't properly URL-encoded, some of the comparisons in real_main() will fail because the repos_url value obtained from "svn info" will be URL-encoded.

Should be able to use urllib.quote for this.