Summary of Issue:

Hangs doing a checkout when subversion password is missing.

Expected behavior and actual behavior:

Tried doing a checkout on a new system that I've never used subversion on before. It hangs when trying
to checkout ww3. Should be prompted to enter your subversion username and password. I was able to get around this by doing a manual checkout of ww3 and entering my password and saving it. Since my password is now saved, the checkout works.

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

Remove .subversion from your home directory.

What is the changeset ID of the code, and the machine you are using:

Release manic-v0.7.0

have you modified the code? If so, it must be committed and available for testing:

No

Screen output or log file showing the error message and context:

login1.stampede2(79)$ ./manage_externals/checkout_externals
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam,
Checking out externals: clm, mosart, ww3,

checkout_externals changes directories numerous times while being run. This makes it tricky when using local paths as the url to refer to git repo on the same filesystem. Depending on what repo you are cloning, cime, clm, standalon sub-external, e.g. fates, it can require one, two or three parent levels. This is trick to get correct and confusing.

To simplify the user experience, should check for and require absolute paths.

I'm opening this enhancement request on behalf of @cecilehannay , but I see her point, too.

While I like the default behavior of checkout_externals in terms of the level of verbosity, there are times when some extra verbosity can be reassuring, especially when you have changed the CESM.cfg file and are rerunning checkout_externals, and want to see that it's actually doing something.

Our thought is that, when you add the -v flag, you should see all of the output from the underlying git checkout or svn switch commands, rather than having that output be suppressed.

From @cacraigucar on November 20, 2017 21:13

While the help message highlights that checkout_externals.py must be run in the root of the source tree, the error message indicates a different error. The error message is about "not being able to find the model description file". It should probably also suggest: "Is this command being run from the root of the source tree?"

Copied from original issue: ESCOMP/CESM#5

Summary of Issue:

If I update the CESM.cfg file for an external to point to a new tag then run checkout_externals, I sometimes get a failure because the given tag hasn't been fetched. This can occur if there are no branches on the remote that contain the given tag (e.g., when I push a tag to the remote but haven't pushed the branch that tag is on).

Expected behavior and actual behavior:

Expected: should fetch all tags

Actual: only fetches tags contained in branches on the remote

Fix incoming

I'm not sure where is the best location for this (top of README_FIRST, perhaps?), but I was thinking that it would be good to clearly call out the location of the manage_externals repository: Since this is included as a subtree in various places, it would help if people knew where to look to find the source of the code. This is useful for issues, and also in case anyone is interested in including this in their own package.

From @jedwards4b on February 20, 2018 20:46

Summary of Issue: I want to call the equivalent of ./checkout_externals --status --verbose from

an external python program, the current structure of checkout.py does not lend itself well to this desire.

Expected behavior and actual behavior:

Steps to reproduce the problem. Include externals description file(s) and link to public repository):

What is the changeset ID of the code, and the machine you are using:

have you modified the code? If so, it must be committed and available for testing:

Screen output or log file showing the error message and context:

Copied from original issue: ESCOMP/CESM#25

I'm opening this enhancement request on behalf of @cecilehannay , but I see her point, too.

It would be good to have a way to see what's currently checked out. This is similar to #14 , and if that were done, this one wouldn't be absolutely necessary. But it would still be helpful if the status command or some other command - or perhaps the status command with the -v option - showed the current branch/tag that is checked out.

The use case is: If you're not sure whether you updated all externals correctly (e.g., "Shoot, did I remember to run checkout_externals before I ran that case?"), it would be helpful to see at a glance what you have checked out.

When "-o" isn't given that status of optional externals isn't reported on. I'd like to see a warning given that mentions these weren't checked out (at least for normal level of output logging).

I'd also like to see a README placed in the optional directory, that says (this directory wasn't checked out, please rerun checkout_externals at the top level with the "-o" option to make this happen if you need this directory).

I think the behavior should also be that if the directory doesn't exist it does the above creating the directory and adding a README file. But, if the directory does exist, it should ignore it (as presumably you ran checkout_externals with a "-o" option previously. But, it should normally print a warning if this is happening (for example "leaving the existing optional external directory tools/PTCLM since it already exists and you didn't specify the "-o" option).

Even if manage_externals is used exclusively in CESM and its components, having a default name of CESM.cfg doesn't always make sense. For example, @mnlevy1981 is planning to use this in place of svn externals in POP to pull in POP's dependencies (cvmix, marbl). I assume he's going to want to be able to run checkout_externals from a POP checkout to get these. Currently this would require a top-level CESM.cfg file, which doesn't really make sense because he wouldn't be pulling in other CESM components.

What about something generic like EXTERNALS.cfg?

Summary of Issue:

Document manage_externals depenecies:

• git - 1.8, 2.3 <= version <= 2.15
• svn - 1.6.11 <= version <= 1.9.4
• python
  • python2 > 2.7.z
  • python3 > 3.4.z

I find the 'e-o' status output confusing: When I see any letters in a status output, intuitively I think it indicates that the state is "dirty" or that there's some other problem. This was particularly an issue for me when my repo got into a dirty state, so checkout_externals refused to update, and printed out the status: It was hard for me to see what was wrong at first, because my eye was drawn to the 'e-o' lines (where I originally assumed 'e' meant 'error').

Personally, I'd be happy for any optional empty directory to be excluded from the output: i.e., any time it would print 'e-o ...' it instead just leaves that line out (perhaps unless you specify some separate flag, or an optional string following the '-s' flag, that forces printing of these empty optional repositories).

A compromise, though, if others see value in having this printed, would be: If any output is 'e-o' (or maybe just do this always for simplicity), include a line in the output like:

A status of 'e-o' just indicates an optional repository that is currently empty; this is not a problem.

From @jedwards4b on February 20, 2018 19:48

Summary of Issue: The RE_TRACKING reg ex in repository_git.py appears to be too restrictive.

Expected behavior and actual behavior:I have a clm checkout that looks like

git status
On branch pio2_cleanup
Your branch and 'mydev/pio2_cleanup' have diverged,
and have 62 and 2 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   src/main/ncdio_pio.F90.in

no changes added to commit (use "git add" and/or "git commit -a")

Current behavior is:

Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm, 
ERROR: DEV_ERROR: regex to detect tracking branch failed.

If I changes the RE_TRACKING to be RE_TRACKING = re.compile(r'[(.+)]')
then I get what I think that I should expect:

Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam, 
sM  ./cime
        modified sandbox, cesmdev/master: ahead 136 --> cime5.4.0-alpha.23
 M  ./components/cam
        modified sandbox, on cam1/trunk_tags/cam5_4_166/components/cam
s   ./components/cice
        clean sandbox, origin/master --> cice5_20180123
    ./components/cism
        clean sandbox, on cism2_1_46
    ./components/cism/glimmer-cism
        clean sandbox, on test_coupling_move_up_v2_n01
sM  ./components/clm
        modified sandbox, mydev/pio2_cleanup: ahead 62, behind 2 --> clm5.0.dev001
    ./components/clm/src/fates
        clean sandbox, on fates_s1.4.1_a3.0.0_rev3
s   ./components/clm/tools/PTCLM
        clean sandbox, PTCLM2_171216c --> PTCLM2_180214
    ./components/mosart
        clean sandbox, on mosart1_0_30
    ./components/pop
        clean sandbox, on pop2/trunk_tags/cesm_pop_2_1_20180205
    ./components/pop/externals/CVMix
        clean sandbox, on v0.72-conv_bugfix.002
    ./components/pop/externals/MARBL
        clean sandbox, on MARBL/trunk_tags/MARBL0_22_0
    ./components/rtm
        clean sandbox, on rtm1_0_65
    ./components/ww3
        clean sandbox, on ww3/trunk_tags/ww3_180115

Steps to reproduce the problem. Include externals description file(s) and link to public repository):

What is the changeset ID of the code, and the machine you are using:

have you modified the code? If so, it must be committed and available for testing:

Screen output or log file showing the error message and context:

Copied from original issue: ESCOMP/CESM#24

I feel we should add some more details about when you need to rerun checkout_externals. Something like:

You need to rerun checkout_externals.py whenever CESM.xml has changed (unless you have already manually updated the relevant external(s) to have the correct branch/tag checked out). Common times when this is needed are:

• After checking out a new branch/tag
• After merging some other branch/tag into your currently checked-out branch

There's debate on whether this should be done or not. But, to be consistent with a lot of our other tools I think we should drop the ".py" extension on the tool. I've argued that it should be in place in the past, since it does tell you this is a python tool. But, since this isn't a tool that we expect (or want) the user to actually make changes to -- it makes sense they shouldn't know that it's a python program. In the future if we change the underlying language from Python to something else -- do we really need the average user to know about that? I think not.

Here's some sample ranting and raving on this subject...

https://unix.stackexchange.com/questions/31760/file-extensions-for-unix-shell-scripts

Since, the script has the magic

#!/usr/bin/env python

you don't really need to know that it's a python program. The main argument I saw for adding the ".py" is when you have tons of different files and you want to be able to classify them based on the content. That doesn't apply here.

Summary of Issue:

The first time I ran the program, I saw

$ ./manage_externals/checkout_externals
Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checkout components: clm,

And it was unclear to me that the last line was a progress update (I thought it was an input prompt)

Something like

Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checking out components:
clm...

Would be much clearer, although the final result would be more screen real estate:

Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checking out components:
clm... done
mosart... done
ww3... done
cime... done
cice... done
pop... done
cism... done
rtm... done
cam... done
cime_config... done

instead of

Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checkout components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,

For consistency, it may also be worth cleaning up Checking status of components but really it's just the phrase Checkout components makes me think I need to provide a response.

Expected behavior and actual behavior:

see above

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

just run checkout_externals

What is the changeset ID of the code, and the machine you are using:

using cesm2_0_beta07.002 from ESCOMP/cesm

have you modified the code? If so, it must be committed and available for testing:

nope, this was out of the box

Screen output or log file showing the error message and context:

see above

Summary of Issue:

If you point to a git hash, sometimes checkout_externals says we're out-of-sync when really we're in-sync. This happens when the abbreviated hash printed by git branch --verbose --verbose, (e.g., (HEAD detached at cade91dfe)) has a different number of characters than the hash listed in the externals description file.

This is a problem if the full 40-character hash is listed in the externals description file, or if the number of characters used in the abbreviation in the externals description file differs from the number of characters printed by git.

Furthermore, running checkout_externals to bring things in sync does not resolve this problem: it continues to mark the given external as out-of-sync.

Expected behavior and actual behavior:

Expected behavior: If we're detached at a hash, and we're at the same hash as the one listed in the externals description file, then checkout_externals should say we're in sync.

Actual behavior: see above.

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

git clone -b cism2_1_46 [email protected]:ESCOMP/cism-wrapper.git
cd cism-wrapper
./manage_externals/checkout_externals

In the cime section of Externals.cfg, change the sha to point to the full 40-character sha: cade91dfea97ca6ac4e2448173e6f5ad4a13b90f.

Then run: ./manage_externals/checkout_externals -S -v

What is the changeset ID of the code, and the machine you are using:

659854797919defb70f5e376f9cdabadc103d0c4

Mac with git version 2.14.1

have you modified the code? If so, it must be committed and available for testing:

No

Screen output or log file showing the error message and context:

Checking status of externals: cism, source_cism, mosart, ctsm, fates, ptclm, cime,
s   ./cime
        clean sandbox, cade91dfe --> cade91dfea97ca6ac4e2448173e6f5ad4a13b90f

Summary of Issue:

If you are in detached head state at the tag given in the externals config file, then make a commit (so you are still in detached head state, but no longer at the given tag), checkout_externals -S says that you're in sync when in fact you are not.

I think the problem is that RE_DETACHED in repository_git.py (RE_DETACHED = re.compile(r'\* \((?:[\w]+[\s]+)?detached (?:at|from) ([\w\-./]+)\)')) looks at a line like this from git branch -vv:

* (HEAD detached from cime5.4.0-alpha.24) 402131faa add to readme

It takes the string after detached from as giving where you are. But it seems like the correct thing to do is:

• If it says detached at, then use the current behavior. (This seems important so that we get the tag name that you're on.)

• If it says detached from, then the string after detached from seems to be irrelevant; instead, use the commit hash after the parentheses. I did something like this in #69 , where I changed RE_DETACHED to r'\* \((?:[\w]+[\s]+)?detached (?:at|from) ([\w\-./]+)\)[\s]+([\w]+)'). But I think the correct thing to do is to actually have two separate regexes: Rename the current RE_DETACHED to RE_DETACHED_AT, and only match detached at, and then have a new RE_DETACHED_FROM, which matches detached from and has only one capture group: the second capture group from the version I had in #69 .

Note that if you actually run checkout_externals without the status flag, you end up in the correct state, since checkout_externals runs the git checkout command even if it thinks you're already in sync. But I still think it's important to fix this problem.

Expected behavior and actual behavior:

Expected behavior: status says you're out-of-sync in the above situation

Actual behavior: status says you're in-sync in the above situation

Steps to reproduce the problem (should include model description file(s) or link to public repository):

git clone [email protected]:ESCOMP/cism-wrapper.git
cd cism-wrapper
git checkout cism2_1_50
# Replace manage_externals with your desired version for testing, or keep it as is
./manage_externals/checkout_externals
cd cime
echo "some change" >> README.md
git commit -am "add to readme"
cd ..
./manage_externals/checkout_externals -S -v

What is the changeset ID of the code, and the machine you are using:

I have tried this with both:

• the head of #84 (195c1d0)

• manic-v0.8.0 (the default in the above cism tag)

Both exhibit this problem

have you modified the code? If so, it must be committed and available for testing:

No

Screen output or log file showing the error message and context:

[roo2:~/temporary/cism-wrapper]$ ./manage_externals/checkout_externals -S -v
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Checking status of externals: cism, source_cism, mosart, ctsm, fates, ptclm, cime,
    ./cime
        clean sandbox, on cime5.4.0-alpha.24
    ./components/clm
        clean sandbox, on clm4_5_18_r274
    ./components/clm/src/fates
        clean sandbox, on fates_s1.4.1_a3.0.0_rev2
    ./components/clm/tools/PTCLM
        clean sandbox, on PTCLM2_171216c
    ./components/mosart
        clean sandbox, on mosart1_0_30
    ./glimmer-cism
        clean sandbox, on cism/branch_tags/longer_var_list_tags/longer_var_list_n01_cism2_1_develop_n02

despite:

[roo2:~/temporary/cism-wrapper/cime]$ git describe
cime5.4.0-alpha.24-1-g7bac08598
[roo2:~/temporary/cism-wrapper/cime]$ git branch -vv
* (HEAD detached from cime5.4.0-alpha.24) 7bac08598 add to readme

We have a bunch of tools where "-s" means "silent", and "-v" is "verbose". "-v" is used that way here, but "-s" should be changed from "status" to "silent". That would mean "-s" would output basically nothing unless there's an error. "-v" could actually give the output that's listed as "-status" now. As it is now, "-s" actually gives more output than "-v". Having all of our tools contain the same expected: -v, -s, -h options helps with people knowing how to use them.

I created ESCOMP/CTSM#215 which I think really belongs here. If when manage_externals is run it would create a file called something like "version_info" this file could be parsed and put as meta-data into NetCDF files to describe what "version" the checkout points to. We currently use subversion keywords to do this sort of thing. The main utility of this is when you've checked out a frozen tag. So just giving what that tag is would be good. If the tag is a branch, it's obviously not as useful, but this is used when it's a tag. A far secondary thing would be for it to add some notes if the checkout is dirty. But, that's secondary, because when I rely on subversion keywords it's knowing that the sandbox is NOT dirty.

We also might need a cime change to make this automatically picked up in cime.

Summary of Issue:

When rerunning checkout_externals, if the top-level externals configuration has changed to update one of the externals, and the new version of that external itself has a changed sub-externals file (i.e., one referenced recursively), we end up in the wrong state: We end up with the old versions of the sub-externals rather than the updated versions.

Expected behavior and actual behavior:

Expected: If an external is updated, then a single run of checkout_externals should give us that updated external as well as the correct versions of all updated sub-externals.

Actual: In the first run of checkout_externals, the sub-externals aren't updated. It takes a second run to update them.

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

git clone [email protected]:ESCOMP/cesm.git
cd cesm
git checkout cesm2_0_beta09
./manage_externals/checkout_externals
sed -i '' -e 's/clm4_5_18_r274/clm5.0.dev001/' Externals.cfg
./manage_externals/checkout_externals -v

Then, notice that ptclm and fates are not up-to-date: ./manage_externals/checkout_externals -S -v

gives (among other things):

s   ./components/clm/src/fates
        clean sandbox, fates_s1.4.1_a3.0.0_rev2 --> fates_s1.4.1_a3.0.0_rev3
s   ./components/clm/tools/PTCLM
        clean sandbox, PTCLM2_171216c --> PTCLM2_180214

Running checkout_externals a second time brings things up to date

What is the changeset ID of the code, and the machine you are using:

v0.8.0 on my mac

have you modified the code? If so, it must be committed and available for testing:

no

Screen output or log file showing the error message and context:

See above

Summary of Issue:

At least for branches (and possibly for tags that have been changed - though ideally that should never happen): If there have been changes on the remote that have not yet been fetched, then checkout_externals -S can show you as being in-sync when really you're out of sync.

It seems like checkout_externals should fetch from the specified remote before showing you the status.

Expected behavior and actual behavior:

Expected behavior: If your config file points to a branch and your local version of the branch is behind the remote's, it should say you're out-of-sync, even if your local version matches the last-fetched version of the remote. i.e., it should do a fetch before determining if you're in sync.

Actual behavior: Status determines that you're in-sync based on the last time you fetched, not based on what's currently on the remote.

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

1. git clone [email protected]:ESCOMP/cesm.git and run checkout_externals

2. Make a new branch in cime, add a commit to the branch, then push it to a remote

3. Change Externals.cfg to point to your new branch

4. From another clone of cime: fetch the branch, make a commit, then push it.

5. Back in your cesm checkout: Run checkout_externals -S. It will incorrectly show that you're in sync.

6. From /path/to/cesm_checkout/cime: Fetch from the remote. Then rerun checkout_externals -S. It will now show (correctly) that you're out-of-sync.

What is the changeset ID of the code, and the machine you are using:

manic-v1.1.1

have you modified the code? If so, it must be committed and available for testing:

No

Screen output or log file showing the error message and context:

n/a

Summary of Issue:

When running checkout_externals -v, the status of sub-externals is shown incorrectly. I have determined that this problem came in with 1ae8c84 (Merge bugfix branch for stale subexternals into master).

The end result is correct, so this isn't a big deal. This just makes checkout_externals -v less useful than it otherwise would be.

Expected behavior and actual behavior:

Expected behavior (and the behavior prior to the above commit):

[roo2:~/temporary/cism-wrapper-temp2]$ ./manage_externals/checkout_externals -v
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Checking status of externals: cism, source_cism, mosart, ctsm, fates, ptclm, cime,
Checking out externals:


    ./components/mosart
        clean sandbox, on mosart1_0_30

    ./components/clm
        clean sandbox, on clm4_5_18_r274

    ./cime
        clean sandbox, on cime5.4.0-alpha.24

Checking out externals:

    ./glimmer-cism
        clean sandbox, on cism/branch_tags/longer_var_list_tags/longer_var_list_n01_cism2_1_develop_n02

Checking out externals:

    ./components/clm/src/fates
        clean sandbox, on fates_s1.4.1_a3.0.0_rev2

    ./components/clm/tools/PTCLM
        clean sandbox, on PTCLM2_171216c

Current, incorrect behavior:

[roo2:~/temporary/cism-wrapper-temp2]$ ./manage_externals/checkout_externals -v
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Checking status of externals: cism, source_cism, mosart, ctsm, fates, ptclm, cime,
Checking out externals:


    ./components/mosart
        clean sandbox, on mosart1_0_30

    ./components/clm
        clean sandbox, on clm4_5_18_r274

    ./cime
        clean sandbox, on cime5.4.0-alpha.24

Processing externals description file : Externals_CISM.cfg
Checking out externals:

---
        -,  -->
    svn switch https://svn-ccsm-models.cgd.ucar.edu/cism/branch_tags/longer_var_list_tags/longer_var_list_n01_cism2_1_develop_n02

Processing externals description file : Externals_CLM.cfg
Checking out externals:

---
        -,  -->
    git checkout fates_s1.4.1_a3.0.0_rev2

---
        -,  -->
    git checkout PTCLM2_171216c

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

git clone [email protected]:ESCOMP/cism-wrapper.git
cd cism-wrapper
git checkout cism2_1_50
rm -rf manage_externals
git clone [email protected]:NCAR/manage_externals.git
./manage_externals/checkout_externals
./manage_externals/checkout_externals -v

What is the changeset ID of the code, and the machine you are using:

See above for notes on changeset ID

have you modified the code? If so, it must be committed and available for testing:

no

Screen output or log file showing the error message and context:

see above

If I have (for example) a CESM repository that points to a git repo for CLM, and do the following:

1. From components/clm: git checkout -b mybranch

2. Make some changes in components/clm

3. From components/clm: git push -u origin mybranch

4. Change CESM.cfg to point to mybranch (modifying the repo_url and setting the branch appropriately)

5. Run checkout_externals

then I end up in detached head state in components/clm.

In fact, any time you run checkout_externals pointing to a branch, you'll end up in detached head state - though @bandre-ucar has convinced me that this is okay (and probably preferable) in the case where you weren't already on the given branch.

However, in this relatively common case where you're already on the given branch, I'd like to end up in a state where I'm still on that branch. There are issues with what to do if the branch on the remote is ahead or behind the local branch. But for starters, I think it would be very helpful to have the following special-purpose logic:

(Solution 1) If the cfg file specifies a branch, and (after fetching from the remote) the sha of the remote branch matches the currently-checked-out sha, then do nothing. (Can this be done by comparing git rev-parse HEAD with git rev-parse remotename/branchname?)

I also like @bandre-ucar 's idea:

(Solution 2) If you give repo_url as "." for a git repository, then all operations will be done on local tags and branches.

However, I think that even if we have (Solution 2), I'd still like to have (Solution 1), because this will facilitate doing cross-component development on multiple machines.

Summary of Issue: checkout_externals currently wants to update all of the externals in the sandbox, but often I want to update only some of the components, or there is a particular component I do not want to update. I think that this can be easily handled with --include and --exclude options to checkout_externals
Expected behavior and actual behavior: only those externals in the include list (or not in the exclude list) would be updated.

The SVN_EXTERNAL_DIRECTORIES indicates that clm will be brought in as a SVN external
but the CESM.cfg file indicates it's to be pulled in as git. So, for the git version, should we delete the SVN_EXTERNAL_DIRECTORIES file?

Running checkout_externals.py gives:

[aliceb@cheyenne2:cesm-demo-externals]>./checkout_externals/checkout_externals.py
Processing model description file : CESM.xml
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checkout components: clm, Processing model description file : CLM.xml
fates, mosart, cime, cice, pop, cism, cam, cime_config,

Which suggests that CLM.xml also checks out an identical set of model components.

However, when I run the svn demo version of the checkout_externals.py, I get the following
output:

./checkout_externals/checkout_externals.py
Processing model description file : CESM.xml
Checking status of components: clm,
Checkout components: clm, Processing model description file : CLM.xml
fates,

This may just require more detailed status output be added to the output from the script
instead of hidden in the checkout_externals.log.

Another point of confusion is that the components/clm contains a README_FIRST and it's own
checkout_externals.py tool that when run duplicates the CESM externals in the components/clm/components subdir:

[components/clm]/checkout_externals/checkout_externals.py
Processing model description file : CESM.xml
Processing model description file : CLM.xml
Checking status of components: cism, clm, fates, ptclm, mosart, cime, rtm,
Checkout components: cism, clm, fates, mosart, cime,

I think somehow CLM checkout_externals.py needs to know that it was checked out as part of CESM or as stand-alone.

Create a tool to generate externals description files.

Functionality

This tool is the inverse operation of checkout_externals.

• checkout_externals - attempts to make the working tree match the externals description file.

• generate_externals - attempts to make the externals description file match the working tree.

Basic workflow:

• Read an existing model description file and it's sub-externals.
• Check the working copies on the file system, updating any modified repositories or branches in the model description.
• Update any referenced sub-externals filenames to add the user specified prefix.
• Write the self consistent set of new externals files.

Command line options

• --external - base externals description file to use, default CESM.cfg
• --prefix - prefix to use for output model description files, default 'user'. Set to an empty string, '', to overwrite the existing files.
• --template - write a minimal configuration template file.

UI issues

• Overwrite existing files by default? This is safe if they are under revision control and un-modified. Otherwise there is a risk of destroying user data.

• What if a sub-external doesn't exist? For example, run generate_externals on a cesm repo where CLM has sub-externals, but CLM has not been checked out. Error? Reference existing file?

It makes sense that you can't run checkout_externals from someone else's directory. But in debugging a problem, it can be helpful to be able to run checkout_externals -S from someone else's directory. However, this isn't possible right now because it tries to write to a log file in the current directory, where you normally won't have write permission.

I'd say that the status command probably shouldn't write to a log file at all, by default. But if it does, then could there be a flag to checkout_externals to suppress writing to the log file, so that you can run it from someone else's directory?

Error for RTM checkout returned when running command:

./checkout_externals/checkout_externals.py -o

Processing model description file : CESM.xml
Processing model description file : CLM.xml
Checking status of components: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Enter passphrase for key '/home/aliceb/.ssh/id_rsa':
Enter passphrase for key '/home/aliceb/.ssh/id_rsa':
Checkout components: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cism, rtm,
ERROR: CalledProcessError in "git checkout rtm1_0_62_cimeupdate_tags_n01"

Details from the checkout_externals.log file don't include the URL of the repo
to help debug the problem:

INFO : 2017-11-09 12:25:23 : In directory: /home/aliceb/checkout_testing/cesm-demo-externals/components/rtm
execute_subprocess running command:
INFO : 2017-11-09 12:25:23 : [u'git', u'fetch', u'--all', u'--tags']
INFO : 2017-11-09 12:25:23 : git fetch --all --tags
DEBUG : 2017-11-09 12:25:28 : Fetching origin
DEBUG : 2017-11-09 12:25:28 :
INFO : 2017-11-09 12:25:28 : In directory: /home/aliceb/checkout_testing/cesm-demo-externals/components/rtm
execute_subprocess running command:
INFO : 2017-11-09 12:25:28 : [u'git', u'checkout', u'rtm1_0_62_cimeupdate_tags_n01']
INFO : 2017-11-09 12:25:28 : git checkout rtm1_0_62_cimeupdate_tags_n01
ERROR : 2017-11-09 12:25:28 : Command '[u'git', u'checkout', u'rtm1_0_62_cimeupdate_tags_n01']' returned non-zero exit status 1
ERROR : 2017-11-09 12:25:28 : Returned : 1
DEBUG : 2017-11-09 12:25:28 : error: pathspec 'rtm1_0_62_cimeupdate_tags_n01' did not match any file(s) known to git.
DEBUG : 2017-11-09 12:25:28 :
ERROR : 2017-11-09 12:25:28 : CalledProcessError in "git checkout rtm1_0_62_cimeupdate_tags_n01"
INFO : 2017-11-09 12:25:28 :
ERROR: CalledProcessError in "git checkout rtm1_0_62_cimeupdate_tags_n01"

manage_externals sporadically fails when checking out an svn external that itself has git externals. We've seen this for CISM, and yesterday I saw it for the first time with CLM (when checking out FATES).

The problem occurs when svn gives this error message at the end of checking out its external from github:

DEBUG : 2017-12-19 11:18:08 : svn: warning: W175002: REPORT request on '/NCAR/fates-release/!svn/vcc/default' failed

and then at the end of the svn checkout process:

DEBUG : 2017-12-19 11:18:08 : Checked out revision 88097.
DEBUG : 2017-12-19 11:18:08 : svn: E205011: Failure occurred processing one or more externals definitions

This leads to the following error from manage_externals:

ERROR : 2017-12-19 11:18:08 : Called process did not run successfully.
Returned status: 1
Failed command:
    svn checkout https://svn-ccsm-models.cgd.ucar.edu/clm2/trunk_tags/clm4_5_16_r253/components/clm /Users/sacks/testing/my_cesm_sandbox/components/clm
Please check the log file "manage_externals.log" for more details.
INFO : 2017-12-19 11:18:08 :
ERROR: Called process did not run successfully.
Returned status: 1
Failed command:
    svn checkout https://svn-ccsm-models.cgd.ucar.edu/clm2/trunk_tags/clm4_5_16_r253/components/clm /Users/sacks/testing/my_cesm_sandbox/components/clm
Please check the log file "manage_externals.log" for more details.

Since both CLM and CISM are moving to pure git, I expect this problem to go away for them. But this could lead to problems if we have other svn externals that themselves pull in externals from github. @mnlevy1981 It looks like this is an issue for POP (which pulls in cvmix from github), but maybe nothing else?

I feel like README_FIRST should start with a "Quick start" section or something like that, which simply says something like:

To get a working version of the model, after cloning this repository, run the following from the top-level of the clone:

./checkout_cesm/checkout_externals.py

(Then you can go into the details below.)

As @bertinia pointed out (in relation to #10), the CLM version of this file should perhaps add something saying that this is only needed if you cloned the clm repo directly – you do not need to run this if you have a copy of CLM inside CESM, CAM, etc.

This issue needs to be documented for developers checking out private repos;

On hobart, I'm getting this error:

./checkout_externals/checkout_externals.py --model CESM.cfg
Processing model description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
The authenticity of host 'github.com (192.30.253.113)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Checkout components: clm,
ERROR: CalledProcessError in "git clone [email protected]:NCAR/clm-demo-externals.git clm"

Similar error when specifying --model CESM.json:

./checkout_externals/checkout_externals.py --model CESM.json
Processing model description file : CESM.json
Checking status of components: clm, mosart, cime, ww3, cice, pop, cism, rtm, cam, cime_config,
Checkout components: clm,
ERROR: CalledProcessError in "git clone [email protected]:NCAR/clm-demo-externals.git clm"

Is there a reason to use the ssh clone command as opposed to the https?
Yes, for private repos, access via ssh is required.

When reviewing the README_FIRST, it points to:

https://help.github.com/articles/connecting-to-github-with-ssh/

and I also tried running the tool:

http://tools.cgd.ucar.edu/make_user_ssh_keys/index.html

on hobart using:

./mk_user_ssh_keys.sh -t github.com -u bertinia -v

But I'm getting permission denied errors from the target cause I think the only
way to add an ssh key on github is via the web interface settings in my profile.

I added the hobart ssh key to my github profile via the web settings and was able to
get the download to work correctly.

We will need more explanation of how to setup the ssh keys in the README_FIRST
if we use the ssh clone link.

According to @goldy2718, we will use https protocol for public released versions of the model.
Developers will still need to add ssh keys to their profiles when downloading private repos.

From @cacraigucar on November 20, 2017 21:2

Going from a git or svn checkout where the screen is inundated with every file checked out, to manage_externals where it silently sits there for minutes can be disconcerting as you are not sure if the command is making progress or not. A simple "starting to check out XXX", "completed check out of XXX" for each of the components would be useful so the user can make sure progress is actually being made. I ended up opening a new window and seeing that something was happening just to make sure progress was being made and it wasn't simply hung.

Copied from original issue: ESCOMP/CESM#4

We need comments in the CESM.cfg and CLM.cfg explaining what they are, what they do, and how they are used.

Summary of Issue:

manage_externals will return an error code even for "--status" if there is a missing Externals file or some other problem running manage_externals.

Expected behavior and actual behavior:

Error code is returned for "--status" if an Externals file is missing. I expect --status to not return an error code.

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

1179 23:15 cd ctsm/
1180 23:15 rm Externals.cfg
1181 23:15 ./manage_externals/checkout_externals --status
1182 23:15 echo $?

What is the changeset ID of the code, and the machine you are using:

manicv1.0.2
81ccf92e8c4da272121fcc96902021be9e7a7b1f

cheyenne

have you modified the code? If so, it must be committed and available for testing:

no

Screen output or log file showing the error message and context:

cheyenne4 erik/ctsm> ./manage_externals/checkout_externals --status
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CISM.cfg

ERROR: External externals description file "Externals_CLM.cfg" does not exist! In directory: /glade/p/work/erik/ctsm
cheyenne4 erik/ctsm> echo $?
1

With the changes in #63 , my preference would be to have the current "verbose" output be the default - definitely for checkout, and probably for status, too. I could imagine having a --quiet flag that reverts to the current default and a --verbose flag that gives the current --verbose --verbose behavior.

We can implement this for v1.0 if there's time, or wait and change this post-v1.0 after some other people have enough experience that they weigh in one way or the other.

Currently, unit testing of repository_git is done by having the calls to git commands extracted into their own functions, and then having unit tests set these methods to test-specific methods on a per-test basis, as in:

        self._repo._git_remote_verbose = self._git_remote_origin_upstream

where _git_remote_origin_upstream is a function defined in the unit test class.

I think it would be easier to write, understand and maintain the unit tests if we introduced a class whose purpose is to wrap various git commands. We would move all of the git static methods from repository_git.py into this new class. By default, a GitRepository object would be set up to hold an instance of the real GitWrapper class (this could be set up in the __init__ method of GitRepository. However, for unit testing, we could have a separate GitWrapperFake class, and we could set

self._git_wrapper = GitWrapperFake()
self._repo.set_git_wrapper(self._git_wrapper)

This GitWrapperFake would have its own fake versions of all of the git commands, which return some specified value. They would all be set up with some default value (in case one isn't specified explicitly in a unit test), but they could all be configured. So we could have, for example, in a given test:

self._git_wrapper.set_git_remote_verbose("return value from git remote verbose")

By having the actual return strings in the given test, rather than needing to chase down the functions being used, I think the tests will be more maintainable.

Rename the repository to reflect future as a suite of tools to manage externals.

Rename the local for subtrees of this repository to match repo name.

Summary of Issue:

Expected behavior and actual behavior:

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

What is the changeset ID of the code, and the machine you are using:

have you modified the code? If so, it must be committed and available for testing:

Screen output or log file showing the error message and context:

Summary of Issue:

My current workflow for CESM development is to set up a CESM sandbox where components/pop is a softlink to a local SVN checkout of POP (and in that checkout, POP's externals/marbl/ directory is another softlink to a git checkout of MARBL). I could use manage_externals to checkout my POP branch directly, but I would still need to update the MARBL checkout manually... so it would be nice to have the option of doing something like

[pop]
tag = none
protocol = softlink
repo_url = /path/to/pop
local_path = components/pop
required = True

Expected behavior and actual behavior:

Expected behavior would be a directory structure where components/pop/ is a softlink to /path/to/pop, but instead I get

Processing externals description file : CESM.cfg

ERROR: Unknown repository protocol "softlink" in "pop".

Steps to reproduce the problem (should include model description file(s) or link to public repository):

In ESCOMP/cesm:

$ git diff
 [pop]
 tag = trunk_tags/cesm_pop_2_1_20171008
-protocol = svn
+protocol = softlink
 repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2
 local_path = components/pop
 required = True

What is the changeset ID of the code, and the machine you are using:

Tested on MARBL, using cesm2_0_beta07.002 tag of ESCOMP/cesm

have you modified the code? If so, it must be committed and available for testing:

N/A

Screen output or log file showing the error message and context:

see above

Summary of Issue:

I have a branch of POP where we pointed to an SVN tag of MARBL; we have decided to point to a git tag instead

Expected behavior and actual behavior:

I expected to be able to change protocol from svn to git in my externals file, but instead I get the error included below. I ended up running rm -rf and then checkout_externals but it seems like tool should be able to handle this case.

Steps to reproduce the problem (should include model description file(s) or link to public repository):

1. Mirror a git repository in subversion (or vice versa)
2. Generate an external pointing to subversion and run checkout_externals
3. Replace subversion protocol with git and try to rerun checkout_externals

What is the changeset ID of the code, and the machine you are using:

This is on hobart, using manic-v0.8.0

have you modified the code? If so, it must be committed and available for testing:

Nope, this is clean code.

Screen output or log file showing the error message and context:

$ ./externals/manage_externals/checkout_externals
Processing externals description file : Externals.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: pop_externals, cvmix, marbl,
    ./externals/CVMix
?   ./externals/MARBL
----------------------------------------------------------------------
The external repositories labeled with 'M' above are not in a clean state.

The following are two options for how to proceed:

(1) Go into each external that is not in a clean state and issue either
    an 'svn status' or a 'git status' command. Either revert or commit
    your changes so that all externals are in a clean state. (Note,
    though, that it is okay to have untracked files in your working
    directory.) Then rerun checkout_externals.

(2) Alternatively, you do not have to rely on checkout_externals. Instead, you
    can manually update out-of-sync externals (labeled with 's' above)
    as described in the configuration file Externals.cfg.

----------------------------------------------------------------------

Summary of Issue:

When untracked files are present in one of the externals managed by manage_externals, checkout_externals refuses to proceed since that external is in a "modified" state. I think it's important that it proceed in this case, because it will be very common for users to have untracked files in their directories.

Expected behavior and actual behavior:

Expected: checkout_externals proceeds despite there being untracked files in one or more external directories.

Actual: checkout_externals bails with a message like this:

Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
 M  ./cime
    ./cime_config
    ./components/cam
    ./components/cice
    ./components/cism
    ./components/clm
    ./components/mosart
    ./components/pop
    ./components/rtm
    ./components/ww3
----------------------------------------------------------------------
Some external repositories that are not in a clean state. Please
ensure all external repositories are clean before updating.
----------------------------------------------------------------------

Steps to reproduce the problem (should include model description file(s) or link to publi c repository):

1. Clone the cesm repo (git clone [email protected]:ESCOMP/cesm.git)

2. Run ./manage_externals/checkout_externals

3. Add an untracked file in one of the externals directories. In my case, I did a workflow that will be very common: creating a case in cime/scripts: ./create_newcase --case mytest --compset A --res f45_g37_rx1

4. Rerun ./manage_externals/checkout_externals

What is the changeset ID of the code, and the machine you are using:

code: cesm repo at a9702c9

Machine: mac (roo2)

have you modified the code? If so, it must be committed and available for testing:

No

Screen output or log file showing the error message and context:

See above

Summary of Issue:

The command line option --model, -m, is a relic from when this was managing the 'model'. We are now being more explicit about managing 'externals'. The command line options should be updated to --externals, -e, to reflect this.

Summary of Issue:

Having tag = in a component block using protocol = svn is a little misleading; first of all, svn "tags" are really just branches combined with a web-hook to make them read-only. Also, it's not really clear where repo_url ends and tag begins. As far as I can tell, all of the following are the same

[pop]
tag = trunk_tags/cesm_pop_2_1_20171008
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2
local_path = components/pop
required = True

[pop]
tag = pop2/trunk_tags/cesm_pop_2_1_20171008
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu
local_path = components/pop
required = True

[pop]
tag = cesm_pop_2_1_20171008
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2/trunk_tags
local_path = components/pop
required = True

Expected behavior and actual behavior:

What I'd rather see is the full URL in repo_url and no tag field:

[pop]
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2/trunk_tags/cesm_pop_2_1_20171008
local_path = components/pop
required = True

Instead of the required field being a bool , what if it were a list of one or more groups so
for example cvmix would have:

required = pop, mom

while fms might have
required = mom, camfv3

then the -o argument to checkout_externals would expect a list of one or more optional components.

The tool seems to assume that the remote you want to work with is named ‘origin’. If I have something like this:

<repo_url>[email protected]:billsacks/clm-demo-externals.git</repo_url>
<branch>mybranch</branch>

and I have done:

git remote add billsacks [email protected]:billsacks/clm-demo-externals.git

I get:

ERROR: Invalid repository in /Users/sacks/cesm_code/cesm-demo-externals/components/clm, url = [email protected]:NCAR/clm-demo-externals.git, should be [email protected]:billsacks/clm-demo-externals.git

I have also encountered this error after merging in a branch that points to an external from a different remote.

My understanding (maybe wrong) is that the name of remotes in git (e.g., “origin”, or “billsacks” as above) is just a convenience, and that anywhere where you could have a short remote name like this it’s also valid to list the full remote URL (like [email protected]:billsacks/clm-demo-externals.git). If that’s right, then I’m thinking that there’s no need for checkout_model.py to assume anything regarding remote alias names (“origin” or anything else): any time it needs to interact with the remote, it can just give the full path to the remote.

Since I'm paranoid, I started worrying about the possibility that the wrong commit will be checked out when checkout_externals tries to checkout a tag. While this doesn't seem like a big risk, it seems possible - especially after someone has been doing some work locally and then reruns checkout_externals after the .cfg file has been updated.

There are two scenarios that I'm worried about:

(1) There happens to be a branch named the same as the given tag

If the reference is ambiguous in this way, git will checkout the branch rather than the tag. This one seems easy to fix: rather than just git checkout TAGNAME, we should do git checkout tags/TAGNAME (tested with git 2.14.1).

(2) This tag is defined differently in different places - i.e., the same tag name points to a different sha in different places.

While this one seems unlikely (I hope), it could cause massive confusion if it did happen.

(2a) Multiple remotes define this tag differently

I believe that the current behavior - git fetch --all --tags will give indeterminate behavior in this case.

To address this, I think we should just do a fetch from the single remote listed in the .cfg file rather than doing a fetch from --all.

(2b) There is already a local tag with this name but pointing to the wrong sha

All of my testing with git 2.14.1 suggests that local tags are overwritten with the tag from the remote when doing a fetch, but some answers on StackOverflow suggest that this may be version-dependent (see the second answer, by torek, here: https://stackoverflow.com/questions/35979642/how-to-checkout-remote-git-tag (especially the section on Tags), and see the comments by torek in the answer by eckes here https://stackoverflow.com/questions/9662249/how-to-overwrite-local-tags-with-git-fetch/21475727 ).

I can see a couple of ways to address this.

We could use the method suggested by torek here https://stackoverflow.com/questions/9662249/how-to-overwrite-local-tags-with-git-fetch/21475727

git fetch <remote> '+refs/tags/*:refs/tags/*'

Though maybe just specifying the single tag of interest rather than *. (I could imagine first fetching all tags from the given remote, because it's useful to update all tags, and then doing something like git fetch <remote> '+refs/tags/TAGNAME:refs/tags/TAGNAME'.)

(NOTE: I haven't tested this method myself.)

Alternatively, it seems we could first delete any local tag with the given name (git tag -d TAGNAME, ignoring errors in case that tag doesn't exist yet locally), then do a fetch. I slightly prefer this approach simply because it's easier to understand.

Summary of Issue:

Add python 3.4 and 3.5 to CI testing.

Create additional output verbosity levels:

Default output:

./manage_externals/checkout_externals 
Processing externals description file : CESM.cfg
Processing externals description file : CLM.cfg
Checking status of externals: cism, clm, fates, ptclm, mosart, cime, rtm, 
Checking out externals: cism, clm, mosart, cime, 
Checking out externals: fates, 

./manage_externals/checkout_externals --status
Processing externals description file : CESM.cfg
Processing externals description file : CLM.cfg
Checking status of externals: cism, clm, fates, ptclm, mosart, cime, rtm, 
m   ./cime
    ./components/cism
    ./components/mosart
e-o ./components/rtm
    ./src/fates
e-o ./tools/PTCLM

Verbose output:

./manage_externals/checkout_externals --verbose
Processing externals description file : CESM.cfg
Processing externals description file : CLM.cfg
Checking status of externals: 
    ./cime:
        Required: https://github.com/CESM-Development/cime.git - mvertens/updates_for_clm.tag2
        Working: https://github.com/CESM-Development/cime.git - cime5.3.0-alpha.35
    ./components/cism
        Synced: https://svn-ccsm-models.cgd.ucar.edu/glc/trunk_tags/cism2_1_37
    ./components/mosart
        Synced:  https://svn-ccsm-models.cgd.ucar.edu/mosart/trunk_tags/mosart1_0_26
    ./components/rtm
        Optional: https://github.com/escom/rtm/ - rtm1_0_23
        Working: empty
    ./src/fates
        Synced: https://github.com/NCAR/fates-release - fates_s1.0.0_a1.0.0
    ./tools/PTCLM
        Optional: https://github.com/escom/ptclm - ptclm20171212
        Working: empty

Checking out externals:
    ./cime:
        https://github.com/CESM-Development/cime.git - mvertens/updates_for_clm.tag2
    ./components/cism
        https://svn-ccsm-models.cgd.ucar.edu/glc/trunk_tags/cism2_1_37
    ./components/mosart
        https://svn-ccsm-models.cgd.ucar.edu/mosart/trunk_tags/mosart1_0_26
    ./components/rtm
        Optional - empty
    ./src/fates
        https://github.com/NCAR/fates-release - fates_s1.0.0_a1.0.0
    ./tools/PTCLM
        Optional - Empty
Checking out externals: 
    ./src/fates
        https://github.com/NCAR/fates-release - fates_s1.0.0_a1.0.0

./manage_externals/checkout_externals --status --verbose
Processing externals description file : CESM.cfg
Processing externals description file : CLM.cfg
Checking status of externals:
    ./cime:
        Required: https://github.com/CESM-Development/cime.git - mvertens/updates_for_clm.tag2
        Working: https://github.com/CESM-Development/cime.git - cime5.3.0-alpha.35
    ./components/cism
        Synced: https://svn-ccsm-models.cgd.ucar.edu/glc/trunk_tags/cism2_1_37
    ./components/mosart
        Synced:  https://svn-ccsm-models.cgd.ucar.edu/mosart/trunk_tags/mosart1_0_26
    ./components/rtm
        Optional: https://github.com/escom/rtm/ - rtm1_0_23
        Working: empty
    ./src/fates
        Synced: https://github.com/NCAR/fates-release - fates_s1.0.0_a1.0.0
    ./tools/PTCLM
        Optional: https://github.com/escom/ptclm - ptclm20171212
        Working: empty

m   ./cime
    ./components/cism
    ./components/mosart
e-o ./components/rtm
    ./src/fates
e-o ./tools/PTCLM

Very verbose output

./manage_externals/checkout_externals --verbose --verbose

• checkout - no change from verbose. Default svn output hides errors that the user needs to see.
• status - former verbose output.

From an earlier conversation, I thought we were going to rename this to remove the .py, in case we ever want to rewrite it in ruby or haskell or ...