esmci / manage_externals Goto Github PK
View Code? Open in Web Editor NEWcesm externals management utility
License: Other
cesm externals management utility
License: Other
Hangs doing a checkout when subversion password is missing.
Tried doing a checkout on a new system that I've never used subversion on before. It hangs when trying
to checkout ww3. Should be prompted to enter your subversion username and password. I was able to get around this by doing a manual checkout of ww3 and entering my password and saving it. Since my password is now saved, the checkout works.
Remove .subversion from your home directory.
Release manic-v0.7.0
No
login1.stampede2(79)$ ./manage_externals/checkout_externals
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam,
Checking out externals: clm, mosart, ww3,
checkout_externals changes directories numerous times while being run. This makes it tricky when using local paths as the url to refer to git repo on the same filesystem. Depending on what repo you are cloning, cime, clm, standalon sub-external, e.g. fates, it can require one, two or three parent levels. This is trick to get correct and confusing.
To simplify the user experience, should check for and require absolute paths.
I'm opening this enhancement request on behalf of @cecilehannay , but I see her point, too.
While I like the default behavior of checkout_externals in terms of the level of verbosity, there are times when some extra verbosity can be reassuring, especially when you have changed the CESM.cfg file and are rerunning checkout_externals, and want to see that it's actually doing something.
Our thought is that, when you add the -v
flag, you should see all of the output from the underlying git checkout
or svn switch
commands, rather than having that output be suppressed.
From @cacraigucar on November 20, 2017 21:13
While the help message highlights that checkout_externals.py must be run in the root of the source tree, the error message indicates a different error. The error message is about "not being able to find the model description file". It should probably also suggest: "Is this command being run from the root of the source tree?"
Copied from original issue: ESCOMP/CESM#5
If I update the CESM.cfg file for an external to point to a new tag then run checkout_externals
, I sometimes get a failure because the given tag hasn't been fetched. This can occur if there are no branches on the remote that contain the given tag (e.g., when I push a tag to the remote but haven't pushed the branch that tag is on).
Expected: should fetch all tags
Actual: only fetches tags contained in branches on the remote
Fix incoming
I'm not sure where is the best location for this (top of README_FIRST, perhaps?), but I was thinking that it would be good to clearly call out the location of the manage_externals repository: Since this is included as a subtree in various places, it would help if people knew where to look to find the source of the code. This is useful for issues, and also in case anyone is interested in including this in their own package.
From @jedwards4b on February 20, 2018 20:46
an external python program, the current structure of checkout.py does not lend itself well to this desire.
Copied from original issue: ESCOMP/CESM#25
I'm opening this enhancement request on behalf of @cecilehannay , but I see her point, too.
It would be good to have a way to see what's currently checked out. This is similar to #14 , and if that were done, this one wouldn't be absolutely necessary. But it would still be helpful if the status command or some other command - or perhaps the status command with the -v option - showed the current branch/tag that is checked out.
The use case is: If you're not sure whether you updated all externals correctly (e.g., "Shoot, did I remember to run checkout_externals before I ran that case?"), it would be helpful to see at a glance what you have checked out.
When "-o" isn't given that status of optional externals isn't reported on. I'd like to see a warning given that mentions these weren't checked out (at least for normal level of output logging).
I'd also like to see a README placed in the optional directory, that says (this directory wasn't checked out, please rerun checkout_externals at the top level with the "-o" option to make this happen if you need this directory).
I think the behavior should also be that if the directory doesn't exist it does the above creating the directory and adding a README file. But, if the directory does exist, it should ignore it (as presumably you ran checkout_externals with a "-o" option previously. But, it should normally print a warning if this is happening (for example "leaving the existing optional external directory tools/PTCLM since it already exists and you didn't specify the "-o" option).
Even if manage_externals is used exclusively in CESM and its components, having a default name of CESM.cfg doesn't always make sense. For example, @mnlevy1981 is planning to use this in place of svn externals in POP to pull in POP's dependencies (cvmix, marbl). I assume he's going to want to be able to run checkout_externals from a POP checkout to get these. Currently this would require a top-level CESM.cfg file, which doesn't really make sense because he wouldn't be pulling in other CESM components.
What about something generic like EXTERNALS.cfg?
Document manage_externals depenecies:
I find the 'e-o' status output confusing: When I see any letters in a status output, intuitively I think it indicates that the state is "dirty" or that there's some other problem. This was particularly an issue for me when my repo got into a dirty state, so checkout_externals refused to update, and printed out the status: It was hard for me to see what was wrong at first, because my eye was drawn to the 'e-o' lines (where I originally assumed 'e' meant 'error').
Personally, I'd be happy for any optional empty directory to be excluded from the output: i.e., any time it would print 'e-o ...' it instead just leaves that line out (perhaps unless you specify some separate flag, or an optional string following the '-s' flag, that forces printing of these empty optional repositories).
A compromise, though, if others see value in having this printed, would be: If any output is 'e-o' (or maybe just do this always for simplicity), include a line in the output like:
A status of 'e-o' just indicates an optional repository that is currently empty; this is not a problem.
From @jedwards4b on February 20, 2018 19:48
git status
On branch pio2_cleanup
Your branch and 'mydev/pio2_cleanup' have diverged,
and have 62 and 2 different commits each, respectively.
(use "git pull" to merge the remote branch into yours)
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: src/main/ncdio_pio.F90.in
no changes added to commit (use "git add" and/or "git commit -a")
Current behavior is:
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm,
ERROR: DEV_ERROR: regex to detect tracking branch failed.
If I changes the RE_TRACKING to be RE_TRACKING = re.compile(r'[(.+)]')
then I get what I think that I should expect:
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam,
sM ./cime
modified sandbox, cesmdev/master: ahead 136 --> cime5.4.0-alpha.23
M ./components/cam
modified sandbox, on cam1/trunk_tags/cam5_4_166/components/cam
s ./components/cice
clean sandbox, origin/master --> cice5_20180123
./components/cism
clean sandbox, on cism2_1_46
./components/cism/glimmer-cism
clean sandbox, on test_coupling_move_up_v2_n01
sM ./components/clm
modified sandbox, mydev/pio2_cleanup: ahead 62, behind 2 --> clm5.0.dev001
./components/clm/src/fates
clean sandbox, on fates_s1.4.1_a3.0.0_rev3
s ./components/clm/tools/PTCLM
clean sandbox, PTCLM2_171216c --> PTCLM2_180214
./components/mosart
clean sandbox, on mosart1_0_30
./components/pop
clean sandbox, on pop2/trunk_tags/cesm_pop_2_1_20180205
./components/pop/externals/CVMix
clean sandbox, on v0.72-conv_bugfix.002
./components/pop/externals/MARBL
clean sandbox, on MARBL/trunk_tags/MARBL0_22_0
./components/rtm
clean sandbox, on rtm1_0_65
./components/ww3
clean sandbox, on ww3/trunk_tags/ww3_180115
Copied from original issue: ESCOMP/CESM#24
I feel we should add some more details about when you need to rerun checkout_externals. Something like:
You need to rerun checkout_externals.py whenever CESM.xml has changed (unless you have already manually updated the relevant external(s) to have the correct branch/tag checked out). Common times when this is needed are:
There's debate on whether this should be done or not. But, to be consistent with a lot of our other tools I think we should drop the ".py" extension on the tool. I've argued that it should be in place in the past, since it does tell you this is a python tool. But, since this isn't a tool that we expect (or want) the user to actually make changes to -- it makes sense they shouldn't know that it's a python program. In the future if we change the underlying language from Python to something else -- do we really need the average user to know about that? I think not.
Here's some sample ranting and raving on this subject...
https://unix.stackexchange.com/questions/31760/file-extensions-for-unix-shell-scripts
Since, the script has the magic
#!/usr/bin/env python
you don't really need to know that it's a python program. The main argument I saw for adding the ".py" is when you have tons of different files and you want to be able to classify them based on the content. That doesn't apply here.
The first time I ran the program, I saw
$ ./manage_externals/checkout_externals
Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checkout components: clm,
And it was unclear to me that the last line was a progress update (I thought it was an input prompt)
Something like
Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checking out components:
clm...
Would be much clearer, although the final result would be more screen real estate:
Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checking out components:
clm... done
mosart... done
ww3... done
cime... done
cice... done
pop... done
cism... done
rtm... done
cam... done
cime_config... done
instead of
Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checkout components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
For consistency, it may also be worth cleaning up Checking status of components
but really it's just the phrase Checkout components
makes me think I need to provide a response.
see above
just run checkout_externals
using cesm2_0_beta07.002
from ESCOMP/cesm
nope, this was out of the box
see above
If you point to a git hash, sometimes checkout_externals
says we're out-of-sync when really we're in-sync. This happens when the abbreviated hash printed by git branch --verbose --verbose
, (e.g., (HEAD detached at cade91dfe)
) has a different number of characters than the hash listed in the externals description file.
This is a problem if the full 40-character hash is listed in the externals description file, or if the number of characters used in the abbreviation in the externals description file differs from the number of characters printed by git.
Furthermore, running checkout_externals
to bring things in sync does not resolve this problem: it continues to mark the given external as out-of-sync.
Expected behavior: If we're detached at a hash, and we're at the same hash as the one listed in the externals description file, then checkout_externals should say we're in sync.
Actual behavior: see above.
git clone -b cism2_1_46 [email protected]:ESCOMP/cism-wrapper.git
cd cism-wrapper
./manage_externals/checkout_externals
In the cime
section of Externals.cfg
, change the sha to point to the full 40-character sha: cade91dfea97ca6ac4e2448173e6f5ad4a13b90f
.
Then run: ./manage_externals/checkout_externals -S -v
659854797919defb70f5e376f9cdabadc103d0c4
Mac with git version 2.14.1
No
Checking status of externals: cism, source_cism, mosart, ctsm, fates, ptclm, cime,
s ./cime
clean sandbox, cade91dfe --> cade91dfea97ca6ac4e2448173e6f5ad4a13b90f
If you are in detached head state at the tag given in the externals config file, then make a commit (so you are still in detached head state, but no longer at the given tag), checkout_externals -S
says that you're in sync when in fact you are not.
I think the problem is that RE_DETACHED
in repository_git.py
(RE_DETACHED = re.compile(r'\* \((?:[\w]+[\s]+)?detached (?:at|from) ([\w\-./]+)\)')
) looks at a line like this from git branch -vv
:
* (HEAD detached from cime5.4.0-alpha.24) 402131faa add to readme
It takes the string after detached from
as giving where you are. But it seems like the correct thing to do is:
If it says detached at
, then use the current behavior. (This seems important so that we get the tag name that you're on.)
If it says detached from
, then the string after detached from
seems to be irrelevant; instead, use the commit hash after the parentheses. I did something like this in #69 , where I changed RE_DETACHED to r'\* \((?:[\w]+[\s]+)?detached (?:at|from) ([\w\-./]+)\)[\s]+([\w]+)')
. But I think the correct thing to do is to actually have two separate regexes: Rename the current RE_DETACHED
to RE_DETACHED_AT
, and only match detached at
, and then have a new RE_DETACHED_FROM
, which matches detached from
and has only one capture group: the second capture group from the version I had in #69 .
Note that if you actually run checkout_externals
without the status flag, you end up in the correct state, since checkout_externals
runs the git checkout
command even if it thinks you're already in sync. But I still think it's important to fix this problem.
Expected behavior: status says you're out-of-sync in the above situation
Actual behavior: status says you're in-sync in the above situation
git clone [email protected]:ESCOMP/cism-wrapper.git
cd cism-wrapper
git checkout cism2_1_50
# Replace manage_externals with your desired version for testing, or keep it as is
./manage_externals/checkout_externals
cd cime
echo "some change" >> README.md
git commit -am "add to readme"
cd ..
./manage_externals/checkout_externals -S -v
I have tried this with both:
Both exhibit this problem
No
[roo2:~/temporary/cism-wrapper]$ ./manage_externals/checkout_externals -S -v
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Checking status of externals: cism, source_cism, mosart, ctsm, fates, ptclm, cime,
./cime
clean sandbox, on cime5.4.0-alpha.24
./components/clm
clean sandbox, on clm4_5_18_r274
./components/clm/src/fates
clean sandbox, on fates_s1.4.1_a3.0.0_rev2
./components/clm/tools/PTCLM
clean sandbox, on PTCLM2_171216c
./components/mosart
clean sandbox, on mosart1_0_30
./glimmer-cism
clean sandbox, on cism/branch_tags/longer_var_list_tags/longer_var_list_n01_cism2_1_develop_n02
despite:
[roo2:~/temporary/cism-wrapper/cime]$ git describe
cime5.4.0-alpha.24-1-g7bac08598
[roo2:~/temporary/cism-wrapper/cime]$ git branch -vv
* (HEAD detached from cime5.4.0-alpha.24) 7bac08598 add to readme
We have a bunch of tools where "-s" means "silent", and "-v" is "verbose". "-v" is used that way here, but "-s" should be changed from "status" to "silent". That would mean "-s" would output basically nothing unless there's an error. "-v" could actually give the output that's listed as "-status" now. As it is now, "-s" actually gives more output than "-v". Having all of our tools contain the same expected: -v, -s, -h options helps with people knowing how to use them.
I created ESCOMP/CTSM#215 which I think really belongs here. If when manage_externals is run it would create a file called something like "version_info" this file could be parsed and put as meta-data into NetCDF files to describe what "version" the checkout points to. We currently use subversion keywords to do this sort of thing. The main utility of this is when you've checked out a frozen tag. So just giving what that tag is would be good. If the tag is a branch, it's obviously not as useful, but this is used when it's a tag. A far secondary thing would be for it to add some notes if the checkout is dirty. But, that's secondary, because when I rely on subversion keywords it's knowing that the sandbox is NOT dirty.
We also might need a cime change to make this automatically picked up in cime.
When rerunning checkout_externals
, if the top-level externals configuration has changed to update one of the externals, and the new version of that external itself has a changed sub-externals file (i.e., one referenced recursively), we end up in the wrong state: We end up with the old versions of the sub-externals rather than the updated versions.
Expected: If an external is updated, then a single run of checkout_externals
should give us that updated external as well as the correct versions of all updated sub-externals.
Actual: In the first run of checkout_externals
, the sub-externals aren't updated. It takes a second run to update them.
git clone [email protected]:ESCOMP/cesm.git
cd cesm
git checkout cesm2_0_beta09
./manage_externals/checkout_externals
sed -i '' -e 's/clm4_5_18_r274/clm5.0.dev001/' Externals.cfg
./manage_externals/checkout_externals -v
Then, notice that ptclm and fates are not up-to-date: ./manage_externals/checkout_externals -S -v
gives (among other things):
s ./components/clm/src/fates
clean sandbox, fates_s1.4.1_a3.0.0_rev2 --> fates_s1.4.1_a3.0.0_rev3
s ./components/clm/tools/PTCLM
clean sandbox, PTCLM2_171216c --> PTCLM2_180214
Running checkout_externals a second time brings things up to date
v0.8.0 on my mac
no
See above
At least for branches (and possibly for tags that have been changed - though ideally that should never happen): If there have been changes on the remote that have not yet been fetched, then checkout_externals -S
can show you as being in-sync when really you're out of sync.
It seems like checkout_externals should fetch from the specified remote before showing you the status.
Expected behavior: If your config file points to a branch and your local version of the branch is behind the remote's, it should say you're out-of-sync, even if your local version matches the last-fetched version of the remote. i.e., it should do a fetch before determining if you're in sync.
Actual behavior: Status determines that you're in-sync based on the last time you fetched, not based on what's currently on the remote.
git clone [email protected]:ESCOMP/cesm.git
and run checkout_externals
Make a new branch in cime, add a commit to the branch, then push it to a remote
Change Externals.cfg
to point to your new branch
From another clone of cime: fetch the branch, make a commit, then push it.
Back in your cesm checkout: Run checkout_externals -S
. It will incorrectly show that you're in sync.
From /path/to/cesm_checkout/cime
: Fetch from the remote. Then rerun checkout_externals -S
. It will now show (correctly) that you're out-of-sync.
manic-v1.1.1
No
n/a
When running checkout_externals -v
, the status of sub-externals is shown incorrectly. I have determined that this problem came in with 1ae8c84 (Merge bugfix branch for stale subexternals into master
).
The end result is correct, so this isn't a big deal. This just makes checkout_externals -v
less useful than it otherwise would be.
Expected behavior (and the behavior prior to the above commit):
[roo2:~/temporary/cism-wrapper-temp2]$ ./manage_externals/checkout_externals -v
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Checking status of externals: cism, source_cism, mosart, ctsm, fates, ptclm, cime,
Checking out externals:
./components/mosart
clean sandbox, on mosart1_0_30
./components/clm
clean sandbox, on clm4_5_18_r274
./cime
clean sandbox, on cime5.4.0-alpha.24
Checking out externals:
./glimmer-cism
clean sandbox, on cism/branch_tags/longer_var_list_tags/longer_var_list_n01_cism2_1_develop_n02
Checking out externals:
./components/clm/src/fates
clean sandbox, on fates_s1.4.1_a3.0.0_rev2
./components/clm/tools/PTCLM
clean sandbox, on PTCLM2_171216c
Current, incorrect behavior:
[roo2:~/temporary/cism-wrapper-temp2]$ ./manage_externals/checkout_externals -v
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Checking status of externals: cism, source_cism, mosart, ctsm, fates, ptclm, cime,
Checking out externals:
./components/mosart
clean sandbox, on mosart1_0_30
./components/clm
clean sandbox, on clm4_5_18_r274
./cime
clean sandbox, on cime5.4.0-alpha.24
Processing externals description file : Externals_CISM.cfg
Checking out externals:
---
-, -->
svn switch https://svn-ccsm-models.cgd.ucar.edu/cism/branch_tags/longer_var_list_tags/longer_var_list_n01_cism2_1_develop_n02
Processing externals description file : Externals_CLM.cfg
Checking out externals:
---
-, -->
git checkout fates_s1.4.1_a3.0.0_rev2
---
-, -->
git checkout PTCLM2_171216c
git clone [email protected]:ESCOMP/cism-wrapper.git
cd cism-wrapper
git checkout cism2_1_50
rm -rf manage_externals
git clone [email protected]:NCAR/manage_externals.git
./manage_externals/checkout_externals
./manage_externals/checkout_externals -v
See above for notes on changeset ID
no
see above
If I have (for example) a CESM repository that points to a git repo for CLM, and do the following:
From components/clm: git checkout -b mybranch
Make some changes in components/clm
From components/clm: git push -u origin mybranch
Change CESM.cfg to point to mybranch (modifying the repo_url and setting the branch appropriately)
Run checkout_externals
then I end up in detached head state in components/clm.
In fact, any time you run checkout_externals pointing to a branch, you'll end up in detached head state - though @bandre-ucar has convinced me that this is okay (and probably preferable) in the case where you weren't already on the given branch.
However, in this relatively common case where you're already on the given branch, I'd like to end up in a state where I'm still on that branch. There are issues with what to do if the branch on the remote is ahead or behind the local branch. But for starters, I think it would be very helpful to have the following special-purpose logic:
(Solution 1) If the cfg file specifies a branch, and (after fetching from the remote) the sha of the remote branch matches the currently-checked-out sha, then do nothing. (Can this be done by comparing git rev-parse HEAD
with git rev-parse remotename/branchname
?)
I also like @bandre-ucar 's idea:
(Solution 2) If you give repo_url as "." for a git repository, then all operations will be done on local tags and branches.
However, I think that even if we have (Solution 2), I'd still like to have (Solution 1), because this will facilitate doing cross-component development on multiple machines.
Summary of Issue: checkout_externals currently wants to update all of the externals in the sandbox, but often I want to update only some of the components, or there is a particular component I do not want to update. I think that this can be easily handled with --include and --exclude options to checkout_externals
Expected behavior and actual behavior: only those externals in the include list (or not in the exclude list) would be updated.
The SVN_EXTERNAL_DIRECTORIES indicates that clm will be brought in as a SVN external
but the CESM.cfg file indicates it's to be pulled in as git. So, for the git version, should we delete the SVN_EXTERNAL_DIRECTORIES file?
Running checkout_externals.py gives:
[aliceb@cheyenne2:cesm-demo-externals]>./checkout_externals/checkout_externals.py
Processing model description file : CESM.xml
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Checkout components: clm, Processing model description file : CLM.xml
fates, mosart, cime, cice, pop, cism, cam, cime_config,
Which suggests that CLM.xml also checks out an identical set of model components.
However, when I run the svn demo version of the checkout_externals.py, I get the following
output:
./checkout_externals/checkout_externals.py
Processing model description file : CESM.xml
Checking status of components: clm,
Checkout components: clm, Processing model description file : CLM.xml
fates,
This may just require more detailed status output be added to the output from the script
instead of hidden in the checkout_externals.log.
Another point of confusion is that the components/clm contains a README_FIRST and it's own
checkout_externals.py tool that when run duplicates the CESM externals in the components/clm/components subdir:
[components/clm]/checkout_externals/checkout_externals.py
Processing model description file : CESM.xml
Processing model description file : CLM.xml
Checking status of components: cism, clm, fates, ptclm, mosart, cime, rtm,
Checkout components: cism, clm, fates, mosart, cime,
I think somehow CLM checkout_externals.py needs to know that it was checked out as part of CESM or as stand-alone.
Create a tool to generate externals description files.
This tool is the inverse operation of checkout_externals.
checkout_externals - attempts to make the working tree match the externals description file.
generate_externals - attempts to make the externals description file match the working tree.
Basic workflow:
Overwrite existing files by default? This is safe if they are under revision control and un-modified. Otherwise there is a risk of destroying user data.
What if a sub-external doesn't exist? For example, run generate_externals on a cesm repo where CLM has sub-externals, but CLM has not been checked out. Error? Reference existing file?
It makes sense that you can't run checkout_externals from someone else's directory. But in debugging a problem, it can be helpful to be able to run checkout_externals -S from someone else's directory. However, this isn't possible right now because it tries to write to a log file in the current directory, where you normally won't have write permission.
I'd say that the status command probably shouldn't write to a log file at all, by default. But if it does, then could there be a flag to checkout_externals to suppress writing to the log file, so that you can run it from someone else's directory?
Error for RTM checkout returned when running command:
./checkout_externals/checkout_externals.py -o
Processing model description file : CESM.xml
Processing model description file : CLM.xml
Checking status of components: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
Enter passphrase for key '/home/aliceb/.ssh/id_rsa':
Enter passphrase for key '/home/aliceb/.ssh/id_rsa':
Checkout components: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cism, rtm,
ERROR: CalledProcessError in "git checkout rtm1_0_62_cimeupdate_tags_n01"
Details from the checkout_externals.log file don't include the URL of the repo
to help debug the problem:
INFO : 2017-11-09 12:25:23 : In directory: /home/aliceb/checkout_testing/cesm-demo-externals/components/rtm
execute_subprocess running command:
INFO : 2017-11-09 12:25:23 : [u'git', u'fetch', u'--all', u'--tags']
INFO : 2017-11-09 12:25:23 : git fetch --all --tags
DEBUG : 2017-11-09 12:25:28 : Fetching origin
DEBUG : 2017-11-09 12:25:28 :
INFO : 2017-11-09 12:25:28 : In directory: /home/aliceb/checkout_testing/cesm-demo-externals/components/rtm
execute_subprocess running command:
INFO : 2017-11-09 12:25:28 : [u'git', u'checkout', u'rtm1_0_62_cimeupdate_tags_n01']
INFO : 2017-11-09 12:25:28 : git checkout rtm1_0_62_cimeupdate_tags_n01
ERROR : 2017-11-09 12:25:28 : Command '[u'git', u'checkout', u'rtm1_0_62_cimeupdate_tags_n01']' returned non-zero exit status 1
ERROR : 2017-11-09 12:25:28 : Returned : 1
DEBUG : 2017-11-09 12:25:28 : error: pathspec 'rtm1_0_62_cimeupdate_tags_n01' did not match any file(s) known to git.
DEBUG : 2017-11-09 12:25:28 :
ERROR : 2017-11-09 12:25:28 : CalledProcessError in "git checkout rtm1_0_62_cimeupdate_tags_n01"
INFO : 2017-11-09 12:25:28 :
ERROR: CalledProcessError in "git checkout rtm1_0_62_cimeupdate_tags_n01"
manage_externals sporadically fails when checking out an svn external that itself has git externals. We've seen this for CISM, and yesterday I saw it for the first time with CLM (when checking out FATES).
The problem occurs when svn gives this error message at the end of checking out its external from github:
DEBUG : 2017-12-19 11:18:08 : svn: warning: W175002: REPORT request on '/NCAR/fates-release/!svn/vcc/default' failed
and then at the end of the svn checkout process:
DEBUG : 2017-12-19 11:18:08 : Checked out revision 88097.
DEBUG : 2017-12-19 11:18:08 : svn: E205011: Failure occurred processing one or more externals definitions
This leads to the following error from manage_externals:
ERROR : 2017-12-19 11:18:08 : Called process did not run successfully.
Returned status: 1
Failed command:
svn checkout https://svn-ccsm-models.cgd.ucar.edu/clm2/trunk_tags/clm4_5_16_r253/components/clm /Users/sacks/testing/my_cesm_sandbox/components/clm
Please check the log file "manage_externals.log" for more details.
INFO : 2017-12-19 11:18:08 :
ERROR: Called process did not run successfully.
Returned status: 1
Failed command:
svn checkout https://svn-ccsm-models.cgd.ucar.edu/clm2/trunk_tags/clm4_5_16_r253/components/clm /Users/sacks/testing/my_cesm_sandbox/components/clm
Please check the log file "manage_externals.log" for more details.
Since both CLM and CISM are moving to pure git, I expect this problem to go away for them. But this could lead to problems if we have other svn externals that themselves pull in externals from github. @mnlevy1981 It looks like this is an issue for POP (which pulls in cvmix from github), but maybe nothing else?
I feel like README_FIRST should start with a "Quick start" section or something like that, which simply says something like:
To get a working version of the model, after cloning this repository, run the following from the top-level of the clone:
./checkout_cesm/checkout_externals.py
(Then you can go into the details below.)
As @bertinia pointed out (in relation to #10), the CLM version of this file should perhaps add something saying that this is only needed if you cloned the clm repo directly – you do not need to run this if you have a copy of CLM inside CESM, CAM, etc.
This issue needs to be documented for developers checking out private repos;
On hobart, I'm getting this error:
./checkout_externals/checkout_externals.py --model CESM.cfg
Processing model description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
The authenticity of host 'github.com (192.30.253.113)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Checkout components: clm,
ERROR: CalledProcessError in "git clone [email protected]:NCAR/clm-demo-externals.git clm"
Similar error when specifying --model CESM.json:
./checkout_externals/checkout_externals.py --model CESM.json
Processing model description file : CESM.json
Checking status of components: clm, mosart, cime, ww3, cice, pop, cism, rtm, cam, cime_config,
Checkout components: clm,
ERROR: CalledProcessError in "git clone [email protected]:NCAR/clm-demo-externals.git clm"
Is there a reason to use the ssh clone command as opposed to the https?
Yes, for private repos, access via ssh is required.
When reviewing the README_FIRST, it points to:
https://help.github.com/articles/connecting-to-github-with-ssh/
and I also tried running the tool:
on hobart using:
./mk_user_ssh_keys.sh -t github.com -u bertinia -v
But I'm getting permission denied errors from the target cause I think the only
way to add an ssh key on github is via the web interface settings in my profile.
I added the hobart ssh key to my github profile via the web settings and was able to
get the download to work correctly.
We will need more explanation of how to setup the ssh keys in the README_FIRST
if we use the ssh clone link.
According to @goldy2718, we will use https protocol for public released versions of the model.
Developers will still need to add ssh keys to their profiles when downloading private repos.
From @cacraigucar on November 20, 2017 21:2
Going from a git or svn checkout where the screen is inundated with every file checked out, to manage_externals where it silently sits there for minutes can be disconcerting as you are not sure if the command is making progress or not. A simple "starting to check out XXX", "completed check out of XXX" for each of the components would be useful so the user can make sure progress is actually being made. I ended up opening a new window and seeing that something was happening just to make sure progress was being made and it wasn't simply hung.
Copied from original issue: ESCOMP/CESM#4
We need comments in the CESM.cfg and CLM.cfg explaining what they are, what they do, and how they are used.
manage_externals will return an error code even for "--status" if there is a missing Externals file or some other problem running manage_externals.
Error code is returned for "--status" if an Externals file is missing. I expect --status to not return an error code.
1179 23:15 cd ctsm/
1180 23:15 rm Externals.cfg
1181 23:15 ./manage_externals/checkout_externals --status
1182 23:15 echo $?
manicv1.0.2
81ccf92e8c4da272121fcc96902021be9e7a7b1f
cheyenne
no
cheyenne4 erik/ctsm> ./manage_externals/checkout_externals --status
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CISM.cfg
ERROR: External externals description file "Externals_CLM.cfg" does not exist! In directory: /glade/p/work/erik/ctsm
cheyenne4 erik/ctsm> echo $?
1
With the changes in #63 , my preference would be to have the current "verbose" output be the default - definitely for checkout, and probably for status, too. I could imagine having a --quiet
flag that reverts to the current default and a --verbose
flag that gives the current --verbose --verbose
behavior.
We can implement this for v1.0 if there's time, or wait and change this post-v1.0 after some other people have enough experience that they weigh in one way or the other.
Currently, unit testing of repository_git
is done by having the calls to git commands extracted into their own functions, and then having unit tests set these methods to test-specific methods on a per-test basis, as in:
self._repo._git_remote_verbose = self._git_remote_origin_upstream
where _git_remote_origin_upstream
is a function defined in the unit test class.
I think it would be easier to write, understand and maintain the unit tests if we introduced a class whose purpose is to wrap various git commands. We would move all of the git static methods from repository_git.py
into this new class. By default, a GitRepository object would be set up to hold an instance of the real GitWrapper class (this could be set up in the __init__
method of GitRepository. However, for unit testing, we could have a separate GitWrapperFake class, and we could set
self._git_wrapper = GitWrapperFake()
self._repo.set_git_wrapper(self._git_wrapper)
This GitWrapperFake would have its own fake versions of all of the git commands, which return some specified value. They would all be set up with some default value (in case one isn't specified explicitly in a unit test), but they could all be configured. So we could have, for example, in a given test:
self._git_wrapper.set_git_remote_verbose("return value from git remote verbose")
By having the actual return strings in the given test, rather than needing to chase down the functions being used, I think the tests will be more maintainable.
Rename the repository to reflect future as a suite of tools to manage externals.
Rename the local for subtrees of this repository to match repo name.
My current workflow for CESM development is to set up a CESM sandbox where components/pop
is a softlink to a local SVN checkout of POP (and in that checkout, POP's externals/marbl/
directory is another softlink to a git checkout of MARBL). I could use manage_externals
to checkout my POP branch directly, but I would still need to update the MARBL checkout manually... so it would be nice to have the option of doing something like
[pop]
tag = none
protocol = softlink
repo_url = /path/to/pop
local_path = components/pop
required = True
Expected behavior would be a directory structure where components/pop/
is a softlink to /path/to/pop
, but instead I get
Processing externals description file : CESM.cfg
ERROR: Unknown repository protocol "softlink" in "pop".
In ESCOMP/cesm:
$ git diff
[pop]
tag = trunk_tags/cesm_pop_2_1_20171008
-protocol = svn
+protocol = softlink
repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2
local_path = components/pop
required = True
Tested on MARBL, using cesm2_0_beta07.002
tag of ESCOMP/cesm
N/A
see above
I have a branch of POP where we pointed to an SVN tag of MARBL; we have decided to point to a git tag instead
I expected to be able to change protocol
from svn
to git
in my externals file, but instead I get the error included below. I ended up running rm -rf
and then checkout_externals
but it seems like tool should be able to handle this case.
checkout_externals
git
and try to rerun checkout_externals
This is on hobart, using manic-v0.8.0
Nope, this is clean code.
$ ./externals/manage_externals/checkout_externals
Processing externals description file : Externals.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: pop_externals, cvmix, marbl,
./externals/CVMix
? ./externals/MARBL
----------------------------------------------------------------------
The external repositories labeled with 'M' above are not in a clean state.
The following are two options for how to proceed:
(1) Go into each external that is not in a clean state and issue either
an 'svn status' or a 'git status' command. Either revert or commit
your changes so that all externals are in a clean state. (Note,
though, that it is okay to have untracked files in your working
directory.) Then rerun checkout_externals.
(2) Alternatively, you do not have to rely on checkout_externals. Instead, you
can manually update out-of-sync externals (labeled with 's' above)
as described in the configuration file Externals.cfg.
----------------------------------------------------------------------
When untracked files are present in one of the externals managed by manage_externals, checkout_externals refuses to proceed since that external is in a "modified" state. I think it's important that it proceed in this case, because it will be very common for users to have untracked files in their directories.
Expected: checkout_externals proceeds despite there being untracked files in one or more external directories.
Actual: checkout_externals bails with a message like this:
Processing externals description file : CESM.cfg
Checking status of components: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam, cime_config,
M ./cime
./cime_config
./components/cam
./components/cice
./components/cism
./components/clm
./components/mosart
./components/pop
./components/rtm
./components/ww3
----------------------------------------------------------------------
Some external repositories that are not in a clean state. Please
ensure all external repositories are clean before updating.
----------------------------------------------------------------------
Clone the cesm repo (git clone [email protected]:ESCOMP/cesm.git
)
Run ./manage_externals/checkout_externals
Add an untracked file in one of the externals directories. In my case, I did a workflow that will be very common: creating a case in cime/scripts
: ./create_newcase --case mytest --compset A --res f45_g37_rx1
Rerun ./manage_externals/checkout_externals
code: cesm repo at a9702c9
Machine: mac (roo2)
No
See above
The command line option --model, -m, is a relic from when this was managing the 'model'. We are now being more explicit about managing 'externals'. The command line options should be updated to --externals, -e, to reflect this.
Having tag =
in a component block using protocol = svn
is a little misleading; first of all, svn "tags" are really just branches combined with a web-hook to make them read-only. Also, it's not really clear where repo_url
ends and tag
begins. As far as I can tell, all of the following are the same
[pop]
tag = trunk_tags/cesm_pop_2_1_20171008
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2
local_path = components/pop
required = True
[pop]
tag = pop2/trunk_tags/cesm_pop_2_1_20171008
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu
local_path = components/pop
required = True
[pop]
tag = cesm_pop_2_1_20171008
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2/trunk_tags
local_path = components/pop
required = True
What I'd rather see is the full URL in repo_url
and no tag
field:
[pop]
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2/trunk_tags/cesm_pop_2_1_20171008
local_path = components/pop
required = True
Instead of the required field being a bool , what if it were a list of one or more groups so
for example cvmix would have:
required = pop, mom
while fms might have
required = mom, camfv3
then the -o argument to checkout_externals would expect a list of one or more optional components.
The tool seems to assume that the remote you want to work with is named ‘origin’. If I have something like this:
<repo_url>[email protected]:billsacks/clm-demo-externals.git</repo_url>
<branch>mybranch</branch>
and I have done:
git remote add billsacks [email protected]:billsacks/clm-demo-externals.git
I get:
ERROR: Invalid repository in /Users/sacks/cesm_code/cesm-demo-externals/components/clm, url = [email protected]:NCAR/clm-demo-externals.git, should be [email protected]:billsacks/clm-demo-externals.git
I have also encountered this error after merging in a branch that points to an external from a different remote.
My understanding (maybe wrong) is that the name of remotes in git (e.g., “origin”, or “billsacks” as above) is just a convenience, and that anywhere where you could have a short remote name like this it’s also valid to list the full remote URL (like [email protected]:billsacks/clm-demo-externals.git). If that’s right, then I’m thinking that there’s no need for checkout_model.py to assume anything regarding remote alias names (“origin” or anything else): any time it needs to interact with the remote, it can just give the full path to the remote.
Since I'm paranoid, I started worrying about the possibility that the wrong commit will be checked out when checkout_externals tries to checkout a tag. While this doesn't seem like a big risk, it seems possible - especially after someone has been doing some work locally and then reruns checkout_externals after the .cfg file has been updated.
There are two scenarios that I'm worried about:
(1) There happens to be a branch named the same as the given tag
If the reference is ambiguous in this way, git will checkout the branch rather than the tag. This one seems easy to fix: rather than just git checkout TAGNAME
, we should do git checkout tags/TAGNAME
(tested with git 2.14.1).
(2) This tag is defined differently in different places - i.e., the same tag name points to a different sha in different places.
While this one seems unlikely (I hope), it could cause massive confusion if it did happen.
(2a) Multiple remotes define this tag differently
I believe that the current behavior - git fetch --all --tags
will give indeterminate behavior in this case.
To address this, I think we should just do a fetch from the single remote listed in the .cfg file rather than doing a fetch from --all
.
(2b) There is already a local tag with this name but pointing to the wrong sha
All of my testing with git 2.14.1 suggests that local tags are overwritten with the tag from the remote when doing a fetch, but some answers on StackOverflow suggest that this may be version-dependent (see the second answer, by torek, here: https://stackoverflow.com/questions/35979642/how-to-checkout-remote-git-tag (especially the section on Tags), and see the comments by torek in the answer by eckes here https://stackoverflow.com/questions/9662249/how-to-overwrite-local-tags-with-git-fetch/21475727 ).
I can see a couple of ways to address this.
We could use the method suggested by torek here https://stackoverflow.com/questions/9662249/how-to-overwrite-local-tags-with-git-fetch/21475727
git fetch <remote> '+refs/tags/*:refs/tags/*'
Though maybe just specifying the single tag of interest rather than *
. (I could imagine first fetching all tags from the given remote, because it's useful to update all tags, and then doing something like git fetch <remote> '+refs/tags/TAGNAME:refs/tags/TAGNAME'
.)
(NOTE: I haven't tested this method myself.)
Alternatively, it seems we could first delete any local tag with the given name (git tag -d TAGNAME
, ignoring errors in case that tag doesn't exist yet locally), then do a fetch. I slightly prefer this approach simply because it's easier to understand.
Add python 3.4 and 3.5 to CI testing.
Create additional output verbosity levels:
./manage_externals/checkout_externals
Processing externals description file : CESM.cfg
Processing externals description file : CLM.cfg
Checking status of externals: cism, clm, fates, ptclm, mosart, cime, rtm,
Checking out externals: cism, clm, mosart, cime,
Checking out externals: fates,
./manage_externals/checkout_externals --status
Processing externals description file : CESM.cfg
Processing externals description file : CLM.cfg
Checking status of externals: cism, clm, fates, ptclm, mosart, cime, rtm,
m ./cime
./components/cism
./components/mosart
e-o ./components/rtm
./src/fates
e-o ./tools/PTCLM
./manage_externals/checkout_externals --verbose
Processing externals description file : CESM.cfg
Processing externals description file : CLM.cfg
Checking status of externals:
./cime:
Required: https://github.com/CESM-Development/cime.git - mvertens/updates_for_clm.tag2
Working: https://github.com/CESM-Development/cime.git - cime5.3.0-alpha.35
./components/cism
Synced: https://svn-ccsm-models.cgd.ucar.edu/glc/trunk_tags/cism2_1_37
./components/mosart
Synced: https://svn-ccsm-models.cgd.ucar.edu/mosart/trunk_tags/mosart1_0_26
./components/rtm
Optional: https://github.com/escom/rtm/ - rtm1_0_23
Working: empty
./src/fates
Synced: https://github.com/NCAR/fates-release - fates_s1.0.0_a1.0.0
./tools/PTCLM
Optional: https://github.com/escom/ptclm - ptclm20171212
Working: empty
Checking out externals:
./cime:
https://github.com/CESM-Development/cime.git - mvertens/updates_for_clm.tag2
./components/cism
https://svn-ccsm-models.cgd.ucar.edu/glc/trunk_tags/cism2_1_37
./components/mosart
https://svn-ccsm-models.cgd.ucar.edu/mosart/trunk_tags/mosart1_0_26
./components/rtm
Optional - empty
./src/fates
https://github.com/NCAR/fates-release - fates_s1.0.0_a1.0.0
./tools/PTCLM
Optional - Empty
Checking out externals:
./src/fates
https://github.com/NCAR/fates-release - fates_s1.0.0_a1.0.0
./manage_externals/checkout_externals --status --verbose
Processing externals description file : CESM.cfg
Processing externals description file : CLM.cfg
Checking status of externals:
./cime:
Required: https://github.com/CESM-Development/cime.git - mvertens/updates_for_clm.tag2
Working: https://github.com/CESM-Development/cime.git - cime5.3.0-alpha.35
./components/cism
Synced: https://svn-ccsm-models.cgd.ucar.edu/glc/trunk_tags/cism2_1_37
./components/mosart
Synced: https://svn-ccsm-models.cgd.ucar.edu/mosart/trunk_tags/mosart1_0_26
./components/rtm
Optional: https://github.com/escom/rtm/ - rtm1_0_23
Working: empty
./src/fates
Synced: https://github.com/NCAR/fates-release - fates_s1.0.0_a1.0.0
./tools/PTCLM
Optional: https://github.com/escom/ptclm - ptclm20171212
Working: empty
m ./cime
./components/cism
./components/mosart
e-o ./components/rtm
./src/fates
e-o ./tools/PTCLM
./manage_externals/checkout_externals --verbose --verbose
From an earlier conversation, I thought we were going to rename this to remove the .py, in case we ever want to rewrite it in ruby or haskell or ...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.