Update J1850G compset #90
Assignees
Projects
Comments
|
Two issues with the proposed compset in the above comment have come to light.
|
alperaltuntas
pushed a commit
to alperaltuntas/cesm
that referenced
this issue
Apr 24, 2019
a465b4f add --quiet argument to improve performance b2f3ae8 Merge pull request ESCOMP#83 from jedwards4b/jedwards/components_arg 3f4c88f fix comment c1b5b09 remove unneeded logic 4fdf180 one more test f78d60f another test bf52ac6 add a test 91d4851 fix pylint issue 987df5a only use components if populated 98a810d add a components arg to checkout only select components 6923119 Merge pull request ESCOMP#90 from ESMCI/issue-86-detached-sync-status b11ad61 Merge branch 'master' into issue-86-detached-sync-status 3b624cf Merge pull request ESCOMP#93 from billsacks/work_on_coverage 2562830 Run a single coverage command rather than two separate commands d1de5f8 Return to starting directory after each test 144f7d9 Merge pull request ESCOMP#92 from billsacks/point_to_esmci 58b8d3e Point to location of repository 0b46d81 Point to correct location for build/coverage status a385070 fix pylint problems dcf17b6 make style cleanup 92d342c Rewrite _current_ref to use plumbing rather than parsing porcelain ca0a5d3 Rework some git repository functions, and major rework of unit tests 719383e Remove commented-out pdb.set_trace() call 376c780 Bugfix: detect and report 'detached from' correctly 21813e9 Add system test demonstrating failure to detect out of sync status. 1a7c59d Merge documentation update into master. 247fee1 Document return values of checkout.py: main git-subtree-dir: manage_externals git-subtree-split: a465b4f
alperaltuntas
pushed a commit
to alperaltuntas/cesm
that referenced
this issue
Apr 24, 2019
commit 025e6cb543cb4ccff0f65bc3d48586ec9a973b2a
Merge: 0c5a2f6 489842b
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon May 21 18:54:03 2018 -0600
Merge pull request #107 from jedwards4b/ignore_empty_git_dir
if you encounter an empty directory clone into it
If when cloning a git repository the root directory already exists but is empty go ahead and clone into it anyway.
User interface changes?: No
Fixes: #104
Testing:
test removed:
unit tests: all pass
system tests: all pass
manual testing: mom6_interface
commit 489842b54bc4a3bb490d0fd5c293b3a2b6701d28
Author: Jim Edwards <jedwards@ucar.edu>
Date: Mon May 21 15:21:56 2018 -0600
if you encounter an empty directory clone into it
commit 0c5a2f696ee73bff09e4744cfa48ba2eb0d46598
Merge: 0427305 7799e99
Author: jedwards4b <jedwards@ucar.edu>
Date: Sat May 19 06:56:37 2018 -0600
Merge pull request #106 from billsacks/remove_logfile_message
Remove message about checking the log file for more details
Now that logging is off by default, this message is misleading
User interface changes?: No
Testing:
test removed:
unit tests: pass
system tests: pass
manual testing: checked error message
commit 7799e993c81e19a6f7685ab4c5bcedde88ece1d8
Author: Bill Sacks <sacks@ucar.edu>
Date: Fri May 18 15:20:14 2018 -0600
Remove message about checking the log file for more details
Now that logging is off by default, this message is misleading
commit 04273058c297127927f0fc85eed1cdc33e1a3af3
Merge: 9af6b02 9bb46aa
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon May 7 19:40:00 2018 -0600
Merge pull request #103 from billsacks/no_logging
Make no-logging be the default
@jedwards4b, @gold2718 and I all feel that no-logging should be the
default. This change is made here. I introduced a new --logging option
that can be used to turn on logging. I have maintained --no-logging as
an option for backwards compatibility - e.g., since cime calls
checkout_externals with '--no-logging' - but now --no-logging doesn't
actually do anything.
User interface changes?: Yes
`--no-logging` is now the default, new `--logging option` to turn on
logging. I have kept `--no-logging` in place to ensure backwards
compatibility.
Fixes: #95 (Should not write to log file when running status command)
As noted in #97, @jedwards4b, @gold2718 and I felt that logging should
always be off by default.
Testing:
test removed: none
unit tests: pass
system tests: pass
manual testing: Ran with `--logging`, `--no-logging`, both and
neither, with python2 and python3 (note that running with both
generates an error message)
commit 9bb46aa55ea01dcc5a0ea0b6fba790b3ba9e2155
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon May 7 15:58:46 2018 -0600
Make no-logging be the default
@jedwards4b, @gold2718 and I all feel that no-logging should be the
default. This change is made here. I introduced a new --logging option
that can be used to turn on logging. I have maintained --no-logging as
an option for backwards compatibility - e.g., since cime calls
checkout_externals with '--no-logging' - but now --no-logging doesn't
actually do anything.
commit 9af6b021d3fc794b1a33dd9b2843696b1261842f
Merge: 60fc03b 7f973ae
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon May 7 15:39:11 2018 -0600
Merge pull request #102 from billsacks/explain_qmark
Explain question mark
This is a minimal fix for #78
User interface changes?: No
Fixes: Partial fix for #78 (weird result when moving from subversion to git)
Testing:
test removed: none
unit tests: pass
system tests: pass
manual testing: triggered message, examined formatting
commit 7f973ae35e0a6a32a30227e0513a36c6daa2f7c0
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon May 7 15:04:32 2018 -0600
Run through make style
commit d077a57d9b4f8fb4667c30c78d39e0d76b59c381
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon May 7 15:02:46 2018 -0600
Add message describing meaning of '?'
This is a minimal fix for
https://github.com/ESMCI/manage_externals/issues/78
commit 60fc03b70ca6fe95e03a4b85040fad0d0605ea73
Merge: bfa4831 28073ec
Author: jedwards4b <jedwards@ucar.edu>
Date: Mon May 7 13:50:27 2018 -0600
Merge pull request #101 from ESMCI/catch_svn_error
catch errors from svn status --xml
The issue is that the svn command actually produces a malformed xml output in this case which manage_externals attempts to parse before printing the error. This solution first attempts to parse the
xml returned from svn and bails and prints the raw output if it cannot. The resulting output is a little ugly but readable.
User interface changes?: No
[ If yes, describe what changed, and steps taken to ensure backward compatibilty ]
Fixes: #100
Testing:
test removed:
unit tests: all pass
system tests:
manual testing: Introduced a bad svn repo and tested
commit 28073ec41fcb88952d3dfabb8f019720b0828c85
Author: Jim Edwards <jedwards@ucar.edu>
Date: Fri May 4 11:17:08 2018 -0600
add exception class
commit 4fb7e47f27b380f67196d0352ad0fa408a8c31db
Author: Jim Edwards <jedwards@ucar.edu>
Date: Fri May 4 10:02:43 2018 -0600
catch errors from svn status --xml
commit bfa483124f636a283dc5a75d019441855175afff
Merge: a465b4f 7d12650
Author: jedwards4b <jedwards@ucar.edu>
Date: Sat Apr 14 16:50:07 2018 -0600
Merge pull request #98 from billsacks/quieter
Make more git and svn commands quiet
We've found that commands that do a lot of output can be slow when
called via python's subprocess. So make even more git and svn commands
quiet. These ones probably don't have a huge impact, but I can't see any
reason to keep them non-quiet.
Also run 'make style', which fixed some earlier commits.
User interface changes?: No
Fixes: none
Testing:
test removed: none
unit tests: pass
system tests: pass
manual testing: none
commit 7d12650b92fc899174dd4b2d7cddd53b1c2999fa
Author: Bill Sacks <sacks@ucar.edu>
Date: Sat Apr 14 16:20:09 2018 -0600
make style
commit afb4f1153b6a97432a4eed86b32661db9c41610f
Author: Bill Sacks <sacks@ucar.edu>
Date: Sat Apr 14 16:18:49 2018 -0600
Make more git and svn commands quieter
We've found that commands that do a lot of output can be slow when
called via python's subprocess. So make even more git and svn commands
quiet. These ones probably don't have a huge impact, but I can't see any
reason to keep them non-quiet.
commit a465b4fb740e7895248fab4a302d402ea2ba5cf7
Author: Jim Edwards <jedwards@ucar.edu>
Date: Fri Apr 13 07:25:49 2018 -0600
add --quiet argument to improve performance
commit b2f3ae8e251371bdb451d566dfa124e0c826a4b5
Merge: 6923119 3f4c88f
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Apr 10 15:22:39 2018 -0600
Merge pull request #83 from jedwards4b/jedwards/components_arg
add a components arg to checkout only select components
Add optional argument to only process components listed on the command line
checkout_externals cam clm
will only process externals associated with the listed components cam and clm.
Also added a test of the new functionality.
User interface changes?: Yes
An additional command line option was added, previous functionality continues to be supported
Fixes: #80
Testing:
test removed:
unit tests: all pass
system tests:
manual testing: Tested with lists of components in Externals.cfg and with ones that were not there.
commit 3f4c88fe2d8c3868fc4051a71f371f1c041ee41a
Author: Jim Edwards <jedwards@ucar.edu>
Date: Tue Apr 10 14:20:31 2018 -0600
fix comment
commit c1b5b09d9d224fd63a26fe6af85a18e37212fd7b
Author: Jim Edwards <jedwards@ucar.edu>
Date: Tue Apr 10 14:16:35 2018 -0600
remove unneeded logic
commit 4fdf1802668dae6bb43aac32e09722a1cd9a655e
Author: Jim Edwards <jedwards@ucar.edu>
Date: Wed Apr 4 06:13:39 2018 -0600
one more test
commit f78d60f3c5e2f02ff6ccc23076a321640c07e537
Author: Jim Edwards <jedwards@ucar.edu>
Date: Tue Apr 3 21:21:29 2018 -0600
another test
commit bf52ac615801bc04ad85ad1ed3a0b4a9cae15dcb
Author: Jim Edwards <jedwards@ucar.edu>
Date: Tue Apr 3 20:54:32 2018 -0600
add a test
commit 91d4851de0b4d1b2d37a2e8c21c4cc86fdd90d83
Author: Jim Edwards <jedwards@ucar.edu>
Date: Tue Apr 3 18:08:42 2018 -0600
fix pylint issue
commit 987df5a3aba092b818d7459d71da8e51304822cd
Author: Jim Edwards <jedwards@ucar.edu>
Date: Tue Apr 3 17:38:44 2018 -0600
only use components if populated
commit 98a810db57659f8bbf02af64ae2e753d5537be5c
Author: Jim Edwards <jedwards@ucar.edu>
Date: Thu Mar 29 10:58:34 2018 -0600
add a components arg to checkout only select components
commit 69231197cdd227a0cece7afd5d4593b765d6ce79
Merge: 3b624cf b11ad61
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Apr 10 12:41:40 2018 -0600
Merge pull request #90 from ESMCI/issue-86-detached-sync-status
git: rework logic for determining in-sync and current ref name
Rework two aspects of the logic for git repositories; both of these can
change behavior in some cases when printing status:
1. Determining whether the local checkout is in-sync with the expected
reference from the configuration file: Now we always convert the
expected reference to a hash and compare that with the
currently-checked-out hash. Previously, we sometimes did the
comparison using names, e.g., just ensuring that you're on the right
branch.
2. Determining the current ref name (e.g., the branch or tag currently
checked out). Now we use a number of plumbing commands rather than
relying on regex parsing of 'git branch -vv'. The previous regex
parsing was fragile and hard to maintain, and was the source of a
number of bugs. In addition, differences between git v. 1 and git
v. 2 meant that the result was incorrect in some cases -
particularly, in the case where we have "detached from foo" (which is
the text that always appeared for a detached head in v 1, but in v 2
means we are no longer at foo).
I have also overhauled the unit tests covering this functionality. Many
tests were no longer needed with the new logic and so I have removed
them. I have added some other tests covering the new functionality.
User interface changes?: Yes
- Subtle changes to status output, as described above
- One particular change is: If we're on a tracking branch,
`checkout_externals -S -v` will show the name of the local branch
rather than the tracked branch. (This is more accurate, because we may
not actually be at the head of the tracking branch.)
Fixes: #86 (Status incorrectly reports in-sync when you have made
commits in detached head state - fixed due to the change in (1)).
Testing:
test removed: many no-longer-relevant unit tests
unit tests: pass
system tests: pass
manual testing: basic manual testing of checking out and running
status, in the context of cesm and ctsm
commit b11ad61bb8ab2efb7bf1786e316fb1afce5a53ce
Merge: d1de5f8 3b624cf
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Apr 10 09:05:01 2018 -0600
Merge branch 'master' into issue-86-detached-sync-status
commit 3b624cf5e9acef16e2d8380c4bf78fae6cc0474b
Merge: 144f7d9 2562830
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Apr 10 09:03:31 2018 -0600
Merge pull request #93 from billsacks/work_on_coverage
Run a single coverage command rather than two separate commands
I was finding that, even with the '--append' flag, some of the coverage
results were getting overwritten in the second run, resulting in a
too-low coverage report.
This turns out to have no effect on the reported coverage on master, but
it fixes the coverage report for #90: without this change, some lines
that are covered only by unit tests are listed as uncovered in the final
coverage report (at least, I think that's what was going on).
User interface changes?: No
Fixes: None
Testing:
test removed: changed 'make coverage' operation
unit tests:
system tests:
manual testing:
commit 2562830f6bbd109c3d5a82d0f271db7209863d93
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Apr 10 08:39:30 2018 -0600
Run a single coverage command rather than two separate commands
I was finding that, even with the '--append' flag, some of the coverage
results were getting overwritten in the second run, resulting in a
too-low coverage report.
commit d1de5f8f5e0a836b0c923fab48f1107057724d36
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Apr 10 08:32:21 2018 -0600
Return to starting directory after each test
Without this, future tests can fail
commit 144f7d99c8c5425ea648084d6ac190fbaf289a3f
Merge: 1a7c59d 58b8d3e
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon Apr 9 15:36:45 2018 -0600
Merge pull request #92 from billsacks/point_to_esmci
Point to esmci in README and for build/coverage status
commit 58b8d3e1e3eefe5a0827d3718a0d81bdca2f7940
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon Apr 9 15:10:52 2018 -0600
Point to location of repository
commit 0b46d8130e73ce0322675a9fa19ca28949569c1c
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon Apr 9 15:05:47 2018 -0600
Point to correct location for build/coverage status
commit a385070bbb8e2400f58239d5a058bff644323598
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon Apr 9 10:51:09 2018 -0600
fix pylint problems
commit dcf17b60fdcb5a17e947d455510e166a3b7b55f4
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon Apr 9 09:15:19 2018 -0600
make style cleanup
commit 92d342c9287cafaaec23f600c62765ffe3eeac79
Author: Bill Sacks <sacks@ucar.edu>
Date: Mon Apr 9 09:01:10 2018 -0600
Rewrite _current_ref to use plumbing rather than parsing porcelain
Previously, _current_ref attempted to do everything with the output from
git branch -vv. This required some complex regex matches that were
fragile and hard to maintain. In addition, differences between git v. 1
and git v. 2 meant that the result was incorrect in some cases -
particularly, in the case where we have "detached from foo" (which is
the text that always appeared for a detached head in v 1, but in v 2
means we are no longer at foo).
I have rewritten _current_ref to use various plumbing commands. This
also resulted in being able to simplify some of the unit tests.
commit ca0a5d34d69d0587b20281017a11a7a8b2bed188
Author: Bill Sacks <sacks@ucar.edu>
Date: Sun Apr 8 10:04:00 2018 -0600
Rework some git repository functions, and major rework of unit tests
- Rename _current_ref_from_branch_command to _current_ref. This now
calls _git_branch_vv rather than expecting it to be called in. This
simplifies work for the caller of this function, and is a step towards
the next rework I want to do.
- Fix _current_ref to give the correct reference if we are detached FROM
something: in this case, give the hash that we're on rather than the
name of the branch/tag from which we are detached.
- Change _check_sync_logic to handle the case where git revparse fails
- Major overhaul of many of the repository_git unit tests. Get tests
passing, remove tests that are now redundant due to earlier work on
this branch, add some new tests needed with the rework on this branch,
and generally rework some of the tests.
commit 719383e9fc2dcfd79fec0860b19e3e687defca5d
Author: Bill Sacks <sacks@ucar.edu>
Date: Fri Apr 6 13:26:25 2018 -0600
Remove commented-out pdb.set_trace() call
commit 376c780225fe5af2f33b8b72bdd942823b7e4924
Author: Ben Andre <andre@ucar.edu>
Date: Wed Apr 4 19:15:28 2018 -0600
Bugfix: detect and report 'detached from' correctly
Bugfix to detect and report when a git repo is detached from a hash, tag, or
branch. This changes the git _check_sync_logic function to compare the
underlying hashes instead of the names of branches and tags.
Testing:
make stest - python2/3 - all system tests pass, including new test
make utest - python2/3 - unit tests have 17 failures because
current_ref_from_branch_command and check_sync_logic is now operating
on the underlying hash rather than the tag and branch names.
commit 21813e9d514ee7379102e1cd800455b46c8cc0b2
Author: Ben Andre <andre@ucar.edu>
Date: Wed Apr 4 17:38:44 2018 -0600
Add system test demonstrating failure to detect out of sync status.
Add a system test demonstrating a bug that checkout_externals fails to detect
out-of-sync status for git when there are commits on a detached head state.
Testing:
make test - python2/3 - 1 new test fails, all existing tests pass.
commit 1a7c59d3a13d204c78ce7e010fe3ce32705a61b6
Merge: f1e9e99 247fee1
Author: Ben Andre <andre@ucar.edu>
Date: Wed Apr 4 16:08:39 2018 -0600
Merge documentation update into master.
Testing:
make test - python2/3 - all tests pass
commit f1e9e990f261a05e60aa96972c3ecd79ceb37c86
Merge: d6423c6 195c1d0
Author: Ben Andre <andre@ucar.edu>
Date: Wed Apr 4 15:32:41 2018 -0600
Merge schema support for git hashes into master.
Add schema support for explicitly specifying a git hash. This schema change is
backwards compatible with older input files, but adds new functionality that
can not be used with prior versions of manage_externals.
According to semantic versioning rules, this is the start of v1.1.
Testing:
make test - python2/3 - all tests pass
commit 247fee1f1706ad961f84e236dd82715a253598a2
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Apr 3 14:30:55 2018 -0600
Document return values of checkout.py: main
commit 195c1d05ebdfc2bd85abb50d452c2afd692a4584
Author: Ben Andre <bjandre@gmail.com>
Date: Mon Mar 19 10:08:59 2018 -0600
Implement explicit use of a hash for git repositories.
Increment the cfg schema to 1.1, and add explicit support for git checkouts via hashes from the config file.
Adds new unit tests for some hash related functionality. Adds checkout of
hashes to some basic systems tests.
Testing:
make test - python2/3 - all tests pass
commit 12dd743ef1027b7f065bfc13860720adbc8d9738
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Mar 9 14:18:13 2018 -0700
Refactor: schema validation output
Cleanup the schema validation error output so it is more useable.
Testing:
make test - python3 - all tests pass
commit fdbc72050ee08b1c36a3a3443c83e874a58f77d6
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Mar 9 14:05:55 2018 -0700
Bugfix: incorrect order of operations validing user input
The order of operations for validating user data was incorrect. Previously
optional fields were filled in, then ran checks for whether only one of branch
or tag was set. This didn't properly check if tags and branches were set
correctly. Now check user input, then populate optional data. Remove additional
data added to test input becouse of this error.
Testing:
make all - python3 all tests pass
commit d6423c68f0948329e048d4c317aa3070164225c2
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Mar 21 21:12:25 2018 -0600
Bugfix: timeout limit for subprocesses
For security reasons svn and git do not use standard in/out/error to request
and process authentication. Instead, they use lower level system interfaces.
Python calling a subprocess can only interact with stdio, and can not intercept
or detect the lower level calls. On some systems, these requests are visible
to the user, on some systems they are not. When subprocess commands expect user
input they will wait and manage_externals appears to hang.
Change the subprocess calls to polling the running process and checking the
current run time. If the run time exceeds a timeout limit, then we display an
error message suggesting that the user verify authentication and then declare a
fatal error.
The default timeout interval of 5 minutes is probably too long and can be
shortened. But since svn requires network interaction and some large checkouts
can take a long time, this is a 'conservative' initial value.
Breakup up the subprocess call routine so it is easier to understand.
Testing:
make test - python2/python3 - all tests pass.
manually test with very short timeout to ensure error message is raised.
commit 7998f60c2fd06a5157e7b12bea736bcf4f2552ec
Author: Ben Andre <andre@ucar.edu>
Date: Wed Mar 28 15:28:24 2018 -0600
Update readme and help output
Update the readme and help output for checkout_externals to:
* clarify basic operation as explicit and data driver
* remove some cesm specific names in examples
* Document format externals description file and comments.
* Note that switching repositiory protocol is not supported. Must manually move
the old repository.
* Document a work-around to avoid safety checks on modified files.
* Document that using the 'externals_only' protocol on the top level repository
requires specifying the local_path as '.'
Testing:
make test - python2 - all tests pass
commit 00b6fb2875671a14e2ea406665d9a125a41995ab
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Mar 9 10:50:51 2018 -0700
Bugfix: add explicit schema version checking
Add explicit schema version checking to the externals description object. Add
corresponding unit tests.
Testing:
make test - python3 - all tests pass
commit 0527869d2629c0430ee8e9ae404b2e865e97bec1
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Mar 6 10:27:34 2018 -0700
Update readme
Update the readme to describe how to tell manage_externals to only operate on
the local clone and not remotes. Try to clarify use of the externals field in
the external description file.
Switch from echo to printf when generating readme to fix formatting problem.
Testing:
make all - python3 - all tests pass
commit 1ae8c84b93f498e42c071d0b7cc200115489cd8b
Merge: bfda7b9 b0c16d7
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Mar 6 09:46:31 2018 -0700
Merge bugfix branch for stale subexternals into master.
Testing:
make test - python3 - all tests pass
commit b0c16d7c585276031212663bc0c56a10e3252a0f
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Feb 28 09:28:49 2018 -0700
Bugfix: stale sub-externals after checkout.
When an external has sub-externals, and the sub-externals description file is
modified by a checkout, the in-memory cached description of the sub-externals
is stale and potentially out date.
Change the logic when doing breadth-first search for sub-externals. Don't rely
on the in-memory sub-externals description, but delete it and re-read the
appropriate externals description file.
Update the test_container_full system test to check for the new behavior.
Testing:
make test - python3 - all tests pass
commit 30a4e442582af4bec5d79030fde6e9012c094f2a
Author: Ben Andre <andre@ucar.edu>
Date: Thu Feb 22 18:36:30 2018 -0700
Finish implementing system test for mixed-use externals
Finish implementing a system level test for mixed use externals as an external.
This removes the unittest.skip decorator from the test_container_full system
test.
Note: this test reveals a previously reported bug that when an external with
sub-externals gits updated, the sub-externals will not be updated because the
sub-external information contained in memory has gone stale.
Testing:
make test - python2 - all tests pass, no skips.
commit ac7ff962231de013d06b4a69ecf5a927f5aba077
Author: Ben Andre <andre@ucar.edu>
Date: Thu Feb 22 17:40:52 2018 -0700
Update mixed-use test repo.
Add a sub-externals description file to the mixed-use container/external test
repo with a single sub-external that needs to be checked out. Make a branch in
the repo with a modified set of externals. Update system test using this repo
to not create a sub-externals file, and to only look for a single sub-external.
Testing:
make test - python2 - pass, one skip.
commit bfda7b9ced37b25de09b4607d4218285bf32f9ea
Author: Ben Andre <andre@ucar.edu>
Date: Wed Feb 21 19:52:21 2018 -0700
Bugfix: regexp for determining git tracking branches
The regexp for determining git tracking branches could not
correctly detect a tracking branch when the local branch was
ahead or behind the remote. Update regexp, document additional
requirement in doc string, and add unit tests for new
requirement.
Testing:
make test - python2 - all tests pass, one skip
commit 5fc8709a77f2408e29a59a1de750a9ef919675e4
Author: Ben Andre <andre@ucar.edu>
Date: Mon Jan 22 15:28:18 2018 -0700
Change default externals description filename to Externals.cfg
Change the default externals description filename to Externals.cfg so it is not cesm specific.
Testing:
make test, python2 - all tests pass, one skip.
manual test with clm repo - ok - explicitly pass old filename
manual test with clm repo - ok - pickup new filename
commit f56be929238c4573094cc2bc531b2cb95ddf2201
Author: Ben Andre <andre@ucar.edu>
Date: Mon Jan 22 14:48:22 2018 -0700
Update copyright year in license
Update the copyright year in the license file.
Testing: none
commit 6c3d4dec7fcceb734b1d740d92d32901f0ced659
Merge: 013b82c 9fee6a0
Author: Ben Andre <andre@ucar.edu>
Date: Thu Jan 18 17:21:48 2018 -0700
Change status symbol and improve error message
(1) Change 'm' to 's' to denote out-of-sync: I found it hard to remember
what was meant by 'm' vs. 'M'. Hopefully it will be easier to
remember 's' vs. 'M'.
(2) Improve the message when checkout_externals fails to run because one
or more externals are in a modified 'M' state.
User interface changes?: Yes
Changed 'm' to 's'
Testing:
python2 make test - all tests pass, on skip system test
commit 9fee6a0c9bc6197079af8fb33a673244748b50dd
Author: Ben Andre <andre@ucar.edu>
Date: Thu Jan 18 16:44:56 2018 -0700
Fix lint issues on branch.
Fix some lint issues for logging strings.
Testing:
python2 - make test - all tests pass, one skipped system test.
manually ran status - ok
Test baseline:
Test namelist changes:
Test status: [bit for bit, roundoff, climate changing]
Fixes: [CIME Github issue #]
User interface changes?: [Yes (describe what changes), No]
Code review:
commit bfd9597752e31a0ed7697938ddd80d6d1f50a3f2
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Jan 9 20:15:26 2018 -0700
Change 'm' to 's' and improve message when externals are dirty
(1) Change 'm' to 's' to denote out-of-sync: I found it hard to remember
what was meant by 'm' vs. 'M'. Hopefully it will be easier to
remember 's' vs. 'M'.
(2) Improve the message when checkout_externals fails to run because one
or more externals are in M state.
commit 013b82c638e0ba296809b2c356dcf15205ad4fb3
Author: Ben Andre <andre@ucar.edu>
Date: Thu Jan 4 20:07:12 2018 -0700
Ignore untracked files when running checkout
Untracked files in a git or svn external have the potential to abort a checkout
and leave the sandbox in a poorly defined state that the user needs to resolve.
The initial implementation refused to update if untracked files were present to
prevent this. This restriction has been deamed too restrictive. It is possible
to request svn to do a dry-run and determine if a checkout or switch will be
successful. Git does not have a mechanism to determine if a checkout will be
successful before running, and there are not any clear workarounds.
For now untracked files are being ignored. If problems arise, restrictions and
checks will be re-added to address specific problems.
Testing:
new unit tests added for new git functionality
existing tests were updated to test for new behavior.
python2 - make test - all tests pass, one skip.
commit 7c8638b2a7ce0904a84f52a9a32e95d5baf8eb36
Author: Bill Sacks <sacks@ucar.edu>
Date: Wed Jan 17 20:10:18 2018 -0700
If we're already in sync, do not show output from the checkout command
However, specifying '--verbose --verbose' will still show output from
the checkout command even if we're already in sync.
This implementation feels like a bit of a hack. On the other hand, I can
convince myself that it makes sense as a general rule to decrease the
verbosity level of the checkout if we're in sync.
commit 8067c59c6cc5ddbfcb4edcab182fcc8ec793f7cc
Author: Ben Andre <andre@ucar.edu>
Date: Wed Jan 17 19:08:19 2018 -0700
Refine verbose output
Refine the verbose output based on PR feedback from Bill Sacks.
* Verbose output is cumulative as the verbose count increases.
* checkout and status have similar behavior
* dump level status output is indented consistently.
Testing:
make test - all tests pass
manual testing of cesm and standalone repos with new verbose checkout and status output.
commit b2ee78f09ddaeca965a62597b5e8e809910bf34e
Author: Ben Andre <andre@ucar.edu>
Date: Wed Jan 17 15:09:48 2018 -0700
Implement intermediate verbosity level
Implement the intermediate verbosity level for status and checkout.
Testing:
make test - python2 - all tests pass, one skip
manual testing of --verbose checkout and status.
commit f535defc286c36d093770bda359a5bbc8c6fc99c
Author: Ben Andre <andre@ucar.edu>
Date: Tue Jan 9 16:13:11 2018 -0700
Rename verbose status functions.
Rename the verbose_status functions to verbose_status_dump in preparation for
adding a new verbose status output level.
Testing:
python2 make test - all tests pass, on test skipped
manual testing - status and status verbose work as expected.
commit 002de3c2324cdf063f73931e4bbcbab05ffe7b0a
Merge: 1dcb2f4 290c205
Author: Ben Andre <andre@ucar.edu>
Date: Wed Jan 17 16:59:06 2018 -0700
Better error messages for failed commands
Merge remote-tracking branch 'wjs/print_command_error'
Two related changes here:
1. For any failed command run via execute_subprocess, print more
information about the problem to stdout
2. After doing an svn switch, check the status of the resulting
repository; if dirty, abort with a helpful message
The immediate purpose of both of these is to give more helpful error
messages if there are failures due to the presence of untracked files,
now that we attempt a checkout even if there are untracked files (via
overwritten by a tracked file in the new version:
git aborts the checkout with an error code; the new information
printed by (1) will help users in this case
svn attempts the checkout and returns a 0 error status, but may
leave you in a conflict state; we check this via (2)
Note that (2) should never happen currently (due to the up-front conservative
check of the repository status, which doesn't even allow untracked files), but
it will become possible after #57.
User interface changes?: No
Testing:
new unit tests added for some utility functions
unit tests: pass
system tests: pass
manual testing: For both git and svn, tested having an untracked file
in the working directory that gets overwritten after checking out a
new branch;
commit 1dcb2f40498ae701da5f71b587d5531ffe8d3f0f
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Jan 10 17:27:52 2018 -0700
Bugfix: initialize logging during system tests
Initialize logging during module setup for systems tests.
Testing:
make tests - all tests pass, one test skip. log file present.
commit 290c205fb502e374fab97f23157674624a3cb5df
Author: Bill Sacks <sacks@ucar.edu>
Date: Wed Jan 10 15:38:13 2018 -0700
Fixes from autopep8 and fixes for pylint
commit 2c98c94f6af41df6882ae73acab80a0002b666cd
Author: Bill Sacks <sacks@ucar.edu>
Date: Wed Jan 10 15:31:44 2018 -0700
Fix error in docstring
commit 5cf62214d6aaa94657816fdef3b32220b0fe95da
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Jan 9 16:04:18 2018 -0700
Abort if we find an svn external in dirty state following switch
Unlike git, we cannot rely on the return status here; instead, we need
to check 'svn status' following the switch.
This is to signal to the user if there was an problem in doing the
switch, such as a conflict in the final state. This can happen if you
had an untracked file in the working directory with the same name as a
tracked file in the new revision.
commit fc093a7bd21935e8d054cfda132b4b2aeba61dcb
Author: Bill Sacks <sacks@ucar.edu>
Date: Tue Jan 9 15:20:11 2018 -0700
Give more complete error message for failed command
The main motivation is to echo the git error message if a checkout
results in an error (e.g., due to a conflict between an untracked file
in the working directory and a tracked file in the new sha).
commit 2e0de468554ff2c951e0aa82c586da9a198d6728
Author: Ben Andre <andre@ucar.edu>
Date: Tue Jan 9 15:13:58 2018 -0700
Add command line option to disable logging
Add command line option, --no-logging, to disable logging to a file. In this
mode, debug and info level output from subprocess commands is dropped, and only
warning, error and critical output is output to the screen.
Testing:
make test - all tests pass, one test skipped
manual testing - run with --no-logging, no log file written to disk. Error
level log written to screen, info level dropped.
commit 2739ebbd4cdd0b47ca6a1f2a77419186d9eb5148
Author: Ben Andre <andre@ucar.edu>
Date: Tue Jan 9 14:51:32 2018 -0700
Update style/lint rules in test makefile
Update makefile to remove style as a prereq for the lint command. When 'make
lint' is run by CI, it will now return the actual lint status.
Add new rule 'stylint' for running autopep8 and lint in one command.
Testing:
make test - all tests pass, one skip
make lint/style/stylint work as expected.
commit bdb29ca5bd33302d8fd6c4f07a0b9f93d3f22cfe
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Jan 5 10:35:18 2018 -0700
Bugfix: fetch all tags from a remote
Add a test to verify all tags are being fetched from a remote. New test fails
when a tag is not referenced by a current branch.
Bugfix: git fetch needs to use the --tags option to fetch tags that are not
related to current branches.
Testing:
python2 - make test - all tests pass, one skip.
commit 15f60d2de372aff6bd9cc286226354e6cf343997
Author: Bill Sacks <sacks@ucar.edu>
Date: Thu Jan 4 11:23:13 2018 -0700
Update documentation
* Document that a 'tag' can also be a sha-1
* fix some typos
* Add information about typical cause of non-clean state
commit d29760f7afc039f2c8a1bd13a02886de43b26454
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Dec 20 20:49:17 2017 -0700
Update screen output.
Update the screen output for status and checkout to indicate externals instead
of components. During checkout step, change text from 'checkout' to action verb
'checking out'.
Testing:
make test - pass with one skip, python2/3
manual testing - clm python2 - ok
commit b1ca38e5dad55366f35d0b61f23d9066d2da2299
Author: Ben Andre <andre@ucar.edu>
Date: Tue Dec 5 16:59:07 2017 -0700
Bugfix in regexp to work with git 1.8 output
The screen output of 'git branch -vv' changed between version 1.8 and 2.10.
Update the regexp parsing this output to be compatible with both versions. Add
new unit test for this output.
Testing:
make test - pass, 1 skip - python2/3
manual testing - escomp-cesm, clm-demo, cesm-demo, checkout, status - ok
commit a93458ba3635605426fecc3a2fd58937399de868
Author: Ben Andre <andre@ucar.edu>
Date: Mon Dec 4 17:20:41 2017 -0700
Add info to error message.
Add info to the error message when the configuration file could not be found.
Suggests that the user verify they ran from the root of the source tree.
Testing:
make test - python2/3 - pass, 1 skip
manual testing - escomp/cesm, clm-demo - ok
commit 7b7ca23516a84bb55f1348702be5a7287a189f65
Author: Ben Andre <andre@ucar.edu>
Date: Mon Dec 4 17:15:54 2017 -0700
Remove old debugging output.
Testing:
make test - python2/3 - pass, one skip
manual testing - escomp/cesm - ok
commit 913bdd47bb1e615a8a6d45f5e5a40cc2450b9e82
Author: Ben Andre <andre@ucar.edu>
Date: Mon Dec 4 16:47:41 2017 -0700
Change checkout process from depth first to breadth first.
Change the order of the checkout process from depth first to breadth first.
This cleans up the output during the initial checkout of a container repository
by doing all checkouts at the current level before doing sub externals. The
disadvantage is that if a checkout dies, a component may be in an incomplete
state. For example, clm is checked out, but it's externals are not. Then
another component checkout fails. This should be easily fixed by a rerun of the
checkout.
Testing:
make test - python2/3 - pass, 1 skip
manual testing - cesm-demo, clm-demo, escomp/cesm - ok checkout, status
commit d74281ec4b63276ce3fae6d8a8ad16912001de8e
Author: Ben Andre <andre@ucar.edu>
Date: Mon Dec 4 16:14:32 2017 -0700
Change config file metadata
Change the 'version' key to 'schema_version' in the config file metadata. In
preparation for freezing 1.0 file format.
User interface changes?: Yes, backwards incompatible change to config file
format.
Testing:
make test - pass, 1 skip - python2 and python3
manual testing - cesm demo, clm demo, escomp/cesm - checkout/status - ok
commit e9e8d2a9247e0bc51be4db2f13a58daebbce858a
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Nov 30 13:49:14 2017 -0700
Basic testing of mix-use repos
Add infrastructure to do basic testing of using mixed use repos as the root
repository.
Add verbose and debug levels of output for make test.
Testing:
python2/3 - make test - pass, 1 skip
manual testing - none, test system changes only
commit 44502a3fac89768b9619b329e50a8835fcdbf763
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Nov 30 12:51:50 2017 -0700
Improve test coverage of svn repositories.
Add test coverage for basic svn repository functionality: switch to a different branch/tag, determine and report dirty and modifed status, and verbose status.
Testing:
python2/3 - make test - pass, two unimplemented skips.
manual testing - not done, only test changes.
commit 089bf2e2676a6439f0970c452eaeb7c6b1857dbb
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 29 19:34:06 2017 -0700
Add svn systems tests and refactor system tests
Break up the systems tests class into a base infrastructure class and several
test classes focused on testing specific conditions for checkout. Test classes
are focused on basic testing, svn testing, and error conditions.
Add svn tests. If svn isn't available because of there is no executable or no
network connection, then the tests will be skipped. Moving forward skipped
tests should be reported...
Testing:
python2/3 - make test including svn - pass
python2 - manual cesm, clm checkout - ok
commit 2324d02fff83f297dbb3d79bcac8c08d8c2ec093
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 29 15:01:15 2017 -0700
Cleanup subprocess calls
Consolidate the check_output and execute_subprocess into a single function.
Clean up error handling to prove more consistent and useful messages.
Testing:
python2/3 - make test - pass
python2 - manual testing cesm, clm checkout and status - ok.
commit 1a7235f5f5164757833dd9efffe0c54e24ffbb66
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 29 14:13:31 2017 -0700
Rename some confusing variables and objects.
The 'source' object was a remanant of the original xml input file naming.
Rename to External to clarify use.
Testing:
python2/3 - make test - pass
python2 - cesm, clm checkout, status - ok
commit 826789f860d670856b24a9d0c3a20b38c92344a8
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 29 13:51:09 2017 -0700
Replace hard coded constants with variables.
Replace multiple different uses of hard coded '.' strings with meaningfully
named constants.
Testing:
python2/3 - make test - pass
python2 - cesm, clm checkout and status - ok
commit 57fe1b22f4fd49efe6dfda1b929888436fa8e343
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 29 11:16:48 2017 -0700
Work on developer doc.
Work on rst developer documentation. Regenerate the readme file since it was
out of date.
Testing: none
commit 383bac5eb6f6f76b90b8675ee432b4b412e041b0
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 29 10:58:55 2017 -0700
Setup sphinx rst based documentation.
commit 89c335fbc3caaa5d5254e5516dd100dc49f439ca
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 29 10:54:35 2017 -0700
Add more developer doc.
commit f4d314761483691374abcaa86c413c6b8c537d94
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 28 21:54:40 2017 -0700
Implement using '.' as a repository url.
Specifying '.' as the repository url in an external description indicates that
the user does not want the tool to interact with remotes. Setting the repo to
'.' and the branch to the currently checked out branch will preserve the repo
state while other repos are updated.
Stop using random test-id for system tests because they were too hard to debug.
Figured out how to use the test name.
Testing:
python2/3 - make test - pass
python2 - cesm/clm manual testing checkout, status - ok
commit 88f804e20e90249c60a5abc661d000337424ac37
Author: Ben Andre <andre@ucar.edu>
Date: Tue Nov 28 15:18:16 2017 -0700
Improve branch and tag error checking.
Introduction of remotes and tag checks didn't cover enough edge cases for
determining if something is a branch or tag, and didn't report helpful error
messages. Untracked remote branches were being reported as invalid refs rather
than valid branches that could be checkoud out.
Testing:
python2/3 - make test - pass
python2/3 - manual checkout, status - ok
commit 83b1c8347f765b398d457629ec08e1f8a49725d4
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Nov 23 16:16:39 2017 -0700
Improve system test infrastructure
Implement assertions for status checking when running the system tests. Add
status assertions to existing tests. Add additional systems, including error
testing to ensure specific conditions are being caught and reported.
Testing:
python2/3 - make test - all tests pass
python2/3 - manual testing checkout, status - ok
commit 3009801362ac2337baeba05e1558f037897e3b37
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 22 17:09:41 2017 -0700
Debugging git remotes and status
More work debugging git remotes and sync state status determination. Add logic
to determine if a tag exists and is unique (not also a branch). Add more unit
tests for determining synce state, verify regexps work, checking tags.
Add debug output to system tests so it is as simple copy/paste to rerun a test
manually.
Testing:
python2/3 - unit/sys tests - pass
python2/3 - manual checkout, status - ok
commit 8bcf33a38f753c3e0251657500fc6dc5e4c0478e
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 21 20:25:46 2017 -0700
Lint cleanup and testing
Cleanup remaining lint errors. Enable lint testing on travis ci. Add more unit
tests. update clean makerule to remove everything but virtual envs.
Testing:
coverage - 82%
python2/3 - unit tests - pass
python2/3 - manual checkout status - ok
commit 9f3405bf89b6e56b872188949a4b06150669df71
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 21 18:52:19 2017 -0700
Refactor repository_svn
Refactor repository_svn to match the have structure, conventions and
organization as repository_git. Update unit tests.
Testing:
python2/3 - unit tests - pass
python2 - clm checkout, status - ok
python3 - cesm checkout, status - ok
commit c5e57b3476babbeb1873eafd585f04d1d0e4b10f
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 21 18:22:46 2017 -0700
Remove git repo determine_ref_type unused code
Remove the _determine_ref_type routine from GitRepository because it isn't
being used.
Testing:
python2/3 - unit tests - pass
python2/3 - manual checkout status - ok
commit d90567cdf401b9f8b24bdce97e87292a421f5607
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 21 13:50:23 2017 -0700
Debug git remotes
Work on debugging with git remotes. Try new checkout procedure determining the
remote name, adding if not in known remotes, fetching and checking out. Add
unit and systems tests.
System tests just verify code runs to completion without errors. Need to finish
implementing system test infrastructure to verify correct results.
Add a new bare test repo that can be used as a remote fork.
Testing:
python2/3 - unit tests - pass
python2/3 - system tests - pass
python2/3 - clm manual testing checkout, status - ok
python2/3 - cesm manual testing checkout, status - ok
commit 368e9b6b8127ddbf2fee1de983fb24092b4cdb93
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 21 12:05:16 2017 -0700
Remove dead code from git repository
Remove code from the git repository class that is unused and untested, and
probably will not be used going forward.
Testing:
python2, python3 - unit tests - pass
python2, python3 - system tests - pass
python2, python3 - clm manual checkout status - ok
python2, python3 - cesm manual checkout status - ok
commit 0d802b4a6e077eaf57ab54234c375a824fccdde4
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 21 11:53:49 2017 -0700
Refactor git repository interface
Refactor the git repository class.
Regroup methods to more clearly group public api, internal worker routines,
external calls to git.
Isolate subcalls to git from application logic by making all git calls occur in
small single purpose standalone functions.
Rename methods more consistently.
Testing:
python2 - unit tests - pass
python2 - system integration tests - pass
python2 - clm manual testing checkout status - ok
python2 - cesm manual testing checkout status - ok
python3 - clm manual testing checkout status - ok
python3 - cesm manual testing checkout status - ok
commit 1cbf2eadb9105724bcf0ba74daf48086bbfcf50d
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 21 16:05:31 2017 -0700
Flush stdout from printlog
Flush stdout everytime printlog is done to ensure user can see progress when a
series of long subprocess calls are being executed.
Testing:
python2/3 - unit tests - pass
python2/3 - manual testing - ok, seems to improve display.
commit fa1e1741a9972b4a9dec1c9135a03b6bde80800e
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 21 15:46:11 2017 -0700
Bugfix handling subprocess output.
python3 would generate an attribute error while handling fatal errors from
subprocess output. It was working on a byte string where a unicode string was
expected. Change to using universal_newlines in subprocess call so that we
always get back a unicode string.
Testing:
python2/3 - unit tests - pass
python2/3 - manual testing ok.
commit ed2e265a4a96ab308f5cfcd6602513bc4a8aa897
Author: Benjamin Andre <andre@ucar.edu>
Date: Mon Nov 20 16:51:28 2017 -0700
Rename modules
Rename some modules for consistency
* checkout_externals - removed .py extension
* externals_status - consistency with exernals_description
* global_constants - clarify intent
* test_unit_externals_status - consistency with module being tested.
Testing:
python2 - unit tests - pass
python2 - manual testing cesm - ok
python2 - manual testing clm - ok
commit 98aed47dd3509f5e5973aa9944025d4e0f7e777a
Author: Benjamin Andre <andre@ucar.edu>
Date: Mon Nov 20 16:37:06 2017 -0700
Add .dir_locals.el for consistent emacs indenting.
commit ba7c6df4503d7d427023632867ab928fc5dd0671
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Nov 17 14:08:39 2017 -0700
UI: consistent use of externals instead of model
Update the command line interface and doc, replacing 'model' with 'externals'
User interface changes: yes
checkout_externals - command line -m --model is now -e --externals
Testing:
python2 test suite - pass
python3 test suite - pass
python2 manual test - checkout status - ok
python3 manual test - checkout status - ok
commit 6ed827764df251784686d8347583f7bb79aaf8ff
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Nov 17 12:21:47 2017 -0700
Add python 3.4 and 3.5 to CI testing.
Add git version info to the pre-script stage for reference.
commit 3402d57cafeadcfe8d0a58aca7946673c8093616
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Nov 16 17:16:33 2017 -0700
Initial implementation of system tests
Initial implementation of system integration tests for manage_externals
checkout script. Includes git bare repositories for testing. The mixed use
container external repository has not been fully initialize with the branches
and configuration files needed to run.
Testing:
python2 - make test - passes, one skip
python3 - make test - passes, one skip
python2 - manual testing checkout status - ok
python3 - manual testing checkout status - ok
commit 7b44d20a6d5b6e40d42d391d0e72542ffbc3fec7
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Nov 16 08:53:53 2017 -0700
Bugfix for python3, enable python3 on travis-ci
Python3 is more strict about not allowing a loop over keys to be modified
inplace. Copy the key list prior to starting a loop when we need to delete
items.
Enable linux-python3 testing on travis-ci.
Testing:
python2 unit tests - pass
python3 unit tests - pass
python2 manual testing checkout, checkout status - ok
python3 manual testing checkout, checkout status - ok
commit e82ded2ff64eba04f2f194cace86d7ce0c56b918
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 15 19:56:16 2017 -0700
Work on coveralls.io integration
Update travis-ci script and test makefile to integrate coverage and
coveralls.iointo automated testing.
Testing:
python2 unit tests - pass
python2 coverage - 45%
commit bb0f5a04d458925c3b82a4cb680ec7014b99e339
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 15 18:36:20 2017 -0700
Convert readme file to markdown
Rename the readme file with a markdown extension, then update a couple of
formatting rules so it will display reasonably on github and the terminal.
Update the text of the discussion about source root to clarify where
checkout_externals should be run from.
Testing: regenerate readme file.
commit b0889713c45985fec488db99150b51a079654b0b
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 15 14:23:55 2017 -0700
Travis-ci integration
* Add travis-ci configuration file.
* Include build status badge in README file.
* Move python dependancies into requirements.txt file.
Testing:
verified that make env and make lint work with requirments file.
commit 731284c7622326e38800649f8ee9c72f4a38fb47
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 15 14:21:21 2017 -0700
Update autogenerated README file.
commit 411baa664c133e1bb041fda052af5fc525004bd6
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Nov 15 08:30:52 2017 -0700
Misc lint cleanup
Cleanup a bunch of lint issues. Still have three files with lint issues:
repository_git, sourcetree, test_unit_externals_description.
Testing:
python2 unit tests - pass
python2 clm manual testing checkout, status - ok
python2 cesm manual testing checkout, status - ok
python3 unit tests - 3 errors
commit dcb6241ad2214140d1aac2ff8533d5bf6f1af592
Author: Benjamin Andre <andre@ucar.edu>
Date: Tue Nov 14 17:49:00 2017 -0700
Rename model description to externals description.
Testing:
unit tests - pass
cesm - manual testing of checkout and status - ok.
clm - manual testing of checkout and status - ok.
commit e55d5eccd9f9aaa3979d911aa58f6dae69ed3d08
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 14 14:39:33 2017 -0700
Refactor the model description objects.
Refactor the model description code to remove the xml, json and yaml readers.
Split the dict and cfg processing into seperate objects. Use a base class to
check and validate the different input formats. Update tests and dependant
code.
Testing:
unit tests - pass
manual testing of checkout and checkout status - ok
commit 288050bf86e60fda326670e7c3af196388a511ce
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 14 12:32:57 2017 -0700
Add initial versions of githup templates.
commit c86298f5f3b1fb4eff8ceeba0b72ba47c58ecfa4
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 14 12:19:31 2017 -0700
Break unit tests into separate files for the modules they are testing.
Testing: unit tests pass. manual test of checkout and checkout status ok.
commit 89e36a082e5a0a44eaf36cc2e179cf1dbace38db
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 14 11:58:34 2017 -0700
Rename the manageexternals package to manic.
Testing: unit tests pass. manual testing of checkout_externals and checkout status ok.
commit 22aa66ef446548aad19738b81bc01cfb8cb76e3e
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Nov 14 11:30:36 2017 -0700
Continue breaking up checkout_externals into modules.
Testing: unit tests pass, manual checkout and checkout status ok.
commit ab558ee9aeab9cadf3312bf5fed95efad1f47165
Author: Benjamin Andre <andre@ucar.edu>
Date: Mon Nov 13 18:53:13 2017 -0700
Initial work breaking checkout_externals into modules.
Create the manageexternals package. Move ModelDescription object into it's own
module. Move some common code shared by ModelDescription and checkout_model
into global and utils modules.
Testing: unit tests pass. manual run of 'checkout_externals' and
'checkout_externals --status' ok..
commit 28de448c52e2fed60f8aeadb15c588ae85b1070c
Author: Benjamin Andre <andre@ucar.edu>
Date: Mon Nov 13 16:29:01 2017 -0700
Change short command line option for status.
Maintain compatibility with the commandline options of other cesm tools by
reserving '-s' for silent. Change the short option for status from '-s' to '-S'.
Fixes: #1
Testing: manually verified change to '-S' works.
commit 446424f0e41206aa39ccb3e9de9d097fadcf8cbf
Author: Benjamin Andre <andre@ucar.edu>
Date: Mon Nov 13 15:24:34 2017 -0700
Change default model file to use cfg format.
commit f2e31c37ba7ce51ab7dc3b451eb4f868aeb63b9c
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Oct 26 23:22:39 2017 -0600
Fix typos in doc.
commit 3e3716b3d55859b557ea8b629e4c4912f381d65d
Author: Ben Andre <andre@ucar.edu>
Date: Thu Oct 26 16:55:01 2017 -0600
Bugfix for full cesm. May need to create an external subrepo during
checkout stage.
Bugfix for git 2.10, only a single porcelain version for git status
command.
Testing: checked out model and created a cesm test.
commit c875565de2b13c7f20269a7fdaad6f43efec52e1
Author: Ben Andre <andre@ucar.edu>
Date: Thu Oct 26 15:14:15 2017 -0600
Tweak how statistics data is stored so that it can be printed with a meaningful
sort order.
Testing: python2 - all unit tests pass, manual testing ok.
commit 27d9567451904d26294c7d2560da2d027c64e8da
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Oct 26 12:09:06 2017 -0600
Refactor path naming and handling
Refactor path naming to be more explicit and consistent during the status and
checkout routines. Try to move any changes in directory down to the lowest
level where they are needed.
Testing: unit tests pass (python2). Manual testing of standalone clm is correct.
commit 1a7344fc2f163df4cec3e3d5b2237ccc404b55a7
Author: Ben Andre <andre@ucar.edu>
Date: Wed Oct 25 16:26:48 2017 -0600
Update readme_first doc with note about authentication.
commit 8cb36b70383a1fe891089fa508c84bf1dad0905f
Author: Ben Andre <andre@ucar.edu>
Date: Wed Oct 25 15:33:08 2017 -0600
Move pylint rc file into the test directory with other development related
files.
Testing: ensure pylint runs without error and generates same results.
commit f58ed188965d1a4478830e2f7a5d3e9a13a06167
Author: Ben Andre <andre@ucar.edu>
Date: Wed Oct 25 15:18:54 2017 -0600
Update test makefile and virtual environment creation to more easily support
python2 and python3 testing. Add code coverage make rule. Update testing
documentation.
Testing: all unit tests pass (python2), checkout_externals manual testing ok.
commit 631d84d65845372dd3f42c66a9500ee95116d267
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Oct 25 11:19:25 2017 -0600
Change logic of svn repo clean/dirty check to match git by marking any working
copy status other than external as dirty.
Testing: unit tests pass. Manually tested checkout_externals with clean/dirty
svn repo.
commit b002f1eff3a65cd7e461e297ad4b3d3031f3dbe3
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Oct 24 17:09:44 2017 -0600
Improve output to user when the externals can't be updated because one is dirty.
Testing: manually ran checkout_externals on dirty and clean repo.
commit 055abdeab26d04330ff2e3c62d6171cc22dcd0b4
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Oct 24 16:58:03 2017 -0600
Get lint working again. Fix some lint issues.
Testing: unit tests pass. checkout_externals runs.
commit 89caaceba3d065c6c89a83e8fea95fbc7f9a25ac
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Oct 24 15:48:43 2017 -0600
Rename checkout_model to checkout_externals. Update all tests, docstrings,
comments, etc.
Testing: all unit tests pass, manual testing of checkout_externals ok.
commit 896b87c903e3b1426cd67729f38559511ad3c9d4
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Oct 24 15:31:35 2017 -0600
Add logic to check for optional fields when building the model description
object from the input file. Remove optional fields from example user input
files. Add unit tests for cfg and yaml input formats.
Testing: all unit tests pass. manual run of xml, yaml, json and cfg input all
appear ok, report same status.
commit 2def85a571c6580664f49d012ae10d1b7cb99af0
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Oct 24 11:22:52 2017 -0600
Initial support for yaml input format.
Testing: unit tests pass, status generates same report for yaml, json, xml, cfg.
commit 43dcbebef5e6b4bc5963d8963bc1e973750a9fab
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Oct 24 09:35:30 2017 -0600
Wrapper to make strings read by config parser unicode.
Testing: unit tests pass. xml, json and cfg input result in same status.
commit 7d48e8d2cc523963614bffae40a2dce0d74d7a74
Author: Ben Andre <bjandre@gmail.com>
Date: Tue Oct 24 09:22:41 2017 -0600
Fix typo in unit tests that caused invalid syntax.
commit 21c274c11c5c35cbe8be11e3eb381c2956f720da
Author: Ben Andre <andre@ucar.edu>
Date: Tue Oct 24 09:17:46 2017 -0600
Maintaining python2/3 compatibility with xml, json, cfg (and yaml) input
formats is going to be extremely difficult. For now abandon 2/3 compatibility
and just focus on input formats. Fix inconsistency between json and other input
formats. Add util function so xml always returns unicode strings.
Testing: minimal, not expected to fully work for all input formats.
commit 10c78c834c5be71ee52395ff613f72a7e68f5e6b
Author: Ben Andre <bjandre@gmail.com>
Date: Mon Oct 23 11:30:09 2017 -0600
Python compatibility.
Fix so checkout_model and tests are compatible with both python2.7 and
python3.6-ish.
Testing: unit tests pass for both python2 and python3. Manual checkout and
status ok with both python2 and 3.
commit 181db136aec2e83c69dc897ab67060a2181e6d46
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Oct 20 13:55:05 2017 -0600
Remove the long list of pylint disabled options in the pylintrc.
Testing: pylint runs, did not add trigger any additional issues.
commit 0039f69eab1385a7b124b14c7ec7a8b8e3ab986a
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Oct 20 13:43:47 2017 -0600
Fix pylint issues.
Testing: unit tests pass, manual testing ok.
commit 877451ff2bc41148b6a3872638bc5492de34c45f
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Oct 20 13:11:33 2017 -0600
Cleanup makefile in the test directory.
Setup and fix make rules for testing, running autopep8, pylint, creating
readme file.
Add README_FIRST with getting started doc.
Testing: unit tests pass.
commit 1f5449d41f22fb41258a53220462fae1ff853cc9
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Oct 20 12:57:55 2017 -0600
Rename the test directory so it isn't hidden.
Testing: unit tests pass.
commit 94b487702d2b938aab29a96fcd034accd790a839
Author: Ben Andre <bjandre@gmail.com>
Date: Fri Oct 20 12:49:43 2017 -0600
Tweak status report output to try to be clearer. Update doc to match output.
Testing: Unit tests pass. Manual testing ok.
commit 5fcb4f43c37669052de8f6908f865f31c35d2d79
Author: Ben Andre <andre@ucar.edu>
Date: Thu Oct 19 17:56:33 2017 -0600
Add optional info to status.
commit fac659b0bd1200adf344160405789279f3f8383e
Author: Ben Andre <andre@ucar.edu>
Date: Thu Oct 19 16:41:45 2017 -0600
Bugfix: svn checkout needed to do a switch if the directory already exists.
commit 04a7acded579b7a3c59f91d26fc383a8e51afbda
Author: Ben Andre <andre@ucar.edu>
Date: Thu Oct 19 16:15:55 2017 -0600
Add verbose status output.
commit 6ca32aec3d1a8ee587919b0613980265c7520345
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Oct 19 14:06:06 2017 -0600
Get model updates working safely.
Add functionality to evalue the state of the model repositories and determine
if all managed repositories are safe to update. If any repo is not safe, then
we inform the user and exit gracefully.
Added new unit tests for evaluating safe update status.
Testing: all tests ok, manual testing of modified and clean repos.
commit dbe6b928d900faf96d336e803ad7ff5e3fa1bdac
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Oct 19 12:04:19 2017 -0600
Finsh implementing status summary information for git and svn.
Add related unit tests for parsing git and svn output. pylint and autopep8
cleanup of status changes.
Testing: unit tests pass (1 skip), manual testing of checkout and status pass.
commit 8c96d0ccaf04557d4eb851856b000f5ab53f56ee
Author: Ben Andre <bjandre@gmail.com>
Date: Thu Oct 19 08:43:21 2017 -0600
Cleanup string literals
Import unicode_literals so we can be python2 and python3 compatible without
explicitly labeling every literal.
Testing: unit tests pass, manual checkout and status tests pass.
commit 6a7b6d316367648f9678c7c631b983a098e14cbf
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Oct 18 21:28:51 2017 -0600
Improve user experience by simplifing screen output.
* Create externals SourceTree objects when the initial objects are created
rather than during status or checkout operations. This makes the output a bit
nicer since the read_model_description output is not intermixed during status
and checkout.
* Print current component when running checkout and status.
* read_model_description prints path to log file, not screen.
* Log more info when making subprocess calls.
All unit tests pass, 1 skip. Manual testing of checkout and status runs as
expected.
commit 59bdf46d0d6024f0c6e67c9c7106fb5413e9bbb0
Author: Ben Andre <bjandre@gmail.com>
Date: Wed Oct 18 21:02:34 2017 -0600
Initial status checking for 'sync state', whether the repos urls or branch
match the model description file. Checking for whether the working copy is
dirty is not implemented. Add unit tests for sync state.
All unit tests pass, 1 skip. Statu…
fischer-ncar
pushed a commit
that referenced
this issue
Mar 3, 2020
Run a single coverage command rather than two separate commands I was finding that, even with the '--append' flag, some of the coverage results were getting overwritten in the second run, resulting in a too-low coverage report. This turns out to have no effect on the reported coverage on master, but it fixes the coverage report for #90: without this change, some lines that are covered only by unit tests are listed as uncovered in the final coverage report (at least, I think that's what was going on). User interface changes?: No Fixes: None Testing: test removed: changed 'make coverage' operation unit tests: system tests: manual testing:
fischer-ncar
pushed a commit
that referenced
this issue
Mar 3, 2020
git: rework logic for determining in-sync and current ref name Rework two aspects of the logic for git repositories; both of these can change behavior in some cases when printing status: 1. Determining whether the local checkout is in-sync with the expected reference from the configuration file: Now we always convert the expected reference to a hash and compare that with the currently-checked-out hash. Previously, we sometimes did the comparison using names, e.g., just ensuring that you're on the right branch. 2. Determining the current ref name (e.g., the branch or tag currently checked out). Now we use a number of plumbing commands rather than relying on regex parsing of 'git branch -vv'. The previous regex parsing was fragile and hard to maintain, and was the source of a number of bugs. In addition, differences between git v. 1 and git v. 2 meant that the result was incorrect in some cases - particularly, in the case where we have "detached from foo" (which is the text that always appeared for a detached head in v 1, but in v 2 means we are no longer at foo). I have also overhauled the unit tests covering this functionality. Many tests were no longer needed with the new logic and so I have removed them. I have added some other tests covering the new functionality. User interface changes?: Yes - Subtle changes to status output, as described above - One particular change is: If we're on a tracking branch, `checkout_externals -S -v` will show the name of the local branch rather than the tracked branch. (This is more accurate, because we may not actually be at the head of the tracking branch.) Fixes: #86 (Status incorrectly reports in-sync when you have made commits in detached head state - fixed due to the change in (1)). Testing: test removed: many no-longer-relevant unit tests unit tests: pass system tests: pass manual testing: basic manual testing of checking out and running status, in the context of cesm and ctsm
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The current J1850G compset is set to:
1850_DATM%CRU_CLM50%BGC-CROP_CICE_POP2_MOSART_CISM2%EVOLVE_SWAV
But it should be:
1850_DATM%CRU_CLM50%BGC-CROP_CICE_POP2%ECO_MOSART_CISM2%EVOLVE_WW3_BGC%BDRD
Also, running a quick ERS test on the J1850G compset with the resolution used in the glacier spin-up runs results in these errors:
This error goes on 240 times. Also seems to be some errors about missing '/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.cruncep_qianFill.0.5d.V4.c130305/Solar6Hrly/clmforc.cruncep.V4.c2011.0.5d.Solr.1917-07.nc' and possibly others. So, maybe the forcing data needs to be updated for this compset as well?
The text was updated successfully, but these errors were encountered: