Testing forward backward compatibility

Problem we're solving

We need to maintain both backward and forward compatibility in table formats and want to add testing to Jenkins to make sure we do not break that. This work is for issue #1019. I believe we can solve this solely through Jenkins.

Description

We need to be able to maintain compatibility for the previous N releases of WT. N currently is two. A release can be found by its git tag and tags are created as major.minor.patch. All tags, with one exception follows this pattern. The one exception is the existence of a tag named release-1.0 which appears as the last entry in the tag list.

It is sufficient to run wt verify successfully on a given URI to determine if it is compatible. We will run a random test/format configuration in every tree we plan to test for compatibility. Those trees would be the current develop, and previous releases. Other projects will copy the other 'wt` executable to their local tree.

When a project tests compatibility against a copied tree from another build, it will need to figure out the URI to pass to wt verify since it is randomly generated. The table name is known to be wt and the script can grep for the data_source type in RUNDIR/CONFIG. The URI passed to wt verify will either be file:wt or table:wt. We can figure this out by:

isfile=`grep datasource RUNDIR/CONFIG | grep -c file || exit 0`
if test $isfile -ne 0; then
    uri="file:wt"
else
    uri="table:wt"
fi

Jenkins does not allow you to have multiple repositories in one workspace. Therefore each release revision will have to be a separate workspace/job. However, with the Copy Artifact Plugin we can access the results of other Jenkins jobs.

We need Jenkins jobs that will be chained:

• wiredtiger-compat-develop:
  • Build from develop.
  • Run test/format.
  • Copy wt from wiredtiger-compat-r2 into local tree R2.
  • Copy wt from wiredtiger-compat-r1 into local tree R1.
  • Run wt verify on both local trees.
  • NOTE: This job is essentially a copy of wiredtiger-test-format except it sets runs=1 and when the downstream jobs run, they are assured this job is successfully complete and the database is in a known state.
• wiredtiger-compat-r2:
  • Build from 2nd last release (git tag | grep -v release | tail -2 | head -1).
  • Run test/format.
  • Copy wt from wiredtiger-compat-develop into local tree DEV.
  • Run wt verify on local tree. (checks forward compatibility.)
• wiredtiger-compat-r1:
  • Build from most recent release (git tag | grep -v release | tail -1).
  • Run test/format.
  • Copy RUNDIR from wiredtiger-compat-develop into local tree DEV.
  • Run wt verify on local tree. (checks forward compatibility.)

Not having used the plugin yet, it is not clear that I can set up multiple copies from multiple projects. If that is not allowed, then the backward compatibility job needs to be broken up for each previous release. If we need to, we can go back more releases and add additional jobs to do so.

Pros

• No code changes to WT or test/format. Therefore no need to know if a new format config value is supported or not, etc.
• Creates an infrastructure to use any method to generate old database directories. The use of test/format could be replaced by wtperf or anything else if we wanted.
• Allows automatic advancement for releases. When the release number changes Jenkins starts testing the new most recent two releases with no editing of any jobs.
• Can easily change N from 2 to some other number just by adding or removing a job and it just starts working.

Cons

Admittedly I am biased and these are pretty weak cons.

• Only able to run through Jenkins.
• Requires recreating test/format database directories on every iteration. We don't save anything from earlier runs or earlier releases.
• Random test/format means we hope we cover all the cases.