/
SystemTests.html
495 lines (452 loc) · 41.2 KB
/
SystemTests.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>System Tests</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css?v=fadd4351" />
<link rel="stylesheet" type="text/css" href="_static/custom.css?v=77160d70" />
<script src="_static/documentation_options.js?v=30d551ce"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Data Files for Testing" href="DataFilesForTesting.html" />
<link rel="prev" title="Writing Performance Tests" href="WritingPerformanceTests.html" />
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-59110517-1', 'auto');
ga('send', 'pageview');
</script>
</head><body>
<div id="navbar" class="navbar navbar-default ">
<div class="container">
<div class="navbar-header">
<!-- .btn-navbar is used as the toggle for collapsed navbar content -->
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="http://www.mantidproject.org">
</a>
<span class="navbar-text navbar-version pull-left"><b>master</b></span>
</div>
<div class="collapse navbar-collapse nav-collapse">
<ul class="nav navbar-nav">
<li class="divider-vertical"></li>
<li><a href="index.html">Home</a></li>
<li><a href="https://download.mantidproject.org">Download</a></li>
<li><a href="https://docs.mantidproject.org">User Documentation</a></li>
<li><a href="http://www.mantidproject.org/contact">Contact Us</a></li>
</ul>
<form class="navbar-form navbar-right" action="search.html" method="get">
<div class="form-group">
<input type="text" name="q" class="form-control" placeholder="Search" />
</div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<p>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="nav-item nav-item-0"><a href="index.html">Documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">System Tests</a></li>
</ul>
</div> </p>
</div>
<div class="container">
<div class="row">
<div class="body col-md-12 content" role="main">
<section id="system-tests">
<span id="systemtests"></span><h1>System Tests<a class="headerlink" href="#system-tests" title="Link to this heading">¶</a></h1>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#overview" id="id2">Overview</a></p></li>
<li><p><a class="reference internal" href="#writing-a-test" id="id3">Writing a Test</a></p>
<ul>
<li><p><a class="reference internal" href="#specifying-validation" id="id4">Specifying Validation</a></p></li>
<li><p><a class="reference internal" href="#no-workspace-validation" id="id5">No Workspace Validation</a></p></li>
<li><p><a class="reference internal" href="#skipping-tests" id="id6">Skipping tests</a></p>
<ul>
<li><p><a class="reference internal" href="#target-platform-based-on-free-memory" id="id7">Target Platform Based on Free Memory</a></p></li>
<li><p><a class="reference internal" href="#id1" id="id8">Target Platform Based on Free Memory</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#set-the-tolerance" id="id9">Set the Tolerance</a></p></li>
<li><p><a class="reference internal" href="#disable-some-checks" id="id10">Disable Some Checks</a></p></li>
<li><p><a class="reference internal" href="#assertions" id="id11">Assertions</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#running-tests-locally" id="id12">Running Tests Locally</a></p>
<ul>
<li><p><a class="reference internal" href="#downloading-the-test-data" id="id13">Downloading the test data</a></p></li>
<li><p><a class="reference internal" href="#visual-studio-xcode" id="id14">Visual Studio/Xcode</a></p></li>
<li><p><a class="reference internal" href="#makefile-like-generators" id="id15">Makefile-like Generators</a></p></li>
<li><p><a class="reference internal" href="#selecting-tests-to-run-from-ide" id="id16">Selecting Tests to Run From IDE</a></p></li>
<li><p><a class="reference internal" href="#debugging-system-tests-in-pycharm" id="id17">Debugging System Tests in Pycharm</a></p></li>
<li><p><a class="reference internal" href="#selecting-tests-to-run" id="id18">Selecting Tests To Run</a></p></li>
<li><p><a class="reference internal" href="#running-the-tests-on-multiple-cores" id="id19">Running the tests on multiple cores</a></p></li>
<li><p><a class="reference internal" href="#reducing-the-size-of-console-output" id="id20">Reducing the size of console output</a></p></li>
<li><p><a class="reference internal" href="#running-a-cleanup-run" id="id21">Running a cleanup run</a></p></li>
<li><p><a class="reference internal" href="#adding-new-data-references-files" id="id22">Adding New Data & References Files</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#best-practice" id="id23">Best Practice</a></p></li>
</ul>
</nav>
<section id="overview">
<h2><a class="toc-backref" href="#id2" role="doc-backlink">Overview</a><a class="headerlink" href="#overview" title="Link to this heading">¶</a></h2>
<p>System tests are high-level tests, which check that Mantid is able to
reproduce accepted, standardised results as part of its calculations,
when executing user stories. The system test suite is written against
Mantid’s Python API.</p>
<p>As part of our nightly-build and nightly-test procedure, Mantid’s system
tests are run as acceptance tests. The nightly-test jobs deploy a
packaged version of Mantid to the target OS, before executing the system
tests scripts on that environment.</p>
</section>
<section id="writing-a-test">
<h2><a class="toc-backref" href="#id3" role="doc-backlink">Writing a Test</a><a class="headerlink" href="#writing-a-test" title="Link to this heading">¶</a></h2>
<p>The (Python) code for the system tests can be found in the git
repository at
<a class="reference external" href="https://github.com/mantidproject/mantid">mantidproject/mantid</a>, under
the <code class="docutils literal notranslate"><span class="pre">Testing/SystemTests</span></code> directory.</p>
<p>System tests inherit from the <code class="xref py py-class docutils literal notranslate"><span class="pre">systemtesting.MantidSystemTest</span></code> class.
The methods that need to be overridden are <code class="docutils literal notranslate"><span class="pre">runTest(self)</span></code>, where the Python
code that runs the test should be placed, and <code class="docutils literal notranslate"><span class="pre">validate(self)</span></code>, which should
simply return a pair of strings: the name of the final workspace that results
from the <code class="docutils literal notranslate"><span class="pre">runTest</span></code> method and the name of a nexus file that should be saved
in the <code class="docutils literal notranslate"><span class="pre">ReferenceResults</span></code> sub-directory in the repository. The test code
itself is likely to be the output of a <em>Save History</em> command, though it can
be any Python code. In the unlikely case of files being used during a system
test, implement the method <code class="docutils literal notranslate"><span class="pre">requiredFiles</span></code> which should return a list of
filenames without paths. The file to validate against should be included as
well. If any of those files are missing the test will be marked as skipped.</p>
<p>The tests should be added to the <code class="docutils literal notranslate"><span class="pre">Testing/SystemTests/tests/framework</span></code>,
with the template result going in the <code class="docutils literal notranslate"><span class="pre">reference</span></code> sub-folder. It will
then be included in the suite of tests from the following night.</p>
<p>Alternatively, any tests relating to testing qt interfaces should be added to
the <code class="docutils literal notranslate"><span class="pre">Testing/SystemTests/tests/qt</span></code> directory.</p>
<section id="specifying-validation">
<h3><a class="toc-backref" href="#id4" role="doc-backlink">Specifying Validation</a><a class="headerlink" href="#specifying-validation" title="Link to this heading">¶</a></h3>
<p>You may need to inform the System Test Suite about the format of that
the benchmark workspace you wish to validate against. By default, the
system tests assume that the second argument returned by the validate
tuple is the name of a nexus file to validate against. However you can
override the validateMethod on a test with any one of three options.</p>
<ul class="simple">
<li><p>WorkspaceToNexus (Benchmark workspace is stored as a Nexus file)
(default)</p></li>
<li><p>WorkspaceToWorkspace (Benchmark workspace is stored as a workspace)</p></li>
<li><p>ValidateAscii (Benchmark workspace is stored as an ascii file)</p></li>
</ul>
<p>For example:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">validateMethod</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s1">'WorkspaceToNeXus'</span>
</pre></div>
</div>
</section>
<section id="no-workspace-validation">
<h3><a class="toc-backref" href="#id5" role="doc-backlink">No Workspace Validation</a><a class="headerlink" href="#no-workspace-validation" title="Link to this heading">¶</a></h3>
<p>If the system test does not need comparison/validation against a
standard workpace, then this step can be skipped. Simply omitting the</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">validate</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
</pre></div>
</div>
<p>method from the system test is sufficient.</p>
</section>
<section id="skipping-tests">
<h3><a class="toc-backref" href="#id6" role="doc-backlink">Skipping tests</a><a class="headerlink" href="#skipping-tests" title="Link to this heading">¶</a></h3>
<p>Tests can be skipped based on arbitrary criteria by implementing the
<code class="docutils literal notranslate"><span class="pre">skipTests</span></code> method and returning True if your criteria are met, and
False otherwise. Examples are the availability of a data file or of
certain Python modules (e.g. for the XML validation tests).</p>
<section id="target-platform-based-on-free-memory">
<h4><a class="toc-backref" href="#id7" role="doc-backlink">Target Platform Based on Free Memory</a><a class="headerlink" href="#target-platform-based-on-free-memory" title="Link to this heading">¶</a></h4>
<p>Some tests consume a large amount memory resources, and are therefore
best executed on hardware where enough memory is available. You can set
a minimum RAM specification by overriding requiredMemoryMB:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">requiredMemoryMB</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">2000</span>
</pre></div>
</div>
<p>The above function limits the test to run on a machine where there is at
least 2GB of free memory.</p>
</section>
<section id="id1">
<h4><a class="toc-backref" href="#id8" role="doc-backlink">Target Platform Based on Free Memory</a><a class="headerlink" href="#id1" title="Link to this heading">¶</a></h4>
<p>Some tests require very large files that cannot be placed in the shared
repository. The <code class="docutils literal notranslate"><span class="pre">requiredFiles()</span></code> method returns a list of these files
so that they test can check that they are all available. If all files
are not available then the tests are skipped.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">requiredFiles</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="p">[</span><span class="s1">'a.nxs'</span><span class="p">,</span> <span class="s1">'b.nxs'</span><span class="p">]</span>
</pre></div>
</div>
<p>The above function limits the test to run on a machine that can find the
files ‘a.nxs’ & ‘b.nxs’</p>
</section>
</section>
<section id="set-the-tolerance">
<h3><a class="toc-backref" href="#id9" role="doc-backlink">Set the Tolerance</a><a class="headerlink" href="#set-the-tolerance" title="Link to this heading">¶</a></h3>
<p>You may specialise the tolerance used by <code class="docutils literal notranslate"><span class="pre">CompareWorkspace</span></code> in your
system test.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="bp">self</span><span class="o">.</span><span class="n">tolerance</span> <span class="o">=</span> <span class="mf">0.00000001</span>
</pre></div>
</div>
<p>By default the tolerance is absolute. It can be changed to relative by another
flag in the <code class="xref py py-class docutils literal notranslate"><span class="pre">systemtesting.MantidSystemTest</span></code> class.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="bp">self</span><span class="o">.</span><span class="n">tolerance_rel_err</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div>
</div>
</section>
<section id="disable-some-checks">
<h3><a class="toc-backref" href="#id10" role="doc-backlink">Disable Some Checks</a><a class="headerlink" href="#disable-some-checks" title="Link to this heading">¶</a></h3>
<p>You may disable some checks performed by the <code class="docutils literal notranslate"><span class="pre">CompareWorkspaces</span></code>
algorithm by appending them to the disableChecking list, which, by
default, is empty.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># A list of things not to check when validating</span>
<span class="bp">self</span><span class="o">.</span><span class="n">disableChecking</span> <span class="o">=</span> <span class="p">[]</span>
</pre></div>
</div>
</section>
<section id="assertions">
<h3><a class="toc-backref" href="#id11" role="doc-backlink">Assertions</a><a class="headerlink" href="#assertions" title="Link to this heading">¶</a></h3>
<p>Additional assertions can be used as the basis for your own comparison
tests. The following assertions are already implemented in the base
class.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">assertTrue</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="s2">""</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">assertEqual</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">expected</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="s2">""</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">assertDelta</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">expected</span><span class="p">,</span> <span class="n">delta</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="s2">""</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">assertLessThan</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">expected</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="s2">""</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">assertGreaterThan</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">expected</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="s2">""</span><span class="p">):</span>
</pre></div>
</div>
</section>
</section>
<section id="running-tests-locally">
<h2><a class="toc-backref" href="#id12" role="doc-backlink">Running Tests Locally</a><a class="headerlink" href="#running-tests-locally" title="Link to this heading">¶</a></h2>
<p>CMake configures a script file called <code class="docutils literal notranslate"><span class="pre">systemtest</span></code> (<code class="docutils literal notranslate"><span class="pre">systemtest.bat</span></code>
on Windows) in the root of the build directory. This file is the driver
script to execute the system tests that runs the lower-level
<code class="docutils literal notranslate"><span class="pre">Testing/SystemTests/scripts/runSystemTests.py</span></code> script but ensures
that the environment is set up correctly for that particular build and
that the required test data has been updated. The script accepts a
<code class="docutils literal notranslate"><span class="pre">-h</span></code> option to print out the standard usage information.</p>
<p>Usage differs depending on whether you are using a single-configuration
generator with CMake, for example Makefiles/Ninja, or a
multi-configuration generator such as Visual Studio or Xcode.</p>
<section id="downloading-the-test-data">
<h3><a class="toc-backref" href="#id13" role="doc-backlink">Downloading the test data</a><a class="headerlink" href="#downloading-the-test-data" title="Link to this heading">¶</a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">systemtest</span></code> script will automatically attempt to download any missing
data files but will time-out after 2 minutes. The time out limit can be set in two
variables <code class="docutils literal notranslate"><span class="pre">ExternalData_TIMEOUT_INACTIVITY</span></code> and <code class="docutils literal notranslate"><span class="pre">ExternalData_TIMEOUT_ABSOLUTE</span></code>.
If using CMake these will need to be added as new string entries (value is in seconds).</p>
</section>
<section id="visual-studio-xcode">
<h3><a class="toc-backref" href="#id14" role="doc-backlink">Visual Studio/Xcode</a><a class="headerlink" href="#visual-studio-xcode" title="Link to this heading">¶</a></h3>
<p>The user must first open command-prompt from, the build directory. The
script requires the developer to select the configuration that will be
used to execute the tests, one of: <em>Release</em>, <em>Debug</em>, <em>RelWithDebInfo</em>
or ‘MinSizeRelease’’. Note that the script does not build the code so
the chosen configuration must have already been built. An example to
execute all of the tests for the release configuration would be (in the
command-prompt):</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>><span class="w"> </span>systemtest<span class="w"> </span>-C<span class="w"> </span>Release
</pre></div>
</div>
</section>
<section id="makefile-like-generators">
<h3><a class="toc-backref" href="#id15" role="doc-backlink">Makefile-like Generators</a><a class="headerlink" href="#makefile-like-generators" title="Link to this heading">¶</a></h3>
<p>The script requires no additional arguments as the configuration is
fixed when running CMake, e.g.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>build
systemtest
</pre></div>
</div>
</section>
<section id="selecting-tests-to-run-from-ide">
<h3><a class="toc-backref" href="#id16" role="doc-backlink">Selecting Tests to Run From IDE</a><a class="headerlink" href="#selecting-tests-to-run-from-ide" title="Link to this heading">¶</a></h3>
<p>System tests can be run from the MSVC IDE using the <code class="docutils literal notranslate"><span class="pre">SystemTests</span></code> target,
which behaves in a similar way to unit test targets. One key advantage is
that it allows you to start Mantid in a debug environment rather than attach
to one midway through.</p>
<p>To select an individual test, or range of tests, go to the <code class="docutils literal notranslate"><span class="pre">SystemTests</span></code>
properties, go to <code class="docutils literal notranslate"><span class="pre">`Command</span> <span class="pre">Arguments`</span></code> and append flags as appropriate.</p>
<p>For example, adding <code class="docutils literal notranslate"><span class="pre">-R</span> <span class="pre">ISIS</span></code> will run any tests which match the regular
expression <code class="docutils literal notranslate"><span class="pre">ISIS</span></code>.</p>
</section>
<section id="debugging-system-tests-in-pycharm">
<h3><a class="toc-backref" href="#id17" role="doc-backlink">Debugging System Tests in Pycharm</a><a class="headerlink" href="#debugging-system-tests-in-pycharm" title="Link to this heading">¶</a></h3>
<p>System tests can be debugged from Pycharm without finding and attaching to
the process, or using a remote debugger.</p>
<p>To do this, create a new Configuration (Run -> Edit Configurations), and add
a new Python configuration with the script path set to <code class="docutils literal notranslate"><span class="pre">runSystemtests.py</span></code>
This is found in <code class="docutils literal notranslate"><span class="pre">/Testing/SystemTests/Scripts/runSystemTests.py</span></code></p>
<p>The parameters for the configuration can be set just like the command line
args when running the tests from the <code class="docutils literal notranslate"><span class="pre">systemtest.bat</span></code>/<code class="docutils literal notranslate"><span class="pre">systemtest</span></code> script, e.g pass
<code class="docutils literal notranslate"><span class="pre">-R="EnginX"</span></code> to run all tests containing the string <code class="docutils literal notranslate"><span class="pre">EnginX</span></code> in their
name.</p>
<p>Note that running the system tests this way will not update the system test
data, so if your data need to be updated, the system tests should be called
via the normal method in the first case.</p>
<p>Do not use the multiprocessing <code class="docutils literal notranslate"><span class="pre">-j</span></code> flag in your configuration parameters
as this will render you unable to debug the system tests directly, as they
will no longer be running under the parent Python process.</p>
<p>N.B. Windows users do not need to specify the configuration with the <code class="docutils literal notranslate"><span class="pre">-C</span></code>
flag as when using the <code class="docutils literal notranslate"><span class="pre">systemtest.bat</span></code> script, as this is not passed to
<code class="docutils literal notranslate"><span class="pre">runSystemtests.py</span></code> and will result in an error.</p>
</section>
<section id="selecting-tests-to-run">
<h3><a class="toc-backref" href="#id18" role="doc-backlink">Selecting Tests To Run</a><a class="headerlink" href="#selecting-tests-to-run" title="Link to this heading">¶</a></h3>
<p>The most important option on the script is the <code class="docutils literal notranslate"><span class="pre">-R</span></code> option. This
restricts the tests that will run to those that match the given regex,
e.g.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>build
systemtest<span class="w"> </span>-R<span class="w"> </span>SNS
<span class="c1"># or for msvc/xcode</span>
systemtest<span class="w"> </span>-C<span class="w"> </span><cfg><span class="w"> </span>-R<span class="w"> </span>SNS
</pre></div>
</div>
<p>would run all of the tests whose name contains SNS.</p>
</section>
<section id="running-the-tests-on-multiple-cores">
<h3><a class="toc-backref" href="#id19" role="doc-backlink">Running the tests on multiple cores</a><a class="headerlink" href="#running-the-tests-on-multiple-cores" title="Link to this heading">¶</a></h3>
<p>Running the System Tests can be sped up by distributing the list of
tests across multiple cores. This is done in a similar way to <code class="docutils literal notranslate"><span class="pre">ctest</span></code>
using the <code class="docutils literal notranslate"><span class="pre">-j</span> <span class="pre">N</span></code> option, where <code class="docutils literal notranslate"><span class="pre">N</span></code> is the number of cores you want
to use, e.g.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./systemtest<span class="w"> </span>-j<span class="w"> </span><span class="m">8</span>
</pre></div>
</div>
<p>would run the tests on 8 cores.</p>
<p>Some tests write or delete in the same directories, using the same file
names, which causes issues when running in parallel. To resolve this,
a global list of test modules (= different Python files in the
<code class="docutils literal notranslate"><span class="pre">Testing/SystemTests/tests/framework</span></code> directory) is first created.
Now we scan each test module line by line and list all the data files
that are used by that module. The possible ways files are being
specified are:
1. if the extensions <code class="docutils literal notranslate"><span class="pre">.nxs</span></code>, <code class="docutils literal notranslate"><span class="pre">.raw</span></code> or <code class="docutils literal notranslate"><span class="pre">.RAW</span></code> are present
2. if there is a sequence of at least 4 digits inside a string
In case number 2, we have to search for strings starting with 4 digits,
i.e. “0123, or strings ending with 4 digits 0123”.
This might over-count, meaning some sequences of 4 digits might not be
used for a file name specification, but it does not matter if it gets
identified as a filename as the probability of the same sequence being
present in another Python file is small, and it would therefore not lock
any other tests. A dict is created with an entry for each module name
that contains the list of files that this module requires.
An accompanying dict with an entry for each data file stores a lock
status for that particular datafile.</p>
<p>Finally, a scheduler spawns <code class="docutils literal notranslate"><span class="pre">N</span></code> threads who each start a loop and
gather a first test module from the master test list which is stored in
a shared dictionary, starting with the number in the module list equal
to the process id.</p>
<p>Each process then checks if all the data files required by the current
test module are available (i.e. have not been locked by another
thread). If all files are unlocked, the thread locks all these files
and proceeds with that test module. If not, it goes further down the
list until it finds a module whose files are all available.</p>
<p>Once it has completed the work in the current module, it unlocks the
data files and checks if the number of modules that remains to be
executed is greater than 0. If there is some work left to do, the
thread finds the next module that still has not been executed
(searches through the tests_lock array and finds the next element
that has a 0 value). This aims to have all threads end calculation
approximately at the same time.</p>
</section>
<section id="reducing-the-size-of-console-output">
<h3><a class="toc-backref" href="#id20" role="doc-backlink">Reducing the size of console output</a><a class="headerlink" href="#reducing-the-size-of-console-output" title="Link to this heading">¶</a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">systemtests</span></code> can be run in “quiet” mode using the <code class="docutils literal notranslate"><span class="pre">-q</span></code> or
<code class="docutils literal notranslate"><span class="pre">--quiet</span></code> option. This will print only one line per test instead of
the full log.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./systemtest<span class="w"> </span>--quiet
Updating<span class="w"> </span>testing<span class="w"> </span>data...
<span class="o">[</span><span class="m">100</span>%<span class="o">]</span><span class="w"> </span>Built<span class="w"> </span>target<span class="w"> </span>StandardTestData
<span class="o">[</span><span class="m">100</span>%<span class="o">]</span><span class="w"> </span>Built<span class="w"> </span>target<span class="w"> </span>SystemTestData
Running<span class="w"> </span>tests...
FrameworkManager-<span class="o">[</span>Notice<span class="o">]</span><span class="w"> </span>Welcome<span class="w"> </span>to<span class="w"> </span>Mantid<span class="w"> </span><span class="m">3</span>.13.20180820.2132
FrameworkManager-<span class="o">[</span>Notice<span class="o">]</span><span class="w"> </span>Please<span class="w"> </span>cite:<span class="w"> </span>http://dx.doi.org/10.1016/j.nima.2014.07.029<span class="w"> </span>and<span class="w"> </span>this<span class="w"> </span>release:<span class="w"> </span>http://dx.doi.org/10.5286/Software/Mantid
<span class="o">[</span><span class="w"> </span><span class="m">0</span>%<span class="o">]</span><span class="w"> </span><span class="m">1</span>/435<span class="w"> </span>:<span class="w"> </span>DOSTest.DOSCastepTest<span class="w"> </span>...............................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.05s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">0</span>%<span class="o">]</span><span class="w"> </span><span class="m">2</span>/435<span class="w"> </span>:<span class="w"> </span>ISISIndirectBayesTest.JumpCETest<span class="w"> </span>....................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.06s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">0</span>%<span class="o">]</span><span class="w"> </span><span class="m">3</span>/435<span class="w"> </span>:<span class="w"> </span>ISISIndirectInelastic.IRISCalibration<span class="w"> </span>...............................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.03s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">0</span>%<span class="o">]</span><span class="w"> </span><span class="m">4</span>/435<span class="w"> </span>:<span class="w"> </span>HFIRTransAPIv2.HFIRTrans1<span class="w"> </span>...........................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">1</span>.30s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">1</span>%<span class="o">]</span><span class="w"> </span><span class="m">5</span>/435<span class="w"> </span>:<span class="w"> </span>DOSTest.DOSIRActiveTest<span class="w"> </span>.............................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.04s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">1</span>%<span class="o">]</span><span class="w"> </span><span class="m">6</span>/435<span class="w"> </span>:<span class="w"> </span>ISISIndirectBayesTest.JumpFickTest<span class="w"> </span>..................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.06s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">1</span>%<span class="o">]</span><span class="w"> </span><span class="m">7</span>/435<span class="w"> </span>:<span class="w"> </span>AbinsTest.AbinsBinWidth<span class="w"> </span>.............................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">1</span>.65s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">1</span>%<span class="o">]</span><span class="w"> </span><span class="m">8</span>/435<span class="w"> </span>:<span class="w"> </span>ISIS_PowderPearlTest.CreateCalTest<span class="w"> </span>..................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">1</span>.65s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">2</span>%<span class="o">]</span><span class="w"> </span><span class="m">9</span>/435<span class="w"> </span>:<span class="w"> </span>ISISIndirectInelastic.IRISConvFit<span class="w"> </span>...................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.56s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">2</span>%<span class="o">]</span><span class="w"> </span><span class="m">10</span>/435<span class="w"> </span>:<span class="w"> </span>LiquidsReflectometryReductionWithBackgroundTest.BadDataTOFRangeTest<span class="w"> </span>.<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">2</span>.94s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">2</span>%<span class="o">]</span><span class="w"> </span><span class="m">11</span>/435<span class="w"> </span>:<span class="w"> </span>DOSTest.DOSPartialCrossSectionScaleTest<span class="w"> </span>.............................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.23s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">2</span>%<span class="o">]</span><span class="w"> </span><span class="m">12</span>/435<span class="w"> </span>:<span class="w"> </span>ISISIndirectBayesTest.JumpHallRossTest<span class="w"> </span>..............................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.07s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">2</span>%<span class="o">]</span><span class="w"> </span><span class="m">13</span>/435<span class="w"> </span>:<span class="w"> </span>ISISIndirectInelastic.IRISDiagnostics<span class="w"> </span>...............................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.03s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">3</span>%<span class="o">]</span><span class="w"> </span><span class="m">14</span>/435<span class="w"> </span>:<span class="w"> </span>HFIRTransAPIv2.HFIRTrans2<span class="w"> </span>...........................................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.83s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">3</span>%<span class="o">]</span><span class="w"> </span><span class="m">15</span>/435<span class="w"> </span>:<span class="w"> </span>DOSTest.DOSPartialSummedContributionsCrossSectionScaleTest<span class="w"> </span>..........<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.15s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">3</span>%<span class="o">]</span><span class="w"> </span><span class="m">16</span>/435<span class="w"> </span>:<span class="w"> </span>ISISIndirectBayesTest.JumpTeixeiraTest<span class="w"> </span>..............................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.07s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">3</span>%<span class="o">]</span><span class="w"> </span><span class="m">17</span>/435<span class="w"> </span>:<span class="w"> </span>ISISIndirectInelastic.IRISElwinAndMSDFit<span class="w"> </span>............................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.29s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">4</span>%<span class="o">]</span><span class="w"> </span><span class="m">18</span>/435<span class="w"> </span>:<span class="w"> </span>MagnetismReflectometryReductionTest.MRFilterCrossSectionsTest<span class="w"> </span>.......<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">5</span>.30s<span class="o">)</span>
<span class="o">[</span><span class="w"> </span><span class="m">4</span>%<span class="o">]</span><span class="w"> </span><span class="m">19</span>/435<span class="w"> </span>:<span class="w"> </span>DOSTest.DOSPartialSummedContributionsTest<span class="w"> </span>...........................<span class="w"> </span><span class="o">(</span>success:<span class="w"> </span><span class="m">0</span>.16s<span class="o">)</span>
</pre></div>
</div>
<p>One can recover the full log when a test fails by using the <code class="docutils literal notranslate"><span class="pre">--ouptut-on-failure</span></code> option.</p>
</section>
<section id="running-a-cleanup-run">
<h3><a class="toc-backref" href="#id21" role="doc-backlink">Running a cleanup run</a><a class="headerlink" href="#running-a-cleanup-run" title="Link to this heading">¶</a></h3>
<p>A cleanup run will go through all the tests and call the
<code class="docutils literal notranslate"><span class="pre">.cleanup()</span></code> function for each test. It will not run the tests
(i.e. call the <code class="docutils literal notranslate"><span class="pre">execute()</span></code> function) themselves. This is achieved
by using the <code class="docutils literal notranslate"><span class="pre">-c</span></code> or <code class="docutils literal notranslate"><span class="pre">--clean</span></code> option, e.g.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./systemtest<span class="w"> </span>-c
</pre></div>
</div>
<p>This is useful if some old data is left over from a previous run,
where some tests were not cleanly exited.</p>
</section>
<section id="adding-new-data-references-files">
<h3><a class="toc-backref" href="#id22" role="doc-backlink">Adding New Data & References Files</a><a class="headerlink" href="#adding-new-data-references-files" title="Link to this heading">¶</a></h3>
<p>The data is managed by CMake’s external data system that is described by
<a class="reference internal" href="DataFilesForTesting.html#datafilesfortesting"><span class="std std-ref">Data Files for Testing</span></a>. Please see <a class="reference internal" href="DataFilesForTesting.html#datafilesfortesting-addinganewfile"><span class="std std-ref">Adding A New File(s)</span></a> for how to add new
files.</p>
</section>
</section>
<section id="best-practice">
<h2><a class="toc-backref" href="#id23" role="doc-backlink">Best Practice</a><a class="headerlink" href="#best-practice" title="Link to this heading">¶</a></h2>
<ul class="simple">
<li><p>Always check your test works locally before making it public.</p></li>
<li><p>User stories should come from the users themselves where possible.</p></li>
<li><p>Take care to set the tolerance to an acceptable level.</p></li>
</ul>
</section>
</section>
</div>
</div>
</div>
<footer class="footer">
<div class="container">
<ul class="nav navbar-nav" style=" float: right;">
<li>
<a href="WritingPerformanceTests.html" title="Previous Chapter: Writing Performance Tests"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">« Writing Perfo...</span>
</a>
</li>
<li>
<a href="DataFilesForTesting.html" title="Next Chapter: Data Files for Testing"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Data Files fo... »</span>
</a>
</li>
<li><a href="#">Back to top</a></li>
</ul>
<p>
</p>
</div>
</footer>
</body>
</html>