-
Notifications
You must be signed in to change notification settings - Fork 0
/
Packaging.html
300 lines (257 loc) · 23.2 KB
/
Packaging.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Packaging</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css?v=fadd4351" />
<link rel="stylesheet" type="text/css" href="_static/custom.css?v=77160d70" />
<script src="_static/documentation_options.js?v=30d551ce"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Architecture" href="Architecture.html" />
<link rel="prev" title="Getting Started without Conda" href="GettingStarted/GettingStartedNoneConda.html" />
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-59110517-1', 'auto');
ga('send', 'pageview');
</script>
</head><body>
<div id="navbar" class="navbar navbar-default ">
<div class="container">
<div class="navbar-header">
<!-- .btn-navbar is used as the toggle for collapsed navbar content -->
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="http://www.mantidproject.org">
</a>
<span class="navbar-text navbar-version pull-left"><b>master</b></span>
</div>
<div class="collapse navbar-collapse nav-collapse">
<ul class="nav navbar-nav">
<li class="divider-vertical"></li>
<li><a href="index.html">Home</a></li>
<li><a href="https://download.mantidproject.org">Download</a></li>
<li><a href="https://docs.mantidproject.org">User Documentation</a></li>
<li><a href="http://www.mantidproject.org/contact">Contact Us</a></li>
</ul>
<form class="navbar-form navbar-right" action="search.html" method="get">
<div class="form-group">
<input type="text" name="q" class="form-control" placeholder="Search" />
</div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<p>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="nav-item nav-item-0"><a href="index.html">Documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Packaging</a></li>
</ul>
</div> </p>
</div>
<div class="container">
<div class="row">
<div class="body col-md-12 content" role="main">
<section id="packaging">
<h1>Packaging<a class="headerlink" href="#packaging" title="Link to this heading">¶</a></h1>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#conda" id="id4">Conda</a></p>
<ul>
<li><p><a class="reference internal" href="#building-conda-packages" id="id5">Building Conda Packages</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#standalone" id="id6">Standalone</a></p>
<ul>
<li><p><a class="reference internal" href="#building-standalone-installers" id="id7">Building Standalone Installers</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#build-packages-from-branch-using-jenkins" id="id8">Build packages from branch using Jenkins</a></p>
<ul>
<li><p><a class="reference internal" href="#options" id="id9">Options</a></p></li>
<li><p><a class="reference internal" href="#example" id="id10">Example</a></p></li>
</ul>
</li>
</ul>
</nav>
<p>This page gives an overview of the different packaging mechansims used to deliver
Mantid to users.</p>
<section id="conda">
<h2><a class="toc-backref" href="#id4" role="doc-backlink">Conda</a><a class="headerlink" href="#conda" title="Link to this heading">¶</a></h2>
<p>Mantid provides packages to be consumed by the <a class="reference external" href="https://conda.io">Conda</a> package manager.
This is separate to the <code class="docutils literal notranslate"><span class="pre">mantid-developer</span></code> environment files that setup a local
development environment. See <a class="reference internal" href="GettingStarted/GettingStarted.html#gettingstarted"><span class="std std-ref">Getting Started</span></a> for more detail on how to setup a development environment.
The packages described here are those deployed for users to install.</p>
<p>Even though it is a single codebase the repository is split across 4 conda
packages:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">mantid</span></code>: Provides Python access to the non-GUI elements of mantid and allows
users to import mantid as a Python library for use in other programs.
This package is the base package that can be installed on its own without the requirement
for any other packages to be installed.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">mantidqt</span></code>: Provides Python access to Qt-widgets library customized for use
with the mantid framework, e.g. instrumentview, sliceviewer, file finder widget etc.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">mantiddocs</span></code>: Provides the built-in help pages for the Qt Help viewer in
workbench.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">mantidworkbench</span></code>: The graphical application produced by the Mantid project that
brings together all of the above packages.</p></li>
</ul>
<p>Each package is built separately for <code class="docutils literal notranslate"><span class="pre">x64</span></code> Windows, Linux and macOS with the
exception of <code class="docutils literal notranslate"><span class="pre">mantiddocs</span></code> where a single, <code class="docutils literal notranslate"><span class="pre">noarch</span></code> package is created for all
operating systems as it only contains static HTML.</p>
<p>All mantid-conda packages are available to users through Mantid’s
<a class="reference external" href="https://anaconda.org/mantid">Conda organization</a>.</p>
<section id="building-conda-packages">
<span id="id1"></span><h3><a class="toc-backref" href="#id5" role="doc-backlink">Building Conda Packages</a><a class="headerlink" href="#building-conda-packages" title="Link to this heading">¶</a></h3>
<p>Conda requires <a class="reference external" href="https://docs.conda.io/projects/conda-build/en/stable/concepts/recipe.html">recipes</a> as input for producing packages.
The recipes are stored in the <a class="reference external" href="https://github.com/mantidproject/mantid/tree/main/conda">main codebase</a> so that
they can evolve as the code develops. The packages are built as part of the
<a class="reference external" href="https://github.com/mantidproject/mantid/blob/main/buildconfig/Jenkins/Conda/nightly_build_and_deploy.jenkinsfile">CI pipeline</a> using a <a class="reference external" href="https://github.com/mantidproject/mantid/blob/main/buildconfig/Jenkins/Conda/package-conda">script</a> to encapsulate
the steps.</p>
<p>To build the packages locally it is recommended that you use a separate
clone of the repository as the build copies the repository content to a temporary
working directory and any additional build directories can interfere with the
packaging. The following steps should produce a set of conda packages in a <code class="docutils literal notranslate"><span class="pre">conda-bld</span></code>
directory relative to the current working directory
(on Windows run this from Git bash):</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/mantidproject/mantid.git<span class="w"> </span>mantid-conda-build
mantid-conda-build/buildconfig/Jenkins/Conda/package-conda<span class="w"> </span><span class="nv">$PWD</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--build-mantid<span class="w"> </span>--build-qt<span class="w"> </span>--build-docs<span class="w"> </span>--build-workbench<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>tee<span class="w"> </span>package.log
<span class="c1"># wait quite a while ...</span>
<span class="c1"># packages will appear in a conda-bld directory</span>
</pre></div>
</div>
<p>You can build the workbench without the documentation by:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>mantid-conda-build/buildconfig/Jenkins/Conda/package-conda<span class="w"> </span><span class="nv">$PWD</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--build-mantid<span class="w"> </span>--build-qt<span class="w"> </span>--build-workbench-without-docs<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>tee<span class="w"> </span>package.log
</pre></div>
</div>
<p>To create a new test environment with packages from the local build:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>mamba<span class="w"> </span>create<span class="w"> </span>-n<span class="w"> </span>local-package-test<span class="w"> </span>-c<span class="w"> </span><span class="nv">$PWD</span>/conda-bld<span class="w"> </span>mantidworkbench
mamba<span class="w"> </span>activate<span class="w"> </span>local-package-test
workbench
</pre></div>
</div>
</section>
</section>
<section id="standalone">
<h2><a class="toc-backref" href="#id6" role="doc-backlink">Standalone</a><a class="headerlink" href="#standalone" title="Link to this heading">¶</a></h2>
<p>Mantid also provides a set of bundled packages that provide a complete install of
<code class="docutils literal notranslate"><span class="pre">MantidWorkbench</span></code> and all of its dependencies without a user having to
first setup conda and then install the relevant conda packages.
The standalone packages are available to users through Mantid’s <a class="reference external" href="https://download.mantidproject.org">download page</a>.</p>
<p>Each installer is simply a conda environment, with <code class="docutils literal notranslate"><span class="pre">mantidworkbench</span></code> installed within
it followed by the appropriate installer technology wrapped around it.
Installers are provided for:</p>
<ul class="simple">
<li><p><cite>Windows</cite>: A <a class="reference external" href="https://sourceforge.net/projects/nsis/">NSIS</a> installer package providing a wizard-style interface
for installing/uninstalling MantidWorkbench.</p></li>
<li><p><cite>macOS</cite>: A <a class="reference external" href="https://en.wikipedia.org/wiki/Apple_Disk_Image">disk image</a> providing a facility to drag-and-drop MantidWorkbench
to the <cite>/Applications</cite> folder</p></li>
<li><p><cite>Linux</cite>: An <code class="docutils literal notranslate"><span class="pre">xz</span></code>-compressed tarball of the package files and directories that
can be extracted to any location.</p></li>
</ul>
<section id="building-standalone-installers">
<h3><a class="toc-backref" href="#id7" role="doc-backlink">Building Standalone Installers</a><a class="headerlink" href="#building-standalone-installers" title="Link to this heading">¶</a></h3>
<p>The installers are built as part of the <a class="reference external" href="https://github.com/mantidproject/mantid/blob/main/buildconfig/Jenkins/Conda/nightly_build_and_deploy.jenkinsfile">CI pipeline</a> using a
<a class="reference external" href="https://github.com/mantidproject/mantid/blob/main/buildconfig/Jenkins/Conda/package-standalone">script</a> to encapsulate the build steps.</p>
<p>To build an installer locally first build a set of conda packages using the
<a class="reference internal" href="#building-conda-packages"><span class="std std-ref">instructions above</span></a>. Once you have the <code class="docutils literal notranslate"><span class="pre">conda-bld</span></code>
directory run the following from the same working directory:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>mantid-conda-build/buildconfig/Jenkins/Conda/package-standalone<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">$PWD</span><span class="w"> </span>--package-suffix<span class="w"> </span>Unstable<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>tee<span class="w"> </span>standalone-package.log
<span class="c1"># wait some time and the installer will appear in the working directory</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">--package-suffix</span></code> argument is an optional string to append to the name
of the final package. We generally pick <code class="docutils literal notranslate"><span class="pre">Unstable</span></code> for installers not built
by the CI pipeline to indicate it has been built outside of the standard process.</p>
</section>
</section>
<section id="build-packages-from-branch-using-jenkins">
<span id="build-packages-from-branch"></span><h2><a class="toc-backref" href="#id8" role="doc-backlink">Build packages from branch using Jenkins</a><a class="headerlink" href="#build-packages-from-branch-using-jenkins" title="Link to this heading">¶</a></h2>
<p>Developers can build packages to test branches using the <code class="docutils literal notranslate"><span class="pre">build_packages_from_branch</span></code> <a class="reference external" href="https://builds.mantidproject.org/job/build_packages_from_branch/">Jenkins job</a>. This job provides the ability to,</p>
<ul class="simple">
<li><p>Run system tests on Windows, Mac, Linux, or all three.</p></li>
<li><p>Build a packages on Windows, Mac, Linux, or all three.</p></li>
<li><p>Publish the package(s) to a given Anaconda channel and label.</p></li>
<li><p>Publish the package(s) to a given Github repository under a specified tag.</p></li>
</ul>
<p>for a given branch of mantid. The branch can be from the main mantid repo or from a remote.</p>
<section id="options">
<h3><a class="toc-backref" href="#id9" role="doc-backlink">Options</a><a class="headerlink" href="#options" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BUILD_DEVEL</span></code> [<code class="docutils literal notranslate"><span class="pre">none</span></code>, <code class="docutils literal notranslate"><span class="pre">all</span></code>, <code class="docutils literal notranslate"><span class="pre">linux-64</span></code>, <code class="docutils literal notranslate"><span class="pre">win-64</span></code>, <code class="docutils literal notranslate"><span class="pre">osx-64</span></code>]: Run the system tests for this OS.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BUILD_PACKAGE</span></code> [<code class="docutils literal notranslate"><span class="pre">none</span></code>, <code class="docutils literal notranslate"><span class="pre">all</span></code>, <code class="docutils literal notranslate"><span class="pre">linux-64</span></code>, <code class="docutils literal notranslate"><span class="pre">win-64</span></code>, <code class="docutils literal notranslate"><span class="pre">osx-64</span></code>]: Build a package on this OS.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PACKAGE_SUFFIX</span></code>: String to append onto the standalone package name, useful for distinguishing builds. By default this is <code class="docutils literal notranslate"><span class="pre">Unstable</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PUBLISH_TO_ANACONDA</span></code>: Set true to publish to the given Anaconda channel and label.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PUBLISH_TO_GITHUB</span></code>: Set true to publish to the Github repository using the specified tag.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ANACONDA_CHANNEL</span></code>: Anaconda channel to upload the package to. By default this is <code class="docutils literal notranslate"><span class="pre">mantid</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ANACONDA_CHANNEL_LABEL</span></code>: Label attached to the uploaded package. By default this is <code class="docutils literal notranslate"><span class="pre">unstable</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">GITHUB_RELEASES_REPO</span></code>: Repository to store the release. By Default this is <code class="docutils literal notranslate"><span class="pre">mantidproject/mantid</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">GITHUB_RELEASES_TAG</span></code>: Name of the tag for the release; only to be used for release candidate builds.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ANACONDA_TOKEN_CREDENTIAL_ID</span></code> [<code class="docutils literal notranslate"><span class="pre">anaconda-cloud-token</span></code>, <code class="docutils literal notranslate"><span class="pre">anaconda-token-ornl</span></code>]: One of two credentials to use for publishing to Anaconda.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">GH_ORGANIZATION_OR_USERNAME</span></code>: Name of the organisation or Github user name who owns the repository with the code to build. By default this is <code class="docutils literal notranslate"><span class="pre">mantidproject</span></code>, if you are building from a fork this will need to change to your username.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BRANCH_NAME</span></code>: Name of the branch to build the packages from.</p></li>
</ul>
</section>
<section id="example">
<h3><a class="toc-backref" href="#id10" role="doc-backlink">Example</a><a class="headerlink" href="#example" title="Link to this heading">¶</a></h3>
<p>Say I’ve implemented a new file searching method on a branch <code class="docutils literal notranslate"><span class="pre">1234_new_file_search</span></code> and I want to test this on IDAaaS, one of the easiest ways to do this would be to build the packages and upload them to Anaconda using the pipeline. These are the steps I’d take to do this.</p>
<ol class="arabic simple">
<li><p>Go to the <code class="docutils literal notranslate"><span class="pre">build_packages_from_branch</span></code> <a class="reference external" href="https://builds.mantidproject.org/job/build_packages_from_branch/">Jenkins job</a>.</p></li>
<li><p>If needed click <code class="docutils literal notranslate"><span class="pre">login</span></code> in the top right of the window.</p></li>
<li><p>Go to <code class="docutils literal notranslate"><span class="pre">Build</span> <span class="pre">with</span> <span class="pre">parameters</span></code> in the side bar.</p></li>
<li><p>Fill out the following options:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BUILD_DEVEL</span></code> = <code class="docutils literal notranslate"><span class="pre">none</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BUILD_PACKAGE</span></code> = <code class="docutils literal notranslate"><span class="pre">linux-64</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PUBLISH_TO_ANACONDA</span></code> = true</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ANACONDA_CHANNEL_LABEL</span></code> = <code class="docutils literal notranslate"><span class="pre">new_file_system_test</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ANACONDA_TOKEN_CREDENTIAL_ID</span></code> = <code class="docutils literal notranslate"><span class="pre">anaconda-cloud-token</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BRANCH_NAME</span></code> = <code class="docutils literal notranslate"><span class="pre">1234_new_file_search</span></code></p></li>
</ul>
</li>
<li><p>Click <code class="docutils literal notranslate"><span class="pre">Build</span></code>. This will take you back to the main job page, the build just set off will be the most recent (highest number) build on the left hand side. It is a good idea to make note of the build number / copy the link somewhere safe. If the build is for testing a pr, make sure to add the link to the testing instructions.</p></li>
<li><p>Once the job has successfully completed, check <a class="reference external" href="https://anaconda.org/mantid">the Mantid Anaconda page</a> to make sure it has uploaded.</p></li>
<li><p>Head to IDAaaS (or any linux system) and run <code class="docutils literal notranslate"><span class="pre">mamba</span> <span class="pre">install</span> <span class="pre">-c</span> <span class="pre">mantid/label/new_file_system_test</span> <span class="pre">mantidworkbench</span></code> in a new environment to install the test package.</p></li>
</ol>
<p>Most often, you won’t need to upload the packages to Anaconda, this is most useful in cases where installing standalone packages is inconvenient. Standalone package builds created by the jenkins job can be found under the jenkins job build artifacts, this is near the top of the page. Say you built a package for Windows using the jenkins job, you should find a <code class="docutils literal notranslate"><span class="pre">mantidworkbench</span></code> exe file in the build artifacts.</p>
</section>
</section>
</section>
</div>
</div>
</div>
<footer class="footer">
<div class="container">
<ul class="nav navbar-nav" style=" float: right;">
<li>
<a href="GettingStarted/GettingStartedNoneConda.html" title="Previous Chapter: Getting Started without Conda"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">« Getting Start...</span>
</a>
</li>
<li>
<a href="Architecture.html" title="Next Chapter: Architecture"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Architecture »</span>
</a>
</li>
<li><a href="#">Back to top</a></li>
</ul>
<p>
</p>
</div>
</footer>
</body>
</html>