getting_started.tmp

.. _getting_started:

**************************
Getting started with Ptypy
**************************

.. _installation:

Installation
============

General installation instructions
---------------------------------

Download and unpack |ptypy|_ from the sources. For example, on Linux
systems you can do::

    $ wget https://github.com/ptycho/ptypy/archive/master.zip
    $ unzip master.zip -d /tmp/; rm master.zip

or make a clone of the |ptypy|_ github repository::

    $ git clone https://github.com/ptycho/ptypy.git /tmp/ptypy-master

which will unpack the master branch of ptypy into `/tmp/ptypy-master` but
you are of course free to place the software wherever it is convenient for you.
In principle, next you only have to navigate to that directory and install with
::

    $ pip install .

However, since |ptypy|_ depends on a number of other packages, we recommend
installing it in a virtual environment using the
`anaconda <https://www.anaconda.com/>`_ package manager. Conveniently, this
installation route allows you to install |ptypy|_ across platforms.


Essential install
^^^^^^^^^^^^^^^^^

Only three Python packages are essential for |ptypy|_ to work:

* `NumPy <https://pypi.python.org/pypi/numpy>`_
  (homepage: `<http://www.numpy.org>`_)
* `SciPy <https://pypi.python.org/pypi/scipy>`_
  (homepage: `<http://www.scipy.org>`_)
* `h5py <https://pypi.python.org/pypi/h5py>`_
  (homepage: `<http://www.h5py.org>`_)

Install the essential version like so
::

    $ conda env create -f dependencies_core.yml
    $ conda activate ptypy_core
    (ptypy_core)$ pip install .


Recommended install for additional functionality
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please note that |ptypy|_ is an alpha release and lacks rigorous import 
checking. There may be parts of code that implicitly ask for any of the
packages listed here. Therefore, we recommend to install these packages.

* `Matplotlib <https://pypi.python.org/pypi/matplotlib>`_
  for any kind of plotting or image generation
  (homepage: `<http://www.matplotlib.org>`_)

* `mpi4py <https://pypi.python.org/pypi/mpi4py>`_
  for CPU-node parallelization, contains python bindings for the
  `Message Passaging Interface <http://www.mcs.anl.gov/research/projects/mpi/>`_, 
  (homepage: `<https://bitbucket.org/mpi4py/mpi4py/>`_)

* `pyzmq <https://pypi.python.org/pypi/pyzmq>`_
  for a threaded server on the main node to perfrom asynchronous 
  client-server communication, contains python bindings for the
  ZeroMQ protocol, 
  (homepage: `<http://github.com/zeromq/pyzmq>`_).
  This package is needed for non-blocking plots of the reconstruction run
  (e.g. for :ref:`plotclient`).

Install the recommended version like so
::

    $ conda env create -f dependencies_full.yml
    $ conda activate ptypy_full
    (ptypy_full)$ pip install .


Recommended install for GPU support with CuPy
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We support an accelerated version of |ptypy|_ for CUDA-capable
GPUs based on our own kernels and the
`CuPy <https://cupy.dev/>`_ package.

Install the dependencies for this version like so.
::

    $ conda env create -f ptypy/accelerate/cuda_cupy/dependencies.yml
    $ conda activate ptypy_cupy
    (ptypy_cupy)$ pip install .


Install for GPU support with PyCUDA
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Alternatively, we also support an accelerated version of |ptypy|_ for CUDA-capable
GPUs based on our own kernels and the
`PyCUDA <https://pypi.org/project/pycuda/>`_ package.

Install the dependencies for this version like so.
::

    $ conda env create -f ptypy/accelerate/cuda_pycuda/dependencies.yml
    $ conda activate ptypy_pycuda
    (ptypy_pycuda)$ pip install .


We use `Reikna <https://pypi.org/project/reikna/>`_ to
provide a filtered FFT, i.e. a FFT that is fused with a pointwise matrix multiplication. 

Optional installation of filtered cufft
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For optimal performance with both CuPy and PyCUDA engines, you can optionally install a version 
of the filtered FFT based on cufft and callbacks. Due to the nature of this extension it
needs to be built for fixed array sizes externally and currently supports array 
sizes of 16, 32, 64, 128, 256, 512, 1024 and 2048.
::

    $ conda activate ptypy_cupy
    (ptypy_cupy)$ cd cufft
    (ptypy_cupy)$ conda env update --file dependencies.yml --name ptypy_cupy
    (ptypy_cupy)$ pip install .


Optional packages
^^^^^^^^^^^^^^^^^
Other very useful packages are

* `Ipython <https://pypi.python.org/pypi/ipython>`_
  (homepage: `<http://www.ipython.org>`_)


.. _quickstart:

Quickstart with a minimal script
================================

!!include!! _script2rst/minimal_script.tut


.. _morescripts:

Utilies/Binaries for convenience
================================

|ptypy| provides a few utility scripts to make life easier for you, 
the user. They are located in ``[ptypy_root]/scripts``. 
In case of an install with conda, these are copied to ``<conda_env_dir>/bin``

.. note:: 
   Due to the early stage of developmnet, 
   these scripts may see substantial changes in further releases, i.e.
   the call signature may change.

Plotting from a reconstruction/dump file (\*.ptyr)
--------------------------------------------------

``ptypy.plot`` is an automatic plotting script that installs on Unix systems
It has the syntax
::

    $ ptypy.plot [-h] [-l LAYOUT] [-t IMFILE] ptyrfile

For our minimal example this translates to
::

    $ ptypy.plot /tmp/ptypy/recons/minimal/minimal_DM.ptyr -t minimal.png

and the image looks like this (:numref:`minimal_result`)

.. figure:: ../img/minimal_result.png
   :width: 90 %
   :figclass: highlights
   :name: minimal_result
   
   Example plot made with ``ptypy.plot`` using the *default* layput

Inspecting a hdf5 compatible file
---------------------------------

Sometimes we want to quickly inspect what is in a *hdf5* file
that was created by |ptypy|. For such cases, we can use ``ptypy.inspect``.
::

    $ ptypy.inspect [-h] [-p PATH] [--report] [-d DEPTH] h5file

For example, a quick view at the top level can be realized with
::

    $ ptypy.inspect /tmp/ptypy/recons/minimal/minimal_DM_0040.ptyr -d 1

which has the following the output::

     * content [Param 5]:
         * obj [dict 1]:
         * pars [Param 9]:
         * positions [dict 1]:
         * probe [dict 1]:
         * runtime [Param 8]:
     * header [dict 2]:
         * description [string = "Ptypy .h5 compatible storage format"]
         * kind [string = "minimal"]

If we are interested solely in the probe we could use ::

    $ ptypy.inspect /tmp/ptypy/recons/minimal/minimal_DM.ptyr -d 1 -p content/probe

which has the following the output::

    * SMF [dict 20]:
        * DataTooSmall [scalar = False]
        * ID [string = "SMF"]
        * _center [array = [64. 64.]]
        * _energy [scalar = 7.2]
        * _origin [array = [-3.59918584e-06 -3.59918584e-06]]
        * _pool [dict 0]:
        * _psize [array = [5.62372787e-08 5.62372787e-08]]
        * _record [scalar = (b'SMF',)]
        * _recs [dict 0]:
        * data [1x128x128 complex64 array]
        * distributed [scalar = False]
        * fill_value [scalar = 0.0]
        * grids [tuple = 2x[1x128x128 float64 array]]
        * layermap [list = [0]]
        * model_initialized [scalar = True]
        * nlayers [scalar = 1]
        * numID [scalar = 1]
        * padding [scalar = 0]
        * padonly [scalar = False]
        * shape [tuple = (1, 128, 128)]

We omitted the result for the complete file to save some space but you
are encouraged to try::

    $ ptypy.inspect /tmp/ptypy/recons/minimal/minimal_DM.ptyr



Create a new template for a reconstruction script
-------------------------------------------------

[WIP] Due to wildcards and links in the parameter tree, this section will be reworked.


.. _plotclient:

Run a plotclient in a separate process
--------------------------------------

|ptypy| supports a Client/Server approach. That means that the
reconstruction process runs on a remote server (cluster) while we can 
monitor the progress on a local machine.

In this case, we need to start a plotting Client on a separate machine). 
You can implement your own plotting client but you may find it convenient
to use the plotting utility ``ptypy.plotclient``::
 
    $ ptypy.plotclient [-h] [-l LAYOUT] [--dump] [--movie] [-i INTERVAL] [-d DIRECTORY]
    
.. note::
   None of the options are currently implemented. The plotting client receives all
   information from the server it connects to. Work in progress ..
   
More script templates
=====================

Besides the script from which section :ref:`quickstart` was generated,
there is a trinity of similar scripts at your disposal that
you can temper with. 

All-in-one
----------
We encourage you to use the script ``[ptypy_root]/templates/ptypy_minimal_prep_and_run.py``
and modify the *recipe* part of the data parameter branch.
Observe what changes in the reconstruction when scan parameters change.

.. literalinclude:: ../../templates/ptypy_minimal_prep_and_run.py
   :language: python
   :linenos:
   :emphasize-lines: 41,43,45
   
Creating a .ptyd data-file
--------------------------

We encourage you to use this script ``[ptypy_root]/templates/ptypy_make_sample_ptyd.py``
to create various different samples and see what happens if the data
processing parameters are changed. If you have become curious, move
forward to :ref:`ptypy_data` and take a look at |ptypy|_'s data management.
Check out the data parameter branch :py:data:`.scan.data` for detailed parameter
descriptions.

.. literalinclude:: ../../templates/ptypy_make_sample_ptyd.py
   :language: python
   :linenos:
   :emphasize-lines: 12-19
   
Loading a data file to run a reconstruction
-------------------------------------------

The script ``[ptypy_root]/templates/minimal_load_and_run.py``
should resembles the case of data analysis after the experiment has taken
place. Take a challenging sample data from before
and alter the reconstruction parameters and algorithms to find out if you
can make the recontruction converge. Check out the engine parameter
branch :py:data:`.engine` for detailed parameter descriptions.

.. literalinclude:: ../../templates/ptypy_minimal_load_and_run.py
   :language: python
   :linenos: