.. _getting_started: ************************** Getting started with Ptypy ************************** .. _installation: Installation ============ General installation instructions --------------------------------- Download and unpack |ptypy|_ from the sources. For example, on Linux systems you can do:: $ wget https://github.com/ptycho/ptypy/archive/master.zip $ unzip master.zip -d /tmp/; rm master.zip or make a clone of the |ptypy|_ github repository:: $ git clone https://github.com/ptycho/ptypy.git /tmp/ptypy-master which will unpack the master branch of ptypy into `/tmp/ptypy-master` but you are of course free to place the software wherever it is convenient for you. In principle, next you only have to navigate to that directory and install with :: $ pip install . However, since |ptypy|_ depends on a number of other packages, we recommend installing it in a virtual environment using the `anaconda `_ package manager. Conveniently, this installation route allows you to install |ptypy|_ across platforms. Essential install ^^^^^^^^^^^^^^^^^ Only three Python packages are essential for |ptypy|_ to work: * `NumPy `_ (homepage: ``_) * `SciPy `_ (homepage: ``_) * `h5py `_ (homepage: ``_) Install the essential version like so :: $ conda env create -f dependencies_core.yml $ conda activate ptypy_core (ptypy_core)$ pip install . Recommended install for additional functionality ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Please note that |ptypy|_ is an alpha release and lacks rigorous import checking. There may be parts of code that implicitly ask for any of the packages listed here. Therefore, we recommend to install these packages. * `Matplotlib `_ for any kind of plotting or image generation (homepage: ``_) * `mpi4py `_ for CPU-node parallelization, contains python bindings for the `Message Passaging Interface `_, (homepage: ``_) * `pyzmq `_ for a threaded server on the main node to perfrom asynchronous client-server communication, contains python bindings for the ZeroMQ protocol, (homepage: ``_). This package is needed for non-blocking plots of the reconstruction run (e.g. for :ref:`plotclient`). Install the recommended version like so :: $ conda env create -f dependencies_full.yml $ conda activate ptypy_full (ptypy_full)$ pip install . Recommended install for GPU support with CuPy ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We support an accelerated version of |ptypy|_ for CUDA-capable GPUs based on our own kernels and the `CuPy `_ package. Install the dependencies for this version like so. :: $ conda env create -f ptypy/accelerate/cuda_cupy/dependencies.yml $ conda activate ptypy_cupy (ptypy_cupy)$ pip install . Install for GPU support with PyCUDA ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Alternatively, we also support an accelerated version of |ptypy|_ for CUDA-capable GPUs based on our own kernels and the `PyCUDA `_ package. Install the dependencies for this version like so. :: $ conda env create -f ptypy/accelerate/cuda_pycuda/dependencies.yml $ conda activate ptypy_pycuda (ptypy_pycuda)$ pip install . We use `Reikna `_ to provide a filtered FFT, i.e. a FFT that is fused with a pointwise matrix multiplication. Optional installation of filtered cufft ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For optimal performance with both CuPy and PyCUDA engines, you can optionally install a version of the filtered FFT based on cufft and callbacks. Due to the nature of this extension it needs to be built for fixed array sizes externally and currently supports array sizes of 16, 32, 64, 128, 256, 512, 1024 and 2048. :: $ conda activate ptypy_cupy (ptypy_cupy)$ cd cufft (ptypy_cupy)$ conda env update --file dependencies.yml --name ptypy_cupy (ptypy_cupy)$ pip install . Optional packages ^^^^^^^^^^^^^^^^^ Other very useful packages are * `Ipython `_ (homepage: ``_) .. _quickstart: Quickstart with a minimal script ================================ .. note:: This tutorial was generated from the python source :file:`[ptypy_root]/tutorial/minimal_script.py` using :file:`ptypy/doc/script2rst.py`. You are encouraged to modify the parameters and rerun the tutorial with:: $ python [ptypy_root]/tutorial/minimal_script.py This tutorial explains the minimal settings to get a reconstruction runnig in |ptypy|_. A |ptypy| script consists of two parts: * Creation of a parameter tree with parameters as listed in :ref:`parameters` and * calling a :any:`Ptycho` instance with this parameter tree, specifying a level that determines how much the Ptycho instance will do. Preparing the parameter tree ---------------------------- We begin with opening an empty python file of arbitrary name in an editor of your choice, e.g.:: $ gedit minimal_script.py Next we create an empty parameter tree. In |ptypy|, parameters are managed by the :any:`Param` class which is a convenience class subclassing Python's dict type. It is designed in such a way that dictionary items can be accessed also as class attributes, making the scripts and code much more readable. :: >>> from ptypy import utils as u >>> p = u.Param() # root level We set the verbosity to a high level, in order to have information on the reconstruction process printed to the terminal. See :py:data:`~ptycho.verbose_level`. :: >>> p.verbose_level = "info" We limit this reconstruction to single precision. The other choice is to use double precision. :: >>> p.data_type = "single" We give this reconstruction the name ``'minimal'`` although it will automatically choose one from the file name of the script if we put in ``None``. (But then the tutorial may not work on your computer as the chosen run name may differ from the one that this tutorial was created with) :: >>> p.run = 'minimal' Next, we set the home path. The :any:`Ptycho` instance will use this path as base for any other relative file path (e.g :py:data:`.io.autosave.path` or :py:data:`.io.rfile`). Relative paths lack a leading "/" (or "C:\\" on windows). Make sure to replace all "/" with "\\" if you run the scripts on a Windows system. :: >>> p.io = u.Param() >>> p.io.home = "/tmp/ptypy/" We want an intermediate result of the reconstruction to be dumped regularly every 20 reconstructions. :: >>> p.io.autosave = u.Param() >>> p.io.autosave.interval = 20 In this tutorial we switch off the threaded plotting client. (alternative one-liners would be `p['io.autoplot.active'] = False` or `p.io['autoplot.active'] = False`) :: >>> p.io.autoplot = u.Param() >>> p.io.autoplot.active = False Since we do not want to plot anything, we don't need the interaction server either. :: >>> p.io.interaction = u.Param() >>> p.io.interaction.active = False Now we have to insert actual parameters associated with a ptychographic scan. PtyPy is designed to support reconstruction from mutliple scans. Each individual scan is represented by a branch in ``scans``. The parameters in these branches are those that differ from the *defaults* in the ``scan`` branch mentioned above. Obviously at least the ``data`` branch will differ from :py:data:`.scan.data`. In this tutorial we create a new scan parameter branch ``MF``. :: >>> p.scans = u.Param() >>> p.scans.MF = u.Param() >>> p.scans.MF.name = 'Vanilla' >>> p.scans.MF.data = u.Param() As data source we have choosen the *'MoonFlowerScan'* test source. That will make |ptypy| use the internal :py:class:`~ptypy.core.data.MoonFlowerScan` class to generate data. This class is meant for testing, and it provides/simulates diffraction patterns without using the more complex generic :py:class:`SimScan` class. :: >>> p.scans.MF.data.name = 'MoonFlowerScan' We set the diffraction frame shape to a small value (128x128px) and limit the number af diffraction patterns at 100. The :py:class:`~ptypy.core.data.MoonFlowerScan` instance will balance the diffraction patterns accordingly. :: >>> p.scans.MF.data.shape = 128 >>> p.scans.MF.data.num_frames = 100 We skip saving the "prepared" data file for now. The |ptypy| data management is described in detail in :ref:`ptypy_data` :: >>> p.scans.MF.data.save = None Needlees to say, we need to specify a reconstruction engine. We choose 40 iterations of difference map algorithm. :: >>> p.engines = u.Param() >>> p.engines.engine00 = u.Param() >>> p.engines.engine00.name = 'DM' >>> p.engines.engine00.numiter = 40 >>> p.engines.engine00.numiter_contiguous = 5 Running ptypy ------------- We import the :any:`Ptycho` class and pass the tree ``p`` at level 5. This level tells :any:`Ptycho` to initialize everything and start the reconstruction using all reconstruction engines in ``p.engines`` immediately upon construction. :: >>> from ptypy.core import Ptycho >>> P = Ptycho(p, level=5) From the terminal log, we note that there was an autosave every 20 iterations and the error reduced from iteration to iteration. .. _morescripts: Utilies/Binaries for convenience ================================ |ptypy| provides a few utility scripts to make life easier for you, the user. They are located in ``[ptypy_root]/scripts``. In case of an install with conda, these are copied to ``/bin`` .. note:: Due to the early stage of developmnet, these scripts may see substantial changes in further releases, i.e. the call signature may change. Plotting from a reconstruction/dump file (\*.ptyr) -------------------------------------------------- ``ptypy.plot`` is an automatic plotting script that installs on Unix systems It has the syntax :: $ ptypy.plot [-h] [-l LAYOUT] [-t IMFILE] ptyrfile For our minimal example this translates to :: $ ptypy.plot /tmp/ptypy/recons/minimal/minimal_DM.ptyr -t minimal.png and the image looks like this (:numref:`minimal_result`) .. figure:: ../img/minimal_result.png :width: 90 % :figclass: highlights :name: minimal_result Example plot made with ``ptypy.plot`` using the *default* layput Inspecting a hdf5 compatible file --------------------------------- Sometimes we want to quickly inspect what is in a *hdf5* file that was created by |ptypy|. For such cases, we can use ``ptypy.inspect``. :: $ ptypy.inspect [-h] [-p PATH] [--report] [-d DEPTH] h5file For example, a quick view at the top level can be realized with :: $ ptypy.inspect /tmp/ptypy/recons/minimal/minimal_DM_0040.ptyr -d 1 which has the following the output:: * content [Param 5]: * obj [dict 1]: * pars [Param 9]: * positions [dict 1]: * probe [dict 1]: * runtime [Param 8]: * header [dict 2]: * description [string = "Ptypy .h5 compatible storage format"] * kind [string = "minimal"] If we are interested solely in the probe we could use :: $ ptypy.inspect /tmp/ptypy/recons/minimal/minimal_DM.ptyr -d 1 -p content/probe which has the following the output:: * SMF [dict 20]: * DataTooSmall [scalar = False] * ID [string = "SMF"] * _center [array = [64. 64.]] * _energy [scalar = 7.2] * _origin [array = [-3.59918584e-06 -3.59918584e-06]] * _pool [dict 0]: * _psize [array = [5.62372787e-08 5.62372787e-08]] * _record [scalar = (b'SMF',)] * _recs [dict 0]: * data [1x128x128 complex64 array] * distributed [scalar = False] * fill_value [scalar = 0.0] * grids [tuple = 2x[1x128x128 float64 array]] * layermap [list = [0]] * model_initialized [scalar = True] * nlayers [scalar = 1] * numID [scalar = 1] * padding [scalar = 0] * padonly [scalar = False] * shape [tuple = (1, 128, 128)] We omitted the result for the complete file to save some space but you are encouraged to try:: $ ptypy.inspect /tmp/ptypy/recons/minimal/minimal_DM.ptyr Create a new template for a reconstruction script ------------------------------------------------- [WIP] Due to wildcards and links in the parameter tree, this section will be reworked. .. _plotclient: Run a plotclient in a separate process -------------------------------------- |ptypy| supports a Client/Server approach. That means that the reconstruction process runs on a remote server (cluster) while we can monitor the progress on a local machine. In this case, we need to start a plotting Client on a separate machine). You can implement your own plotting client but you may find it convenient to use the plotting utility ``ptypy.plotclient``:: $ ptypy.plotclient [-h] [-l LAYOUT] [--dump] [--movie] [-i INTERVAL] [-d DIRECTORY] .. note:: None of the options are currently implemented. The plotting client receives all information from the server it connects to. Work in progress .. More script templates ===================== Besides the script from which section :ref:`quickstart` was generated, there is a trinity of similar scripts at your disposal that you can temper with. All-in-one ---------- We encourage you to use the script ``[ptypy_root]/templates/ptypy_minimal_prep_and_run.py`` and modify the *recipe* part of the data parameter branch. Observe what changes in the reconstruction when scan parameters change. .. literalinclude:: ../../templates/ptypy_minimal_prep_and_run.py :language: python :linenos: :emphasize-lines: 41,43,45 Creating a .ptyd data-file -------------------------- We encourage you to use this script ``[ptypy_root]/templates/ptypy_make_sample_ptyd.py`` to create various different samples and see what happens if the data processing parameters are changed. If you have become curious, move forward to :ref:`ptypy_data` and take a look at |ptypy|_'s data management. Check out the data parameter branch :py:data:`.scan.data` for detailed parameter descriptions. .. literalinclude:: ../../templates/ptypy_make_sample_ptyd.py :language: python :linenos: :emphasize-lines: 12-19 Loading a data file to run a reconstruction ------------------------------------------- The script ``[ptypy_root]/templates/minimal_load_and_run.py`` should resembles the case of data analysis after the experiment has taken place. Take a challenging sample data from before and alter the reconstruction parameters and algorithms to find out if you can make the recontruction converge. Check out the engine parameter branch :py:data:`.engine` for detailed parameter descriptions. .. literalinclude:: ../../templates/ptypy_minimal_load_and_run.py :language: python :linenos: