Differences between revisions 1 and 9 (spanning 8 versions)

Under Construction

EMAN2 is built with conda-build using binaries from https://anaconda.org, packaged into an installer with constructor as of v2.2.

conda is the package manager.
https://anaconda.org is the online repository of binaries.
conda-build is the tool to build from source.
constructor is the tool to package eman2 and dependency binaries into a single installer file.

EMAN2 is distributed as a single installer which includes all its dependencies. However, EMAN2 is not available as a conda-package on https://anaconda.org. In other words it is not possible to install EMAN2 by typing conda install eman2.

Conda

Packages that are available on https://anaconda.org can be installed into any conda environment by issuing the command conda install <package>. Conda installs the package along with its dependencies. In order for packages to benefit from this automation, they need to be packaged in a specific way. That can be done with conda-build. conda-build builds packages according to instructions provided in a recipe. A recipe consists of a file with package metadata, meta.yaml, and any other necessary resources like build scripts, (build.sh, bld.bat), patches and so on.

Recipes, Feedstocks and anaconda.org channel: cryoem

Most of EMAN2 dependencies can be found on anaconda's channels, defaults and conda-forge. A few that do not exist or need to be customized have been built and uploaded to channel cryoem. The recipes are hosted in separate repositories on GitHub. Every recipe repository follows the feedstock approach of conda-forge. See here for a complete list.

Build Strategies

It is possible to utilize conda for building and installing EMAN2 in a few ways. One way is to just install binaries with conda and point to the right locations of dependencies during cmake configuration. Another way is to make use of the newly added features in EMAN2's cmake which find the dependencies automatically. These features are activated only when the build is performed by conda-build. CMake knows the build is a conda-build build only through an environment variable. So, it is possible to set the specific environment variable manually and still activate those features without actually using conda-build. Third way is to use a recipe to run conda-build. Basic instructions for all three strategies follow.

1. Use conda for binaries only

Install dependencies

Manually with
```
conda install <package>
```

Or with a single command

conda install eman-deps -c cryoem -c defaults -c conda-forge

Build and install EMAN2 manually into home directory with cmake, make and make install.
Resulting installation is under $HOME/EMAN2 by default.

This is detailed on EMAN WIKI. Also, see build_no_envars.sh.

2. Use conda to install EMAN2 into a conda environment

Install dependencies

Manually with
```
conda install <package>
```

Or with a single command

conda install eman-deps -c cryoem -c defaults -c conda-forge

Set environment variables.

export CONDA_BUILD_STATE=BUILD
export PREFIX=<path-to-anaconda-installation-directory> # $HOME/miniconda2/ or $HOME/anaconda2/
export SP_DIR=$PREFIX/lib/python2.7/site-packages

Build and install EMAN2 manually into conda environment with cmake, make and make install. See build_with_envars.sh.
Resulting installation is under Anaconda/Miniconda installation, $HOME/anaconda2/ or $HOME/miniconda2/ by default.

3. Use recipe for a fully automated conda build

conda build <path-to-eman-recipe-directory>
conda install eman2 --use-local -c cryoem -c defaults -c conda-forge

Resulting installation is under Anaconda/Miniconda installation, $HOME/anaconda2/ or $HOME/miniconda2/ by default.

:TODO: Advantages/disadvantages/comparison of the strategies.

Tests

The build strategies described in section Build Strategies are tested on CI (Continuous Integration) servers for MacOSX (TravisCI) and Linux (CircleCI). For Windows (Appveyor), only the recipe strategy is tested. The tests are triggered for every commit that is pushed to GitHub.

Local Build Tests and Automated Daily Snapshot Binaries with Jenkins

:TODO: Jenkins.

Binary Distribution

Constructor

Packaging is done with constructor, a tool for making installers from conda packages. In order to slightly customize the installers the project was forked. The customized project is at https://github.com/cryoem/constructor. The input files for constructor are maintained at https://github.com/cryoem/docker-images https://github.com/cryoem/build-scripts.

The installer has additional tools like conda, conda-build, pip bundled. The installer is setup so that the packages are kept in the installed EMAN2 conda environment cache for convenience.

Build Machines

The binary packages are built on three physical machines. The operating systems are Mac OSX 10.10, CentOS 7 and Windows 10. CentOS 6 binaries are built in a CentOS 6 docker container on the CentOS 7 machine. Various build related scripts are on https://github.com/cryoem/build-scripts.

Cron

:TODO:

Windows

[Windows]

Docker

Docker images and helper scripts are at https://github.com/cryoem/docker-images https://github.com/cryoem/build-scripts.

:FIXME: Runs as root on Linux. chown doesn't work, the resulting installer has root ownership.

:TODO: More explanation.

:TODO: Review and edit below

Pydusa, OpenMPI and Fftw-mpi

The version of OpenMPI provided with EMAN2 installer may not work with user's batch queueing system, meaning they would not be able to run jobs on more than one node at a time. In that case, installer-provided OpenMPI needs to be removed, pydusa and fftw-mpi need to be installed against OpenMPI that is present in the system. Two scripts, one that uninstalls installer provided OpenMPI, pydusa and fftw-mpi, and another that builds fftw-mpi and pydusa against the system OpenMPI are provided. Instructions [here].

Only OpenMPI is force-removed.

So the system or newly installed version is used
Force-removing fftw-mpi and, specifically, pydusa leaves the system broken.
Even if not removed, later installations overwrite, so there isn't any need to uninstall anyways.

-  ⇤ ← Revision 1 as of 2017-06-01 21:33:49 → 
  Size: 6436
  Editor: TunayDurmaz
  Comment: build documentation savepoint
+   ← Revision 9 as of 2017-06-02 23:06:13 → ⇥
  Size: 7337
  Editor: TunayDurmaz
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 21:
-**:TODO:** Advantages/disadvantages/comparison of the strategies.
-Line 23:
+Line 22:
-**:TODO:** Resulting file hierarchies, EMAN2DIR

It is possible to utilize conda for building and installing EMAN2 in a few ways. One way is to just install binaries with conda and point to the right locations of dependencies during cmake configuration. Another way is to make use of the newly added features in EMAN2's cmake which find the dependencies automatically. These features are activated only when the build is performed by `conda-build`. Cmake knows the build is a `conda-build` build only through an environment variable. So, it is possible to set the specific environment variable manually and still activate those features without actually using `conda-build`. Third way is to use a recipe to run `conda-build`. Basic instructions for all three strategies follow.
+It is possible to utilize conda for building and installing EMAN2 in a few ways. One way is to just install binaries with conda and point to the right locations of dependencies during cmake configuration. Another way is to make use of the newly added features in EMAN2's cmake which find the dependencies automatically. These features are activated only when the build is performed by {{{conda-build}}}. CMake knows the build is a {{{conda-build}}} build only through an environment variable. So, it is possible to set the specific environment variable manually and still activate those features without actually using {{{conda-build}}}. Third way is to use a recipe to run {{{conda-build}}}. Basic instructions for all three strategies follow.
-Line 28:
+Line 25:
-. Install dependencies with `conda install`. 2. Build and install EMAN2 manually into **root/home directory** with `cmake`, `make` and `make install`.
-Line 30:
+Line 26:
-This is detailed on [EMAN WIKI](http://blake.bcm.edu/emanwiki/EMAN2/COMPILE_EMAN2_MAC_OS_X). Also, see [build_no_envars.sh](https://github.com/cryoem/eman2/blob/master/ci_support/build_no_envars.sh).
+. Install dependencies
    a. Manually with {{{
conda install <package>
}}}
    a. Or with a single command{{{
conda install eman-deps -c cryoem -c defaults -c conda-forge
}}}

 2. Build and install EMAN2 manually into '''home directory''' with {{{cmake}}}, {{{make}}} and {{{make install}}}.

 3. Resulting installation is under '''$HOME/EMAN2''' by default.

This is detailed on [[http://blake.bcm.edu/emanwiki/EMAN2/COMPILE_EMAN2_MAC_OS_X|EMAN WIKI]]. Also, see [[https://github.com/cryoem/eman2/blob/master/ci_support/build_no_envars.sh|build_no_envars.sh]].
-Line 33:
+Line 41:
-. Install dependencies with `conda install`. 2. Set environment variables, `shell export CONDA_BUILD_STATE=BUILD export PREFIX=<path-to-anaconda-installation-directory> # $HOME/miniconda2/ or $HOME/anaconda2/ export SP_DIR=$PREFIX/lib/python2.7/site-packages ` 3. Build and install EMAN2 manually into **conda environment** with `cmake`, `make` and `make install`. See [build_with_envars.sh](https://github.com/cryoem/eman2/blob/master/ci_support/build_with_envars.sh).
+. Install dependencies
    a. Manually with {{{
conda install <package>
}}}
    a. Or with a single command{{{
conda install eman-deps -c cryoem -c defaults -c conda-forge
}}}

 2. Set environment variables. {{{
export CONDA_BUILD_STATE=BUILD
export PREFIX=<path-to-anaconda-installation-directory> # $HOME/miniconda2/ or $HOME/anaconda2/
export SP_DIR=$PREFIX/lib/python2.7/site-packages
}}}

 3. Build and install EMAN2 manually into '''conda environment''' with {{{cmake}}}, {{{make}}} and {{{make install}}}. See [[https://github.com/cryoem/eman2/blob/master/ci_support/build_with_envars.sh\|build_with_envars.sh]].

 4. Resulting installation is under Anaconda/Miniconda installation, '''$HOME/anaconda2/''' or '''$HOME/miniconda2/''' by default.
-Line 36:
+Line 61:
-. `conda build <path-to-eman-recipe-directory>`
+{{{
conda build <path-to-eman-recipe-directory>
conda install eman2 --use-local -c cryoem -c defaults -c conda-forge
}}}

Resulting installation is under Anaconda/Miniconda installation, '''$HOME/anaconda2/''' or '''$HOME/miniconda2/''' by default.

''':TODO:''' Advantages/disadvantages/comparison of the strategies.
-Line 39:
+Line 72:
-The build strategies described in [section Build Strategies](#build-strategies) are tested on CI (Continuous Integration) servers for MacOSX ([TravisCI](https://travis-ci.org/cryoem/eman2/builds)) and Linux ([CircleCI](https://circleci.com/gh/cryoem/eman2)). For Windows (Appveyor), only the recipe strategy is tested. The tests are triggered for every commit that is pushed to GitHub.
-Line 41:
+Line 73:
+The build strategies described in section Build Strategies are tested on CI (Continuous Integration) servers for MacOSX ([[https://travis-ci.org/cryoem/eman2/builds|TravisCI]]) and Linux ([[https://circleci.com/gh/cryoem/eman2|CircleCI]]). For Windows (Appveyor), only the recipe strategy is tested. The tests are triggered for every commit that is pushed to GitHub.

=== Local Build Tests and Automated Daily Snapshot Binaries with Jenkins ===
-Line 43:
+Line 78:
-Line 44:
+Line 80:
-Line 45:
+Line 82:
-Packaging is done with `constructor`, a tool for making installers from conda packages. In order to slighltly customize the installers the project was forked. The customized project is at https://github.com/cryoem/constructor. The input files for `constructor` are maintained at https://github.com/cryoem/docker-images.
-Line 47:
+Line 83:
-The installer includes additional tools such as `conda`, `conda-build`, `pip`. The installer is setup so that the packages are kept in the installed EMAN2 conda environment cache for convenience.
+Packaging is done with {{{constructor}}}, a tool for making installers from conda packages. In order to slightly customize the installers the project was forked. The customized project is at https://github.com/cryoem/constructor. The input files for {{{constructor}}} are maintained at --(https://github.com/cryoem/docker-images )-- https://github.com/cryoem/build-scripts.

The installer has additional tools like {{{conda}}}, {{{conda-build}}}, {{{pip}}} bundled. The installer is setup so that the packages are kept in the installed EMAN2 conda environment cache for convenience.
-Line 50:
+Line 89:
-The binary packages are built on three physical machines. The operating systems are `Mac OSX 10.10`, `CentOS 7` and `Windows 10`. `CentOS 6` binaries are built in a `CentOS 6` docker container on the `CentOS 7` machine.
-Line 52:
+Line 90:
-https://github.com/cryoem/buils-scripts
+The binary packages are built on three physical machines. The operating systems are Mac OSX 10.10, CentOS 7 and Windows 10. CentOS 6 binaries are built in a CentOS 6 docker container on the CentOS 7 machine. Various build related scripts are on https://github.com/cryoem/build-scripts.
-Line 58:
+Line 96:
-[[Windows]]
+[Windows]
-Line 61:
+Line 99:
-Docker images and helper scripts are at https://github.com/cryoem/docker-images.
+Docker images and helper scripts are at --(https://github.com/cryoem/docker-images )-- https://github.com/cryoem/build-scripts.
-Line 63:
+Line 101:
-Runs as root on Linux. `chown` doesn't work, the resulting installer has root ownership.
+:FIXME: Runs as root on Linux. `chown` doesn't work, the resulting installer has root ownership.
-Line 67:
+Line 105:
+''':TODO: Review and edit below'''
-Line 68:
+Line 109:
-The version of `OpenMPI` provided with EMAN2 installer may not work with user's batch queueing system, meaning they would not be able to run jobs on more than one node at a time. In that case, installer-provided `OpenMPI` needs to be removed, `pydusa` and `fftw-mpi` need to be installed against `OpenMPI` that is present in the system. Two scripts, one that uninstalls installer provided `OpenMPI`~~, `pydusa` and `fftw-mpi`,~~ and another that builds `fftw-mpi` and `pydusa` against the system `OpenMPI` are provided. The installer prints a detailed message regarding this issue.
-Line 70:
+Line 110:
-Only OpenMPI is force-removed. 1. So the system or newly installed version is used 1. Force-removing fftw-mpi and, specifically, pydusa leaves the system broken. 1. Even if not removed, later installations overwrite, so there isn't any need to uninstall anyways.
+The version of OpenMPI provided with EMAN2 installer may not work with user's batch queueing system, meaning they would not be able to run jobs on more than one node at a time. In that case, installer-provided OpenMPI needs to be removed, pydusa and fftw-mpi need to be installed against OpenMPI that is present in the system. Two scripts, one that uninstalls installer provided OpenMPI--(, pydusa and fftw-mpi,)-- and another that builds fftw-mpi and pydusa against the system OpenMPI are provided. Instructions [here].

Only OpenMPI is force-removed.
 1. So the system or newly installed version is used
 1. Force-removing fftw-mpi and, specifically, pydusa leaves the system broken.
 1. Even if not removed, later installations overwrite, so there isn't any need to uninstall anyways.