Contents
GitHub webhooks are setup to send notifications to blake. Blake forwards those to three build machines. Currently, webhooks are being listened to only on Linux. Linux runs the server that drives the Jenkins jobs.
Jenkins server runs on Linux, Jenkins agents on Linux, Mac and Windows. The server instance manages Jenkins, the agents run the jobs.
On Linux, to start Jenkins, run command:
docker stop jenkins-master docker rm jenkins-master docker run -d -u root --name jenkins-master -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home -e PLUGINS_FORCE_UPGRADE=true -e TRY_UPGRADE_IF_NO_MARKER=true --restart unless-stopped jenkins/jenkins:lts
Binary builds require conda-build and constructor. Packaging is done with constructor, a tool for making installers from conda packages.
Nodes: http://10.10.11.176:8080/computer/
http://10.10.11.176:8080/computer/linux/configure
http://10.10.11.176:8080/computer/mac/configure
http://10.10.11.176:8080/computer/win/configure
JavaPath: "D:\Downloads\openjdk-11+28_windows-x64_bin\jdk-11\bin\java.exe"
Prefix Start Agent Command: "D: &&"
http://10.10.11.176:8080/manage/configure
Jenkins URL: http://10.10.11.176:8080/
System Admin e-mail address: eman-bot (linux)<eman.github@gmail.com>
http://10.10.11.176:8080/manage/configureSecurity/
Credentials: http://10.10.11.176:8080/manage/credentials/
ID Name gh-eman-bot eman-bot (gh-eman- jenkins-key-eman eman (jenkins-key-eman) jenkins-key-eman2 eman2 (jenkins-key-eman2)
http://10.10.11.176:8080/job/cryoem-eman2-trigger/: Triggers job cryoem-eman2 on agents
http://10.10.11.176:8080/job/cryoem-eman2/: Test(?) and binary builds
http://10.10.11.176:8080/job/eman-feedstock-trigger-from-eman-master/
http://10.10.11.176:8080/job/eman-feedstock-building-eman-v2.99/
JenkinsCI: Jenkinsfile
Manually triggered by including "[ci build]" anywhere in the last commit message. Manually triggered builds on master branch are uploaded as continuous builds and builds triggered from any other branch are uploaded to testing area.
Dependencies not available on anaconda or conda-forge are available on cryoem. The binaries are built and uploaded using conda-forge's conda-smithy. conda-smithy takes care of generating feedstocks, registering them on GitHub and online CI services and building conda recipes.
conda is the package manager.
https://anaconda.org is the online repository of binaries.
conda-build is the tool to build from source.
constructor is the tool to package eman2 and dependency binaries into a single installer file.
EMAN2 is distributed as a single installer which includes all its dependencies.
Packages that are available on https://anaconda.org can be installed into any conda environment by issuing the command conda install <package>. Conda installs the package along with its dependencies. In order for packages to benefit from this automation, they need to be packaged in a specific way. That can be done with conda-build. conda-build builds packages according to instructions provided in a recipe. A recipe consists of a file with package metadata, meta.yaml, and any other necessary resources like build scripts, (build.sh, bld.bat), patches and so on.
Most of EMAN2 dependencies can be found on anaconda's channels, defaults and conda-forge. A few that do not exist or need to be customized have been built and uploaded to channel cryoem. The recipes are hosted in separate repositories on GitHub. Every recipe repository follows the feedstock approach of conda-forge. See here for a complete list.
Conda smithy uses tokens to authenticate with GitHub.
Conda-smithy commands:
1 conda create -n smithy conda-smithy
2 conda activate smithy
3 conda smithy init <recipe_directory>
4 conda smithy register-github <feedstock_directory> --organization cryoem
5 conda smithy register-ci --organization cryoem --without-azure --without-drone
6 conda smithy rerender --no-check-uptodate
libpython can be linked statically or dynamically when python is built. It is important for python extensions to be aware of the type of linking in order to avoid segfaults. This can be accomplished by querying Py_ENABLE_SHARED.
1 python -c "import sysconfig; print(sysconfig.get_config_var('Py_ENABLE_SHARED'))"
In EMAN, it is done in cmake/FindPython.cmake
OpenGL detection when Anaconda's compilers are used is done using a cmake toolchain file.
Mac: slave clock sync https://blog.shameerc.com/2017/03/quick-tip-fixing-time-drift-issue-on-docker-for-mac docker run --rm --privileged alpine hwclock -s
Windows: OPENGL: https://github.com/conda/conda-recipes/blob/master/qt5/notes.md
Not sure if this is true