Under Construction

Contents

  1. Under Construction
  2. CMake
  3. Anaconda
  4. Continuous Integration
  5. Jenkins Setup
    1. Linux
    2. Mac
    3. Windows
  6. Distribution
    1. Binaries on cryoem.bcm.edu
    2. EMAN2 on anaconda.org
    3. EMAN2 Docker images

CMake

  1. Dependencies are from Anaconda. CMake runs conda to detect environment location and uses it to find packages.

  1. Make targets can be listed with make help. Some convenience targets: 

    $ make help
The following are some of the valid targets for this Makefile:
... .....
... .....
... PythonFiles
... test-rt
... test-py-compile
... test-verbose-broken
... test-progs
... test-verbose
... .....
... .....

  2. libpython can be linked statically or dynamically when python is built. It is important for python extensions to be aware of the type of linking in order to avoid segfaults. (Need a little more details, refs?)

  3. opengl ???

  4. cmake <source-dir> -DENABLE_WARNINGS=OFF

Anaconda

Feedstocks, conda-smithy, https://anaconda.org/cryoem/

  1. ftgl
  2. fftw-mpi
  3. openmpi
  4. pydusa and pydusa-feedstock
  5. constructor and constructor-feedstock
  6. eman-deps-feedstock, eman-deps-cli
  7. eman-packaging-feedstock

File structure, dependencies ? ABI compat

  1. conda
  2. conda-build
  3. constructor

Continuous Integration

  1. GitHub webhooks

  2. Test source build and conda recipe build (conda-build).
  3. Binary builds on local build machines.
    1. Manually triggered build on master and non-master.
    2. Cron builds.
    3. Release binaries manually copied from cont. builds.

  4. CircleCI: .circleci/config.yml

  5. TravisCI: .travis.yml

  6. Appveyor: appveyor.yml

  7. JenkinsCI: Jenkinsfile, binary builds

Jenkins Setup

  1. Triggers

    1. GitHub webhooks

    2. Cron
    3. Binary build trigger
  2. Jenkins master needs PATH prepended with $CONDA_PREFIX/bin
  3. Jenkins Docker image, docker-coompose or docker stack deploy
  4. docker-compose.yml at home dir in build machines

  5. TZ: https://stackoverflow.com/a/46384925

  6. plugins
  7. config, jcasc, config.xml, users.xml, jobs/*.xml?, gpg encrypt
  8. Agent nodes setup, agent nodes auto-start
  9. Server and agent per machine vs single server and os agents
    1. Master only
    2. Master and agent per machine
    3. Single master and OS agents

Linux

  1. systemctl

docker run -d -u root -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock jenkins/jenkins:lts &

docker run -d -u root --name jenkins-master -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock -e PLUGINS_FORCE_UPGRADE=true -e TRY_UPGRADE_IF_NO_MARKER=true --restart unless-stopped cryoem/jenkins:dev

cron:

0 0 * * * bash /home/eman2/workspace/cronjobs/cleanup_harddisk.sh

$ cat Desktop/docker.txt docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts docker run -p 8080:8080 -p 50000:50000 --restart unless-stopped jenkins/jenkins:lts docker run -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home jenkins/jenkins:lts docker run -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /var/jenkins_home:/home/eman2/jenkins_home jenkins/jenkins:lts

# Working docker run -u root -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home jenkins/jenkins:lts docker run -u root -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home jenkins

docker run -d -u root -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock jenkins/jenkins:lts

sudo docker run -it -v /var/jenkins_home:/home/eman2/jenkins_home jenkins

startup: right-click ???

Mac

  1. plist

docker run -d --name jenkins-master -p 8080:8080 -p 50000:50000 -v /Users/eman/workspace/jenkins_home:/var/jenkins_home --restart unless-stopped jenkins/jenkins:lts

Auto startup: plist https://imega.club/2015/06/01/autostart-slave-jenkins-mac/ /Users/eman/Library/LaunchAgents

slave clock sync https://blog.shameerc.com/2017/03/quick-tip-fixing-time-drift-issue-on-docker-for-mac docker run --rm --privileged alpine hwclock -s

client 0 free swap space

$ cat Desktop/docker.txt docker run -p 8080:8080 -v /Users/eman/workspace/jenkins_home:/var/jenkins_home jenkins docker run -it -p 8080:8080 -v /Users/eman/workspace/jenkins_home:/var/jenkins_home --restart unless-stopped jenkins

# Working docker run -it -p 8080:8080 -v /Users/eman/workspace/jenkins_home:/var/jenkins_home --restart unless-stopped jenkins/jenkins:lts

# Blue Ocean docker run \

# Latest docker run -d --name jenkins-master -p 8080:8080 -p 50000:50000 -v /Users/eman/workspace/jenkins_home:/var/jenkins_home --restart unless-stopped jenkins/jenkins:lts

FROM jenkins/jenkins:lts COPY plugins.txt /usr/share/jenkins/ref/plugins.txt RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

plugins.txt: ace-editor:latest bouncycastle-api:latest branch-api:latest chef-identity:latest

Settings: tokens slaves

Windows

Move jenkins_home http://tech.nitoyon.com/en/blog/2014/02/25/jenkins-home-win/

Run as service: Open Task Manager(Ctrl+Shift+Esc), New task, Browse to agent.jnlp and run as admin does this work? This is when starting via Web Launcher doesn't work.

currently, task scheduler works need to have miniconda pn path, set it during miniconda installation, but do not(?) register python.

While installing miniconda register python and add to PATH. Then, conda init in cmd (git init cmd.exe) and git windows (git init bash). And, maybe restart???

BUG: miniconda3 conda-build=3.17.8 adds vc14 even if python2 is requested in build reqs

OPENGL: https://github.com/conda/conda-recipes/blob/master/qt5/notes.md

Distribution

Binaries on cryoem.bcm.edu

EMAN2 on anaconda.org

EMAN2 Docker images