Differences between revisions 79 and 80
⇤ ← Revision 79 as of 2022-11-28 03:13:21 →
Size: 11420
Editor: TunayDurmaz
Comment:
← Revision 80 as of 2022-11-28 03:17:41 → ⇥
Size: 11442
Editor: TunayDurmaz
Comment:
Deletions are marked like this. Additions are marked like this.
Line 63: Line 63:
=== Jenkins Setup on Linux === === Jenkins Server Setup (on Linux) ===
Line 90: Line 90:
PATH



== Jenkins Setup ==
 {{{#!wiki comment
Not sure if this is true
Line 97: Line 93:
}}}

Contents

  1. Binary Distribution
    1. Jenkins Setup
      1. Jenkins Server Setup (on Linux)
    2. Jenkins Setup
    3. Linux
    4. Mac
    5. Windows
  2. Anaconda
    1. Conda
    2. Recipes, Feedstocks and anaconda.org channel: cryoem
    3. Feedstocks
      1. General instructions
      2. Conda-smithy Workflow
  3. Build System Notes
    1. CMake
        1. Jenkins Docker
        2. Docker

Binary Distribution

  • GitHub webhooks are setup to send notifications to blake. Blake forwards those to three build machines. Currently, webhooks are being listened to only on Linux. Linux runs the server that drives the Jenkins jobs.

Jenkins Setup

Jenkins server runs on Linux, Jenkins agents on Linux, Mac and Windows. The server instance manages Jenkins, the agents run the jobs.

  1. On Linux, to start Jenkins, run command:

    docker stop jenkins-master

docker rm jenkins-master

docker run -d -u root --name jenkins-master -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home -e PLUGINS_FORCE_UPGRADE=true -e TRY_UPGRADE_IF_NO_MARKER=true --restart unless-stopped jenkins/jenkins:lts
  2. Install Java on agents.
  3. Setup passwordless ssh connections from Linux server to agents.

  4. Nodes: http://10.10.11.176:8080/computer/

    1. http://10.10.11.176:8080/computer/linux/configure

      1. name: linux
      2. remoteFS: /home/eman2/linux-1-agent/
      3. numExecutors: 100
      4. host: eman-centos7-01
      5. credentialsId: jenkins-key-eman2
      6. label: eman
      7. envVars
        1. DEPLOY_PATH: /opt/web2py_apps/web2py/applications/cryoem/static/software/
        2. HOME_DIR: /home/eman2
        3. PATH+EXTRA: /home/eman2/miniconda3/bin

    2. http://10.10.11.176:8080/computer/mac/configure

      1. name: mac
      2. remoteFS: /Users/eman/mac-1-agent
      3. numExecutors: 100
      4. host: eman-macmini
      5. credentialsId: jenkins-key-eman
      6. label: eman
      7. envVars
        1. DEPLOY_PATH: /opt/web2py_apps/web2py/applications/cryoem/static/software/
        2. HOME_DIR: /Users/eman/
        3. PATH+EXTRA: /Users/eman/miniconda3/bin

    3. http://10.10.11.176:8080/computer/win/configure

      1. name: win
      2. remoteFS: D:\workspace\win-1-agent
      3. numExecutors: 100
      4. host: BM-WIN-01
      5. credentialsId: jenkins-key-eman

      6. JavaPath: "D:\Downloads\openjdk-11+28_windows-x64_bin\jdk-11\bin\java.exe"

      7. Prefix Start Agent Command: D: &&

      8. label: eman
      9. envVars
        1. DEPLOY_PATH: /opt/web2py_apps/web2py/applications/cryoem/static/software/
        2. HOME_DIR: D:
        3. PATH+EXTRA: D:\Miniconda3;D:\Miniconda3\Scripts

Jenkins Server Setup (on Linux)

  1. http://10.10.11.176:8080/manage/configure

    1. Labels: main master

    2. Jenkins URL: http://10.10.11.176:8080/

    3. System Admin e-mail address: eman-bot (linux)<eman.github@gmail.com>

    4. SSH Servers
      1. Name: Installer-Server
      2. Hostname: cryoem.bcm.edu
      3. Username: eman-binary-uploader
      4. Remote Directory: /opt/web2py_apps/web2py/applications/cryoem/static/software/

  2. http://10.10.11.176:8080/manage/configureSecurity/

    1. Agents
      1. TCP port for inbound agents
        1. Fixed: 50000
    2. Git Host Key Verification Configuration
      1. Host Key Verification Strategy
        1. Known hosts file

  3. Credentials: http://10.10.11.176:8080/manage/credentials/ 

    • ID                  Name
gh-eman-bot         eman-bot (gh-eman-
jenkins-key-eman    eman (jenkins-key-eman)
jenkins-key-eman2   eman2 (jenkins-key-eman2)

Jenkins Setup

  1. Login info: 10.10.11.176:8080/ username: eman2
  2. Jobs
  3. Settings, Plugins ?
  4. Binary builds require conda-build, constructor

Packaging is done with constructor, a tool for making installers from conda packages.

  1. JenkinsCI: Jenkinsfile

    1. Secrets like ssh keys are stored locally in Jenkins
    2. Some env vars need to be set by agents:
      1. HOME_DIR, DEPLOY_PATH, PATH+EXTRA (to add miniconda to PATH).
      2. PATH+EXTRA is not set on win. (?)
        1. Now, it is set on win, too. 
    3.  Launch method: via SSH
   Advanced:
     Prefix Start Agent Command: "D: && "
    4. On windows for sh calls in jenkins to work "Git for Windows" might need to be installed.
  1. Binary builds on local build machines.

    1. Manually triggered by including "[ci build]" anywhere in the last commit message. Manually triggered builds on master branch are uploaded as continuous builds and builds triggered from any other branch are uploaded to testing area.

    2. Any branch in the form of "release-" triggers continuous builds without having to include "[ci build]" in the commit message. Once the release branch is ready, release binaries are manually copied from cont. builds folder into the release folder on the server.

Linux

docker run -d -u root --name jenkins-master -p 8080:8080 -p 50000:50000 --restart unless-stopped -v /home/eman2/jenkins_home:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock -e PLUGINS_FORCE_UPGRADE=true -e TRY_UPGRADE_IF_NO_MARKER=true --restart unless-stopped cryoem/jenkins:dev

Mac

slave clock sync https://blog.shameerc.com/2017/03/quick-tip-fixing-time-drift-issue-on-docker-for-mac docker run --rm --privileged alpine hwclock -s

Windows

OPENGL: https://github.com/conda/conda-recipes/blob/master/qt5/notes.md

Anaconda

Dependencies not available on anaconda or conda-forge are available cryoem. The binaries are built and uploaded using conda-forge's conda-smithy. conda-smithy takes care of generating feedstocks, registering them on GitHub and online CI services and building conda recipes.

EMAN2 is built with conda-build using binaries from https://anaconda.org, packaged into an installer with constructor as of v2.2.

  1. conda is the package manager.

  2. https://anaconda.org is the online repository of binaries.

  3. conda-build is the tool to build from source.

  4. constructor is the tool to package eman2 and dependency binaries into a single installer file.

EMAN2 is distributed as a single installer which includes all its dependencies.

Conda

Packages that are available on https://anaconda.org can be installed into any conda environment by issuing the command conda install <package>. Conda installs the package along with its dependencies. In order for packages to benefit from this automation, they need to be packaged in a specific way. That can be done with conda-build. conda-build builds packages according to instructions provided in a recipe. A recipe consists of a file with package metadata, meta.yaml, and any other necessary resources like build scripts, (build.sh, bld.bat), patches and so on.

Recipes, Feedstocks and anaconda.org channel: cryoem

Most of EMAN2 dependencies can be found on anaconda's channels, defaults and conda-forge. A few that do not exist or need to be customized have been built and uploaded to channel cryoem. The recipes are hosted in separate repositories on GitHub. Every recipe repository follows the feedstock approach of conda-forge. See here for a complete list.

Feedstocks

General instructions

  1. Existing feedstocks
  2. Files to edit: recipe/, conda-build.yaml, conda-forge.yaml
  3. conda create -n smithy conda-smithy -c conda-forge
  4. conda-smithy rerender
  5. More info in conda-smithy/README.md, conda smithy -h, conda-forge.org/docs
  6. New feedstocks
  7. conda-smithy/README.md, conda smithy -h

Conda-smithy Workflow

Conda smithy uses tokens to authenticate with GitHub. 

  •    1 # conda-forge.yml
   2 
   3 channels:
   4   sources: [cryoem, defaults, conda-forge]
   5   targets:
   6   - [cryoem, main]
   7 
   8 github:
   9   user_or_org: cryoem
  10   repo_name: <package>-feedstock
  11 
  12 provider:
  13   linux: circle
  14   osx: travis
  15   win: appveyor
  16 
  17 azure:
  18   build_id: blank

       1 # recipe/conda_build_config.yaml
   2 
   3 channel_sources:
   4 - cryoem, defaults,conda-forge
   5 channel_targets:
   6 - cryoem dev

Conda-smithy commands: 

  •    1 conda create -n smithy conda-smithy
   2 conda activate smithy
   3 conda smithy init <recipe_directory>
   4 conda smithy register-github <feedstock_directory> --organization cryoem
   5 conda smithy register-ci --organization cryoem --without-azure --without-drone
   6 conda smithy rerender --no-check-uptodate

Build System Notes

CMake

  1. libpython can be linked statically or dynamically when python is built. It is important for python extensions to be aware of the type of linking in order to avoid segfaults. This can be accomplished by querying Py_ENABLE_SHARED. 

       1 python -c "import sysconfig; print(sysconfig.get_config_var('Py_ENABLE_SHARED'))"

    In EMAN, it is done in cmake/FindPython.cmake

  2. OpenGL detection when Anaconda's compilers are used is done using a cmake toolchain file.

  3. glext.h file needed for OpenGL related module compilation is already present on Linux and Mac. On Windows, it is manually copied once into C:\Program Files\Microsoft SDKs\Windows\v6.0A\Include\gl. On Appveyor it is downloaded as part of env setup every time a test is run.

Jenkins Docker

  1. docker-compose.yml at home dir in build machines

  2. TZ: https://stackoverflow.com/a/46384925

  3. Agent nodes setup
  4. Server and agent per machine vs single server and os agents

Docker

Docker images and helper scripts are at https://github.com/cryoem/docker-images https://github.com/cryoem/build-scripts.

Command to run docker with GUI support, CentOS7: 

xhost + local:root

docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY cryoem/eman-nvidia-cuda8-centos7

# When done with eman
xhost - local:root

:FIXME: Runs as root on Linux. chown doesn't work, the resulting installer has root ownership.

EMAN2/BuildSystem (last edited 2022-12-12 03:42:44 by TunayDurmaz)