Differences between revisions 6 and 7
Revision 6 as of 2010-09-22 07:08:28
Size: 5364
Editor: root
Comment:
Revision 7 as of 2010-09-22 07:15:32
Size: 5504
Editor: root
Comment:
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:
For ease of editing, EMEN2 uses YAML as the configuration file format. YAML is a markup language similar to XML or JSON for structured documents. The default configuration file (emen2/config/config.sample.yml) is presented below, with comments detailing each setting. EMEN2 uses JSON for the config file format, although YAML is also supported if a suitable parser is found. The default configuration file (emen2/db/config.base.json) is presented below, with comments detailing each setting.
Line 6: Line 6:
For compatibility with the low level Berkeley DB tools, you will generally use the environment variable $DB_HOME to point to the path of the actual EMEN2 database environment. Alternatively, you could specify it in the configuration file with the EMEN2DBPATH setting. Any config.json files found in the database directory, $EMEN2DBHOME, will be loaded on top of the base config, overriding any values. I present a typical installation's config.json file later on in this page.
Line 8: Line 8:
The most important settings are the paths specified for various EMEN2 subsystems. By default, the &root element (first item below) is used as a "prefix" for the various subsystem directories (applog, archive, emen2files, db_hotbackup), so &root as /home/emen2 would result in /home/emen2/applog, /home/emen2/archive, etc. Only the "paths" section is treated specially -- items without leading slashes are interpreted relative to $EMEN2DBHOME, items with leading slashes are left as absolute path references.
Line 10: Line 10:
If you wish to specify an absolute path to an element, without the &root prefix, remove the "prefix: -*root" item from that block. = Base Configuration =
Line 13: Line 13:
---
# run cmdlineutils/expandyaml.py to see contents of this file after processing
Line 16: Line 14:
root: {
 // These paths either absolute, or relative to EMEN2DBHOME, depending on the presence of leading slash
 "paths":{
Line 18: Line 18:
  # Default directory for emen2 items: db, applog, archive, db_hotbackup, emen2files, emen2tiles
  # This should be an absolute path
  root: &root
    /home/emen2/
  // Log directory: access.log, error.log, debug.log (see logging section)
  "LOGPATH": "applog",
  "EMEN2JOBFILE": "applog/emen2job",
  "EMEN2ERRLOG":"applog/emen2errlog",
  "DEBUGLOG":"applog/debug.log",
Line 23: Line 24:
  # path to Berkeley DB installation
  bdbprefix: &bdbprefix
    /usr/local/BerkeleyDB.4.8/
  // Berkeley DB Hot Backup directory, for incremental backups and failover. Backup offsite.
  "HOTBACKUP": "db_hotbackup",
Line 27: Line 27:
  # path to configuration files -- loaded after this one and overrides it
  configfiles:
    - /etc/emen2config.yml
  // Berkeley DB Log Archive directory. You will want to carefully back these up.
  "LOG_ARCHIVE": "log_archive",
Line 31: Line 30:
  // Thumbnails and other files derived from primary data
  "TILEPATH": "tiles",
  
  // Temporary directory to use for graphs and other temporary files
  "TMPPATH": "tmp",
  
  /* Raw data storage area. You may want to place this on a large raid array.
  You can also set this up to split file storage by date, e.g.:
   '0': '/raid1/',
   '20080501': '/raid2/',
   '20100101': '/raid3/'
  Files from beginning -> May 2008 will be on /raid1, May 2008 to Jan 2010 on /raid2, and files after Jan 2010 on /raid3 */
  "BINARYPATH":{
   "0": "emen2data"
  },
Line 32: Line 46:
base:
  prefix:
      - *root
  // Path to SSL certificates
  "SSLPATH": "ssl",
Line 36: Line 49:
  # EMEN2 Database Environment
  # I recommend using the $DB_HOME environment variable for this setting, as $DB_HOME/config.yml will be loaded as the default environment
  # as well as convenience with the Berkeley DB command line utils, without specifying -h
  # EMEN2DBPATH: /home/emen2/db
  
  // Alternate views and templates to be loaded
  "TEMPLATEDIRS": ["templates"],
  "VIEWPATHS": ["views"],
Line 42: Line 53:
  # Log directory: access.log, error.log, debug.log (see logging section)
  LOGPATH: &logpath
      applog
      
  # Berkeley DB Log Archive directory. Backup offsite.
  ARCHIVEPATH:
      log_archive
      
  # Berkeley DB Hot Backup directory, for incremental backups and failover. Backup offsite.
  BACKUPPATH:
      db_backup
  // Load additional files. Typically this will only be used by the system installed base config file.
  "CONFIGFILES" :[
   "/etc/emen2config.json",
   "config.json"
  ]
 },
Line 54: Line 60:
  # Cache for "map" interface for large image files
  TILEPATH:
      emen2tiles
 // Mail settings. If both of these are set, EMEN2 will send emails during certain events.
 "mailsettings":{
  "MAILADMIN": false,
  "MAILHOST": false
 },
Line 58: Line 66:
 // Web-server related settings
 "network":{
  
  // Number of web server threads to run
  "EMEN2EXTURI": "",
Line 59: Line 72:
# Raw data storage area. You may want to place this on a large raid array.
# You can also set this up to split file storage by date, e.g.:
    # 0: /raid1/
    # 20080501: /raid2/
    # 20100101: /raid3/
# Files from beginning -> May 2008 will be on /raid1, May 2008 to Jan 2010 on /raid2, and files after Jan 2010 on /raid3
# You will need to use an absolute path here
binaries:
   BINARYPATH:
      0: /home/emen2/emen2data
    
  "NUMTHREADS": 5,
  "EMEN2PORT": 8080,
Line 71: Line 75:
# Command to invoke to run EMAN2 tile generator for micrographs. These will be cached in the TILEPATH directory.
EMAN2:
   E2TILEFILE: emen2/clients/e2tilefile.py --buildspec
  // HTTPS port. You will need to place the SSL certificate and key files in the SSL directory
  "EMEN2HTTPS": 0,
Line 75: Line 78:
  // The EMEN2 URI root -- e.g., "/testserver" or "http://ncmidb.bcm.edu/challenge". Useful for running behind a reverse proxy.
  "EMEN2WEBROOT": ""
 },
Line 76: Line 82:
params:
  # Database version. This may be moved out of the config file at some point.
  VERSION: 20100325
 // You probably don't want to change any of these.
 "params":{
  "BLOCKLENGTH": 100000,
  "VERSION": 20100911,
  "TIMESTR": "%Y/%m/%d %H:%M:%S",
  "MAXRECURSE": 50,
  "MAXIDLE": 604800
 },
Line 80: Line 91:
  # Number of web server threads to run. Note: these are python threads, so limited to single CPU
  NUMTHREADS: 1
  
  # URI and port configuration.
  EMEN2EXTURI: http://localhost:8080
  EMEN2HOST: localhost
  EMEN2PORT: 8080
 // Some basic settings and interface strings
 "customization":{
Line 88: Line 94:
  # Web root is useful is you are proxying behind apache: e.g. /proxy/virtualroot/db -> /db... and external proxy port
  EMEN2WEBROOT: ""
  EMEN2EXTPORT: 8080
  // Short string for database identification. This shows up several places in the UI
  "EMEN2DBNAME": "EMEN2",
Line 92: Line 97:
  # SSL
  EMEN2HTTPS: False
  EMEN2PORT_HTTPS: 65532
  
  # You will want to leave these at their default settings
  TIMESTR: '%Y/%m/%d %H:%M:%S'
  MAXRECURSE: 50
  BLOCKLENGTH: 100000
  MAXIDLE: 604800
  USETXN: True
  RECOVER: True
  // New users will be added to these groups
  "GROUP_DEFAULTS": ["create"],
  
  // Logo in emen2/web/static/images to use
  "EMEN2LOGO": "emen2logo-small.png",
  
  // These words will not be indexed
  "UNINDEXED_WORDS": ["in", "of", "for", "this", "the", "at", "to", "from", "at", "for", "and", "it", "or"]
 },
Line 104: Line 107:
 // Bookmarks to some key records.
 "bookmarks":{
  "BOOKMARKS":{
   "EQUIPMENT":0,
   "GROUPS":0,
   "PROJECTS":0
  }
 },
Line 105: Line 116:
customization:
  # Short string for database identification. This shows up several places in the UI
  EMEN2DBNAME: "EMEN2"
  
  # Logo in emen2/tweb/images to use
  EMEN2LOGO: "emen2logo-small.png"
  
  # Default text to show on login screen
  GREETING:
      <p>Welcome!</p>
 // Load the default templates.
 "viewsettings":{
  "TEMPLATEDIRS_DEFAULT":true
 },
Line 116: Line 121:
  # Template to use for anonymous uses (e.g. before login)
  HOME_NOAUTH: '/pages/home.noauth'
  
  # Words to ignore in full-text indexing.
  UNINDEXED_WORDS: ["in", "of", "for", "this", "the", "at", "to", "from", "at", "for", "and", "it", "or"]
 // Verbose logging
 "logging":{
  "DEBUG":false,
  "LOG_LEVEL":"LOG_INFO"
 },
Line 122: Line 127:
  # Default groups for new users
  GROUP_DEFAULTS: ["create"]
  
  # Unused at the moment
  SPARSETREE: 1
 // Path to EMAN2 python; if you are using an EMAN2 binary you will want to set this.
 "EMAN2":{
  "EMAN2PYTHON":"python"
 },
Line 128: Line 132:
 // Path to ImageMagick programs. Used for generating thumbnails.
 "IMAGEMAGICK":{
  "CONVERTPATH":"convert"
 }
}
}}}
Line 129: Line 139:
# Record IDs to use in the top toolbar in the web UI
bookmarks:
  BOOKMARKS:
    GROUPS: 1
    PROJECTS: 1
    EQUIPMENT: 1
= Example Custom Configuration =
Line 136: Line 141:
This is a $EMEN2DBHOME/config.json file based on our EMEN2 installation. Again, note that the "paths" section is treated specially -- paths without a leading slash are interpreted relative to $EMEN2DBHOME, paths with a leading slash are left as absolute.
Line 137: Line 143:
# Set mailhost to 0 to leave mail subsystem off, otherwise set to outgoing SMTP server
mailsettings:
  MAILADMIN: root@localhost
  MAILHOST: 0
{{{
Line 142: Line 145:
{
    "EMAN2": {
        "EMAN2PYTHON": "/usr/local/bin/python"
    },
    "bookmarks": {
        "BOOKMARKS": {
            "BANNER": 448491,
            "BANNER_NOAUTH": 449989,
            "EQUIPMENT": 270940,
            "GROUPS": 0,
            "PROJECTS": 136
        }
    },
    "mailsettings": {
        "MAILADMIN": "webadmin@blake.grid.bcm.edu",
        "MAILHOST": "blake.grid.bcm.edu"
    },
    "network": {
        "EMEN2DBNAME": "NCMI Database",
        "EMEN2EXTURI": "http://ncmidb.bcm.edu",
        "EMEN2LOGO": "ncmi2006-logo2-white.png",
        "EMEN2PORT": 80,
        "EMEN2WEBROOT": "",
        "NUMTHREADS": 5
    },
    "paths": {
        "BINARYPATH": {
            "0": "/raid1/emen2",
            "20050101": "/raid2/emen2",
            "20070321": "/raid3/emen2"
        },
        "LOG_ARCHIVE": "/home/emen2/log_archive",
        "TEMPLATEDIRS": [
            "/home/emen2/db/overlay/templates"
        ],
        "VIEWPATHS": [
            "/home/emen2/db/overlay/views"
        ]
    }
}
Line 143: Line 186:


# Probably don't want to touch these unless you have an unusual Berkeley DB setup, e.g. db48_recover instead of db_recover
dbutils:
    prefix:
        - *bdbprefix
    db_recover: bin/db_recover
    db_hotbackup: bin/db_hotbackup
    db_archive: bin/db_archive
    db_checkpoint: bin/db_checkpoint
 
 
# Additional template and view directories to add to search paths
viewsettings:
  TEMPLATEDIRS_DEFAULT: True
  TEMPLATEDIRS:
    - 'templates'
  VIEWPATHS:
    - 'views'


# Logging files. See &logpath for base directory.
logging:
  prefix:
    - *logpath
  EMEN2ERRLOG: emen2errlog
  EMEN2JOBFILE: emen2job
  DEBUGLOG: debug.log
  DEBUG: True
  
  

EMEN2 Configuration

EMEN2 uses JSON for the config file format, although YAML is also supported if a suitable parser is found. The default configuration file (emen2/db/config.base.json) is presented below, with comments detailing each setting.

Any config.json files found in the database directory, $EMEN2DBHOME, will be loaded on top of the base config, overriding any values. I present a typical installation's config.json file later on in this page.

Only the "paths" section is treated specially -- items without leading slashes are interpreted relative to $EMEN2DBHOME, items with leading slashes are left as absolute path references.

Base Configuration

{
        // These paths either absolute, or relative to EMEN2DBHOME, depending on the presence of leading slash
        "paths":{

                // Log directory: access.log, error.log, debug.log (see logging section) 
                "LOGPATH": "applog",
                "EMEN2JOBFILE": "applog/emen2job",
                "EMEN2ERRLOG":"applog/emen2errlog",
                "DEBUGLOG":"applog/debug.log",

                // Berkeley DB Hot Backup directory, for incremental backups and failover. Backup offsite.
                "HOTBACKUP": "db_hotbackup",

                // Berkeley DB Log Archive directory. You will want to carefully back these up.
                "LOG_ARCHIVE": "log_archive",

                // Thumbnails and other files derived from primary data
                "TILEPATH": "tiles",
                
                // Temporary directory to use for graphs and other temporary files
                "TMPPATH": "tmp",
                
                /* Raw data storage area. You may want to place this on a large raid array.
                You can also set this up to split file storage by date, e.g.:
                        '0': '/raid1/', 
                        '20080501': '/raid2/',
                        '20100101': '/raid3/'
                Files from beginning -> May 2008 will be on /raid1, May 2008 to Jan 2010 on /raid2, and files after Jan 2010 on /raid3 */               
                "BINARYPATH":{
                        "0": "emen2data"
                },

                // Path to SSL certificates
                "SSLPATH": "ssl",

                // Alternate views and templates to be loaded 
                "TEMPLATEDIRS": ["templates"],
                "VIEWPATHS": ["views"],

                // Load additional files. Typically this will only be used by the system installed base config file.
                "CONFIGFILES" :[
                        "/etc/emen2config.json",
                        "config.json"
                ]
        },

        // Mail settings. If both of these are set, EMEN2 will send emails during certain events.
        "mailsettings":{
                "MAILADMIN": false,
                "MAILHOST": false
        },

        // Web-server related settings
        "network":{
                
                // Number of web server threads to run
                "EMEN2EXTURI": "",

                "NUMTHREADS": 5,
                "EMEN2PORT": 8080,

                // HTTPS port. You will need to place the SSL certificate and key files in the SSL directory
                "EMEN2HTTPS": 0,

                // The EMEN2 URI root -- e.g., "/testserver" or "http://ncmidb.bcm.edu/challenge". Useful for running behind a reverse proxy.
                "EMEN2WEBROOT": ""
        },

        // You probably don't want to change any of these.
        "params":{
                "BLOCKLENGTH": 100000,
                "VERSION": 20100911,
                "TIMESTR": "%Y/%m/%d %H:%M:%S",
                "MAXRECURSE": 50,
                "MAXIDLE": 604800
        },

        // Some basic settings and interface strings
        "customization":{

                // Short string for database identification. This shows up several places in the UI
                "EMEN2DBNAME": "EMEN2",

                // New users will be added to these groups
                "GROUP_DEFAULTS": ["create"],
                
                // Logo in emen2/web/static/images to use
                "EMEN2LOGO": "emen2logo-small.png",
                
                // These words will not be indexed
                "UNINDEXED_WORDS": ["in", "of", "for", "this", "the", "at", "to", "from", "at", "for", "and", "it", "or"]
        },

        // Bookmarks to some key records.
        "bookmarks":{
                "BOOKMARKS":{
                        "EQUIPMENT":0,
                        "GROUPS":0,
                        "PROJECTS":0
                }
        },

        // Load the default templates.
        "viewsettings":{
                "TEMPLATEDIRS_DEFAULT":true
        },

        // Verbose logging
        "logging":{
                "DEBUG":false,
                "LOG_LEVEL":"LOG_INFO"
        },

        // Path to EMAN2 python; if you are using an EMAN2 binary you will want to set this.
        "EMAN2":{
                "EMAN2PYTHON":"python"
        },

        // Path to ImageMagick programs. Used for generating thumbnails.
        "IMAGEMAGICK":{
                "CONVERTPATH":"convert"
        }
}

Example Custom Configuration

This is a $EMEN2DBHOME/config.json file based on our EMEN2 installation. Again, note that the "paths" section is treated specially -- paths without a leading slash are interpreted relative to $EMEN2DBHOME, paths with a leading slash are left as absolute.

{
    "EMAN2": {
        "EMAN2PYTHON": "/usr/local/bin/python"
    }, 
    "bookmarks": {
        "BOOKMARKS": {
            "BANNER": 448491, 
            "BANNER_NOAUTH": 449989, 
            "EQUIPMENT": 270940, 
            "GROUPS": 0, 
            "PROJECTS": 136
        }
    }, 
    "mailsettings": {
        "MAILADMIN": "webadmin@blake.grid.bcm.edu", 
        "MAILHOST": "blake.grid.bcm.edu"
    }, 
    "network": {
        "EMEN2DBNAME": "NCMI Database", 
        "EMEN2EXTURI": "http://ncmidb.bcm.edu", 
        "EMEN2LOGO": "ncmi2006-logo2-white.png", 
        "EMEN2PORT": 80, 
        "EMEN2WEBROOT": "", 
        "NUMTHREADS": 5
    }, 
    "paths": {
        "BINARYPATH": {
            "0": "/raid1/emen2", 
            "20050101": "/raid2/emen2", 
            "20070321": "/raid3/emen2"
        }, 
        "LOG_ARCHIVE": "/home/emen2/log_archive", 
        "TEMPLATEDIRS": [
            "/home/emen2/db/overlay/templates"
        ], 
        "VIEWPATHS": [
            "/home/emen2/db/overlay/views"
        ]
    }
}

EMEN2/config (last edited 2010-09-22 07:15:32 by root)