Differences between revisions 2 and 7 (spanning 5 versions)
Revision 2 as of 2010-03-29 08:51:30
Size: 4981
Editor: root
Comment:
Revision 7 as of 2010-09-22 07:15:32
Size: 5504
Editor: root
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
## page was renamed from EMEN2/config.yml
Line 3: Line 4:
For ease of editing, EMEN2 uses YAML as the configuration file format. YAML is a markup language similar to XML or JSON for structured documents. The default configuration file (emen2/config/config.sample.yml) is presented below, with comments detailing each setting. EMEN2 uses JSON for the config file format, although YAML is also supported if a suitable parser is found. The default configuration file (emen2/db/config.base.json) is presented below, with comments detailing each setting.
Line 5: Line 6:
For compatibility with the low level Berkeley DB tools, you will generally use the environment variable $DB_HOME to point to the path of the actual Database Environment. Alternatively, you could specify it in the configuration file with the EMEN2DBPATH setting. Any config.json files found in the database directory, $EMEN2DBHOME, will be loaded on top of the base config, overriding any values. I present a typical installation's config.json file later on in this page.
Line 7: Line 8:
The most important settings are the paths specified for various EMEN2 subsystems. By default, the &root element (first item below) is used as a "prefix" for the various subsystem directories (applog, archive, emen2files, db_hotbackup), so &root as /home/emen2 would result in /home/emen2/applog, /home/emen2/archive, etc. Only the "paths" section is treated specially -- items without leading slashes are interpreted relative to $EMEN2DBHOME, items with leading slashes are left as absolute path references.
Line 9: Line 10:
If you wish to specify an absolute path to an element, without the &root prefix, remove the "prefix: -*root" item from that block. = Base Configuration =
Line 12: Line 13:
---
# run cmdlineutils/expandyaml.py to see contents of this file after processing
Line 15: Line 14:
root: {
 // These paths either absolute, or relative to EMEN2DBHOME, depending on the presence of leading slash
 "paths":{
Line 17: Line 18:
  # Default directory for emen2 items: db, applog, archive, db_hotbackup, emen2files, emen2tiles
  # This should be an absolute path
  root: &root
    /home/emen2/
  // Log directory: access.log, error.log, debug.log (see logging section)
  "LOGPATH": "applog",
  "EMEN2JOBFILE": "applog/emen2job",
  "EMEN2ERRLOG":"applog/emen2errlog",
  "DEBUGLOG":"applog/debug.log",
Line 22: Line 24:
  # path to Berkeley DB installation
  bdbprefix: &bdbprefix
    /usr/local/BerkeleyDB.4.8/
  // Berkeley DB Hot Backup directory, for incremental backups and failover. Backup offsite.
  "HOTBACKUP": "db_hotbackup",
Line 26: Line 27:
  # path to configuration files -- loaded after this one and overrides it
  configfiles:
    - /etc/emen2config.yml
  // Berkeley DB Log Archive directory. You will want to carefully back these up.
  "LOG_ARCHIVE": "log_archive",
Line 30: Line 30:
  // Thumbnails and other files derived from primary data
  "TILEPATH": "tiles",
  
  // Temporary directory to use for graphs and other temporary files
  "TMPPATH": "tmp",
  
  /* Raw data storage area. You may want to place this on a large raid array.
  You can also set this up to split file storage by date, e.g.:
   '0': '/raid1/',
   '20080501': '/raid2/',
   '20100101': '/raid3/'
  Files from beginning -> May 2008 will be on /raid1, May 2008 to Jan 2010 on /raid2, and files after Jan 2010 on /raid3 */
  "BINARYPATH":{
   "0": "emen2data"
  },
Line 31: Line 46:
base:
  prefix:
      - *root
  // Path to SSL certificates
  "SSLPATH": "ssl",
Line 35: Line 49:
  # Log directory: access.log, error.log, debug.log (see logging section)
  LOGROOT: &logroot
      applog
      
  # Berkeley DB Log Archive directory. Backup offsite.
  ARCHIVEPATH:
      db_archive
      
  # Berkeley DB Hot Backup directory, for incremental backups and failover. Backup offsite.
  HOTBACKUP:
      db_hotbackup
  // Alternate views and templates to be loaded
  "TEMPLATEDIRS": ["templates"],
  "VIEWPATHS": ["views"],
Line 47: Line 53:
  # Cache for "map" interface for large image files
  TILEPATH:
      emen2tiles
  // Load additional files. Typically this will only be used by the system installed base config file.
  "CONFIGFILES" :[
   "/etc/emen2config.json",
   "config.json"
  ]
 },
Line 51: Line 60:
 // Mail settings. If both of these are set, EMEN2 will send emails during certain events.
 "mailsettings":{
  "MAILADMIN": false,
  "MAILHOST": false
 },
Line 52: Line 66:
# Raw data storage area. You may want to place this on a large raid array.
# You can also set this up to split file storage by date, e.g.:
    # 0: /raid1/
    # 20080501: /raid2/
    # 20100101: /raid3/
# Files from beginning -> May 2008 will be on /raid1, May 2008 to Jan 2010 on /raid2, and files after Jan 2010 on /raid3
# You will need to use an absolute path here
binaries:
   BINARYPATH:
      0: /home/emen2/emen2data
    
 // Web-server related settings
 "network":{
  
  // Number of web server threads to run
  "EMEN2EXTURI": "",
Line 64: Line 72:
# Command to invoke to run EMAN2 tile generator for micrographs. These will be cached in the TILEPATH directory.
EMAN2:
   E2TILEFILE: emen2/clients/e2tilefile.py --buildspec
  "NUMTHREADS": 5,
  "EMEN2PORT": 8080,
Line 68: Line 75:
  // HTTPS port. You will need to place the SSL certificate and key files in the SSL directory
  "EMEN2HTTPS": 0,
Line 69: Line 78:
params:
  # Database version. This may be moved out of the config file at some point.
  VERSION: 20100325
  // The EMEN2 URI root -- e.g., "/testserver" or "http://ncmidb.bcm.edu/challenge". Useful for running behind a reverse proxy.
  "EMEN2WEBROOT": ""
 },
Line 73: Line 82:
  # Number of web server threads to run. Note: these are python threads, so limited to single CPU
  NUMTHREADS: 1
  
  # URI and port configuration.
  EMEN2EXTURI: http://localhost:8080
  EMEN2HOST: localhost
  EMEN2PORT: 8080
 // You probably don't want to change any of these.
 "params":{
  "BLOCKLENGTH": 100000,
  "VERSION": 20100911,
  "TIMESTR": "%Y/%m/%d %H:%M:%S",
  "MAXRECURSE": 50,
  "MAXIDLE": 604800
 },
Line 81: Line 91:
  # Web root is useful is you are proxying behind apache: e.g. /proxy/virtualroot/db -> /db... and external proxy port
  EMEN2WEBROOT: ""
  EMEN2EXTPORT: 8080
 // Some basic settings and interface strings
 "customization":{
Line 85: Line 94:
  # SSL
  EMEN2HTTPS: False
  EMEN2PORT_HTTPS: 65532
  
  # You will want to leave these at their default settings
  TIMESTR: '%Y/%m/%d %H:%M:%S'
  MAXRECURSE: 50
  BLOCKLENGTH: 100000
  MAXIDLE: 604800
  USETXN: True
  RECOVER: True
  // Short string for database identification. This shows up several places in the UI
  "EMEN2DBNAME": "EMEN2",
Line 97: Line 97:
  // New users will be added to these groups
  "GROUP_DEFAULTS": ["create"],
  
  // Logo in emen2/web/static/images to use
  "EMEN2LOGO": "emen2logo-small.png",
  
  // These words will not be indexed
  "UNINDEXED_WORDS": ["in", "of", "for", "this", "the", "at", "to", "from", "at", "for", "and", "it", "or"]
 },
Line 98: Line 107:
customization:
  # Short string for database identification. This shows up several places in the UI
  EMEN2DBNAME: "EMEN2"
  
  # Logo in emen2/tweb/images to use
  EMEN2LOGO: "emen2logo-small.png"
  
  # Default text to show on login screen
  GREETING:
      <p>Welcome!</p>
 // Bookmarks to some key records.
 "bookmarks":{
  "BOOKMARKS":{
   "EQUIPMENT":0,
   "GROUPS":0,
   "PROJECTS":0
  }
 },
Line 109: Line 116:
  # Template to use for anonymous uses (e.g. before login)
  HOME_NOAUTH: '/pages/home.noauth'
  
  # Words to ignore in full-text indexing.
  UNINDEXED_WORDS: ["in", "of", "for", "this", "the", "at", "to", "from", "at", "for", "and", "it", "or"]
 // Load the default templates.
 "viewsettings":{
  "TEMPLATEDIRS_DEFAULT":true
 },
Line 115: Line 121:
  # Default groups for new users
  GROUP_DEFAULTS: ["create"]
  
  # Unused at the moment
  SPARSETREE: 1
 // Verbose logging
 "logging":{
  "DEBUG":false,
  "LOG_LEVEL":"LOG_INFO"
 },
Line 121: Line 127:
 // Path to EMAN2 python; if you are using an EMAN2 binary you will want to set this.
 "EMAN2":{
  "EMAN2PYTHON":"python"
 },
Line 122: Line 132:
# Record IDs to use in the top toolbar in the web UI
bookmarks:
  BOOKMARKS:
    GROUPS: 1
    PROJECTS: 1
    EQUIPMENT: 1
 // Path to ImageMagick programs. Used for generating thumbnails.
 "IMAGEMAGICK":{
  "CONVERTPATH":"convert"
 }
}
}}}
Line 129: Line 139:
= Example Custom Configuration =
Line 130: Line 141:
# Set mailhost to 0 to leave mail subsystem off, otherwise set to outgoing SMTP server
mailsettings:
  MAILADMIN: root@localhost
  MAILHOST: 0
This is a $EMEN2DBHOME/config.json file based on our EMEN2 installation. Again, note that the "paths" section is treated specially -- paths without a leading slash are interpreted relative to $EMEN2DBHOME, paths with a leading slash are left as absolute.
Line 135: Line 143:
{{{
Line 136: Line 145:
{
    "EMAN2": {
        "EMAN2PYTHON": "/usr/local/bin/python"
    },
    "bookmarks": {
        "BOOKMARKS": {
            "BANNER": 448491,
            "BANNER_NOAUTH": 449989,
            "EQUIPMENT": 270940,
            "GROUPS": 0,
            "PROJECTS": 136
        }
    },
    "mailsettings": {
        "MAILADMIN": "webadmin@blake.grid.bcm.edu",
        "MAILHOST": "blake.grid.bcm.edu"
    },
    "network": {
        "EMEN2DBNAME": "NCMI Database",
        "EMEN2EXTURI": "http://ncmidb.bcm.edu",
        "EMEN2LOGO": "ncmi2006-logo2-white.png",
        "EMEN2PORT": 80,
        "EMEN2WEBROOT": "",
        "NUMTHREADS": 5
    },
    "paths": {
        "BINARYPATH": {
            "0": "/raid1/emen2",
            "20050101": "/raid2/emen2",
            "20070321": "/raid3/emen2"
        },
        "LOG_ARCHIVE": "/home/emen2/log_archive",
        "TEMPLATEDIRS": [
            "/home/emen2/db/overlay/templates"
        ],
        "VIEWPATHS": [
            "/home/emen2/db/overlay/views"
        ]
    }
}
Line 137: Line 186:

# Probably don't want to touch these unless you have an unusual Berkeley DB setup, e.g. db48_recover instead of db_recover
dbutils:
    prefix:
        - *bdbprefix
    db_recover: bin/db_recover
    db_hotbackup: bin/db_hotbackup
    db_archive: bin/db_archive
    db_checkpoint: bin/db_checkpoint
 
 
# Additional template and view directories to add to search paths
viewsettings:
  TEMPLATEDIRS:
    - 'templates'
  VIEWPATHS:
    - 'views'


# Logging files. See &logroot for base directory.
logging:
  prefix:
    - *logroot
  EMEN2ERRLOG: emen2errlog
  EMEN2JOBFILE: emen2job
  DEBUGLOG: debug.log
  DEBUG: True
  
  

EMEN2 Configuration

EMEN2 uses JSON for the config file format, although YAML is also supported if a suitable parser is found. The default configuration file (emen2/db/config.base.json) is presented below, with comments detailing each setting.

Any config.json files found in the database directory, $EMEN2DBHOME, will be loaded on top of the base config, overriding any values. I present a typical installation's config.json file later on in this page.

Only the "paths" section is treated specially -- items without leading slashes are interpreted relative to $EMEN2DBHOME, items with leading slashes are left as absolute path references.

Base Configuration

{
        // These paths either absolute, or relative to EMEN2DBHOME, depending on the presence of leading slash
        "paths":{

                // Log directory: access.log, error.log, debug.log (see logging section) 
                "LOGPATH": "applog",
                "EMEN2JOBFILE": "applog/emen2job",
                "EMEN2ERRLOG":"applog/emen2errlog",
                "DEBUGLOG":"applog/debug.log",

                // Berkeley DB Hot Backup directory, for incremental backups and failover. Backup offsite.
                "HOTBACKUP": "db_hotbackup",

                // Berkeley DB Log Archive directory. You will want to carefully back these up.
                "LOG_ARCHIVE": "log_archive",

                // Thumbnails and other files derived from primary data
                "TILEPATH": "tiles",
                
                // Temporary directory to use for graphs and other temporary files
                "TMPPATH": "tmp",
                
                /* Raw data storage area. You may want to place this on a large raid array.
                You can also set this up to split file storage by date, e.g.:
                        '0': '/raid1/', 
                        '20080501': '/raid2/',
                        '20100101': '/raid3/'
                Files from beginning -> May 2008 will be on /raid1, May 2008 to Jan 2010 on /raid2, and files after Jan 2010 on /raid3 */               
                "BINARYPATH":{
                        "0": "emen2data"
                },

                // Path to SSL certificates
                "SSLPATH": "ssl",

                // Alternate views and templates to be loaded 
                "TEMPLATEDIRS": ["templates"],
                "VIEWPATHS": ["views"],

                // Load additional files. Typically this will only be used by the system installed base config file.
                "CONFIGFILES" :[
                        "/etc/emen2config.json",
                        "config.json"
                ]
        },

        // Mail settings. If both of these are set, EMEN2 will send emails during certain events.
        "mailsettings":{
                "MAILADMIN": false,
                "MAILHOST": false
        },

        // Web-server related settings
        "network":{
                
                // Number of web server threads to run
                "EMEN2EXTURI": "",

                "NUMTHREADS": 5,
                "EMEN2PORT": 8080,

                // HTTPS port. You will need to place the SSL certificate and key files in the SSL directory
                "EMEN2HTTPS": 0,

                // The EMEN2 URI root -- e.g., "/testserver" or "http://ncmidb.bcm.edu/challenge". Useful for running behind a reverse proxy.
                "EMEN2WEBROOT": ""
        },

        // You probably don't want to change any of these.
        "params":{
                "BLOCKLENGTH": 100000,
                "VERSION": 20100911,
                "TIMESTR": "%Y/%m/%d %H:%M:%S",
                "MAXRECURSE": 50,
                "MAXIDLE": 604800
        },

        // Some basic settings and interface strings
        "customization":{

                // Short string for database identification. This shows up several places in the UI
                "EMEN2DBNAME": "EMEN2",

                // New users will be added to these groups
                "GROUP_DEFAULTS": ["create"],
                
                // Logo in emen2/web/static/images to use
                "EMEN2LOGO": "emen2logo-small.png",
                
                // These words will not be indexed
                "UNINDEXED_WORDS": ["in", "of", "for", "this", "the", "at", "to", "from", "at", "for", "and", "it", "or"]
        },

        // Bookmarks to some key records.
        "bookmarks":{
                "BOOKMARKS":{
                        "EQUIPMENT":0,
                        "GROUPS":0,
                        "PROJECTS":0
                }
        },

        // Load the default templates.
        "viewsettings":{
                "TEMPLATEDIRS_DEFAULT":true
        },

        // Verbose logging
        "logging":{
                "DEBUG":false,
                "LOG_LEVEL":"LOG_INFO"
        },

        // Path to EMAN2 python; if you are using an EMAN2 binary you will want to set this.
        "EMAN2":{
                "EMAN2PYTHON":"python"
        },

        // Path to ImageMagick programs. Used for generating thumbnails.
        "IMAGEMAGICK":{
                "CONVERTPATH":"convert"
        }
}

Example Custom Configuration

This is a $EMEN2DBHOME/config.json file based on our EMEN2 installation. Again, note that the "paths" section is treated specially -- paths without a leading slash are interpreted relative to $EMEN2DBHOME, paths with a leading slash are left as absolute.

{
    "EMAN2": {
        "EMAN2PYTHON": "/usr/local/bin/python"
    }, 
    "bookmarks": {
        "BOOKMARKS": {
            "BANNER": 448491, 
            "BANNER_NOAUTH": 449989, 
            "EQUIPMENT": 270940, 
            "GROUPS": 0, 
            "PROJECTS": 136
        }
    }, 
    "mailsettings": {
        "MAILADMIN": "webadmin@blake.grid.bcm.edu", 
        "MAILHOST": "blake.grid.bcm.edu"
    }, 
    "network": {
        "EMEN2DBNAME": "NCMI Database", 
        "EMEN2EXTURI": "http://ncmidb.bcm.edu", 
        "EMEN2LOGO": "ncmi2006-logo2-white.png", 
        "EMEN2PORT": 80, 
        "EMEN2WEBROOT": "", 
        "NUMTHREADS": 5
    }, 
    "paths": {
        "BINARYPATH": {
            "0": "/raid1/emen2", 
            "20050101": "/raid2/emen2", 
            "20070321": "/raid3/emen2"
        }, 
        "LOG_ARCHIVE": "/home/emen2/log_archive", 
        "TEMPLATEDIRS": [
            "/home/emen2/db/overlay/templates"
        ], 
        "VIEWPATHS": [
            "/home/emen2/db/overlay/views"
        ]
    }
}

EMEN2/config (last edited 2010-09-22 07:15:32 by root)