|
Size: 6100
Comment:
|
← Revision 30 as of 2010-09-22 07:18:44 ⇥
Size: 2107
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| ## page was renamed from EMEN2/Backups | |
| Line 3: | Line 4: |
| There are several ways to backup your EMEN2 environment. | An EMEN2 installation consists of several components: |
| Line 5: | Line 6: |
| At the simplest level, you can just copy all the files directly to another location. This is a "cold backup," and is the simplest and most reliable backup mechanism, but requires that you stop all database writes for the duration of the process to ensure the integrity of the archive. Alternatively, you can perform a "hot backup," which can be performed at any time, even while the database is active. A hot backup copies database log files to an existing cold backup to bring it up-to-date with the current state. I recommend running cold backups about once a week, with hot backups daily. | * EMEN2 database environment ([[EMEN2/config.yml|DB_HOME]]) * File storage area ([[EMEN2/config.yml|BINARYPATH]]) * Image browser cache area ([[EMEN2/config.yml|TILEPATH]]) * Web server log, log file archive and database backup ([[EMEN2/config.yml|LOGPATH]], [[EMEN2/config.yml|ARCHIVEPATH]], [[EMEN2/config.yml|BACKUPPATH]]) <<TableOfContents>> = EMEN2 Database Environment = There are several ways to backup your EMEN2 database environment. In the simplest case, you can perform a normal copy of the database environment. This is a "cold backup", and is the easiest and most reliable mechanism, but requires that you stop all database writes for the duration of the process to ensure the integrity of the archive (see "long answer"). If your uptime requirements are not stringent, performing cold backups once a night or once a week using normal shell tools (see simple answer below) may be all you need. If you want to perform very frequent backups, or do not want to stop the database environment, you can perform a "hot backup," which can be performed even while the database is active. |
| Line 8: | Line 22: |
| == Cold Backup == | = File Storage Area, Logs, etc. = |
| Line 10: | Line 24: |
| To create a cold backup, shut down any open database processes, and use the EMEN2 backup utility with the "--cold" argument. | The other directories are just normal files on disk and can handled with standard shell tools. = Backup: Simple Answer = Shutdown the web server (see [[EMEN2/emen2control.py]]) and perform a normal cold backup of everything by rsync'ing to a remote backup server. This will be sufficient for most installations. Example (default config, everything in /home/emen2): |
| Line 13: | Line 35: |
| backup.py --cold }}} This will run a database checkpoint, and create a cold backup in the path specified by [[EMEN2/config.yml|config:BACKUP]]. To prevent overwriting an existing cold backup, it will not run if the target directory exists. You should move or remove an existing cold backup first. You can update this cold backup by running a hot backup. Example: {{{ # python ./cmdlineutils/backup.py --cold ... snip: startup ... Opening Database Environment: /home/emen2/db/ Cold Backup: Checkpoint Cold Backup: Copying data: /home/emen2/db/data -> /home/emen2/db_backup/data Cold Backup: Copying config: /home/emen2/db/config.yml -> /home/emen2/db_backup/config.yml Cold Backup: Copying config: /home/emen2/db/DB_CONFIG -> /home/emen2/db_backup/DB_CONFIG Cold Backup: Copying log: /home/emen2/db/log/log.0000000311 -> /home/emen2/db_backup/log/log.0000000311 |
[emen2@ncmidb ~/emen2]# python ./emen2control.py --shutdown [emen2@ncmidb ~/emen2]# python ./emen2control.py --recover [emen2@ncmidb ~]# cd /home/emen2/ [emen2@ncmidb ~]# rsync -vr db db_backup log_archive applog emen2files emen2tiles emen2 emen2@remotebackup:~/emen2backup/ .... |
| Line 33: | Line 42: |
| == Hot Backup == | = Hot backup, incremental backup, log files, etc. = |
| Line 35: | Line 44: |
| {{{ backup.py --hot }}} {{{ # python ./cmdlineutils/backup.py --cold ... snip: startup ... Opening Database Environment: /home/emen2/db/ Hot Backup: Log archive Log Archive: Checkpoint Log Archive: /home/emen2/db/log/log.0000000303 -> /home/emen2/log_archive/log.0000000303 Log Archive: /home/emen2/db/log/log.0000000304 -> /home/emen2/log_archive/log.0000000304 Log Archive: /home/emen2/db/log/log.0000000305 -> /home/emen2/log_archive/log.0000000305 Log Archive: /home/emen2/db/log/log.0000000306 -> /home/emen2/log_archive/log.0000000306 Log Archive: /home/emen2/db/log/log.0000000307 -> /home/emen2/log_archive/log.0000000307 Log Archive: /home/emen2/db/log/log.0000000308 -> /home/emen2/log_archive/log.0000000308 Log Archive: /home/emen2/db/log/log.0000000309 -> /home/emen2/log_archive/log.0000000309 Log Archive: /home/emen2/db/log/log.0000000310 -> /home/emen2/log_archive/log.0000000310 Hot Backup: Copying log: /home/emen2/db/log/log.0000000303 -> /home/emen2/db_backup/log/log.0000000303 Hot Backup: Copying log: /home/emen2/db/log/log.0000000304 -> /home/emen2/db_backup/log/log.0000000304 Hot Backup: Copying log: /home/emen2/db/log/log.0000000305 -> /home/emen2/db_backup/log/log.0000000305 Hot Backup: Copying log: /home/emen2/db/log/log.0000000306 -> /home/emen2/db_backup/log/log.0000000306 Hot Backup: Copying log: /home/emen2/db/log/log.0000000307 -> /home/emen2/db_backup/log/log.0000000307 Hot Backup: Copying log: /home/emen2/db/log/log.0000000308 -> /home/emen2/db_backup/log/log.0000000308 Hot Backup: Copying log: /home/emen2/db/log/log.0000000309 -> /home/emen2/db_backup/log/log.0000000309 Hot Backup: Copying log: /home/emen2/db/log/log.0000000310 -> /home/emen2/db_backup/log/log.0000000310 Hot Backup: Copying log: /home/emen2/db/log/log.0000000311 -> /home/emen2/db_backup/log/log.0000000311 Log Archive: Checkpoint Log Archive: Removing /home/emen2/db/log/log.0000000303 Log Archive: Removing /home/emen2/db/log/log.0000000304 Log Archive: Removing /home/emen2/db/log/log.0000000305 Log Archive: Removing /home/emen2/db/log/log.0000000306 Log Archive: Removing /home/emen2/db/log/log.0000000307 Log Archive: Removing /home/emen2/db/log/log.0000000308 Log Archive: Removing /home/emen2/db/log/log.0000000309 Log Archive: Removing /home/emen2/db/log/log.0000000310 }}} == Log Archive == EMEN2 uses Berkeley DB as the underlying database technology. To provide guarantees about transaction atomicity and durability, Berkeley DB writes changes to a log file on stable storage before a transaction is marked as committed. Database files are not updated until this step is complete. In the event of a crash or hardware failure, the database files can be checked against the log files to correct any errors or missing data. The log files are stored in $DB_HOME/log as log.XX, where XX is a sequential integer starting from 1. With default settings, the files are 8 MB each. As one log file is finished, the next log file in the sequence is created and used as the active log file. Log files that are not being currently open or used may be archived after a "checkpoint" is made. This frees up disk space in the DB_HOME environment, and lets the administrator move the logs to long term archival storage. The EMEN2 backup utility will archive finished logs when run with the "--archive" flag. Finished log files will be moved to the location specified by the "ARCHIVE" configuration setting. {{{ backup.py --archive }}} Example: {{{ # python ./cmdlineutils/backup.py --archive ... snip: startup ... Opening Database Environment: /home/emen2/db/ Log Archive: Checkpoint Log Archive: /home/emen2/db/log/log.0000000303 -> /home/emen2/log_archive/log.0000000303 Log Archive: /home/emen2/db/log/log.0000000304 -> /home/emen2/log_archive/log.0000000304 Log Archive: /home/emen2/db/log/log.0000000305 -> /home/emen2/log_archive/log.0000000305 }}} The log archive directory should be part of your normal backup procedures. In a "worst case scenario" failure, these log files will be necessary to rebuild the database. You may want to run this periodically as a cron job, combined with an action to copy the log archive directory to a different system. In addition to log archiving, you will want to maintain hot and cold backups. |
For a more detailed discussion, see [[EMEN2/BackupDiscussion|Backup Discussion]] |
EMEN2 Maintenance and Backups
An EMEN2 installation consists of several components:
EMEN2 database environment (DB_HOME)
File storage area (BINARYPATH)
Image browser cache area (TILEPATH)
Web server log, log file archive and database backup (LOGPATH, ARCHIVEPATH, BACKUPPATH)
Contents
EMEN2 Database Environment
There are several ways to backup your EMEN2 database environment.
In the simplest case, you can perform a normal copy of the database environment. This is a "cold backup", and is the easiest and most reliable mechanism, but requires that you stop all database writes for the duration of the process to ensure the integrity of the archive (see "long answer"). If your uptime requirements are not stringent, performing cold backups once a night or once a week using normal shell tools (see simple answer below) may be all you need.
If you want to perform very frequent backups, or do not want to stop the database environment, you can perform a "hot backup," which can be performed even while the database is active.
File Storage Area, Logs, etc.
The other directories are just normal files on disk and can handled with standard shell tools.
Backup: Simple Answer
Shutdown the web server (see EMEN2/emen2control.py) and perform a normal cold backup of everything by rsync'ing to a remote backup server. This will be sufficient for most installations.
Example (default config, everything in /home/emen2):
[emen2@ncmidb ~/emen2]# python ./emen2control.py --shutdown
[emen2@ncmidb ~/emen2]# python ./emen2control.py --recover
[emen2@ncmidb ~]# cd /home/emen2/
[emen2@ncmidb ~]# rsync -vr db db_backup log_archive applog emen2files emen2tiles emen2 emen2@remotebackup:~/emen2backup/
....
Hot backup, incremental backup, log files, etc.
For a more detailed discussion, see Backup Discussion