WARNING !

EMAN2 stores much of the data and other information during processing in an embedded database system based on BerkeleyDB. These databases exist in directories called EMAN2DB. You may be tempted to rename, delete or otherwise manipulate the files in these directories. If you feel thus tempted, you need to be aware of a number of limitations and restrictions associated with the flexibility and convenience of such database systems. Failure to heed these warnings could potentially result in data loss and a variety of apparently bizzare things happening. Regular flat files (HDF5, MRC, SPIDER, IMAGIC, etc) saved by EMAN2 are completely safe, of course, and you can do what you want with them. This warning applies strictly to the EMAN2DB directory and its contents :

Brief technical explanation

Details on the database are discussed in Eman2DataStorage

Q: Why in the blankety-blank-blank did you decide to switch to this ? It's a pain to deal with !

A: This system is an embedded database and stores information about a project, as well as much of the actual image data. Originally, we planned to use just HDF, a convenient cross-platform interdisciplinary format. However, upon testing, we found that A) its performance for large image sets was awful and B) the sorts of issues that often happen with shared filesystems multiple processes accessing the file, etc, caused corruption in the HDF files somewhat frequently, and they could not be recovered. If you do the sorts of thing that cause BDB to have problems with normal flat files, like MRC, IMAGIC, SPIDER, etc., you will get file corruption in the form of images being mispositioned, out of order, and other strange phenomena. The BDB system permits you to safely use as many threads/processes as you like within a single computer, and has specific procedures to follow when you need to go between multiple machines. It also gives a dramatic performance boost to many many tasks. So, that's why we use it...

However, it comes with a few limitations. Like most databases, it uses a memory & disk cache to give faster access to information and coordinate access to the data from multiple programs (on the same machine). This cache consists of a set of files stored in /tmp (which must be physically attached to the local machine). If you try to access the same database from two different machines at the same time via a shared network filesystem, each machine establishes an independent cache in /tmp, and both think they have exclusive access to the files. This produces a situation where the machines can easily disagree about the contents of a file, and can cause database corruption. The 'e2bdb.py -c' program will safely close the cache on one machine, so it can be reliably accessed from another machine. It is also possible in some cases to open the databases read-only from multiple machines at once, with no cache, however this is a special case used in some situations on clusters, and not a general rule.

The files in the EMAN2DB directories are not normal flat image files, but are actually proprietary database files. Moving them around or otherwise messing with them will confuse the database. Just like you wouldn't create a MySQL database and go moving around its database files wily-nily, you shouldn't mess with files in the EMAN2DB directory, particularly if there is an active cache.