Diff for "Eman2AppMetadata"

Differences between revisions 74 and 78 (spanning 4 versions)

Contents

Parameters/Metadata for applications stored in the project

Parameters/Metadata for applications stored in the project

These are metadata parameters associated with a 'Project', like a single particle reconstruction, and may be application specific or global. If an application wishes to make parameters available for other applications to see, it should put them in 'bdb:project' in the local directory and name them as described below. Documentation for application specific parameters not really intended for use by other applications appears here as well. This does NOT document parameters stored in the header of individual images, though we strive to use the same names whereever possible.

These parameters can be found in bdb:project

Parameters starting with 'global.' are those which could be used equally by any application, other applications wishing to publish parameters for use by others should prefix them with 'appname.'

global.apix	float	Default A/pix value for a project
global.microscope_voltage	float	Default microscope voltage in kV for a project
global.microscope_cs	float	Default microscope Cs in mm for a project
global.spr_raw_file_names	list of strings	All micrograph and/or CCD data file names that are associated with a single particle reconstruction project
global.spr_particle_file_names	list of strings	A list of all raw particle files associated with the project
global.num_cpus	int	Number of cpus available to this project
global.memory_available	int	Maximum memory usage for this project
global.particle_mass	float	The mass of the particle in kilodaltons
workflow.process_ids	list of ints	A list of process ids that is managed by the workflow system

These parameters are application specific and subject to change

Dictionary names should begin with the application name, the contents of each dictionary can be freely defined by the application.

e2ctf.parms

filename

EMAN2Ctf object

An EMAN2Ctf object associated with each filename (without extension)

e2ctf.misc

envelope

list of (x,y) tuples

Computed Envelope*StrucFact curve from CTF maxima from all sets processed during one run

e2boxer.cache

This is the cache associated with e2boxer.py.

Not that by filename_DD it is meant to be a string constructed from the original file name, for instance if the file name was 123458.mrc then filename_DD would be literally "123458_DD".

filename_DD	Python dictionary	Image Database Dictionary
filename_DD.auto_boxer_state_TS	string	Greenwich Mean time stamp of the associated Autoboxer when it was last used to autobox this image. This information can be used to force reboxing if the associated autoboxer has been updated
filename_DD.auto_boxer_unique_id	string	stores the unique ID of an associated Autoboxer instance which is also stored in the boxer_cache database
filename_DD.auto_boxes	boxertools.TrimBox list	stores the current set of automatically selected boxes by the associated Autoboxer instance
filename_DD.coarse_flat_image	EMData	stores the image generated by the boxertools.CoarsenedFlattenedImage object that is associated with this image
filename_DD.frozen_state	boolean	indicates whether or not the image is frozen. If the image is frozen then no changes can be made to the currently associated boxes
filename_DD.image_tag	string	Stores the original name of the image
filename_DD.manual_boxes	boxertools.TrimBox list	stores boxes that have been manually placed in this image
filename_DD.moved_boxes	Python list of 2D coordinates	effectively a memorization of all the box movements enforced by the user. Can be used to perform collision detection and automatically correct the results of automatic boxing
filename_DD.quality	integer [0-4]	in the special case of the quality being equal to zero this means the image is excluded from boxing and that no boxes should ever be generated for it. Otherwise this is just metadata that may be acted upon autonomously by the user
filename_DD.reference_boxes	boxertools.TrimBox list	stores the reference boxes located in this image that are also being used by the associated Autoboxer instance
filename_DD.template_ts	string	Greenwich Mean time stamp that records when the associated template (that was used to generate the correlation image, which is potentially stored in the database also) was created
autoboxer_ts	Python dictionary	the key here is literally the string "autobxer_" followed the a unique time stamp as a string. This dictionary object stores the essential parameters of a an Autoboxer instance
autoboxer_ts.autoboxer	boxertools.AutoBoxer or boxertools.TrimAutoBoxer	the Autoboxer instance itself (or similar)
autoboxer_ts.autoboxer_type	string	this is the unique associated with the Autoboxer type, for example "Swarm" or "Gauss"
autoboxer_ts.convenience_name	string	the user may optionally supply a convenience name for this autoboxer in the e2boxer interface, otherwise the name is generated automatically
current_autoboxer	boxertools.AutoBoxer or boxertools.TrimAutoBoxer	the most recently use Autoboxer instance
current_autoboxer_type	string	the type of the most recently used Autoboxer instance. This should become deprecated.

e2classaverage.indices

class number (integer stored as a string)*

list of integers

The indexes of particles that went into each class. Generated by e2classaverage if correct command line arguments are passed. Used automatically be e2refine and accessed by e2eulerxplor

* Stored as a string because of some database complication during development

emform.e2boxer

This dictionary stores the most recently used parameters associated with boxing in e2workflow.py

interface_boxsize	int	The box size that is used to launch e2boxer from within the project/workflow setting
output_boxsize	int	The last box size that was used for generating boxed output with e2boxer
force	boolean	The force flagged used for writing output in e2boxer
write_coord_files	boolean	The write coordinates flagged used for writing output in e2boxer
write_box_images	boolean	The write box images flag used for writing output in e2boxer
normproc	string	The normalization processor name used writing boxed output in e2boxer
outformat	string	The image format for boxed images generated when writing output in e2boxer
filenames	string list	The last set of filenames that was used to launch or spawn e2boxer

emform.e2ctf

This dictionary stores the most recently used parameters associated with ctf correction in e2workflow.py

ac	float	Amplitude contrast constant used to launch e2ctf from within the project/workflow setting
oversamp	float	Oversample factor used to launch e2ctf from within the project/workflow setting
nosmooth	boolean	Last nosmooth program argument used to run e2ctf from within the project/workflow setting
autohp	boolean	Last autohp program argument used to run e2ctf from within the project/workflow setting
wiener	boolean	Last wiener program argument used to run e2ctf from within the project/workflow setting
phaseflip	boolean	Last phaseflip program argument used to run e2ctf from within the project/workflow setting
filenames	string list	The last set of filenames that was used to launch or spawn e2ctf

emform.e2initialmodel

filenames	string list	The last set of filenames that was used to launch or spawn e2initialmodel
iter	int	Last iterations argument used to run e2initialmodel
symname	string	Symmetry type, i.e. c,d,icos etc, used to create the symmetry argument last used to run e2initialmodel
symnumber	string	Symmetry number ("","1","2" and so on) used to create the symmetry argument last used to run e2initialmodel
tries	int	Last tries argument used to run e2initialmodel

This dictionary stores the most recently used parameters associated with initial model generation in e2workflow.py

emform.e2refine

This dictionary stores the most recently used parameters associated with 3D refinement in e2workflow.py

amnshellsgauss	int	The nshellsgauss parameter of the mask.auto3d processor
amnshells	int	The nshells parameter of the mask.auto3d processor
amradius	int	The radius parameter of the mask.auto3d processor
amthreshold	float	The isosurface threshold parameter of the mask.auto3d processor
automask3d	boolean	The last state of the automask3d checkbox on the main page, which cause e2refine to use the mask.auto3d processor following e2make3d
classalign	string	The aligner used for alignment during iterative class averaging
classalignargs	string	classalign (The aligner) parameters
classaligncmp	string	The comparator used by the main aligner during iterative class averaging
classaligncmpargs	string	classaligncmp (comparator) parameters
classaverager	string	Averager used for averaging the images in each class
classcmp	string	The main comparator used to quality and exclude bad particles in iterative class averaging
classcmpargs	string	classcmp (comparator) parameters
classiter	int	The number of class averaging iterations
classkeep	float	The keep threshold used for excluding bad particles in iterative class averaging
classkeepsig	bool	The sigma thresholding flag used in conjunction with classkeep
classnormproc	string	The normalization processor used in class averaging
classralign	string	The refinement aligner used in iterative class averagin
classralignargs	string	classralign (aligner) parameters
classraligncmp	string	The comparator used by the refinement aligner in iterative class averaging
classraligncmpargs	string	classraligncmp (comparator) parameters
filenames	string list	The most recently selected/specified files used to create the input stack from within the workflow setting
iter	int	The number of refinement iterations
lowmem	boolean	A low memory flag used to indicate memory should be used as sparsely as possible
m3diter	int	The number of iterations used my make3d when performing the Fourier inversion method of 3D reconstruction
m3dkeep	float	The keep threshold used by e2make3d for the purpose of slice exclusion during 3D reconstruction
m3dpreprocess	string:args	The normalization processor applied prior to slice insertion during 3D reconstruction
model	string list (length 1)	The seeding 3D model, this is not an absolute path, but rather the file tag (see get_file_tag in EMAN2.py) of an image in the initial models database
orientgen	string	The method (OrientationGenerator) used to generate orientations in the asymmetric unit of the 3D model
orientopt	string	Reflects the user's choice between number based and angle based orientation generation (eventually becomes an OrientationGenerator parameter)
orientopt_entry	float	A value closely tied with orientopt, this is either an angle of, or the total number of, desired orientations
pad	int	The amount of padding used by the Fourier inversion 3D reconstruction technique
particle_set_choice	string	Stores which data set the user chose to supply as main input into the refinement process (e.g. Wiener, phase flipped, etc)
projector	string	The projector used for generating projections
recon	string	The reconstructor used for performing 3D reconstruction
sep	int	The number of classes each particles can be associated with
shrink	int	The shrink factor applied to particles prior to generation of the similarity matrix (e2simmx.py)
simalign	string	The main aligner used during similarity matrix generation
simalignargs	string	simalign (aligner) parameters
simaligncmp	string	The comparator used by the main aligner during similarity matrix generation
simaligncmpargs	string	simaligncmp (comparator) parameters
simcmp	string	The comparator used to generate the final score which is stored in the similarity matrix
simcmpargs	string	simcmp (comparator) parameters
simralign	string	The refinement aligners used during similarity matrix generation
simralignargs	string	simralign (aligner) parameters
simraligncmp	string	The comparator used by the refine align in similarity matrix generation
simraligncmpargs	string	simraligncmp (comparator) arguments
symname	string	Symmetry type, i.e. c,d,icos etc
symnumber	string	Symmetry number ("","1","2" and so on)
usefilt_choice	string	Most recently chosen usefilt option
usefilt_string	string	Most recently specified usefilt filename

emform.e2refine2d

filenames	string list	The most recently selected/specified files used to create the input stack from within the workflow setting
initial	string	Name of file containing initial class averages
iter	int	The number of refinement iterations
iterclassav	int	The number of iterations to perform when performing class averaging
naliref	int	e2refine2d argument
nbasisfp	int	e2refine2d argument
ncls	int	The number of classes to produce
normproj	boolean	e2refine2d argument
parallel	int	The number of available CPUs
particle_set_choice	string	Stores which data set the user chose to supply as main input into the refinement process (e.g. Wiener, phase flipped, etc)
shrink	int	Shrink factor for speed
simalign	string	The main aligner used during similarity matrix generation
simalignargs	string	simalign (aligner) parameters
simaligncmp	string	The comparator used by the main aligner during similarity matrix generation
simaligncmpargs	string	simaligncmp (comparator) parameters
simcmp	string	The comparator used to generate the final score which is stored in the similarity matrix
simcmpargs	string	simcmp (comparator) parameters
simralign	string	The refinement aligners used during similarity matrix generation
simralignargs	string	simralign (aligner) parameters
simraligncmp	string	The comparator used by the refine align in similarity matrix generation
simraligncmpargs	string	simraligncmp (comparator) arguments

This dictionary stores the most recently used parameters associated with 2D refinement in e2workflow.py

These parameters can be found in bdb:raw_data#

The raw_directory is used by e2workflow.py for storing imported micrograph/CCD data. The image file tag (e.g. the file tag of /home/somewhere/something.mrc is "something") may be used to access the image, for example

   1 a = EMData("bdb:raw_data#something",0)
   2 b = EMData()
   3 b.read_image("bdb:raw_data#something",0)

These parameters can be found in bdb:particles#

The short story is e2boxer stores particles in this directory, it uses a naming convention where "_ptcls" is appended to the end of the input image names.

e2ctf stores particles in this directory and appends "_ctf_flip" or "_ctf_wiener", depending whether the output data are phase flipped or Wiener filtered.

The longer story is - The particles directory is used by e2ctf, e2boxer and e2workflow for storing boxed particle data that has optionally been filtered. In general e2boxer will be the first program that stores data here, and it can be done in a number of ways. People can save directly to the particles directory from e2boxer using the 'write box particle images' button (name may be different) and by making sure the output format is 'bdb'. e2boxer can also be used from the command line for writing output (output format is a command line argument). This is precisely what occurs from the context of e2workflow, which spawns output writing processes using e2boxer.

Note that a naming convention is employed by e2boxer - if the input image names is something.mrc or "bdb:raw_data#something" then the output particles will be written using the filename "something_ptcls". You can access these images from Python using any of these approaches:

   1 a = EMData.read_images("bdb:raw_data#something_ptcls",[])
   2 n = EMUtil.get_image_count("bdb:raw_data#something_ptcls")
   3 images = []
   4 for i in range(n):
   5     images.append(EMData("bdb:raw_data#something_ptcls",i))

e2ctf will also store filtered particle images in the this directory, and these will be either phase flipped or Wiener filtered. Note that the Wiener filtered data are also phase flipped. These particles are stored in the database with "_ctf_flip" and "_ctf_wiener" appended to the input file names, respectively.

These parameters can be found in bdb:e2boxercache#

The e2boxercache directory has many dedicated particle stacks in it. The currently existing stacks are listed in this table

exclusion_image	EMData database	A binary exclusion image, in shrunken image coordinates
image_thumb	EMData database	A thumbnail of the image
subsampled_image	EMData database	A subsampled version of the image generated by the Gauss mode - Sparx group please add details
coarse_flat_image	EMData database	A coarsened version of the image that has its background flattened
coarse_sigma_image	EMData database	A specialized image that stores the inverse of the local pixel standard deviation - should probably called inverse_sigma_image - this image is in shrunken image coordinates and is used to normalize the correlation image (Roseman). Note that pixels representing 0 standard deviation are left unchanged
flcf_image	EMData database	Locally normalized correlation image (Roseman, Ultramicroscopy)

-  ⇤ ← Revision 74 as of 2009-01-21 21:10:09 → 
  Size: 21506
  Editor: DavidWoolford
  Comment:
+   ← Revision 78 as of 2009-05-08 17:40:32 → ⇥
  Size: 21860
  Editor: DavidWoolford
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 17:
-||global.micrograph_ccd_filenames||list of strings|| All micrograph and/or CCD data file names that are associated with a project ||
+||global.spr_raw_file_names||list of strings|| All micrograph and/or CCD data file names that are associated with a single particle reconstruction project ||
||global.spr_particle_file_names||list of strings|| A list of all raw particle files associated with the project ||
-Line 36:
+Line 37:
+Not that by __filename_DD__ it is meant to be a string constructed from the original file name, for instance if the file name was 123458.mrc then __filename_DD__ would be literally "123458_DD".
-Line 128:
+Line 131:
-|| model|| string || The seeding 3D model, this is not an absolute path, but rather the file tag (see get_file_tag in EMAN2.py) of an image in the initial models database ||
+|| model|| string list (length 1)|| The seeding 3D model, this is not an absolute path, but rather the file tag (see get_file_tag in EMAN2.py) of an image in the initial models database ||