Differences between revisions 55 and 56
Revision 55 as of 2014-07-23 20:55:36
Size: 12253
Editor: SteveLudtke
Comment:
Revision 56 as of 2015-01-12 18:44:04
Size: 12111
Editor: SteveLudtke
Comment:
Deletions are marked like this. Additions are marked like this.
Line 58: Line 58:
The specification for reading/writing images is:
{{{#!python
# note that optional arguments [ ] below require all previous arguments to be specified

# Read multiple images at once, class-method
imagelist=EMData.read_images(filename,[image#_list],[header_only])

# Read a single image
img=EMData()
img.read_image(filename,[image#],[header_only],[Region])
# or
img=EMData(filename,[image#],[header_only])


# write a single image
EMData.write_image(filename,image#,[filetype],[header_only],[Region],[Datatype])
}}}

Where filename, is the name of the file containing the image data, in any supported format, image# is the zero-indexed image number within the file, image#_list is a python list or tuple of image numbers, header_only is a boolean flag indicating that only the header should be read/written from/to the file, Region is a Region(x0,y0,xsize,ysize) or Region(x0,y0,z0,xsize,ysize,zsize) object.

Filetype can be : IMAGE_UNKNOWN, IMAGE_AMIRA, IMAGE_IMAGIC, IMAGE_PIF, IMAGE_SPIDER, IMAGE_VTK, IMAGE_DM3, IMAGE_GATAN2, IMAGE_LST, IMAGE_PNG, IMAGE_TIFF, IMAGE_XPLOR, IMAGE_DM4, IMAGE_HDF, IMAGE_MRC, IMAGE_SAL, IMAGE_EM, IMAGE_ICOS, IMAGE_PGM, IMAGE_SINGLE_SPIDER, IMAGE_V4L. If IMAGE_UNKNOWN is used on write, then the file extension will be used to determine the filetype. Note that since MRC format does not distinguish between 3-D volumes and stacks of 2-D images, the '.mrcs' extension MUST be used for stack files, and the '.mrc' extension MUST be used for non-stack volume data.

Datatype can be: EM_CHAR, EM_FLOAT, EM_INT, EM_UINT, EM_USHORT, EM_DOUBLE, EM_FLOAT_COMPLEX, EM_SHORT, EM_UCHAR, EM_USHORT_COMPLEX, EM_SHORT_COMPLEX. While SHORT_COMPLEX types are defined, they should never be actually used. FLOAT_COMPLEX is only really usable for HDF files. Strongly suggest not reading/writing complex images, and simply recomputing the FFT instead. Not all file formats support all data types!

If no image#_list is specified to read_images, then ALL images in the file will be read in.
Line 85: Line 111:
Region I/O permits reading or writing sub-images/volumes from within a file. This is useful when processing huge files (like full 4k tomograms) on machines with limited RAM. The region specification is the same as in EMData::get_clip() function. For region reading, it is possible to specify extending outside the actual image dimensions (missing areas are filled with 0), though this generally isn't a good idea. For region writing, the region must be completely inside image bounds. Region I/O permits reading or writing sub-images/volumes from within a file. It is not supported for all file formats. This is useful when processing huge files (like full 4k tomograms) on machines with limited RAM. For region reading, it is possible to specify a Region extending outside the actual image dimensions, though this generally isn't a good idea. For region writing, the region must be completely inside image bounds.
Line 98: Line 124:
Line 104: Line 129:
img.write_image('short-image.mrc', 0, EMUtil.ImageType.IMAGE_MRC, False, None, EMUtil.EMDataType.EM_SHORT) #write mrc file in short (16bit)
img.write_image('byte-image.mrc', 0, EMUtil.ImageType.IMAGE_MRC, False, None, EMUtil.EMDataType.EM_UCHAR) #write mrc file in byte (8bit)
img.write_image('short-image.mrc', 0, IMAGE_MRC, False, None, EM_SHORT) #write mrc file in short (16bit)
img.write_image('byte-image.mrc', 0, IMAGE_MRC, False, None, EM_UCHAR) #write mrc file in byte (8bit)
img.write_image('byte-image.spi', 0, IMAGE_UNKNOWN, False, None, EM_FLOAT) #write mrc file in byte (8bit)
Line 107: Line 133:
In the last write_image() funciton call, 'byte-image.mrc' is the output file name. The second argument is the index of the image in stack files. MRC format only supports Z-stacks, not multiple image stacks, so this number must always be 0 for MRC. The third argument, EMUtil.ImageType.IMAGE_MRC is the file type you are writing to, type 'help(EMUtil.ImageType)' in python will print out all types supported by the specific version of EMAN2 you are using. The fourth argument, False tells it to write both header and image data. If this were true, only the header would be written. The fifth argument, None means we are not doing Region I/O. This could optionally be a ''Region()'' object specifying only part of the image data should be written to disk (any other data being left unchanged). The last argument, EMUtil.EMDataType.EM_UCHAR specify the data storage type for this image file.

=== WRITING TO THE HEADER OF AN IMAGE ===
For some reason, you have to specify the type of an image via a monstrous flag if you want to write to the header of the image without actually opening/loading it.
Say you load ONLY the header of an image (in python) by doing:

{{{
a=EMData('myimage.hdf',0,True) <--- "0" means "load the first image in the file/stack", while "True" means "load the header only".
}}}

And then you define a new header parameter:

{{{
a['my_new_parameter'] = 'whatever_value'
}}}
To write out the new header into the image, you cannot simply say ''a.write_image('myimage.hdf',0,True)'', but actually have to do it this way:

{{{
img.write_image("test.hdf",0,EMUtil.ImageType.IMAGE_HDF,True)
}}}

'''EMUtil.ImageType.IMAGE_HDF'' can either correspond to the image format you are writing to, or be left undefined: ''EMUtil.ImageType.IMAGE_UNKNWON'', '''BUT''' you have to write it out nonetheless (it is what it is...).

Note that this is NOT the case if you load the ENTIRE image (opposed to just the header). If you load the ENTIRE image, you can reasonably set or reset a value on the header as follows:
{{{
img=EMData('my_file.hdf',0)
img['my_parameter']=whatever_value
img.write_image('test.hdf',0)
}}}

Table of supported image formats in EMAN2

Type

Extension

Read

Write

3D

Image Stacks

Region I/O

Comments

Primary EMAN2 Format

HDF5

hdf

Y

Y

Y

Y

Y

HDF5 is an international standard for scientific data (http://www.hdfgroup.org/HDF5/). It supports arbitrary metadata (header info) and is very portable. This is the standard interchange format for EMAN2. Chimera can read EMAN2 style HDF files.

Cryo-EM Formats

DM2 (Gatan)

dm2

Y

N

N

N

N

Proprietary Gatan format (older version)

DM3 (Gatan)

dm3

Y

N

N

N

N

Proprietary Gatan format from Digital Micrograph

DM4 (Gatan)

dm4

Y

N

Y

Y

N

Proprietary Gatan format from Digital Micrograph, used with K2 cameras

SER (FEI)

ser

Y

N

N

Y

N

Proprietary FEI format (Falcon camera ?)

EM

em

Y

Y

Y

N

Y

As produced by the EM software package

ICOS

icos

Y

Y

Y

N

Y

Old icosahedral format

Imagic

img/hed

Y

Y

Y

Y

Y

This format stores header and image data in 2 separate files. Region I/O is only available for 2D. The Imagic format in EMAN2 is fully compatible with Imagic4D standard since the 2.0 release.

MRC

mrc

Y

Y

Y

N

Y

Largely compatible with CCP4. Note that some programs will treat 3D MRC files as stacks of 2D imagess (like IMOD). This behavior is partially supported in EMAN, but be aware that it is impossible to store metadata about each image in the stack when doing this, so it is not suitable as an export format for single particle work. EMAN2 support reading of FEI MRC, which is an extended MRC format for tomography. The extra header information will be read into the header. All FEI MRC images will be 2-byte integer.

Spider Stack

spi

Y

Y

Y

Y

Y

To read the overall image header in a stacked spider file, use image_index = -1.

Spider Single

spi

Y

Y

Y

N

Y

Specify "--outtype=spidersingle" to use with e2proc2d/3d

SER

ser

Y

N

N

Y

N

Also known as TIA (Emospec) file format, used by FEI Tecnai and Titan microscope for acquiring and displaying scanned images and spectra

BDB

N/A

Y

Y

Y

Y

Y

This entry is for EMAN2's (retired) embedded database system. While it is still possible to read/write BDB's for backwards compatibility, we do not suggest any new use of this format in EMAN2 (SPARX still uses it for many operations)

Other Supported Formats

Amira

am

Y

Y

Y

N

N

A native format for the Amira visualization package

DF3

df3

Y

Y

Y

N

N

File format for POV-Ray, support 8,16,32 bit integer per pixel

FITS

fts

Y

N

Y

N

N

Widely used file format in astronomy

JPEG

jpg/jpeg

N

Y

N

N

N

Note that JPEG images use lossy compression and are NOT suitable for quantitative analysis. PNG (lossless compression) is a better alternative unless file size is of critical importance.

LST

lst

Y

Y

Y

Y

N

ASCII file contains a list of image file names and numbers. Used in EMAN1 to avoid large files. Not commonly used in EMAN2

LSTFAST

lsx/lst

Y

Y

Y

Y

N

Optomized version of LST

OMAP

omap

Y

N

Y

N

N

Also called DSN6 map, 1 byte integer per pixel

PGM

pgm

Y

Y

N

N

N

Standard graphics format with 8 bit greyscale images. No compression.

PIF

pif

Y

Y

Y

Y

N

Purdue Image Format. This will read most, but not all PIF images. Recent support added for mode 40 and 46 (boxed particles). Some of the FFT formats cannot be read by EMAN2. PIF writing is normally done in FLOAT mode, which is not used very often in PIF. PIF technically permits only images with odd dimensions, EMAN does not enforce this.

PNG

png

Y

Y

N

N

N

Excellent format for presentations. Lossless data compression, 8 bit or 16 bit per pixel

SAL

hdr/img

Y

N

N

N

N

Scans-A-Lot. Old proprietary scanner format. Separate header and data file

SITUS

situs

Y

Y

Y

N

N

Situs-specific ASCII format on a cubic lattice. Used by Situs programs

TIFF

tiff/tif

Y

Y

N

N

N

Good format for use with programs like photoshop. Some variants are good for quantitative analysis, but JPEG compression should be avoided.

V4L

v4l

Y

N

N

N

N

Used by some video-capture boards in Linux. Acquires images from the V4L2 interface in real-time(video4linux).

VTK

vtk

Y

Y

Y

N

N

Native format from Visualization Toolkit

XPLOR

xplor

Y

Y

Y

N

N

8 bytes integer, 12.5E float ASCII format

File format conversions

  • Most EMAN2 programs will read and write any of the formats above directly without conversion.
  • e2proc2d.py and e2proc3d.py provide options for saving in specific formats with specific data modes.

  • The Save as button in the e2display.py and e2projectmanager.py browsers can be used to save into an arbitrary format.

Special issues for MRC/CCP4 files

MRC/CCP4 format supports a single 1-D, 2-D or 3-D image, with an associated header. At some point in time, someone decided it would be a good idea to store sets of 2-D particle images as "stacks" in 3-D. That is, a set of NZ identically sized NX x NY images are stacked to make a single 3-D pseudo-volume image. The problem is that the original format was not designed for this, and there is no consistent way a program can tell if an MRC file contains a true volume (like a 3-D reconstruction) or a stack of 2-D images. While a number of developers have recently agreed upon a standard way of doing this in future, the last 30 years of files floating around in the community don't have this information stored in a consistent way. So, EMAN2 programs requiring a set of 2-D particle images will only read MRC stack files in specific situations.

To convert or operate on such 2-D -> 3-D stack files, e2proc2d.py has a number of special options. These options are compatible with most other options in e2proc2d.py :

  • --threed2twod - reads an MRC-style stack file and outputs to a 'normal' set of 2-D images (used by all other file formats supporting multiple images)

  • --twod2threed - reads a set of 2-D images and outputs an MRC-style stack

  • --threed2threed - reads an MRC-style stack file and outputs to another MRC-style stack file

Note: The term stack file is used universally to refer to a single file containing multiple images, regardless of whether they are 'stacked' into a single 3-D image, or actually a set of 2-D images, each with its own header. It is even possible in some formats (HDF and IMAGIC) to have stacks of 3-D volumes.

Reading and Writing images in Python (for programmers)

The main image object in EMAN2 is called EMData(). EMData objects represent an image in an arbitrary file format in the computer's memory, with arbitrary associated tag-based metadata.

Simple Image Reading/Writing

The specification for reading/writing images is:

   1 # note that optional arguments [ ] below require all previous arguments to be specified
   2 
   3 # Read multiple images at once, class-method
   4 imagelist=EMData.read_images(filename,[image#_list],[header_only])
   5 
   6 # Read a single image
   7 img=EMData()
   8 img.read_image(filename,[image#],[header_only],[Region])
   9 # or
  10 img=EMData(filename,[image#],[header_only])
  11 
  12 
  13 # write a single image
  14 EMData.write_image(filename,image#,[filetype],[header_only],[Region],[Datatype])

Where filename, is the name of the file containing the image data, in any supported format, image# is the zero-indexed image number within the file, image#_list is a python list or tuple of image numbers, header_only is a boolean flag indicating that only the header should be read/written from/to the file, Region is a Region(x0,y0,xsize,ysize) or Region(x0,y0,z0,xsize,ysize,zsize) object.

Filetype can be : IMAGE_UNKNOWN, IMAGE_AMIRA, IMAGE_IMAGIC, IMAGE_PIF, IMAGE_SPIDER, IMAGE_VTK, IMAGE_DM3, IMAGE_GATAN2, IMAGE_LST, IMAGE_PNG, IMAGE_TIFF, IMAGE_XPLOR, IMAGE_DM4, IMAGE_HDF, IMAGE_MRC, IMAGE_SAL, IMAGE_EM, IMAGE_ICOS, IMAGE_PGM, IMAGE_SINGLE_SPIDER, IMAGE_V4L. If IMAGE_UNKNOWN is used on write, then the file extension will be used to determine the filetype. Note that since MRC format does not distinguish between 3-D volumes and stacks of 2-D images, the '.mrcs' extension MUST be used for stack files, and the '.mrc' extension MUST be used for non-stack volume data.

Datatype can be: EM_CHAR, EM_FLOAT, EM_INT, EM_UINT, EM_USHORT, EM_DOUBLE, EM_FLOAT_COMPLEX, EM_SHORT, EM_UCHAR, EM_USHORT_COMPLEX, EM_SHORT_COMPLEX. While SHORT_COMPLEX types are defined, they should never be actually used. FLOAT_COMPLEX is only really usable for HDF files. Strongly suggest not reading/writing complex images, and simply recomputing the FFT instead. Not all file formats support all data types!

If no image#_list is specified to read_images, then ALL images in the file will be read in.

   1 # Create a new EMData object and initializes it with the first image in "myimage.hdf". 
   2 # This will work with any supported file format, not just HDF
   3 img=EMData("myimage.hdf")
   4 
   5 # Replace the data in EMdata object 'img' with the 3rd image from "myimage.hdf" (the first is #0)
   6 img.read_image("myimage.hdf",2)
   7 
   8 # Write an EMData object to disk as the 3rd image in "image.hdf"
   9 img.write_image("image.hdf",2)
  10 
  11 # Read all of the images from the SPIDER stack file (also works with single image files) "test.spi"
  12 # lst will become a list of EMData objects
  13 lst=EMData.read_images("test.spi")
  14 
  15 # Count the number of images available in a stack file
  16 n=EMUtil.get_image_count("myimage.hdf")
  17 
  18 # Create a new EMData object with ONLY HEADER INFORMATION from the 5th image 
  19 # in the "myimage.hdf" stack file. Any image processing operations on this object
  20 # will cause EMAN2 to crash, because it doesn't have data loaded for the actual image.
  21 # This can be useful when all you need is the header information from a bunch of images.
  22 hdr=EMData("myimage.hdf",4,True)

Region I/O

Region I/O permits reading or writing sub-images/volumes from within a file. It is not supported for all file formats. This is useful when processing huge files (like full 4k tomograms) on machines with limited RAM. For region reading, it is possible to specify a Region extending outside the actual image dimensions, though this generally isn't a good idea. For region writing, the region must be completely inside image bounds.

   1 # Read a subimage with origin (1,1,1) and size 8x8x8
   2 img = EMData
   3 region = Region(1,1,1,8,8,8)
   4 img.read_image("3dimage.hdf",0,False,region)

Storage type

Internally EMAN2 stores all images as 32-bit (single precision) floating point. Many file formats also support other storage modes. The various formats are defined in a dictionary imported from EMAN2.py: file_mode_map. There is also a file_mode_range dictionary which contains the numeric limits for each type. If you set the header values renfer_min and render_max in each image before writing, this will control how the float data is scaled to the specified mode. ie - if render_min is 0 and render_max is 1.0, then the 0-1 range in the internal image will be mapped to the full available scale of (integer mode) output formats. Note also that not all file formats support all modes.

Here are some examples of how to write in alternative formats:

img = EMData(128,128)
img.write('float-image.mrc')  #by default, image will be write as float
img.write_image('short-image.mrc', 0, IMAGE_MRC, False, None, EM_SHORT) #write mrc file in short (16bit)
img.write_image('byte-image.mrc', 0, IMAGE_MRC, False, None, EM_UCHAR) #write mrc file in byte (8bit)
img.write_image('byte-image.spi', 0, IMAGE_UNKNOWN, False, None, EM_FLOAT) #write mrc file in byte (8bit)

EMAN2/ImageFormats (last edited 2024-02-01 04:08:01 by SteveLudtke)