Porting an EMAN1 refinement project to EMAN2

Note : This is a simplified version of the tutorial, and assumes you are familiar with EMAN1. The full tutorials provide a much better way of learning EMAN2. This page will get you started if you just want to rapidly switch a project from EMAN1. Just remember if you find yourself asking 'but what does THAT mean' when you read this, you're reading the wrong page :^)

WARNING: This page has been updated for EMAN2.1 which is now in alpha-testing. Some of the information below is not valid with EMAN2.0x

A Quickstart Guide

This quickstart makes use of the workflow, as the goal here is presumably to get the best results in EMAN2, and that is most easily accomplished in the workflow. It will take you all the way through a canonical reconstruction starting with data you've already processed in EMAN1 (or any other image processing package for that matter). This is written targeting primarily Linux/Mac users, but hopefully windows users will be able to follow along.

This may appear daunting, but there is a bit more detail here than in a typical 'quickstart' guide. It isn't nearly as complicated as its length would indicate.

Please make sure you have read this Important Warning.

For purposes of this tutorial will call the directory where your eman1 data resides 'eman1'. You should also make another directory where your EMAN2 project will reside. We will call this 'eman2', but you can use any name you like.

Outline

Since this rather long for a quickstart guide, we'll start out with a short outline of the steps:

• Use e2projectmanager.py

• Fill in project parameters
• Import particle data
• Run automatic CTF determination
• Run automatic structure factor determination
• Rerun automatic CTF determination
• Generate output
• Make a particle set
• Run 2-D refinement
• Make initial model
• Run 3-D refinement
• Even/odd test

Initial Setup

1. It is very important that you cd eman2 (again, 'eman2' is the name of your project directory) before starting the EMAN2 GUI. Whatever directory you are in when you run e2workflow.py will become your project directory.

2. Run e2projectmanager.py. A window containing an expandable list of tasks to complete and a large empty area will appear.

3. The basic project parameters are accessed via a menu item. On the Mac this will be on the menu-bar at the top of the screen. On Linux/Win it will appear at the top of the projectmanager window. Use Project->Edit Project, and enter your basic project parameters.

Importing Particle Data

1. There are 3 possibilities for getting your data into EMAN2. You MUST reprocess the CTF in EMAN2, rather than use the already processed data from EMAN1 (unless you don't plan on doing any CTF correction), but on the bright side, this process is much easier in EMAN2. So, the data we will import is the un phase-flipped data. It must also be completely unfiltered. Just the original, raw data. Pick only one of the following 3 sections in bold:

   • If you have the original micrographs/CCD frames, and EMAN1 style .box files :

     1. There are two tools under Raw Data' you can use to import the micrographs into the project:

        • Import Micrographs will just copy your images into a Micrographs directory and make sure they are in HDF format.

        • Evaluate & Import Micrographs will allow you to visually assess each image and its power spectrum in various ways, then press the Import button only for those you decide have sufficient quality. This tool can also simultaneously provide an estimate of the defocus, which will largely eliminate fitting failures during later CTF processing.

     2. There are a few options you can select as well.

        • It is important to get Inversion right. The final particles MUST appear white on a darker background, so whether you check this box depends on whether you are using stain or cryo, and how the data was digitized.

        • Filter X-ray Pixels is important for CCD data, but should not be used if you are using a phase-plate or film data.

     3. Under Particles, select Box Coordinate Import. Assuming your particle and box filenames match, everything should work well.

     4. You could actually go to the interactive particle picker and check/update the boxes at this point if you like, but we will just assume they are correct, and use Generate Output under Particles. Again, there are some options:

        • Box size is the most important one. In EMAN2, it is VERY important, if you want to obtain optimal results, to use a box size that is 1.5-2x the size of the maximum dimension of your particle. If you are dealing with a huge virus particle, you may need to skimp a bit on this, to avoid having a, say, 1500 box size, but some padding is essential for CTF correction to work well, and to avoid artifacts. See this list of good sizes to use. If you decide to change the box size later, it generally requires starting a new project from scratch, so consider this number very carefully.

        • You should select write_ptcls, normalize.edgemean and hdf.

        • Only select invert here if you made a mistake at the earlier step. Remember, particles MUST be light on a darker background.

   • If you only have or want to use particle stacks sorted by micrograph:

     1. Double check to make sure your particle stacks have light particles on a darker background. If they have inverted contrast, you will need to invert them before proceeding. You can do this with e2proc2d.py <file> <file> --mult=-1 --inplace or any other method you like. Having this correct is CRITICAL to getting good results in EMAN2.

     2. Use Particles->Particle Import.

   • If you only have a start.hed file containing phase-flipped particles from EMAN1 :

     • This is not an optimal situation, but it is still possible (assuming you have EMAN1 installed).
     • If the start.hed file contains particles which have been phase-flipped. You must first use EMAN1's 'applyctf' program to invert the phases a second time so the images are un-flipped again.
     • In EMAN1, use proc2d with the ctfsplit option to turn your start.hed file into a set of individual particle stacks (for each micrograph)
     • Now that you have particle stacks for individual images, follow the instructions above to import the stacks.

CTF Fitting/Correction (even for stain and phase plate)

1. Now that your particles have been imported into the EMAN2 project, you can begin CTF correction. It is important to note at this point that CTF correction in EMAN2 entails a lot more than just CTF correction. It also makes an SSNR evaluation for each micrograph, which is used to perform weighting and other tasks. So, even if you do not feel you need to do CTF correction (neg stain or phase plates for example), we strongly encourage you to go through this process anyway.

2. Use CTF->Automated Fitting. Since you want to do initial fitting for all images, you have two approaches. You can use Browse and select all of the image files manually, or instead, you can leave the 'particles' line blank and check the 'allparticles' checkbox. minptcl and minqual allow you to filter out certain image sets. Oversamp should be set to 2. Make sure the other values are correct. Normally you will want autohp, and curdefocushint checked. autofit must be selected. Fitting should only take 1-2 seconds per image.

3. When it's done, select Interactive Tuning. At this point you do not need to check all of the images, but just a few you plan to use for structure factor determination. Click on the Num Particles column to sort in decreasing order. Then select the first 5-10 images in the list.

4. Four windows will appear: a control panel, a 2-D power spectrum view, a particle view, and a 2-D plot. In this case all we need to do is make sure the defocus is roughly correct. This is the most common error in automatic CTF fitting. Either it will be completely correct, or the defocus will be way off. If you find an image with the wrong defocus, adjust it manually to roughly the correct position, and hit the refit button. If it is incorrect again after refit, then manually adjust to the correct value and Save Params.

5. NOTE: If you are working with phase-plate data, hopefully your images will all have defocus of 0. The autofitting program is incapable of determining defocuses where fewer than 2 zeros appear in the power spectrum. So, for phase plate images, you will need to enter a defocus manually for all images.

6. Next, we need to generate a structure factor (while you could use your own structure factor curve by replacing strucfac.txt with your own file, normally making one will give better results). When you open the Generate Structure Factor tool, it should already have the same micrographs you used in the previous step selected, so you only need to reselect if some of those images were bad. Generating a structure factor will usually take only a few seconds.

7. The generated structure factor goes into a file called strucfac.txt. You can plot this file vs any other structure factor to see how it compares if you wish. Unlike EMAN2.0x, in EMAN2.1 this text file is used for all later CTF correction situations requiring a structure factor.

8. Repeat automatic CTF fitting for all images as you did it earlier. Usually having a structure factor will improve fitting accuracy, however, there is generally no point in iterating the process (by making the structure factor again).

9. You should really run Interactive Tuning again. You are welcome to check all of the images, or just a few of them. If you check a few, and find that the fitting is poor, you should probably check the rest as well. If this happens, please email sludtke@bcm.edu, as such failures are valuable in improving the software. You may optionally go through the data and adjust the Quality slider to indicate relative quality of each image. This is an entirely user controlled parameter, and is not actually required directly by any of the programs. Note that you do NOT need to press the Save Parm button after adjusting the quality slider, but do if you adjust anything else.

10. Once you are happy with all of the CTF parameters, use CTF->Generate Output. Usually selecting refinebysnr will slightly improve the defocus values. You will usually want phaseflip and wiener checked. phasefliphp' can be useful if you have a lot of ice gradients or if working in negative stain, but in some cases can have very negative effects. You'll have to play with it.

Particle Sets

1. When performing a single particle reconstruction, often it is useful to work with only part of your data for specific steps. For example, when generating an initial model, the process goes much faster if you only use the best fraction of your data, and generally ~1000 particles or so is sufficient for this process (fewer if the contrast is high). When performing a 3-D reconstruction, you may wish to assess the quality of the model if only the best fraction of the data is used, vs the entire data set. The Sets interface allows you to create named subsets of your data for the later stages of processing. You must make at least 1 set to proceed though the Workflow.

2. Select Particle Sets->Build Particle Sets. Select the images you want to use in the set, and give your set a name. Filetype should be lst. If you check excludebad it will exclude particles which have been marked as bad using one of several different mechanisms. (In EMAN2.1alpha1 it is tricky to do manual bad particle selection, a new tool for this will appear later...)

Reference Free Class-averages

1. If you have a good 3-D starting model already, and wish to skip directy to 3-D refinement, you can skip this and move down to Running a 3-D Refinement.

2. It is often useful to make some reference-free 2-D class-averages to assess your particle's structural variability, and to compare to projections of the 3-D map later. They are also useful to build an initial model. This is the equivalent of the refine2d.py command in EMAN1.

3. Select Reference Free Class Averages->Generate Classes.

   • The input file should be a set and usually will be the ctf_flip or ctf_flip_hp file.

   • Generally the default options are fine for a first try, though if you have multiple CPUs/cores available on your computer, you may wish to put thread:n where n is the number of CPUs in the parallel box. You could even run on a cluster with MPI or other parallelism scheme, if you like, though this job won't take all that long to run in most cases. After a trial run, you may play with the options a bit.

4. When the job has finished, you will find the results in a directory called r2d_xx. Each time you run 2d refinement, it will increment xx. There are many files in this directory, but the main file to look at is either classes_xx or allrefs_xx. You can use the file browser to look at the results from the final iteration.

5. The images in 'classes_xx' are in arbitrary order. 'allrefs_xx' contains the particles sorted in order of mutual similarity and aligned to each other in 2-D. If you wish to see which particles went into a specific class-average, you can simply double-click on that particle, and an additional window will open showing the contributing particles. It will also show particles which were classified in that class, but were excluded from the final average. These will be marked with a blue square in the corner. Note that these particles are not excluded from the set or from any subsequent process, they were simply excluded from that specific average.
6. You may also wish to look at 'basis_xx', which contains the MSA basis vectors, and can tell you something about the symmetry and shape of your particles, if you understand how MSA works.

7. If you observe some class-averages which clearly contain entirely bad particles, you can use this as a mechanism for permanently excluding those particles from future sets you create. To do this, you use the 'Evaluate Particle Sets' interface in the workflow. You can also use this interface to select particles from specific class-averages for further rounds of 2-D or 3-D refinement. It's detailed use is beyond the scope of this quickstart guide, but the corresponding command-line program is e2evalparticles.py. (note that this is broken in EMAN2.1alpha1)

Making an Initial 3-D Model

1. EMAN2 uses an approach for initial particle generation that was always suggested in EMAN1, but had to be performed manually. This process is fully automated in EMAN2. It does not work on 100% of projects, but I would estimate it works on ~80%. If you have a project which you cannot get working this way, or do not trust your results, I suggest using Random Conical Tilt, or Single Particle Tomography to get an initial model. Both methods are available in EMAN2, including corresponding GUI tools.

2. First you need to select a good subset of your class-averages to use in initial model generation. To do this, open the classes_xx file in the browser, then middle-click on the image display to bring up a control-panel (alt-click without a 3-button mouse). Select Del mode in the control panel, then click on images you do not wish to use in initial model generation to delete them. You will want to use ~10-20 averages for your initial model. Try to pick averages representing diverse views of your particle, but do not include anything that is questionable or bad. When you have the desired averages, use the Save button to put them into a file: good-classes.hdf.

3. Select Make Model from the Workflow. Press 'Browse to Add' and select the good-classes.hdf file you just created. Hilight this file once it's added.

4. If you don't know the symmetry for sure. I suggest starting with the highest symmetry you suspect is possible. Do not rely on it to identify the symmetry for you.
5. This process basically starts with a completely random pattern of blobs, and does a full single particle refinement, using the class-averages as particles. It does this for N different starting models, and iterates each M times. Some structures converge faster than others, M=8 is a reasonable value for most structures, but you may find yourself needing to try twice that many for difficult cases. N doesn't really matter too much, as if you don't get a good model on the first pass, you can always try N more. I suggest starting with Iterations=8 and Tries=5.
6. When this is complete, you will find several files in the initial_models directory. There are 4 types of files:
   • model_xx_yy : This is the refined model you would use as a starting model
   • model_xx_yy_aptcl : Contains the class-averages alternating with the corresponding projections of the model after the final iteration. Poor agreement between pairs is an indication of a bad initial model.
   • model_xx_yy_proj : Projections from the final round of refinement covering the asymmetric triangle.
   • model_xx_yy_init : The initial model for this refinement. Just in case you want to see it.
   • xx is the run number. If you run the initial model generator a second time, this number will be incremented.
   • yy is the model number. In theory these are sorted in order of quality at the end of the run, so 01 will be the best. However, this isn't very reliable. You should check all of the models, and all of the _aptcl files to find the best one.
7. If you didn't get a good model, run it again, until you're happy. Note that these models are not going to be perfect. They just need to be vaguely the correct shape, so when you run a 'real' refinement it will converge to the right thing. If you have no success with this approach, random conical tilt and single particle tomography are both viable alternatives, though both will require additional data collection.

Running a 3-D Refinement

1. At last, to the meat of the matter. 3-D refinements in EMAN2 have more options than in EMAN1, though the fundamental concepts are basically the same. When you select Run - e2refine, you will first see a dialog asking which types of data you wish to use for the refinement. There are some minor issues with the usefilt option, so I suggest using None. Normally for the particle data, you would use either phase-flipped or phase-flipped-hp.

2. The options in the Particles tab should all be fairly obvious to EMAN1 users, and based on what you've already done.

3. The options in the Model tab allow you to select the initial model, and the automasking parameters applied at the end of each refinement cycle. If the initial model you want to use isn't in the list, you can Browse to Add.

4. In the Project3d tab, you basically select the symmetry and angular step for generating projections. You may be able to use a smaller angular step with EMAN2 than you used in EMAN1, due to the 2 stage option in the next tab. Normally the angular step is at most ~6 degrees.

5. The simmx tab is the first place where there are real differences with EMAN1.

   • The 2 stage simmx option is one of the biggest innovations in EMAN2. It is somewhat similar to the 'tree' option in EMAN1, but done properly so the results are robust. If a non zero integer is entered in this box, then the particle orientation will take place in two stages, and during the first stage it will shrink the data by the factor entered in the box. This is done completely differently than in EMAN1. The larger the project, the greater the speedup using this option will achieve. For very small projects, the speedup may be as little as 2x, but for very large projects, it can be over 30x.

   • In EMAN1, there were 3 related options phasecls, fscls, and dfilt. Each representing a different choice for a similarity metric to use during alignment and classification of the particles. dfilt was almost always the best option. In EMAN2, this has been broken down into several options. You can select which algorithm is used for the first stage of alignment, and which similarity metric this algorithm uses, then you can select the algorithm (refine or none) used for the second stage of alignment and the similarity metric for that. Finally, you select the similarity metric used for classification after alignment. These parameters can potentially all be different. The text at the top of this tab has some suggestions.

6. The class-averaging tab has options related to making class-averages.

   • Some of these options are duplicated from the previous page with a similar purpose.

   • There are basically 3 choices for averager: ctf.auto - Full CTF correction without Wiener filter, ctfw.auto - Full CTF correction with Wiener filter, mean - a simple average of all of the particles with no weighting or CTF correction. Unlike EMAN1, even if you Wiener filter the class-averages, when the 3-D reconstruction is performed, the 2-D Wiener filter can be undone, and reapplied properly in 3-D.

   • Set proj SF on avg causes the 1-D structure factor of each projection to be applied (via filtration) to the corresponding class-average.

   • Averaging iterations is equivalent to classiter in EMAN1, but the numbers are not skewed. In EMAN1, classiter=3 would perform 1 iteration. In EMAN2 1 is the correct value to use.

   • Class separation is the equivalent to EMAN1 sep.

7. Finally, the Make3d tab has options related to 3-D reconstruction. For Reconstructor, you have basically 2 choices fourier or wiener_fourier. If you are using the setsf option, which imposes the computed power spectrum on the map, then it really doesn't matter which you pick. Reconstruction Iterations can be 2 or 3. 3 is very slightly more accurate and takes longer. Post Processor can be set to apply any single filter that you like. A common choice would be to check setsf, then use filter.lowpass.gauss with cutoff_freq=F, and F=1/resolution in A. If you are not using setsf, clearly you shouldn't filter as strongly, as the map will already be filtered. The other options should all be familiar.

8. When you say OK it will start the refinement, and also print the refinement command on the console. If you are planning to run using MPI on a different computer, the best approach is to launch the refinement from the GUI with no options set for Parallel, then immediately kill the job with kill <pid>. Then you can paste the command it ran, with the appropriate --parallel option added into your script file. Be sure to run e2bdb.py -c before launching the job.

9. Refinement results go into sequentially named refine_xx' directories. Unlike EMAN1, you cannot continue a refinement from where it left off in the same directory. Each time you run a refinement, it starts from scratch in a new subdirectory. This makes it much easier to track exactly what was done each time it was run. Simply use the final threed_filt_xx' file from the previous round of refinement as a starting model, and it's equivalent to EMAN1.

10. When the refinement is done, e2eotest is run similarly. The Resolution item will show you both the even/odd test results as well as the traditional EMAN1 convergence plot. The Eulers item allows you to look at the distribution of particle orientations, and compare class-averages to projections.

This is the end of this quickstart guide. There is a lot of documentation around for the details not covered here, but this should get you started. Good luck !