Differences between revisions 8 and 31 (spanning 23 versions)
Revision 8 as of 2016-11-10 18:27:15
Size: 3396
Editor: MuyuanChen
Comment:
Revision 31 as of 2019-12-03 13:43:43
Size: 24026
Editor: SteveLudtke
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
== Tomogram Segmentation ==
Availability: EMAN2 daily build after 2016-10
== Tomogram Annotation/Segmentation ==
Availability: EMAN2.2+, or daily build after 2016-10
This is the tutorial for a Convolutional neural network(CNN) based semi-automated cellular tomogram annotation protocol.
Line 4: Line 5:
Programs in the tomogram segmentation requires Theano, which is not distributed with EMAN2. To use the protocol, one needs to build EMAN2 from source and install Theano manually. If you are using this in a publication, please cite:
 M. Chen et al. (2017). Convolutional Neural Networks for Automated Annotation of Cellular Cryo-Electron Tomograms. Nature Methods, http://dx.doi.org/10.1038/nmeth.4405
Line 6: Line 8:
http://deeplearning.net/software/theano/install.html A stable (thus not up-to-date) online version of the tutorial can also be found at Protocol exchange: https://www.nature.com/protocolexchange/protocols/6171
Line 8: Line 10:
== Getting Started ==
First, make an empty directory and get into that directory in command line. Then run e2projectmanager.py from the command line. While a GUI window will show up, it is still a good idea to keep the command line window open to view the messages.
Updates:
 * 10/2018: Now it uses Tensorflow backend which should be included during installation. Also particle boxing interface and directory structure are updated to incorporate into the new [[http://blake.bcm.edu/emanwiki/e2tomo | e2tomo ]] pipeline.
 * 01/2018: Slides of a recent tutorial in Rutgers boot camp can be found here. [[attachment:tomoseg_tutorial_rutgers.pdf]]
 * 10/2017: We have switched to gpuarray backend for CUDA9 support. It should still work with CUDA7.5+ and the old cuda backend in Theano.
Line 11: Line 15:
Click the '''Workflow Mode''' drop-down menu next to navigate to the TomoSeg panel. == Installation ==
CNN based tomogram annotation makes use of the !TensorFlow deep learning toolkit. At the time of this writing, tensorflow effectively requires CUDA, the compute framework on NVidia GPUs. While future support for OpenCL (an open alternative to CUDA) seems likely, and while it is possible to use TensorFlow on a CPU, at present we cannot recommend running this tool on Mac/Windows. You may do so, but training the networks may be VERY slow.
Line 13: Line 18:
{{attachment:tomoseg_panel.png|TomoSeg Panel|width="400"}} To make use of accelerated TensorFlow, your machine must be properly configured for CUDA. This is outside the installation of EMAN2, and requires specific drivers, etc. on your machine. Some extra information on how to use CUDA and GPU can be found at http://blake.bcm.edu/emanwiki/EMAN2/Install/BinaryInstallAnaconda#Using_the_GPU
Line 15: Line 20:
== Import Tomograms ==
Click "Import Tomogram Files" on the left panel. On the panel showed up on the right, click "Browse" next to import_files, and select the tomogram you would like to segment in the browser window, and click "Ok". If you want to bin the tomogram before processing, write the shrinking factor in the text box next to "shrink". Make sure that the "import_tomos" and "tomoseg_auto" box is checked. Finally, click "Launch" and wait the pre-process to finish.
== Setup workspace ==
If you used EMAN2 for your tomogram alignment/reconstruction ([[EMAN2/e2TomoSmall]]), simply 'cd' to the project folder and launch e2projectmanager.py. If you will be using tomograms produced by other software:
Line 18: Line 23:
{{attachment:import_tomo.png|Import Tomogram|width="400"}} Make an empty folder and cd into that folder at the command-line. Run e2projectmanager.py.
 
Select the '''Tomo''' Workflow Mode.
Line 20: Line 27:
== Select Positive Samples == == Import or Preprocess ==
If your tomograms were produced using the EMAN2 workflow:
 * You should preprocess your tomograms to prepare them for segmentation.
  * Select Segmentation -> Preprocess Tomograms
  * Press '''Browse''', '''Sel All''' then '''OK'''
  * All of the other options should normally be fine, and you can press '''Launch'''
Line 22: Line 34:
Open "Box training references". Press browse, and select your imported tomogram. Leave “boxsize” at -1, and press Launch. If your tomograms were produced by other software, and you wish to do the segmentation in EMAN2:
 * If your original tomograms are in multiple folders, you can repeat the next step.
 * Select Raw Data -> Import Tomograms
  * Use '''Browse''' to select all of the tomograms you wish to segment, and press '''OK'''
  * EMAN2 uses the convention that tomograms have "dark" or negative contrast, and individual particles and 3-D averages will have "white" or positive contrast. If your tomograms have positive contrast (features look bright when displayed) check the '''Invert''' box.
  * If your tomograms are larger than ~1k x 1k, you should consider downsampling, or '''Shrink'''ing them for segmentation. Normally 1k x 1k is more than sufficient for accurate segmentation. ie - if your original tomograms are 4k x 4k, enter ''4'' in the '''shrink''' box.
  * In deciding how much to shrink the original tomograms, it is only important that the features you wish to recognize are still recognizable by eye. If shrinking produces a tomogram where a feature you wish to identify is only 1-2 pixels in size, then you are shinking too much.
  * '''Launch'''
 * Once imported, you should preprocess your tomograms to prepare them for segmentation.
  * Select Segmentation -> Preprocess Tomograms
  * Press '''Browse''', '''Sel All''' then '''OK'''
  * All of the other options should normally be fine, and you can press '''Launch'''
Line 24: Line 47:
In a moment, three windows will appear on your screen, which will be familiar if you’ve boxed particles before. The only difference between this boxing and the other is that you can box in 2D on slices of a 3D image. {{attachment:import_tomo.png|Import Tomogram|width="600"}}
Line 26: Line 49:
On the window named “e2boxer”, make sure your box size is 64. None of the other options need to be changed.  * Note: There may be cases where it is impossible to pick a single scale which works well to annotate features at different scales. For example, if you are trying to annotate icosahedral viruses (large scale) and actin filaments (require small scale to identify) in the same tomogram, you may need to import the tomogram twice, with 2 different "shrink" factors, segment each feature at the appropriate scale, then rescale/merge the final segmentation maps. This is a topic for advanced users, and we encourage you to learn the basic pipeline with a single feature before attempting this.
Line 28: Line 51:
On the window containing your tomogram, you can begin selecting boxes. Go up and down in the tomogram using the arrow keys, select and drag boxes using the left mouse button, and delete boxes using Shift + left mouse button. As you select boxes, they will appear in the (Particles) window. == Overview ==
The network training process may seem complicated at first, but in reality it is extremely simple. You will be training a neural network to look at your tomogram slice by slice, and for each pixel it will decide whether it does or does not look like part of the feature you are trying to identify. This feature is termed the Feature of Interest (FOI). To accomplish this task, we must provide a few manual annotations to teach the network what it should be looking for. We must also provide some regions from the tomogram which do not contain the FOI as negative examples. That is, you are giving the network several examples of: "I am looking for THIS", and several more examples of: "It does NOT look like THIS".
Line 30: Line 54:
Select around 10 boxes containing your structure. If your structure appears differently throughout the cell (e.g. microtubules), be sure to include a variety of views in the boxes. For the network to perform this task well, it is critical that you be absolutely certain about the regions you select for training. The largest error people make when training the network is to include regions containing ambiguous density. If you select a tile containing a clear feature you wish to include, but the tile also contains some density you are unsure of, then you will be in a difficult situation. If you mark the ambiguous region as a particle, and it actually is something else, the trained network will not be very selective for the FOI. If you do not mark the ambiguous density, and it turns out to actually be the FOI, then you are trying to train the network to distinguish between the object you said was the ROI and the object you said was not (but was), which may cause a training failure, or a network which misses many FOIs. The correct solution is ''not to include any tiles containing ambiguous density''! Only include training tiles you are certain that you can annotate accurately in 2-D.
Line 32: Line 56:
When selecting boxes, ensure that your structure is clear in the (Particles) window. You will have to manually segment these boxes, so if you can’t see your structure, your segmentation will be more difficult, and your final segmentation will suffer as a result. It is better to have fewer boxes that you can segment better than more boxes you segment worse. There will be other cases where it may be difficult to distinguish between two features. For example, at low resolution, the strong line of an actin filament, may look somewhat like the edge of a membrane on an organelle. If you train only a single network, and it is trained to recognize actin, then you apply it to a cellular tomogram, you will likely find many other features, such as membrane edges, and possibly the edge of the holy carbon film, get mis-identified as actin. While you may be able to more carefully train the actin network, or alter the threshold when segmenting, usually the better solution is to train a second network to recognize membranes and a third network to recognize the C-film hole, then compete the 3 networks against one another. While the vesicle may look somewhat like actin at low resolution, it should look more like a vesicle than actin, so the result of the competitive networks will be a more accurate segmentation of all 3 features.
Line 34: Line 58:
After getting an appropriate number of boxes, press “Write output” in the e2boxer window. We will begin the tutorial by training only a single network. You can then go back and follow the same process to train additional networks for other features, then finally at the end, use a program designed to compete these solutions against one another.
Line 36: Line 60:
 1 Select your boxes in the “Raw Data” window.
 2 Change the “Box Size” to the box size you want.
 3 Select the “Force Overwrite” box.
== Select Training Examples ==
In this stage we will be manually selecting a few 64x64 pixel tiles from several tomograms, some of which containing the feature of interest (FOI) you wish to train the neural network to identify. We will identify some 2-D tiles which contain the FOI and other tiles which do not contain the FOI at all. In the next step we will annotate the tiles containing
Line 40: Line 63:
 4 Leave the “Output Suffix” as _ptcls. Select '''Box training references'''. Press browse, and select the __preproc version of any one imported tomogram. Leave '''boxsize''' as -1, and press Launch.
Line 42: Line 65:
 5 In “Normalize Images”, select “None”. In a moment, three windows will appear: the tomogram view, the selected particles view, and a control-panel (labeled '''Options'''). Unlike the 2-D particle picking interface used in single particle analysis, which is almost identical, this program allows you to move through the various Z-slices of the tomogram. The examples you select will be 2-D, drawn from arbitrary slices throughout the 3-D tomogram. Additionally, this program permits selecting multiple classes of tiles from the same tomogram. For example, you may have one class of tiles containing the FOI and another class which does not contain the FOI. It is critical that you use a consistent naming convention for these different classes of particles, since we would like to have training references from multiple tomograms used for the network training.
Line 44: Line 67:
 6 Press “OK”.  * In the window named '''e2spt-boxer''', make sure your box size is set to 64. The '''Options''' window also contains the number 64, but this is the radius for the eraser, and its value does not matter.
 * In the options window you should see a list with something like ''particles_00'' in it. The tile selecting program can be used to select several different "classes" of particles from the same image/volume. Select this first element in the list and press '''Rename'''. Call this first set ''good_1''. If you go back and select tiles for a second FOI, you would call those tiles ''good_2''. Note that there is nothing magical about this name, but it needs to be self consistent among different tomograms.
 * Press the '''New''' button, and rename the new set ''bad_1''. We will use this to select tiles which do not contain the FOI at all
 * Click on ''good_1'' to make it the selected set, then:
 * In the window containing your tomogram, begin selecting boxes. You can move through the slices of the tomogram using the vertical slider in the lower right corner of the tomogram window, and zoom in and out with the scroll wheel. Select and reposition tiles with the left mouse button. Hold down shift when clicking to delete existing tiles. As you select tiles, they will appear in the '''(Particles)''' window.
 * Select roughly 10 - 20 tiles containing the FOI you wish to annotate. You will repeat this training process from scratch for different features, so for now, focus just on a single feature type. If the same feature appears different in different layers or different regions of the cell, be sure to cover the full range of visual differences in the representative regions you select. If the same particle appears completely different in different orientations, you may consider training two different networks for the two different appearances to improve accuracy. For example, if the particle is long and cylindrical, an end-on view will appear to be a small circle, and a side view will appear to be a rod. You will likely get superior results if you train separate networks for these very different views.
 * Highlight ''bad_1''
 * Select quite a few regions in the tomogram which do NOT contain the FOI, typically ~100. Include as many _different_ things which aren't the FOI as possible. For example, it doesn't do a lot of good to pick 100 different regions of similar looking tomogram background.
 * When you are done selecting ''good'' and ''bad'' references from a single tomogram, go back to the '''Options''' window, and put a checkmark next to only the ''good_1'' set. Then press '''Save'''. Accept the provided default name.
 * Repeat this process for ''bad_1'': put a check next to only ''bad_1'' and press save.
 * You can now close this program (closing the window containing the tomogram is the easiest way)
 * You have now created two new files in the ''particles/'' folder. You may repeat this process for other tomograms spanning the defocus range in your project. Each time you do this you will create a pair of new files in ''particles/''

{{attachment:gen_ptcls.png|Particle Output|width="300"}}

Note: The box size for training tiles MUST be 64x64. If the feature of interest does not fit in a 64x64 tile, you will need to create a more down-sampled copy of your tomogram for the recognition to work well. You can do this manually with e2proc3d.py. If you create a down-sampled version of a tomogram, use the exact same name and location as the original tomogram, but add a suffix to the name preceded by __ (double underscore). eg - ''tomograms/tomo_12345.hdf'' might become ''tomograms/tomo_12345__shrink2.hdf''

== Manually Annotate Samples ==

The next step is to manually identify the FOI
 
 * '''Segmentation -> Segment training references'''
  * '''particles''' should be set to the ''good_1'' file you just created in the ''particles/'' folder.
  * '''output''' -> leave blank
  * '''segment''' -> leave this checked
  * '''Launch'''
 * Two windows will appear, one containing images, and the other a control panel. You can navigate through the images using the up/down arrows, or using the '''N#''' slider . The control panel will open with the '''Draw''' tab selected.
 * Using the left mouse button, draw on each of the FOI to exactly cover the feature of interest as best you can. If necessary you can always can go back to the FOI selection window to check the surroundings of each feature to insure accurate segmentation
 * Generally speaking, the CNN can recognize the FOI if you can recognize the same feature in 2D patches by eye without ambiguity.
 * The key in this step is consistentcy. If you annotate a feature in a particular way you should annotate other features in the same way. If the pen is too large, reduce both '''Pen Size''' and '''Pen Size2'''.
 * Ideally you should be pixel-accurate in this process, but a small amount of 'slop' at the edges is likely inevitable, and should not pose a major problem. It IS critical however, that you correctly annotate all possible FOIs in every frame. If you miss a single FOI, it may significantly degrade the performance of the network. That is, if your annotation is consistently a little larger than the FOI, this will pose much less of a problem than completely missing a single FOI in the training tiles.
 * When you are finished, simply close the windows. The segmentation file will be saved automatically as "*_seg.hdf" with the same base file name as your ROIs.

{{attachment:segment_ptcls.png |Segment Particles|width="400"}}

== Build Training Set ==
This step is simply combining the files we have created through this step in a specific way to prepare for network training.

 * '''Segmentation -> Build training set'''
  * '''particles_raw''' -> select ''particles/*_good_1.hdf''
  * '''particles_label''' -> select ''particles/*_good_1_seg.hdf''
  * '''boxes_negative''' -> select ''particles/*_bad.hdf''
  * '''trainset_output''' -> leave blank
  * '''ncopy''' -> default of 10 should be fine. This controls the number of randomly oriented training replicas to create
  * '''Launch'''

This will create a new file with _trainset appended to the name. It takes only a few seconds to run.

== Train Neural Network ==

 * '''Segmentation -> Train the neural network'''
  * '''trainset''' -> the ''particles/*_trainset'' file you created in the previous step
  * '''device''' -> If you are running on a machine with CUDA properly configured with Tensorflow, enter ''gpu'' in this box. Otherwise you will have to use ''cpu''
  * defaults should be fine for all other options, '''Launch'''
  * If you use ''cpu'' this process may take a ''long'' time to run. On a GPU it should take 15-20 minutes.

 * Any errors or other issues will appear in the console window where you ran e2projectmanager.
  * If you get an error, for example:
   * ''terminate called after throwing an instance of Xbyak::Error''
   * ''zsh: killed e2tomoseg_convnet.py''
  * This generally means you have an incompatible version of TensorFlow installed in your environment. It may be incompatible with the version of CUDA or the driver installed on your computer, or there may be some other fundamental incompatibility. If you encounter this, you can try:
   * ''conda install tensorflow-gpu'' or ''conda install tensorflow-cpu'' as appropriate to update to the latest version.
   * If you need to go to an intermediate version, such as 1.9 (which seems to have pretty good forwards and backwards compatibility at the time this is being written, you can append ''=1.9'' to either of the previous commands
   * if you continue to encounter problems, the commands: ''nvidia-smi'' and ''nvcc --version'' can give you more information. You may then want to report the issue to the EMAN2 mailing list for help.
 * If the program runs correctly, you will see "Training..." on the console. The first time you train a network there may be a several minute delay, but eventually numbers will begin to appear in the console. If these numbers are extremely large, it implies something is wrong. Contact us for help if you have issues.
 * After the training is finished, it is recommended to look at the training result before proceeding.
  * Open the ''trainout_*.hdf'' using the EMAN2 file browser. You should see something like:

{{attachment:nnet_trainout.png |Training Results |width="400"}}

 * Zoom in or out a bit so there are 3*N images displayed in each row. For each image triplet, the first one is the tile from the tomogram, the second is the corresponding manual segmentation, and the third is the neural network output for the same FOI after training. The neural network is considered well trained if the third image matches the second image reasonably well. For the negative particles, both the second and the third images should be largely blank (the 3rd may contain some very weak features, this is ok).
 * If the training result looks completely wrong, go back and check your positive and negative training set first. Most significant errors are caused by training set errors, i.e. having some positive particles in the negative training set, or one of the positive training set is not correctly segmented.
 * If the training result looks suboptimal (the segmentation is not clear enough but not totally wrong), you may consider continuing the training for a few rounds. To do this, go back to the '''Train the neural network panel''', choose the previously trained network for the from_trained box and launch the training again. It is usually better to set a smaller '''learning rate''' in the continued training. Consider change value in the learnrate to ~0.001, or the displayed learning rate value at the last iteration of the previous training.
Once you are satisfied with the result, go to the next step to segment the whole tomogram.

== Apply to Tomograms ==

Finally, open '''Apply the neural network panel'''. Choose the tomogram you used to generate the boxes in the '''tomograms''' box, choose the saved neural network file (not the "trainout_" file, which is only used for visualization), and set the output filename. You can change the number of threads to use by adjusting the '''thread''' option. Keep in mind that using more threads will consume more memory as the tomogram slices are read in at the same time. For example, processing a 1k x 1k x 512 downsampled tomogram on 10 cores would use ~5 GB of RAM. Processing an unscaled 4k x 4k x 1k tomogram would increase RAM usage to ~24 GB. When this process finishes, you can open the output file in your favourite visualization software to view the segmentation.

{{attachment:segment_out.png |Segmentation Result |width="400"}}

To segment a different feature, just repeat the entire process for the each feature of interest. Make sure to use different file names (eg - _good2 and _bad2)! The trained network should generally work well on other tomograms using a similar specimen with similar microscope settings (clearly the A/pix value must be the same).

== Merging multiple annotation results ==

Merging the results from multiple networks on a single tomogram can help resolve ambiguities, or correct regions which were apparently misassigned. For example, in the microtubule annotation shown above, the carbon edge is falsely recognized as a microtubule. An extra neural network can be trained to specifically recognize the carbon edge and its result can be competed against the microtubule annotation. A multi-level mask is produced after merging multiple annotation result in which the integer values in a voxel identify the type of feature the voxel contains. To merge multiple annotation results, simply run in the terminal:

  mergetomoseg.py <annotation #1> <annotation #2> <...> --output <output mask file>

== Tips in selecting training samples ==
 * Annotate samples correctly, as a bad segmentation in the training set can damage the overall performance. In the microtubule case, if you annotate the spacing between microtubules, instead of the microtubules themselves (it is actually quite easy to make such mistake when annotating microtubule bundles), the neural network can behave unpredictably and sometimes just refuse to segment anything. Here is the training result on an incorrect and correct segmentation in one training set. Note the top one (22) is annotating the space between microtubules.
{{attachment:seg_compare.png |Good vs bad segments |width="200"}}
 * Make sure there are no positive samples in the negative training set. If your target feature is everywhere and it is hard to find negative regions, you can add additional positive samples which include various features other than the target feature (annotating only the target feature).
 * You can bin your tomogram differently to segment different features. Just import multiple copies of raw tomogram with different shrink factors, and unbin the segmentation using math.fft.resample processor. It is particularly useful when you have features with different lengthscales in one tomogram, and it is impossible to both fit the large features into a 64*64 box and still have the smaller features visible at the same scale.
 * In some cases, there is a significant missing wedge in the x-y plane slices (you can visualize this by clicking Amp button when looking at the slices in EMAN2). So the resolvability on x direction is different than that on y direction. It is important to provide features running in different directions in the training set, otherwise the neural net may only pick up features in one direction based on the Fourier patten. Also, you may want to check the stage of the microscope, since this may suggest the sample is not tilted exactly around the x axis.
 * It is also vital to cover various states of the target feature. For example, if you want to segment single layer membranes, you may want to have some cell membrane, some small vesicles, and some vesicles with darker density inside, so the neural network can grab the concept of membrane. Just imagine how you would teach someone with no biological knowledge about the features you are looking for. On the other hand, it is possible to ask the neural network to separate different types of those same features. In the membrane example, it is possible to train the neural network to segment vesicles from cell membranes based on the curvature, or recognize dense vesicles based on the difference of intensity on both side of the membrane, given carefully picked training set.
Line 47: Line 165:
== Manually Annotate Samples ==
== Select Negative Samples ==
== Build Training Set ==
== Train Neural Network ==
== Apply to Tomograms ==
Line 53: Line 166:

Tomogram Annotation/Segmentation

Availability: EMAN2.2+, or daily build after 2016-10 This is the tutorial for a Convolutional neural network(CNN) based semi-automated cellular tomogram annotation protocol.

If you are using this in a publication, please cite:

A stable (thus not up-to-date) online version of the tutorial can also be found at Protocol exchange: https://www.nature.com/protocolexchange/protocols/6171

Updates:

  • 10/2018: Now it uses Tensorflow backend which should be included during installation. Also particle boxing interface and directory structure are updated to incorporate into the new e2tomo pipeline.

  • 01/2018: Slides of a recent tutorial in Rutgers boot camp can be found here. tomoseg_tutorial_rutgers.pdf

  • 10/2017: We have switched to gpuarray backend for CUDA9 support. It should still work with CUDA7.5+ and the old cuda backend in Theano.

Installation

CNN based tomogram annotation makes use of the TensorFlow deep learning toolkit. At the time of this writing, tensorflow effectively requires CUDA, the compute framework on NVidia GPUs. While future support for OpenCL (an open alternative to CUDA) seems likely, and while it is possible to use TensorFlow on a CPU, at present we cannot recommend running this tool on Mac/Windows. You may do so, but training the networks may be VERY slow.

To make use of accelerated TensorFlow, your machine must be properly configured for CUDA. This is outside the installation of EMAN2, and requires specific drivers, etc. on your machine. Some extra information on how to use CUDA and GPU can be found at http://blake.bcm.edu/emanwiki/EMAN2/Install/BinaryInstallAnaconda#Using_the_GPU

Setup workspace

If you used EMAN2 for your tomogram alignment/reconstruction (EMAN2/e2TomoSmall), simply 'cd' to the project folder and launch e2projectmanager.py. If you will be using tomograms produced by other software:

Make an empty folder and cd into that folder at the command-line. Run e2projectmanager.py.

Select the Tomo Workflow Mode.

Import or Preprocess

If your tomograms were produced using the EMAN2 workflow:

  • You should preprocess your tomograms to prepare them for segmentation.
    • Select Segmentation -> Preprocess Tomograms

    • Press Browse, Sel All then OK

    • All of the other options should normally be fine, and you can press Launch

If your tomograms were produced by other software, and you wish to do the segmentation in EMAN2:

  • If your original tomograms are in multiple folders, you can repeat the next step.
  • Select Raw Data -> Import Tomograms

    • Use Browse to select all of the tomograms you wish to segment, and press OK

    • EMAN2 uses the convention that tomograms have "dark" or negative contrast, and individual particles and 3-D averages will have "white" or positive contrast. If your tomograms have positive contrast (features look bright when displayed) check the Invert box.

    • If your tomograms are larger than ~1k x 1k, you should consider downsampling, or Shrinking them for segmentation. Normally 1k x 1k is more than sufficient for accurate segmentation. ie - if your original tomograms are 4k x 4k, enter 4 in the shrink box.

    • In deciding how much to shrink the original tomograms, it is only important that the features you wish to recognize are still recognizable by eye. If shrinking produces a tomogram where a feature you wish to identify is only 1-2 pixels in size, then you are shinking too much.
    • Launch

  • Once imported, you should preprocess your tomograms to prepare them for segmentation.
    • Select Segmentation -> Preprocess Tomograms

    • Press Browse, Sel All then OK

    • All of the other options should normally be fine, and you can press Launch

Import Tomogram

  • Note: There may be cases where it is impossible to pick a single scale which works well to annotate features at different scales. For example, if you are trying to annotate icosahedral viruses (large scale) and actin filaments (require small scale to identify) in the same tomogram, you may need to import the tomogram twice, with 2 different "shrink" factors, segment each feature at the appropriate scale, then rescale/merge the final segmentation maps. This is a topic for advanced users, and we encourage you to learn the basic pipeline with a single feature before attempting this.

Overview

The network training process may seem complicated at first, but in reality it is extremely simple. You will be training a neural network to look at your tomogram slice by slice, and for each pixel it will decide whether it does or does not look like part of the feature you are trying to identify. This feature is termed the Feature of Interest (FOI). To accomplish this task, we must provide a few manual annotations to teach the network what it should be looking for. We must also provide some regions from the tomogram which do not contain the FOI as negative examples. That is, you are giving the network several examples of: "I am looking for THIS", and several more examples of: "It does NOT look like THIS".

For the network to perform this task well, it is critical that you be absolutely certain about the regions you select for training. The largest error people make when training the network is to include regions containing ambiguous density. If you select a tile containing a clear feature you wish to include, but the tile also contains some density you are unsure of, then you will be in a difficult situation. If you mark the ambiguous region as a particle, and it actually is something else, the trained network will not be very selective for the FOI. If you do not mark the ambiguous density, and it turns out to actually be the FOI, then you are trying to train the network to distinguish between the object you said was the ROI and the object you said was not (but was), which may cause a training failure, or a network which misses many FOIs. The correct solution is not to include any tiles containing ambiguous density! Only include training tiles you are certain that you can annotate accurately in 2-D.

There will be other cases where it may be difficult to distinguish between two features. For example, at low resolution, the strong line of an actin filament, may look somewhat like the edge of a membrane on an organelle. If you train only a single network, and it is trained to recognize actin, then you apply it to a cellular tomogram, you will likely find many other features, such as membrane edges, and possibly the edge of the holy carbon film, get mis-identified as actin. While you may be able to more carefully train the actin network, or alter the threshold when segmenting, usually the better solution is to train a second network to recognize membranes and a third network to recognize the C-film hole, then compete the 3 networks against one another. While the vesicle may look somewhat like actin at low resolution, it should look more like a vesicle than actin, so the result of the competitive networks will be a more accurate segmentation of all 3 features.

We will begin the tutorial by training only a single network. You can then go back and follow the same process to train additional networks for other features, then finally at the end, use a program designed to compete these solutions against one another.

Select Training Examples

In this stage we will be manually selecting a few 64x64 pixel tiles from several tomograms, some of which containing the feature of interest (FOI) you wish to train the neural network to identify. We will identify some 2-D tiles which contain the FOI and other tiles which do not contain the FOI at all. In the next step we will annotate the tiles containing

Select Box training references. Press browse, and select the preproc version of any one imported tomogram. Leave boxsize as -1, and press Launch.

In a moment, three windows will appear: the tomogram view, the selected particles view, and a control-panel (labeled Options). Unlike the 2-D particle picking interface used in single particle analysis, which is almost identical, this program allows you to move through the various Z-slices of the tomogram. The examples you select will be 2-D, drawn from arbitrary slices throughout the 3-D tomogram. Additionally, this program permits selecting multiple classes of tiles from the same tomogram. For example, you may have one class of tiles containing the FOI and another class which does not contain the FOI. It is critical that you use a consistent naming convention for these different classes of particles, since we would like to have training references from multiple tomograms used for the network training.

  • In the window named e2spt-boxer, make sure your box size is set to 64. The Options window also contains the number 64, but this is the radius for the eraser, and its value does not matter.

  • In the options window you should see a list with something like particles_00 in it. The tile selecting program can be used to select several different "classes" of particles from the same image/volume. Select this first element in the list and press Rename. Call this first set good_1. If you go back and select tiles for a second FOI, you would call those tiles good_2. Note that there is nothing magical about this name, but it needs to be self consistent among different tomograms.

  • Press the New button, and rename the new set bad_1. We will use this to select tiles which do not contain the FOI at all

  • Click on good_1 to make it the selected set, then:

  • In the window containing your tomogram, begin selecting boxes. You can move through the slices of the tomogram using the vertical slider in the lower right corner of the tomogram window, and zoom in and out with the scroll wheel. Select and reposition tiles with the left mouse button. Hold down shift when clicking to delete existing tiles. As you select tiles, they will appear in the (Particles) window.

  • Select roughly 10 - 20 tiles containing the FOI you wish to annotate. You will repeat this training process from scratch for different features, so for now, focus just on a single feature type. If the same feature appears different in different layers or different regions of the cell, be sure to cover the full range of visual differences in the representative regions you select. If the same particle appears completely different in different orientations, you may consider training two different networks for the two different appearances to improve accuracy. For example, if the particle is long and cylindrical, an end-on view will appear to be a small circle, and a side view will appear to be a rod. You will likely get superior results if you train separate networks for these very different views.
  • Highlight bad_1

  • Select quite a few regions in the tomogram which do NOT contain the FOI, typically ~100. Include as many _different_ things which aren't the FOI as possible. For example, it doesn't do a lot of good to pick 100 different regions of similar looking tomogram background.
  • When you are done selecting good and bad references from a single tomogram, go back to the Options window, and put a checkmark next to only the good_1 set. Then press Save. Accept the provided default name.

  • Repeat this process for bad_1: put a check next to only bad_1 and press save.

  • You can now close this program (closing the window containing the tomogram is the easiest way)
  • You have now created two new files in the particles/ folder. You may repeat this process for other tomograms spanning the defocus range in your project. Each time you do this you will create a pair of new files in particles/

Particle Output

Note: The box size for training tiles MUST be 64x64. If the feature of interest does not fit in a 64x64 tile, you will need to create a more down-sampled copy of your tomogram for the recognition to work well. You can do this manually with e2proc3d.py. If you create a down-sampled version of a tomogram, use the exact same name and location as the original tomogram, but add a suffix to the name preceded by (double underscore). eg - tomograms/tomo_12345.hdf might become tomograms/tomo_12345shrink2.hdf

Manually Annotate Samples

The next step is to manually identify the FOI

  • Segmentation -> Segment training references

    • particles should be set to the good_1 file you just created in the particles/ folder.

    • output -> leave blank

    • segment -> leave this checked

    • Launch

  • Two windows will appear, one containing images, and the other a control panel. You can navigate through the images using the up/down arrows, or using the N# slider . The control panel will open with the Draw tab selected.

  • Using the left mouse button, draw on each of the FOI to exactly cover the feature of interest as best you can. If necessary you can always can go back to the FOI selection window to check the surroundings of each feature to insure accurate segmentation
  • Generally speaking, the CNN can recognize the FOI if you can recognize the same feature in 2D patches by eye without ambiguity.
  • The key in this step is consistentcy. If you annotate a feature in a particular way you should annotate other features in the same way. If the pen is too large, reduce both Pen Size and Pen Size2.

  • Ideally you should be pixel-accurate in this process, but a small amount of 'slop' at the edges is likely inevitable, and should not pose a major problem. It IS critical however, that you correctly annotate all possible FOIs in every frame. If you miss a single FOI, it may significantly degrade the performance of the network. That is, if your annotation is consistently a little larger than the FOI, this will pose much less of a problem than completely missing a single FOI in the training tiles.
  • When you are finished, simply close the windows. The segmentation file will be saved automatically as "*_seg.hdf" with the same base file name as your ROIs.

Segment Particles

Build Training Set

This step is simply combining the files we have created through this step in a specific way to prepare for network training.

  • Segmentation -> Build training set

    • particles_raw -> select particles/*_good_1.hdf

    • particles_label -> select particles/*_good_1_seg.hdf

    • boxes_negative -> select particles/*_bad.hdf

    • trainset_output -> leave blank

    • ncopy -> default of 10 should be fine. This controls the number of randomly oriented training replicas to create

    • Launch

This will create a new file with _trainset appended to the name. It takes only a few seconds to run.

Train Neural Network

  • Segmentation -> Train the neural network

    • trainset -> the particles/*_trainset file you created in the previous step

    • device -> If you are running on a machine with CUDA properly configured with Tensorflow, enter gpu in this box. Otherwise you will have to use cpu

    • defaults should be fine for all other options, Launch

    • If you use cpu this process may take a long time to run. On a GPU it should take 15-20 minutes.

  • Any errors or other issues will appear in the console window where you ran e2projectmanager.
    • If you get an error, for example:
      • terminate called after throwing an instance of Xbyak::Error

      • zsh: killed e2tomoseg_convnet.py

    • This generally means you have an incompatible version of TensorFlow installed in your environment. It may be incompatible with the version of CUDA or the driver installed on your computer, or there may be some other fundamental incompatibility. If you encounter this, you can try:

      • conda install tensorflow-gpu or conda install tensorflow-cpu as appropriate to update to the latest version.

      • If you need to go to an intermediate version, such as 1.9 (which seems to have pretty good forwards and backwards compatibility at the time this is being written, you can append =1.9 to either of the previous commands

      • if you continue to encounter problems, the commands: nvidia-smi and nvcc --version can give you more information. You may then want to report the issue to the EMAN2 mailing list for help.

  • If the program runs correctly, you will see "Training..." on the console. The first time you train a network there may be a several minute delay, but eventually numbers will begin to appear in the console. If these numbers are extremely large, it implies something is wrong. Contact us for help if you have issues.
  • After the training is finished, it is recommended to look at the training result before proceeding.
    • Open the trainout_*.hdf using the EMAN2 file browser. You should see something like:

Training Results

  • Zoom in or out a bit so there are 3*N images displayed in each row. For each image triplet, the first one is the tile from the tomogram, the second is the corresponding manual segmentation, and the third is the neural network output for the same FOI after training. The neural network is considered well trained if the third image matches the second image reasonably well. For the negative particles, both the second and the third images should be largely blank (the 3rd may contain some very weak features, this is ok).
  • If the training result looks completely wrong, go back and check your positive and negative training set first. Most significant errors are caused by training set errors, i.e. having some positive particles in the negative training set, or one of the positive training set is not correctly segmented.
  • If the training result looks suboptimal (the segmentation is not clear enough but not totally wrong), you may consider continuing the training for a few rounds. To do this, go back to the Train the neural network panel, choose the previously trained network for the from_trained box and launch the training again. It is usually better to set a smaller learning rate in the continued training. Consider change value in the learnrate to ~0.001, or the displayed learning rate value at the last iteration of the previous training.

Once you are satisfied with the result, go to the next step to segment the whole tomogram.

Apply to Tomograms

Finally, open Apply the neural network panel. Choose the tomogram you used to generate the boxes in the tomograms box, choose the saved neural network file (not the "trainout_" file, which is only used for visualization), and set the output filename. You can change the number of threads to use by adjusting the thread option. Keep in mind that using more threads will consume more memory as the tomogram slices are read in at the same time. For example, processing a 1k x 1k x 512 downsampled tomogram on 10 cores would use ~5 GB of RAM. Processing an unscaled 4k x 4k x 1k tomogram would increase RAM usage to ~24 GB. When this process finishes, you can open the output file in your favourite visualization software to view the segmentation.

Segmentation Result

To segment a different feature, just repeat the entire process for the each feature of interest. Make sure to use different file names (eg - _good2 and _bad2)! The trained network should generally work well on other tomograms using a similar specimen with similar microscope settings (clearly the A/pix value must be the same).

Merging multiple annotation results

Merging the results from multiple networks on a single tomogram can help resolve ambiguities, or correct regions which were apparently misassigned. For example, in the microtubule annotation shown above, the carbon edge is falsely recognized as a microtubule. An extra neural network can be trained to specifically recognize the carbon edge and its result can be competed against the microtubule annotation. A multi-level mask is produced after merging multiple annotation result in which the integer values in a voxel identify the type of feature the voxel contains. To merge multiple annotation results, simply run in the terminal:

  • mergetomoseg.py <annotation #1> <annotation #2> <...> --output <output mask file>

Tips in selecting training samples

  • Annotate samples correctly, as a bad segmentation in the training set can damage the overall performance. In the microtubule case, if you annotate the spacing between microtubules, instead of the microtubules themselves (it is actually quite easy to make such mistake when annotating microtubule bundles), the neural network can behave unpredictably and sometimes just refuse to segment anything. Here is the training result on an incorrect and correct segmentation in one training set. Note the top one (22) is annotating the space between microtubules.

Good vs bad segments

  • Make sure there are no positive samples in the negative training set. If your target feature is everywhere and it is hard to find negative regions, you can add additional positive samples which include various features other than the target feature (annotating only the target feature).
  • You can bin your tomogram differently to segment different features. Just import multiple copies of raw tomogram with different shrink factors, and unbin the segmentation using math.fft.resample processor. It is particularly useful when you have features with different lengthscales in one tomogram, and it is impossible to both fit the large features into a 64*64 box and still have the smaller features visible at the same scale.
  • In some cases, there is a significant missing wedge in the x-y plane slices (you can visualize this by clicking Amp button when looking at the slices in EMAN2). So the resolvability on x direction is different than that on y direction. It is important to provide features running in different directions in the training set, otherwise the neural net may only pick up features in one direction based on the Fourier patten. Also, you may want to check the stage of the microscope, since this may suggest the sample is not tilted exactly around the x axis.
  • It is also vital to cover various states of the target feature. For example, if you want to segment single layer membranes, you may want to have some cell membrane, some small vesicles, and some vesicles with darker density inside, so the neural network can grab the concept of membrane. Just imagine how you would teach someone with no biological knowledge about the features you are looking for. On the other hand, it is possible to ask the neural network to separate different types of those same features. In the membrane example, it is possible to train the neural network to segment vesicles from cell membranes based on the curvature, or recognize dense vesicles based on the difference of intensity on both side of the membrane, given carefully picked training set.

Acknowledgement

Darius Jonasch, the first user of the tomogram segmentation protocol, provided many useful advices to make the workflow user-friendly. He also wrote a tutorial of the earlier version of the protocol, on which this tutorial is based.

EMAN2/Programs/tomoseg (last edited 2019-12-03 13:43:43 by SteveLudtke)