Differences between revisions 1 and 23 (spanning 22 versions)
Revision 1 as of 2010-03-29 10:10:48
Size: 35
Editor: root
Comment:
Revision 23 as of 2010-04-21 05:40:17
Size: 6859
Editor: root
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
Coming soon emen2client.py serves several functions.

1. Provides a simple command-line interface for bulk upload and download from the database.

2. Contains a number of classes that are useful in building EMEN2 client applications. It is used by other programs, such as EMDash. See [[EMEN2/API]] for details.

3. Provides a mechanism for uploading EMAN/EMAN2 processing data, including CTF parameters and box coordinates.

The program is a relatively recent development, and replaces a number of older programs (uploadclient, downloadclient, libemen2client, etc.) that performed similar functions in the past.

<<TableOfContents>>

= Download emen2client.py =

Use the link below to download emen2client.py:

* [[http://blake.bcm.edu/emanwiki/EMEN2/emen2client.py?action=AttachFile&do=get&target=emen2client.py|emen2client.py]]

This version is updated frequently (currently about once every two weeks), and this page is "primary" download location. I try to thoroughly test each version before publishing it here. If you find any problems, please contact Ian Rees, ian.rees@bcm.edu, and include the command you ran, how it failed, and the complete program output.

= Usage =

{{{
./emen2client.py <action>
Actions available: upload, download, sync
For detailed help: ./emen2client.py <action> --help
}}}

When run from the command line, emen2client.py offers several utility "actions" that can be taken. You invoke an action by specifying it as the first command line argument.

The "upload" action is used for uploading data (ccd frames, volumes, etc.) to the database. Similarly, "download" transfers files from the database to your local machine. These two actions are documented below.

The "sync" action is new, and provides some support for uploading EMAN/EMAN2 image processing data to database records, e.g. CTF parameters. This is documented on a separate page: [[EMEN2/EMAN2_Integration|EMAN2 Integration]]


== Download Usage ==

The "download" action is used to batch download files from the database. A Record ID is required; in most cases, this will be the Record ID of a grid_imaging session containing CCD frames. There are a number of options, detailed below, although in most cases the default options are fine. You can optionally specify a filename or pattern to filter the list of files that will be downloaded.
{{{
Usage: emen2client.py download [options] <recid> [filename-pattern]

Options:
  -h, --help show this help message and exit
  -U USERNAME, --username=USERNAME
                        Username
  -P PASSWORD, --password=PASSWORD
                        Password (Note: specifying passwords in shell commands
                        is not secure)
  -H HOST, --host=HOST Host endpoint URI
  -C CTXID, --ctxid=CTXID
                        Valid Context ID
  --recurse=RECURSE Recursion level
  -o, --overwrite Overwrite existing files (default is to skip)
  -r, --rename If a file already exists, save with format
                        'duplicate.recid:original_filename.dm3'
  -s, --sidecar Include sidecar file with EMEN2 metadata in JSON
                        format
  --gzip Decompress gzip'd files. Requires gzip in path
                        (default)
  --no-gzip Do not decompress gzip'd files.
}}}

Example:

{{{
[irees@mbp ~/data]$ emen2client.py download 123456

Username: ianrees
Password:

1 of 1: 123456
 Checking for items to download
 Found 92 items in 93 records

 1 of 92: 2009110400033
  Downloading frame1.dm3.gz -> frame1.dm3...

 2 of 92: 2009110400032
  Downloading frame2.dm3.gz -> frame2.dm3...

 3 of 92: 2009110400031
  Downloading frame3.dm3.gz -> frame3.dm3...

.... snip ....
 
}}}

== Upload Usage ==

The "upload" action will upload files into the database. A record type, record ID, and list of files must be specified. If you are uploading CCD frames, use "ccd", if you are uploading SerielEM tomography stacks, use "stack." Other types are also available, e.g. "volume" to upload a number of density maps.


{{{
Usage: emen2client.py upload [options] <record type> <recid> <files to upload>

 Record type can be any valid database protocol.
 Some record types have special, application-specific handlers, e.g.:
 
  ccd CCD Frames
  scan Scanned micrographs
  stack Tomograms

 Other values, e.g. "volume", will create child records of that type, with 1 file per child record.

 Alternatively, you can use "none" for record type and the files will be attached directly to the specified record ID.
 
  

Options:
  -h, --help show this help message and exit
  -U USERNAME, --username=USERNAME
                        Username
  -P PASSWORD, --password=PASSWORD
                        Password (Note: specifying passwords in shell commands
                        is not secure)
  -H HOST, --host=HOST Host endpoint URI
  -C CTXID, --ctxid=CTXID
                        Valid Context ID
  -q, --noninteractive Do not prompt for parameter values
  -s, --sidecar Write sidecar file after upload
  -f, --force Force re-upload even if a sidecar is found
  --metafile Attempt to read JAMES/JADAS metadata files (default)
  --no-metafile Ignore metadata files
}}}

Example:

{{{
[irees@mbp ~/data]$ emen2client.py upload ccd 123456 *.dm3

Username: ianrees
Password:

3 Files to upload:
 1.dm3
 2.dm3
 3.dm3

----- tem_magnification_set -----
 Description: Magnification set on microscope
 Units: K

 Common values:
  0) 60.0
  1) 50.0
  2) 40.0
  3) 20.0
  4) 80.0
  5) 100.0
  6) 15.0
  7) 30.0
  8) 25.0

  10) None or N/A
  11) Enter a different not listed above

 Selection (0-11): 0

----- ctf_defocus_set -----
 Description: Intended defocus while imaging, underfocus positive
 Units: um

 Common values:
  0) 1.0
  1) -2.0
  2) 2.0
  3) 1.5
  4) 3.0
  5) 2.5
  6) -3.0
  7) -1.5
  8) -2.5
  9) 0.8

  10) None or N/A
  11) Enter a different not listed above

 Selection (0-11): 0


... snip: additional prompts ommitted for clarity ...


1 of 3: 1.dm3
 Preparing for upload
 Checking and committing records

 1.dm3
  Done. Uploaded 1.dm3 to record 417171 @ 15210.92 kb/sec

2 of 3: 2.dm3
 Preparing for upload
 Checking and committing records

 2.dm3
  Done. Uploaded 2.dm3 to record 417172 @ 15080.45 kb/sec

3 of 3: 3.dm3
 Preparing for upload
 Checking and committing records

 3.dm3
  Done. Uploaded 3.dm3 to record 417173 @ 14253.12 kb/sec
}}}


== EMAN Synchronization ==

emen2client.py provides growing support for uploading processed image data back into the database. See [[EMEN2/EMAN2_Integration|EMAN2 Integration]] for details.

emen2client.py

emen2client.py serves several functions.

1. Provides a simple command-line interface for bulk upload and download from the database.

2. Contains a number of classes that are useful in building EMEN2 client applications. It is used by other programs, such as EMDash. See EMEN2/API for details.

3. Provides a mechanism for uploading EMAN/EMAN2 processing data, including CTF parameters and box coordinates.

The program is a relatively recent development, and replaces a number of older programs (uploadclient, downloadclient, libemen2client, etc.) that performed similar functions in the past.

Download emen2client.py

Use the link below to download emen2client.py:

* emen2client.py

This version is updated frequently (currently about once every two weeks), and this page is "primary" download location. I try to thoroughly test each version before publishing it here. If you find any problems, please contact Ian Rees, ian.rees@bcm.edu, and include the command you ran, how it failed, and the complete program output.

Usage

./emen2client.py <action>
Actions available: upload, download, sync
For detailed help: ./emen2client.py <action> --help

When run from the command line, emen2client.py offers several utility "actions" that can be taken. You invoke an action by specifying it as the first command line argument.

The "upload" action is used for uploading data (ccd frames, volumes, etc.) to the database. Similarly, "download" transfers files from the database to your local machine. These two actions are documented below.

The "sync" action is new, and provides some support for uploading EMAN/EMAN2 image processing data to database records, e.g. CTF parameters. This is documented on a separate page: EMAN2 Integration

Download Usage

The "download" action is used to batch download files from the database. A Record ID is required; in most cases, this will be the Record ID of a grid_imaging session containing CCD frames. There are a number of options, detailed below, although in most cases the default options are fine. You can optionally specify a filename or pattern to filter the list of files that will be downloaded.

Usage: emen2client.py download [options] <recid> [filename-pattern]

Options:
  -h, --help            show this help message and exit
  -U USERNAME, --username=USERNAME
                        Username
  -P PASSWORD, --password=PASSWORD
                        Password (Note: specifying passwords in shell commands
                        is not secure)
  -H HOST, --host=HOST  Host endpoint URI
  -C CTXID, --ctxid=CTXID
                        Valid Context ID
  --recurse=RECURSE     Recursion level
  -o, --overwrite       Overwrite existing files (default is to skip)
  -r, --rename          If a file already exists, save with format
                        'duplicate.recid:original_filename.dm3'
  -s, --sidecar         Include sidecar file with EMEN2 metadata in JSON
                        format
  --gzip                Decompress gzip'd files. Requires gzip in path
                        (default)
  --no-gzip             Do not decompress gzip'd files.

Example:

[irees@mbp ~/data]$ emen2client.py download 123456

Username: ianrees
Password: 

1 of 1: 123456
        Checking for items to download
        Found 92 items in 93 records

        1 of 92: 2009110400033
                Downloading frame1.dm3.gz -> frame1.dm3...

        2 of 92: 2009110400032
                Downloading frame2.dm3.gz -> frame2.dm3...

        3 of 92: 2009110400031
                Downloading frame3.dm3.gz -> frame3.dm3...

.... snip ....

Upload Usage

The "upload" action will upload files into the database. A record type, record ID, and list of files must be specified. If you are uploading CCD frames, use "ccd", if you are uploading SerielEM tomography stacks, use "stack." Other types are also available, e.g. "volume" to upload a number of density maps.

Usage: emen2client.py upload [options] <record type> <recid> <files to upload>

        Record type can be any valid database protocol.
        Some record types have special, application-specific handlers, e.g.:
        
                ccd                     CCD Frames
                scan            Scanned micrographs
                stack           Tomograms

        Other values, e.g. "volume", will create child records of that type, with 1 file per child record.

        Alternatively, you can use "none" for record type and the files will be attached directly to the specified record ID.
        
                

Options:
  -h, --help            show this help message and exit
  -U USERNAME, --username=USERNAME
                        Username
  -P PASSWORD, --password=PASSWORD
                        Password (Note: specifying passwords in shell commands
                        is not secure)
  -H HOST, --host=HOST  Host endpoint URI
  -C CTXID, --ctxid=CTXID
                        Valid Context ID
  -q, --noninteractive  Do not prompt for parameter values
  -s, --sidecar         Write sidecar file after upload
  -f, --force           Force re-upload even if a sidecar is found
  --metafile            Attempt to read JAMES/JADAS metadata files (default)
  --no-metafile         Ignore metadata files

Example:

[irees@mbp ~/data]$ emen2client.py upload ccd 123456 *.dm3

Username: ianrees
Password: 

3 Files to upload:
        1.dm3
        2.dm3
        3.dm3

----- tem_magnification_set -----
        Description: Magnification set on microscope
        Units: K

        Common values:
                0) 60.0
                1) 50.0
                2) 40.0
                3) 20.0
                4) 80.0
                5) 100.0
                6) 15.0
                7) 30.0
                8) 25.0

                10) None or N/A
                11) Enter a different not listed above

        Selection (0-11): 0

----- ctf_defocus_set -----
        Description: Intended defocus while imaging, underfocus positive
        Units: um

        Common values:
                0) 1.0
                1) -2.0
                2) 2.0
                3) 1.5
                4) 3.0
                5) 2.5
                6) -3.0
                7) -1.5
                8) -2.5
                9) 0.8

                10) None or N/A
                11) Enter a different not listed above

        Selection (0-11): 0


... snip: additional prompts ommitted for clarity ...


1 of 3: 1.dm3
        Preparing for upload
        Checking and committing records

        1.dm3
                Done. Uploaded 1.dm3 to record 417171 @ 15210.92 kb/sec

2 of 3: 2.dm3
        Preparing for upload
        Checking and committing records

        2.dm3
                Done. Uploaded 2.dm3 to record 417172 @ 15080.45 kb/sec

3 of 3: 3.dm3
        Preparing for upload
        Checking and committing records

        3.dm3
                Done. Uploaded 3.dm3 to record 417173 @ 14253.12 kb/sec

EMAN Synchronization

emen2client.py provides growing support for uploading processed image data back into the database. See EMAN2 Integration for details.

EMEN2/emen2client.py (last edited 2011-05-20 15:58:47 by IanRees)