BRT flow
Batchruntomo pipeline overview:
Takes an
input_dir
containing 1 or more .mrc or .st filesIMOD
batchruntomo
(BRT) https://bio3d.colorado.edu/imod/doc/man/batchruntomo.html is run on each file in dir.BRT has a large number of parameters, which are provided using an .adoc parameter file.
Some of these parameters are provided to the pipeline on a per run basis, the remaining of these parameters are defaulted using the templated adoc file.
There are two templates,
../templates/plastic_brt.adoc
and../templates/cryo_brt.adoc
(Note, currently these are identical, we expect this to change.) The template chosen is defined by parameter adoc_template.The run time parameters are interpolated into the template, and this file is then run with BRT.
BRT produces two output files that we care about: _ali.mrc and _rec.mrc
- Using the ali (alignment) file we generate a tilt movie:
look up dimensionality of alignment file, and generate n sections in
gen_ali_x()
we assemble these sections into a single aligned mrc file, in
gen_ali_asmbl()
we convert this into a stack of jpegs, in
gen_mrc2tiff()
we compile these jpegs into a tilt movie in
gen_tilt_movie()
ADDITIONALLY:
we take the mid point of the stack jpeg to use as the display thumbnail.
we clean up intermediate files now.
- Using the _rec (reconstructed) file we generate a reconstructed movie (in a similar fashion to the above)
look up dimensionality of the reconstructed (_rec) file,
create a stack of averaged mrc files in
gen_clip_avgs()
create single mrc using the above averaged stack, in
consolidate_ave_mrcs()
convert this mrc to a stack of jpegs, in
gen_ave_jpgs_from_ave_mrc()
compile jpegs into reconstructed movie, in
gen_recon_movie()
clean up after ourselves in
cleanup_files()
Use average (created above) reconstructed mrc file to create input for volslicer.k in
gen_ave_8_vol()
- need to produce pyramid files with reconstructed mrc.
convert mrc file to Zarr, in gen_zarr()
Now we need to copy the outputs to the right place, and tell the API where they are. We use JSON to talk to the API.
build a json datastructure, containing the locations of the inputs we key on “primaryFilePath”, and we append every output that’s generated for this input into the “assets” json key.
Finally, we
POST
the JSON datastructure to the API, and cleanup temp dirs.
- em_workflows.brt.flow.gen_dimension_command(file_path: FilePath, ali_or_rec: str) str
- looks up the z dimension of an mrc file.ali_or_rec is nasty, str to denote whether you’re using the _ali file or the _rec file.
- Todo:
this is duplicate, see utils.lookup_dims()
- em_workflows.brt.flow.gen_ali_x(file_path: FilePath, z_dim) None
chops an mrc input into its constituent Z sections.
eg if an mrc input has a z_dim of 10, 10 sections will be generated.
the i-i syntax is awkward, and may not be required. Eg possibly replace i-i with i.
eg:
newstack -secs {i}-{i} path/BASENAME_ali*.mrc WORKDIR/hedwig/BASENAME_ali{i}.mrc
- em_workflows.brt.flow.gen_ali_asmbl(file_path: FilePath) None
Use IMOD
newstack
to assemble, eg:newstack -float 3 {BASENAME}_ali*.mrc ali_{BASENAME}.mrc
- em_workflows.brt.flow.gen_mrc2tiff(file_path: FilePath) None
This generates a lot of jpegs (-j) which will be compiled into a movie. (That is, the -jpeg switch is set to produce jpegs) eg:
mrc2tif -j -C 0,255 ali_BASENAME.mrc BASENAME_ali
- em_workflows.brt.flow.gen_thumbs(file_path: FilePath, z_dim) dict
Use GraphicsMagick to create thumbnail images, eg:
gm convert -size 300x300 BASENAME_ali.{MIDDLE_I}.jpg -resize 300x300 -sharpen 2 -quality 70 keyimg_BASENAME_s.jpg
- em_workflows.brt.flow.gen_tilt_movie(file_path: FilePath) dict
generates the tilt movie, eg:
ffmpeg -f image2 -framerate 4 -i ${BASENAME}_ali.%03d.jpg -vcodec libx264 -pix_fmt yuv420p -s 1024,1024 tiltMov_${BASENAME}.mp4
- em_workflows.brt.flow.gen_recon_movie(file_path: FilePath) dict
compiles a stack of jpgs into a movie. eg:
ffmpeg -f image2 -framerate 8 -i WORKDIR/hedwig/BASENAME_mp4.%04d.jpg -vcodec libx264 -pix_fmt yuv420p -s 1024,1024 WORKDIR/hedwig/keyMov_BASENAME.mp4
- em_workflows.brt.flow.gen_clip_avgs(file_path: FilePath, z_dim: str) None
give _rec mrc file, generate a stack of mrcs, averaged to assist viewing.
produces base_ave001.mrc etc, base_ave002.mrc etc,
inputs for newstack (for recon movie) and binvol (for volslicer)
eg:
for i in range(2, dimensions.z-2): IZMIN = i-2 IZMAX = i+2 clip avg -2d -iz IZMIN-IZMAX -m 1 BASENAME_rec.mrc BASENAME_ave${i}.mrc
- em_workflows.brt.flow.consolidate_ave_mrcs(file_path: FilePath) dict
consumes base_ave001.mrc etc, base_ave002.mrc etc,
creates ave_base.mrc the (averagedVolume asset)
eg:
newstack -float 3 BASENAME_ave* ave_BASENAME.mrc
- em_workflows.brt.flow.gen_ave_8_vol(file_path: FilePath)
creates volume asset, for volslicer, eg:
binvol -binning 2 WORKDIR/hedwig/ave_BASENAME.mrc WORKDIR/avebin8_BASENAME.mrc
- em_workflows.brt.flow.gen_ave_jpgs_from_ave_mrc(file_path: FilePath)
generates a load of jpgs from the ave_base.mrc with the format {base}_mp4.123.jpg OR {base}_mp4.1234.jpg depending on size of stack.
These jpgs can later be compiled into a movie. eg:
mrc2tif -j -C 100,255 WORKDIR/hedwig/ave_BASNAME.mrc hedwig/BASENAME_mp4
- em_workflows.brt.flow.cleanup_files(file_path: FilePath, pattern=str)
Given a
FilePath
and unix filepattern
, iterate through directory removing all files that match the pattern
- em_workflows.brt.flow.gen_dimension_command(fp_in: Path) str
- looks up the z dimension of an mrc file.ali_or_rec is nasty, str to denote whether you’re using the _ali file or the _rec file.
- Todo:
this is duplicate, see utils.lookup_dims()
- em_workflows.brt.flow.gen_ali_x(fp_in: Path, z_dim) None
chops an mrc input into its constituent Z sections.
eg if an mrc input has a z_dim of 10, 10 sections will be generated.
the i-i syntax is awkward, and may not be required. Eg possibly replace i-i with i.
eg:
newstack -secs {i}-{i} path/BASENAME_ali*.mrc WORKDIR/hedwig/BASENAME_ali{i}.mrc
- em_workflows.brt.flow.gen_ali_asmbl(fp_in: Path) None
Use IMOD
newstack
to assemble, eg:newstack -float 3 {BASENAME}_ali*.mrc ali_{BASENAME}.mrc
- em_workflows.brt.flow.gen_mrc2tiff(fp_in: Path) None
This generates a lot of jpegs (-j) which will be compiled into a movie. (That is, the -jpeg switch is set to produce jpegs) eg:
mrc2tif -j -C 0,255 ali_BASENAME.mrc BASENAME_ali
- em_workflows.brt.flow.gen_clip_avgs(in_fp: Path, z_dim: str) None
give _rec mrc file, generate a stack of mrcs, averaged to assist viewing.
produces base_ave001.mrc etc, base_ave002.mrc etc,
inputs for newstack (for recon movie) and binvol (for volslicer)
eg:
for i in range(2, dimensions.z-2): IZMIN = i-2 IZMAX = i+2 clip avg -2d -iz IZMIN-IZMAX -m 1 BASENAME_rec.mrc BASENAME_ave${i}.mrc
- em_workflows.brt.flow.consolidate_ave_mrcs(fp_in: Path) Path
consumes base_ave001.mrc etc, base_ave002.mrc etc,
creates ave_base.mrc the (averagedVolume asset)
eg:
newstack -float 3 BASENAME_ave* ave_BASENAME.mrc
- em_workflows.brt.flow.gen_ave_jpgs_from_ave_mrc(ave_mrc: Path)
generates a load of jpgs from the ave_base.mrc with the format {base}_mp4.123.jpg OR {base}_mp4.1234.jpg depending on size of stack.
These jpgs can later be compiled into a movie. eg:
mrc2tif -j -C 100,255 WORKDIR/hedwig/ave_BASNAME.mrc hedwig/BASENAME_mp4
- em_workflows.brt.flow.cleanup_files(file_path: ~pathlib.Path, pattern=<class 'str'>, keep_file: ~pathlib.Path | None = None)
Given a
FilePath
and unix filepattern
, iterate through directory removing all files that match the pattern