BRT flow

Batchruntomo pipeline overview:

  • Takes an input_dir containing 1 or more .mrc or .st files

  • IMOD batchruntomo (BRT) https://bio3d.colorado.edu/imod/doc/man/batchruntomo.html is run on each file in dir.

  • BRT has a large number of parameters, which are provided using an .adoc parameter file.

  • Some of these parameters are provided to the pipeline on a per run basis, the remaining of these parameters are defaulted using the templated adoc file.

  • There are two templates, ../templates/plastic_brt.adoc and ../templates/cryo_brt.adoc (Note, currently these are identical, we expect this to change.) The template chosen is defined by parameter adoc_template.

  • The run time parameters are interpolated into the template, and this file is then run with BRT.

  • BRT produces two output files that we care about: _ali.mrc and _rec.mrc

  • Using the ali (alignment) file we generate a tilt movie:

    • look up dimensionality of alignment file, and generate n sections in gen_ali_x()

    • we assemble these sections into a single aligned mrc file, in gen_ali_asmbl()

    • we convert this into a stack of jpegs, in gen_mrc2tiff()

    • we compile these jpegs into a tilt movie in gen_tilt_movie()

    • ADDITIONALLY:

    • we take the mid point of the stack jpeg to use as the display thumbnail.

    • we clean up intermediate files now.

  • Using the _rec (reconstructed) file we generate a reconstructed movie (in a similar fashion to the above)

    • look up dimensionality of the reconstructed (_rec) file,

    • create a stack of averaged mrc files in gen_clip_avgs()

    • create single mrc using the above averaged stack, in consolidate_ave_mrcs()

    • convert this mrc to a stack of jpegs, in gen_ave_jpgs_from_ave_mrc()

    • compile jpegs into reconstructed movie, in gen_recon_movie()

    • clean up after ourselves in cleanup_files()

  • Use average (created above) reconstructed mrc file to create input for volslicer.k in gen_ave_8_vol()

  • need to produce pyramid files with reconstructed mrc.

    • convert mrc file to Zarr, in gen_zarr()

  • Now we need to copy the outputs to the right place, and tell the API where they are. We use JSON to talk to the API.

  • build a json datastructure, containing the locations of the inputs we key on “primaryFilePath”, and we append every output that’s generated for this input into the “assets” json key.

  • Finally, we POST the JSON datastructure to the API, and cleanup temp dirs.

em_workflows.brt.flow.gen_dimension_command(file_path: FilePath, ali_or_rec: str) → str
looks up the z dimension of an mrc file.
ali_or_rec is nasty, str to denote whether you’re using the _ali file or the _rec file.
Todo:

this is duplicate, see utils.lookup_dims()

em_workflows.brt.flow.gen_ali_x(file_path: FilePath, z_dim) → None

  • chops an mrc input into its constituent Z sections.

  • eg if an mrc input has a z_dim of 10, 10 sections will be generated.

  • the i-i syntax is awkward, and may not be required. Eg possibly replace i-i with i.

  • eg:

    newstack -secs {i}-{i} path/BASENAME_ali*.mrc WORKDIR/hedwig/BASENAME_ali{i}.mrc
em_workflows.brt.flow.gen_ali_asmbl(file_path: FilePath) → None

Use IMOD newstack to assemble, eg:

newstack -float 3 {BASENAME}_ali*.mrc ali_{BASENAME}.mrc
em_workflows.brt.flow.gen_mrc2tiff(file_path: FilePath) → None

This generates a lot of jpegs (-j) which will be compiled into a movie. (That is, the -jpeg switch is set to produce jpegs) eg:

mrc2tif -j -C 0,255 ali_BASENAME.mrc BASENAME_ali
em_workflows.brt.flow.gen_thumbs(file_path: FilePath, z_dim) → dict

Use GraphicsMagick to create thumbnail images, eg:

gm convert -size 300x300 BASENAME_ali.{MIDDLE_I}.jpg -resize 300x300                 -sharpen 2 -quality 70 keyimg_BASENAME_s.jpg
em_workflows.brt.flow.gen_tilt_movie(file_path: FilePath) → dict

generates the tilt movie, eg:

ffmpeg -f image2 -framerate 4 -i ${BASENAME}_ali.%03d.jpg -vcodec libx264                 -pix_fmt yuv420p -s 1024,1024 tiltMov_${BASENAME}.mp4
em_workflows.brt.flow.gen_recon_movie(file_path: FilePath) → dict

compiles a stack of jpgs into a movie. eg:

ffmpeg -f image2 -framerate 8 -i WORKDIR/hedwig/BASENAME_mp4.%04d.jpg -vcodec libx264                 -pix_fmt yuv420p -s 1024,1024 WORKDIR/hedwig/keyMov_BASENAME.mp4
em_workflows.brt.flow.gen_clip_avgs(file_path: FilePath, z_dim: str) → None

  • give _rec mrc file, generate a stack of mrcs, averaged to assist viewing.

  • produces base_ave001.mrc etc, base_ave002.mrc etc,

  • inputs for newstack (for recon movie) and binvol (for volslicer)

eg:

for i in range(2, dimensions.z-2):
    IZMIN = i-2
    IZMAX = i+2
    clip avg -2d -iz IZMIN-IZMAX  -m 1 BASENAME_rec.mrc BASENAME_ave${i}.mrc
em_workflows.brt.flow.consolidate_ave_mrcs(file_path: FilePath) → dict

  • consumes base_ave001.mrc etc, base_ave002.mrc etc,

  • creates ave_base.mrc the (averagedVolume asset)

  • eg:

    newstack -float 3 BASENAME_ave* ave_BASENAME.mrc
em_workflows.brt.flow.gen_ave_8_vol(file_path: FilePath)

  • creates volume asset, for volslicer, eg:

    binvol -binning 2 WORKDIR/hedwig/ave_BASENAME.mrc WORKDIR/avebin8_BASENAME.mrc
em_workflows.brt.flow.gen_ave_jpgs_from_ave_mrc(file_path: FilePath)

  • generates a load of jpgs from the ave_base.mrc with the format {base}_mp4.123.jpg OR {base}_mp4.1234.jpg depending on size of stack.

  • These jpgs can later be compiled into a movie. eg:

    mrc2tif -j -C 100,255 WORKDIR/hedwig/ave_BASNAME.mrc hedwig/BASENAME_mp4
em_workflows.brt.flow.cleanup_files(file_path: FilePath, pattern=str)

Given a FilePath and unix file pattern, iterate through directory removing all files that match the pattern

em_workflows.brt.flow.gen_dimension_command(fp_in: Path) → str
looks up the z dimension of an mrc file.
ali_or_rec is nasty, str to denote whether you’re using the _ali file or the _rec file.
Todo:

this is duplicate, see utils.lookup_dims()

em_workflows.brt.flow.gen_ali_x(fp_in: Path, z_dim) → None

  • chops an mrc input into its constituent Z sections.

  • eg if an mrc input has a z_dim of 10, 10 sections will be generated.

  • the i-i syntax is awkward, and may not be required. Eg possibly replace i-i with i.

  • eg:

    newstack -secs {i}-{i} path/BASENAME_ali*.mrc WORKDIR/hedwig/BASENAME_ali{i}.mrc
em_workflows.brt.flow.gen_ali_asmbl(fp_in: Path) → None

Use IMOD newstack to assemble, eg:

newstack -float 3 {BASENAME}_ali*.mrc ali_{BASENAME}.mrc
em_workflows.brt.flow.gen_mrc2tiff(fp_in: Path) → None

This generates a lot of jpegs (-j) which will be compiled into a movie. (That is, the -jpeg switch is set to produce jpegs) eg:

mrc2tif -j -C 0,255 ali_BASENAME.mrc BASENAME_ali
em_workflows.brt.flow.gen_clip_avgs(in_fp: Path, z_dim: str) → None

  • give _rec mrc file, generate a stack of mrcs, averaged to assist viewing.

  • produces base_ave001.mrc etc, base_ave002.mrc etc,

  • inputs for newstack (for recon movie) and binvol (for volslicer)

eg:

for i in range(2, dimensions.z-2):
    IZMIN = i-2
    IZMAX = i+2
    clip avg -2d -iz IZMIN-IZMAX  -m 1 BASENAME_rec.mrc BASENAME_ave${i}.mrc
em_workflows.brt.flow.consolidate_ave_mrcs(fp_in: Path) → Path

  • consumes base_ave001.mrc etc, base_ave002.mrc etc,

  • creates ave_base.mrc the (averagedVolume asset)

  • eg:

    newstack -float 3 BASENAME_ave* ave_BASENAME.mrc
em_workflows.brt.flow.gen_ave_jpgs_from_ave_mrc(ave_mrc: Path)

  • generates a load of jpgs from the ave_base.mrc with the format {base}_mp4.123.jpg OR {base}_mp4.1234.jpg depending on size of stack.

  • These jpgs can later be compiled into a movie. eg:

    mrc2tif -j -C 100,255 WORKDIR/hedwig/ave_BASNAME.mrc hedwig/BASENAME_mp4
em_workflows.brt.flow.cleanup_files(file_path: ~pathlib.Path, pattern=<class 'str'>, keep_file: ~pathlib.Path | None = None)

Given a FilePath and unix file pattern, iterate through directory removing all files that match the pattern