File_path module

file_path.log(msg: str) → None

Convenience method to write an INFO message to a Prefect log.

class file_path.FilePath(share_name: str, input_dir: Path, fp_in: Path)

The FilePath class is used to track the directory structure of the input and output files when running an image pipeline. The output _asset_dir and a temporary (fast-disk) _working_dir are created for each input file. These members are @properties without setters to keep them immutable, as should the entire class, probably. It is important that each file have its own _working_dir to avoid any collisions during the asynchronous processing of the pipeline. Very many output files are created in the _working_dir, but only the outputs we care about are added to the FilePath for copying to the _asset_dir later in the pipeline. An “asset” is a resource the Hedwig Web application uses. For example an asset might be an image, or a movie, or output of the pipeline, that the web application users care about.

Todo:

Consider making entire class immutable

property assets_dir: Path

the top level directory where results are left.

other subdirs are attached here containing the outputs of individual files

property working_dir: Path

A pathlib.Path of the temporary (high-speed) directory where the working files will be stored. This is a property without a setter to make it immutable. :return: pathlib.Path

get_environment() → str

The workflows can operate in one of several environments, named HEDWIG_ENV for historical reasons, eg prod, qa or dev. This function looks up that environment. Raises exception if no environment found.

make_work_dir() → Path

a temporary dir to house all files in the form: {Config.tmp_dir}{fname.stem}. eg: /gs1/home/macmenaminpe/tmp/tmp7gcsl4on/tomogram_fname/ Will be rm’d upon completion.

make_assets_dir() → Path

proj_dir comes in the form {mount_point}/RMLEMHedwigQA/Projects/Lab/PI/ want to create: {mount_point}/RMLEMHedwigQA/Assets/Lab/PI/

copy_to_assets_dir(fp_to_cp: Path) → Path

Copy FilePath to the assets (reported output) dir

  • fp is the Path to be copied.

  • assets_dir is the root dir (the proj_dir with s/Projects/Assets/)

gen_output_fp(output_ext: str | None = None, out_fname: str | None = None) → Path

cat working_dir to input_fp.name, but swap the extension to output_ext the reason for having a working_dir default to None is sometimes the output dir is not the same as the input dir, and working_dir is used to define output in this case.

gen_asset(asset_type: str, asset_fp) → Dict

Construct and return an asset (dict) based on the asset “type” and FilePath :param asset_type: a string that details the type of output file :param asset_fp: the originating FilePath to “hang” the asset on :return: the resulting “asset” in the form of a dict

gen_prim_fp_elt(exceptions_as_str: str | None = None) → Dict

creates a single primaryFilePath element, to which assets can be appended.

Todo:

Is following “todo” comment out of date?

Todo:

input_fname_b is optional, sometimes the input can be a pair of files.

eg:

[
 {
  "primaryFilePath": "Lab/PI/Myproject/MySession/Sample1/file_a.mrc",
  "thumbnailIndex": 0,
  "fileMetadata": null,
  "imageSet": []
 }
]
copy_workdir_to_assets() → Path

  • copies all of working dir to Assets dir.

  • tests to see if the destination dir exists prior to copy

  • removes work dir upon completion.

  • returns newly created dir

copy_workdir_logs_to_assets() → Path

  • copies all working dir logs to Assets dir.

  • tests to see if the destination dir exists prior to copy

  • removes work dir upon completion.

  • returns newly created dir

rm_workdir()

Removes the the entire working directory

static run(cmd: List[str], log_file: str, env: Dict | None = None, *, copy_env: bool = True) → int

Runs a Unix command as a subprocess

  • If final returncode is not 0, raises a RuntimeError

Parameters:

  • cmd – list of strings representing the command to run

  • log_file – path to the log file to write the stdout and stderr to

  • env – dictionary of additional environment variables to pass to the subprocess

  • copy_env – if True, the subprocess inherits the parent’s environment

Returns:

the return code of the subprocess