Config module

config.setup_pytools_log()

config.SLURM_exec(asynchronous: bool = False, **cluster_kwargs)

brings up a dynamically sized cluster.

Docs: https://jobqueue.dask.org/en/latest/generated/dask_jobqueue.SLURMCluster.html

We can view the sbatch script using the following command, to know how the job is started by slurm: python -c “from em_workflows import config; c = config.SLURM_exec(); print(c.job_script())”

The processes determins number of dask workers, and nthreads = cores / processes The memory limit is also divided among the workers

More about the cluster: https://bigskywiki.niaid.nih.gov/big-sky-architecture

class config.Config

Bases: object

imod_root = '/data/apps/software/spack/linux-rocky9-x86_64_v3/gcc-11.3.1/imod-5.1.1-vyv6iidgdilzyxoqumqmdbyokzi4cdlx/IMOD/'

bioformats2raw = '/data/apps/software/spack/linux-rocky9-x86_64_v3/gcc-11.3.1/bioformats2raw-0.9.4-yj7uyq6r7zduyd34h75nt6kootcuxrg4/bioformats2raw/bin/bioformats2raw'

brt_binary = '/data/apps/software/spack/linux-rocky9-x86_64_v3/gcc-11.3.1/imod-5.1.1-vyv6iidgdilzyxoqumqmdbyokzi4cdlx/IMOD//bin/batchruntomo'

header_loc = '/data/apps/software/spack/linux-rocky9-x86_64_v3/gcc-11.3.1/imod-5.1.1-vyv6iidgdilzyxoqumqmdbyokzi4cdlx/IMOD//bin/header'

mrc2tif_loc = '/data/apps/software/spack/linux-rocky9-x86_64_v3/gcc-11.3.1/imod-5.1.1-vyv6iidgdilzyxoqumqmdbyokzi4cdlx/IMOD//bin/mrc2tif'

newstack_loc = '/data/apps/software/spack/linux-rocky9-x86_64_v3/gcc-11.3.1/imod-5.1.1-vyv6iidgdilzyxoqumqmdbyokzi4cdlx/IMOD//bin/newstack'

ffmpeg_loc = '/data/apps/software/spack/linux-rocky9-x86_64_v3/gcc-11.3.1/ffmpeg-6.0-zq2bmekz3iolxjshigm6b6q2w64kn5h2/bin/ffmpeg'

gm_loc = '/data/apps/software/spack/linux-rocky9-x86_64_v3/gcc-11.3.1/graphicsmagick-1.3.43-5cc6lqtchmgntmy66i56rs55nk6aqopp/bin/gm'

HIGH_SLURM_EXECUTOR = <prefect_dask.task_runners.DaskTaskRunner object>

SLURM_EXECUTOR = <prefect_dask.task_runners.DaskTaskRunner object>

user = 'runner'

tmp_dir = '/data/scratch/runner'

local_storage = LocalFileSystem(basepath='/home/runner/.prefect/local-storage')

pickle_serializer = PickleSerializer(type='pickle', picklelib='pickle', picklelib_version=None)

static proj_dir(share_name: str) → str

Parameters:

share_name – FileShareEnum string

Returns:

Projects folder mount point based on the file-share name

static assets_dir(share_name: str) → str

repo_dir = PosixPath('/home/runner/work/image_portal_workflows/image_portal_workflows/em_workflows')

template_dir = PosixPath('/home/runner/work/image_portal_workflows/image_portal_workflows/em_workflows/templates')