Config module

config.setup_pytools_log()

config.SLURM_exec(asynchronous: bool = False, **cluster_kwargs)

brings up a dynamically sized cluster.

Docs: https://jobqueue.dask.org/en/latest/generated/dask_jobqueue.SLURMCluster.html

We can view the sbatch script using the following command, to know how the job is started by slurm: python -c “from em_workflows import config; c = config.SLURM_exec(); print(c.job_script())”

The processes determins number of dask workers, and nthreads = cores / processes The memory limit is also divided among the workers

More about the cluster: https://bigskywiki.niaid.nih.gov/big-sky-architecture

class config.Config

Bases: object

imod_root = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or'

bioformats2raw = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/bioformats2raw-0.7.0-7kt7dff7f7fxmdjdk57u6xjuzmsxqodn/bin/bioformats2raw'

brt_binary = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or/bin/batchruntomo'

header_loc = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or/bin/header'

mrc2tif_loc = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or/bin/mrc2tif'

newstack_loc = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or/bin/newstack'

HIGH_SLURM_EXECUTOR = <prefect_dask.task_runners.DaskTaskRunner object>

SLURM_EXECUTOR = <prefect_dask.task_runners.DaskTaskRunner object>

user = 'runner'

tmp_dir = '/gs1/Scratch/runner_scratch/'

local_storage = LocalFileSystem(basepath='/home/runner/.prefect/local-storage')

pickle_serializer = PickleSerializer(type='pickle', picklelib='pickle', picklelib_version=None)

static proj_dir(share_name: str) → str

Parameters:

share_name – FileShareEnum string

Returns:

Projects folder mount point based on the file-share name

static assets_dir(share_name: str) → str

repo_dir = PosixPath('/home/runner/work/image_portal_workflows/image_portal_workflows/em_workflows')

template_dir = PosixPath('/home/runner/work/image_portal_workflows/image_portal_workflows/em_workflows/templates')