Config module
- config.setup_pytools_log()
- config.SLURM_exec(asynchronous: bool = False, **cluster_kwargs)
brings up a dynamically sized cluster.
Docs: https://jobqueue.dask.org/en/latest/generated/dask_jobqueue.SLURMCluster.html
We can view the sbatch script using the following command, to know how the job is started by slurm: python -c “from em_workflows import config; c = config.SLURM_exec(); print(c.job_script())”
The processes determins number of dask workers, and nthreads = cores / processes The memory limit is also divided among the workers
More about the cluster: https://bigskywiki.niaid.nih.gov/big-sky-architecture
- class config.Config
Bases:
object
- imod_root = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or'
- bioformats2raw = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/bioformats2raw-0.7.0-7kt7dff7f7fxmdjdk57u6xjuzmsxqodn/bin/bioformats2raw'
- brt_binary = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or/bin/batchruntomo'
- header_loc = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or/bin/header'
- mrc2tif_loc = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or/bin/mrc2tif'
- newstack_loc = '/gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/imod-4.12.47-2fcggru32s3f4jl3ar5m2rztuqz5h2or/bin/newstack'
- HIGH_SLURM_EXECUTOR = <prefect_dask.task_runners.DaskTaskRunner object>
- SLURM_EXECUTOR = <prefect_dask.task_runners.DaskTaskRunner object>
- user = 'runner'
- tmp_dir = '/gs1/Scratch/runner_scratch/'
- local_storage = LocalFileSystem(basepath='/home/runner/.prefect/local-storage')
- pickle_serializer = PickleSerializer(type='pickle', picklelib='pickle', picklelib_version=None)
- static proj_dir(share_name: str) str
- Parameters:
share_name – FileShareEnum string
- Returns:
Projects folder mount point based on the file-share name
- static assets_dir(share_name: str) str
- repo_dir = PosixPath('/home/runner/work/image_portal_workflows/image_portal_workflows/em_workflows')
- template_dir = PosixPath('/home/runner/work/image_portal_workflows/image_portal_workflows/em_workflows/templates')