Managing Prefect Server

Continuous Deployment (dev to qa to prod)

Server images are promotoed from one environment to the next, i.e. dev -> qa -> prod.

For example to promote the image from dev to qa:

spaces task -f hedwig.spaces-solution.yaml promote-image -- dev qa

We can then deploy the aws infrastructure for qa:

SPACES_SOLUTION_ENV=qa spaces task -f hedwig.spaces-solution.yaml build-deploy

This can then again be applied for the qa to prod changes.

Managing Prefect Worker

Deploying workflows

Once the prefect server is running and workpool is created. You can login to respective BigSky instance (dev, qa, prod) and deploy the workflows.

Make sure the configurations are correct:

1. Update prefect.yaml to change the user and/or directory names

   # by default pull.directory setting is set for prod environment
   directory: /gs1/home/hedwig_prod/image_portal_workflows
   
   # change it to dev or qa, based on your environment
   

2. Check HPC worker daemon:

   systemctl status hedwig_listener_prod
   

   Certain scenarios require the deamon to be restarted or reloaded, although typically we do not need to perform this step. (see helper_scripts/.service file) The systemctl should restart the worker if killed or on crash.

3. Check prefect config with view

   prefect config view
   
   # Update config iff required
   export PREFECT_API_KEY=xyz
   export PREFECT_API_URL=abc.com
   

4. Deploy flows with prefect deploy

   prefect deploy
   # Or deploy all based on prefect.yaml using the following setting
   # However, this will also deploy pytest_runner workflow in other envs (where it's not needed)
   # prefect deploy --all
   

Troubleshooting:

• Prefect Server is running, but workpool is not available

  In HPC, you can use the following command to create a work pool. prefect work-pool create “workpool” Enssure prefect server is running, and workpool is not available. If not create a workpool by going to the website where the prefect server is hosted. Go to Work Pools tab and create a workpool with the name workpool. This is the name of the workpool that is defined in prefect.yaml > definitions > work_pools > name file.

• IMOD unable to find env

Unable to run command.
Cannot run program "env" (in directory "?"): error=2, No such file or directory

Note directory “?”, this implies that something is trying to run in a directory that does not exist. Ensure that the daemon is taken down, ensure that ps aux | grep hedwig does not list any processes that may be running, ensure that the service file is correct, ensure that the daemon is reloaded and started.