Simplifying Workflow Management with Snakemake and Task Execution Service (TES)

Microsoft

Feb 14, 2025

In this guide, we will walk you through the process of setting up Snakemake and configuring it to run with a Task Execution Service (TES). By following these steps, you will be able to install Snakemake, set up the necessary environment, and configure TES to run your workflows efficiently. This is part of a larger series we have Using TES and TES Rust.

Below describes the snakemake example.

Client Requirements

You can install all client dependencies using either Conda or the faster alternative, Mamba.

conda env create -f environment.yml

Creating TES Configuration

Next, you need to set the environment variables in your shell. These variables will help configure TES for your workflows:

export TES_ENDPOINT="<Your TES endpoint>" 
export TES_USERNAME="<Your TES username>" 
export TES_PASSWORD="<Your TES password>" 
export STORAGE_ACCT="<Your Azure Storage account password>" 
export AZURITE_CONNECTION_STRING="<Your Azure Storage Connection String>"

Additionally, you will need to update the Snakefile with the name of your storage account.

Running a Snakemake Workflow with TES

With Snakemake installed and TES configured, you are now ready to run your Snakemake workflow on TES. Here is an example command to execute the workflow using TES as the executor:

snakemake --executor tes --tes-url $TES_ENDPOINT --tes-user $TES_USERNAME --tes-password $TES_PASSWORD --default-storage-provider azure --default-storage-prefix 'az://inputs/snakemake' --storage-azure-account-name $STORAGE_ACCT -j1 --envvars AZURITE_CONNECTION_STRING --use-conda --verbose

Viewing Results

Once the workflow is complete, you can view the results in the Azure Blob Storage container outputs/crankshaft-tes. These files can be accessed through the Azure portal or your preferred command-line tool.