St. LEO V: Speech -> Text on LEO5

This repository provides a tool for transcribing audio (MP3) files on the LEO5 compute cluster at the University of Innsbruck. It uses AI models to perform speech-to-text, speaker diarization (identifying who is speaking), and optional translation. An optional workflow including encryption at rest enhances security.

Table of Contents

Quick Start

  1. Place your MP3 audio file in a directory on LEO5 (e.g., in your $SCRATCH space).
  2. Copy the SLURM script (/scratch/llm/audio_transcription/scripts/stt_leo5.slurm) to your $HOME or $SCRATCH directory.
  3. Edit the "User configuration" section in your copy of the SLURM script to set the path to your audio file and other parameters.
  4. Submit the job: sbatch stt_leo5.slurm.
  5. Check your email for the transcript (for non-encrypted data).

Overview of Models

Please refer to the documentation of these models for details on supported languages and other features.

Prerequisites

  • Access the LEO5 cluster.
  • A Hugging Face account with access to the following gated models:
  • A Hugging Face access token with READ permissions.
  • gocryptfs installed on your local machine for working with encrypted data.
    # For Debian/Ubuntu
    sudo apt install gocryptfs
    

Setup

On LEO5

The necessary software is provided in a Singularity container at /scratch/llm/audio_transcription/container. The scripts are located in /scratch/llm/audio_transcription/scripts. All files are in a central file location on LEO5 for ease of use for everyone. No installation is required on LEO5.

The stt.sh script automatically caches Hugging Face and PyTorch models in your $SCRATCH directory to avoid filling up your $HOME directory.

On Your Local Machine (for encrypted data)

Set up an encrypted workspace on your local machine. See the Working with Encrypted Data section for detailed instructions.

How to Use

1. Prepare Your Audio Files

Ensure the audio files are in MP3 format. If that is not the case, convert them before use.
Create a directory on LEO5 (preferably in $SCRATCH) and place your audio files there.

2. Configure the SLURM Script

Copy the SLURM script to a location where you have write permissions (e.g., your $HOME or $SCRATCH directory).

Edit the "User configuration" section of the SLURM script:

Parameter Description
ENCRYPTED Set to 1 for yes, otherwise 0.
WORKSPACE Path to the directory containing your audio files.
FILENAME Name of the file (in the workspace) to process.
NUM_SPEAKERS The number of speakers in the audio (can be 1).
MAX_SPEAKERS The maximum number of speakers if the exact number is unknown.
LANGUAGE The primary language spoken in the audio.
TASK transcribe, translate, or translate_transcript.
TARGET_LANGUAGE Target language for translation.
MAIL_ADDRESS Email addresses to send the transcript to (not used for encrypted data).
HF_TOKEN Hugging Face access token (only for the first use of models – delete after!)
PASSWORD Password for the encrypted directory. Delete after every submission!

3. Run the Job

Submit the SLURM script using sbatch:

sbatch stt_leo5.slurm

4. Retrieve Your Results

  • For non-encrypted data: The transcript will be emailed to the specified recipients.
  • For encrypted data: The results will be stored in the encrypted directory on LEO5. You will need to sync them back to your local machine to view them.

The audio file and the transcript are removed from LEO5 after the job completes.

Error messages are written to a file called errors.txt.

Working with Encrypted Data

This workflow ensures that the data remains encrypted from your local machine to the LEO5 cluster and back.

Note that there is a script available that guides through all the steps: workflow_leo5.sh.

Using ssh-keys is highly recommended. It is also recommended to add the LEO5 user name and the UIBK mail address to the script in order not be prompted for them at every use.

1. One-Time Setup on Your Personal Device

  • Create the workspace:

    cd ~/Documents
    mkdir -p stt-workspace/cryptdir stt-workspace/mountpoint
    cd stt-workspace
    
  • Initialize the encrypted directory: This command sets up the cryptdir as an encrypted filesystem and prompts you to create a password.

    gocryptfs -init cryptdir
    

    Important: Back up the master key that is displayed. Without it, you cannot recover your data if you forget the password.

2. One-Time Setup on LEO5

  • Create the corresponding workspace: Create the same directory structure in your $SCRATCH space on LEO5.
    mkdir -p $SCRATCH/stt-workspace/cryptdir
    

3. Recurring Workflow

  • a) Mount and Add Data (Local):

    cd ~/Documents/stt-workspace
    gocryptfs cryptdir mountpoint
    cp ~/Downloads/interview.mp3 mountpoint/
    fusermount -u mountpoint
    
  • b) Sync to LEO5 (Local):

    rsync -av cryptdir/ YOUR_USERNAME@leo5:/scratch/YOUR_USERNAME/stt-workspace/cryptdir/
    

    (Replace YOUR_USERNAME with your LEO5 username.)

  • c) Run the Job (LEO5): Execute the SLURM job on LEO5: sbatch stt_leo5.slurm

  • d) Sync Results Back (Local):

    cd ~/Documents/stt-workspace
    rsync -av YOUR_USERNAME@leo5:/scratch/YOUR_USERNAME/stt-workspace/cryptdir/ cryptdir/
    

    (Do not forget the trailing slashes.)

  • e) Access the Results (Local):

    gocryptfs cryptdir mountpoint
    

    Your results will be inside the mountpoint directory. Remember to unmount when you are done:

    fusermount -u mountpoint
    

Important Notes

  • For best results, speakers should alternate. Overlapping speech can negatively impact diarization. To avoid too many short passages in the transcript, short snippets that might be just a "yes", "aha", "okay" or the like get cut out. If not all speakers get properly recognized, increase the number of speakers – some noises might be attributed to a speaker and get cut out.
  • Audio files longer than approximately 2 hours will be split for diarization. Files longer than 4 hours are not currently supported.
  • The models are primarily trained on English. Transcription quality may be lower for other languages or when multiple languages are mixed.
  • The first time a model is used, it will be downloaded, which may increase the job's runtime.

Future Work

  • Support for splitting audio files longer than 4 hours.
  • Explore using A40 GPUs (interactively) for more advanced translation models.

Usage on other (HPC) Systems

Since on other systems the container is not provided, it has to be built with the definition file. The path to the Python environment package list file has to be adapted therein.

The scripts need to be placed in a directory with read and execute permission.

Adjusting the SLURM file

  • The SLURM file is very generic and is expected to work on most systems, depending on the node scheduling policy.

  • The paths to the directory containing the container and to the scripts have to be adapted.

  • Singularity or Apptainer must be available; it is currently loaded as a LEO5-specific module.

  • All necessary directories (for models and caches) have to be bound to the container; e.g. with the SINGULARITY_BIND variable.

  • At the srun command, the path to stt.sh has to be adjusted.

It is also an option to run without SLURM by directly executing stt.sh after updating all settings therein.

Funding and Contributors

This work got support from EuroCC Austria, the Austrian National Competence Centre for High-Performance Computing, High-Performance Data Analytics and Artificial Intelligence, which has received funding from the European High Performance Computing Joint Undertaking (JU) and Germany, Bulgaria, Austria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, Greece, Hungary, Ireland, Italy, Lithuania, Latvia, Poland, Portugal, Romania, Slovenia, Spain, Sweden, France, Netherlands, Belgium, Luxembourg, Slovakia, Norway, Türkiye, Republic of North Macedonia, Iceland, Montenegro, Serbia under grant agreement No 101101903.

This project was developed by Andreas Lindner with support from the HPC team at the University of Innsbruck.

Disclaimer

Please note that the transcription of audio recordings of people constitutes the processing of personal data, which may also include sensitive content. Ensure that you comply with the applicable legal requirements when using this software.