# St. LEO V: Speech -> Text on LEO5

![St. LEO-V](St_LEO_V.png)

This repository provides a tool for transcribing audio (MP3) files on the LEO5 compute cluster at the University of Innsbruck. It uses AI models to perform speech-to-text, speaker diarization (identifying who is speaking), and optional translation. An optional workflow including encryption at rest enhances security.

## Table of Contents

- [Quick Start](#quick-start)
- [Overview of Models](#overview-of-models)
- [Prerequisites](#prerequisites)
- [Setup](#setup)
  - [On LEO5](#on-leo5)
  - [On Your Local Machine (for encrypted data)](#on-your-local-machine-for-encrypted-data)
- [How to Use](#how-to-use)
  - [1. Prepare Your Audio Files](#1-prepare-your-audio-files)
  - [2. Configure the SLURM Script](#2-configure-the-slurm-script)
  - [3. Run the Job](#3-run-the-job)
  - [4. Retrieve Your Results](#4-retrieve-your-results)
- [Working with Encrypted Data](#working-with-encrypted-data)
  - [1. One-Time Setup on Your Personal Device](#1-one-time-setup-on-your-personal-device)
  - [2. One-Time Setup on LEO5](#2-one-time-setup-on-leo5)
  - [3. Recurring Workflow](#3-recurring-workflow)
- [Important Notes](#important-notes)
- [Future Work](#future-work)
- [Usage on other (HPC) Systems](#usage-on-other-hpc-systems)
- [Funding and Contributors](#funding-and-contributors)

## Quick Start

1.  **Place your MP3 audio file** in a directory on LEO5 (e.g., in your `$SCRATCH` space).
2.  **Copy the SLURM script** (`/scratch/llm/audio_transcription/scripts/stt_leo5.slurm`) to your `$HOME` or `$SCRATCH` directory.
3.  **Edit the "User configuration"** section in your copy of the SLURM script to set the path to your audio file and other parameters.
4.  **Submit the job:** `sbatch stt_leo5.slurm`.
5.  **Check your email** for the transcript (for non-encrypted data).


## Overview of Models

*   **Transcription:** [Whisper](https://huggingface.co/docs/transformers/en/model_doc/whisper) by OpenAI
*   **Speaker Diarization:** [pyannote.audio](https://huggingface.co/pyannote)
*   **Translation:** [Tower Plus](https://huggingface.co/collections/Unbabel/tower-plus) by Unbabel

Please refer to the documentation of these models for details on supported languages and other features.

## Prerequisites

*   Access the LEO5 cluster.
*   A Hugging Face account with access to the following gated models:
    *   [`pyannote/speaker-diarization-3.1`](https://huggingface.co/pyannote/speaker-diarization-3.1)
    *   [`pyannote/segmentation-3.0`](https://huggingface.co/pyannote/segmentation-3.0)
*   A Hugging Face access token with `READ` permissions.
*   `gocryptfs` installed on your local machine for working with encrypted data.
    ```bash
    # For Debian/Ubuntu
    sudo apt install gocryptfs
    ```

## Setup

### On LEO5

The necessary software is provided in a Singularity container at `/scratch/llm/audio_transcription/container`. The scripts are located in `/scratch/llm/audio_transcription/scripts`. All files are in a central file location on LEO5 for ease of use for everyone. No installation is required on LEO5.

The `stt.sh` script automatically caches Hugging Face and PyTorch models in your `$SCRATCH` directory to avoid filling up your `$HOME` directory.

### On Your Local Machine (for encrypted data)

Set up an encrypted workspace on your local machine. See the [Working with Encrypted Data](#working-with-encrypted-data) section for detailed instructions.

## How to Use

### 1. Prepare Your Audio Files

Ensure the audio files are in MP3 format. If that is not the case, convert them
before use.  
Create a directory on LEO5 (preferably in `$SCRATCH`) and place your audio files there.

### 2. Configure the SLURM Script

Copy the SLURM script to a location where you have write permissions (e.g., your `$HOME` or `$SCRATCH` directory).

Edit the "User configuration" section of the SLURM script:

| Parameter | Description |
| --- | --- |
| `ENCRYPTED` | Set to `1` for yes, otherwise `0`. |
| `WORKSPACE` | Path to the directory containing your audio files. |
| `FILENAME` | Name of the file (in the workspace) to process. |
| `NUM_SPEAKERS` | The number of speakers in the audio (can be 1). |
| `MAX_SPEAKERS` | The maximum number of speakers if the exact number is unknown. |
| `LANGUAGE` | The primary language spoken in the audio. |
| `TASK` | `transcribe`, `translate`, or `translate_transcript`. |
| `TARGET_LANGUAGE` | Target language for translation. |
| `MAIL_ADDRESS` | Email addresses to send the transcript to (not used for encrypted data). |
| `HF_TOKEN` | Hugging Face access token (only for the first use of models – **delete after!**) |
| `PASSWORD` | Password for the encrypted directory. **Delete after every submission!** |

### 3. Run the Job

Submit the SLURM script using `sbatch`:

```bash
sbatch stt_leo5.slurm
```


### 4. Retrieve Your Results

*   **For non-encrypted data:** The transcript will be emailed to the specified recipients.
*   **For encrypted data:** The results will be stored in the encrypted directory on LEO5. You will need to sync them back to your local machine to view them.

The audio file and the transcript are removed from LEO5 after the job completes.

Error messages are written to a file called `errors.txt`.


## Working with Encrypted Data

This workflow ensures that the data remains encrypted from
your local machine to the LEO5 cluster and back.

Note that there is a script available that guides through all the steps:
[`workflow_leo5.sh`](scripts/workflow_leo5.sh).

Using ssh-keys is highly recommended.
It is also recommended to add the LEO5 user name and the UIBK mail address to
the script in order not be prompted for them at every use.

### 1. One-Time Setup on Your Personal Device

*   **Create the workspace:**
    ```bash
    cd ~/Documents
    mkdir -p stt-workspace/cryptdir stt-workspace/mountpoint
    cd stt-workspace
    ```

*   **Initialize the encrypted directory:**
    This command sets up the `cryptdir` as an encrypted filesystem and prompts you to create a password.
    ```bash
    gocryptfs -init cryptdir
    ```
    **Important:** Back up the master key that is displayed. Without it, you cannot recover your data if you forget the password.

### 2. One-Time Setup on LEO5

*   **Create the corresponding workspace:**
    Create the same directory structure in your `$SCRATCH` space on LEO5.
    ```bash
    mkdir -p $SCRATCH/stt-workspace/cryptdir
    ```

### 3. Recurring Workflow

*   **a) Mount and Add Data (Local):**
    ```bash
    cd ~/Documents/stt-workspace
    gocryptfs cryptdir mountpoint
    cp ~/Downloads/interview.mp3 mountpoint/
    fusermount -u mountpoint
    ```

*   **b) Sync to LEO5 (Local):**
    ```bash
    rsync -av cryptdir/ YOUR_USERNAME@leo5:/scratch/YOUR_USERNAME/stt-workspace/cryptdir/
    ```
    *(Replace `YOUR_USERNAME` with your LEO5 username.)*

*   **c) Run the Job (LEO5):**
    Execute the SLURM job on LEO5: `sbatch stt_leo5.slurm`

*   **d) Sync Results Back (Local):**
    ```bash
    cd ~/Documents/stt-workspace
    rsync -av YOUR_USERNAME@leo5:/scratch/YOUR_USERNAME/stt-workspace/cryptdir/ cryptdir/
    ```
    *(Do not forget the trailing slashes.)*

*   **e) Access the Results (Local):**
    ```bash
    gocryptfs cryptdir mountpoint
    ```
    Your results will be inside the `mountpoint` directory. Remember to unmount when you are done:
    ```bash
    fusermount -u mountpoint
    ```

## Important Notes

*   For best results, speakers should alternate. Overlapping speech can negatively impact diarization. To avoid too many short passages in the transcript, short snippets that might be just a "yes", "aha", "okay" or the like get cut out. If not all speakers get properly recognized, increase the number of speakers – some noises might be attributed to a speaker and get cut out.
*   Audio files longer than approximately 2 hours will be split for diarization. Files longer than 4 hours are not currently supported.
*   The models are primarily trained on English. Transcription quality may be lower for other languages or when multiple languages are mixed.
*   The first time a model is used, it will be downloaded, which may increase the job's runtime.

## Future Work

*   Support for splitting audio files longer than 4 hours.
*   Explore using A40 GPUs (interactively) for more advanced translation models.


## Usage on other (HPC) Systems

Since on other systems the container is not provided, it has to be built with
the [definition file](container/stt_container.def).
The path to the [Python environment package list file](container/stt_env.txt)
has to be adapted therein.

The [`scripts`](scripts) need to be placed in a directory with read and execute
permission.

### Adjusting the SLURM file

- The SLURM file is very generic and is expected to work on most systems,
depending on the node scheduling policy.

- The paths to the directory containing the container and to the
[`scripts`](scripts) have to be adapted.

- Singularity or Apptainer must be available; it is currently loaded as a
LEO5-specific module.

- All necessary directories (for models and caches) have to be bound to the
container; e.g. with the `SINGULARITY_BIND` variable.

- At the `srun` command, the path to [`stt.sh`](scripts/stt.sh) has to be adjusted.


It is also an option to run without SLURM by directly executing
[`stt.sh`](scripts/stt.sh) after updating all settings therein.


## Funding and Contributors

This work got support from EuroCC Austria, the Austrian National Competence
Centre for High-Performance Computing, High-Performance Data Analytics and
Artificial Intelligence, which has received funding from the European High
Performance Computing Joint Undertaking (JU) and Germany, Bulgaria, Austria,
Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, Greece, Hungary,
Ireland, Italy, Lithuania, Latvia, Poland, Portugal, Romania, Slovenia, Spain,
Sweden, France, Netherlands, Belgium, Luxembourg, Slovakia, Norway, Türkiye,
Republic of North Macedonia, Iceland, Montenegro, Serbia under grant agreement
No 101101903.

This project was developed by Andreas Lindner with support from the HPC team at the University of Innsbruck.


## Disclaimer

Please note that the transcription of audio recordings of people constitutes the processing of personal data, which may also include sensitive content. Ensure that you comply with the applicable legal requirements when using this software.