Processing Audio Data in LakeInsight

Demo Content：

1. Read audio data from LakeSoul multimodal lakehouse

2. Use Daft with Whisper model to transcribe audio to text

You can access the LakeInsight Demo Environment and open demo/display_audio.ipynb to view the demo code.

1. Environment Initialization and Parameter Configuration

This step imports the components needed for the demo:

• Ray Data: Read audio data from LakeSoul tables.
• LakeSoul Ray Connector: Register the ray.data.read_lakesoul() interface.
• Daft: Organize the audio transcription process as DataFrame computation tasks.
• faster-whisper: Perform the actual speech recognition.
• Pandas / HTML: Generate result tables with embedded audio players.

Key Parameters:

• TABLE: The LakeSoul table to read.
• NAMESPACE: LakeSoul namespace.
• LIMIT: Number of audio samples to process in this demo.
• MODEL: Path to the local faster-whisper model directory.

import os
import warnings
import logging
logging.disable(logging.WARNING)

import daft
import lakesoul.ray  # Registers ray.data.read_lakesoul.
import ray
import base64
import html
import re
import pandas as pd
from daft import DataType, col
from IPython.display import HTML, display


TABLE = "librispeech_minimal_parquet_s3"

NAMESPACE = "default"
LIMIT = 4
MODEL = "/opt/models/faster-whisper-base.en"

2. Text Normalization and WER Calculation

The answer in the dataset is the human-provided ground truth, while Whisper outputs the predicted text.

The two may differ only in capitalization or punctuation, for example:

• Ground truth: A ROUTE SLIGHTLY LESS DIRECT THAT'S ALL
• Prediction: a route slightly less direct. That's all.

Before comparison, we need to normalize case and remove common punctuation to avoid treating formatting differences as recognition errors.

def normalize(text):
    return re.sub(r"[^A-Z0-9']+", " ", text.upper()).strip()

3. Read Audio Data from LakeSoul

Use ray.data.read_lakesoul() to read the LakeSoul table.

This example requires the following fields:

• sample_id: Unique sample identifier for locating and correlating records.
• context.bytes: FLAC audio binary stored in LakeSoul.
• answer: The ground truth transcription text provided by the original dataset.

After reading, convert nested fields to a structure more suitable for Daft processing:

• context.bytes → audio_bytes
• answer → reference_text. batch_size and thread_count are set to small values to control concurrency and memory usage in the demo environment.

if not ray.is_initialized():
    ray.init(
        num_cpus=1,
        include_dashboard=False, # Suppress Ray logs
        log_to_driver=False,
        logging_level=logging.ERROR,
    )

# Read LakeSoul table
source = ray.data.read_lakesoul(
    TABLE,
    namespace=NAMESPACE,
    batch_size=LIMIT,
    thread_count=1,
    retain_partition_columns=True,
)

4. Transcribe with Daft and faster-whisper

Daft itself does not perform speech recognition; it organizes DataFrame computation and schedules UDFs via Ray.

WhisperTranscriber is a Daft stateful UDF:

1. The Actor loads the faster-whisper model once during initialization.
2. Each row receives a segment of FLAC audio binary.
3. The binary is temporarily written to an audio file.
4. faster-whisper is called for English speech recognition.
5. All recognized segments are merged to produce the prediction field.

max_concurrency=1 restricts the same model instance to processing one task at a time, reducing CPU and memory pressure.

# Configure Daft executor
daft.set_runner_ray(noop_if_initialized=True)

@daft.cls(cpus=1, max_concurrency=1)
class WhisperTranscriber:
    def __init__(self): # Load Whisper model
        from faster_whisper import WhisperModel

        self.model = WhisperModel(
            MODEL,
            device="cpu", # CPU inference
            compute_type="int8",
        )
    # Audio transcription method 
    @daft.method(return_dtype=DataType.string())
    def transcribe(self, audio_bytes: bytes) -> str:
        import tempfile

        with tempfile.NamedTemporaryFile(suffix=".flac") as audio_file:
            audio_file.write(audio_bytes)
            audio_file.flush()
            segments, _ = self.model.transcribe( # Execute Whisper
                audio_file.name,
                language="en",
                beam_size=1,
                temperature=0,
            )
            return " ".join(segment.text.strip() for segment in segments)


transcriber = WhisperTranscriber()

limited_rows = source.limit(LIMIT)
print("Rows processed:", limited_rows.count())


audio_df = (
    daft.from_ray_dataset(limited_rows) # Convert ray dataset to daft dataframe
    .select(
        col("context")["bytes"].alias("audio_bytes"),
        col("answer").alias("reference_text"),
    ).into_partitions(1) # Use single partition to save memory
)

results = (
    audio_df
    .with_column(
        "prediction",
        transcriber.transcribe(col("audio_bytes")), # Transcribe audio
    )
    .select(
        "audio_bytes",
        "reference_text",
        "prediction",
      )
    .to_arrow()
    .to_pylist()
)

Rows processed: 4

5. Display and Verify Transcription Results

The final result table includes:

• Audio: Browser audio player generated from the audio binary.
• Ground Truth: The reference answer stored in the LakeSoul table.
• Whisper Transcription: The predicted text re-transcribed by Whisper.

comparison_rows = []

for row in results:
    reference = row["reference_text"] # Dataset ground truth
    prediction = row["prediction"]    # Re-transcribed text after recognition
    audio_base64 = base64.b64encode(row["audio_bytes"]).decode("ascii") # Encode binary to base64 for embedding in notebook HTML

    audio_player = (
        '<audio controls preload="none">'
        f'<source src="data:audio/flac;base64,{audio_base64}" '
        'type="audio/flac">'
        "</audio>"
    )

    comparison_rows.append({
      "Audio": audio_player,
      "Ground Truth": html.escape(reference),
      "Whisper Transcription": html.escape(prediction),
  })

comparison_df = pd.DataFrame(comparison_rows) # Convert list to Pandas DataFrame

display(
    HTML(
        comparison_df.to_html(
            escape=False,
            index=False,
        )
    )
)

Audio	Ground Truth	Whisper Transcription
	A ROUTE SLIGHTLY LESS DIRECT THAT'S ALL	a route slightly less direct, that's all.
	NANCY'S CURLY CHESTNUT CROP SHONE IN THE SUN AND OLIVE'S THICK BLACK PLAITS LOOKED BLACKER BY CONTRAST	Nancy's curly chestnut crop, shown in the sun and olive's thick black plates looked blacker by contrast.
	THERE BEFELL AN ANXIOUS INTERVIEW MISTRESS FITZOOTH ARGUING FOR AND AGAINST THE SQUIRE'S PROJECT IN A BREATH	There befell an anxious interview, Mistress Fitzhuth arguing for and against the Squire's project in a breath.
	ROBIN FITZOOTH SAW THAT HIS DOUBTS OF WARRENTON HAD BEEN UNFAIR AND HE BECAME ASHAMED OF HIMSELF FOR HARBORING THEM	Robin Fitzhuth saw that his doubts of warrant and had been unfair, and he became ashamed of himself for harboring them.