From Waveform to Log-Mel: The ASR Preprocessing Pipeline | Sylvia Xiao

Automatic Speech Recognition (ASR) models don’t usually consume raw audio waveforms directly. Instead, many systems—including classic hybrids and modern encoder–decoder Transformers—first convert audio into a time–frequency representation. This preprocessing step is one of the most important “front-ends” in speech: it turns a 1D signal into a 2D representation that makes phonetic structure easier to learn.

This post walks through the end-to-end pipeline:

waveform → framing (windows)

STFT / FFT → spectrum (frequency bins)

Mel filterbank (fbank) → 80/128 mel bands

log compression → log-mel

(optionally) convolutional downsampling → shorter sequences for Transformers

Along the way we clarify:

what STFT really does (it’s not “averaging FFTs”)

what an FFT output looks like and what its axes mean

why FFT for real audio keeps only half the bins

the difference between fbank and log-mel

why some papers say “fbank” even when they mean log-mel

1) What ASR starts with: the waveform

A digital audio recording is a sequence of samples over time:

Sampling rate fs = 16,000 Hz (common in ASR)

Every sample is a real number (e.g., float in [-1, 1])

If the audio length is L seconds, the waveform has:

shape: [T]

where T = L * fs

Example: L = 30 s, fs = 16k → T = 480,000 samples.

Physical meaning (time domain):

x[n] is the microphone’s pressure/electrical signal at time t = n/fs.

2) Why we “window” audio: speech is non-stationary

Human speech changes rapidly: vowels, consonants, transitions, pauses, and noise all vary across time. A single FFT over the entire 30 seconds would erase when things happened.

So we assume speech is approximately stationary inside a short window (a “frame”), such as:

Window length: 25 ms → N = 0.025 * 16000 = 400 samples

Hop length: 10 ms → H = 0.010 * 16000 = 160 samples

This creates overlapping frames:

Frame 0: samples [0 : 400)

Frame 1: samples [160 : 560)

Frame 2: samples [320 : 720)

Frame count (approx):

~ L / hop

1 s / 10ms ≈ 100 frames

30 s ≈ 3000 frames (exact count depends on padding/edge handling)

At this stage, you can imagine:

shape: [num_frames, N] ≈ [3000, 400]

3) FFT: turning one frame into a frequency “recipe”

3.1 What FFT outputs

For one frame of length (often equal to the window length), the FFT returns:

Each is a complex number:

magnitude → “how much of that frequency”
phase → “timing offset / alignment”

But speech features usually do not use complex values directly. Instead we compute:

Magnitude spectrum: (non-negative real)

Power spectrum: (non-negative real)

3.2 What are frequency “bins”?

Each FFT index k corresponds to a frequency:

Frequency resolution is:

Example: fs=16000, N_fft=400:

Δf = 16000 / 400 = 40 Hz

bins represent 0, 40, 80, ..., 8000 Hz (for non-negative half; more below)

Axes after FFT (for a single frame):

x-axis: frequency (Hz) or bin index k

y-axis: magnitude or power

4) STFT: sliding FFT across time (not “averaging”)

STFT (Short-Time Fourier Transform) is simply:

“Do FFT on every windowed frame, then stack the results over time.”

So for the whole signal:

Each time frame → one spectrum

All frames → a time–frequency matrix

If we use magnitude or power, we get a real-valued spectrogram:

shape: [num_frames, num_freq_bins]

Where num_freq_bins depends on N_fft.

4.1 Window function: not averaging, but weighting

Before FFT, we typically multiply the frame by a window function (e.g., Hann). This reduces spectral leakage caused by hard-cutting a segment. This is not averaging FFTs; it’s weighting samples inside the frame.

5) Why FFT for real audio keeps only half the bins

Audio waveforms are real-valued signals. For real inputs, the FFT is conjugate symmetric:

Meaning:

the negative-frequency half contains no new information

it’s a mirror (complex conjugate) of the positive-frequency half

So we keep only the non-negative frequencies:

(inclusive)

total bins =

Example: N_fft=400:

bins kept =

corresponds to 0 Hz to 8000 Hz (Nyquist)

This is why many libraries provide “rFFT” (real FFT) which directly returns bins.

6) From linear frequency bins to Mel bands: fbank

The STFT spectrum has linear frequency spacing. Human perception is not linear: we resolve low frequencies more finely than high frequencies.

A Mel filterbank applies triangular filters on the linear-frequency spectrum and sums energy into mel-spaced bands:

Input: num_freq_bins (e.g., 201)

Output: M mel bands (commonly 80 or 128)

This output is called fbank (filterbank energies):

shape: [num_frames, M] (e.g., [3000, 80])

values: non-negative real energies

Important:

The Mel step is where you “reduce” frequency dimensionality (201 → 80/128).

It’s a weighted aggregation over frequency bins.

7) Log compression: log-mel (why ASR models like it)

Now take the mel energies and apply a log transform:

7.1 Why log helps

Dynamic range compression: speech energy varies by orders of magnitude; log makes values more stable.

Closer to human loudness perception: decibels are log-like.

Easier optimization: models learn better when input scales are well-behaved.

Key distinction:

fbank = mel energies before log

log-mel = mel energies after log

They usually have the same shape, just different numeric scales.

8) Shape evolution summary (typical ASR setup)

Assume:

fs=16000

window = 25 ms → N=400

hop = 10 ms → H=160

N_fft = 400

mel bands M = 80 (or 128)

For L = 30 s:

Waveform

shape: [480000]

Framing (conceptually)

shape: [~3000, 400]

STFT magnitude/power (keep non-negative frequencies)

num_freq_bins = N_fft/2 + 1 = 201

shape: [~3000, 201]

Mel filterbank (fbank)

shape: [~3000, 80] (or [~3000, 128])

Log compression (log-mel)

shape: [~3000, 80] (or [~3000, 128])

(Optional) Convolutional downsampling before Transformer encoder

Many models reduce time length (e.g., ×2, ×4, ×8) for speed/memory.

Example: ×2 downsampling: ~3000 → ~1500 frames.

9) “Whisper uses log-mel, but Qwen-ASR says it uses fbank”—why?

In speech literature, the term “fbank” is often used loosely to mean:

Mel filterbank features (sometimes including the log step)

So a model description that says “fbank” may still be using log-mel-like features under the hood—especially if it uses typical ASR feature extractors.

Practical rule:

If a paper says “fbank” but shows typical ASR configs (mel bands + CMVN/log), it may be equivalent to “log-mel.”

Always confirm by checking the feature extractor implementation or config.

10) Mental model (the simplest intuition)

Time domain waveform = “what the microphone recorded”

STFT spectrogram = “which frequencies are present at each moment”

Mel = “compress frequencies to a human-like scale”

Log = “compress energy range to something learnable”

This is why the pipeline is so common: it turns raw audio into a stable, interpretable representation that neural networks (and especially Transformers) can learn from efficiently.

Appendix: Glossary

FFT: Converts one fixed-length frame from time → frequency (complex values).

STFT: Applies FFT to many overlapping frames → time–frequency matrix.

Bin: A discrete frequency slot in FFT output.

Nyquist frequency: fs/2; the highest frequency representable at sampling rate fs.

fbank: Mel filterbank energies (often used as shorthand for mel features).

log-mel: log of mel energies, typically used as model input.

eps (ε): small constant to prevent log(0).