# Audio classification

> Tag a clip with the sounds it contains: a log-mel spectrogram goes through a CNN (PANNs) or a patch transformer (AST), frame scores are pooled to one independent sigmoid per class, and AudioSet's 527 classes are scored by mAP.

Source: https://www.mlfoundry.in/learn/audio-classification  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**Audio classification:** assigns one or more labels to a clip of sound: an event ("dog bark", "siren"), a scene, a spoken keyword, an instrument. Most modern systems treat the log-mel spectrogram as an image. Key models are **PANNs** (Kong et al., "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition", IEEE/ACM TASLP 2020) and the **Audio Spectrogram Transformer** (AST; Gong, Chung and Glass, Interspeech 2021). Both were trained on **AudioSet** (Gemmeke et al., ICASSP 2017).

The input is 1–10 s of audio; the output is a score per class. Real clips overlap several sources, so the task is usually **multi-label**: a street clip can be "car", "speech" and "siren" at once. Labels are often **weak**: AudioSet says a 10 s clip *contains* a dog bark, not when. Events can be 100 ms long inside a 10 s clip, and the same class sounds very different across recordings.

Before deep learning, systems pooled MFCCs or other hand-made features over the clip and fed them to GMMs or SVMs. CNNs on log-mel images took over once large labelled sets appeared. Transformers followed once ImageNet-pretrained **Vision Transformers (ViT)** could be adapted to spectrograms.

## Two architectures

> Figure: A clip becomes a log-mel spectrogram; SpecAugment masks bands and frames during training only. PANNs CNN14 (32 kHz, 64 bands × 1001 frames) runs six blocks of two 3×3 convolutions (64 → 2048 channels), averages over frequency, adds max- and mean-pooling over time, and ends in FC 2048 → 527 sigmoids. AST (16 kHz, 128 × 1024) cuts 1212 overlapping 16×16 patches (stride 10), prepends a CLS token and uses a 12-layer, 768-d transformer initialised from ImageNet. Both output independent sigmoids, scored by mAP.

**PANNs CNN14** is a VGG-like CNN (see **CNN architecture**) with 14 weight layers: 12 convolutions and 2 fully connected. Its input is 64 log-mel bands at 32 kHz with a 1024-sample window and a 320-sample hop, so 10 s gives 1001 frames. After the last block it averages over frequency, then sums max- and mean-pooling over time to get one 2048-d clip embedding. The PANNs paper also released the pretrained weights, which became a standard feature extractor and starting point for fine-tuning.

**AST** has no convolutions. A 128 × 1024 log-mel (10 s at a 10 ms hop) is split into 16 × 16 patches with stride 10, so neighbours overlap by 6. That gives $12\times101=1212$ patches, each linearly projected to 768-d with a learned position embedding. A 12-layer transformer encoder runs over them (see **Self-attention**), and a linear head reads the CLS token. Initialising from an ImageNet-trained vision transformer matters: the position embeddings are adapted from the 2-D image grid.

**Shapes for one 10 s AudioSet clip**

1. PANNs: $10\times32{,}000=320{,}000$ samples, hop 320 → $1000$ hops + 1 = 1001 frames × 64 bands.
2. AST: 16 kHz, 10 ms shift → 1024 frames (padded or cut) × 128 bands.
3. AST patches: $\lfloor(128-16)/10\rfloor+1=12$ along frequency, $\lfloor(1024-16)/10\rfloor+1=101$ along time → $1212$ patches.
4. Plus one CLS token → a $1213\times768$ sequence for each of the 12 layers; attention cost grows with $1213^2\approx1.5$M pairs.

$$
\hat y_c=\sigma\big(w_c^\top h+b_c\big),\qquad \mathcal L=-\frac1C\sum_{c=1}^{C}\Big[y_c\log\hat y_c+(1-y_c)\log(1-\hat y_c)\Big]
$$

- $\hat y_c$: probability that class $c$ is present; one independent sigmoid per class, so scores need not sum to 1
- $h$: pooled clip embedding (CNN14: 2048-d; AST: the 768-d CLS output)
- $\mathcal L$: **Binary cross-entropy** averaged over $C$ classes ($C=527$ for AudioSet); softmax + cross-entropy is used only when classes are mutually exclusive, e.g. ESC-50 or keyword spotting

## From frames to a clip score

$$
y^{\max}=\max_t p_t,\qquad y^{\text{mean}}=\frac1T\sum_t p_t,\qquad y^{\text{lin}}=\frac{\sum_t p_t^2}{\sum_t p_t}
$$

- $p_t$: frame-level probability of one class at frame $t$ (decision-level pooling); pooling can also be applied to embeddings before the classifier (feature-level)
- $y^{\text{lin}}$: linear softmax pooling: each frame is weighted by its own probability. Wang, Li and Metze (ICASSP 2019) compared five pooling functions for weakly labelled sound event detection and found it best at localising events

Max pooling says "present if any frame is confident", so one false frame causes a false tag. Mean pooling says "present on average", which dilutes a 100 ms bark in a 10 s clip. Attention pooling learns the weights instead. Frame-level outputs also give rough timestamps, which turns a tagger into a weakly supervised sound event detector.

$$
\mathrm{AP}_c=\frac{1}{|P_c|}\sum_{k:\,\text{clip}_k\in P_c}\mathrm{Precision@}k,\qquad \mathrm{mAP}=\frac1C\sum_c \mathrm{AP}_c
$$

- $P_c$: clips that truly contain class $c$; clips are ranked by $\hat y_c$ and precision is read at the rank of each positive
- $\mathrm{mAP}$: threshold-free, and every class counts equally however rare it is. It is the standard AudioSet metric (see **Precision, recall & ROC**)

**Datasets and results**

- **AudioSet:** about 2.1M human-labelled 10 s YouTube clips, 527 classes in the released labels (632 in the ontology); balanced train ≈ 22k clips; heavily imbalanced (Speech and Music dominate)
- **ESC-50:** 2,000 environmental clips of 5 s, 50 classes × 40, 5 folds (Piczak, 2015); human accuracy 81.3%
- **UrbanSound8K:** 8,732 urban clips of up to 4 s, 10 classes, 10 folds (Salamon et al., 2014)
- **Speech Commands v2:** 105,829 one-second 16 kHz utterances of 35 words (Warden, 2018); the keyword-spotting benchmark
- **AudioSet mAP:** PANNs CNN14 0.431, Wavegram-Logmel-CNN 0.439; AST 0.459 single model, 0.485 ensemble
- **AST transfer:** ESC-50 95.6% accuracy (AudioSet-pretrained); Speech Commands v2 98.1%

## SpecAugment

SpecAugment (Park et al., Interspeech 2019) augments the spectrogram itself. It applies a time warp, then $m_F$ frequency masks of width $f\sim U[0,F]$ bands and $m_T$ time masks of width $t\sim U[0,T]$ frames, filled with the mean. It was designed for speech recognition: a LAS model with LM fusion reached 2.5%/5.8% WER on LibriSpeech test-clean/test-other. It is now standard for audio tagging too, usually with mixup (see **Data augmentation**). Masks force the model not to rely on any single band or moment.

> Figure: A real multi-label classifier on synthetic 1 s clips of four classes (siren, bark, rain, bird): 40-band log-mel, a logistic layer per class on each frame, and pooling to a clip score. Defaults (clip 5, siren + rain, 2 + 2 masks, F = 8, T = 12): mAP over the 12 test clips is 0.939 with max, 0.932 with mean and 0.942 with linear softmax. Tag accuracy at 0.5 is 83.3%, 66.7% and 87.5%. With masks off, all three reach mAP 1.000, but accuracy stays at 91.7%, 75.0% and 93.8%: mean pooling ranks correctly yet scores short events below 0.5. On clip 2 (bark only) without masks, max tags a false siren (0.57), mean misses the bark (0.40), and linear softmax gets both right (bark 0.90, siren 0.24).

Worked example: one class, 10 frames, a 2-frame event; then AP over 6 clips

| Quantity | Computation | Value |
| --- | --- | --- |
| frame probabilities | 0.05, 0.1, 0.9, 0.95, 0.2, 0.05, 0.05, 0.1, 0.05, 0.05 | Σp = 2.5, Σp² = 1.785 |
| max pooling | max p_t | 0.95 |
| mean pooling | 2.5 / 10 | 0.25 (below 0.5: missed) |
| linear softmax | 1.785 / 2.5 | 0.714 |
| clip scores (truth) | 0.92 (✓), 0.81 (✗), 0.77 (✓), 0.40 (✓), 0.35 (✗), 0.10 (✗) | positives at ranks 1, 3, 4 |
| AP | (1/1 + 2/3 + 3/4) / 3 | 0.8056 |

```python
import numpy as np
rng = np.random.default_rng(0)

def spec_augment(x, F=27, n_freq=2, T=100, n_time=2):
    """x: (n_mels, frames) normalised log-mel. Masks are set to 0 (the mean)."""
    x = x.copy()
    n_mels, frames = x.shape
    for _ in range(n_freq):
        f = rng.integers(0, F + 1); f0 = rng.integers(0, n_mels - f + 1)
        x[f0:f0 + f] = 0
    for _ in range(n_time):
        t = rng.integers(0, T + 1); t0 = rng.integers(0, frames - t + 1)
        x[:, t0:t0 + t] = 0
    return x

def average_precision(scores, labels):
    order = np.argsort(-scores)
    hits = np.cumsum(labels[order])
    prec = hits / np.arange(1, len(scores) + 1)
    return (prec * labels[order]).sum() / labels.sum()

x = spec_augment(rng.standard_normal((128, 1024)))
print((x == 0).mean().round(3))                       # 0.344 of cells masked

frame_p = np.array([0.05, 0.1, 0.9, 0.95, 0.2, 0.05, 0.05, 0.1, 0.05, 0.05])
print(frame_p.max(), frame_p.mean(), (frame_p**2).sum() / frame_p.sum())  # 0.95 0.25 0.714

scores = np.array([0.92, 0.81, 0.77, 0.40, 0.35, 0.10])
labels = np.array([1, 0, 1, 1, 0, 0])
print(average_precision(scores, labels))              # 0.8056; mAP = mean over classes
```

SpecAugment masks, three poolings and average precision in numpy (run with numpy 2.5). A pretrained tagger is one call away in Hugging Face, e.g. pipeline("audio-classification", model="MIT/ast-finetuned-audioset-10-10-0.4593"), not run here.

> **AudioSet numbers are hard to compare:** AudioSet is distributed as YouTube IDs, and videos disappear, so every group trains and evaluates on a slightly different subset. Labels are also noisy and imbalanced: Music and Speech each have hundreds of thousands of clips, while many classes have only a few dozen. Compare mAP only on the same evaluation copy, and use balanced sampling or loss weighting in training.

> **Pretrain on AudioSet, fine-tune small:** Small datasets such as ESC-50 (2,000 clips) are too small to train a large model from scratch. The standard recipe is to pretrain on AudioSet, then fine-tune or train a linear probe. With an AudioSet-pretrained AST, ESC-50 accuracy reaches 95.6%, above the 81.3% human baseline reported with the dataset.

**Strengths**

- CNN (PANNs): cheap, works on any clip length, good on small data thanks to locality
- AST: global attention over the whole clip from layer 1; best AudioSet mAP of the two
- Both: a pretrained AudioSet model gives strong embeddings for other tasks

**Limitations**

- CNN: receptive field grows only with depth; pooling choice matters
- AST: quadratic cost in the 1212 patches; depends on ImageNet initialisation
- Both: weak, noisy, imbalanced labels limit what any architecture can learn

- **Keyword spotting:** "Hey …" wake words on phones and speakers: small CNNs on 40-band log-mels or MFCCs over 1 s windows, trained on Speech Commands-style data
- **Sound event detection:** safety and monitoring (glass break, alarms, gunshots), wildlife and bird surveys, machine-fault detection
- **Media tagging:** content search, music and speech detection, automatic captions of non-speech sounds

**Question:** Why does AudioSet tagging use one sigmoid per class rather than a softmax over 527 classes?

- A. Sigmoids are faster to compute
- B. Clips can contain several sounds at once; independent sigmoids let any number of classes be present
- C. Softmax cannot be trained with 527 classes
- D. mAP requires probabilities that sum to 1

**Answer:** B. Clips can contain several sounds at once; independent sigmoids let any number of classes be present — A softmax forces classes to compete for a total probability of 1, which suits single-label tasks (ESC-50, keywords). AudioSet is multi-label: a clip can be Speech, Music and Dog at once. Each sigmoid is trained with its own binary cross-entropy.

**Question:** A 10 s clip contains one 0.2 s bark, and the frame model outputs about 0.95 on those frames and 0.02 elsewhere. Which pooling gives the most useful clip score?

- A. Mean pooling
- B. Max pooling
- C. Linear softmax pooling
- D. Pooling does not matter for weak labels

**Answer:** B. Max pooling — With 1000 frames (20 bark, 980 background), mean pooling gives $(20\cdot0.95+980\cdot0.02)/1000=0.039$, and linear softmax gives $\sum p^2/\sum p=0.478$. Both fall below 0.5, because 980 background frames dilute the event. Max keeps 0.95. The price is that a single confident false frame would also produce a tag, which is why learned attention pooling is common.

**Key points**

- Audio classification = log-mel image + CNN (PANNs CNN14, mAP 0.431) or patch transformer (AST, 0.459 single / 0.485 ensemble) on AudioSet.
- Multi-label: one sigmoid per class with binary cross-entropy; softmax only for exclusive classes (ESC-50, keywords).
- Frame scores are pooled to the clip: max, mean, attention or linear softmax $\sum p^2/\sum p$; weak labels make this choice matter.
- SpecAugment masks F bands and T frames (plus time warp) during training; mixup is the other standard augmentation.
- Evaluate with mAP (threshold-free, class-balanced); pretrain on AudioSet and fine-tune on small sets like ESC-50.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
