Audio classification

Tag a clip with the sounds it contains: a log-mel spectrogram goes through a CNN (PANNs) or a patch transformer (AST), frame scores are pooled to one independent sigmoid per class, and AudioSet's 527 classes are scored by mAP.

8 min read759 wordscore0 viewsupdated 2026-10-15

Definition

Audio classification assigns one or more labels to a clip of sound: an event ("dog bark", "siren"), a scene, a spoken keyword, an instrument. Most modern systems treat the log-mel spectrogram as an image. Key models are PANNs (Kong et al., "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition", IEEE/ACM TASLP 2020) and the Audio Spectrogram Transformer (AST; Gong, Chung and Glass, Interspeech 2021). Both were trained on AudioSet (Gemmeke et al., ICASSP 2017).

The input is 1–10 s of audio; the output is a score per class. Real clips overlap several sources, so the task is usually multi-label: a street clip can be "car", "speech" and "siren" at once. Labels are often weak: AudioSet says a 10 s clip contains a dog bark, not when. Events can be 100 ms long inside a 10 s clip, and the same class sounds very different across recordings.

Before deep learning, systems pooled MFCCs or other hand-made features over the clip and fed them to GMMs or SVMs. CNNs on log-mel images took over once large labelled sets appeared. Transformers followed once ImageNet-pretrained Vision Transformers (ViT) could be adapted to spectrograms.

Two architectures

clip → log-mel → CNN or transformer → 527 sigmoids (AudioSet)10 s cliplog-mel + masksSpecAugmenttraining only:time warp, F-band andT-frame masksAudioSet527 classes~2M 10 s clipsweak clip labelsPANNs CNN14Kong et al. 2020mAP 0.43164 × 1001 in64128256512102420486 blocks × (2 conv 3×3 + BN + ReLU), 2×2 avg pool after blocks 1–5poolmean over freq,max + mean (time)FC 2048→ 527sigmoidASTGong et al. 2021mAP 0.459128 × 1024 in16×16 patches, stride 10CLS12 × 101 = 1212768-d + pos. emb.Transformer12 layers, d 768ImageNet initCLS →linear 527sigmoidmulti-label output: independent σ per class, BCE lossDog 0.91Bark 0.84Speech 0.62Music 0.07Rain 0.03scores need not sum to 1 · threshold per class, or rank clips per class and report mAP
fig.A clip becomes a log-mel spectrogram; SpecAugment masks bands and frames during training only. PANNs CNN14 (32 kHz, 64 bands × 1001 frames) runs six blocks of two 3×3 convolutions (64 → 2048 channels), averages over frequency, adds max- and mean-pooling over time, and ends in FC 2048 → 527 sigmoids. AST (16 kHz, 128 × 1024) cuts 1212 overlapping 16×16 patches (stride 10), prepends a CLS token and uses a 12-layer, 768-d transformer initialised from ImageNet. Both output independent sigmoids, scored by mAP.

PANNs CNN14 is a VGG-like CNN (see CNN architecture) with 14 weight layers: 12 convolutions and 2 fully connected. Its input is 64 log-mel bands at 32 kHz with a 1024-sample window and a 320-sample hop, so 10 s gives 1001 frames. After the last block it averages over frequency, then sums max- and mean-pooling over time to get one 2048-d clip embedding. The PANNs paper also released the pretrained weights, which became a standard feature extractor and starting point for fine-tuning.

AST has no convolutions. A 128 × 1024 log-mel (10 s at a 10 ms hop) is split into 16 × 16 patches with stride 10, so neighbours overlap by 6. That gives 12×101=121212\times101=1212 patches, each linearly projected to 768-d with a learned position embedding. A 12-layer transformer encoder runs over them (see Self-attention), and a linear head reads the CLS token. Initialising from an ImageNet-trained vision transformer matters: the position embeddings are adapted from the 2-D image grid.

Shapes for one 10 s AudioSet clip

  1. 1PANNs: 10×32,000=320,00010\times32{,}000=320{,}000 samples, hop 320 → 10001000 hops + 1 = 1001 frames × 64 bands.
  2. 2AST: 16 kHz, 10 ms shift → 1024 frames (padded or cut) × 128 bands.
  3. 3AST patches: ⌊(128−16)/10⌋+1=12\lfloor(128-16)/10\rfloor+1=12 along frequency, ⌊(1024−16)/10⌋+1=101\lfloor(1024-16)/10\rfloor+1=101 along time → 12121212 patches.
  4. 4Plus one CLS token → a 1213×7681213\times768 sequence for each of the 12 layers; attention cost grows with 12132≈1.51213^2\approx1.5M pairs.
y^c=σ(wc⊤h+bc),L=−1C∑c=1C[yclog⁡y^c+(1−yc)log⁡(1−y^c)]\hat y_c=\sigma\big(w_c^\top h+b_c\big),\qquad \mathcal L=-\frac1C\sum_{c=1}^{C}\Big[y_c\log\hat y_c+(1-y_c)\log(1-\hat y_c)\Big]
y^c\hat y_c
probability that class cc is present; one independent sigmoid per class, so scores need not sum to 1
hh
pooled clip embedding (CNN14: 2048-d; AST: the 768-d CLS output)
L\mathcal L
Binary cross-entropy averaged over CC classes (C=527C=527 for AudioSet); softmax + cross-entropy is used only when classes are mutually exclusive, e.g. ESC-50 or keyword spotting

From frames to a clip score

ymax⁡=max⁡tpt,ymean=1T∑tpt,ylin=∑tpt2∑tpty^{\max}=\max_t p_t,\qquad y^{\text{mean}}=\frac1T\sum_t p_t,\qquad y^{\text{lin}}=\frac{\sum_t p_t^2}{\sum_t p_t}
ptp_t
frame-level probability of one class at frame tt (decision-level pooling); pooling can also be applied to embeddings before the classifier (feature-level)
yliny^{\text{lin}}
linear softmax pooling: each frame is weighted by its own probability. Wang, Li and Metze (ICASSP 2019) compared five pooling functions for weakly labelled sound event detection and found it best at localising events

Max pooling says "present if any frame is confident", so one false frame causes a false tag. Mean pooling says "present on average", which dilutes a 100 ms bark in a 10 s clip. Attention pooling learns the weights instead. Frame-level outputs also give rough timestamps, which turns a tagger into a weakly supervised sound event detector.

APc=1∣Pc∣∑k: clipk∈PcPrecision@k,mAP=1C∑cAPc\mathrm{AP}_c=\frac{1}{|P_c|}\sum_{k:\,\text{clip}_k\in P_c}\mathrm{Precision@}k,\qquad \mathrm{mAP}=\frac1C\sum_c \mathrm{AP}_c
PcP_c
clips that truly contain class cc; clips are ranked by y^c\hat y_c and precision is read at the rank of each positive
mAP\mathrm{mAP}
threshold-free, and every class counts equally however rare it is. It is the standard AudioSet metric (see Precision, recall & ROC)

Datasets and results

AudioSet
about 2.1M human-labelled 10 s YouTube clips, 527 classes in the released labels (632 in the ontology); balanced train ≈ 22k clips; heavily imbalanced (Speech and Music dominate)
ESC-50
2,000 environmental clips of 5 s, 50 classes × 40, 5 folds (Piczak, 2015); human accuracy 81.3%
UrbanSound8K
8,732 urban clips of up to 4 s, 10 classes, 10 folds (Salamon et al., 2014)
Speech Commands v2
105,829 one-second 16 kHz utterances of 35 words (Warden, 2018); the keyword-spotting benchmark
AudioSet mAP
PANNs CNN14 0.431, Wavegram-Logmel-CNN 0.439; AST 0.459 single model, 0.485 ensemble
AST transfer
ESC-50 95.6% accuracy (AudioSet-pretrained); Speech Commands v2 98.1%

SpecAugment

SpecAugment (Park et al., Interspeech 2019) augments the spectrogram itself. It applies a time warp, then mFm_F frequency masks of width f∼U[0,F]f\sim U[0,F] bands and mTm_T time masks of width t∼U[0,T]t\sim U[0,T] frames, filled with the mean. It was designed for speech recognition: a LAS model with LM fusion reached 2.5%/5.8% WER on LibriSpeech test-clean/test-other. It is now standard for audio tagging too, usually with mixup (see Data augmentation). Masks force the model not to rely on any single band or moment.

test clip 5/12 · truth: siren + rain · 1 s → 40 mel × 61 frames02k8ksirenbarkrainbird1.50per-frame sigmoid p_t(class) from a logistic layer on each standardised log-mel frameclip score (max)siren0.99 ✓bark0.15 ✓rain1.00 ✓bird0.26 ✓| = threshold 0.5✓/✗ vs the clip labelsmAP over 12 test clips × 4 classes (2 freq + 2 time masks each):max 0.939· mean 0.932· linear softmax 0.942tag accuracy at 0.5: max 83.3%· mean 66.7%· linear softmax 87.5%max pooling trusts a single frame: one confident false frame is enough for a false tag
test clip
pooling
SpecAugment
mask widths are drawn uniformly from [0, F] bands and [0, T] frames; masked cells are set to 0, the standardised mean
fig.A real multi-label classifier on synthetic 1 s clips of four classes (siren, bark, rain, bird): 40-band log-mel, a logistic layer per class on each frame, and pooling to a clip score. Defaults (clip 5, siren + rain, 2 + 2 masks, F = 8, T = 12): mAP over the 12 test clips is 0.939 with max, 0.932 with mean and 0.942 with linear softmax. Tag accuracy at 0.5 is 83.3%, 66.7% and 87.5%. With masks off, all three reach mAP 1.000, but accuracy stays at 91.7%, 75.0% and 93.8%: mean pooling ranks correctly yet scores short events below 0.5. On clip 2 (bark only) without masks, max tags a false siren (0.57), mean misses the bark (0.40), and linear softmax gets both right (bark 0.90, siren 0.24).
QuantityComputationValue
frame probabilities0.05, 0.1, 0.9, 0.95, 0.2, 0.05, 0.05, 0.1, 0.05, 0.05Σp = 2.5, Σp² = 1.785
max poolingmax p_t0.95
mean pooling2.5 / 100.25 (below 0.5: missed)
linear softmax1.785 / 2.50.714
clip scores (truth)0.92 (✓), 0.81 (✗), 0.77 (✓), 0.40 (✓), 0.35 (✗), 0.10 (✗)positives at ranks 1, 3, 4
AP(1/1 + 2/3 + 3/4) / 30.8056
Worked example: one class, 10 frames, a 2-frame event; then AP over 6 clips
python
1import numpy as np
2rng = np.random.default_rng(0)
3
4def spec_augment(x, F=27, n_freq=2, T=100, n_time=2):
5 """x: (n_mels, frames) normalised log-mel. Masks are set to 0 (the mean)."""
6 x = x.copy()
7 n_mels, frames = x.shape
8 for _ in range(n_freq):
9 f = rng.integers(0, F + 1); f0 = rng.integers(0, n_mels - f + 1)
10 x[f0:f0 + f] = 0
11 for _ in range(n_time):
12 t = rng.integers(0, T + 1); t0 = rng.integers(0, frames - t + 1)
13 x[:, t0:t0 + t] = 0
14 return x
15
16def average_precision(scores, labels):
17 order = np.argsort(-scores)
18 hits = np.cumsum(labels[order])
19 prec = hits / np.arange(1, len(scores) + 1)
20 return (prec * labels[order]).sum() / labels.sum()
21
22x = spec_augment(rng.standard_normal((128, 1024)))
23print((x == 0).mean().round(3)) # 0.344 of cells masked
24
25frame_p = np.array([0.05, 0.1, 0.9, 0.95, 0.2, 0.05, 0.05, 0.1, 0.05, 0.05])
26print(frame_p.max(), frame_p.mean(), (frame_p**2).sum() / frame_p.sum()) # 0.95 0.25 0.714
27
28scores = np.array([0.92, 0.81, 0.77, 0.40, 0.35, 0.10])
29labels = np.array([1, 0, 1, 1, 0, 0])
30print(average_precision(scores, labels)) # 0.8056; mAP = mean over classes
SpecAugment masks, three poolings and average precision in numpy (run with numpy 2.5). A pretrained tagger is one call away in Hugging Face, e.g. pipeline("audio-classification", model="MIT/ast-finetuned-audioset-10-10-0.4593"), not run here.

Use when

  • +CNN (PANNs): cheap, works on any clip length, good on small data thanks to locality
  • +AST: global attention over the whole clip from layer 1; best AudioSet mAP of the two
  • +Both: a pretrained AudioSet model gives strong embeddings for other tasks

Avoid when

  • −CNN: receptive field grows only with depth; pooling choice matters
  • −AST: quadratic cost in the 1212 patches; depends on ImageNet initialisation
  • −Both: weak, noisy, imbalanced labels limit what any architecture can learn

Where it's used

Keyword spotting

"Hey …" wake words on phones and speakers: small CNNs on 40-band log-mels or MFCCs over 1 s windows, trained on Speech Commands-style data

Sound event detection

safety and monitoring (glass break, alarms, gunshots), wildlife and bird surveys, machine-fault detection

Media tagging

content search, music and speech detection, automatic captions of non-speech sounds

Check yourself

Why does AudioSet tagging use one sigmoid per class rather than a softmax over 527 classes?

Check yourself

A 10 s clip contains one 0.2 s bark, and the frame model outputs about 0.95 on those frames and 0.02 elsewhere. Which pooling gives the most useful clip score?

Remember

  • ✓Audio classification = log-mel image + CNN (PANNs CNN14, mAP 0.431) or patch transformer (AST, 0.459 single / 0.485 ensemble) on AudioSet.
  • ✓Multi-label: one sigmoid per class with binary cross-entropy; softmax only for exclusive classes (ESC-50, keywords).
  • ✓Frame scores are pooled to the clip: max, mean, attention or linear softmax ∑p2/∑p\sum p^2/\sum p; weak labels make this choice matter.
  • ✓SpecAugment masks F bands and T frames (plus time warp) during training; mixup is the other standard augmentation.
  • ✓Evaluate with mAP (threshold-free, class-balanced); pretrain on AudioSet and fine-tune on small sets like ESC-50.
← back to Deep Learning roadmap