Definition
Audio classification assigns one or more labels to a clip of sound: an event ("dog bark", "siren"), a scene, a spoken keyword, an instrument. Most modern systems treat the log-mel spectrogram as an image. Key models are PANNs (Kong et al., "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition", IEEE/ACM TASLP 2020) and the Audio Spectrogram Transformer (AST; Gong, Chung and Glass, Interspeech 2021). Both were trained on AudioSet (Gemmeke et al., ICASSP 2017).
The input is 1–10 s of audio; the output is a score per class. Real clips overlap several sources, so the task is usually multi-label: a street clip can be "car", "speech" and "siren" at once. Labels are often weak: AudioSet says a 10 s clip contains a dog bark, not when. Events can be 100 ms long inside a 10 s clip, and the same class sounds very different across recordings.
Before deep learning, systems pooled MFCCs or other hand-made features over the clip and fed them to GMMs or SVMs. CNNs on log-mel images took over once large labelled sets appeared. Transformers followed once ImageNet-pretrained Vision Transformers (ViT) could be adapted to spectrograms.
Two architectures
PANNs CNN14 is a VGG-like CNN (see CNN architecture) with 14 weight layers: 12 convolutions and 2 fully connected. Its input is 64 log-mel bands at 32 kHz with a 1024-sample window and a 320-sample hop, so 10 s gives 1001 frames. After the last block it averages over frequency, then sums max- and mean-pooling over time to get one 2048-d clip embedding. The PANNs paper also released the pretrained weights, which became a standard feature extractor and starting point for fine-tuning.
AST has no convolutions. A 128 × 1024 log-mel (10 s at a 10 ms hop) is split into 16 × 16 patches with stride 10, so neighbours overlap by 6. That gives patches, each linearly projected to 768-d with a learned position embedding. A 12-layer transformer encoder runs over them (see Self-attention), and a linear head reads the CLS token. Initialising from an ImageNet-trained vision transformer matters: the position embeddings are adapted from the 2-D image grid.
Shapes for one 10 s AudioSet clip
- 1PANNs: samples, hop 320 → hops + 1 = 1001 frames × 64 bands.
- 2AST: 16 kHz, 10 ms shift → 1024 frames (padded or cut) × 128 bands.
- 3AST patches: along frequency, along time → patches.
- 4Plus one CLS token → a sequence for each of the 12 layers; attention cost grows with M pairs.
- probability that class is present; one independent sigmoid per class, so scores need not sum to 1
- pooled clip embedding (CNN14: 2048-d; AST: the 768-d CLS output)
- Binary cross-entropy averaged over classes ( for AudioSet); softmax + cross-entropy is used only when classes are mutually exclusive, e.g. ESC-50 or keyword spotting
From frames to a clip score
- frame-level probability of one class at frame (decision-level pooling); pooling can also be applied to embeddings before the classifier (feature-level)
- linear softmax pooling: each frame is weighted by its own probability. Wang, Li and Metze (ICASSP 2019) compared five pooling functions for weakly labelled sound event detection and found it best at localising events
Max pooling says "present if any frame is confident", so one false frame causes a false tag. Mean pooling says "present on average", which dilutes a 100 ms bark in a 10 s clip. Attention pooling learns the weights instead. Frame-level outputs also give rough timestamps, which turns a tagger into a weakly supervised sound event detector.
- clips that truly contain class ; clips are ranked by and precision is read at the rank of each positive
- threshold-free, and every class counts equally however rare it is. It is the standard AudioSet metric (see Precision, recall & ROC)
Datasets and results
- AudioSet
- about 2.1M human-labelled 10 s YouTube clips, 527 classes in the released labels (632 in the ontology); balanced train ≈ 22k clips; heavily imbalanced (Speech and Music dominate)
- ESC-50
- 2,000 environmental clips of 5 s, 50 classes × 40, 5 folds (Piczak, 2015); human accuracy 81.3%
- UrbanSound8K
- 8,732 urban clips of up to 4 s, 10 classes, 10 folds (Salamon et al., 2014)
- Speech Commands v2
- 105,829 one-second 16 kHz utterances of 35 words (Warden, 2018); the keyword-spotting benchmark
- AudioSet mAP
- PANNs CNN14 0.431, Wavegram-Logmel-CNN 0.439; AST 0.459 single model, 0.485 ensemble
- AST transfer
- ESC-50 95.6% accuracy (AudioSet-pretrained); Speech Commands v2 98.1%
SpecAugment
SpecAugment (Park et al., Interspeech 2019) augments the spectrogram itself. It applies a time warp, then frequency masks of width bands and time masks of width frames, filled with the mean. It was designed for speech recognition: a LAS model with LM fusion reached 2.5%/5.8% WER on LibriSpeech test-clean/test-other. It is now standard for audio tagging too, usually with mixup (see Data augmentation). Masks force the model not to rely on any single band or moment.
| Quantity | Computation | Value |
|---|---|---|
| frame probabilities | 0.05, 0.1, 0.9, 0.95, 0.2, 0.05, 0.05, 0.1, 0.05, 0.05 | Σp = 2.5, Σp² = 1.785 |
| max pooling | max p_t | 0.95 |
| mean pooling | 2.5 / 10 | 0.25 (below 0.5: missed) |
| linear softmax | 1.785 / 2.5 | 0.714 |
| clip scores (truth) | 0.92 (✓), 0.81 (✗), 0.77 (✓), 0.40 (✓), 0.35 (✗), 0.10 (✗) | positives at ranks 1, 3, 4 |
| AP | (1/1 + 2/3 + 3/4) / 3 | 0.8056 |
1import numpy as np2rng = np.random.default_rng(0)34def spec_augment(x, F=27, n_freq=2, T=100, n_time=2):5 """x: (n_mels, frames) normalised log-mel. Masks are set to 0 (the mean)."""6 x = x.copy()7 n_mels, frames = x.shape8 for _ in range(n_freq):9 f = rng.integers(0, F + 1); f0 = rng.integers(0, n_mels - f + 1)10 x[f0:f0 + f] = 011 for _ in range(n_time):12 t = rng.integers(0, T + 1); t0 = rng.integers(0, frames - t + 1)13 x[:, t0:t0 + t] = 014 return x1516def average_precision(scores, labels):17 order = np.argsort(-scores)18 hits = np.cumsum(labels[order])19 prec = hits / np.arange(1, len(scores) + 1)20 return (prec * labels[order]).sum() / labels.sum()2122x = spec_augment(rng.standard_normal((128, 1024)))23print((x == 0).mean().round(3)) # 0.344 of cells masked2425frame_p = np.array([0.05, 0.1, 0.9, 0.95, 0.2, 0.05, 0.05, 0.1, 0.05, 0.05])26print(frame_p.max(), frame_p.mean(), (frame_p**2).sum() / frame_p.sum()) # 0.95 0.25 0.7142728scores = np.array([0.92, 0.81, 0.77, 0.40, 0.35, 0.10])29labels = np.array([1, 0, 1, 1, 0, 0])30print(average_precision(scores, labels)) # 0.8056; mAP = mean over classesUse when
- +CNN (PANNs): cheap, works on any clip length, good on small data thanks to locality
- +AST: global attention over the whole clip from layer 1; best AudioSet mAP of the two
- +Both: a pretrained AudioSet model gives strong embeddings for other tasks
Avoid when
- −CNN: receptive field grows only with depth; pooling choice matters
- −AST: quadratic cost in the 1212 patches; depends on ImageNet initialisation
- −Both: weak, noisy, imbalanced labels limit what any architecture can learn
Where it's used
Keyword spotting
"Hey …" wake words on phones and speakers: small CNNs on 40-band log-mels or MFCCs over 1 s windows, trained on Speech Commands-style data
Sound event detection
safety and monitoring (glass break, alarms, gunshots), wildlife and bird surveys, machine-fault detection
Media tagging
content search, music and speech detection, automatic captions of non-speech sounds
Check yourself
Why does AudioSet tagging use one sigmoid per class rather than a softmax over 527 classes?
Check yourself
A 10 s clip contains one 0.2 s bark, and the frame model outputs about 0.95 on those frames and 0.02 elsewhere. Which pooling gives the most useful clip score?
Remember
- ✓Audio classification = log-mel image + CNN (PANNs CNN14, mAP 0.431) or patch transformer (AST, 0.459 single / 0.485 ensemble) on AudioSet.
- ✓Multi-label: one sigmoid per class with binary cross-entropy; softmax only for exclusive classes (ESC-50, keywords).
- ✓Frame scores are pooled to the clip: max, mean, attention or linear softmax ; weak labels make this choice matter.
- ✓SpecAugment masks F bands and T frames (plus time warp) during training; mixup is the other standard augmentation.
- ✓Evaluate with mAP (threshold-free, class-balanced); pretrain on AudioSet and fine-tune on small sets like ESC-50.