Google Research has released MSEB, a Massive Sound Embedding Benchmark designed to test how well audio encoders perform across classification, clustering, retrieval, and segmentation. The project provides a full Python package, a set of abstract type definitions, and a suite of evaluators that run on a CPU without requiring large dataset downloads.
Installation and structure
The package installs via pip and exposes three distinct layers. The types module defines the data shapes for sound files, embeddings, scores, and task metadata. The encoder module contains the MultiModalEncoder abstract base class that any custom model must inherit from. The evaluators package holds specific modules for each task family.
Importing the library reveals the four evaluators used in this guide. These modules depend only on NumPy and scikit-learn. Other evaluators, such as those for reranking or transcription, require heavier dependencies like Whisper, TensorFlow, and Apache Beam. Consequently, the examples below run on a standard CPU without needing a GPU or external data.
Type definitions and validation
The benchmark enforces strict contracts at the data level. A Sound object holds a waveform array and a SoundContextParams dictionary. This context includes the identifier, sample rate, duration, language code, and optional text transcript. The SoundEmbedding object stores the vector array and timestamp pairs, along with statistics on input and embedding sizes.
Timestamps define the relationship between frames and utterances. If the count of timestamps matches the count of embeddings, the model is frame-aligned. A single timestamp indicates an utterance-level vector. The Score class validates metric names and ensures the minimum value does not exceed the maximum. It rejects malformed inputs immediately, preventing invalid numbers from reaching the leaderboard.
SR = 16000
def type_contract():
t = np.arange(SR) / SR
waveform = (0.5 * np.sin(2 * np.pi * 440 * t)).astype(np.float32)
sound = types.Sound(
waveform=waveform,
context=types.SoundContextParams(id="demo_000", sample_rate=SR, length=len(waveform),
language="en_us", text="a 440 Hz tone"),
)
print(f" Sound id={sound.context.id!r} {sound.waveform.shape} @ {sound.context.sample_rate} Hz"
f" -> {sound.size_bytes:,} bytes")
embedding = types.SoundEmbedding(
embedding=np.zeros((1, 16), dtype=np.float32), # (N, D): one utterance-level vector
timestamps=np.array([[0.0, 1.0]], dtype=np.float32), # (M, 2): [start, end] in seconds
context=sound.context,
encoding_stats=types.EncodingStats(input_size_bytes=sound.size_bytes, embedding_size_bytes=16 * 4),
)
print(f" SoundEmbedding embedding{embedding.embedding.shape} timestamps{embedding.timestamps.shape}"
f" -> {embedding.size_bytes} bytes")
print(f" compression_ratio = {embedding.encoding_stats.compression_ratio:.5f}"
f" ({1 / embedding.encoding_stats.compression_ratio:,.0f}x smaller than the audio)")
print(" N embeddings and M timestamps: M == N is frame-aligned, M == 1 is utterance-level.")
print(" `embedding` may also hold N strings instead of vectors - step 8 uses exactly that.")
score = types.Score(metric="Accuracy", description="Overall classification accuracy",
value=0.875, min=0.0, max=1.0)
print(f"\n Score {score.metric}={score.value} in [{score.min}, {score.max}] :: {score.description}")
for bad, why in [(dict(metric="", description="d", value=0.5, min=0.0, max=1.0), "empty metric name"),
(dict(metric="m", description="d", value=0.5, min=1.0, max=0.0), "min > max")]:
try:
types.Score(**bad)
except Exception as e:
print(f" rejected at construction ({why}): {type(e).__name__}: {e}")
return f"Sound {sound.size_bytes:,} B -> embedding {embedding.size_bytes} B"
type_contract()Implementing two different encoders
The framework requires models to inherit from MultiModalEncoder. This class defines _setup for loading weights and _encode for processing batches of sound objects. Two distinct implementations demonstrate how different acoustic features affect the benchmark results.
The first encoder, EnergyEnvelopeEncoder, calculates the average energy across equal time slices. It measures loudness and quietness but ignores timbre. The second encoder, SpectralProfileEncoder, computes the mean log-magnitude spectrum pooled into specific bands. This approach describes the timbre of the sound.
class EnergyEnvelopeEncoder(encoder_lib.MultiModalEncoder):
"""Baseline: average energy in `n_bins` equal time slices. Loud/quiet, nothing about timbre."""
def __init__(self, n_bins: int = 16):
super().__init__()
self.n_bins = n_bins
def _setup(self):
self._ready = True # a real encoder loads weights here
def _check_input_types(self, batch):
for item in batch:
if not isinstance(item, types.Sound):
raise ValueError(f"{type(self).__name__} takes types.Sound, got {type(item).__name__}")
def _encode(self, batch) -> list[types.SoundEmbedding]:
out = []
for sound in batch:
slices = np.array_split(sound.waveform.astype(np.float32), self.n_bins)
vec = np.array([[float(np.sqrt(np.mean(s ** 2) + 1e-12)) for s in slices]], dtype=np.float32)
vec /= np.linalg.norm(vec) + 1e-9
out.append(types.SoundEmbedding(
embedding=vec,
timestamps=np.array([[0.0, sound.context.length / sound.context.sample_rate]], dtype=np.float32),
context=sound.context))
return out
class SpectralProfileEncoder(encoder_lib.MultiModalEncoder):
"""Contender: mean log-magnitude spectrum pooled into `n_bands` bands. Describes timbre."""
def __init__(self, n_bands: int = 16, frame: int = 512):
super().__init__()
self.n_bands, self.frame = n_bands, frame
def _setup(self):
self._window = np.hanning(self.frame).astype(np.float32)
def _check_input_types(self, batch):
for item in batch:
if not isinstance(item, types.Sound):
raise ValueError(f"{type(self).__name__} takes types.Sound, got {type(item).__name__}")
def _encode(self, batch) -> list[types.SoundEmbedding]:
out = []
for sound in batch:
w = sound.waveform.astype(np.float32)
n_frames = max(1, len(w) // self.frame)
spectra = [np.abs(np.fft.rfft(w[i * self.frame:(i + 1) * self.frame] * self._window))
for i in range(n_frames)]
mean_spectrum = np.log1p(np.mean(spectra, axis=0))
vec = np.array([[float(b.mean()) for b in np.array_split(mean_spectrum, self.n_bands)]],
dtype=np.float32)
vec /= np.linalg.norm(vec) + 1e-9
out.append(types.SoundEmbedding(
embedding=vec,
timestamps=np.array([[0.0, sound.context.length / sound.context.sample_rate]], dtype=np.float32),Running the evaluators
Once the encoders are defined, the code drives the four main evaluators against the synthetic embeddings. The classification evaluator checks if the vectors separate distinct categories. The clustering evaluator tests how well the vectors group similar sounds together. The retrieval evaluator measures how quickly a system can find a specific sound within a database. The segmentation evaluator assesses the precision of time-bound predictions.
Calling the metric functions directly reveals what each evaluator rewards. The loudness encoder may perform well on tasks requiring volume detection but fail on timbre-based retrieval. Conversely, the spectral encoder excels where frequency content matters. The results show that the best model depends entirely on the specific task asked.
Final submission format
The benchmark concludes by assembling the TaskMetadata object required for a real submission. This structure bundles the encoder implementation, the evaluation results, and the performance scores into a single package. The output confirms that the two encoders trade places depending on which evaluator is active, proving the value of a multi-task benchmark through numbers rather than description.
What it means
Developers can now test audio models without downloading terabytes of data. The framework allows immediate comparison of different encoding strategies
