Kyutai and Mirelo have released MuScriptor, an open-weight model designed to transcribe multi-instrument audio mixes into MIDI. The tool addresses a persistent bottleneck in automatic music transcription where separating individual instruments from a full mix remains difficult.
What is MuScriptor?
The model functions as a decoder-only Transformer. It takes a mel-spectrogram of a short audio clip and predicts MIDI-like tokens for pitch, timing, and instrument identity. This approach treats transcription as a language-modelling task, using the same tokenization scheme as MT3.
The release provides three weight variants on Hugging Face. The small version contains 103M parameters, the medium version holds 307M parameters, and the large version reaches 1.4B parameters. The inference code operates under the MIT license. The model weights are licensed under CC BY-NC 4.0, which prohibits commercial use.
The Three-Stage Training Pipeline
The development process relies on data quality rather than architectural novelty. Training proceeds through three distinct stages, with each phase building on the previous one.
- Pre-training uses D<sub>Synth</sub>, a dataset of roughly 1.45M MIDI files. An on-the-fly pipeline synthesizes these files during training. Augmentations include pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts combined with random detuning produce near-infinite audio realizations.
- Fine-tuning uses D<sub>Real</sub>, an internal set of 170,000 recordings. This brings the total to more than 11,000 hours of audio with aligned note annotations. Most alignments derive from audio-symbolic synchronization using interpolation and dynamic time warping. Poor pairs are filtered based on warping distance and a maximum time-dilation factor.
- Reinforcement learning post-training uses D<sub>RL</sub>, consisting of 300 manually verified tracks. The team applies a GRPO-like method combining REINFORCE with group-relative advantage normalization. The reward function sums three F-scores: onset, frame, and offset. This forces the model to prioritise cleaner transcriptions.
Performance Benchmarks
The model was evaluated against existing baselines on multi-instrument datasets. Results indicate a consistent improvement in recall and precision across different instrument types.
- Recall: The model achieves an average recall of 84.5% on the MusicNet test set, outperforming previous state-of-the-art methods by roughly 4 percentage points.
- Precision: Frame-level precision reaches 78.2%, showing that the model effectively reduces the number of false positive note detections.
- Instrument Separation: On the MIREX 2024 dataset, MuScriptor correctly identified the onset of the secondary instrument in 62% of complex mix scenarios, a figure previously considered unattainable for decoder-only architectures.
What it means
For producers and arrangers, this tool offers a practical alternative to manual MIDI entry. Previously, isolating a specific instrument from a dense mix required hours of manual editing or expensive commercial software. MuScriptor provides a free, open-source option that can generate a usable MIDI skeleton in seconds. While the output still requires human refinement for professional use, it significantly lowers the barrier for sketching ideas or backing tracks.
