Acoustical Society of Japan (ASJ) Student and Young Forum Advent Calendar 2023 Day 24

Suno AI is a generative AI service that automatically produces full songs with vocals simply by providing lyrics, musical style descriptors, and a track title.

Suno AI

While various music generation models have been released recently, Suno AI went especially viral because of its unique lyric-driven interface and the immense entertainment value of generating expressive vocal tracks.

Rather than discussing how to use Suno AI (which is covered across many blogs), this article explains how to build a system like Suno AI from an engineering and machine learning perspective.

My personal technical takeaways regarding Suno AI:

• Building a system like Suno AI is entirely possible by combining currently published technologies and open architectures.
• However, the main bottlenecks lie in large-scale training data curation, legal rights, and ethical considerations.

Specifications of Suno AI

The input/output specifications of Suno AI are straightforward:

• Input: 1. Lyrics (text), 2. Musical style prompt (text), 3. Song title (text)
• Output: Stereo audio track (48 kHz)

All inputs are text prompts. (Whether the song title is actually used during conditioned generation remains unclear.)

Recent Trends in Music Generation AI

Most recent music generation models share a common architectural framework:

Generative models learn far more effectively when the representation size is compact. Raw audio waveforms (especially complex polyphonic music) are massive in dimension and temporally dense compared to images.

Modern audio generation frameworks therefore adopt a two-stage paradigm: Compression Model + Generative Model:

1. First, an Audio Compression Model (neural audio codec) is trained to compress high-rate raw audio into compact discrete acoustic tokens.
2. Next, a Generative Model (autoregressive Transformer or diffusion model) is trained on these compact discrete tokens conditioned on text.
3. At inference time, the generative model outputs token sequences, and the neural codec decoder reconstructs high-fidelity audio from them.

(This mirrors the latent diffusion architecture used in image synthesis like Stable Diffusion.)

What Model Architecture Does Suno AI Likely Use?

Since Suno’s proprietary code is not public, let’s examine the generated signal characteristics:

Generated songs are downloaded as 128 kbps MP3 files at 48 kHz. Looking at the spectrogram, frequencies above 16 kHz are cut off. A 16 kHz acoustic bandwidth strongly points toward an internal sampling rate of 32 kHz (Nyquist frequency: 16 kHz).

In 32 kHz neural audio generation, Meta’s open-source MusicGen (generative model) and EnCodec 32kHz (neural compression codec) are prominent reference architectures:

• MusicGen (Generative Model)
• EnCodec (Neural Audio Codec)

Furthermore, Suno previously released Bark, a general audio generation model based on GPT-style Transformers that explicitly uses EnCodec tokens as its audio representation:

Bark

This strongly suggests that Suno AI builds upon a Transformer-based sequence generation architecture conditioned on text, predicting hierarchical neural codec tokens like EnCodec.

How to Train Conditioned Vocal Music Generation

The standout capability of Suno AI is generating synchronized vocal singing along with instrumental backing tracks from lyrics. As demonstrated by OpenAI’s Jukebox in 2019, training a sequence model on paired audio and aligned lyric text allows the network to learn both instrumental composition and vocal articulation simultaneously:

OpenAI Jukebox

With modern neural codecs (like EnCodec) and scaled Transformer backbones (like MusicGen/Bark), generation fidelity and musical coherence improve dramatically over older 2019 architectures.

Constructing the Training Dataset

Training such a model requires three paired data modalities:

1. Audio recordings (music tracks)
2. Corresponding lyrics (text)
3. Style/Genre descriptions (text)

OpenAI’s Jukebox gathered ~1.2 million tracks with lyrics and metadata through web crawling:

(From OpenAI Jukebox paper)

Generating Style Annotations (Text)

When crawling massive audio datasets, detailed natural language descriptions (e.g. “upbeat 80s synth-pop with energetic drums”) are rarely present.

To solve this, modern pipelines leverage joint audio-text representation models like CLAP (Contrastive Language-Audio Pretraining). Models like MusicLDM use CLAP to extract rich semantic embeddings directly from raw audio, bypassing the need for manual style captions during training:

• MusicLDM
• CLAP

(From MusicLDM paper)

Extracting and Aligning Lyrics

For uncaptioned music, an automated pipeline can:

1. Isolate the vocal stem using state-of-the-art music source separation (e.g., HT Demucs).
2. Transcribe lyrics from the isolated vocal track using Automatic Speech/Lyrics Recognition models (e.g., Whisper).

Even if automated transcription contains occasional phonetic noise (e.g., 85–90% word accuracy), generative training with slightly noisy conditioning labels often regularizes models effectively without significantly hurting musical generation quality.

System Pipeline Architecture

Combining these components yields a complete pipeline for a Suno-like music generation system:

Bonus: Exploring Suno AI Behaviors

Copyright & Copying Protection

Suno AI incorporates safety filters that reject verbatim lyrics from famous commercial copyrighted songs:

Handling Short Lyrics and Hallucinations

When prompted with very brief lyrics, the generative model fills remaining song duration by generating pseudo-vocal vocables or gibberish that phonetically mimics singing:

Out-of-Distribution Lyric Experiments

Testing unusual lyrics (e.g., medical diagnoses like “complex bone fracture”):