Audio generation has seen significant progress, yet synthesizing unified audio where speech and environmental sounds are naturally composited remains a challenge. Current methods either rely on disjoint pipelines that fail to capture fine-grained interactions or require structured inputs and external text rewriting, limiting the flexibility of free-form text prompts. In this paper, we introduce a new task: Free-Form-Text-Prompt-to-Unified-Audio generation, which aims to directly synthesize unified audio containing speech, sound, and their composites from unconstrained natural language. To address this task, we propose PlanAudio, an encoder-free, autoregressive LLM-based framework. PlanAudio leverages the intrinsic language understanding of LLMs and introduces a Semantic latent chain-of-thought (Latent CoT) mechanism. This mechanism enables the model to perform implicit semantic planning in a continuous latent space before audio generation, effectively bridging high-level semantic understanding and low-level acoustic synthesis. Furthermore, we create PlanAudio-Bench, a specialized benchmark for evaluating composite audio scenarios. We conduct evaluation across the scenarios of speech, sound, and their compositions. Results demonstrate that PlanAudio outperforms existing pipeline and unified baselines. Our analysis further reveals the superiority of Semantic Latent CoT over other CoT mechanisms and highlights the importance of continuous multi-scenario training curricula.
Architecture
Samples in composite scenario
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
A violin plays a short melody, followed by a man speaking: "Hopefully you all know that. But if you just slide your hand up into second position. This is something I do all the time but I never thought about teaching it this way.". The violin then plays another short melody.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
A hip-hop beat plays as a man raps: "With the letters on one slot devil, it's magic. How I keep it clean, no static. I'm Mike Maverick. Sean's bright, prismatic. I rock beats. Indeed, I got what you need.".
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
Music plays as a man begins to speak: "Hello, my name is Dr. Ian Dunbar. Most dog training programs have a single host, even though there are nearly as many ways to train dogs as there are dog trainers.".
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
A man speaks, "some interesting riffs. It's starting on the fifth fret on the E string. And when you play the note, you go low.", as a bass guitar plays a single note, followed by another note that slides down in pitch.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
A man speaks, saying "And the last tune is called The Hurricane. The Hurricane.", followed by the sound of a violin playing.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
Music plays as a man says: "It has a front camera with wide angle which is 40% wider than normal one and you can record high dimension videos in screen size of 1080 pixels 30 frames per second".
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
Upbeat music plays as a man speaks: "First and ever, the only dryer that is touch. Touch the screen, slide for speed. Touch the screen, slide for temperature. And touch the screen,".
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
An electric guitar plays with a clean tone, then a man says: "Okay, so I have everything set to flat and I have my gain at 9 o'clock. Here's what the overdrive channel sounds like.", after which the guitar plays again with a distorted effect.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2 (Speech+Sound)
Rhythmic electronic music plays, then stops, and a man says: "So, let's get started. I used to native instruments massive and".
Samples in sound scenario
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
Man speaking continuously with hissing in the background.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
A woman laughs with some wind blowing and sheep bleating.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
A car engine idles and then the horn blows.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
A machine works while something is rasping.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
A loud burst followed by laughter.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
Food frying in a pan.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
A jackhammer operating.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
A man is speaking as birds are chirping.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
AudioLDM2
A flock of sheep baaing.
Samples in speech scenario
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
A woman speaks, "He summoned his engineer.", with no noise in the background.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
A man is speaking: "He was quiet, and I might almost say silent, during the whole evening; but I could see from the testimony of his altered countenance how strong is the passion for life that dwells in the human breast.".
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
"He had broken a thickness of ice, the formation of many a winter; had had his reasons for a long silence.", a woman says.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
A woman speaks: "The leisure rendered by the wife in such cases is, of course, not a simple manifestation of idleness or indolence.", there's no other sounds in the background.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
A woman speaks: "The Fairview band was engaged to discourse as much harmony as it could produce, and the resources of the great house were taxed to entertain the guests.", there's no other sounds in the background.
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
A woman speaks, saying "He looked at us in a queer way.".
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
A male says: "All passed so quickly, there was so much going on around him, the Tree quite forgot to look to himself.".
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
In a quite room, a man begins to speak: "Paul takes pride in his ministry, not to his own praise but to the praise of God.".
Prompt
Ours
VoiceLDM-s
VoiceLDM-m
PromptTTS++
An adult male speaks in a silent background, saying: "With such a radical opposition of sentiment, both factions were on the alert to seize every available vantage ground."
