Search

MusicGen

๋‚ ์งœ
2023/06/08
Blog/HuggingFace
๊นƒํ—™
๋ฆฌ๋ทฐ์–ด
ํ•™ํšŒ

Simple and Controllable Music Generation

TL;DR

โ€ข
Conditional Music Generation
โ€ข
MusicGen
โ—ฆ
(Simple) single-stage transformer LM
โ–ช
์ด์ „ ์—ฐ๊ตฌ๋“ค๊ณผ ๋‹ฌ๋ฆฌ, cascading several models ํ•„์š” ์—†์Œ
โ–ช
hierarchically or upsampling
โ—ฆ
(Controllable) conditioned on textual description or melodic features
โ€ข
Contribution
โ—ฆ
simple and efficient model to generate high quality music at 32 kHz
1.
generate consistent music with a single-stage language model through an efficient codebook interleaving strategy
2.
ํ…์ŠคํŠธ์™€ ๋ฉœ๋กœ๋””๋ฅผ condition์œผ๋กœ ๋ฐ›์•„ ์ƒ์„ฑํ•˜๋Š” ๋‹จ์ผ ๋ชจ๋ธ ์ œ์‹œ, ์ผ๊ด€๋˜๊ณ  condition์— ์ถฉ์‹คํ•จ ์ž…์ฆ
3.
๊ด‘๋ฒ”์œ„ํ•œ objective and human evaluations ์ œ๊ณต

Motivation (or problem to solve)

โ€ข
์Œ์•… ์ƒ์„ฑ์˜ ์–ด๋ ค์›€
โ—ฆ
modeling long range sequences
โ–ช
์Šคํ”ผ์น˜ : 16kHz
โ–ช
์Œ์•… : full frequency spectrum / 44.1kHz or 48kHz
โ—ฆ
์—ฌ๋Ÿฌ ์•…๊ธฐ ์‚ฌ์šฉ โ†’ complex structures
โ—ฆ
์‚ฌ๋žŒ๋“ค์€ disharmony์— ์˜ˆ๋ฏผํ•จ
โ€ข
์Œ์•… ์ œ์ž‘์ž
โ—ฆ
๊ฑด๋ฐ˜, ์•…๊ธฐ, ๋ฉœ๋กœ๋””, ์žฅ๋ฅด ๋“ฑ ๋‹ค์–‘ํ•œ ๋ฐฉ๋ฒ•์œผ๋กœ ์ƒ์„ฑ ๊ณผ์ •์„ ์ œ์–ดํ•  ์ˆ˜ ์žˆ๋Š” ๊ธฐ๋Šฅ ํ•„์ˆ˜์ 
โ€ข
๊ธฐ์กด ๋ชจ๋ธ
โ—ฆ
multi-streams : high cost

Proposed Method

โ€ข
MUSICGEN consists in an autoregressive transformer-based decoder
โ—ฆ
The (language) model is over the quantized units from an EnCodec audio tokenizer
โ—ฆ
EnCodec : SoundStream์—์„œ ์ตœ์ข… Conv1D ์ง์ „์— LSTM ์ถ”๊ฐ€
SoundStream
EnCodec

1. Audio tokenization

โ€ข
RVQ, EnCodec ์‚ฌ์šฉ โ†’ RVQ์—์„œ ๊ฐ quantizer๋Š” ์ด์ „ quantizer๊ฐ€ ๋‚จ๊ธด ์–‘์žํ™” ์˜ค์ฐจ๋ฅผ ์ธ์ฝ”๋”ฉํ•˜๋ฏ€๋กœ ์„œ๋กœ ๋‹ค๋ฅธ ์ฝ”๋“œ๋ถ์— ๋Œ€ํ•œ ์–‘์žํ™” ๊ฐ’์€ ์ผ๋ฐ˜์ ์œผ๋กœ ๋…๋ฆฝ์ ์ด์ง€ ์•Š์œผ๋ฉฐ ์ฒซ ๋ฒˆ์งธ ์ฝ”๋“œ๋ถ์ด ๊ฐ€์žฅ ์ค‘์š”

2. Codebook interleaving patterns

โ€ข
EnCodec ๋ชจ๋ธ์—์„œ ์–ป์€ ํ‘œํ˜„ย Q์˜ ์ฃผ์š” ๋ฌธ์ œ๋Š” ๊ฐ timestep์— ๋Œ€ํ•ดย K๊ฐœ์˜ ์ฝ”๋“œ๋ถ์ด ์žˆ๋‹ค(?)

3. Model conditioning

โ€ข
Text conditoning
1.
T5 encoder: ์‚ฌ์ „ ํ•™์Šต๋œ ํ…์ŠคํŠธ ์ธ์ฝ”๋”
2.
FLAN-T5 : instruct-based ์–ธ์–ด ๋ชจ๋ธ
3.
CLAP : joint text-audio representation, provides better-quality generations
โ€ข
Melody conditioning
โ—ฆ
์Œ์•…์— ๋Œ€ํ•ด text๋ณด๋‹ค ์ž์—ฐ์Šค๋Ÿฌ์šด ์ ‘๊ทผ ๋ฐฉ์‹์€ ๋‹ค๋ฅธ ์˜ค๋””์˜ค ํŠธ๋ž™์˜ ๋ฉœ๋กœ๋”” ๊ตฌ์กฐ, ํœ˜ํŒŒ๋žŒ์ด๋‚˜ ํฅ์–ผ๊ฑฐ๋ฆผ
โ—ฆ
conditioning on the raw chromagram often led to reconstructing the original sample, resulting in overfitting
โ—ฆ
information bottleneck by choosing the dominant time-frequency bin in each time step

4. Model architecture

โ€ข

Experiments

CC3M caption

โ€ข
olympic athlete is seen during the men 's handball preliminary group b match
โ€ข
tiger in front of wild mountain
โ€ข
a_man_walks_alone_on_a_dock_in_a_mountain_lake
โ€ข
3d_vector_deluxe_alphabet_of_randomly_rotated_thin_golden_symbols