[Paper 리뷰] Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer
Generative Pre-trained Speech Language Model with Efficient Hierarchical TransformerSpeech language model은 여전히 neural audio codec의 long acoustic sequence를 modeling 하는데 한계가 있음Generative Pre-trained Speech Transformer (GPST)Audio waveform을 2가지의 discrete speech representation으로 quantize 하고 hierarchical transformer architecture에 integrate 함End-to-End unsupervised manner로 train 됨으로써 다양한 speaker ident..
[Paper 리뷰] SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
SpeechX: Neural Codec Language Model as a Versatile Speech TransformerAudio-text prompt 기반의 speech model은 text-to-speech 외의 다양한 task를 처리하는 데는 한계가 있음SpeechXZero-shot Text-to-Speech, Speech Editing, Noise Suppression, Target Speaker Extraction 등의 다양한 task를 지원하는 speech modelNeural codec language modeling과 task-dependent prompting에 기반한 multi-task learning을 도입논문 (TASLP 2024) : Paper Link1. Introducti..
[Paper 리뷰] Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal SupervisionMinimal supervision으로 train 할 수 있는 multi-speaker text-to-speech model이 필요함SPEAR-TTSText to High level semantic token (Reading), Semantic token to Low-level acoustic token (Speaking)의 2가지 discrete speech representation을 combining 하여 text-to-speech를 sequence-to-sequence task로 casting특히 abundant audio-only data를 사용하여 Speak..
[Paper 리뷰] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the WildSpeech editing, zero-shot text-to-speech를 위해 token infilling neural codec language model을 구성할 수 있음VocieCraftTransformer decoder architecture와 causal masking, delayed stacking을 결합하여 existing sequence 내에서 generation을 수행하는 token rearrangement를 도입추가적으로 speech editing evaluation을 위한 RealEdit dataset을 제공논문 (ACL 2024) : Paper Link1. Int..
[Paper 리뷰] TacoLM: Gated Attention Equipped Codec Language Model are Efficient Zero-shot Text-to-Speech Synthesizers
TacoLM: Gated Attention Equipped Codec Language Model are Efficient Zero-shot Text to Speech SynthesizersNeual codec language model은 zero-shot text-to-speech에서 우수한 성능을 보이고 있음BUT, autoregressive nature와 text-audio 간의 implicit alignment로 인해 속도의 한계가 있음TacoLMTraining/inference 속도를 향상하고 model size를 줄이기 위해 gated attention mechanism을 도입추가적으로 각 decoder layer마다 gated cross-attention layer를 적용하여 합성 품질과 ef..
[Paper 리뷰] Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
Voicebox: Text-Guided Multilingual Universal Speech Generation at ScaleLarge-scale generative model은 고품질의 output을 생성할 수 있지만, scale과 task generalization 측면에서 한계가 있음Voicebox주어진 audio context와 text를 기반으로 speech를 infill 하도록 train 된 non-autoregressive flow-matching modelIn-context learning을 통해 cross-lingual zero-shot synthesis, noise removal, content editing, style conversion 등의 다양한 task를 지원논문 (NeurI..