[Paper 리뷰] RFWave: Multi-Band Rectified Flow for Audio Waveform Reconstruction
RFWave: Multi-Band Rectified Flow for Audio Waveform ReconstructionDiffusion model은 waveform reconstruction에 효과적이지만 상당한 sampling step이 필요하므로 latency 문제가 존재함RFWaveComplex spectrogram을 생성하고 frame-level에서 모든 subband를 simultaneously process 함Straight transport trajectory를 위해 Rectified Flow를 도입논문 (ICLR 2025) : Paper Link1. IntroductionAudio waveform reconstruction은 raw audio data에서 derive 된 low-dimen..
[Paper 리뷰] PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation
PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform GenerationHigh-resolution waveform signal의 natural periodic feature를 explicitly disentangle 할 수 있는 generator가 필요함PeriodWaveVector field를 추정할 때 waveform signal의 periodic feature를 capture 하는 period-aware flow matching estimator를 도입Waveform signal의 periodic feature를 capture 하는 multi-period estimator를 활용추가적으로 waveform generation에서 hig..
[Paper 리뷰] DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific FactorsLarge-scale Latent Diffusion Model은 various modality에 대해 우수한 content generation 성능을 보여주고 있지만, text-to-speech에서는 phoneme, duration에 의존해야 하므로 scalability의 한계가 있음DiTTo-TTSDomain-specific factor를 제거한 Latent Diffusion Model 기반의 text-to-speech model기존 U-Net 대신 Diffusion Transformer를 채택하고 speech length predicto..
[Paper 리뷰] UniAudio: Towards Universal Audio Generation with Large Language Models
UniAudio: Towards Universal Audio Generation with Large Language Models다양한 task를 unified manner로 처리할 수 있는 universal audio generation model이 필요함UniAudioLarge Language Model-based audio generation model을 구성해 phoneme, text description, audio 등의 다양한 input condition을 기반으로 speech, sound, music, singing voice 등을 생성Model performance와 efficiency를 향상하기 위한 audio tokenization과 language model architecture를 설..
[Paper 리뷰] Whisper: Robust Speech Recognition via Large-Scale Weak Supervision
Whisper: Robust Speech Recognition via Large-Scale Weak SupervisionAudio transcripting을 위해 large data로 speech processing system을 training 할 수 있음Whisper680,000 hours의 multilingual, multitask supervision으로 model을 scalingDataset에 대한 specific fine-tuning 없이도 뛰어난 accuracy, robustness를 달성논문 (ICML 2023) : Paper Link1. IntroductionSpeech recognition은 wav2vec 2.0의 unsupervised pre-training technique을 주로 ..
[Paper 리뷰] MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec TransformerLarge-scale text-to-speech system은 autoregressive/non-autoregressive 방식으로 나눌 수 있음- Autoregressive 방식은 robustness와 duration controllability 측면에서 한계가 있음- Non-auotregressive 방식은 training 중에 text, speech 간의 explicit alignment information이 필요함MaskGCTText, speech supervision 간의 explicit alignment information과 phone-level duratio..
[Paper 리뷰] RepCodec: A Speech Representation Codec for Speech Tokenization
RepCodec: A Speech Representation Codec for Speech TokenizationDiscrete speech tokenization은 large language model에서 유용하게 활용되지만 discretization으로 인해 information loss가 발생함RepCodecSpeech encoder에서 speech representation을 reconstruction 하여 vector quantization codebook을 학습Speech encoder, Codec encoder, Vector quantization codebook으로 구성된 pipeline을 통해 speech waveform을 semantic token으로 변환논문 (ACL 2024) : P..
[Paper 리뷰] BnTTS: Few-Shot Speaker Adaptation in Low-Resource Setting
BnTTS: Few-Shot Adaptation in Low-Resource SettingLow-resource language에 대한 text-to-speech model이 필요함BnTTSXTTS architecture를 기반으로 하는 speaker adaptation-based text-to-speech modelLow-resource language의 phonetic, linguistic character를 반영하도록 multilingual pipeline에 integrate 함논문 (NAACL 2025) : Paper Link1. IntroductionText-to-Speech (TTS)에서 zero-shot, one-shot adaptation은 여전히 한계가 있음대표적인 zero-shot TT..
[Paper 리뷰] Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised DisentanglementVoice imitation은 annotated data에 크게 의존하고 timbre/style을 disentangle 하는데 어려움이 있음VevoContent-Style Modeling을 통해 text/speech content token을 input으로 하고 style reference로 prompt 되는 content-style token을 생성Acoustic Modeling을 통해 content-style token을 기반으로 flow-matching transformer를 사용해 timbre reference로 prompt 되는 acoustic repr..