PolySinger: Singing-Voice to Singing-Voice Translation From English to Japanese

Abstract

The speech domain prevails in the spotlight for several natural language processing (NLP) tasks while the singing domain remains less explored. The culmination of NLP is the speech-to-speech translation (S2ST) task, referring to translation and synthesis of human speech. A disparity between S2ST and the possible adaptation to the singing domain, which we describe as singing-voice to singing-voice translation (SVST), is becoming prominent as the former is progressing ever faster while the latter is at a standstill. Singing voice synthesis (SVS) and singing voice conversion (SVC) are related tasks, but despite limited attention has been paid to multilingual singing writing and song translation. This paper endeavors to determine what is required for successful SVSVT and proposes PolySinger (Polyglot Singer) as the first system for SVSVT, performing lyric translation from English to Japanese. A cascaded approach is proposed to establish a framework with a high degree of control which can potentially diminish the disparity between SVSVT and S2ST. The performance of PolySinger is evaluated by a mean opinion score test with native Japanese speakers. Results and in-depth discussions with test subjects reveal that while the foundation for SVSVT has been laid, several shortcomings must be overcome, which are discussed for the future of SVSVT.

System Architecture

Key Technologies

Evaluation Results

Mean Opinion Score (MOS) test conducted with 6 native Japanese speakers. The participants were asked to evaluate the quality of the synthesized Japanese vocal performances on a 5 point scale. The same songs were tested with both a baseline and a fine-tuned translation model.

Audio Samples and Lyrics

BibTeX

@INPROCEEDINGS{PolySinger,
    AUTHOR = {Silas Antonisen and Iván López-Espejo},
    TITLE = {{PolySinger: Singing-Voice to Singing-Voice Translation from English to Japanese}},
    BOOKTITLE = {Proceedings of the 25th International Society for Music Information Retrieval (ISMIR) Conference},
    YEAR = {2024},
    ADDRESS = {San Francisco, CA, USA}
    }