MULTI-SPEAKER EXPRESSIVE SPEECH SYNTHESIS VIA MULTIPLE FACTORS
DECOUPLING
0. Contents
- Abstract
- Demos -- expressive speech with a specific style and emotion for target speakers
- Demos -- long expressive speech with a specific style and various emotions for target speakers
1. Abstract
This paper aims to synthesize target speaker's speech with desired speaking style and emotion by transferring the style and emotion from reference speech recorded by other speakers. Specifically, we address this challenging problem with a two-stage framework composed of a text-to-style-and-emotion (Text2SE) module and a style-and-emotion-to-wave (SE2Wave) module, bridging by neural bottleneck (BN) features. To further solve the multi-factor (speaker timbre, speaking style and emotion) decoupling problem, we adopt the multi-label binary vector (MBV) and mutual information (MI) minimization to respectively discretize the extracted embeddings and disentangle these highly entangled factors in both Text2SE and SE2Wave modules. Moreover, we introduce a semi-supervised training strategy to leverage data from multiple speakers, including emotion-labelled data, style-labelled data, and unlabeled data. To better transfer the fine-grained expressiveness from references to the target speaker in the non-parallel transfer, we introduce a reference-candidate pool and propose an attention based reference selection approach. Extensive experiments demonstrate the good design of our model.
2. Demos -- expressive speech with a specific style and emotion for target speakers
Convert the emotion and style expresssions from different source speakers to the target speakers without emotional and stylistic training data.
Target speaker: M1
| Emotion | Target style example | Target emotion example | Target speaker example | MR-Tacotron | Referee | Proposed |
|---|---|---|---|---|---|---|
| Poet + Happy | Text: 长风破浪会有时，直挂云帆济沧海。(English: Someday, with my sail piercing the clouds; I will mount the wind, break the waves, and traverse the vast, rolling sea.) | |||||
| Poet + Sad | Text: 君不见，高堂明镜悲白发，朝如青丝暮成雪。(English: Don't you see? sadly, the high hall mirror saw white hair, as black as green hair in the morning and as white as snow in the evening.) | |||||
| Poet + Angry | Text: 天生我材必有用，千金散尽还复来。(English: Heaven has made us talents, we're not made in vain. A thousand gold coins spent, more will turn up again.) | |||||
| Poet + Surprise | Text: 从此山河两相阅，缠尽青山尝清河。(English: From then on, the mountains and rivers read each other and wrapped around the green mountains to taste the Qinghe River.) | |||||
| Poet + Fear | Text: 山不厌高，海不厌深。周公吐哺，天下归心。(English: The water can never be too deep, and the mountain too high. The Duke of Zhou spits and feeds, and the world returns to the heart.) | |||||
| Fairytales + Happy | Text: 不久以后，王后果然生下了一个可爱的小公主。(English: Before long, the queen gave birth to a lovely little princess.) | |||||
| Fairytales + Sad | Text: 她对恶魔说，求求你，千万不要伤害公主，我什么都可以给你。(English: She said to the devil, Please, don't hurt the princess. I can give you anything.) | |||||
| Fairytales + Angry | Text: 给我杀了她！把她的心和舌头都带回来，做为你杀死她的证据。(English: Kill her! Bring back her heart and tongue as proof that you killed her.) | |||||
| Fairytales + Surprise | Text: 进入小木屋后，里面竟然整齐排列着七张小小的床。(English: After entering the cabin, there were seven small beds neatly arranged inside.) | |||||
| Fairytales + Fear | Text: 听到猫头鹰叫声的白雪公主，越走越觉得森林好可怕。(English: Snow White, who heard the cry of an owl, felt more and more terrible in the forest as she walked.) | |||||
| Novels + Happy | Text: 众大臣见状大喜，同时拜伏在地，大声道："太后吉祥！太后圣明！太后威武！"(English: The ministers were overjoyed and bowed to the ground at the same time, shouting: "the Empress Dowager is auspicious!" the Empress Dowager is holy! the Empress Dowager is mighty! ") | |||||
| Novels + Sad | Text: 那少妇摇了摇头，道："昨日前线传来消息说，这次御驾亲征已然惨败。" (English: The young woman shook her head and said, "the news came from the front yesterday that this expedition has been a fiasco.") | |||||
| Novels + Angry | Text: "气死我了！母后！武德侯害死皇兄，咱们还等什么？快快下令诛杀他全家满门，给皇兄报仇啊！"(English: "I'm so angry! mother! Wu Dehou killed his elder brother, what are we waiting for? quickly order his whole family to be killed and avenge his brother!") | |||||
| Novels + Surprise | Text: 惊呼，武德侯有功与国家，太后万万不可呀！(English: Exclaimed, Wu de Hou meritorious service to the country, the Empress Dowager must not!) | |||||
| Novels + Fear | Text: 一名长者走上前来，颤声道："这……这到底是怎么回事？为何有官兵杀来？"(English: An elderly man came forward and said in a trembling voice, "this …"... what on earth is going on? why are the officers and soldiers here? ") | |||||
Target speaker: F1
| Emotion | Target style example | Target emotion example | Target speaker example | MR-Tacotron | Referee | Proposed |
|---|---|---|---|---|---|---|
| Poet + Happy | Text: 长风破浪会有时，直挂云帆济沧海。(English: Someday, with my sail piercing the clouds; I will mount the wind, break the waves, and traverse the vast, rolling sea.) | |||||
| Poet + Sad | Text: 君不见，高堂明镜悲白发，朝如青丝暮成雪。(English: Don't you see? sadly, the high hall mirror saw white hair, as black as green hair in the morning and as white as snow in the evening.) | |||||
| Poet + Angry | Text: 天生我材必有用，千金散尽还复来。(English: Heaven has made us talents, we're not made in vain. A thousand gold coins spent, more will turn up again.) | |||||
| Poet + Surprise | Text: 从此山河两相阅，缠尽青山尝清河。(English: From then on, the mountains and rivers read each other and wrapped around the green mountains to taste the Qinghe River.) | |||||
| Poet + Fear | Text: 山不厌高，海不厌深。周公吐哺，天下归心。(English: The water can never be too deep, and the mountain too high. The Duke of Zhou spits and feeds, and the world returns to the heart.) | |||||
| Fairytales + Happy | Text: 不久以后，王后果然生下了一个可爱的小公主。(English: Before long, the queen gave birth to a lovely little princess.) | |||||
| Fairytales + Sad | Text: 她对恶魔说，求求你，千万不要伤害公主，我什么都可以给你。(English: She said to the devil, Please, don't hurt the princess. I can give you anything.) | |||||
| Fairytales + Angry | Text: 给我杀了她！把她的心和舌头都带回来，做为你杀死她的证据。(English: Kill her! Bring back her heart and tongue as proof that you killed her.) | |||||
| Fairytales + Surprise | Text: 进入小木屋后，里面竟然整齐排列着七张小小的床。(English: After entering the cabin, there were seven small beds neatly arranged inside.) | |||||
| Fairytales + Fear | Text: 听到猫头鹰叫声的白雪公主，越走越觉得森林好可怕。(English: Snow White, who heard the cry of an owl, felt more and more terrible in the forest as she walked.) | |||||
| Novels + Happy | Text: 众大臣见状大喜，同时拜伏在地，大声道："太后吉祥！太后圣明！太后威武！"(English: The ministers were overjoyed and bowed to the ground at the same time, shouting: "the Empress Dowager is auspicious!" the Empress Dowager is holy! the Empress Dowager is mighty! ") | |||||
| Novels + Sad | Text: 那少妇摇了摇头，道："昨日前线传来消息说，这次御驾亲征已然惨败。" (English: The young woman shook her head and said, "the news came from the front yesterday that this expedition has been a fiasco.") | |||||
| Novels + Angry | Text: "气死我了！母后！武德侯害死皇兄，咱们还等什么？快快下令诛杀他全家满门，给皇兄报仇啊！"(English: "I'm so angry! mother! Wu Dehou killed his elder brother, what are we waiting for? quickly order his whole family to be killed and avenge his brother!") | |||||
| Novels + Surprise | Text: 惊呼，武德侯有功与国家，太后万万不可呀！(English: Exclaimed, Wu de Hou meritorious service to the country, the Empress Dowager must not!) | |||||
| Novels + Fear | Text: 一名长者走上前来，颤声道："这……这到底是怎么回事？为何有官兵杀来？"(English: An elderly man came forward and said in a trembling voice, "this …"... what on earth is going on? why are the officers and soldiers here? ") | |||||
3. Demos -- long expressive speech with a specific style and various emotions for target speakers
We synthesize long segments of speech to help feel style attributes and emotional variation.
| Style | Text | Target speaker | Generated wave |
|---|---|---|---|
| Fairytales |
（happy）不久以后，王后果然生下了一个可爱的小公主。(English: Before long, the queen gave birth
to a lovely little princess.) （happy）白雪公主非常善良，有爱心，她经常和动物一起玩耍。(English: Snow White is very kind and caring. She often plays with animals.) （sad）可是，好景不长，白雪公主的母亲生病去世了。(English: But the good times didn't last long. Snow White's mother got sick and died.) （sad）国王向白雪公主介绍新王后时，她还正为死去的母后感到悲伤呢。(English: When the king introduced the new queen to Snow White, she was still grieving for her dead mother.) （angry）于是，她就命令宫廷的武士说，我不想再看到白雪公主了。(English: So she ordered the samurai of the court to say, I don't want to see Snow White anymore.) （angry）给我杀了她！把她的心和舌头都带回来，做为你杀死她的证据。(English: Kill her! Bring back her heart and tongue as proof that you killed her.) （fear）听到猫头鹰叫声的白雪公主，越走越觉得森林好可怕。(English: Snow White, who heard the cry of an owl, felt more and more terrible in the forest as she walked.) （surprise）突然，眼前有一栋小木屋，于是便又惊又喜的叫着，啊，是小木屋。(English: Suddenly, there was a cabin in front of him, so he shouted with surprise and delight, Oh, it's a cabin.) （surprise）进入小木屋后，里面竟然整齐排列着七张小小的床。(English: After entering the cabin, there were seven small beds neatly arranged in it.) |
||
| Fairytales |
（happy）不久以后，王后果然生下了一个可爱的小公主。(English: Before long, the queen gave birth
to a lovely little princess.) （happy）白雪公主非常善良，有爱心，她经常和动物一起玩耍。(English: Snow White is very kind and caring. She often plays with animals.) （sad）可是，好景不长，白雪公主的母亲生病去世了。(English: But the good times didn't last long. Snow White's mother got sick and died.) （sad）国王向白雪公主介绍新王后时，她还正为死去的母后感到悲伤呢。(English: When the king introduced the new queen to Snow White, she was still grieving for her dead mother.) （angry）于是，她就命令宫廷的武士说，我不想再看到白雪公主了。(English: So she ordered the samurai of the court to say, I don't want to see Snow White anymore.) （angry）给我杀了她！把她的心和舌头都带回来，做为你杀死她的证据。(English: Kill her! Bring back her heart and tongue as proof that you killed her.) （fear）听到猫头鹰叫声的白雪公主，越走越觉得森林好可怕。(English: Snow White, who heard the cry of an owl, felt more and more terrible in the forest as she walked.) （surprise）突然，眼前有一栋小木屋，于是便又惊又喜的叫着，啊，是小木屋。(English: Suddenly, there was a cabin in front of him, so he shouted with surprise and delight, Oh, it's a cabin.) （surprise）进入小木屋后，里面竟然整齐排列着七张小小的床。(English: After entering the cabin, there were seven small beds neatly arranged in it.) |
||
| Novels |
（sad）皇上昨日被武德侯所害，惨死皇宫，满城悲伤。(English: The emperor was killed by Wu Dehou yesterday. The palace died
tragically, and the city was filled with grief.) （angry）只见一名少年急急奔上台阶，生气的说。(English: A teenager rushed up the steps and said angrily.) （angry）"气死我了！母后！武德侯害死皇兄，咱们还等什么？快快下令诛杀他全家满门，给皇兄报仇啊！"(English: "I'm so angry! Queen Mother! what are we waiting for when Wu Dehou killed his brother? quickly order his whole family to be slaughtered and avenge his brother!") （netural）良久良久，那老妇终于咬住下唇，举起颤抖不止的手，轻轻的挥了挥。(English: For a long time, the old woman finally bit her lower lip, raised her trembling hand and waved it gently.) （happy）众大臣见状大喜，同时拜伏在地，大声道："太后吉祥！太后圣明！太后威武！"(English: The ministers were overjoyed and bowed to the ground at the same time, shouting: "the Empress Dowager is lucky! the Empress Dowager is holy! the Empress Dowager is mighty!") （happy）皇上终于可以瞑目了。(English: The emperor can finally rest in peace.) （netural）城中百姓或躲炕下，或藏窖中，无一人敢探头张望。(English: No one in the city dared to look around either under the Kang or in the cellar.) （angry）大军开至王府胡同，当先一将喝道："下马！快给我下马！"(English: When the army drove to Wangfu Hutong, the first general shouted, "dismount! dismount me!") （fear）听着可怕骇人的轰天巨响，每一下撞击声都敲进他们的心窝深处，似要将他们的魂胆撞碎。(English: Listening to the terrible and terrible bang, every crash knocked into the depths of their hearts, as if to break their hearts.) （sad）几个妇人挤在一起，泣不成声。(English: Several women huddled together and burst into tears.) （fear）一名长者走上前来，颤声道："这……这到底是怎么回事？为何有官兵杀来？"(English: An elderly man came forward and said in a trembling voice, "this. What on earth is going on? why are the officers and soldiers here?") （sad）那少妇摇了摇头，道："昨日前线传来消息，说这次御驾亲征已然惨败。"(English: The young woman shook her head and said, "there was news from the front yesterday that the expedition had failed miserably.") |
||
| Novels |
（sad）皇上昨日被武德侯所害，惨死皇宫，满城悲伤。(English: The emperor was killed by Wu Dehou yesterday. The palace died
tragically, and the city was filled with grief.) （angry）只见一名少年急急奔上台阶，生气的说。(English: A teenager rushed up the steps and said angrily.) （angry）"气死我了！母后！武德侯害死皇兄，咱们还等什么？快快下令诛杀他全家满门，给皇兄报仇啊！"(English: "I'm so angry! Queen Mother! what are we waiting for when Wu Dehou killed his brother? quickly order his whole family to be slaughtered and avenge his brother!") （netural）良久良久，那老妇终于咬住下唇，举起颤抖不止的手，轻轻的挥了挥。(English: For a long time, the old woman finally bit her lower lip, raised her trembling hand and waved it gently.) （happy）众大臣见状大喜，同时拜伏在地，大声道："太后吉祥！太后圣明！太后威武！"(English: The ministers were overjoyed and bowed to the ground at the same time, shouting: "the Empress Dowager is lucky! the Empress Dowager is holy! the Empress Dowager is mighty!") （happy）皇上终于可以瞑目了。(English: The emperor can finally rest in peace.) （netural）城中百姓或躲炕下，或藏窖中，无一人敢探头张望。(English: No one in the city dared to look around either under the Kang or in the cellar.) （angry）大军开至王府胡同，当先一将喝道："下马！快给我下马！"(English: When the army drove to Wangfu Hutong, the first general shouted, "dismount! dismount me!") （fear）听着可怕骇人的轰天巨响，每一下撞击声都敲进他们的心窝深处，似要将他们的魂胆撞碎。(English: Listening to the terrible and terrible bang, every crash knocked into the depths of their hearts, as if to break their hearts.) （sad）几个妇人挤在一起，泣不成声。(English: Several women huddled together and burst into tears.) （fear）一名长者走上前来，颤声道："这……这到底是怎么回事？为何有官兵杀来？"(English: An elderly man came forward and said in a trembling voice, "this. What on earth is going on? why are the officers and soldiers here?") （sad）那少妇摇了摇头，道："昨日前线传来消息，说这次御驾亲征已然惨败。"(English: The young woman shook her head and said, "there was news from the front yesterday that the expedition had failed miserably.") |