Xiaoxu Zhu - Speech AI Researcher

Publications

Speaker Disentanglement of Speech Pre-trained Model Based on Interpretability

Xiaoxu Zhu, Junhua Li

arxiv, 2025

pdf abstract bibtex

Speech pretrained models contain task-specific information across different layers, but decoupling content and timbre information remains challenging as removing speaker-specific information often causes content loss. Current research lacks direct metrics to quantify timbre residual in model encodings, relying on indirect evaluation through downstream tasks. This paper addresses these challenges through interpretability-based speaker disentanglement in speech pretraining models. We quantitatively evaluate timbre residual in model embeddings and improve speaker disentanglement using interpretive representations. Our contributions include: (1) InterpTRQE-SptME Benchmark - a timbre residual recognition framework using interpretability. The benchmark concatenates content embeddings with timbre embeddings for speaker classification, then applies Gradient SHAP Explainer to quantify timbre residual. We evaluate seven speech pretraining model variations. (2) InterpTF-SptME method - an interpretability-based timbre filtering approach using SHAP Noise and SHAP Cropping techniques. This model-agnostic method transforms intermediate encodings to remove timbre while preserving content. Experiments on VCTK dataset with HuBERT LARGE demonstrate successful content preservation and significant speaker disentanglement optimization. Results show the SHAP Noise method can reduce timbre residual from 18.05% to near 0% while maintaining content integrity, contributing to enhanced performance in content-related speech processing tasks and preventing timbre privacy leakage.
```
@article{zhu2024speaker,
  title={Speaker Disentanglement of Speech Pre-trained Model Based on Interpretability},
  author={Zhu, Xiaoxu and Li, Junhua},
  journal={arXiv preprint arXiv:2507.17851},
  year={2024}
}
```
A polyphone BERT for Polyphone Disambiguation in Mandarin Chinese

Song Zhang, Ken Zheng, Xiaoxu Zhu, Baoxiang Li

Interspeech, 2022

pdf abstract bibtex

Grapheme-to-phoneme (G2P) conversion is an indispensable part of the Chinese Mandarin text-to-speech (TTS) system, and the core of G2P conversion is to solve the problem of polyphone disambiguation, which is to pick up the correct pronunciation for several candidates for a Chinese polyphonic character. In this paper, we propose a Chinese polyphone BERT model to predict the pronunciations of Chinese polyphonic characters. Firstly, we create 741 new Chinese monophonic characters from 354 source Chinese polyphonic characters by pronunciation. Then we get a Chinese polyphone BERT by extending a pre-trained Chinese BERT with 741 new Chinese monophonic characters and adding a corresponding embedding layer for new tokens, which is initialized by the embeddings of source Chinese polyphonic characters. In this way, we can turn the polyphone disambiguation task into a pre-training task of the Chinese polyphone BERT. Experimental results demonstrate the effectiveness of the proposed model, and the polyphone BERT model obtain 2% (from 92.1% to 94.1%) improvement of average accuracy compared with the BERT-based classifier model, which is the prior state-of-the-art in polyphone disambiguation.
```
@inproceedings{zhang2022polyphone,
  title={A polyphone BERT for Polyphone Disambiguation in Mandarin Chinese},
  author={Zhang, Song and Zheng, Ken and Zhu, Xiaoxu and Li, Baoxiang},
  booktitle={Proc. Interspeech 2022},
  year={2022},
  note={Accepted for INTERSPEECH 2022}
}
```
Multimodal Sentiment Analysis via Efficient Multimodal Transformer and Modality-Aware Adaptive Training Strategy

C. Ding, D. Zong, B. Li, S. Zhang, X. Zhu, G. Zhong, D. Zhou

IEEE/ACM MuSe-Mimic, 2023

pdf abstract bibtex

In this paper, we present the solution to the MuSe-Mimic subchallenge of the 4th Multimodal Sentiment Analysis Challenge. This sub-challenge aims to predict the level of approval, disappointment and uncertainty in user-generated video clips. In our experiments, we found that naive joint training of multiple modalities by late fusion would result in insufficient learning of unimodal features. Moreover, different modalities contribute differently to MuSe-Mimic. Relying solely on multimodal features or treating unimodal features equally may limit the model's generalization performance. To address these challenges, we propose an efficient multimodal transformer equipped with a modality-aware adaptive training strategy to facilitate optimal joint training on multimodal sequence inputs. This framework holds promise in leveraging cross-modal interactions while ensuring adequate learning of unimodal features. Our model achieves the mean Pearson's Correlation Coefficient of .729 (ranking 2nd), outperforming official baseline result of .473. Our code is available at https://github.com/dingchaoyue/Multimodal-Emotion-Recognition-MER-and-MuSe-2023-Challenges.
```
@inproceedings{ding2023multimodal,
  title={Multimodal Sentiment Analysis via Efficient Multimodal Transformer and Modality-Aware Adaptive Training Strategy},
  author={Ding, C. and Zong, D. and Li, B. and Zhang, S. and Zhu, X. and Zhong, G. and Zhou, D.},
  booktitle={Proceedings of the 4th International on Multimodal Sentiment Analysis Workshop and Challenge},
  pages={263--270},
  year={2023},
  organization={ACM}
}
```

About Me

Education

Tsinghua University

Peter the Great St. Petersburg Polytechnic University

Harbin Institute of Technology

Work Experience

SenseTime

Shanghai AI Lab

Cheetah Mobile (OrionStar)

Siemens (St. Petersburg)

Publications

Speaker Disentanglement of Speech Pre-trained Model Based on Interpretability

A polyphone BERT for Polyphone Disambiguation in Mandarin Chinese

Multimodal Sentiment Analysis via Efficient Multimodal Transformer and Modality-Aware Adaptive Training Strategy

Contributions

Research Projects

Patents

Competitions & Awards

Open Source Contributions

LPCNet - Pre-compute GRU B Conditioning

Contact