Xiaoxu Zhu - Speech AI Researcher

Publications

Robust Residual Finite Scalar Quantization for Neural Compression

Xiaoxu Zhu, Jiakui Li, Ken Zheng, Guiping Zhong, Huimeng Wang, Shiyin Kang, Dahua Lin

Manuscript in preparation for ICASSP 2026

pdf abstract bibtex

Finite Scalar Quantization (FSQ) has emerged as a promising alternative to Vector Quantization (VQ) in neural compression, offering simplified training and improved stability. However, naive application of FSQ in residual quantization frameworks suffers from the residual magnitude decay problem, where subsequent FSQ layers receive progressively weaker signals, severely limiting their effectiveness. We propose Robust Residual Finite Scalar Quantization (RFSQ), a general framework that addresses this fundamental limitation through two novel conditioning strategies: learnable scaling factors and invertible layer normalization. Our approach maintains the simplicity of FSQ while enabling effective multi-stage residual quantization. Comprehensive experiments on ImageNet demonstrate that RFSQ variants significantly outperform strong baselines including VQ-EMA, FSQ, and LFQ, achieving up to 45% improvement in perceptual loss and 28.7% reduction in L1 reconstruction error. The proposed LayerNorm strategy shows the most consistent improvements across different configurations, establishing RFSQ as a superior quantization method for image compression.
```
@article{zhu2025robust,
  title={Robust Residual Finite Scalar Quantization for Neural Compression},
  author={Zhu, Xiaoxu},
  journal={arXiv preprint arXiv:2508.15860},
  year={2025}
}
```
Speaker Disentanglement of Speech Pre-trained Model Based on Interpretability

Xiaoxu Zhu, Junhua Li, Aaron J. Li, Yiming Ren, Baoxiang Li

Manuscript in preparation for ICASSP 2026

pdf abstract bibtex

Speech pretrained models contain task-specific information across different layers, but decoupling content and timbre information remains challenging as removing speaker-specific information often causes content loss. Current research lacks direct metrics to quantify timbre residual in model encodings, relying on indirect evaluation through downstream tasks. This paper addresses these challenges through interpretability-based speaker disentanglement in speech pretraining models. We quantitatively evaluate timbre residual in model embeddings and improve speaker disentanglement using interpretive representations. Our contributions include: (1) InterpTRQE-SptME Benchmark - a timbre residual recognition framework using interpretability. The benchmark concatenates content embeddings with timbre embeddings for speaker classification, then applies Gradient SHAP Explainer to quantify timbre residual. We evaluate seven speech pretraining model variations. (2) InterpTF-SptME method - an interpretability-based timbre filtering approach using SHAP Noise and SHAP Cropping techniques. This model-agnostic method transforms intermediate encodings to remove timbre while preserving content. Experiments on VCTK dataset with HuBERT LARGE demonstrate successful content preservation and significant speaker disentanglement optimization. Results show the SHAP Noise method can reduce timbre residual from 18.05% to near 0% while maintaining content integrity, contributing to enhanced performance in content-related speech processing tasks and preventing timbre privacy leakage.
```
@article{zhu2025speaker,
  title={Speaker Disentanglement of Speech Pre-trained Model Based on Interpretability},
  author={Zhu, Xiaoxu and Li, Junhua},
  journal={arXiv preprint arXiv:2507.17851},
  year={2025}
}
```
InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue

Wenwen Tong, Hewei Guo, Dongchuan Ran, ..., Xiaoxu Zhu, ..., Shiyin Kang, Lewei Lu

Technical Report, 2025

pdf abstract bibtex

We introduce InteractiveOmni, a unified and open-source omni-modal large language model for audio-visual multi-turn interaction, ranging from 4B to 8B parameters, designed to lead the field of lightweight models by offering comprehensive omni-modal understanding and speech generation capabilities. To achieve this, we integrate the vision encoder, audio encoder, large language model, and speech decoder into a unified model for understanding and generation tasks. We design a multi-stage training strategy to ensure robust cross-modal capabilities, including pre-training for omni-modal understanding, followed by post-training with speech conversation and audio-visual interaction. To enable human-like long-term conversational ability, we meticulously curate a multi-turn training dataset that enhances the model's ability to handle complex and multi-turn interactions. To effectively evaluate the multi-turn memory and speech interaction capabilities, we construct the multi-modal multi-turn memory benchmark and the multi-turn speech interaction benchmark. Experiments demonstrate that InteractiveOmni significantly outperforms leading open-source models and provides a more intelligent multi-turn audio-visual experience, particularly in its long-term memory capabilities. Notably, InteractiveOmni-4B is comparable to the much larger model like Qwen2.5-Omni-7B on general benchmarks, and it can retain 97% of the performance of the InteractiveOmni-8B while utilizing only 50% of the model size. Achieving state-of-the-art results against similarly sized models across image, audio, video understanding, and speech generation tasks, InteractiveOmni is an accessible, open-source foundation for next-generation intelligent interactive systems.
```
@misc{tong2025interactiveomni,
  title={InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue},
  author={Wenwen Tong and Hewei Guo and Dongchuan Ran and Jiangnan Chen and Jiefan Lu and Kaibin Wang and Keqiang Li and Xiaoxu Zhu and Jiakui Li and Kehan Li and Xueheng Li and Lumin Li and Chenxu Guo and Jiasheng Zhou and Jiandong Chen and Xianye Wu and Jiahao Wang and Silei Wu and Lei Chen and Hanming Deng and Yuxuan Song and Dinghao Zhou and Guiping Zhong and Ken Zheng and Shiyin Kang and Lewei Lu},
  year={2025},
  eprint={2510.13747},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  institution={SenseTime Research},
  note={arXiv:2510.13747v1 [cs.CV] 15 Oct 2025, GitHub: https://github.com/SenseTime-FVG/InteractiveOmni}
}
```
A polyphone BERT for Polyphone Disambiguation in Mandarin Chinese

Song Zhang, Ken Zheng, Xiaoxu Zhu, Baoxiang Li

Interspeech, 2022

pdf abstract bibtex

Grapheme-to-phoneme (G2P) conversion is an indispensable part of the Chinese Mandarin text-to-speech (TTS) system, and the core of G2P conversion is to solve the problem of polyphone disambiguation, which is to pick up the correct pronunciation for several candidates for a Chinese polyphonic character. In this paper, we propose a Chinese polyphone BERT model to predict the pronunciations of Chinese polyphonic characters. Firstly, we create 741 new Chinese monophonic characters from 354 source Chinese polyphonic characters by pronunciation. Then we get a Chinese polyphone BERT by extending a pre-trained Chinese BERT with 741 new Chinese monophonic characters and adding a corresponding embedding layer for new tokens, which is initialized by the embeddings of source Chinese polyphonic characters. In this way, we can turn the polyphone disambiguation task into a pre-training task of the Chinese polyphone BERT. Experimental results demonstrate the effectiveness of the proposed model, and the polyphone BERT model obtain 2% (from 92.1% to 94.1%) improvement of average accuracy compared with the BERT-based classifier model, which is the prior state-of-the-art in polyphone disambiguation.
```
@inproceedings{zhang2022polyphone,
  title={A polyphone BERT for Polyphone Disambiguation in Mandarin Chinese},
  author={Zhang, Song and Zheng, Ken and Zhu, Xiaoxu and Li, Baoxiang},
  booktitle={Proc. Interspeech 2022},
  year={2022},
  note={Accepted for INTERSPEECH 2022}
}
```
Multimodal Sentiment Analysis via Efficient Multimodal Transformer and Modality-Aware Adaptive Training Strategy

C. Ding, D. Zong, B. Li, S. Zhang, X. Zhu, G. Zhong, D. Zhou

IEEE/ACM MuSe-Mimic, 2023

pdf abstract bibtex

In this paper, we present the solution to the MuSe-Mimic subchallenge of the 4th Multimodal Sentiment Analysis Challenge. This sub-challenge aims to predict the level of approval, disappointment and uncertainty in user-generated video clips. In our experiments, we found that naive joint training of multiple modalities by late fusion would result in insufficient learning of unimodal features. Moreover, different modalities contribute differently to MuSe-Mimic. Relying solely on multimodal features or treating unimodal features equally may limit the model's generalization performance. To address these challenges, we propose an efficient multimodal transformer equipped with a modality-aware adaptive training strategy to facilitate optimal joint training on multimodal sequence inputs. This framework holds promise in leveraging cross-modal interactions while ensuring adequate learning of unimodal features. Our model achieves the mean Pearson's Correlation Coefficient of .729 (ranking 2nd), outperforming official baseline result of .473. Our code is available at https://github.com/dingchaoyue/Multimodal-Emotion-Recognition-MER-and-MuSe-2023-Challenges.
```
@inproceedings{ding2023multimodal,
  title={Multimodal Sentiment Analysis via Efficient Multimodal Transformer and Modality-Aware Adaptive Training Strategy},
  author={Ding, C. and Zong, D. and Li, B. and Zhang, S. and Zhu, X. and Zhong, G. and Zhou, D.},
  booktitle={Proceedings of the 4th International on Multimodal Sentiment Analysis Workshop and Challenge},
  pages={263--270},
  year={2023},
  organization={ACM}
}
```

About Me

Education

Tsinghua University

Peter the Great St. Petersburg Polytechnic University

Harbin Institute of Technology

Work Experience

SenseTime

Shanghai AI Lab

Cheetah Mobile (OrionStar)

Siemens (St. Petersburg)

Publications

Robust Residual Finite Scalar Quantization for Neural Compression

Speaker Disentanglement of Speech Pre-trained Model Based on Interpretability

InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue

A polyphone BERT for Polyphone Disambiguation in Mandarin Chinese

Multimodal Sentiment Analysis via Efficient Multimodal Transformer and Modality-Aware Adaptive Training Strategy

Contributions

Research Projects

Patents

Competitions & Awards

Open Source Contributions

LPCNet - Pre-compute GRU B Conditioning

Contact