Yekun Chai

Contact:
chaiyekun (at) gmail.com

I am a staff engineer at Baidu NLP, where I have contributed to Baidu’s LLM series such as ERNIE 4.0, ERNIE 3.5 and ERNIE-Code, and their industry-driven generative AI products such as ERNIE-Bot (文心一言) and Comate (文心快码). Before that, I honed my skills in RL and NLP at Institute of Automation, Chinese Academy of Sciences. I pursued my academic studies in NLP at University of Edinburgh, where I was fortunate to be supervised by Dr. Adam Lopez and Dr. Naomi Saphra.

My research endeavors revolve around transformers and generative AI, with a particular emphasis on:

Pre-training large-scale foundation models across languages, modalities, and tasks.
Efficient alignment, reasoning, and inference at scale.
Multimodal deep generative models.

news

Sep 21, 2024	Our papers on pixel-based pre-training, training data attribution, and tokenization robustness have been accepted to EMNLP 2024 and Findings.
May 02, 2024	One paper on GiLOT, an XAI approach for LLMs, has been accepted to ICML 2024.
Feb 20, 2024	One paper on HumanEval-XL, a multilingual code generation benchmark has been accepted to LREC-COLING 2024. We’ve released the code and data.
Jan 16, 2024	One paper on reward models with tool-augmented feedback has been accepted to ICLR 2024 (spotlight). Dive into our research and code now!
Sep 23, 2023	One paper on XAI has been accepted to NeurIPS 2023 datasets and benchmarks track. Code is available here.

selected publications

EMNLP

Autoregressive Pre-Training on Pixels and Texts

Yekun Chai, Qingyi Liu^{^}, Jingwu Xiao^{^}, Shuohuan Wang, and 2 more authors

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024

Abs HTML PDF Code Poster Slides

The integration of visual and textual information represents a promising direction in the advancement of language models. In this paper, we explore the dual modality of language—both visual and textual—within an autoregressive framework, pre-trained on both document images and texts. Our method employs a multimodal training strategy, utilizing visual data through next patch prediction with a regression head and/or textual data through next token prediction with a classification head. We focus on understanding the interaction between these two modalities and their combined impact on model performance. Our extensive evaluation across a wide range of benchmarks shows that incorporating both visual and textual data significantly improves the performance of pixel-based language models. Remarkably, we find that a unidirectional pixel-based model trained solely on visual data can achieve comparable results to state-of-the-art bidirectional models on several language understanding tasks. This work uncovers the untapped potential of integrating visual and textual modalities for more effective language modeling. We release our code, data, and model checkpoints at https://github.com/ernie-research/pixelgpt.
ICLRSpotlight

Tool-Augmented Reward Modeling

Lei Li^{*^}, Yekun Chai^*†, Shuohuan Wang, Yu Sun, and 3 more authors

In The Twelfth International Conference on Learning Representations , Nov 2024

PDF Code Poster Slides

Spotlight
ACL-Findings

ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages

Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, and 2 more authors

In Findings of the Association for Computational Linguistics: ACL 2023 , Jul 2023

Abs PDF Code

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We release our code and pre-trained checkpoints.