Yekun Chai

Contact:
chaiyekun (at) gmail.com

I am a staff research engineer working on large language models at Baidu NLP. Before that, I was associated with Institute of Automation, Chinese Academy of Sciences (CASIA). I graduated from University of Edinburgh under the supervision of Adam Lopez and Naomi Saphra.

My research endeavors revolve around transformers and generative AI, with a particular emphasis on:

Pre-training large-scale foundation models across languages, modalities, and tasks.
AI alignment, reasoning, and scalable oversight.
Multimodal deep generative models.

news

Sep 21, 2024	Our papers on pixel-based pre-training, training data influence, and LLM tokenization have been accepted to EMNLP 2024 and Findings.
May 02, 2024	One paper on GiLOT, an XAI approach for LLMs, has been accepted to ICML 2024.
Feb 20, 2024	One paper on HumanEval-XL, a multilingual code generation benchmark has been accepted to LREC-COLING 2024. We’ve released the code and data.
Jan 16, 2024	One paper on reward models with tool-augmented feedback has been accepted to ICLR 2024 (spotlight). Dive into our research and code now!
Sep 23, 2023	One paper on XAI has been accepted to NeurIPS 2023 datasets and benchmarks track. Code is available here.

selected publications

EMNLP

Autoregressive Pre-Training on Pixels and Texts

Yekun Chai, Qingyi Liu^{^}, Jingwu Xiao^{^}, Shuohuan Wang, and 2 more authors

2024

PDF Code Poster Slides
ICLRSpotlight

Tool-Augmented Reward Modeling

Lei Li^{*^}, Yekun Chai^*†, Shuohuan Wang, Yu Sun, and 3 more authors

In The Twelfth International Conference on Learning Representations , 2024

PDF Code Poster Slides

Spotlight
ACL-Findings

ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages

Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, and 2 more authors

In Findings of the Association for Computational Linguistics: ACL 2023 , Jul 2023

Abs PDF Code

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We release our code and pre-trained checkpoints.