Yekun Chai

Contact:
chaiyekun (at) gmail.com

I am a staff research engineer working on large language models at Baidu NLP. Before that, I was associated with Institute of Automation, Chinese Academy of Sciences (CASIA). I graduated from University of Edinburgh under the supervision of Adam Lopez and Naomi Saphra.

My research endeavors revolve around transformers and generative AI, with a particular emphasis on:

Pre-training large-scale foundation models across languages, modalities, and tasks.
AI alignment, reasoning, and scalable oversight.
Multimodal deep generative models.

news

Sep 21, 2024	Our papers on pixel-based pre-training, training data influence, and LLM tokenization have been accepted to EMNLP 2024 and Findings.
May 02, 2024	One paper on GiLOT, an XAI approach for LLMs, has been accepted to ICML 2024.
Feb 20, 2024	One paper on HumanEval-XL, a multilingual code generation benchmark has been accepted to LREC-COLING 2024. We’ve released the code and data.
Jan 16, 2024	One paper on reward models with tool-augmented feedback has been accepted to ICLR 2024 (spotlight). Dive into our research and code now!
Sep 23, 2023	One paper on XAI has been accepted to NeurIPS 2023 datasets and benchmarks track. Code is available here.

selected publications

arXiv

MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions

Yekun Chai^*†, Haoran Sun^{*^}, Huang Fang, Shuohuan Wang, and 2 more authors

2024

Bib PDF Code

@misc{chai2024marlhfreinforcementlearninghuman,
  title = {MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions},
  author = {Chai, Yekun and Sun, Haoran and Fang, Huang and Wang, Shuohuan and Sun, Yu and Wu, Hua},
  year = {2024},
  eprint = {2410.02743},
  archiveprefix = {arXiv},
  primaryclass = {cs.CL},
  url = {https://arxiv.org/abs/2410.02743},
}

EMNLP

Autoregressive Pre-Training on Pixels and Texts

Yekun Chai, Qingyi Liu^{^}, Jingwu Xiao^{^}, Shuohuan Wang, and 2 more authors

2024

Bib PDF Code

@misc{chai2024autoregressivepretrainingpixelstexts,
  title = {Autoregressive Pre-Training on Pixels and Texts},
  author = {Chai, Yekun and Liu, Qingyi and Xiao, Jingwu and Wang, Shuohuan and Sun, Yu and Wu, Hua},
  year = {2024},
  eprint = {2404.10710},
  archiveprefix = {arXiv},
  primaryclass = {cs.CL},
  url = {https://arxiv.org/abs/2404.10710},
}

ICLRSpotlight

Tool-Augmented Reward Modeling

Lei Li^{*^}, Yekun Chai^*†, Shuohuan Wang, Yu Sun, and 3 more authors

In The Twelfth International Conference on Learning Representations , 2024

Bib PDF Code Poster Slides

Spotlight

@inproceedings{li2024toolaugmented,
  title = {Tool-Augmented Reward Modeling},
  author = {Li, Lei and Chai, Yekun and Wang, Shuohuan and Sun, Yu and Tian, Hao and Zhang, Ningyu and Wu, Hua},
  booktitle = {The Twelfth International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=d94x0gWTUX},
}

ACL Findings
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages

Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, and 2 more authors

In Findings of the Association for Computational Linguistics: ACL 2023 , Jul 2023

Abs Bib PDF Code

Findings

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We release our code and pre-trained checkpoints.
@inproceedings{chai-etal-2023-ernie-code, url = {https://aclanthology.org/2023.findings-acl.676}, title = {{ERNIE}-Code: Beyond {E}nglish-Centric Cross-lingual Pretraining for Programming Languages}, author = {Chai, Yekun and Wang, Shuohuan and Pang, Chao and Sun, Yu and Tian, Hao and Wu, Hua}, editor = {Rogers, Anna and Boyd-Graber, Jordan and Okazaki, Naoaki}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2023}, month = jul, year = {2023}, address = {Toronto, Canada}, publisher = {Association for Computational Linguistics}, doi = {10.18653/v1/2023.findings-acl.676}, pages = {10628--10650}, }