Yiyang Feng (冯乙洋)

1. 

   EACL 2026

   Tracking the Limits of Knowledge Propagation: How LLMs Fail at Multi-Step Reasoning with Conflicting Knowledge

   Yiyang Feng, Zeming Chen, Haotian Wu, Jiawei Zhou, and Antoine Bosselut

   EACL 2026 (Main Conference) Virtual Oral

   Giving a model updated facts can make its multi-step reasoning worse than giving none, and more facts make it worse. TRACK measures this on Wiki, Code, and Math questions with multiple knowledge conflicts.

   Abstract PDF Code Data Website Talk Thread BibTeX

   A common solution for mitigating outdated or incorrect information in Large Language Models (LLMs) is to provide updated facts in-context or through knowledge editing. However, these methods introduce knowledge conflicts when the knowledge update fails to overwrite the model’s parametric knowledge, which propagate to faulty reasoning. Current benchmarks for this problem largely focus only on single knowledge updates and fact recall without evaluating how these updates affect downstream reasoning. In this work, we introduce TRACK (Testing Reasoning Amid Conflicting Knowledge), a new benchmark for studying how LLMs propagate new knowledge through multi-step reasoning when it conflicts with the model’s initial parametric knowledge. Spanning three reasoning-intensive scenarios (WIKI, CODE, and MATH), TRACK introduces multiple, realistic conflicts to mirror real-world complexity. Our results reveal that providing updated facts to models for reasoning can worsen performance compared to providing no updated facts at all, and that this performance degradation exacerbates as more updated facts are provided. We show this failure stems from both inability to faithfully integrate updated facts, and also flawed reasoning even when knowledge is integrated. TRACK provides a rigorous new benchmark to measure and guide future progress on propagating conflicting knowledge in multi-step reasoning.

   @inproceedings{feng-etal-2026-tracking,
     title = {Tracking the Limits of Knowledge Propagation: How {LLM}s Fail at Multi-Step Reasoning with Conflicting Knowledge},
     author = {Feng, Yiyang and Chen, Zeming and Wu, Haotian and Zhou, Jiawei and Bosselut, Antoine},
     editor = {Demberg, Vera and Inui, Kentaro and Marquez, Llu{\'i}s},
     booktitle = {Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)},
     month = mar,
     year = {2026},
     address = {Rabat, Morocco},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2026.eacl-long.273/},
     doi = {10.18653/v1/2026.eacl-long.273},
     pages = {5813--5847},
     isbn = {979-8-89176-380-7},
   }

2. 

   EMNLP 2025

   Unraveling Misinformation Propagation in LLM Reasoning

   Yiyang Feng*, Yichen Wang*, Shaobo Cui, Boi Faltings, Mina Lee, and Jiawei Zhou

   Findings of EMNLP 2025

   A user’s false premise spreads through the model’s reasoning even when the model knows the correct fact, and explicit instructions to correct it succeed less than half the time. Correcting early works best, and fine-tuning on early corrections helps.

   Abstract PDF arXiv Code Website Talk Thread BibTeX

   Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning, positioning them as promising tools for supporting human problem-solving. However, what happens when their performance is affected by misinformation, i.e., incorrect inputs introduced by users due to oversights or gaps in knowledge? Such misinformation is prevalent in real-world interactions with LLMs, yet how it propagates within LLMs’ reasoning process remains underexplored. Focusing on mathematical reasoning, we present a comprehensive analysis of how misinformation affects intermediate reasoning steps and final answers. We also examine how effectively LLMs can correct misinformation when explicitly instructed to do so. Even with explicit instructions, LLMs succeed less than half the time in rectifying misinformation, despite possessing correct internal knowledge, leading to significant accuracy drops (10.02% – 72.20%), and the degradation holds with thinking models (4.30% – 19.97%). Further analysis shows that applying factual corrections early in the reasoning process most effectively reduces misinformation propagation, and fine-tuning on synthesized data with early-stage corrections significantly improves reasoning factuality. Our work offers a practical approach to mitigating misinformation propagation.

   @inproceedings{feng-etal-2025-unraveling,
     title = {Unraveling Misinformation Propagation in {LLM} Reasoning},
     author = {Feng*, Yiyang and Wang*, Yichen and Cui, Shaobo and Faltings, Boi and Lee, Mina and Zhou, Jiawei},
     editor = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
     month = nov,
     year = {2025},
     address = {Suzhou, China},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2025.findings-emnlp.627/},
     pages = {11683--11707},
     isbn = {979-8-89176-335-7},
   }

3. 

   ACL 2024

   Unveiling the Art of Heading Design: A Harmonious Blend of Summarization, Neology, and Algorithm

   Shaobo Cui*, Yiyang Feng*, Yisong Mao, Yifan Hou, and Boi Faltings

   Findings of ACL 2024

   LOgogram: a benchmark of 6,653 abstracts for writing paper headings, a description plus an acronym spelled out by letters of that description. Metrics cover summarization, neology, and this symbolic constraint, under fine-tuning, in-context learning, and RL.

   Abstract PDF BibTeX

   Crafting an appealing heading is crucial for attracting readers and marketing work or products. A popular way is to summarize the main idea with a refined description and a memorable acronym. However, there lacks a systematic study and a formal benchmark including datasets and metrics. Motivated by this absence, we introduce LOgogram, a novel benchmark comprising 6,653 paper abstracts with corresponding descriptions and acronyms. To measure the quality of heading generation, we propose a set of evaluation metrics from three aspects: summarization, neology, and algorithm. Additionally, we explore three strategies for heading generation (generation ordering, tokenization of acronyms, and framework design) under various prevalent learning paradigms (supervised fine-tuning, in-context learning with Large Language Models (LLMs), and reinforcement learning) on our benchmark. Our experimental results indicate the difficulty in identifying a practice that excels across all summarization, neologistic, and algorithmic aspects.

   @inproceedings{cui-etal-2024-unveiling,
     title = {Unveiling the Art of Heading Design: A Harmonious Blend of Summarization, Neology, and Algorithm},
     author = {Cui*, Shaobo and Feng*, Yiyang and Mao, Yisong and Hou, Yifan and Faltings, Boi},
     editor = {Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek},
     booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
     month = aug,
     year = {2024},
     address = {Bangkok, Thailand},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.findings-acl.368/},
     pages = {6149--6174},
   }

1. 

   ACL 2025

   Conditional Dichotomy Quantification via Geometric Embedding

   Shaobo Cui, Wenqing Liu, Yiyang Feng, Jiawei Zhou, and Boi Faltings

   ACL 2025 (Main Conference) Oral

   Measures how strongly two outputs oppose each other under the same context, using complex-valued embeddings (DoGE), with datasets for debate, defeasible inference, and causal reasoning.

   Abstract PDF Code BibTeX

   Conditional dichotomy, the contrast between two outputs conditioned on the same context, is vital for applications such as debate, defeasible inference, and causal reasoning. Existing methods that rely on semantic similarity often fail to capture the nuanced oppositional dynamics essential for these applications. Motivated by these limitations, we introduce a novel task, Conditional Dichotomy Quantification (ConDQ), which formalizes the direct measurement of conditional dichotomy and provides carefully constructed datasets covering debate, defeasible natural language inference, and causal reasoning scenarios. To address this task, we develop the Dichotomy-oriented Geometric Embedding (DoGE) framework, which leverages complex-valued embeddings and a dichotomous objective to model and quantify these oppositional relationships effectively. Extensive experiments validate the effectiveness and versatility of DoGE, demonstrating its potential in understanding and quantifying conditional dichotomy across diverse NLP applications. Our code and datasets are available at https://github.com/cui-shaobo/conditional-dichotomy-quantification.

   @inproceedings{cui-etal-2025-conditional,
     title = {Conditional Dichotomy Quantification via Geometric Embedding},
     author = {Cui, Shaobo and Liu, Wenqing and Feng, Yiyang and Zhou, Jiawei and Faltings, Boi},
     editor = {Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher},
     booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = jul,
     year = {2025},
     address = {Vienna, Austria},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2025.acl-long.383/},
     doi = {10.18653/v1/2025.acl-long.383},
     pages = {7765--7791},
     isbn = {979-8-89176-251-0},
   }

2. 

   AAAI 2025

   Nuance Matters: Probing Epistemic Consistency in Causal Reasoning

   Shaobo Cui, Junyou Li, Luca Mouchel, Yiyang Feng, and Boi Faltings

   AAAI 2025

   Asks whether 21 LLMs rank fine-grained causal intermediates by polarity and intensity consistently with themselves. They mostly do not.

   Abstract PDF arXiv BibTeX

   Previous research on causal reasoning often overlooks the subtleties crucial to understanding causal reasoning. To address this gap, our study introduces the concept of causal epistemic consistency, which focuses on the self-consistency of Large Language Models (LLMs) in differentiating intermediates with nuanced differences in causal reasoning. We propose a suite of novel metrics – intensity ranking concordance, cross-group position agreement, and intra-group clustering – to evaluate LLMs on this front. Through extensive empirical studies on 21 high-profile LLMs, including GPT-4, Claude3, and LLaMA3-70B, we have favoring evidence that current models struggle to maintain epistemic consistency in identifying the polarity and intensity of intermediates in causal reasoning. Additionally, we explore the potential of using internal token probabilities as an auxiliary tool to maintain causal epistemic consistency. In summary, our study bridges a critical gap in AI research by investigating the self-consistency over fine-grained intermediates involved in causal reasoning.

   @inproceedings{cui2024nuancemattersprobingepistemic,
     title = {Nuance Matters: Probing Epistemic Consistency in Causal Reasoning},
     author = {Cui, Shaobo and Li, Junyou and Mouchel, Luca and Feng, Yiyang and Faltings, Boi},
     booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
     volume = {39},
     number = {22},
     month = apr,
     year = {2025},
     pages = {23715--23723},
     url = {https://ojs.aaai.org/index.php/AAAI/article/view/34542},
     doi = {10.1609/aaai.v39i22.34542},
   }

3. 

   ACL 2024

   Exploring Defeasibility in Causal Reasoning

   Shaobo Cui, Lazar Milikic, Yiyang Feng, Mete Ismayilzada, Debjit Paul, Antoine Bosselut, and Boi Faltings

   Findings of ACL 2024

   delta-CAUSAL: about 11K cause-effect pairs with supporters and defeaters that strengthen or weaken them, and CESAR, a causal-strength metric that tracks those changes.

   Abstract PDF arXiv BibTeX

   Defeasibility in causal reasoning implies that the causal relationship between cause and effect can be strengthened or weakened. Namely, the causal strength between cause and effect should increase or decrease with the incorporation of strengthening arguments (supporters) or weakening arguments (defeaters), respectively. However, existing works ignore defeasibility in causal reasoning and fail to evaluate existing causal strength metrics in defeasible settings. In this work, we present delta-CAUSAL, the first benchmark dataset for studying defeasibility in causal reasoning. delta-CAUSAL includes around 11K events spanning ten domains, featuring defeasible causality pairs, namely, cause-effect pairs accompanied by supporters and defeaters. We further show that current causal strength metrics fail to reflect the change of causal strength with the incorporation of supporters or defeaters in delta-CAUSAL. To this end, we propose CESAR (Causal Embedding aSsociation with Attention Rating), a metric that measures causal strength based on token-level causal relationships. CESAR achieves a significant 69.7% relative improvement over existing metrics, increasing from 47.2% to 80.1% in capturing the causal strength change brought by supporters and defeaters. We further demonstrate even Large Language Models (LLMs) like GPT-3.5 still lag 4.5 and 10.7 points behind humans in generating supporters and defeaters, emphasizing the challenge posed by delta-CAUSAL.

   @inproceedings{cui2024deltacausal,
     title = {Exploring Defeasibility in Causal Reasoning},
     author = {Cui, Shaobo and Milikic, Lazar and Feng, Yiyang and Ismayilzada, Mete and Paul, Debjit and Bosselut, Antoine and Faltings, Boi},
     editor = {Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek},
     booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
     month = aug,
     year = {2024},
     address = {Bangkok, Thailand},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.findings-acl.384/},
     pages = {6433--6452},
   }

1. 

   arXiv 2026

   Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents

   Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian, and Jiawei Zhou

   arXiv preprint, 2026

   Skills induced per subtask lift agents above their no-memory baseline, while skills induced per whole task mostly push them below it. A skill utility score predicts which skills will transfer before any new task is run.

   Abstract PDF Code Data Website BibTeX

   Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience. In practice, induced skills may transfer unreliably and can even harm the agent that retrieves them. When agent-induced skills transfer reliably across tasks remains an open question. We conduct a comprehensive and controlled study of how the way skills are induced shapes their transfer across tasks. Specifically, we compare task-level with subtask-level skill induction and text with code skill formats, the two axes along which existing methods differ. Task-level skills mostly reduce the agent’s performance below its no-memory baseline while subtask-level skills raise it above on average, and text skills transfer better than code skills. To further understand our findings, we examine two complementary properties of the induced skills: specificity, which measures how closely a skill matches real tasks, and abstractness, which measures how evenly its relevance spreads across tasks. Neither property alone predicts task success, but their combined effect does, which we propose as a skill utility score. The score correlates consistently with task success when skills are transferred, and subtask-level and text skills score higher. Computing skill utility only needs the skills and task descriptions, so our score serves as a lightweight diagnostic of a skill memory for practitioners.

   @misc{feng2026breakdownpasson,
     title = {Break It Down, Pass It On: Cross-Task Skill Transfer in {LLM} Agents},
     author = {Feng, Yiyang and Bijoy, Biddut Sarker and Balasubramanian, Niranjan and Zhou, Jiawei},
     year = {2026},
     month = aug,
     eprint = {2608.20274},
     archiveprefix = {arXiv},
     primaryclass = {cs.AI},
     url = {https://arxiv.org/abs/2608.20274},
   }

First or co-first author

1. 

   arXiv 2026

   Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents

   Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian, and Jiawei Zhou

   arXiv preprint, 2026

   Skills induced per subtask lift agents above their no-memory baseline, while skills induced per whole task mostly push them below it. A skill utility score predicts which skills will transfer before any new task is run.

   Abstract PDF Code Data Website BibTeX

   Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience. In practice, induced skills may transfer unreliably and can even harm the agent that retrieves them. When agent-induced skills transfer reliably across tasks remains an open question. We conduct a comprehensive and controlled study of how the way skills are induced shapes their transfer across tasks. Specifically, we compare task-level with subtask-level skill induction and text with code skill formats, the two axes along which existing methods differ. Task-level skills mostly reduce the agent’s performance below its no-memory baseline while subtask-level skills raise it above on average, and text skills transfer better than code skills. To further understand our findings, we examine two complementary properties of the induced skills: specificity, which measures how closely a skill matches real tasks, and abstractness, which measures how evenly its relevance spreads across tasks. Neither property alone predicts task success, but their combined effect does, which we propose as a skill utility score. The score correlates consistently with task success when skills are transferred, and subtask-level and text skills score higher. Computing skill utility only needs the skills and task descriptions, so our score serves as a lightweight diagnostic of a skill memory for practitioners.

   @misc{feng2026breakdownpasson,
     title = {Break It Down, Pass It On: Cross-Task Skill Transfer in {LLM} Agents},
     author = {Feng, Yiyang and Bijoy, Biddut Sarker and Balasubramanian, Niranjan and Zhou, Jiawei},
     year = {2026},
     month = aug,
     eprint = {2608.20274},
     archiveprefix = {arXiv},
     primaryclass = {cs.AI},
     url = {https://arxiv.org/abs/2608.20274},
   }

2. 

   arXiv 2026

   Averaging Bias: Human Faithfulness Annotations are not Locally Faithful

   Huajian Zhang*, Yiyang Feng*, and Jiawei Zhou

   arXiv preprint, 2026

   Human faithfulness labels on four summarization benchmarks follow the average of per-sentence judgments rather than the all-sentences-faithful rule, so many summaries labeled faithful contain local factual errors.

   Abstract PDF Code BibTeX

   Evaluation of faithfulness of text summarization treats a model generated summary as faithful only if every of its sentences is supported by the source document: a strict conjunctive rule under which a single unsupported sentence makes the whole summary unfaithful. Yet most faithfulness benchmarks collect only one global human annotation label per summary. We ask whether such global human labels actually implement the conjunctive rule. We hypothesize that annotators may accept a summary as faithful when most sentences are faithful, not only when all are faithful. To test our hypothesis, we use five large language model (LLM) judges as per-sentence raters across four widely used faithfulness benchmarks. We find that global human labels correlate better with the average of per-sentence LLM judgments than with the implementation of the strict conjunctive rule. A manual review confirms that a substantial fraction of summaries labeled faithful by humans contain genuine local factual errors. We call this tendency Averaging Bias. Our results reveal that human labels on widely used faithfulness benchmarks contain measurable Averaging Bias, calling for carefully structured designs for trustworthy human annotations.

   @misc{zhang2026averagingbias,
     title = {Averaging Bias: Human Faithfulness Annotations are not Locally Faithful},
     author = {Zhang*, Huajian and Feng*, Yiyang and Zhou, Jiawei},
     year = {2026},
     month = jul,
     eprint = {2608.00205},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
     url = {https://arxiv.org/abs/2608.00205},
   }

3. 

   EACL 2026

   Tracking the Limits of Knowledge Propagation: How LLMs Fail at Multi-Step Reasoning with Conflicting Knowledge

   Yiyang Feng, Zeming Chen, Haotian Wu, Jiawei Zhou, and Antoine Bosselut

   EACL 2026 (Main Conference) Virtual Oral

   Giving a model updated facts can make its multi-step reasoning worse than giving none, and more facts make it worse. TRACK measures this on Wiki, Code, and Math questions with multiple knowledge conflicts.

   Abstract PDF Code Data Website Talk Thread BibTeX

   A common solution for mitigating outdated or incorrect information in Large Language Models (LLMs) is to provide updated facts in-context or through knowledge editing. However, these methods introduce knowledge conflicts when the knowledge update fails to overwrite the model’s parametric knowledge, which propagate to faulty reasoning. Current benchmarks for this problem largely focus only on single knowledge updates and fact recall without evaluating how these updates affect downstream reasoning. In this work, we introduce TRACK (Testing Reasoning Amid Conflicting Knowledge), a new benchmark for studying how LLMs propagate new knowledge through multi-step reasoning when it conflicts with the model’s initial parametric knowledge. Spanning three reasoning-intensive scenarios (WIKI, CODE, and MATH), TRACK introduces multiple, realistic conflicts to mirror real-world complexity. Our results reveal that providing updated facts to models for reasoning can worsen performance compared to providing no updated facts at all, and that this performance degradation exacerbates as more updated facts are provided. We show this failure stems from both inability to faithfully integrate updated facts, and also flawed reasoning even when knowledge is integrated. TRACK provides a rigorous new benchmark to measure and guide future progress on propagating conflicting knowledge in multi-step reasoning.

   @inproceedings{feng-etal-2026-tracking,
     title = {Tracking the Limits of Knowledge Propagation: How {LLM}s Fail at Multi-Step Reasoning with Conflicting Knowledge},
     author = {Feng, Yiyang and Chen, Zeming and Wu, Haotian and Zhou, Jiawei and Bosselut, Antoine},
     editor = {Demberg, Vera and Inui, Kentaro and Marquez, Llu{\'i}s},
     booktitle = {Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)},
     month = mar,
     year = {2026},
     address = {Rabat, Morocco},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2026.eacl-long.273/},
     doi = {10.18653/v1/2026.eacl-long.273},
     pages = {5813--5847},
     isbn = {979-8-89176-380-7},
   }

4. 

   EMNLP 2025

   Unraveling Misinformation Propagation in LLM Reasoning

   Yiyang Feng*, Yichen Wang*, Shaobo Cui, Boi Faltings, Mina Lee, and Jiawei Zhou

   Findings of EMNLP 2025

   A user’s false premise spreads through the model’s reasoning even when the model knows the correct fact, and explicit instructions to correct it succeed less than half the time. Correcting early works best, and fine-tuning on early corrections helps.

   Abstract PDF arXiv Code Website Talk Thread BibTeX

   Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning, positioning them as promising tools for supporting human problem-solving. However, what happens when their performance is affected by misinformation, i.e., incorrect inputs introduced by users due to oversights or gaps in knowledge? Such misinformation is prevalent in real-world interactions with LLMs, yet how it propagates within LLMs’ reasoning process remains underexplored. Focusing on mathematical reasoning, we present a comprehensive analysis of how misinformation affects intermediate reasoning steps and final answers. We also examine how effectively LLMs can correct misinformation when explicitly instructed to do so. Even with explicit instructions, LLMs succeed less than half the time in rectifying misinformation, despite possessing correct internal knowledge, leading to significant accuracy drops (10.02% – 72.20%), and the degradation holds with thinking models (4.30% – 19.97%). Further analysis shows that applying factual corrections early in the reasoning process most effectively reduces misinformation propagation, and fine-tuning on synthesized data with early-stage corrections significantly improves reasoning factuality. Our work offers a practical approach to mitigating misinformation propagation.

   @inproceedings{feng-etal-2025-unraveling,
     title = {Unraveling Misinformation Propagation in {LLM} Reasoning},
     author = {Feng*, Yiyang and Wang*, Yichen and Cui, Shaobo and Faltings, Boi and Lee, Mina and Zhou, Jiawei},
     editor = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
     month = nov,
     year = {2025},
     address = {Suzhou, China},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2025.findings-emnlp.627/},
     pages = {11683--11707},
     isbn = {979-8-89176-335-7},
   }

5. 

   ACL 2024

   Unveiling the Art of Heading Design: A Harmonious Blend of Summarization, Neology, and Algorithm

   Shaobo Cui*, Yiyang Feng*, Yisong Mao, Yifan Hou, and Boi Faltings

   Findings of ACL 2024

   LOgogram: a benchmark of 6,653 abstracts for writing paper headings, a description plus an acronym spelled out by letters of that description. Metrics cover summarization, neology, and this symbolic constraint, under fine-tuning, in-context learning, and RL.

   Abstract PDF BibTeX

   Crafting an appealing heading is crucial for attracting readers and marketing work or products. A popular way is to summarize the main idea with a refined description and a memorable acronym. However, there lacks a systematic study and a formal benchmark including datasets and metrics. Motivated by this absence, we introduce LOgogram, a novel benchmark comprising 6,653 paper abstracts with corresponding descriptions and acronyms. To measure the quality of heading generation, we propose a set of evaluation metrics from three aspects: summarization, neology, and algorithm. Additionally, we explore three strategies for heading generation (generation ordering, tokenization of acronyms, and framework design) under various prevalent learning paradigms (supervised fine-tuning, in-context learning with Large Language Models (LLMs), and reinforcement learning) on our benchmark. Our experimental results indicate the difficulty in identifying a practice that excels across all summarization, neologistic, and algorithmic aspects.

   @inproceedings{cui-etal-2024-unveiling,
     title = {Unveiling the Art of Heading Design: A Harmonious Blend of Summarization, Neology, and Algorithm},
     author = {Cui*, Shaobo and Feng*, Yiyang and Mao, Yisong and Hou, Yifan and Faltings, Boi},
     editor = {Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek},
     booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
     month = aug,
     year = {2024},
     address = {Bangkok, Thailand},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.findings-acl.368/},
     pages = {6149--6174},
   }

Co-author

1. 

   TMLR 2026

   Self-Improvement of Large Language Models: A Technical Overview and Future Outlook

   Haoyan Yang, Mario Xerri, Solha Park, Huajian Zhang, Yiyang Feng, Sai Akhil Kogilathota, and Jiawei Zhou

   Transactions on Machine Learning Research (TMLR), 2026 Survey Certification

   Organizes LLM self-improvement as a closed loop of data acquisition, data selection, model optimization, and inference refinement, with autonomous evaluation running throughout.

   Abstract PDF arXiv Code Website BibTeX

   As large language models (LLMs) continue to advance, improving them solely through human supervision is becoming increasingly costly and limited in scalability. As models approach human-level capabilities in certain domains, human feedback may no longer provide sufficiently informative signals for further improvement. At the same time, the growing ability of models to make autonomous decisions and execute complex actions naturally enables abstractions in which components of the model development process can be progressively automated. Together, these challenges and opportunities have driven increasing interest in self-improvement, where models autonomously generate data, evaluate outputs, and iteratively refine their own capabilities. In this paper, we present a system-level perspective on self-improving language models and introduce a unified framework that organizes existing techniques. We conceptualize the self-improvement system as a closed-loop lifecycle, consisting of four tightly coupled processes: data acquisition, data selection, model optimization, and inference refinement, along with an autonomous evaluation layer throughout the process. Within this framework, the model itself plays a central role in driving each stage: collecting or generating data, selecting informative signals, updating its parameters, and refining outputs, while the autonomous evaluation layer continuously monitors progress and guides the improvement cycle across stages.

   @article{yang2026selfimprovement,
     title = {Self-Improvement of Large Language Models: A Technical Overview and Future Outlook},
     author = {Yang, Haoyan and Xerri, Mario and Park, Solha and Zhang, Huajian and Feng, Yiyang and Kogilathota, Sai Akhil and Zhou, Jiawei},
     journal = {Transactions on Machine Learning Research},
     issn = {2835-8856},
     year = {2026},
     month = sep,
     url = {https://openreview.net/forum?id=7Sc51w6mIC},
     note = {Survey Certification},
   }

2. 

   arXiv 2026

   FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline

   Haotian Wu*, Shufan Jiang*, Chios Chen, Yiyang Feng, Hehai Lin, Heqing Zou, Yao Shu, and Chengwei Qin

   arXiv preprint, 2026

   A multi-agent pipeline that builds customizable role-playing benchmarks. With it, reasoning models score higher on role-playing but also hallucinate more.

   Abstract PDF BibTeX

   As large language models (LLMs) advance in role-playing (RP) tasks, existing benchmarks quickly become obsolete due to their narrow scope, outdated interaction paradigms, and limited adaptability across diverse application scenarios. To address this gap, we introduce FURINA-Builder, a novel multi-agent collaboration pipeline that automatically constructs fully customizable RP benchmarks at any scale. It enables evaluation of arbitrary characters across diverse scenarios and prompt formats, as the first benchmark builder in RP area for adaptable assessment. FURINA-Builder simulates dialogues between a test character and other characters drawn from a well-constructed character-scene pool, while an LLM judge selects fine-grained evaluation dimensions and adjusts the test character’s responses into final test utterances. Using this pipeline, we build FURINA-Bench, a new comprehensive role-playing benchmark featuring both established and synthesized test characters, each assessed with dimension-specific evaluation criteria. Human evaluation and preliminary separability analysis justify our pipeline and benchmark design. We conduct extensive evaluations of cutting-edge LLMs and find that o3 and DeepSeek-R1 achieve the best performance on English and Chinese RP tasks, respectively. Across all models, established characters consistently outperform synthesized ones, with reasoning capabilities further amplifying this disparity. Interestingly, we observe that model scale does not monotonically reduce hallucinations. More critically, for reasoning LLMs, we uncover a novel trade-off: reasoning improves RP performance but simultaneously increases RP hallucinations. This trade-off extends to a broader Pareto frontier between RP performance and reliability for all LLMs. These findings demonstrate the effectiveness of FURINA-Builder and the challenge posed by FURINA-Bench.

   @misc{wu2026furina,
     title = {{FURINA}: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline},
     author = {Wu*, Haotian and Jiang*, Shufan and Chen, Chios and Feng, Yiyang and Lin, Hehai and Zou, Heqing and Shu, Yao and Qin, Chengwei},
     year = {2026},
     month = apr,
     eprint = {2510.06800},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
     url = {https://arxiv.org/abs/2510.06800},
   }

3. 

   ACL 2025

   Conditional Dichotomy Quantification via Geometric Embedding

   Shaobo Cui, Wenqing Liu, Yiyang Feng, Jiawei Zhou, and Boi Faltings

   ACL 2025 (Main Conference) Oral

   Measures how strongly two outputs oppose each other under the same context, using complex-valued embeddings (DoGE), with datasets for debate, defeasible inference, and causal reasoning.

   Abstract PDF Code BibTeX

   Conditional dichotomy, the contrast between two outputs conditioned on the same context, is vital for applications such as debate, defeasible inference, and causal reasoning. Existing methods that rely on semantic similarity often fail to capture the nuanced oppositional dynamics essential for these applications. Motivated by these limitations, we introduce a novel task, Conditional Dichotomy Quantification (ConDQ), which formalizes the direct measurement of conditional dichotomy and provides carefully constructed datasets covering debate, defeasible natural language inference, and causal reasoning scenarios. To address this task, we develop the Dichotomy-oriented Geometric Embedding (DoGE) framework, which leverages complex-valued embeddings and a dichotomous objective to model and quantify these oppositional relationships effectively. Extensive experiments validate the effectiveness and versatility of DoGE, demonstrating its potential in understanding and quantifying conditional dichotomy across diverse NLP applications. Our code and datasets are available at https://github.com/cui-shaobo/conditional-dichotomy-quantification.

   @inproceedings{cui-etal-2025-conditional,
     title = {Conditional Dichotomy Quantification via Geometric Embedding},
     author = {Cui, Shaobo and Liu, Wenqing and Feng, Yiyang and Zhou, Jiawei and Faltings, Boi},
     editor = {Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher},
     booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = jul,
     year = {2025},
     address = {Vienna, Austria},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2025.acl-long.383/},
     doi = {10.18653/v1/2025.acl-long.383},
     pages = {7765--7791},
     isbn = {979-8-89176-251-0},
   }

4. 

   AAAI 2025

   Nuance Matters: Probing Epistemic Consistency in Causal Reasoning

   Shaobo Cui, Junyou Li, Luca Mouchel, Yiyang Feng, and Boi Faltings

   AAAI 2025

   Asks whether 21 LLMs rank fine-grained causal intermediates by polarity and intensity consistently with themselves. They mostly do not.

   Abstract PDF arXiv BibTeX

   Previous research on causal reasoning often overlooks the subtleties crucial to understanding causal reasoning. To address this gap, our study introduces the concept of causal epistemic consistency, which focuses on the self-consistency of Large Language Models (LLMs) in differentiating intermediates with nuanced differences in causal reasoning. We propose a suite of novel metrics – intensity ranking concordance, cross-group position agreement, and intra-group clustering – to evaluate LLMs on this front. Through extensive empirical studies on 21 high-profile LLMs, including GPT-4, Claude3, and LLaMA3-70B, we have favoring evidence that current models struggle to maintain epistemic consistency in identifying the polarity and intensity of intermediates in causal reasoning. Additionally, we explore the potential of using internal token probabilities as an auxiliary tool to maintain causal epistemic consistency. In summary, our study bridges a critical gap in AI research by investigating the self-consistency over fine-grained intermediates involved in causal reasoning.

   @inproceedings{cui2024nuancemattersprobingepistemic,
     title = {Nuance Matters: Probing Epistemic Consistency in Causal Reasoning},
     author = {Cui, Shaobo and Li, Junyou and Mouchel, Luca and Feng, Yiyang and Faltings, Boi},
     booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
     volume = {39},
     number = {22},
     month = apr,
     year = {2025},
     pages = {23715--23723},
     url = {https://ojs.aaai.org/index.php/AAAI/article/view/34542},
     doi = {10.1609/aaai.v39i22.34542},
   }

5. 

   ACL 2024

   Exploring Defeasibility in Causal Reasoning

   Shaobo Cui, Lazar Milikic, Yiyang Feng, Mete Ismayilzada, Debjit Paul, Antoine Bosselut, and Boi Faltings

   Findings of ACL 2024

   delta-CAUSAL: about 11K cause-effect pairs with supporters and defeaters that strengthen or weaken them, and CESAR, a causal-strength metric that tracks those changes.

   Abstract PDF arXiv BibTeX

   Defeasibility in causal reasoning implies that the causal relationship between cause and effect can be strengthened or weakened. Namely, the causal strength between cause and effect should increase or decrease with the incorporation of strengthening arguments (supporters) or weakening arguments (defeaters), respectively. However, existing works ignore defeasibility in causal reasoning and fail to evaluate existing causal strength metrics in defeasible settings. In this work, we present delta-CAUSAL, the first benchmark dataset for studying defeasibility in causal reasoning. delta-CAUSAL includes around 11K events spanning ten domains, featuring defeasible causality pairs, namely, cause-effect pairs accompanied by supporters and defeaters. We further show that current causal strength metrics fail to reflect the change of causal strength with the incorporation of supporters or defeaters in delta-CAUSAL. To this end, we propose CESAR (Causal Embedding aSsociation with Attention Rating), a metric that measures causal strength based on token-level causal relationships. CESAR achieves a significant 69.7% relative improvement over existing metrics, increasing from 47.2% to 80.1% in capturing the causal strength change brought by supporters and defeaters. We further demonstrate even Large Language Models (LLMs) like GPT-3.5 still lag 4.5 and 10.7 points behind humans in generating supporters and defeaters, emphasizing the challenge posed by delta-CAUSAL.

   @inproceedings{cui2024deltacausal,
     title = {Exploring Defeasibility in Causal Reasoning},
     author = {Cui, Shaobo and Milikic, Lazar and Feng, Yiyang and Ismayilzada, Mete and Paul, Debjit and Bosselut, Antoine and Faltings, Boi},
     editor = {Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek},
     booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
     month = aug,
     year = {2024},
     address = {Bangkok, Thailand},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.findings-acl.384/},
     pages = {6433--6452},
   }