Agents and multi-agent systems (MAS) have evolved from
autonomous problem-solving entities to integrating generative
AI and LLMs, ultimately leading to cooperative AI frameworks
that enhance adaptability, scalability, and collaboration. 
**Main Takeaways**
* Multi-agent systems have evolved from rule-based autonomy to cooperative AI,
emphasizing collaboration, negotiation, and ethical alignment.
* The rise of Agentic AI, driven by LLMs, introduces new opportunities for flexible
decision-making but raises challenges in efficiency and complexity.
* Integrating cooperative AI with generative models requires balancing
adaptability, transparency, and computational feasibility in multi-agent
environments 

**Context & History** 
The field of multi-agent systems emerged in the late 1980s/early 1990s,
with its main influences coming from two disparate areas [1,2]. One was the
field of AI robotics, which had begun to seriously address the issue of integrated agent architectures: how do we assemble several cognitive components of intelligence (planning, reasoning, learning, vision,…) into an integrated computational agent? The second was the nascent area of distributed AI, which studied how multiple AI systems could be made to solve problems cooperatively, by dynamically sharing information and tasks. By the mid1990s, these ideas had given rise to the new field, driven by the vision of having (semi)autonomous AI systems – agents – working on behalf of individual users in pursuit of their users’ goals, possibly interacting with other such agents in order to do so. A key insight
was that, since delegated goals might not necessarily be in harmony, it would be necessary to equip such agents with the ability to reason socially. Thus, while AI historically emphasised components of intelligence such as reasoning and problem-solving, the new field of multi-agent systems emphasised social skills such as cooperation, coordination, argumentation and negotiation. In order to underpin those skills, the development of models of Theory of Mind became central to the field. 
By the late 1990s, the field had its own conferences and journal, and was
firmly established as a key sub-field of AI. The area flourished from the late
1990s, with enormous energy devoted to (e.g.) communication languages
for autonomous agents, protocols for cooperation, coordination, and
negotiation, and the underpinning theory of these social skills. With
respect to the latter, while in the early years the practical reasoning paradigm of AI planning had been the dominant influence on the theory of multiagent systems, by the early part of this century, game theory had become the dominant theoretical foundation. Game theory, which emerged from the field of economics, is the theory of interaction between self-interested agents. Although originally devised as a tool for studying interactions between humans and human organisations, it nevertheless seemed a natural framework for studying interactions between artificial agents. A huge body of work emerged, studying (for example), how auctions might be used to allocate scarce resources, the theory
of negotiation between self-interested artificial agents, and how agents might optimally form teams to solve problems and share the associated benefits of cooperation. Interestingly, although learning in multi-agent systems was a key component of the field from the outset, it was not the centre of attention within the field in the first decade or so. 
This initial boom period for multiagent systems lasted roughly from
the mid 1990s to around 2010-15. By the end of that time, though,
some uncomfortable questions were beginning to be asked. While the field
had generated impressive quantities of scientific results, applications seemed to be thin on the ground. For sure there were some high-profile applications. The field of security games, which emerged from multi-agent systems, used ideas from game theory to allocate scarce security resources to defend high-profile targets such as airports. This work led to deployed
applications at US airports and ports. Automated high-frequence trading
systems, which plan and execute the bulk of trades on the world’s markets,
are multi-agent systems on a global scale. And agent-based modelling,
which models socio-technical systems at the level of individual decisionmakers, received a huge boost after the 2008 financial crisis, and again after the 2020 COVID-19 pandemic, where it was demonstrated to be an important tool for modelling the spread of contagion: financial in the first case; epidemiological and social policy in the second. But for all these successes, the core vision of multi-agent systems - where agents function in the context of other agents is an active area of research, exploring social concepts such as norms, organisations, practices and also values, is ongoing work, but for a large part outside the AAMAS community, but within social simulation research, and with results informing and
shaping policy-making in several areas from public health to transportation,
and urban transformation. 
While applications of multi-agent systems research (AI agents interacting
with other AI agents) has not, as yet, lived up to early expectations,
individual dialogue agents such as Alexa, Siri, Cortana are now an everyday
reality, and trace their historical roots both to work on intelligent agents
in the 1990s and work from the NLP community on dialog systems. Many
other applications of this thread of work have achieved success over the past three decades: automated call center assistants, customer service assistants, smartphone virtual assistants, smart speaker assistants, home robot assistants, that can converse with human users and accomplish tasks like ticket booking, restaurant reservations, online shopping, medical and health assistance, and sales assistance, empowered by AI modules like
speech recognition, natural language understanding, dialog management (i.e. state tracking and dialog policy), and natural language generation and speech synthesis. 
As ML surged in the early part of this century, activity in this area increased
within the multi-agent systems field. Multi-agent reinforcement
learning (MARL) grew to become the single biggest area within the field,
possibly driven in part by the fact that developing MARL experiments can be done relatively quickly and without recourse to expensive hardware. At the time of writing, while MARL represents a significant sub-field of ML as a whole, it seems to lack any clear unifying vision or direction – or application. 
**Current State & Trends** 
The emergence of LLMs from 2020 onwards has also led to increased
interest in agents [3]. LLMs can be used as part of a workflow to
automate routine tasks, and the general capabilities of such “agents” for
planning and problem solving is widely discussed. In this context, the concept of Agentic AI refers to the integration of generative AI and LLMs into autonomous agent frameworks aiming to leverage the generative capabilities of such models to enhance interaction, creativity, and real-time decision making in dynamic environments. As we write this (late 2024) there has been an explosion of startup companies hoping to commercialise such agents. Despite this renewed enthusiasm, the original aims of AAMAS from 30 years ago, such as building robust, autonomous multiagent systems capable of complex coordination and long-term reasoning, have not been fully realized. The extent to which this new wave of agent activity is informed by what went before is unclear. 
The challenge now is to understand what multi-agent systems mean in
the era of LLMs. The current direction of agentifying LLMs may lead to
overly complex and unnecessary architectures and heavy computational
costs, whereas adopting a multi-agent paradigm to the development and use of LLMs may offer a sustainable way to compose, diversify, and integrate approaches effectively. Even though distribution was one of the original drivers for the MAS field, this is still a largely unexplored direction under the current paradigm. Another trend nowadays is to recover ideas from classical cognitive architectures to add common sense skills to autonomous agents. 
An emerging trend is multi-agent architectures, which structure
AI components into modular systems that improve transparency,
adaptability, and ethical alignment. The focus on cooperative agents
highlights a shift toward AI that prioritizes collaboration, negotiation, and shared decision-making. By applying modularity, encapsulation,
and separation of concerns, these architectures enable scalable teamwork
between autonomous agents and humans, making them ideal for
hybrid AI applications requiring trust, explainability, and domain-specific
expertise. 
**Research Challenges** 
* Identify challenges and benefits of embedding GenAI-driven agents
into MAS, focusing on enhancing collaboration without disrupting
existing dynamics.
* Investigate how LLM-powered agents can improve negotiation and
decision-making in dynamic multiagent environments while ensuring
ethical alignment and safety.
* Develop architectures that integrate LLM-driven agents while maintaining
scalability, transparency, and computational efficiency in multiagent settings. 
**Community Opinion** 
The survey responses indicate a
majority of respondents finding this
theme relevant to their research, with
a growing interest in integrating Large
Language Models (LLMs) into multiagent systems. Many participants
already use AI agents, with LLMs being
the most common technique (29.34%),
highlighting their expanding role in AI driven applications. 
The potential of multi-agent systems
leveraging LLMs is seen in areas such
as collaborative problem-solving
(68.86%), distributed decision-making
(54.49%), and social simulations
(41.32%). However, challenges persist,
including misalignment between LLMs’
general knowledge and specific system
needs (59.88%), lack of interpretability
(59.28%), and security risks (50.90%).
These concerns suggest a need for
improved explainability, alignment
strategies, and robust security measures
to ensure effective deployment. 
There is also a debate on the necessity
of agentifying LLMs—while 51.5%
believe multi-agent LLM paradigms
are essential for sustainable AI,
42.33% disagree that they introduce
unnecessary complexity. The
computational cost-benefit balance
remains uncertain, with responses
divided on whether LLMs outweigh
their costs. 
The textual responses highlight a broad
spectrum of perspectives on integrating
Large Language Models (LLMs) into
multi-agent systems (MAS), with some
advocating for hybrid approaches
rather than relying solely on LLMs. Many
respondents stressed the need for
diverse AI architectures, emphasizing
modular, multi-technology systems
where LLMs play a role but do not
dominate. Governance, coordination,
and adaptability emerge as key
advantages of MAS, while concerns
include increased complexity, lack
of theoretical guarantees, and high
computational costs. Several responses
criticize the overemphasis on LLMs,
questioning whether they are truly
essential or merely a current trend.
Others highlight practical challenges
such as grounding, alignment, and
robust communication protocols,
pointing out the need for new
frameworks that integrate symbolic
reasoning, structured governance,
and scalable architectures. Overall,
the discussion reflects a critical but
open stance toward agentifying LLMs,
suggesting that context, application
domain, and technological diversity will
shape their effectiveness in multi-agent
environments. 
In summary, the survey reflects
optimism about LLM-driven multiagent systems, but also underscores
the need for addressing key challenges
before widespread adoption.
 
1. Yoav Shoham. Agent-oriented programming. Artificial Intelligence. Artificial Intelligence. 60 (1): 51–92.
2. Michael Wooldridge. An Introduction to Multi-agent Systems 2e. Wiley, 2009.
3. Julia Wiesinger, Patrick Marlow and Vladimir Vuskovic. Agents. Google whitepaper. https://archive.org/details/google-ai-agents-whitepaper

AI agents

The ability to reason has been a salient characteristic of human
intelligence, and there is a critical need for verifiable reasoning
in AI systems. 
**Main Takeaways** 
* Reasoning has always been seen as a core characteristic of human intelligence.
Reasoning is used to derive new information from given base knowledge; this
new information is guaranteed correct when sound formal reasoning is used,
otherwise it is merely plausible.
* AI research has led to a range of automated reasoning techniques. These
reasoning techniques have given rise to AI algorithms and systems, including
SAT, SMT, and constraints solvers as well as probabilistic graphical models, all of
which play a key role in critical real-world applications.
* While large pre-trained systems (such as LLMs) have made impressive
advancements in their reasoning capabilities, more research is needed to
guarantee correctness and depth of the reasoning performed by them; such
guarantees are particularly important for autonomously operating AI agents 

**Context & History** 
Reasoning is a core component of
human intelligence. From the dawn
of humanity, abductive reasoning
has been used to predict danger and
inductive reasoning made it possible
to learn regularities governing the
world. Beginning in Ancient Greece,
deductive reasoning techniques were
developed to draw valid conclusions
that follow logically from premises
known to be true. The development
of reasoning methods with such a
priori guarantees was a key factor in
the advancement of modern science,
mathematics, and engineering; notably,
according to philosophers such as
Charles Sanders Peirce, the interplay
between abduction, deduction, and
induction forms the basis of the
scientific method and hence all modern
science. Attempts to mechanize
logical reasoning can be traced back to
13th-century philosopher Ramon Lull
and lie at the heart of the concept of
computation. Probabilistic reasoning
and inference have also profoundly
impacted reasoning, often relying on
the celebrated theorem by Thomas
Bayes on inverse probability that also
forms the basis for many machine
learning and statistics approaches.
Finally, the evaluation of correct (sound)
reasoning lies at the heart of most
quantitative assessments of human
cognition. 
Not surprisingly, reasoning has been
central to the AI enterprise. Indeed,
the earliest research in AI – from Logic
Theorist onwards [1] – had a strong
focus on reasoning [2]. Since the 1960s,
AI has also embraced probabilistic
reasoning and models, initially for
medical diagnosis [3]. Since then,
the reasoning tasks addressed in AI 
research have covered the gamut from
planning and temporal reasoning to
diagnosis and explanation. While early
AI has paid attention to both plausible
reasoning (case-based, analogical,
qualitative) and sound formal reasoning
with guarantees (logical, probabilistic,
constraint-based), over the years,
the focus has shifted more towards
reasoning with formal guarantees.
There are good reasons for this when
designing AI systems and techniques
that compensate for human limitations
and weaknesses since reasoning
with guarantees is challenging for
humans. This has led to practically
impactful applications of AI systems
such as SAT, SMT, and constraints
solvers, including the verification of
correctness properties of computer
hardware and software, the safety of
communications protocols, the design
of new proteins, and, more recently, the
robustness of neural networks against
adversarial attacks. It has also resulted
in probabilistic graphical models [4,
5], which are powerful modeling and
inference tools that have found their
way into numerous applications of
reasoning in medicine, robotics, and
beyond. 
**Current State & Trends** 
The emergence of the Internet and the
associated technology that made it
possible to capture the human digital
footprint at scale, as well as the leaps in
computing power, have made possible
novel approaches to learning bottomup from data. Of particular interest are
large pre-trained models, such as LLMs,
that have shown surprising abilities in
plausible reasoning. Unlike the earlier
research on reasoning in AI, LLMs
have focused on plausible reasoning patterns as they emerge automatically
after large-scale training on petabyte
corpora. While the results have been
quite remarkable so far, the reasoning in
this context has been of the “plausible”
variety with no guarantees. 
Meanwhile, sound formal reasoning
techniques remain key to important
and impactful applications of cuttingedge AI technology for the verification
of computer hardware and software,
as well as for real-world planning and
resource allocation problems. They
are also increasingly recognized as a
crucial basis for the formal verification
of machine learning techniques such
as neural networks, e.g., in the context
of local robustness against adversarial
attacks [6]. Significant research activity
takes place in these areas, focusing on
improving various types of reasoning
algorithms (notably with respect to their
computational complexity), leveraging
learning within sound formal reasoning,
and combining reasoning and learning
techniques [7, 8]. 
**Research Challenges** 
Bringing some of the rigorous a priori or
post hoc guarantees back into plausible
reasoning patterns turbocharged by
the pre-trained models has become an
active and promising area of research
– especially where AI systems need to
work autonomously in safety-critical
domains. Research on so-called “large
reasoning models” as well as on neurosymbolic approaches is addressing
these challenges. 
Furthermore, even though formal
reasoning with correctness guarantees
is currently considerably less in vogue
than the use of generative AI techniques
for plausible reasoning, formidable and essential challenges also remain in that
area. In this context, the combination
of machine learning techniques with
formal reasoning techniques holds
considerable promise for economically
and socially valuable breakthroughs,
notably in the area of AI safety and
transparency.
The questions and challenges we face
range from the philosophical:
* What exactly is “reasoning”?

to the practical:
* Can LLM ‘reasoning’ be trusted?

and include: 
What does the future hold for the
advancement and role of symbolic
reasoning?
* To what extent can LLMs or other
generative models reproduce or
replace symbolic reasoning?
* To what degree will symbolic
reasoning be necessary or sufficient
to overcome the current limitations
of LLMs?
* How well can AI reasoning,
especially LLM ‘reasoning,’ be
explained and understood?
* How can computers better
understand and simulate human
reasoning?
* What is the role of collaborative
reasoning between humans and
computers?
* How best can LLMs and symbolic
reasoning be integrated into “neurosymbolic reasoning”?
* Are further breakthroughs, beyond
both LLMs and traditional symbolic
reasoning, required to achieve AGIlevel reasoning?
* What forms of reasoning can best
support humans when dealing with
various challenges, e.g., in medical,
scientific, engineering, and legal
domains? 

**Community Opinion** 
The AAAI community appears to
strongly agree on the importance
of reasoning in AI systems. In our
community survey, slightly over 55%
of the respondents chose to answer
specific questions related to the topic
of reasoning. Of these, 79% indicated
that the topic of reasoning is relevant
to their research (with 44.7% marking
it as “very relevant”). Of the properties
required for referring to a process
as reasoning, 77.5% of the survey
participants marked “Knowledge can
be incorporated”, 72.5% “Explanations
can be provided,” and 56.9% “Involves
multiple steps to arrive at a conclusion”.
Interestingly, merely 37.4% indicated
“Guaranteed correctness of inference
results/outcomes”, and only 23.7% that
“A formal system and solver is used,”
which reflects the recent focus on
informal, plausible reasoning, likely in
the context of generative AI methods.
This suggests that an effort may be
warranted to better communicate the
importance and success of formal,
sound reasoning techniques. Finally,
44.7% of respondents agreed that
“Reasoning involves a search process.” 
There was broad agreement among
survey participants that focusing
reasoning research in AI on humanlevel reasoning is valuable (41.6%) or
even essential (47%); similarly, a focus
on domain-specific reasoning abilities
was seen by 49.6% of respondents as
valuable, and by 42.8% as essential.
This clearly reflects the importance
attributed to a research focus on
reasoning. 
The community also sees an exciting
potential of synergy offered by logical
and probabilistic models of reasoning
that were developed in AI prior to large
pre-trained models. This is clearly
reflected in the fact that 76.9% of survey
participants marked the integration
of learning and reasoning approaches
as very important (6 or 7 on a scale
of 7); interestingly, the percentage
of respondents that considered
Explainability and verifiability as very
important was similarly high (at 71.7%). 
Finally, 61.8% of survey participants
estimated the minimal percentage of
symbolic AI techniques required for
reaching human-level reasoning to be
at least 50% (with 24.8% estimating it
at 75% or more, compared to 38.2%
estimating it at 25% or below). What
remains unclear is the degree to
which AI researchers and practitioners
realize that decidedly superhuman
levels of reasoning are required for
and displayed in the prominent and
successful applications of formal AI
reasoning techniques for scientific
and mathematical discovery and
engineering applications, as well as in AI
safety. 
1. Newell, A. & Simon, H. (1956). The logic theory machine: A complex information processing system. IRE Transactions on Information Theory 2: 61-79. 
2. Brachman, R. and Levesque, H. (2004) Knowledge Representation and Reasoning (1st Ed). Morgan Kaufman. 
3. Russell, S. J., & Norvig, P. (2016). Artificial intelligence: a modern approach (4th Ed). Pearson. 
4. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufman. 
5. Koller, D. and Friedmann, N. (2009) Probabilistic Graphical Models. The MIT Press. 
6. König, M. et al. (2024) Critically Assessing the State of the Art in Neural Network Verification. Journal of Machine Learning Research 25(12): 1-53 
7. Guo, D. et. al. (2025) DeepSeek-R: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2501.12948 
8. Kambhampati, S. (2024). Can Large Language Models Reason and Plan? Annals of New York Academy of Sciences. March 2024.

AI Reasoning

Improving factually and trustworthiness of AI systems is the
single largest topic of AI research today, and while significant
progress has been made, most scientists are pessimistic that
the problems will be solved in the near future. 
**Main Takeaways** 
* An AI system is factual if it refrains from outputting false statements. Improving
factually of AI systems based on neural-network large language models is
arguably the biggest area of AI research today.
* Trustworthiness extends trustworthiness to include criteria such as human
understandability, robustness, and the incorporation of human values. Lack of
trustworthiness has always been an obstacle for deploying AI systems in critical
applications.
* Approaches to improving the factuality and trustworthiness of AI systems
include fine-tuning, retrieval-augmented generation, verification of machine
outputs, and replacing complex models with simple understandable models. 

**Context & History** 
A factual AI system does not output
erroneous information or hallucinate
answers. Before the era of generative
AI, problems with factuality arose when
systems were trained on bad data,
as captured by the slogan, “garbage
in, garbage out”. Work on methods
for improving data quality has a long
history in AI [1]. 
Generative AI, and in particular
large language models, employ
reconstructive memory - that is, they
rebuild memories as needed on the
basis of distributed bits of information
rather than retrieve memories from
a fixed store. The earliest generative
LLMs made an impact with their ability
to generate coherent but entirely
imaginary stories [2]. Factuality of LLMs
on a given domain was improved by
fine-tuning the model on domain data
[3]. 
Trustworthiness is a broader concept
than factuality because it includes
criteria such as understandability,
robustness, and respect of human
values. A traditional approach for
improving understandability of AI
systems is to replace complex blackbox models with simple humanunderstandable models - such as
naive Bayes [4] or generalized linear
regression [5]. Research on robustness
of machine learning studies how the
outputs of a model vary with small
changes in its training data. For
example, contrastive learning is a
method to train deep neural nets with
increased robustness [6]. Further
discussion of robustness in generative
AI appears in
this report’s section on Reasoning.
Discussion of respect of human values
by AI systems appears in many other
sections of this report and so will not be
discussed here. 

**Current State & Trends** 
As noted, fine-tuning remains the
main approach used by scientists
and engineers to improve factuality
of generative AI systems. In addition
to fine-tuning on domain-specific
documents, modern fine-tuning
includes reinforcement learning with
human feedback from thousands of
people. The cost of employing such
large numbers of human evaluators
is a major bottleneck for scaling AI
systems, and so there is much interest
in discovering methods to reduce the
amount of human feedback needed [7]. 
The second main technique for
improving factuality of generative AI
is retrieval-augmented generation
(RAG) [8]. In response to a question,
the system gathers a set of relevant
documents using traditional
information retrieval algorithms. The
AI system is then prompted to generate
an answer by combing through and
summarizing the retrieved documents.
While RAG can improve factuality, it
is dependent on the quality of data
retrieved. For example, if the target
document set is the entire web, it
can end up incorporating incorrect
information and even satirical stories in
its answer. 
Related to RAG is enabling the
generative AI system to use tools for
fact checking. Tools used by generative
AI systems include calculators, factual
databases such as citation indexes, and
formal planning and reasoning systems
[9]. A recent approach to improving
factuality is to provide the system with
a set of rules that state constraints on
space of answers. The output from the
model is verified against these rules and
inconsistent responses are culled [10].
Amazon Web Services already supports
this approach with “automated
reasoning checks” [11]. 
A third technique for improving
factuality of generative AI is chainof-thought (CoT), where a series of
prompts breaks down a question into
smaller units [12]. CoT often includes
steps where the model is asked to
reflect back on its tentative conclusions
and see if any are hallucinations. CoT
is discussed in more detail in the
Reasoning section of this report. 
The impact of data quality on factuality
was mentioned above. In addition
to fine-tuning on human curated
data, there is recent work on creating
synthetic data that is guaranteed to be
high quality for fine-tuning [13]. 
Trustworthiness, we noted, generalizes
factually and includes understandability
and robustness. One approach to
making neural network models more
understandable is to factor them into a
set of recognizers for high level features
and then combine the features using an
understandable model such as additive
regression [15]. Another approach is to
tease out how concepts and rules are
actually represented in a trained [16].
Understandability can also be
improved by employing CoT techniques
to ask a generative AI system to explain
the steps in its reasoning [17] or tell
the user when the system is uncertain
about a conclusion [18]. Finally, a
generative AI system can be asked not
to output a single answer, but instead
to distill a complex set of information into a simple human understandable
representation such as a decision tree
[19]. 
**Research Challenges** 
Factuality is far from solved. There
are a growing number of benchmark
dataset designed to test the factuality
of LLMs. One of the latest, SimpleQA
from Google, is a collection of simple,
unambiguous, timeless, and challenging
factual questions and answers
[14]. As of December 2024, the best
models from OpenAI and Anthropic
correctly answered less than half of the
questions. 
Robustness in generative AI can
be improved, as noted above, by
employing robust loss functions such
as contrastive learning. Adversarial
training, which applies perturbations in
the embedding space during training,
can improve both robustness and
generalization [20]. In addition, the
techniques for factuality generally
improve robustness as well. 
**Community Opinion** 
Over 75% of the AAAI community
strongly agreed that factuality and
trustworthiness were relevant or very
relevant to their own research. 
All six approaches mentioned in
the survey for improving factuality - external fact-checking tools,
reinforcement, improving data quality,
data curation, synthetic training, and
new neural net architectures - found
support. The greatest demand was
for more research on new neural net
architectures (73% marked important
or very important), followed closely by
external fact-checking tools (70%). 
For trustworthiness, new neural
net architectures were also viewed
as most important (77% important
or very important), followed by
enabling models to describe their
reasoning processes (70%) and use
of understandable models instead of
neural networks (61%). Notably, few
viewed research on giving AI systems
human-like personalities as important
to improving trustworthiness (24%).
Finally, the community mostly agreed
(59%) that trustworthiness as currently
formulated is ill-defined. Most also
disagreed (around 60%) that either
factuality or trustworthiness would
soon be solved. 
The AAAI community suggested
a number of additional aspects of
factuality and trustworthiness that were
not covered above. These included:
* The ability to understand and
present different sides of the same
issues, including pros and cons for
each one.
* Understanding that trustworthiness
depends upon the context of the
domain, organizational objectives,
and user objectives. It is wrong
to think of an AI system as simply
being trustworthy or untrustworthy
without regard to context.
* Transparency needs to go beyond
the models used to the actual
sources of training data. This should
include multi-source verification of
facts ingested by the models.
* The focus of work should be
on risk and mitigation rather
than on “solving” factuality and
trustworthiness.
* Attention needs to paid to giving
AI agents the ability to update
their knowledge while maintaining
reliability.
 

1. Budach, Lukas, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Sina Noack, Hendrik Patzlaff, Hazar Harmouch and Felix Naumann (2022). The Effects of Data Quality on Machine
Learning Performance. https://arxiv.org/pdf/2207.14529
2. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI. Retrieved from https://cdn.openai.com/betterlanguage-models/language_models_are_unsupervised_multitask_learners.pdf
3. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. https://doi.org/10.48550/
arXiv.1810.04805
4. Pedro Domingos & Michael Pazzani (1997). On the Optimality of the Simple Bayesian Classifier under Zero-One Loss. Machine Learning, 29(2-3), 103–130. https://doi.
org/10.1023/A:1007413511361
5. Caruana, Rich, Yin Lou, Johannes Gehrke, Paul Koch, M. Sturm and Noémie Elhadad (2015). Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day
Readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015). https://people.dbmi.columbia.edu/noemie/
papers/15kdd.pdf
6. Hadsell, R., Chopra, S., & LeCun, Y. (2006). Dimensionality reduction by learning an invariant mapping. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision
and Pattern Recognition (CVPR 2006), 2, 1735–1742. https://doi.org/10.1109/CVPR.2006.100
7. Hu, C., Hu, Y., Cao, H., Xiao, T., & Zhu, J. (2024). Teaching language models to self-improve by learning from language feedback. Findings of the Association for Computational Linguistics
(ACL 2024). Retrieved from https://aclanthology.org/2024.findings-acl.364/
8. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for
knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems (NeurIPS 2020), 33, 9459–9474. Retrieved from https://arxiv.org/abs/2005.11401
9. Guan, L., Valmeekam, K., Sreedharan, S., & Kambhampati, S. (2023). Leveraging pre-trained large language models to construct and utilize world models for model-based task
planning. Proceedings of the 33rd International Conference on Automated Planning and Scheduling (ICAPS 2023). Retrieved from https://arxiv.org/abs/2305.14909
10. Backes, J., Bolignano, P., Cook, B., Dodge, C., Gacek, A., Luckow, K., Rungta, N., Tkachuk, O., & Varming, C. (2018). Semantic-based automated reasoning for AWS access policies using
SMT. In 2018 Formal Methods in Computer-Aided Design (FMCAD) (pp. 1–9). IEEE. https://doi.org/10.23919/FMCAD.2018.8602994
11. Barth, Antje (2024). Prevent factual errors from LLM hallucinations with mathematically sound Automated Reasoning checks (preview). Posted 3 Dec 2024, retrieved 8 Feb 2025. AWS
News Blog, permalink https://aws.amazon.com/blogs/aws/prevent-factual-errors-from-llm-hallucinations-with-mathematically-sound-automated-reasoning-checks-preview
12. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Advances in
Neural Information Processing Systems (Vol. 35, pp. 24824–24837). https://proceedings.neurips.cc/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
13. Ding, Bosheng, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu and Shafiq R. Joty (2024). Data Augmentation using LLMs: Data
Perspectives, Learning Paradigms and Challenges. Annual Meeting of the Association for Computational Linguistics (2024). https://aclanthology.org/2024.findings-acl.97.pdf
14. Wei, Jason, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus (2024). Measuring short-form factuality in large language
models. https://doi.org/10.48550/arXiv.2411.04368
15. Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., & Hinton, G. E. (2021). Neural additive models: Interpretable machine learning with neural nets. Advances in
Neural Information Processing Systems, 34, 2021. https://proceedings.neurips.cc/paper/2021/hash/251bd0442dfcc53b5a761e050f8022b8-Abstract.html?utm_source=chatgpt.com
16. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Tamkin,
A., Durmus, E., Hume, T., Mosconi, F., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., & Henighan, T. (2024). Scaling Monosemanticity: Extracting
Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
17. Yeo, W. J., Ng, X. X., Le, T. K. C., & Lu, X. (2024). How interpretable are reasoning explanations from prompting large language models? Findings of the Association for Computational
Linguistics: NAACL 2024. Retrieved from https://aclanthology.org/2024.findings-naacl.138
18. Lin, S., Hilton, J., & Evans, O. (2022). Teaching Models to Express Their Uncertainty in Words. Transactions on Machine Learning Research. https://openreview.net/pdf?id=8s8K2UZGTZ
19. Chen, Y., Zhang, L., Wang, H., & Li, J. (2025). Zero-Shot Decision Tree Construction via Large Language Models. arXiv preprint arXiv:2501.16247. Retrieved from https://arxiv.org/
abs/2501.16247
20.Liu, X., Cheng, H., He, P., Chen, W., Wang, Y., Poon, H., & Gao, J. (2020). Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994. Retrieved from https://
arxiv.org/abs/2004.08994

AI Factuality & Trustworthiness

AI & Sustainability

Artificial General Intelligence (AGI)

How should we challenge exaggerated claims about AI’s
capabilities and set realistic expectations? 
**Main Takeaways** 
* Over the last 70 years, against a background of constant delivery of new and important technologies, many AI innovations have generated excessive hype.
* Like other technologies these hype trends have followed the general Gartner Hype Cycle characterization.
* The current Generative AI Hype Cycle is the first introduction to AI for perhaps the majority of people in the world and they do not have the tools to gauge the validity of many claims 

**Context & History** 
Artificial intelligence, or AI, is the field
that studies the synthesis and analysis
of computational agents that act
intelligently [6]. AI has gone through
hype cycles multiple times since the
1956 workshop that established the
name AI and set the course for early
computer science departments to
include AI as a major component of
their research and teaching. All hype
bubbles eventually burst, as the
essence of hype is that it is beyond
reality. Over the decades this has led to
AI winters where funding has dried up
for all of AI or for specific aspects of AI
such as neural networks or robotics. 
A study published 2017 on trends with
the public perception of AI over a 30-
year period found that discussion about
AI had sharply increased since 2009
and that discussions in the public press
had been consistently more optimistic
than pessimistic [4]. The study also
found that hopes about AI applications
of AI in healthcare and education
were increasing over time. Another
finding was that concerns were growing
about loss of control of AI, ethical
implications, and the negative impact of
AI on work. 
Perhaps the difference in recent years
with prior periods is that hype has
gone beyond the pages of academic
conferences, conference papers, and
scientific magazines, out into both the
mainstream media, and social media.
AI and Artificial Intelligence have
become common words that nontechnical people have heard about
and a common subject for leaders
of almost all countries to talk about.
Governments, for the first time, have AI
policies. 
One of the problems is that AI is
actually a wide-reaching term that can
be used in many different ways. But
now in common parlance it is used
as if it refers to a single thing. In their
2024 book [5] Narayanan and Kapoor
likened it to the language of transport
having only one noun, ‘vehicle’, say, to
refer to bicycles, skate boards, nuclear
submarines, rockets, automobiles, 18
wheeled trucks, container ships, etc.
It is impossible to say almost anything
about ‘vehicles’ and their capabilities
in those circumstances, as anything
one says will be true for only a small
fraction of all ‘vehicles’. This lack of
distinction compounds the problem
of hype, as particular statements get
overgeneralized. 
The hype also sets expectations for
ordinary people. Many are fearful that
they will lose their jobs to AI in the
short term. Social scientists then work
to solve labor disruptions, e.g., for
displaced truck drivers [6], based on
predictions about AI (and in this case,
self-driving trucks) and its adoption that
turn out to be wildly optimistic. There
are no deployed self-driving trucks in
the predicted time frame. 
Hype in response to a technology
trigger is not restricted to AI. Indeed
the business intelligence company
Gartner, has deliberately made
a practice of using a graphical
representation of hype levels through
five stages that are common for many
technologies: (1) technology trigger,
(2) peak of inflated expectations, (3)
trough of disillusionment, (4) slope
of enlightenment, and (5) plateau
of productivity. They have used this
framework to track many technologies,
including quantum computers, block
chain, autonomous vehicles, nanotechnology, etc. In November 2024 they
[1] estimated that hype for Generative
AI had just passed its peak and was on
the downswing. 
The question for AI professionals is
how to respond to this hype, how to
question it, and how to help others
understand what is hubris, while
maintaining their own intellectual
modesty and probity. This is hard to do
in the middle of outsized claims about
one’s own field, and often it is up to
future historians to carefully dissect
past scientific arguments. 
Historian Thomas Haigh has tried to
do such a dissection, almost in real
time, in a recent series of articles in the
Communications of the ACM. In [2] he
gives a post-mortem on the impact of
over-hype in AI that resulted in what is
known as the AI-winter in the 1980s. His
one line summary is: “Fallout from an
exploding bubble of hype triggered the
real AI Winter in the late 1980s.” In [3] he
makes a comparison between the hype
of today and of those earlier times.
He summarizes this particular opinion
piece with the line: “From engines of
logic to engines of bullshit?” 
**Research Challenges** 
Many of us who have worked in AI for
decades face the challenge of trying to
remain honest brokers when we see
that many of the public statements of
people quite new to the field are out of
line with reality. 
The big question is whether, given
the dynamics of social media and the
search for clicks, professional opinions
and peer reviewed research papers
have any impact on dampening the
overclaims and the ways they distort
common understanding of where AI is,
and what is its potential in one year, five
years, ten years, etc. 
If we are currently left out of the
conversations how can we change that? 
**Community Opinion** 
The Community Survey gives
perspectives on the reactions to the AI
Perception vs Reality theme. First, the
results of the survey are summarized
here. 36% of the survey respondents
chose to answer the questions for this
theme. This is the summary breakdown
of the responses to each question: 
**1. How relevant is this Theme for your
own research?** 72% of respondents
said it was somewhat relevant (24%),
relevant (29%) or very relevant (19%). 
**2. The current perception of AI
capabilities matches the reality of AI
research and development.** 79% of
respondents disagreed (47%) or strongly
disagreed (32%). 
**3. In what way is the mismatch
hindering AI research?** 90% of
respondents agreed that it is hindering
research: 74% agreeing that the
directions of AI research are driven by
the hype, 12% saying that theoretical AI
research is suffering as a result, and 4%
saying that less students are interested
in academic research. 
**4. Should there be a communitydriven initiative to counter the hype
by fact-checking claims about AI?** 78%
yes; 51% agree and 27% strongly agree. 
**5. Should there be a communitydriven initiative to organize public
debates on AI perception vs reality,
with video recordings to be made
available to all?** 74% yes; 46% agree
and 28% strongly agree. 
**6. Should there be a communitydriven initiative to build and maintain
a repository of predictions about
future AI’s capabilities, to be checked
regularly for validating their accuracy?**
59% yes; 40% agree and 29% strongly
agree. 
**7. Should there be a communitydriven initiative to educate the public
(including the press and the VCs) about
the diversity of AI techniques and
research areas?** 87% yes; 45% agree
and 42% strongly agree. 
**8. Should there be a communitydriven initiative to develop a method
to produce an annual rating of the
maturity of the AI technology for
several tasks?** 61% yes; 42% agree and
19% strongly agree. 
Since the respondents to this theme
are self-selected (about a third of all
respondents), that bias must be kept
in mind. Of those who responded,
a strong and consistent (though not
completely monolithic) portion felt that
the current perception of AI capabilities
was overblown, that it had a real impact
on the field, and that the field should
find a way to educate people about the
realities.

1. Chandrasekaran, A. [2024]. What’s Driving the Hype Cycle for Generative AI, 2024. https://www.gartner.com/en/articles/hype-cycle-for-genai
2. Haigh, T. [2024]. How the AI Boom Went Bust. Vol. 67 No. 2, pp 22-26.
3. Haigh, T. [2025]. Artificial Intelligence Then and Now. Vol. 68 No. 2, pp 24-29.
4. Fast, E. and Horvitz, E. [2017]. Long-term trends in the public perception of artificial intelligence, AAAI’17: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence,
February 4, 2017, pp 963-969. https://ojs.aaai.org/index.php/AAAI/article/view/10635
5. Narayanan, A. and Kapoor, S. [2024]. AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference. Princeton University Press.
6. Poole, D. L. and Mackworth, A. K. [2023]. Artificial Intelligence: Foundations of Computational Agents, 3rd edition. Cambridge University Press.
7. Wang, S., Mack, E. A., Van Fossen, J. A., Medwid, M., Cotten, S. R., Chang, C. H., Mann, J., Miller, S. R., Savolainen, P.T. and Baker, N. [2023]. Assessing alternative occupations for truck
drivers in an emerging era of autonomous vehicles. Transportation Research Interdisciplinary Perspectives, Vol 19. May. https://doi.org/10.1016/j.trip.2023.100793

AI Perception vs. Reality

As AI becomes increasingly powerful and ubiquitous, it is disrupting skills and displacing workers. NSF’s National AI Institute for Adult Learning and Online Education (AI-ALOE) posits that AI can be part of the solution to the growing problem if we can use AI for reskilling, upskilling, and workforce development at scale. The long-term vision of AI-ALOE is to develop and use AI technologies to enhance the proficiency of online education for all adult learners, using in-person education as a benchmark. The day-to-day mission of AI-ALOE is to conduct responsible research into AI that is grounded in theories of human cognition and learning and derived from the scientific process of learning engineering. I will describe ongoing research at AI-ALOE.

AI for Reskilling, Upskilling, and Workforce Development

Small Data: A New Paradigm for the Next Generation of AI

An agent acts in its world to achieve its objectives. Intelligence allows the agent to make decisions and act. In natural domains, sensing is limited, so acting is gambling. It’s a myth that passive learning and more data are all we need. An agent cannot learn from observations alone. It needs a real body to carry out experiments in its world, testing hypotheses, to determine causation, refining its model of the world’s dynamics. The agent is acting as a scientist: refining its model through experiments and acting appropriately to achieve its objectives. Its objectives depend on its preferences and values, and those of other agents its actions impact. Determining which values to use, and how preferences can be acquired fairly, is a major non-technical challenge. We address three primary questions: What should an agent believe? What should an agent do, given its beliefs, preferences, and abilities? What should the preferences of an agent be? Integrating these issues motivates the design of our latest AI textbook, Artificial Intelligence: Foundations of Computational Agents, (3rd Ed. 2023).

Alan Mackworth is a Professor Emeritus of Computer Science at the University of British Columbia. He works on artificial intelligence with applications in constraint satisfaction, cognitive robotics, assistive technology, hybrid systems and constraint-based agents. He invented the world’s first soccer-playing robots. He has co-authored two books: Computational Intelligence: A Logical Approach (1998) and Artificial Intelligence: Foundations of Computational Agents (2023, 3rd Ed.). Alan co-founded the UBC Cognitive Systems Program, the Centre for AI, Decision-making and Action (CAIDA) and the AI network of BC (AInBC). He has served as President of AAAI, IJCAI and CAIAC. He is a Fellow of AAAI, CAIAC, AGE-WELL, CIFAR and the Royal Society of Canada.

David Poole is a Professor Emeritus of Computer Science at the University of British Columbia. He is known for his work on combining logic and probability, probabilistic inference, relational probabilistic models, statistical relational AI and semantic science. He is a co-author of two AI textbooks (Cambridge University Press, 3rd edition 2023, and Oxford University Press, 1998), and co-author of ” Statistical Relational Artificial Intelligence: Logic, Probability, and Computation”. He is a former chair of the Association for Uncertainty in Artificial Intelligence, the winner of the Canadian AI Association (CAIAC) 2013 Lifetime Achievement Award, and is a Fellow of the Association for the Advancement Artificial Intelligence (AAAI) and CAIAC.

The Essence of Intelligence is Appropriate Action (not thinking, reasoning, learning or language) and other things every student of AI should know

Professor Edward Feigenbaum: a Tribute to and Lecture by a Pioneer of AI on his 90th Birthday

We stand at the dawn of the AI era, a technological revolution poised to be the most consequential of our generation, presenting both unprecedented opportunities and profound challenges. But this promise is shadowed by significant challenges. To build a future we want, we must move beyond the hype and the headlines to confront the most pressing open problems—technical, sociotechnical, and multidisciplinary. This talk will review the rapid progress, dissect challenges ahead, and argue that our greatest task isn’t simply building smarter machines, but fostering the human wisdom to guide them towards a future that is not only intelligent but also equitable, safe, and profoundly human.

Navigating the AI Horizon: Promises, Perils, and the Power of Collaboration

Large language models have revolutionized textual reasoning, yet their ability to act meaningfully in multimodal, real-world environments remains limited. They struggle to ground their decisions in visual context, adapt to changing goals, and plan actions over time—shortcomings that stem from a lack of structured, goal-driven reasoning and insufficient representations of the physical world.

In this talk, I present a unified framework for building embodied agents that can see, simulate, and reason. I begin by introducing methods for learning world simulators from data, arguing that visual reasoning—like textual reasoning—benefits from step-by-step processing. Inverting a physics simulator becomes key: agents must infer structured 3D neural representations of objects, parts, motions, and scenes directly from raw video. I describe methods for extracting such representations using generative priors, injecting them into vision-language models (VLMs), and scaling up their supervision via generative 3D simulation and fast, modular physics engines. These simulators enable agents to anchor their predictions in grounded physical reality, reducing hallucinations and improving control.

Complementing this simulation capability, I explore techniques that enable agents to reason over time and adapt their behavior. By integrating structured memory systems, agents learn to retain and retrieve relevant experiences to inform long-horizon plans. Language-based reflective feedback allows them to refine their strategies beyond what sparse rewards offer, forming abstractions that generalize across tasks. When trained to ground their reasoning directly in the visual environment, agents gain the ability to set subgoals, explore effectively, and verify their own hypotheses.

Together, these advances point toward autonomous systems that simulate before they act, reflect after they fail, and maintain an ongoing awareness of goals, constraints, and context. I will illustrate these capabilities across web automation, robotics, and interactive assistance, showing how agents that see, simulate, and reason offer a promising path toward general-purpose embodied intelligence.

Towards Embodied Agents that See, Simulate, and Reason

From Workflows to Water Coolers: AI That Can Navigate Human Nature

Physicists aim to explain the Universe in terms of a compact, interpretable set of principles. Deducing those principles from experiments poses many challenging and problems which are ripe for application of AI and present opportunities to develop new AI techniques. I will describe how AI has changed the way particle physicists work and speculate about the role of AI in the future of fundamental physics. Finally, I will describe my experience in science communication, as an author, podcaster and television producer.

Fundamental physics and science communication

The pursuit of high-efficiency Artificial General Intelligence (AGI) requires more than brute-force scaling of model size and data. While scaling remains a key driver of capability, equally important are scalable architectural and principles—designs that continue to work, improve, and remain controllable as we vary model scale, domains, and modalities. Central to our approach is the concept of the “Specialized Generalist” – a pathway that achieves deep expertise across multiple domains without sacrificing broad generalization capabilities. In this talk, we introduce the “Specialized Generalist” paradigm and our implementation of it, SAGE (Synergistic Architecture for Generalized Expertise), a three-layer architecture designed to balance specialization and generalization in a systematic way. We will describe how SAGE’s Base Model, Synergy Fusion, and Exploration-Evolution layers interact in practice, focusing on concrete mechanisms for coordinating domain-specific expertise with broad general reasoning. We will share empirical results and recent advances in large reasoning models, embodied AI, and scientific applications to further illustrate the approach. A central motivation is to support “AGI for Science” by building a stable plateau of capabilities that can reliably assist with complex scientific workflows rather than isolated demos. Finally, we will outline the safety and governance questions that arise when deploying Specialized Generalist systems in high-impact settings, and discuss what we have learned so far about monitoring, alignment, and operational safeguards.

Quest of AI towards Specializable Generalist: From Reasoning to Scientific Discovery

There has been a lot of exciting recent progress on new and powerful machine learning algorithms and architectures: how to learn. But for autonomous agents acting in the dynamic, uncertain world, it is at least as important to be able to identify which concepts and subproblems to focus on: what to learn.
This talk presents methods for identifying what to learn within the framework of reinforcement learning, focusing especially on applications in multiagent systems and robotics.

From How to learn to What to learn in Multiagent Systems and Robotics

Large language models (LLMs) have emerged as powerful tools for developing educational systems. While previous studies have explored modeling student mistakes, a critical gap remains in understanding whether LLMs can generate correct solutions that represent student responses to free-response problems. In this paper, we compare the distribution of solutions produced by LLMs with various sampling and prompting techniques and those generated by students, using conversations where students teach math problems to a conversational robot. Our study reveals discrepancies between the correct solutions produced by LLMs and by students. We discuss the practical implications of these findings for the design and evaluation of LLM-supported educational systems.

Can LLMs simulate the same correct solutions to free-response math problems as real students?

The recent rise of reasoning-tuned Large Language Models (LLMs)—which generate chains of thought (CoTs) before giving the final answer—has attracted significant attention and offers new opportunities for gaining insights into human label variation, which refers to plausible differences in how multiple annotators label the same data instance. Prior work has shown that LLM-generated explanations can help align model predictions with human label distributions, but typically adopt a *reverse* paradigm: producing explanations based on given answers. In contrast, CoTs provide a *forward* reasoning path that may implicitly embed rationales for each answer option, before generating the answers. We thus propose a novel LLM-based pipeline enriched with linguistically-grounded discourse segmenters to extract supporting and opposing statements for each answer option from CoTs with improved accuracy. We also propose a rank-based HLV evaluation framework that prioritizes the ranking of answers over exact scores, which instead favor direct comparison of label distributions. Our method outperforms a direct generation method as well as baselines on three datasets, and shows better alignment of ranking methods with humans, highlighting the effectiveness of our approach.

Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation

Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), as it reflects their ability to derive valid conclusions from given premises. While the combination of inference-time scaling in combination with dedicated outcome or process reward models has opened up new avenues to enhance LLMs performance in complex reasoning tasks, this space is under-explored in deductive logical reasoning. In this paper, we present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train the ORMs we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples. Additionally, we propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM. In particular, we propose an echo generation technique that leverages LLMs’ tendency to reflect incorrect assumptions made in prompts to extract additional training data covering previously unexplored error types. While a standard CoT chain may contain errors likely to be made by the reasoner, the echo strategy deliberately steers the model toward incorrect reasoning. We show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO and ProverQA datasets across four different LLMs.

Logical Reasoning with Outcome Reward Models for Test-Time Scaling

Maintaining mutual understanding is a key component in human-human conversation to avoid conversation breakdowns, in which repair, particularly Other-Initiated Repair (OIR, when one speaker signals trouble and prompts the other to resolve), plays a vital role. However, Conversational Agents (CAs) still fail to recognize user-initiated repair requests, leading to breakdowns or disengagement. This work proposes a multimodal approach to automatically detect OIR requests in Dutch dialogues by integrating linguistic and prosodic features grounded in Conversation Analysis. The results show that prosodic cues complement linguistic features and significantly improve the results of pre-trained text and audio embeddings, offering insights into how different features interact. Future directions include incorporating visual cues, exploring large language models (LLMs), and applying the model in CA systems.

Mm, Wat?" Detecting Other-initiated Repair Requests in Dialogue

Large language models (LLMs) show promise in legal question answering (QA), yet Thai legal QA systems face challenges due to limited data and complex legal structures. We introduce ThaiLegal, a novel benchmark featuring two datasets: (1) ThaiLegal-CCL, covering Thai financial laws, and (2) ThaiLegal-Tax, containing Thailand's official tax rulings. Our benchmark also consists of specialized evaluation metrics suited for Thai legal QA. We evaluate retrieval-augmented generation (RAG) and long-context LLM (LCLM) approaches across three key dimensions: (1) the benefits of domain-specific techniques like hierarchy-aware chunking and cross-referencing, (2) comparative performance of RAG components, e.g., retrievers and LLMs, and (3) the potential of long-context LLMs to replace traditional RAG systems. Our results reveal that domain-specific components slightly improve over naive methods. At the same time, existing retrieval models still struggle with complex legal queries, and long-context LLMs have limitations in consistent legal reasoning. Our study highlights current limitations in Thai legal NLP and lays a foundation for future research in this emerging domain.

NitiBench: Benchmarking LLM Frameworks on Thai Legal Question Answering Capabilities

The approaches that guide Large Language Models (LLMs) to emulate human reasoning during response generation have emerged as an effective method for enabling them to solve complex problems in a step-by-step manner, thereby achieving superior performance. However, most existing approaches using few-shot prompts to generate responses heavily depend on the provided examples, limiting the utilization of the model’s inherent reasoning capabilities. Moreover, constructing task-specific few-shot prompts is often costly and may lead to inconsistencies across different tasks. In this work, we introduce Template Oriented Reasoning (TORSO), which elicits the model to utilize internal reasoning abilities to generate proper responses across various tasks without the need for manually crafted few-shot examples. Our experimental results demonstrate that TORSO achieves strong performance on diverse LLMs benchmarks with reasonable rationales.

TORSO: Template-Oriented Reasoning Towards General Tasks

Chain-of-Thought (CoT) reasoning has significantly advanced Large Language Models (LLMs) in solving complex tasks. However, its autoregressive paradigm leads to significant computational overhead, hindering its deployment in latency-sensitive applications. To address this, we propose **DART** (**D**istilling **A**utoregressive **R**easoning to Silent **T**hought), a self-distillation framework that enables LLMs to replace autoregressive CoT with non-autoregressive Silent Thought (ST). Specifically, DART introduces two training pathways: the CoT pathway for traditional reasoning and the ST pathway for generating answers directly from a few ST tokens. The ST pathway utilizes a lightweight Reasoning Evolvement Module (REM) to align its hidden states with the CoT pathway, enabling the ST tokens to evolve into informative embeddings. During inference, only the ST pathway is activated, leveraging evolving ST tokens to deliver the answer directly. Extensive experimental results demonstrate that DART achieves comparable reasoning performance to existing baselines while offering significant efficiency gains, serving as a feasible alternative for efficient reasoning.

DART: Distilling Autoregressive Reasoning to Silent Thought

As AI systems take on collaborative roles, they must reason about shared goals and beliefs—not just generate fluent language. The Rational Speech Act (RSA) framework offers a principled approach to pragmatic reasoning, but existing extensions face challenges in scaling to multi-turn, collaborative scenarios. In this paper, we introduce Collaborative Rational Speech Act (CRSA), an information-theoretic (IT) extension of RSA that models multi-turn dialog by optimizing a gain function adapted from rate-distortion theory. This gain is an extension of the gain model that is maximized in the original RSA model but takes into account the scenario in which both agents in a conversation have private information and produce utterances conditioned on the dialog. We demonstrate the effectiveness of CRSA on referential games and template-based doctor–patient dialogs in the medical domain. Empirical results show that CRSA yields more consistent, interpretable, and collaborative behavior than existing baselines—paving the way for more pragmatic and socially aware language agents.

Collaborative Rational Speech Act: Pragmatic Reasoning for Multi-Turn Dialog

### Abstract Fine-grained understanding and species-specific, multimodal question answering are vital for advancing biodiversity conservation and ecological monitoring. However, existing multimodal large language models (MM-LLMs) face challenges when it comes to specialized topics like avian species, making it harder to provide accurate and contextually relevant information in these areas. To address this limitation, we introduce the **MAviS-Dataset**, a large-scale multimodal avian species dataset that integrates image, audio, and text modalities for over 1,000 bird species, comprising both pretraining and instruction-tuning subsets. Building on the MAviS-Dataset, we present **MAviS-Chat**, an audio-visual multimodal LLM designed for fine-grained species understanding, multimodal question answering, and scene-specific description generation Finally, for quantitative evaluation, we introduce **MAviS-Bench**, a benchmark of over 25,000 QA pairs designed to assess avian species-specific fine-grained perceptual and reasoning abilities of MM-LLMs across different modalities. Experimental results show that MAviS-Chat outperforms the baseline MiniCPM-o-2.6 by a large margin, achieving state-of-the-art open-source results and demonstrating the effectiveness of our MAviS-Dataset. Our results highlight the necessity of domain-adaptive MM-LLMs for ecological applications. *Our code, training data, evaluation benchmark, and models will be publicly released to support future research.*

MAviS: A Multimodal Conversational Assistant For Avian Species

Language use is shaped by pragmatics; i.e., reasoning about communicative goals and norms in context. As language models (LMs) are increasingly used as conversational agents, it becomes ever more important to understand their pragmatic reasoning abilities. We propose an evaluation framework derived from \textit{Wavelength}, a popular communication game where a speaker and a listener communicate about a broad range of concepts in a granular manner. We study a range of LMs on both language comprehension and language production using direct and Chain-of-Thought (CoT) prompting, and further explore a Rational Speech Act (RSA) approach to incorporating Bayesian pragmatic reasoning into LM inference. We find that state-of-the-art LMs, but not smaller ones, achieve strong performance on language comprehension, obtaining similar-to-human accuracy and exhibiting high correlations with human judgments even without CoT prompting or RSA. On language production, CoT can outperform direct prompting, and using RSA provides significant improvements over both approaches. Our study helps identify the strengths and limitations in LMs' pragmatic reasoning abilities and demonstrates the potential for improving them with RSA, opening up future avenues for understanding conceptual representation, language understanding, and social reasoning in LMs and humans.

On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts

With the growing spread of misinformation online, understanding how true news evolves into fake news has become crucial for early detection and prevention. However, previous research has often assumed fake news inherently exists rather than exploring its gradual formation. To address this gap, we propose textbfFUSE (textbfFake news evoltextbfUtion textbfSimulation framtextbfEwork), a novel Large Language Model (LLM)-based simulation approach explicitly focusing on fake news evolution from real news. Our framework model a social network with four distinct types of LLM agents commonly observed in daily interactions: textit spreaders who propagate information, textitcommentators who provide interpretations, textitverifiers who fact-check, and textitstanders who observe passively to simulate realistic daily interactions that progressively distort true news. To quantify these gradual distortions, we develop textbfFUSE-EVAL, a comprehensive evaluation framework measuring truth deviation along multiple linguistic and semantic dimensions. Results show that FUSE effectively captures fake news evolution patterns and accurately reproduces known fake news, aligning closely with human evaluations. Experiments demonstrate that FUSE accurately reproduces known fake news evolution scenarios, aligns closely with human judgment, and highlights the importance of timely intervention at early stages. Our framework is extensible, enabling future research on broader scenarios of fake news:https://anonymous.4open.science/r/FUSE-7022/README.md.

The Stepwise Deception: Simulating the Evolution from True News to Fake News with LLM Agents

Polite speech poses a fundamental alignment challenge for large language models (LLMs). Humans deploy a rich repertoire of linguistic strategies to balance informational and social goals – from positive approaches that build rapport (compliments, expressions of interest) to negative strategies that minimize imposition (hedging, indirectness). We investigate whether LLMs employ a similarly context-sensitive repertoire by comparing human and LLM responses in both constrained and open-ended production tasks. We find that larger models ( ge70B parameters) successfully replicate key preferences from the computational pragmatics literature, and human evaluators surprisingly prefer LLM-generated responses in open-ended contexts. However, further linguistic analyses reveal that models disproportionately rely on negative politeness strategies even in positive contexts, potentially leading to misinterpretations. While modern LLMs demonstrate an impressive handle on politeness strategies, these subtle differences raise important questions about pragmatic alignment in AI systems.

Comparing human and LLM politeness strategies in free production

A Large Language Model (LLM) offers versatility across domains and tasks, purportedly benefiting users with a wide variety of behaviors and preferences. We question this perception about an LLM when users have inherently subjective behaviors and preferences, as seen in their ubiquitous and idiosyncratic browsing of websites or apps. The sequential behavior logs of pages, thus generated, form something akin to each user's self-constructed "language", albeit without the structure and grammar imbued in natural languages. We ask: (i) Can a small LM represent the "language of browsing" better than a large LM? (ii) Can an LM with a single set of parameters (or, single LM) adequately capture myriad users' heterogeneous, subjective behaviors and preferences? (iii) Can a single LM with high average performance, yield low variance in performance to make alignment good at user level? We introduce clusterwise LM training, HeTLM (Heterogeneity aware Training of Language Model), appropriate for subjective behaviors. We find that (i) a small LM trained using a page-level tokenizer outperforms large pretrained or finetuned LMs; (ii) HeTLM with heterogeneous cluster specific set of parameters outperforms a single LM of the same family, controlling for the number of parameters; and (iii) a higher mean and a lower variance in generation ensues, implying improved alignment.

 Subjective Behaviors and Preferences in LLM: Language of Browsing

Large language models (LLMs) continue to face challenges in reliably solving reasoning tasks, particularly tasks that involve precise rule following, as often found in mathematical reasoning tasks. This paper introduces a novel neurosymbolic method that improves LLM reasoning by encoding hidden states into neurosymbolic vectors, enabling problem-solving within a neurosymbolic vector space. The results are decoded and merged with the original hidden state, significantly boosting the model’s performance on numerical reasoning tasks. By offloading computation through neurosymbolic representations, this method enhances efficiency, reliability, and interpretability. Our experimental results demonstrate an average of 88.6% lower cross-entropy loss and 15.4 times more problems correctly solved on a suite of mathematical reasoning tasks compared to chain-of-thought prompting and supervised fine-tuning (LoRA), while not hindering the LLM’s performance on other tasks. We make our code available at https://anonymous.4open.science/r/Neurosymbolic-LLM-A498

Improving Rule-based Reasoning in LLMs using Neurosymbolic Representations

Stance detection, a critical task in Natural Language Processing (NLP), aims to identify the attitude expressed in text toward specific targets. Despite advancements in Large Language Models (LLMs), challenges such as limited interpretability and handling nuanced content persist. To address these issues, we propose the Multi-Path Reasoning Framework (MPRF), a novel framework that generates, evaluates, and integrates multiple reasoning paths to improve accuracy, robustness, and transparency in stance detection. Unlike prior work that relies on single-path reasoning or static explanations, MPRF introduces a structured end-to-end pipeline: it first generates diverse reasoning paths through predefined perspectives, then dynamically evaluates and optimizes each path using LLM-based scoring, and finally fuses the results via weighted aggregation to produce interpretable and reliable predictions. Extensive experiments on the SEM16, VAST, and PStance datasets demonstrate that MPRF outperforms existing models. Ablation studies further validate the critical role of MPRF’s components, highlighting its effectiveness in enhancing interpretability and handling complex stance detection tasks.

MPRF: Interpretable Stance Detection through Multi-Path Reasoning Framework

Multimodal Stance Detection (MSD) aims to determine a user’s stance—support, oppose, or neutral—toward a target by analyzing multimodal content such as text and images from social media. Existing MSD methods struggle with generalizing to unseen targets and handling modality inconsistencies. To address these challenges, we propose the Target-driven Multi-modal Alignment and Dynamic Weighting Model (T-MAD), which combines target-driven multi-modal alignment and dynamic weighting mechanisms to capture target-specific relationships and balance modality contributions. The model incorporates iterative reasoning to iteratively refine predictions, achieving robust performance in both in-target and zero-shot settings. Experiments on the MMSD and MultiClimate datasets show that T-MAD outperforms state-of-the-art models, with optimal results achieved using RoBERTa, ViT, and an iterative depth of 5. Ablation studies further confirm the importance of multi-modal alignment and dynamic weighting in enhancing model effectiveness.

T-MAD: Target-driven Multimodal Alignment for Stance Detection

Transformer encoders are critical for a wide range of Natural Language Processing (NLP) tasks, yet their compute–efficiency remains poorly understood. We present the first comprehensive empirical investigation of compute-optimal pretraining for encoder transformers using the Masked Language Modeling (MLM) objective. Across hundreds of carefully controlled runs we vary model size, data size, batch size, learning rate, and masking ratio, with increasing compute budget. The compute-optimal data-to-model ratio of Transformer encoder models is 10 to 100 times larger than the ratio of auto-regressive models. Using these recipes, we train OptiBERT, a family of compute-optimal BERT-style models that matches or surpasses leading baselines—including ModernBERT and NeoBERT—on GLUE and MTEB while training with dramatically less FLOPS. These models are also speed and memory efficient.

Training compute-optimal transformer encoder models

To enable broader deployment of Large Language Models (LLMs), it is essential to identify the best-performing model under strict memory constraints. We present AMQ, Automated Mixed-Precision Weight-Only Quantization, a framework that assigns layer-wise quantization bit-widths to optimally balance model quality and memory usage. However, the combinatorial search space, with over 10¹⁰⁰ possible configurations, makes conventional black-box optimization infeasible. AMQ overcomes this challenge through four key innovations: (1) **search space pruning** using prior knowledge to exclude unpromising configurations, (2) **quantization proxy** to bypass costly format conversions during search, (3) **quality predictor** to minimize evaluation overhead, and (4) **iterative search-and-update** strategy for fast and stable convergence. By integrating these components, AMQ efficiently explores the quality–efficiency landscape, reaching the Pareto frontier and yielding LLMs that are both compact and high-performing.

AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models

Modern LLMs struggle with efficient updates, as each new pretrained model version requires repeating expensive alignment processes. This challenge also applies to domain- or language-specific models, where fine-tuning on specialized data must be redone for every new base model release. In this paper, we explore the transfer of fine-tuning updates between model versions. Specifically, we derive the *diff vector* (representing the weight changes from fine-tuning) from one *source* model version and apply it to the base model of a different *target* version. Through empirical evaluations on various open-weight model versions, we show that transferring diff vectors can significantly improve the performance of the target base model. For example, transferring the fine-tuning updates from Llama 3.0 8B improves Llama 3.1 8B by 46.9% on IFEval and 15.7% on LiveCodeBench without additional training, even surpassing Llama 3.1 8B Instruct. Furthermore, we demonstrate performance gains on multilingual tasks, with 4.7% and 15.5% improvements on Global MMLU for Malagasy and Turkish, respectively. We observe that these merged models provide stronger initializations for further fine-tuning. Lastly, our controlled experiments suggest that fine-tuning transfer is most effective when source and target models lie in a linearly connected region of parameter space, and we provide a theoretical analysis of our method. Taken together, fine-tuning transfer offers a cost-efficient and practical strategy for continuous LLM development. Our code will be available at https://anonymous.4open.science/r/finetuning-transfer.

Efficient Model Development through Fine-tuning Transfer

Activation sparsity provides a dynamic, input-dependent alternative to weight pruning for accelerating inference in large language models (LLMs), effectively reducing unnecessary computations and memory accesses during the forward pass. Despite its promise, existing activation sparsification methods suffer from two major limitations: (1) solely relying on activation magnitude for sparsification, ignoring the coupling influence with the corresponding weights, (2) applying uniform sparsity rates across all blocks without considering block-wise sparsity sensitivity. To address these issues, this paper proposes a novel training-free weight-aware activation sparsity framework, called **WAS**. Firstly, with analyzing the coupling relationshape between weight and activation, we introduce a weight-aware scoring method to measure the activation importance in sparsification. Then, a novel constrained Bayesian optimization algorithm is further devised to set a suitable sparsity ratio for all blocks based on the sparsity sensitivity. Finally, we implement a custom GPU sparsity kernel to support the resulting sparsity patterns for wall-clock decoding speed-ups. Our **WAS** achieves competitive performance at 60% model-level sparsity and significantly outperforms prior methods at higher sparsity levels, achieving up to 1.68× inference speed-up—at no retraining or weight update. Codes are available at https://anonymous.4open.science/r/WAS

Weight-Aware Activation Sparsity with Constrained Bayesian Optimization Scheduling for Large Language Models

This paper investigates how LLMs encode inputs with typos. We hypothesize that specific neurons and attention heads recognize typos and fix them internally using local and global contexts. We introduce a method to identify typo neurons and typo heads that work actively when inputs contain typos. Our experimental results suggest the following: 1) LLMs can fix typos with local contexts when the typo neurons in either the early or late layers are activated, even if those in the other are not. 2) Typo neurons in the middle layers are the core of typo-fixing with global contexts. 3) Typo heads fix typos by widely considering the context not focusing on specific tokens. 4) Typo neurons and typo heads work not only for typo-fixing but also for understanding general contexts.

Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors

Large Language Models (LLMs) often don't perform as expected under Domain Shift or after Instruct-tuning. A reliable indicator of LLM performance in these settings could assist in decision-making. We present a method that uses the known performance in high-resource domains and fine-tuning settings to predict performance in low-resource domains or base models respectively. In our paper, we formulate the task of performance prediction, construct a dataset for and train regression models to predict the said change in performance. Our proposed methodology is lightweight and, in practice, can help researchers & practitioners decide if resources should be allocated for data labeling and LLM Instruct-tuning.

DA-Pred: Performance Prediction for Text Summarization under Domain-Shift and Instruct-Tuning

Sparse Autoencoders (SAEs) have been proposed as an unsupervised approach to learn a decomposition of a model’s latent space. This enables useful applications, such as fine-grained steering of model outputs without requiring labeled data. Current steering methods identify SAE features to target by analyzing the input tokens that activate them. However, recent work has highlighted that activations alone do not fully describe the effect of a feature on the model’s output. In this work we draw a distinction between two types of features: input features, which mainly capture patterns in the model’s input, and output features, those that have a human-understandable effect on the model’s output. We propose input and output scores to characterize and locate these types of features, and show that high values for both scores rarely co-occur in the same features. These findings have practical implications: After filtering out features with low output scores, steering with SAEs results in a 2–3x improvement, matching the performance of existing supervised methods.

SAEs Are Good for Steering – If You Select the Right Features

Large language models (LLMs) have achieved impressive performance across natural language processing (NLP) tasks. As real-world applications increasingly demand longer context windows, continued pretraining and supervised fine-tuning (SFT) on long-context data has become a common approach. While the effects of data length in continued pretraining have been extensively studied, their implications for SFT remain unclear. In this work, we systematically investigate how SFT data length influences LLM behavior on short-context tasks. Counterintuitively, we find that long-context SFT improves short-context performance, contrary to the commonly observed degradation from long-context pretraining. To uncover the underlying mechanisms of this phenomenon, we first decouple and analyze two key components, Multi-Head Attention (MHA) and Feed-Forward Network (FFN), and show that both independently benefit from long-context SFT. We further study their interaction and reveal a knowledge preference bias: long-context SFT promotes contextual knowledge, while short-context SFT favors parametric knowledge, making exclusive reliance on long-context SFT suboptimal. Finally, we demonstrate that hybrid training mitigates this bias, offering explainable guidance for fine-tuning LLMs.

When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models

Bit-flip errors (BFEs) are hardware faults where individual bits in memory or processing units are unintentionally flipped. These errors pose a significant threat to neural network reliability because even small changes in model parameters can lead to large shifts in outputs. Large language models (LLMs) are particularly vulnerable on resource-constrained or outdated hardware. Such hardware often lacks error-correction mechanisms and faces aging issues, leading to instability under the vast parameter counts and heavy computational loads of LLMs. While the impact of BFEs on traditional networks like CNNs is relatively well-studied, their effect on the complex architecture of transformers remains largely unexplored. Firstly, this paper presents a comprehensive systematic analysis of BFE vulnerabilities in key LLM components, revealing distinct sensitivities across parameters, activations, and gradients during fine-tuning and inference. Secondly, based on our findings, we introduce a novel defense strategy FlipGuard: (i) exponent bit protection, and (ii) a self-correction based fine-tuning mechanism, to address BFE consequences. FlipGuard minimizes performance degradation while significantly enhancing robustness against BFEs. Experiments demonstrate a 9.27 reduction in accuracy drop under 1 BFEs on the SST-2 dataset using BERT, and a 36.35-point improvement in perplexity on the Wikitext-103 dataset using GPT-2, compared to unprotected models. These results show the potential of our approach in enabling reliable LLM deployment on diverse and less reliable hardware platforms.

Bit-Flip Error Resilience in LLMs: A Comprehensive Analysis and Defense Framework

Large Language Models (LLMs) are capable of generating persuasive Natural Language Explanations (NLEs) to justify their answers. However, the faithfulness of these explanations should not be readily trusted at face value. Recent studies have proposed various methods to measure the faithfulness of NLEs, typically by inserting perturbations at the explanation or feature level. We argue that these approaches are neither comprehensive nor correctly designed according to the established definition of faithfulness. Moreover, we highlight the risks of grounding faithfulness findings on out-of-distribution samples. In this work, we leverage a causal mediation technique called activation patching, to measure the faithfulness of an explanation towards supporting the explained answer. Our proposed metric, Causal Faithfulness quantifies the consistency of causal attributions between explanations and the corresponding model outputs as the indicator of faithfulness. We experimented across models varying from 2B to 27B parameters and found that models that underwent alignment-tuning tend to produce more faithful and plausible explanations. We find that Causal Faithfulness is a promising improvement over existing faithfulness tests by taking into account the model’s internal computations and avoiding out-of-distribution concerns that could otherwise undermine the validity of faithfulness assessments.

Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Models

We investigate how large language models (LLMs) perform latent multi-hop reasoning in prompts like "Wolfgang Amadeus Mozart's mother's spouse is". To analyze this process, we introduce logit flow, an interpretability method that traces how logits propagate across layers and positions toward the final prediction. Using logit flow, we identify four distinct stages in single-hop knowledge prediction: (A) entity subject enrichment, (B) entity attribute extraction, (C) relation subject enrichment, and (D) relation attribute extraction. Extending this analysis to multi-hop reasoning, we find that failures often stem from the relation attribute extraction stage, where conflicting logits reduce prediction accuracy. To address this, we propose back attention, a novel mechanism that enables lower layers to leverage higher-layer hidden states from different positions during attention computation. With back attention, a 1-layer transformer achieves the performance of a 2-layer transformer. Applied to five LLMs, back attention improves accuracy on five reasoning datasets, demonstrating its effectiveness in enhancing latent multi-hop reasoning ability.

Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models

Large pre-trained language models have demonstrated impressive capabilities, but there is still much to learn about how they operate. In this study, we conduct an investigation of the autoregressive transformer’s ability to perform basic addition operations. Specifically, by using causal analysis we found that a few different attention heads in the middle layers control the addition carry, with each head processing carries of different lengths. Due to the lack of global focus on the sequence within these attention heads, the model struggles to handle long-sequence addition tasks. By performing inference intervention on mistral-7B, partial task performance can be restored, with the accuracy on 20-digit long-sequence additions from 2% to 38%. Through fine-tuning, a new mechanism branches out for handling complex cases, yet it still faces challenges with length generalization. Our research reveals how the models perform basic arithmetic task, and further provides insights into the debate on whether these models are merely statistical.

How do autoregressive transformers solve full addition?

Various studies have attempted to remove sensitive or private knowledge from a language model to prevent its unauthorized exposure. However, prior studies have overlooked the inherent complexity and interconnectedness of knowledge, which requires careful examination. To resolve this problem, we first define a new concept called superficial unlearning, which refers to the phenomenon where an unlearning method either fails to erase the interconnected knowledge it should remove or unintentionally erases irrelevant knowledge. Based on the definition, we introduce a novel benchmark, FaithUn, to analyze and evaluate the faithfulness of unlearning in real-world knowledge QA settings. Furthermore, we propose a novel unlearning method, KLUE, which updates only knowledge-related neurons to achieve faithful unlearning. KLUE leverages a regularized explainability method to localize contextual knowledge neurons, updating only these neurons using carefully selected unforgotten samples. Experimental results demonstrate that existing unlearning methods fail to ensure faithful unlearning, while our method shows significant effectiveness in real-world QA unlearning.

FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge

Despite the remarkable capabilities of large language models, current training paradigms inadvertently foster sycophancy—alignment with user-provided information, regardless of factual accuracy. In this paper, we introduce SMART (Sycophancy Mitigation through Adaptive Reasoning Trajectories), reconceptualizing sycophancy as a reasoning optimization problem rather than an output alignment issue. SMART employs a two-stage approach: (1) Uncertainty-Aware Adaptive Monte Carlo Tree Search (UA-MCTS), which dynamically adjusts exploration based on state-level uncertainty; and (2) progress-based reinforcement learning that distills these improved reasoning patterns into model adaptation. Through extensive experiments, we show that SMART significantly outperforms existing baselines in effectively reducing sycophancy while maintaining performance on out-of-distribution inputs. These findings demonstrate the importance of optimizing internal reasoning processes for developing aligned truthful AI assistant.

Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories

Topic modeling is a powerful unsupervised tool for knowledge discovery. However, existing work struggles with generating limited-quality topics that are uninformative and incoherent, which hindering interpretable insights from managing textual data. In this paper, we improve the original variational autoencoder framework by incorporating contextual and graph information to address the above issues. First, the encoder utilizes topic fusion techniques to combine contextual and bag-of-words information well, and meanwhile exploits the constraints of topic alignment and topic sharpening to generate informative topics. Second, we develop a simple word co-occurrence graph information fusion strategy that efficiently increases topic coherence. On three benchmark datasets, our new framework generates more coherent and diverse topics compared to various baselines, and achieves strong performance on both automatic and manual evaluations.

Neural Topic Modeling via Contextual and Graph Information Fusion

Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs). However, existing training-based model editing methods often struggle to effectively incorporate new knowledge while preserving unrelated general knowledge. To address this challenge, we propose a novel framework called Geometric Knowledge Editing (GeoEdit). GeoEdit utilizes the geometric relationships of parameter updates from fine-tuning to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. By employing a direction-aware knowledge identification method, we avoid updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model's generalization ability. For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a ``forget-then-learn'' editing strategy for opposite directions. Additionally, we introduce an importance-guided task vector fusion technique that filters out redundant information and provides adaptive neuron-level weighting, further enhancing model editing performance. Extensive experiments on two publicly available datasets demonstrate the superiority of GeoEdit over existing state-of-the-art methods.

Underline digital video library

Browse The Future of AI Research Webinars

Browse AAAI 2026 Invited Talks

Browse Underline Originals

No results!

Browse Natural Language Processing

Browse Machine Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES