The Evolving Landscape of Reasoning in AI

It has been a few months since our last exploration into the realm of Reasoning in AI, and the field has witnessed significant advancements. This blog emerged from a recent discussion with Dr. Lucian Galescu, Openstream.ai Head of Reasoning & Dialogue Research, to highlight the latest developments, challenges, and prospects in this dynamic area.

Credit: Audio generated with NotebookLM

The Rise of Large Reasoning Models (LRMs)

The conversation began by acknowledging the rapid proliferation of new and more powerful Large Language Models (LLMs), with a particular emphasis on those specifically designated as Large Reasoning Models (LRMs). Notable examples include DeepSeek-R1, which garnered considerable attention, Anthropic’s Claude 3.7 Sonnet (a versatile model functioning as both a regular LLM and an LRM), OpenAI’s o3 series, and the recently released open-weight Qwen QwQ with 32 billion parameters, reportedly exhibiting performance comparable to R1. Self-reported results on reasoning benchmarks indicate notable improvements in these models.

However, it was cautioned that extravagant claims should be viewed with discernment, as benchmark performance might not always translate to real-world use cases. Instances were cited where impressive results were achieved through access to training data or prohibitively expensive computational resources.

Are New Models Truly Better at Reasoning?

Despite these caveats, there is evidence suggesting genuine progress in reasoning capabilities. One anecdotal example shared involved o3-mini successfully solving a problem that had stumped previous models, including o1. While the explanation contained some flaws, the overall performance was considered impressive.

Furthermore, independent research supports the notion of improvement. A study by researchers at the University of Washington and the Allen Institute for AI examined scaling effects on constraint satisfaction problems, finding that larger models generally perform better, and increased test-time computation also yields benefits. For instance, o1 can utilize longer chains of thought for more complex problems, leading to enhanced performance.

Understanding "Chain of Thought" (CoT)

The discussion clarified the concept of Chain of Thought (CoT), a technique where an LLM generates intermediate steps toward a final answer rather than directly producing it. While users can explicitly ask any LLM for a step-by-step explanation, reasoning models now often generate these internal chains of thought autonomously. These intermediate steps typically guide the LLM towards a more accurate solution.

Another performance-enhancing technique involves sampling multiple solutions and employing a selection method such as majority voting or best-of-N. While OpenAI's specific techniques remain proprietary, indirect clues from visible CoT summaries suggest that these approaches are likely utilized by o1 and o3, contributing to their improved performance.

The "Curse of Complexity"

Despite the advancements, the aforementioned study also revealed a significant limitation: the "curse of complexity". While clear improvements are observed for easier problems, there exists a threshold of problem complexity beyond which all models experience a performance collapse, hardly solving any problems at all. This phenomenon has been recognized in prior studies, for example, with planning and scheduling problems where performance deteriorates sharply for solutions requiring more than 4-5 steps.

DeepSeek-R1's Performance

The study did evaluate DeepSeek-R1, which had generated considerable excitement. The findings indicated that while DeepSeek-R1 significantly outperformed the generally available o1-mini, it still lagged behind the full version of o1. Given that o3 is considered superior to o1, R1 is likely even further behind.

Reasoning Models vs. Standard LLMs

The study's comparison of reasoning and standard LLMs revealed that reasoning models are clearly superior on reasoning tasks, solving more problems and tackling more complex ones. An interesting observation was made: improved performance on logical reasoning seems to correlate with performance on other reasoning tasks, which is encouraging for the generalizability of these models. However, there is some evidence suggesting that reasoning models might perform slightly worse on certain non-reasoning tasks. This implies that a wholesale switch to the best reasoning models might not be advisable yet, especially considering their higher cost and potential for increased latency. The example of o3-mini taking almost 5 minutes to solve a problem highlights the latency concern, which can be critical for applications with stringent response time requirements, such as conversational agents.

When Should Reasoning Models Be Preferred?

While a definitive answer remains elusive, OpenAI has suggested utilizing reasoning models for tasks demanding attention to detail, resolving ambiguities, or high-level planning, such as in agent system orchestrators. They also recommend them for validation and in LLM-as-judge applications, where a powerful LLM evaluates the outputs of another. However, independent data supporting these specific use cases is currently lacking, emphasizing the need for further investigation.

Benchmarks and Real-World Applications

It was emphasized that while LLMs/LRMs are trained as generalists, their ability to generalize beyond their training data is crucial. Benchmarks serve a valuable purpose but do not encompass all real-life scenarios. In task-oriented dialogue, for instance, the reasoning required might involve determining the next question to ask to gather necessary information, rather than directly solving a fully specified problem as in typical reasoning benchmarks. Spending significant test-time compute might be beneficial when a solution is known to exist, but it could be inefficient when the input information is insufficient.

Future Directions in Reasoning with LLMs

The development of LRMs is expected to continue, with the potential for smaller and faster models specializing in specific types of reasoning. However, it was noted that even with improvements, these models are still simulating reasoning and will likely continue to exhibit hallucinations and logical errors, which can undermine user trust.

Therefore, neuro-symbolic approaches, which integrate neural networks with symbolic reasoning, are considered a promising future direction. An increasing number of publications and new companies in this space suggest a growing interest in combining the strengths of both approaches.

While significant progress has been made in the field of AI reasoning with the emergence of powerful LRMs and advanced techniques like Chain of Thought, challenges such as the curse of complexity and the need for robust and reliable reasoning remain. The integration of neuro-symbolic methods is a key area to watch in the quest for more dependable and trustworthy AI reasoning systems.