青源Talk125期｜NPHardEval：一个通过计算复杂性评估大型语言模型推理能力的动态基准

​

NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes

NPHardEval：一个通过计算复杂性评估大型语言模型（LLM）推理能力的动态基准

推理能力是当前LLM最重要的特征之一，它也在复杂决策任务中扮演了不可或缺的角色。因此，研究LLM的推理能力至关重要。然而，现有的基准在对LLM的推理能力进行严格评估方面是不充分的。它们也容易过拟合，因为这些公开且静态的基准允许模型可能针对特定基准指标调整其响应，从而夸大其性能。为了解决这些现存问题，我们的研究引入了一个新的基准，名为NPHardEval。该基准旨在通过900个算法问题评估LLM的推理能力，涵盖上至NP-Hard计算复杂性类别，为LLM的推理能力提供了严格的衡量标准。通过这项研究，我们揭示了LLM当前的真实推理能力，通过比较它们在不同复杂类别中的性能，提供了客观严谨的视角，并为未来增强这些模型的推理能力奠定了基础。此外，该基准设计了一个动态更新机制，数据点每月更新一次。这种定期更新在减少模型过拟合于基准的风险上发挥了重要作用，促进了对LLM推理能力更准确可靠的评估。The NPHardEval benchmark evaluates the complex reasoning abilities of Large Language Models (LLMs), addressing the limitations of current benchmarks. It comprises 900 algorithmic questions across a spectrum of complexity, including up to NP-Hard class, providing a comprehensive measure of LLM reasoning capabilities. This study highlights the current reasoning performance of LLMs and contributes to understanding their capabilities in complex tasks. NPHardEval features a dynamic update mechanism with monthly data refreshes, reducing the risk of model overfitting and ensuring a more accurate assessment of LLM reasoning abilities.

项目地址： https://arxiv.org/abs/2312.14890

范犁洲，密歇根大学信息学博士候选人。研究领域主要集中在数据档案、机器学习、自然语言处理，以及在社交媒体和医疗保健等领域应用人工智能。其最近的工作尝试将大型语言模型和多智能体系统应用在以上领域。他也对分析复杂人类行为和提高其中的效率有浓厚的兴趣。Lizhou Fan, currently a doctoral candidate in Information at the University of Michigan, exhibits a keen interest in analyzing and enhancing efficiency within complex human behaviors. His research ambit primarily focuses on Data Archives, Machine Learning, Natural Language Processing, and the application of Artificial Intelligence in domains like Social Media and Healthcare. Recently, he has expanded his interests to include the development and application of large language models and multi-agent systems across various fields.

华文越，目前是新泽西州立罗格斯大学新布朗斯维克分校计算机科学的博士候选人，主要研究大语言模型及其应用。她的研究包括用于推荐系统的大语言模型、基于LLM的智能体和多智能体系统，特别是在计算社会科学中的应用。Wenyue Hua, currently a doctoral candidate in Computer Science at the Rutgers University, New Brunswick, works mainly on large language models and its applications. Her research includes LLM for recommender system, LLM-based agents and multi-agent system, especially in their applications in computational social science. 关注华文越：https://link.baai.ac.cn/@150191 

​Read More