Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments

Abstract

Enterprise systems are crucial for enhancing productivity and decision-making among employees and customers. Integrating LLM based systems into enterprise systems enables intelligent automation, personalized experiences, and efficient information retrieval, driving operational efficiency and strategic growth. However, developing and evaluating such systems is challenging due to the inherent complexity of enterprise environments, where data is fragmented across multiple sources and governed by sophisticated access controls.

Our benchmark features EnerpriseBench which provides a Enteprise Simulation Environment along with 500 Realistic Tasks for comprehensive agent assessment. Through extensive evaluation across multiple domains, EnterpriseBench reveals significant gaps between current LLM agent capabilities and enterprise requirements, establishing new benchmarks for real-world AI deployment readiness.

Introduction

The deployment of LLM agents in enterprise environments presents unique challenges that current benchmarks fail to address. While existing evaluation frameworks focus on isolated capabilities like question-answering or code generation, real enterprise scenarios require agents to navigate complex, interconnected business systems with authentic data relationships and domain-specific constraints.

Why Enterprise-Specific Evaluation Matters

Enterprise environments are characterized by:

• Multi-domain Integration: Tasks often span HR, IT, Sales, and Engineering departments
• Complex Data Relationships: Information is interconnected across multiple business systems
• Domain-Specific Constraints: Each department has unique workflows, terminology, and requirements
• Realistic Scale: Enterprise data volumes and complexity far exceed academic benchmarks

EnterpriseBench addresses these gaps by providing the first comprehensive framework specifically designed for enterprise LLM agent evaluation.

Figure 1: EnterpriseBench agent workflow showing the complete task execution process from user query through planning, execution, and task completion within the enterprise environment.

EnterpriseBench Framework

Architecture Overview

EnterpriseBench consists of three core components working together to provide comprehensive enterprise agent evaluation:

1. Enterprise Sandbox Environment

• Realistic Data: Synthetic but authentic business data across 10+ domains
• Interconnected Systems: Data relationships mirror real enterprise architectures
• Scalable Infrastructure: Supports various task types and complexity levels
• Privacy-Compliant: Synthetic data ensures privacy while maintaining realism

2. Comprehensive Task Suite

• Search Tasks: Information retrieval, conversation analysis, and database queries
• CRUD Tasks: Create, Read, Update, Delete operations on enterprise data
• Multi-Domain Coverage: Tasks spanning HR, IT, Sales, and Engineering departments
• Realistic Complexity: Enterprise-grade scenarios with authentic data relationships

3. Advanced Evaluation System

• Automated Assessment: AI-powered evaluation with multiple scoring criteria
• Performance Metrics: Comprehensive evaluation framework for agent capabilities
• Interactive Interfaces: Streamlit-powered demos for real-time testing
• Comparative Analysis: Benchmarking across different LLM architectures

Supported Domains

EnterpriseBench covers comprehensive business domains with authentic data and realistic task scenarios:

Evaluation Methods

Search-Based Evaluation

Search tasks evaluate an agent’s ability to find, analyze, and synthesize information across enterprise systems:

• Information Retrieval: Locate specific data points across multiple systems
• Conversation Analysis: Extract insights from communication threads
• Database Queries: Navigate complex data relationships
• Cross-Domain Search: Find information spanning multiple departments

CRUD-Based Evaluation

CRUD tasks assess an agent’s capability to perform standard business operations:

• Create: Generate new records, documents, or communications
• Read: Access and interpret existing business data
• Update: Modify records while maintaining data integrity
• Delete: Remove outdated or incorrect information safely

Performance Results

Table 3: EnterpriseBench Evaluation - Comparison of performance across agents using different models and planning strategies with LangChain and DSPy frameworks

Interactive Demos

EnterpriseBench provides interactive Streamlit applications for hands-on agent evaluation across different enterprise scenarios:

How to Cite

If you use EnterpriseBench in your research, please cite our work:

@inproceedings{vishwakarma-etal-2025-llms,
    title = "Can {LLM}s Help You at Work? A Sandbox for Evaluating {LLM} Agents in Enterprise Environments",
    author = "Vishwakarma, Harsh  and
      Agarwal, Ankush  and
      Patil, Ojas  and
      Devaguptapu, Chaitanya  and
      Chandran, Mahesh",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.466/",
    pages = "9178--9212",
    ISBN = "979-8-89176-332-6",
}