The AI Agent Index

Stephen Casper et al.

Audio by Paper2Audio.

Abstract

Leading AI developers and startups are increasingly deploying agentic AI systems that can plan and execute complex tasks with limited human involvement. However, there is currently no structured framework for documenting the technical components, intended uses, and safety features of agentic systems. To fill this gap, we introduce the AI Agent Index, the first public database to document information about currently deployed agentic AI systems . For each system that meets the criteria for inclusion in the index, we document the system's components (e.g., base model, reasoning implementation, tool use), application domains (e.g., computer use, software engineering), and risk management practices (e.g., evaluation results, guardrails), based on publicly available information and correspondence with developers. We find that while developers generally provide ample information regarding the capabilities and applications of agentic systems, they currently provide limited information regarding safety and risk management practices. The AI Agent Index is available online at aiagentindex.mit.edu URL with raw data at this link.

1. Introduction

'Agentic' AI systems that can be instructed to plan and directly execute complex tasks with only limited human involvement are transitioning from research prototypes to real-world products (e.g., Devin, h2oGPTe, Simple AI, XBOW) . These systems—which are generally comprised of foundation models augmented with scaffolding for reasoning, planning, memory, and tool use (Sumers are being deployed in a growing number of domains (see Figure 7).

Figure 7 summary: The figure is a pie chart that visualizes the distribution of indexed systems across various application domains. A significant portion of the systems is dedicated to software engineering and computer use. Other categories, including universal agents, research, robotics, and a general 'other' category, constitute a smaller proportion of the indexed systems. Software engineering and computer use have a similar proportion of systems.

The performance of agentic systems is steadily improving on benchmarks, and these systems are being integrated into broader swathes of economic activity. As a result, their real-world impacts are mounting. Alongside the significant opportunities presented by agentic systems, researchers have also raised noteworthy concerns, including cybersecurity risks, loss of control, and physical harm where agents operate robotic systems.

Despite growing efforts to study trends in the development of agentic AI systems, including evaluating their performance and cost, assessing their potential harms, and increasing visibility into their operation, many practical questions remain unanswered: Which organizations are developing agentic systems?

In which domains are they being deployed?

What infrastructure do agentic systems require?

How is their performance and safety evaluated?

What guardrails are used to mitigate risks?

To empirically answer these questions and improve public understanding of agentic AI systems, we introduce and release the AI Agent Index, a comprehensive sample of deployed agentic AI systems (n = 67). The index, which is constructed from a combination of publicly available data and correspondence with developers, documents publiclyavailable information on the intended uses of agentic systems, their technical components (including reasoning, planning, and memory implementation, base models, observa- tion and action space, and user interface), safety features (including accessibility of system components, usage controls and restrictions, and red-teaming practices), and details regarding the organizations developing and deploying agentic systems (including entity type and country of origin).

In addition to collecting and systematizing information about agentic AI systems, the index also sheds light on the availability of such information. Specifically, we find that while relatively detailed information is available regarding the features and applications of agentic systems (Figure 1) , strikingly limited information is available regarding their safety evaluations and guardrails (Figure 2) .

Figure 1 summary: The figure is a bar chart. It shows the percentage of AI agent developers who provide public documentation and release their underlying code. A higher percentage of developers provide public documentation compared to those who release their code.

Figure 2 summary: The figure is a horizontal bar chart. It shows the percentage of agentic systems that disclose a formal safety policy, report external safety evaluations, and report safety testing by the developer. The figure indicates that only a small fraction of indexed agentic systems publicly disclose a formal safety policy or report external safety evaluations. The percentage of systems reporting safety testing by the developer is also relatively low.

In this paper, we make three contributions: 1. We introduce a structured framework for documenting the technical, safety, and policy-relevant features of agentic AI systems. 2. We identify currently deployed agentic systems that meet our criteria (described below) and publicly document these systems according to our framework. 3. We discuss key findings from the index, shedding light on geographic spread, academic vs. industry development, openness, and risk management of agentic systems.

The index is available on the web at aiagentindex.mit.edu URL raw data accessible here.

2. Background

There is no widely accepted definition of "AI agent". The notion of artificial agency has a long and contentious history, spanning multiple decades and diverse disciplines. These include cybernetics.

Wiener, artificial life, rational agency, software engineering, reinforcement learning, and philosophy. While there have been notable attempts to define the term "agent", including in the context of computational systems, we do not decide among these definitions or offer an alternative definition. Instead, we follow Chan et al. (2023), and loosely characterize agentic AI systems as ones that exhibit, to some significant degree, a combination of the following properties: a) Underspecification: the system can accomplish a goal provided to it without a precise specification of how to do so. b) Directness of impact: the system's actions can affect the world with little to no human mediation. c) Goal-directedness: the system acts as if in the pursuit of a particular objective. d) Long-term planning: the system can solve problems by reasoning about how to approach them, constructing plans, and executing them step by step.

2.1. Agentic Architectures, Applications, and Opportunities

Contemporary AI agents are generally compound systems comprised of a foundation model augmented by external resources, known as "scaffolding", which enable effective planning, memory, and tool use. Planning of complex series of actions is typically facilitated through chain-of-thought-based reasoning processes. Memory relies on information stored in the base model and/or in external storage modules. Tool use is enabled through API calls and natural language dialogue between the base model and external software, databases, and other affordances.

These agentic architectures are increasingly applied to a variety of domains, including programming, machine learning research, experimentation in the natural sciences, and consumer activities such as online retail, travel planning, and general-purpose web browsing. Progress in these applications is being evaluated by a growing suite of benchmarks, which measure performance in computer use, software engineering, and virtual work environments.

2.2. Safety Risks and Ethical Concerns

Given that agentic AI systems are built on foundation models, they are susceptible to many of the risks associated with such models, including harms arising from hallucinations, biased outputs, and leakage of private data. Agentic systems, however, also present new risks that stem specifically from their agentic properties, i.e., underspecification, directness of impact, goal-directedness, and long-term planning. For example, while chatbots often cause harm by human users acting upon model outputs (e.g., deploying modelgenerated malicious code), agentic AI systems can directly cause harm (e.g., autonomously hacking websites).

Additionally, as agentic AI systems undertake more complex and long-horizon tasks, with limited human oversight, users are likely to repose greater trust in those systems, potentially developing asymmetric relationships of dependence. Moreover, agentic systems developed and operated by large platform companies could enable those companies to exert greater influence and control over users and third parties with whom they interact (e.g., vendors accessed through platform-controlled agents).

2.3. Documentation Frameworks

Many frameworks have been developed to document the features of AI systems, the resources used to build them, and the contexts in which they are deployed. These in- clude datasheets, model cards, reward reports, ecosystem graphs, and data provenance cards. In addition, several databases have been created to collect information regarding contemporary AI systems and their real-world impacts, such as the Foundation Model Transparency Index, the AI Incident Database, and the AI Risk Repository. Currently, however, there are no equivalent frameworks for documenting agentic AI systems. This lack of structured information limits both researchers' ability to study and build agentic systems, as well as policymakers' capacity to design appropriate governance mechanisms.

The AI Agent Index fills this gap. By collecting and communicating technical, safety, and policy-relevant information concerning agentic AI systems, the index aims to inform different stakeholders in distinct ways. Specifically, the index: 1. Enables users to better understand the capabilities and limitations of agentic systems with which they interact. 2. Provides developers more comprehensive and granular information about currently deployed agentic systems. 3. Supports auditors and red-teams in deciding the scope and focus of their evaluations of agentic systems. 4. Offers an evidence base to policymakers designing governance mechanisms for agentic systems. 5. Improves public awareness and understanding of agentic systems.

3. Methods

What does the index include? As discussed in Section 2, there is no widely-accepted definition of "AI agent." We do not propose one here. Given our focus on the societal impacts of agentic AI systems, we draw on the four characteristics introduced by Citation 2023 discussed in Section 2. Importantly, to address the practical questions outlined in Section 1, we primarily document the features of agentic AI systems that are either deployed as products or available open source.

The full decision graph we used to determine inclusion in the index is shown in Figure 3. Notably, we restricted the index to agentic systems and did not include language models themselves, or agent development frameworks (unless the framework was built around a qualifying flagship system, in which case we indexed that system). We also created a single index entry per named and versioned system. Different releases (e.g., "HelpfulAgent1.1" vs "HelpfulAgent1.2") and different configurations (e.g., "HelpfulAgent-Claude3.5- Sonnet" vs. "HelpfulAgent-GPT4o") were indexed under the same entry. The final node in our decision graph (Figure 3) facilitates the inclusion of systems that otherwise would not strictly fit the criteria at our discretion. In practice, we only invoked this for systems from leading companies that were announced but have not (yet) been externally deployed, such as OpenAI o3 or Project Mariner. In total, we indexed 67 systems. Limitations of our methods are discussed in Section 6.

Figure 3 summary: The figure is a decision graph. The graph outlines the criteria used for determining whether to include a system in the index of AI agents. The decision process begins with a named agentic AI system and proceeds through a series of questions regarding its capabilities, deployment, and economic value. Systems are assessed based on their ability to perform diverse tasks, their deployment status, whether they are open source, and their economic competitiveness. Systems meeting certain criteria are included in the index, while those failing to meet the criteria are excluded. A final consideration allows for the inclusion of systems that may not strictly fit the criteria but are deemed relevant, such as those from leading companies that have been announced but not yet deployed. In total, a specific number of systems were indexed.

The AI Agent Index represents a snapshot in time as of December 31, 2024. New developments in the AI agent research and product ecosystem occur weekly. To improve thoroughness and consistency, we only indexed systems announced by, and available in, 2024.

What does the index not include? Our selection criteria led us to exclude the following types of systems:

Non-"agentic" models such as Llama-3.2-90B-Vision-Instruct.

Unnamed systems often comprised of simple baseline implementations introduced under frameworks or benchmarks such as CORE-Bench, AgentHarm, or The Agent Company.

Non-"agentic" development frameworks without a qualifying flagship model such as AutoGPT, Beam, Dust, GumLoop, Lindy, OpenAI Swarm, Qwen-Agent, or Spell.

Systems that cannot open-endedly accomplish a diverse range of tasks such as systems that propose solutions to git requests (e.g., MentatBot, Engine, Globant Code-Fixer Agent). • Systems that do not have a meaningfully higher degree of agency than ChatGPT-4o 1 (based on the four aspects of agency from Citation 2023) such as Taskade, Vonage AI Virtual Assistant, Talkdesk, IBM WatsonX, and ActionAgents.

Systems that are not open source or products deployed for commercial or other consequential applications such as Falcon-UI or Honey-Comb.

Open source systems that could not be used competitively off the shelf, often due to age or narrow scope such as GeniA, ReAct, Pearl, or Moatlesss.

Systems deployed after the cutoff date of December 31, 2024 such as Deepseek-R1, Doubao-1.5-pro, or OpenAI Operator. How was information collected? From August 2024 to January 2025, we identified agentic AI systems using web searches, academic literature review, benchmark leaderboards (e.g., SWE-bench and GAIA), and additional resources that compile lists of agentic systems (e.g., aiagentslist.com URL vyokky.github.io URL and letta.com URL

On a rolling basis, we created the first drafts of agent cards according to the template outlined next in Section 4. After each first draft was completed, we contacted the developers of each agent to request feedback and potential corrections.

We received a 36% response rate. After editing each draft to incorporate feedback, we updated and finalized agent cards in January 2025 to ensure that all reflected the state of the field as of December 31, 2024. For all web sources cited in all agent cards (excluding stable papers, videos, and social media posts), we cited stable archived versions of websites preceding and as close to December 31, 2024 as possible using web.archive.org URL perma.cc URL

4. Agent Card Components

Each agent card contains 33 fields of information, divided into 6 categories:

1. Basic information

Website Short description Intended uses: What does the developer state that the system is intended for?

Date(s) deployed

2. Developer

Website Legal name Entity type Safety policies: What safety and/or responsibility policies are in place?

Table summary: The table row describes the country which is the location of the developer or the first author's first affiliation.

3. System components

Backend model: What model(s) are used to power the system? Publicly available model specification: Is there formal documentation on the system's intended uses and how it is designed to behave in them?

Reasoning, planning, and memory implementation: How does the system 'think'?

Observation space: What is the system able to observe while 'thinking'? Action space/tools: What direct actions can the system take?

User interface: How do users interact with the system?

Development cost and compute: What is known about the development costs?

4. Guardrails and oversight

Accessibility of components: Weights: Are model parameters available? Data: Is data available?

Code: Is code available?

Scaffolding: Is system scaffolding available?

Documentation: Is documentation available? Controls and guardrails: What notable methods are used to protect against harmful actions? Customer and usage restrictions: Are there know your-customer measures or other restrictions on customers? Monitoring and shutdown procedures: Are there any notable methods or protocols that allow for the system to be shut down if it is observed to behave harmfully?

5. Evaluations

Notable benchmark evaluations (e.g., on SWE-Bench Verified)

Bespoke testing (e.g., demos) Safety: Have safety evaluations been conducted by the developers? What were the results? Publicly reported external red-teaming or comparable auditing: Personnel: Who were the red teamers/auditors? Scope, scale, access, and methods: What ac cess did red-teamers/auditors have and what actions did they take? Findings: What did the red-teamers/auditors conclude?

6. Ecosystem

Interoperability with other systems: What tools or integrations are available? Usage statistics and patterns: Are there any no table observations about usage?

7. Additional notes: If any

We populated each field in each card with written notes based on publicly available information. When no information was available, we recorded "None" or "Unknown."

5. Findings

In addition to compiling specific information regarding each of the 67 indexed systems, the AI Agent Index offers a high-level perspective of this emerging field. Noting the limitations and biases discussed next (in Section 6) , here, we offer a bird's eye view of the state of the art for AI agents.

Agentic systems are being deployed at a steadily increasing rate. Systems that meet our criteria for inclusion in the index have had (initial) deployments dating back to early 2023. However, Figure 4 shows that they have been deployed at an increasing rate with approximately half of the indexed systems deployed in the second half of 2024.

Figure 4 summary: This is a bar chart showing the timeline of agent releases. The number of agent releases has generally increased over time. The number of agents released was relatively low in the first few months shown, and then increased noticeably. The number of agents released was highest in the most recent months.

Most indexed systems are created by developers located in the USA. We considered the 'developer country' of each agent to be the national location of either (a) the developer organization if the developer was a company or (b) the first author's first listed affiliation if the agent was created as part of an academic research collaboration. We plot the number of agents from each country in Figure 5. Of the 67 agents, 45 were created by developers in the USA.

Figure 5 summary: The figure is a pie chart. It represents the country distribution of agent developers. The chart indicates that the majority of agentic systems are created by developers in the USA, with a significantly larger proportion compared to other countries. China has the second largest proportion, followed by the UK, Israel, and Japan. Other countries including Singapore, Canada, Sweden and France have a relatively small representation.

While most agentic systems are developed by companies, a significant fraction are developed in academia. In Figure 6, we show the developers of agents broken down based on whether they are projects from academic labs or companies in industry. 18 (26.9%) are academic while 49 (73.1%) are from companies.

Figure 6 summary: The figure is a pie chart. It shows the distribution of developer types for agentic systems, categorized as either academic or industry. The data indicates that a significantly larger proportion of agentic systems are developed by companies compared to academic institutions.

The majority of indexed systems specialize in software engineering and/or computer use. We divided the 67 systems into 6 categories: Software : agents that assist in coding and software engineering (e.g., Yang et al.).

Computer use : agents designed to open-endedly interact with computer interfaces (e.g., ). • Universal : agents designed to be a general-purpose reasoning engine (e.g., OpenAI).

Research : agents designed to assist with scientific research (e.g.,).

Robotics : agents designed for robotic control (e.g., Kim et al.).

Other : systems that are designed for niche applications (e.g., LinkedIn Talent Agents) .

We plot the breakdown by domain in Figure 7. 50 of the 67 agents (74.6%) specialize in either software engineering or computer use. We also note that there exist many 'agentic' systems for customer service, which do not meet our criteria for inclusion in the index. See Section 3 for discussion and examples.

The majority of indexed systems have released code and/or documentation. Developers are relatively publicly forthcoming about details related to usage and capabilities. In Figure 1, we show results: 33 (49.3%) release code, and 47 (70.1%) release documentation. We also observed that systems developed as academic projects are released with a high degree of openness, with 16 of the 18 (88.8%) releasing code.

There is limited publicly available information about safety testing and risk management practices. In contrast to the relatively high degree of openness that developers exercise around their systems' capabilities and usage, we find scant public information about safety policies, internal safety evaluations, and external safety evaluations. In Figure 2, we show that only 13 (19.4%), 5 (7.5%), and 6 (9%) indexed systems have publicly available information on each of these, respectively. We also note that most of the systems that have undergone formal, publicly-reported safety testing are from a small number of large companies (e.g., Anthropic, Google DeepMind, OpenAI).

6. Limitations and Concerns

Defining agentic systems. The term "AI agent" is contentious, as discussed in Section 2. In particular, the term has been criticized for inappropriately anthropomorphizing certain AI systems, which could potentially lead to unrealistic expectations from, or over-reliance on, such systems. Recognizing this concern, we do not weigh in on this debate, advocate a particular definition of "AI agent", or propose alternative terminology. Instead, we focus on empirically documenting a growing class of deployed AI systems that exhibit "agentic" characteristics (as described in Citation 1) and have a potential for significant impact. Through the index, we communicate our findings as plainly and openly as possible.

Scope and timing of index. The index is not a comprehensive or exhaustive database of all agentic systems or related resources, such as language models and development frameworks for building agentic systems. The field of agentic AI is highly decentralized and poorly documented. Accordingly, there may also be systems that meet the selection criteria specified in Section 3 but do not appear in the index. In particular, the index is likely to disproportionately document agentic systems that are publicly available or publicly released, compared with systems used internally within organizations (which, by definition, are not publicly accessible). In addition, the index only includes systems described in the English language and includes relatively few systems from non-western developers. The index represents a snapshot in time on December 31, 2024 and does not include systems that were obsolete by this date or were released thereafter. Moreover, while the agent cards in the index collect 33 fields of information, these are not exhaustive and exclude, for example, records of real-world safety incidents (to the extent such incidents have occurred). Incomplete or inaccurate information. In total, the index contains over 2,200 fields of information reviewed by multiple authors. Nonetheless, despite our best efforts to manually verify the completeness and accuracy of all agent cards, mistakes may have occurred. In addition, the response rate of developers to our requests for feedback was 36%. Accordingly, it is possible that some developers may, for example, have in place internal safety documents or practices that we could not discover from publicly available documentation, or were not informed about through correspondence. Recognizing these concerns, we have established a structured process for facilitating further corrections to the index. These can be submitted at this link.

Promoting problematic practices. The findings we present in Section 5— particularly the lack of transparency around the safety features of agentic systems—could arguably promote problematic risk management practices. For example, developers could choose to 'game' an index like ours through perfunctory, selective disclosure of information recorded in the index. Due in part to this concern, we do not use this index to make developer scorecards. Instead, we see our findings as offering basic information to key stakeholders, including users, developers, auditors, and policymakers. In doing so, we hope to lay the groundwork for more targeted assessments of impacts and risks from agentic systems in future work.

7. Discussion and Future Work

The agentic AI ecosystem is difficult to document. The extensive data collection process undertaken for the current paper (see Section 3) sheds light on the significant challenges involved in documenting agentic AI systems. During this process, we encountered a diverse range of AI systems, across multiple domains, in different places in the research– product spectrum, and accompanied by varying levels of information and documentation. The differences were often most stark when comparing systems developed in industry and systems developed in academia, the latter of which are typically simpler and more open. On occasion, these features of the agentic AI ecosystem made it challenging to determine whether a particular system meets our criteria for inclusion in the index. Most importantly, the fact that we ultimately produced an "AI Agent Index" should not be taken to suggest that this ecosystem lends itself to clean taxonomization and indexing (it does not). We expect these documentation challenges to persist for the foreseeable future.

Future documentation work should be appropriately scoped. Our research design—including both the selection of information fields to be collected and the methods for collecting data—offers lessons for future attempts to document the agentic AI ecosystem. From the outset, we sought to collect information on agentic systems that had been generally overlooked by previous survey papers and overviews of the field, such as the accessibility of documentation and code, information regarding red-teaming and safety policies, and the country of developers (see Section 4) . Future documentation work can build on this approach, examining a broader range of technical, safety, and policy-relevant features of agentic AI systems. To ensure tractability, we recommend that future work surveying the agent ecosystem be appropriately scoped either in breadth or depth. For example, selection criteria could be revised to demand a high threshold for "agency" or anticipated societal impact.

Documentation can inform governance and policy. Our findings (discussed in Section 5) may inform the scope and methods of AI governance and policymaking: The majority of indexed agentic systems were developed in industry, suggesting that governance interventions should consider the incentives of corporate developers (distinct from those of academic labs).

Most indexed systems were developed by US-based organizations, indicating that governance efforts focused on US contexts could have more leverage than efforts in other countries or regions.

The prominence of software engineering and computeruse agents suggests that policy researchers and practitioners should prioritize these domains when designing governance frameworks.

Very few developers disclose information about safety or risk management, underscoring the importance of establishing transparency and disclosure mechanisms as a key first step in the governance of agentic systems.

To address knowledge and accountability gaps uncovered by our findings, policymakers could consider: Structured bug bounties: Incentivizing external redteaming promotes the proactive discovery of vulnerabilities, adapting approaches used in cybersecurity.

Systematic testing of agents: Governance bodies and academic labs could coordinate risk assessments of agentic systems.

Centralized oversight of indices: Regulatory or standard-setting institutions could establish and maintain indices of agentic systems like this one. Integration with model registries: Incorporate indices of agentic systems into broader registry frameworks, ensuring unified reporting of agentic systems, common safety benchmarks, and clearer accountability mechanisms.

Impact Statement

This work was undertaken to improve our collective understanding of the emerging field of agentic AI. Its contributions revolve around the compilation and analysis of publicly available information, supplemented by correspondence with developers. In Section 6, we discuss how transparency standards can be 'gamed,' and note that this was one reason that we did not score developers using the index. Taken together, we hope the methodology and findings introduced by the AI Agent Index inform progress toward better risk management practices and governance frameworks for agentic AI systems.

You have reached the end of the paper.