AI Agent Tool Taxonomies Unveiled at CAISI NIST Workshop

January 2024 — Approximately 140 AI experts convened at a workshop hosted by the Consortium for AI Safety and Innovation (CAISI) and the National Institute of Standards and Technology (NIST) to address the growing need for standardized taxonomies of tools used in AI agent systems. The event, part of the AI Safety and Innovation Consortium (AISIC), aimed to create a shared vocabulary for developers, deployers, and researchers to improve transparency and risk management.

Key Takeaways from the Workshop

Participants identified multiple approaches to categorize AI agent tools, including:

1. Functionality-focused: What actions does the tool enable?
2. Access patterns: Can tools access external resources or be configured with write permissions?
3. Risk-based: How critical is the tool to potential harms? Are actions reversible?
4. Reliability: Can the tool be used consistently?
5. Modality: Is the tool text-based, robotic, or multimodal?
6. Monitoring: What level of observability does the tool provide?
7. Autonomy: How much discretion does the agent have in tool use?

Workshop participants emphasized that no single taxonomy suffices. Instead, multidimensional frameworks combining these approaches may offer the most promise.

Proposed Taxonomies

1. Functionality-Oriented Taxonomy

This taxonomy helps developers communicate capabilities and constraints, such as filtering risky search results.

2. Constrained Tool Access Patterns

This framework complements risk assessments, aligning with resources like NIST AI 600-1.

Next Steps

CAISI and NIST encourage stakeholders to adapt these taxonomies and provide feedback via CAISI-agents@nist.gov. The effort underscores the importance of transparency in AI agent development as tools grow more sophisticated and pervasive.

"Tool taxonomies are one method to improve transparency on capabilities and deployments along the AI agent value chain," the organizers noted.