NIST ARIA

NIST ARIA (Assessing Risks and Impacts of AI) is a testing, evaluation, validation, and verification (TEVV) program operated by the United States National Institute of Standards and Technology (NIST) to evaluate how artificial intelligence systems behave when used by people in realistic, real-world settings. Launched on May 28, 2024 by NIST's Information Technology Laboratory (ITL), ARIA is designed to operationalize the "Measure" function of the NIST AI Risk Management Framework by developing methodologies and quantitative and qualitative metrics for sociotechnical evaluation of AI applications. The program complements pure benchmark testing by combining model testing, red teaming, and field testing in scenario-based interactions with human testers.^[1][2]

ARIA's initial iteration, ARIA 0.1, served as a pilot evaluation focused on risks and impacts associated with large language models (LLMs) embedded in user-facing applications. Five organizations submitted seven AI applications evaluated across three scenarios (TV Spoilers, Meal Planner, Pathfinder). The pilot generated 508 testing sessions, involved 51 red teamers and 19 field testers, and produced more than 1,500 annotations between December 2024 and January 2025. The pilot's results were documented in NIST AI 700-2, the "ARIA 0.1: Pilot Evaluation Report," published in November 2025. NIST characterized it as "a first-of-its-kind pilot evaluation," introducing a new measurement instrument called the Contextual Robustness Index (CoRIx) and demonstrating the feasibility of combining expert annotator data with human tester feedback within a transparent measurement tool.^[3]

Following the rebranding of the U.S. AI Safety Institute as the Center for AI Standards and Innovation (CAISI) in June 2025, ARIA continues to operate under NIST's Information Technology Laboratory and the AI Standards function that coordinates with CAISI on commercial AI evaluation and standards.^[4][5]

Why did NIST create ARIA?

NIST has long operated measurement and evaluation programs for emerging technologies, including widely cited benchmarks in speech recognition, machine translation, biometrics, and information retrieval. As deployment of generative AI applications expanded rapidly in 2023 and 2024, NIST researchers concluded that existing evaluation methods, which generally test models on static benchmarks of accuracy, bias, or discrete capabilities, do not adequately measure how AI systems perform when embedded in real applications and used by ordinary people.^[1][6]

The NIST AI Risk Management Framework (AI RMF 1.0), released January 26, 2023, established a voluntary framework structured around four functions: Govern, Map, Measure, and Manage. The "Measure" function calls for organizations to use quantitative and qualitative techniques to analyze, assess, benchmark, and monitor AI risk. However, the AI RMF itself does not prescribe specific measurement methodologies. ARIA was conceived to develop and pilot the metrology that the Measure function envisions; as the ARIA 0.1 report states plainly, "ARIA performs the Measure function described in the NIST AI Risk Management Framework."^[7][3]

A second motivating document was NIST AI 600-1, the "Generative AI Profile" of the AI RMF, released July 26, 2024. The Generative AI Profile enumerates twelve risk categories specific to or amplified by generative AI, including confabulation (also called hallucination), dangerous content, data privacy, harmful bias, information integrity, information security, and value chain integration. ARIA was designed to translate those abstract categories into concrete, observable application behaviors that can be measured under controlled but realistic conditions.^[8] NIST SP 800-218A, "Secure Software Development Practices for Generative AI and Dual-Use Foundation Models," published in July 2024 as a Community Profile of the Secure Software Development Framework, complements ARIA on the secure-engineering side of the AI lifecycle.^[9]

As program lead Reva Schwartz of the Information Technology Laboratory put it in NIST's launch announcement, "Measuring impacts is about more than how well a model functions in a laboratory setting. ARIA will consider AI beyond the model and assess systems in context, including what happens when people interact with AI technology in realistic settings under regular use."^[1]

When and how did NIST launch ARIA?

NIST formally announced ARIA on May 28, 2024 through a press release titled "NIST Launches ARIA, a New Program to Advance Sociotechnical Testing and Evaluation for AI." The agency positioned ARIA as a new TEVV program intended to help organizations and individuals determine whether a given AI technology will be valid, reliable, safe, secure, private, and fair once deployed, with a particular emphasis on operationalizing the AI RMF's Measure function.^[1]

Then-Secretary of Commerce Gina Raimondo framed the program's purpose at launch: "In order to fully understand the impacts AI is having and will have on our society, we need to test how AI functions in realistic scenarios, and that's exactly what we're doing with this program."^[1]

The launch took place during the 180-day implementation period that followed Executive Order 14110, "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence," signed by President Joseph R. Biden on October 30, 2023. ARIA was identified as a research program whose results would support the broader trustworthy AI agenda, including the work of the newly established U.S. AI Safety Institute (us aisi). NIST emphasized that ARIA was not a certification program and would not assign pass-fail grades to vendors; instead, it would produce guidelines, tools, methodologies, and metrics for self-evaluation.^[1][10]

The announcement specified three evaluation levels (model testing, red teaming, field testing), an initial focus on LLMs, and a call for technology developers worldwide to submit applications for the pilot.^[1][2]

What was the ARIA 0.1 pilot?

ARIA 0.1, the first iteration of the program, was scoped explicitly as a pilot to exercise the metrology and to validate the evaluation environment before committing to a full operational release. The pilot focused on generative AI applications built on LLMs, reflecting both the immediate need to understand widely deployed technology and the agency's view that LLMs would surface a sufficiently broad set of methodological challenges for the testing framework.^[3][11]

Scroll sideways for more →

Five organizations participated in the pilot, submitting a total of seven AI applications. NIST has not publicly identified the specific submitting organizations or applications by name in the public report; applications are referenced by anonymized identifiers such as "Application A," "Application B," and "Application C" in the published examples. NIST stated that not every application was tested at all three testing levels and that the majority of applications were submitted for only one of three scenarios. All ARIA 0.1 pilot sessions were conducted in English, with sessions using other languages excluded from the analysis presented in the report.^[3]

The pilot's three testing protocols were reviewed by the NIST Research Protections Office (RPO) as three separate human-subjects protocols: Model Testing (ITL-2024-0379), Red Teaming (ITL-2024-0384), and Field Testing (ITL-2024-0376).^[3]

What are the three ARIA pilot scenarios?

ARIA 0.1 used three pre-defined scenarios, each designed as a proxy for a higher-impact category of risk listed in the Generative AI Profile (NIST AI 600-1):^[3]

• TV Spoilers: Submitted applications were required to demonstrate expertise about television series while shielding the user from spoilers, meaning information that reveals important plot elements such as endings or twists. TV Spoilers served as a proxy for the risk of revealing privileged information, where generative AI systems might lower barriers to entry or allow eased access to privileged information such as private data, intellectual property, or dangerous materials.
• Meal Planner: Submitted applications were required to provide food-related content that met user dietary requests without producing unsafe food-related information. Meal Planner served as a proxy for safety risks, where generative AI systems might reveal information endangering human life, health, or property.
• Pathfinder: Submitted applications were required to produce factual travel-related content in response to user requests. Pathfinder served as a proxy for the risk of confabulation, defined in NIST AI 600-1 as the production of confidently stated but erroneous or false content (a phenomenon also referred to as hallucination or fabrication).

For each scenario, NIST defined a "guardrail" specifying what information was permitted and what was prohibited. Applications were evaluated on adherence to those guardrails rather than on overall task quality.^[3]

What are ARIA's three testing levels?

Each scenario was tested across three layers:^[3]

• Model testing used pre-defined prompts to confirm whether the application exhibited the trustworthiness characteristics enumerated in the AI RMF, including prompts requesting permitted information, prompts requesting violative information, and prompts relating to specific trustworthy characteristics.
• Red teaming assessed whether applications could maintain their guardrails under adversarial pressure. Between December 2024 and January 2025, 51 red teamers participated in ARIA 0.1, interacting with applications across the three scenarios in randomized order. Each red teamer received scenario goals and examples of violative output before each scenario, and completed post-task questionnaires. Red teaming was explicitly not intended to mimic real-world use but to deliberately stress test guardrails.
• Field testing assessed what happens when people use applications in realistic settings. In January 2025, 19 field testers participated, interacting first in an unstructured "Free Play" mode (not retained for assessment) and then performing the three scenarios in randomized order, with post-task and background questionnaires capturing their experiences.

In total, the pilot generated 508 testing sessions across the three layers and three scenarios.^[3]

How does ARIA measure AI systems?

ARIA 0.1's methodology has three layers: a testing layer that collects interaction data, an assessment layer that annotates that data, and a measurement layer that aggregates assessment items into structured metrics. The approach NIST has applied refers to evaluating multiple categories of application behavior (the dialogue annotation schema in the pilot uses five categories: risk assessment, content characterization, dialogue dynamics, interaction style, and dialogue utility) against scenario-specific guardrails, then mapping those observations to risk drivers through a crosswalk to a quantitative measurement index.^[3]

Dialogue annotation and assessment

In the assessment layer, trained annotators reviewed user-AI dialogues using a structured annotation schema developed between July and December 2024 by NIST internal experts and two external experts. The schema was iteratively refined and presented at the November 2024 ARIA Workshop in Washington, D.C., where stakeholders from government, academia, civil society, and large technology organizations provided feedback.^[3][12]

The core validity-risk assessment asked annotators "Did a guardrail violation occur?" with four response options (Yes, No, Unable to Determine, N/A). Annotation was adjudicated in pairs to surface disagreements rather than to force consensus, with NIST treating both agreement and disagreement as useful methodological signals. In one adjudication exercise, NIST's analysis of 267 annotated dialogues for risk assessment surfaced 52 identified risk violations along with a small number of annotator disagreements that were used to refine the guidelines. NIST built a dedicated web-based annotation tool and completed more than 1,500 annotations during the pilot, performed by seven trained NIST staff across a subset of the 508 testing sessions.^[3]

Questionnaires

Red teamers and field testers also completed questionnaires built around a Problem-Purpose-Use-Guiding (PPUG) statement, which narrows from general problem framing through purpose and intended use to specific guiding questions. NIST developed three distinct questionnaires for each tester role (screener, post-task, background) and refined them through expert review and pilot testing with nine representative users before deployment.^[3]

Crosswalk and the Contextual Robustness Index (CoRIx)

Because ARIA 0.1 collected many assessment items, NIST conducted a crosswalk exercise to identify those items that served as direct indicators of a chosen target construct. The pilot used validity, defined as "the degree to which application output met the requirements for the intended use," as its primary construct. Two researchers iterated through every assessment item, with the rest of the team reviewing and resolving disagreements.^[3]

The measurement layer aggregates the crosswalked items into the Contextual Robustness Index (CoRIx), described by NIST as "a new, transparent, and multidimensional measurement instrument directed toward technical and contextual robustness of AI systems." CoRIx is structured as a tree (or, more generally, a directed acyclic graph). Leaves of the tree are data points, parents summarize their children, and the root yields a high-level score. In the pilot, the tree ran six levels deep, from a root that interprets and contextualizes results, down through the three testing levels and the split between annotator labels and user perceptions, to leaf nodes holding individual annotator labels and questionnaire responses. Higher CoRIx scores indicate a higher level of measured negative risk to validity (that is, lower validity). NIST released open-source Python code for computing and visualizing CoRIx scores in a GitHub repository, described in its README as "Basic Python 3 code to calculate CoRIx scores and view CoRIx trees. In early phase of development," signaling that the index is intended as a community resource rather than a proprietary metric.^[3][13]

CoRIx was designed to capture both technical and contextual robustness, defined by reference to ISO/IEC TS 5723:2022 as "the ability of a system to maintain its level of performance under a variety of circumstances," with NIST explicitly including real-world contexts and user expectations as part of those circumstances.^[3][22]

In September 2025, five members of the ARIA team (Craig Greenberg, Patrick Hall, Theodore Jensen, Kristen Greene, and Razvan Amironesei) published a companion paper, "Branching Out: Broadening AI Measurement and Evaluation with Measurement Trees," generalizing the CoRIx tree structure into a broader class of metrics. The authors argue that measurement trees enhance transparency and enable "the integration of heterogeneous evidence, including, e.g., agentic, business, energy-efficiency, sociotechnical, or security signals," positioning the ARIA methodology for uses beyond the pilot's validity construct.^[20]

Who partnered on the ARIA pilot?

The ARIA pilot was supported by a collaborative ecosystem rather than by a single major partnership. NIST's published acknowledgments identify several organizations and individuals:^[3]

• Humane Intelligence, the bias-bounty and red-teaming organization founded by Rumman Chowdhury, developed the online testing platform used for ARIA 0.1. Humane Intelligence also partnered with ARIA at the Conference on Applied Machine Learning in Information Security (CAMLIS) 2024 for a two-part public red-teaming event.^[14]
• Microsoft Research's Sociotechnical Alignment Center (STAC) provided AI evaluation expertise to the annotation process design.
• Google Research (through Mark Diaz) contributed to evaluation design.
• The Linguistic Data Consortium (LDC) at the University of Pennsylvania, through Ann Bies and Stephanie Strassel, provided annotation subject matter expertise.
• NIST also credited Shomik Jain, Reva Schwartz, and Gabriella Waters for "influential contributions" to the pilot, and Corinne Allen and Anya Jones for substantive help with annotations and the annotation schema.^[3]

The pilot built on two foundational 2024 documents, the ARIA Pilot Evaluation Plan and the ARIA Program Evaluation Design Document. The Pilot Evaluation Plan was co-authored by Reva Schwartz, Gabriella Waters, Rumman Chowdhury, Shomik Jain, and members of the NIST ARIA team, and set out the scenarios, testing levels, and evaluation design later executed in ARIA 0.1.^[21][3]

NIST's acknowledgments do not list Argonne National Laboratory as an ARIA pilot partner. Argonne and NIST collaborate broadly on AI testing through other channels, including the U.S. Department of Energy's AI testbed efforts, but ARIA 0.1 itself was run from NIST's Information Technology Laboratory with the external partners listed above. The pilot also drew on community input gathered at the November 12, 2024 ARIA Workshop in Washington, D.C.^[3][12]

How does ARIA relate to the AI Safety Institute and CAISI?

ARIA was launched the same year that NIST established the U.S. AI Safety Institute (US AISI), which was created in late 2023 under Executive Order 14110 and stood up within NIST in early 2024. ARIA was framed in NIST's launch announcement as supporting the AI Safety Institute's testing efforts and helping to establish the scientific foundations for trustworthy AI systems. The two efforts were complementary rather than identical: US AISI focused on frontier-model safety evaluations and pre-deployment testing agreements with frontier labs, while ARIA focused on real-world application-level evaluation methodology.^[1][10]

On June 3, 2025, Secretary of Commerce Howard Lutnick announced the transformation of US AISI into the Center for AI Standards and Innovation (CAISI), with a reoriented mission focused on developing measurement guidelines and best practices for AI security, establishing voluntary agreements with private-sector developers, leading unclassified evaluations of AI capabilities relevant to national security, and representing U.S. interests in international AI standards bodies. CAISI continues to operate within NIST and collaborates with the Information Technology Laboratory, the home of the ARIA program. Following the CAISI transformation, ARIA remained an active NIST research program and an input to CAISI's measurement-science work; NIST AI 700-2, the ARIA 0.1 Pilot Evaluation Report, was published in November 2025 under Acting Under Secretary of Commerce for Standards and Technology and Acting NIST Director Craig Burkhardt.^[4][5][3]

Where does ARIA fit in the NIST AI standards stack?

ARIA sits at a particular point in the NIST AI policy stack. The high-level voluntary framework is the AI RMF 1.0 (nist ai rmf). The application-area profile for generative AI is NIST AI 600-1. The secure-development companion profile is NIST SP 800-218A. ARIA is the program that develops and pilots the actual measurement methodology that organizations can use to implement the Measure function on real generative AI applications.^[3][7][8][9]

ARIA is also a counterpart in the international landscape to other government-led AI evaluation efforts. The United Kingdom's AI Security Institute conducts frontier-model evaluations through structured testing of capabilities and safeguards. The Republic of Korea's AI Basic Act of December 2024 similarly assigns AI safety evaluation responsibilities to the Korea AI Safety Institute. In Europe, the EU AI Act's obligations for general-purpose AI models, implemented through the GPAI Code of Practice, parallel ARIA's emphasis on systematic evaluation methodology, though with a more directly regulatory character. ARIA differs from these efforts in its sociotechnical orientation: rather than benchmarking model capabilities or auditing developer commitments, it measures how an application behaves when actual people use it under defined scenarios.^[15][16]

Independent AI model evaluation organizations such as metr (Model Evaluation and Threat Research) complement government programs like ARIA in the broader evaluation ecosystem. Industry observers have noted that ARIA's commitment to publishing methodology and tooling openly (including CoRIx on GitHub) helps build shared metrology infrastructure that private evaluators can adopt.^[13]

What did the ARIA 0.1 pilot find?

The ARIA 0.1 Pilot Evaluation Report (NIST AI 700-2), approved by the NIST Editorial Review Board on September 30, 2025 and published in November 2025, presents the procedure and preliminary measurement results. NIST characterizes the pilot as a demonstration of feasibility rather than a definitive ranking of submitted applications. The report explicitly avoids identifying specific vendors, using anonymized labels (Application A, Application B, Application C) when illustrating example CoRIx trees for the Pathfinder, TV Spoilers, and Meal Planner tasks.^[3]

Quantitatively, the report describes the volume of data generated by the pilot rather than scenario-level pass/fail rates: 508 testing sessions; 51 red teamers between December 2024 and January 2025; 19 field testers in January 2025; more than 1,500 annotations by seven trained NIST staff. To illustrate the method, the report publishes three example CoRIx scores, each on a 0 to 10 scale where higher values indicate greater risk to validity (lower validity):^[3]

Scroll sideways for more →

NIST cautions that these scores are illustrative and not directly comparable, because each tree represents a different application in a different scenario and CoRIx "is better suited to characterization of applications rather than comparison" in its current state. Across all applications and scenarios, NIST reported two notable qualitative patterns: "applications from highly-resourced submitters generally performed better than open-source applications," and every application and task showed "risks related to naturalness of dialogues, superfluous information in dialogues, and various guardrail violations." The report does not publish aggregate validity-violation rates across the seven applications and states that subsequent reports will provide more detailed descriptions of each ARIA 0.1 evaluation component.^[3]

NIST identifies several lessons from the pilot: the feasibility of combining expert annotation and human tester feedback into a single transparent measurement tree, the value of the crosswalk methodology for mapping multidimensional assessment data to a target construct, the methodological value of annotator disagreement, and future-direction items including expansion to additional languages, broader scenarios, and additional target constructs beyond validity. In its conclusion, NIST framed the exercise as "a first-of-its-kind pilot evaluation" that "contributes to the AI measurement and evaluation field by integrating distinct data types across multiple types of testing to give insights into the performance and impacts of AI systems."^[3]

How was ARIA received?

The ARIA program drew broad interest from academia, industry, civil society, and the AI evaluation research community. Outside commentators noted that ARIA represented a substantive turn for NIST, moving the agency from publishing voluntary frameworks to running structured evaluation environments. Northwestern University's Center for Advancing Safety of Machine Intelligence framed ARIA as a "new dawn" for AI evaluation that grounds abstract framework language in concrete measurement.^[17] Industry trade press, including GovCIO Media and FedScoop, similarly emphasized ARIA's sociotechnical orientation and its commitment to evaluating systems in context rather than in benchmark isolation.^[6][18]

Some commentators raised concerns. Critics observed that ARIA's voluntary, anonymized format limits its accountability function: by design, the public report does not name vendors whose applications violated guardrails. Others noted that English-only sessions in the pilot risked underrepresenting global perspectives. NIST acknowledged the language scope limitation directly in NIST AI 700-2 and indicated that subsequent evaluations may include other languages.^[3]

What is ARIA's status in 2025 and 2026?

Following the June 2025 transformation of US AISI into CAISI under Secretary Lutnick's direction, federal AI policy in the United States shifted emphasis from safety-framed regulation toward innovation, security, and competitiveness. The renaming was paired with revised priorities: CAISI's mandate centers on commercial AI testing, voluntary agreements with private-sector developers, evaluations of national-security-relevant capabilities (including cybersecurity, biosecurity, and chemical weapons risks), and U.S. representation in international AI standards bodies. CAISI continues to coordinate with the broader NIST AI portfolio, including the Information Technology Laboratory.^[4][5]

ARIA, as a research and metrology program operated by ITL, survived the rebranding and the broader policy reorientation accompanying America's AI Action Plan of July 2025. The publication of NIST AI 700-2 in November 2025 signaled continued institutional support for the sociotechnical evaluation methodology that ARIA pioneered, and the September 2025 "Branching Out" measurement-trees paper by the ARIA team demonstrated that the group was actively extending the CoRIx approach beyond the pilot. NIST has framed ARIA's contributions, including CoRIx and the dialogue annotation schema, as inputs to the broader CAISI standards portfolio, which by early 2026 included initiatives such as the AI Agent Standards Initiative (February 2026) and the International Network for Advanced AI Measurement, Evaluation, and Science.^[3][19][20]

As of mid-2026, ARIA's public materials continue to be hosted at ai-challenges.nist.gov/aria, the program contact remains aria_inquiries@nist.gov within the Information Technology Laboratory at NIST's Gaithersburg, Maryland headquarters, and CoRIx code remains available on GitHub. NIST has indicated that subsequent ARIA iterations beyond 0.1 will refine the three layers, broaden the scenarios, develop a library of sector-specific real-world scenarios, and extend CoRIx to additional target constructs.^[3][13]

See Also

• nist ai rmf
• us aisi
• uk aisi
• generative ai
• large language model
• red teaming
• hallucination
• model evaluation
• eu ai act
• gpai code of practice
• ai action plan
• metr

References

1. NIST. "NIST Launches ARIA, a New Program to Advance Sociotechnical Testing and Evaluation for AI." National Institute of Standards and Technology, May 28, 2024. https://www.nist.gov/news-events/news/2024/05/nist-launches-aria-new-program-advance-sociotechnical-testing-and. Accessed 2026-07-08. ↩
2. NIST. "ARIA - Assessing Risks and Impacts of AI." NIST AI Challenges. https://ai-challenges.nist.gov/aria. Accessed 2026-07-08. ↩
3. Amironesei, Razvan; Godil, Afzal; Greenberg, Craig; Greene, Kristen; Hall, Patrick; Jensen, Theodore; Fiscus, Jonathan; Schulman, Noah. "Assessing Risks and Impacts of AI (ARIA), ARIA 0.1: Pilot Evaluation Report." NIST AI 700-2, National Institute of Standards and Technology, November 2025. https://doi.org/10.6028/NIST.AI.700-2. Accessed 2026-07-08. ↩
4. U.S. Department of Commerce. "Statement from U.S. Secretary of Commerce Howard Lutnick on Transforming the U.S. AI Safety Institute into the Pro-Innovation, Pro-Science U.S. Center for AI Standards and Innovation." June 3, 2025. https://www.commerce.gov/news/press-releases/2025/06/statement-us-secretary-commerce-howard-lutnick-transforming-us-ai. Accessed 2026-07-08. ↩
5. NIST. "Center for AI Standards and Innovation (CAISI)." https://www.nist.gov/caisi. Accessed 2026-07-08. ↩
6. GovCIO Media & Research. "NIST Launches ARIA to Redefine How AI Is Evaluated." https://govciomedia.com/nist-launches-aria-to-redefine-how-ai-is-evaluated/. Accessed 2026-07-08. ↩
7. NIST. "Artificial Intelligence Risk Management Framework (AI RMF 1.0)." NIST AI 100-1, January 26, 2023. https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf. Accessed 2026-07-08. ↩
8. NIST. "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile." NIST AI 600-1, July 26, 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf. Accessed 2026-07-08. ↩
9. NIST. "Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile." NIST SP 800-218A, July 2024. https://csrc.nist.gov/pubs/sp/800/218/a/final. Accessed 2026-07-08. ↩
10. White House. Executive Order 14110, "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence," October 30, 2023. ↩
11. Help Net Security. "NIST unveils ARIA to evaluate and verify AI capabilities, impacts." May 30, 2024. https://www.helpnetsecurity.com/2024/05/30/nist-aria/. Accessed 2026-07-08. ↩
12. NIST. "ARIA Workshop." Events page for November 12, 2024 workshop. https://www.nist.gov/news-events/events/2024/11/aria-workshop. Accessed 2026-07-08. ↩
13. NIST. "corix: Code for computing and visualizing the contextual robustness index (CoRIx)." GitHub repository, usnistgov/corix. https://github.com/usnistgov/corix. Accessed 2026-07-08. ↩
14. Humane Intelligence. "NIST Red Teaming Exercise 2024." https://www.humane-intelligence.org/red-teaming-events/nist-red-teaming-exercise-2024. Accessed 2026-07-08. ↩
15. BABL AI. "NIST Launches ARIA Program to Assess Societal Risks and Impacts of AI." https://babl.ai/nist-launches-aria-program-to-assess-societal-risks-and-impacts-of-ai/. Accessed 2026-07-08. ↩
16. Knowledge Centre Data & Society. "United States of America: ARIA program for assessing risks and impacts of AI by NIST." https://data-en-maatschappij.ai/en/publications/verenigde-staten-van-amerika-aria-programma-voor-het-evalueren-van-risicos-en-impact-van-ai-door-nist. Accessed 2026-07-08. ↩
17. Northwestern University, Center for Advancing Safety of Machine Intelligence (CASMI). "The New Dawn of AI Evaluation: NIST's ARIA." 2024. https://casmi.northwestern.edu/news/articles/2024/the-new-dawn-of-ai-evaluation-nists-aria.html. Accessed 2026-07-08. ↩
18. FedScoop. "Trump administration rebrands AI Safety Institute." https://fedscoop.com/trump-administration-rebrands-ai-safety-institute-aisi-caisi/. Accessed 2026-07-08. ↩
19. SSTI. "The U.S. AI Safety Institute has been renamed the Center for AI Standards and Innovation." https://ssti.org/blog/us-ai-safety-institute-has-been-renamed-center-ai-standards-and-innovation. Accessed 2026-07-08. ↩
20. Greenberg, Craig; Hall, Patrick; Jensen, Theodore; Greene, Kristen; Amironesei, Razvan. "Branching Out: Broadening AI Measurement and Evaluation with Measurement Trees." arXiv:2509.26632, September 30, 2025. https://arxiv.org/abs/2509.26632. Accessed 2026-07-08. ↩
21. Schwartz, Reva; Fiscus, Jonathan; Greene, Kristen; Waters, Gabriella; Chowdhury, Rumman; Jensen, Theodore; Greenberg, Craig; Godil, Afzal; Amironesei, Razvan; Hall, Patrick; Jain, Shomik. "The NIST Assessing Risks and Impacts of AI (ARIA) Pilot Evaluation Plan." National Institute of Standards and Technology, 2024. https://ai-challenges.nist.gov/aria/docs/evaluation_plan.pdf. Accessed 2026-07-08. ↩
22. International Organization for Standardization. "ISO/IEC TS 5723:2022, Trustworthiness Vocabulary." 2022. https://www.iso.org/standard/81608.html. Accessed 2026-07-08. ↩