- The U.S. Department of Defense (DoD) completed a pilot program, the Crowdsourced AI Red-Teaming (CAIRT) Assurance Program, to test AI chatbots in military medical services. The initiative evaluated large-language models (LLMs) for summarizing clinical notes and providing medical advice, involving over 200 participants from various military health organizations.
- The program identified over 800 potential vulnerabilities and biases in AI systems, highlighting challenges in deploying AI for critical military medical tasks. These findings will inform the development of benchmark datasets to evaluate future AI tools and ensure they meet rigorous military standards.
- The CAIRT program used crowdsourced red-teaming, engaging diverse stakeholders, including clinical providers and future beneficiaries, to accelerate vulnerability identification and build trust in AI systems. The effort was conducted in partnership with the nonprofit Humane Intelligence.
- The pilot underscores the DoD’s commitment to balancing AI innovation with accountability, ensuring tools are both effective and secure. It aligns with broader national security priorities, as AI development is increasingly seen as critical to maintaining U.S. military competitiveness.
- The findings will guide future research, policy development and best practices for AI in military medicine. The DoD’s cautious yet forward-thinking approach serves as a model for responsible AI integration, emphasizing transparency, oversight and collaboration to mitigate risks.
The U.S. Department of Defense (DoD) has taken a significant step forward in integrating cutting-edge artificial intelligence (AI) technologies into military operations, concluding a pilot program that tested the use of AI chatbots in military medical services. The initiative, known as the Crowdsourced AI Red-Teaming (CAIRT) Assurance Program, marks a critical milestone in the Pentagon’s efforts to harness AI for national defense while addressing its potential risks and vulnerabilities.
The pilot, led by the DoD’s Chief Digital and Artificial Intelligence Office (CDAO), focused on evaluating large-language models (LLMs) for two key applications: summarizing clinical notes and providing medical advice to military personnel. Over 200 participants, including clinical providers, healthcare analysts and experts from the Defense Health Agency (DHA) and the Uniformed Services University of the Health Sciences, collaborated to test three prominent chatbot models. Their goal was to identify potential weaknesses, biases and vulnerabilities in these AI systems before they could be deployed in real-world military medical scenarios.
The results were both revealing and sobering. The exercise uncovered over 800 findings of potential vulnerabilities and biases, highlighting the challenges of relying on AI for critical tasks in military medicine. These findings will now serve as the foundation for developing benchmark datasets, which the DoD plans to use to evaluate future AI tools and vendors. According to the DoD, this effort will ensure that any AI systems deployed in the future meet the rigorous performance and security standards required for military applications.
Dr. Matthew Johnson, the CDAO’s lead for the initiative, emphasized the importance of this pilot in shaping the DoD’s approach to generative AI (GenAI). “Since applying GenAI for such purposes within the DoD is in earlier stages of piloting and experimentation, this program acts as an essential pathfinder for generating a mass of testing data, surfacing areas for consideration, and validating mitigation options,” Johnson said. The findings will not only guide future research and development but also inform policies and best practices for the responsible use of AI in military medicine.
The CAIRT program, conducted in collaboration with the technology nonprofit Humane Intelligence, represents a novel approach to AI testing. By leveraging crowdsourced red-teaming – a method that uses adversarial techniques to test system robustness – the DoD was able to engage a diverse group of stakeholders, including potential future beneficiaries of these technologies. This approach not only accelerates the identification of vulnerabilities but also fosters a sense of ownership and trust among those who may eventually rely on these systems.
The CDAO, established in June 2022, has been at the forefront of the Pentagon’s push to integrate AI and digital technologies into defense operations. Its mission is to accelerate the adoption of data, analytics and AI across the DoD, ensuring that the U.S. military remains at the cutting edge of technological innovation. This latest pilot underscores the office’s commitment to balancing innovation with accountability, ensuring that AI tools are both effective and secure.
Essential to national security
The DoD’s efforts come at a time of heightened global competition in AI development. Last November, a bipartisan congressional commission urged the U.S. to prioritize AI development, likening the effort to the Manhattan Project. The commission recommended that the secretary of defense designate AI projects with the highest national priority, reflecting the growing recognition of AI as a critical component of national security.
Meanwhile, tech giants like Meta are also stepping up their involvement in defense applications. The company recently began offering its AI model, Llama, to the U.S. military and defense contractors for national security purposes. This trend highlights the increasing convergence of private-sector innovation and military needs, raising important questions about oversight, ethics and the potential risks of relying on AI in high-stakes environments.
The DoD’s chatbot pilot is a reminder that while AI holds immense promise, it is not without its challenges. The discovery of over 800 potential vulnerabilities underscores the need for rigorous testing and oversight, particularly when these technologies are applied to critical areas like military medicine. As the Pentagon continues to explore the potential of AI, it must remain vigilant in addressing these risks, ensuring that innovation does not come at the expense of security or reliability.
In an era defined by rapid technological advancement, the U.S. military’s cautious yet forward-thinking approach to AI serves as a model for responsible innovation. By prioritizing transparency, accountability and collaboration, the DoD is laying the groundwork for a future in which AI enhances, rather than undermines, national security.
As the CDAO continues to refine its AI strategies, the lessons learned from this pilot will undoubtedly shape the future of military medicine – and perhaps even the broader landscape of AI in defense. For now, the Pentagon’s message is clear: AI is a powerful tool, but its success depends on our ability to use it wisely.
Sources include:
Please contact us for more information.