Leading AI Language Models Fall Short of Upcoming EU Regulations, Stanford Study Warns

Sounding a note of caution, a team of Stanford researchers warned that the world's leading ten AI language models are poised to fail the stringent standards laid out by the European Union's forthcoming AI Act. Should they not meet the regulations, these AI entities could face significant regulatory risks and potentially heavy financial penalties.

The EU’s AI Act, approved in a parliamentary vote on June 14th, is currently on the pathway to becoming official law. As the world’s first comprehensive set of AI regulations, it stands to impact over 450 million individuals, while also serving as an example that nations such as the US and Canada are likely to draw inspiration from in crafting their own AI regulations.

Implications for Foundation Models

Despite recent clarifications that exempt foundation models like GPT-4 from the "high-risk" AI category, generative AI models are still subject to an array of requirements under the AI Act. These include mandatory registration with relevant authorities and essential transparency disclosures, areas where many models fall short.

The price of non-compliance is hefty: fines could exceed €20,000,000 or amount to 4% of a company's worldwide revenue. Furthermore, open-source generative AI models are required to meet the same standards as their closed-source counterparts, raising questions within the open-source community of legal risk and exposure.

The Non-compliant Landscape

In the study, researchers evaluated ten leading AI models against the draft AI Act's 12 fundamental compliance requirements. Alarmingly, most models scored less than 50% in overall compliance.

Notably, closed-source models like OpenAI's GPT-4 only garnered 25 out of a possible 48 points. Google's PaLM 2 fared slightly better with a score of 27, while Cohere’s Command LLM managed just 23. Anthropic’s Claude languished near the bottom with a meager 7 points.

On the other hand, open-source model Hugging Face’s BLOOM performed the best, securing 36 points. However, other open-source models, such as Meta’s LLaMA and Stable Diffusion v2, merely achieved 21 and 22 points respectively.

Noteworthy Patterns and Observations

A notable trend emerged from the aggregate results: open-source models generally outperformed closed-source models in several critical areas, including data sources transparency and resource utilization. Conversely, closed-source models excelled in areas such as comprehensive documentation and risk mitigation.

However, the study points out significant areas of uncertainty. One such area is the murky "dimensions of performance" for complying with the numerous requirements set forth in the AI Act. Moreover, the question of enforcement remains unresolved, and the researchers warn that a lack of technical expertise could hinder the EU’s ability to regulate these foundation models effectively.

The Way Forward

Despite these concerns, the researchers advocate for the implementation of the EU's AI Act. They argue it would "act as a catalyst for AI creators to collectively establish industry standards that enhance transparency" and bring about "significant positive change in the foundation model ecosystem."