State Releases Generative AI Procurement Guidelines

Seven months after Gov. Gavin Newsom signed Executive Order N-12-23 directing further study of generative AI technologies, new guidelines will cement how state agencies and vendors need to approach future procurements.

The guidelines — titled GenAI Guidelines for Public Sector Procurement, Uses and Training — contain updated definitions of AI and generative AI (GenAI) and mandates surrounding the incidental and intentional purchase and implementation of the tools across agencies and programs. The guidelines were developed by the Government Operations Agency (GovOps), California Department of Technology (CDT), Department of General Services (DGS), Office of Data and Innovation (ODI) and the Department of Human Resources (CalHR).

Agency responsibilities for the incidental purchase of AI include assigning an executive-level team member for continuous monitoring and evaluation, mandatory training for the agency’s executive and procurement team and annual review of training and policy to ensure acceptable tool use.

The guidelines around intentional procurements are more extensive and include the identification of business needs, possible implications and a “comprehensive discovery process;” creation of a culture of engagement and communication between state staff and end users; assessment of risks and impacts; the preparation of quality data inputs and model testing; and the creation of a team responsible for the continuous evaluation of GenAI use and implications across operations.

Additionally, state entities will be required to complete a Generative Artificial Intelligence Risk Assessment (SIMM 5305-F) to determine the level of risk exposure associated with a planned GenAI deployment. The new rules also list requirements for vendors around the identification and disclosure of “any GenAI technology” that is part of a procurement.

As of the release of the guidelines, “all IT, non-IT, and telecommunications solicitations, regardless of acquisition type or method, must contain the GenAI disclosure notification language requiring vendors to identify any GenAI technology. This must be fully implemented on or before April 30, 2024,” according to a CDT letter that accompanied the announcement.

Where training is concerned, three phases have been identified. The first phase includes executive-level leadership, legal, labor and privacy specialists. The second phase includes program staff and technical experts. The third phase includes the general workforce and end users. Purchasing officials will have access to the training resources related to identifying AI purchases beginning March 29, according to the publication.