Chatbot Not Found: Long Beach Shares Lessons Learned From 'Elby'

It’s possible most residents in Long Beach, Calif., have never heard of Elby. The city-enabled chatbot, known as Ask Elby!, never got much air time.

An unremarkable test phase last year prompted the city to pause the rollout of Elby, following survey satisfaction feedback along the lines of “very middle of the road,” Ryan Kurtzman, the city’s technology partnerships officer, said.

“It really didn’t noticeably improve the user experience, or resolution of issues over traditional search bars or other means of addressing resident inquiries,” Kurtzman said during a panel last week at the second annual GovAI Coalition Summit in San Jose, Calif. The session, “Progress Lab: Real-World Lessons from Failure,” was devoted to public-sector tech officials discussing projects which did not get off the ground, or were hobbled by any number of pitfalls.

Following an internal test of the chatbot, Long Beach launched a one-month external test with 10 residents, which led to more than 15,000 queries in that month and more than 4,000 users.

“We learned a lot,” Kurtzman said, adding, part of what was learned was the AI-powered chatbot often supplied old data, incorrect data, missing links and other bits of digital dead-ends. “We were getting people saying they were getting press releases and news from 2020 instead of 2024. Things like that. Overall, the system just didn’t really perform in the way that we had hoped.”

Long Beach decided against proceeding with a full implementation of the project, and instead focused on improving the core technology, which is the website and its search functions, “rather than adding on a new layer of AI to that,” Kurtzman said.

“I would never call this project a failure, by any means. I consider this very much a learning lesson for us,” he said. “Throughout the process we learned a lot about what we needed to do, what we needed to invest in, and what we didn’t need to focus on.”

Kurtzman and his counterparts in Fairfield, Calif., and South Bend, Ind., recounted similar outcomes where technology projects were derailed — sometimes due to larger structural problems at city hall — and did not lead to the often-promised improvements in operations, efficiencies or resident services.

In Fairfield, Eudora Fleischman, the city’s IT infrastructure and security manager, recounted what could best be described as a communications breakdown as she explained the rocky rollout of a new dispatching system for the city’s fire stations.

“None of the IT department actually knew that this project was actually going on,” Fleischman told the room. She ultimately put together an internal city committee to flesh out the project and determine its full scope and cost, which was then presented to the City Council and approved.

“And then we were able to have clarity, direction, and get started on the project, as a unified whole,” said Fleischman, who noted the dispatching system has been in place for two years.

“Considering that this required major construction across all of the fire departments in the city, tying into lights, utilities, and automatic shutoff valves for gas stoves and barbecues, it was kind of a big deal that communication was not there,” Fleischman said.

Anmol Rathore, technology equity manager in the Department of Innovation and Technology at South Bend, recalled an upgrade to the city’s 311 service which worked internally for about a month “and did not scale.”

“We realized we needed to look at the problem more closely,” she told the panel. “You can optimize for everything and still fail. Because you have to optimize for the right things.”

The modernization project offered access to multiple languages with a service available any time of the day, among other features. However, Rathore said, “The cost outweighed the benefits.”

Officials stressed pilot projects should be designed with performance measures, in order to validate whether the technology solution is addressing the issue for which it was intended. And the projects should be structured for adequate data gathering.

“It’s really important to be honest with yourselves,” Kurtzman said, noting the common pressure to have a pilot and then move to a wider scale and full implementation. “It’s OK to admit you’re not ready sometimes.”

“We have to constantly remind ourselves every day that we are in the people business, we are not in the technology business. … We have to work for our people,” Rathore said. Once you take a look at the organization, she said, you may start to see the “cracks, within your own organization, in your processes, in your knowledge base management system. And that is a good starting point.”

Kurtzman said he’d like to return to the chatbot project and “validate the problem” with Elby.

“We are planning to go out and explore this again, but really identifying what the needs are first,” he said.

This article was originally published by Government Technology, Industry Insider – California's sister publication.