The anticipated cost of a whole new Grant Delivery System (GDS), according to the state Department of General Services (DGS) and the California Department of Technology (CDT), was simply too high. But something had to be done, CSAC’s Chief Information Officer Gurinder Bains told Techwire in a recent interview. CSAC’s technology was badly outdated.
“The GDS was the system of the past, and it was written in classic ASP (Active Server Pages),” Bains said. “It had multiple single points of failure. If a database failed, someone had to contact CDT to switch servers. At times, CSAC was down almost 16 hours per week while doing backups.”
The code, Bains said, “was classic ASP with SQL nested. … Anytime a new Legislature comes in and they wanted to add $20 to a grant, it was nightmarish because you don’t know, out of a million lines of code, where we are going to change the code. So with disparate systems, different languages, no one knew how to support, (there was) no architecture. Over time, programs keep adding, and people just keep writing the code without thinking where it’s going to go.”
Rebuffed by CDT and DGS with his initial pitch for a new system with a pricetag exceeding $6 million, Bains gathered his IT team — about 30 technologists serving a department of about 115 people — and broke the project up into smaller pieces. It was a textbook use of agile methodology, with smaller, easier-to-manage Requests for Offers, rather than larger, more complex Requests for Proposals.
A project that had been in the planning stages for more than five years was now moving toward fruition. Bains said he also emphasized the importance of security in the new plan: “We have millions of students’ data — and not only students’ data, but their parents’ data as well. So in case we were hacked … it would have cost the state lots of money.”
Bains said “I put my case forward” with the Governor’s Office, CDT and the Department of Finance. Eventually, he said, the project won CDT’s preliminary “Stage Gate 1 approval” to proceed. “That was cause for great celebration,” he said. Within a year, he said, CSAC’s team moved successfully through Stage Gates 2 and 3.
Bains then tested the waters with some large vendors, but he said their responses and cost estimates didn’t line up with his needs.
“Then I discussed with our business partners (within CSAC), like, we have tons of knowledge within our organization, but the challenge we would face moving forward is that people will be super-busy. We all will be drinking from a firehose if we just wanted to start the project on our own. But once I discussed with our business partners … and my boss, I was like, ‘If we don’t go through RFP, but break this project into multiple RFOs — and I, the CIO of CSAC, will act as the system integrator — I think we can finish this project within time and well below budget.”
Bains explained how the revision played out: “CDT’s Project Approval Lifecycle only goes with RFPs, Requests for Proposal, which is just, you put the bid out, have vendors go for it, and then pay millions of dollars and then complete the project.
“But when I went with the RFO approach and said I will be acting as the system integrator, some of the people at CDT liked it; some said, ‘Oh, you’re just putting a target on your back — you know, there will be consequences if (you) won’t be able to finish.’ I said, ‘I know, but I have run so many projects — my background is with Sun Microsystems … the Legislative Data Center. … We run many projects, and I’m very confident I will be able to run as long as I’m supported all around.’ So CDT … gave approval.
“So we went with multiple RFOs. We hired a project manager, scrum master, back-end engineers, front-end engineers, Oracle DBA … So we built a plan and hired all these people, and the rest is history. And now, in May 2021, within two years, we were able to finish everything that we had promised. We came up with the latest and greatest technology — containers, self-replicating containers.
“One of the problems was our students were not able to access our application with other than IE7 (Internet Explorer 7 browser), so we went with angular HTML5 Java script bootstrap, so now these applications are working on all platforms, even these latest technologies like cellphones, tablets, laptops, Chrome — whatever you want, it just works.
“And on databases, we put in Oracle appliances. … They’re automatically disaster-recovery and getting backed up all the way to Vacaville, which is CDT’s cloud storage.”
Sanjib Nayak, founder and chief executive officer of Sacramento-based xFusion Technologies, played a key role on the vendor side.
“xFusion analyzed CSAC’s needs and its RFOs, strategized with CSAC, and determined that the lower costs were attainable,” Nayak told Techwire in an interview. “We brought our methodology approach and design artifacts, and also some of the technology artifacts … We have the cloud ready to go. We know how to do it; we’ve done it many times. … A lot of different technology components, we just brought with us.” Nayak said he included some proprietary technology at no cost and with no license fees.
The challenge, Nayak said, was “how we can take this big elephant and cut it up into small pieces … so the business user doesn’t need to wait two years to get it. In eight months, we took that system live. We never changed any team members. They were all there from day one.”
Now, two years after the inception of the GDSM (Grant Delivery System Modernization), “everything is working,” CIO Bains said.
“It was a hard project. It was a total of seven applications in one portal … so we went with a student landing page, we implemented identity management, and used open source there. CDT was a little reluctant — ‘Oh, open source’ — but we told them this is the best way to go, and the least expensive. And once they looked at … the open source, it’s called WSO2, they (CDT officials) were all on board. They were like, ‘Yeah, this looks like good technology. Move forward.’
“Now, on this WSO2 identity management system, in the last two years almost 500,000 new accounts were created, and people are using it on a daily basis to check their grants. And we have centralized that identity management system all the UCs, CSU, private schools and high schools — all the administrators have accounts there, and they’re using the system as well.
“The system is much more secure than it was. It has built-in APIs, so for example, if a college wanted to know the status of a student, they were just sending us their Social Security number, then we were running the batch jobs, which would take one, two days — sometimes up to seven days. Now those things are taking less than seconds.
“We have leaped forward in so many things, and the only reason is because all these local vendors supported us, and staff stepped up their game and, over time, CDT helped, the Governor’s Office helped, the Department of Finance kept the money flowing, and the Legislature helped, and execs helped, and eventually, the project is up and running.”
Bains said the COVID-19 remote work protocols that began last March were little more than a speed bump. Staff and contractors were working remotely within three days, he said, using Microsoft Teams for communication.
“We didn’t see any lack of interest or slowdown,” he said.
“I think this is one of the major achievements from CSAC,” Bains said. “What we did — we converted a 20-year-old application to a modernized application which is using AWS, which is using containers, which is using Oracle appliances. So we’re using kind of a hybrid approach with database on-prem, and also applications in the cloud with identity management implemented — and no single point of failure anywhere.”
Bains said CSAC has also beefed up its phone system to accommodate more demand and remote access. It’s also added an advanced Google chatbot, with plans to add multilingual capabilities.
“I think we were able to do all this because we’re a smaller organization, and people were listening, and I didn’t have to twist anyone’s arm. Things were not lost in communication.”
