Chief Information Officer Jon Walton said the county had had conversations with two vendors about shifting to a cloud-based authentication model.
“The genesis of our conversation was with ZScaler (and) Okta – those were our first steps toward that model,” Walton told Techwire in a recent interview. “But we were just starting that journey when this whole thing hit. … I was talking with other CIOs: It feels like we’ve progressed two or three years of IT maturity into like two or three weeks.”
Walton said the timing was fortuitous. San Mateo had been one of the first counties to do a soft activation of its emergency operations center, and its county manager and elected officials were proactive in encouraging employees to work from home. Telework, of course, ramped up following Gov. Gavin Newsom’s March 19 stay-at-home order, and now 60 to 70 percent of county workers work largely or entirely from home. The county uses two small data centers for its more traditional applications that haven’t migrated to cloud yet, but during the past five years, it had already moved most applications to the cloud.
But before telework could happen, there was the challenge of getting the hardware – the laptops – into the hands of county workers so they could work remotely while keeping government up and running.
“A lot of CIOs are seeing this,” Walton said. “When it hit, it disrupted the supply chain. When you have 30 percent of your users that used to telework and all of a sudden it’s 70 percent, that 40 percent all want laptops, and Dell, Microsoft, nobody could deliver the equipment. We put in four orders three weeks ago, and we’re getting ETA of end of April.”
Walton says swift reaction was the key to giving the county a leg up on hardware.
“We activated so early, and we were lucky because we got our supply chain requests in first. But even then, we placed an order for almost 2,000 laptops, and we’ve a total of 400 so far. And the rest of them are like, ‘TBD, we’ll let you know.’ It’s the same story over and over again. I’ve been on calls with state CIOs and everything else, and nobody can get equipment right now.
“We pushed out the laptops we could. For something quick, even an iPad will do, but if you’re sitting somebody down at a 13-inch screen on a tablet for eight hours, that just doesn’t work. We just had a conversation with our Risk Management-HR Department about, ‘OK, we’ve got all these requests now, people wanting wireless keyboards and mice and large-screen monitors delivered to their house, because if they’re going to work like this for another month or two, it’s going to turn into all sorts of ergonomic issues.”
Getting the laptops was a challenge – but, in the meantime, county business still had to be done, so officials spun up a VDI (virtual desktop infrastructure) environment, Walton said. “We’re just telling people, ‘Look, you’re going to have to use your own devices. A Chromebook, even – buy a Chromebook off Amazon and we’ll reimburse you for it if we have to. Then just use a Chrome browser to get into a VDI session to do your work.” Before sending laptops home, however, they did a full disk encryption, using ZScaler’s ZPA, Okta and related products to secure local devices and encrypt traffic through VPN tunnels. The county also spun up virtual desktops on VMware-Horizon on the Azure Cloud, which took about two weeks to launch.
ZScaler’s regional sales manager for State, Local and Education (SLED) West, Vaishali Patel, explained the challenge from the vendor’s point of view.
“Due to COVID-19, San Mateo County needed to quickly deploy a remote access solution that could scale to support thousands of users and provide fast, secure access to keep employees safe and productive as they work from home,” she told Techwire. “The county had already used Zscaler Internet Access (ZIA), a secure internet and web gateway delivered from the cloud, to help employees … while using the Internet but to also keep them (and the county) safe from malicious or compromised websites.”
The CIO said the help of vendors was essential to a quick campaign.
“For us, Dell has been really good,” he said. “We were fortunate that we’ve been a longtime Dell customer, so a lot of the things we needed to have done on the front end were done. When the laptops showed up for us, they were almost ready to go. ... I would recommend to any CIO – the more you can offload to your partner-vendors to do that, to say, ‘Configure my machines as close to done as possible, so when they show up they’re almost ready to go,’ I think that’s important.”
Walton lifted the hood and cited the other components and products that the county relied on.
“VMware has been a big, important customer to us, and Nutanix,” he said. “We run VMware on our Nutanix infrastructure. I’ve been running data centers and servers for 25 years, and this is the first time my entire server team hasn’t been in the office for three weeks. They’re all running the servers off their iPads and laptops from home. Zero downtime. With VMware, we’re running 1,200 virtual servers across two data centers, and it’s just solid. It performs and it doesn’t go down. If you’re going to do a hybrid cloud approach and still run some of your infrastructure internally, then I try to preach the gospel of hyperconvergence, simplified infrastructure.”
“For the application side, we’re a big partner of both AWS and Azure, so all of our backups are in the cloud now,” Walton said. “Our big applications, like our HR systems, our financial systems, email, most of our big health and public safety applications are all cloud-based now. Takes a lot of burden off our internal infrastructure, which is a good thing.”
Patel said, “Zscaler has been able to help counties the size of San Mateo, with several thousand users, as well as larger counties with 100,000+ employees such as L.A. County, to quickly deploy a robust and agile cloud-native remote access solution.”
And, Patel added, it helped that San Mateo was already looking to open up its remote-access capabilities.
She noted that “VPN and other legacy remote access technology cannot scale to cover an entire workforce and take months to deploy. … Second, the (county) team recognized the importance of application performance, and providing the best experience for the user possible. They also recognized the need to reduce bandwidth requirements and to deliver secure access through any device – whether government-issued or BYOD (bring your own device).”
“For us,” Walton said, “the beauty is they’re very secure and they’re really easy to use – we could push it out to all of our desktops and in integrates really well with Okta, which is what we use for dual-factor authentication. So anytime you connect to wireless or Wi-Fi, you’re automatically connected to the county network. And there’s so much more visibility and simplicity and controls in this ZScaler ZPA client. We can push it out, monitor it, connect, reconnect people – all automatically, behind the scenes.”
Walton said the way the solutions came together “has really opened us for us such a multitude of ways to access the data and information we need. Even though moving people home has been a big challenge, it’s gone smoothly.”
