AWS
Tracking the Effectiveness of Cloud Adoption
Some organizations have been more successful than others at realizing the value of cloud services. Organizations have long been using Key Performance indicators (KPIs) to measure the health and progress of business endeavors. What makes KPIs to track cloud adoption unique? How do we interpret and use them to make decisions and steer clear of pitfalls? In this post, we’ll look at the challenges and best practices for measuring and tracking the effectiveness of cloud adoption initiatives.
Choosing the wrong KPI can lead to undesirable outcomes. A year into one company’s cloud adoption, they realized that the quality of the software being delivered hadn’t really improved and developer productivity was nearly identical to on-premises. Over 80% of technical employees were cloud certified and yet the results weren’t impressive. What was going wrong? Well, first – training hours and certification counts are, what we call “vanity” metrics. While simple to measure, they provide a false sense of progress. These measures did not really track how the company was progressing towards one of their key goals, which in this case was to improve software delivery quality. A better metric would have been the “software release error rate”, which, when attributed to a team or a developer, would have provided actionable insights to improve quality. Additionally, another desired outcome was to decrease the time it took to provision infrastructure to the software development teams. It turned out that the company was following the same burdensome on-premises workflow requiring multiple approvals to request, provision and deploy infrastructure. They were not effectively leveraging the secure automated tools (“infrastructure as code”) that cloud provides which can significantly speed up provisioning infrastructure. Once the company began using the right KPIs and tools, they were quickly on their way to realizing the benefits of their cloud investments.
This example illustrates two principles that are critical to successful cloud adoption. First, it is important to choose the right KPIs to track progress towards desired objectives. Second, operating in the cloud requires companies to re-think and adapt their on-premises processes to take full advantage of the cloud’s capabilities.
Common Challenges in Tracking Cloud Adoption
The cloud is not a strategy in itself; it’s a remarkably powerful tool for accomplishing business outcomes. A common error is to think of cloud adoption merely as a “technology” initiative, while your real objectives are to improve business agility, operational resilience, and staff productivity and to reduce costs. In this case any attempt to measure progress of cloud adoption has to be broader than just IT operational metrics and should be tied to your primary business objectives. But, are the business objectives clearly known and unambiguous? Unfortunately, it is common to find that leaders have strong but conflicting opinions on cloud adoption objectives. Additionally, each enterprise is complex, unique and the pace of transformation and maturity will vary. What may work in one organization may not be appropriate for another. For example, a family-owned company that intends to be in business for the next 100 years will have different KPIs than a private equity firm that intends to resell the same business in a few years.
Cloud computing has prompted a shift in business practices. Infrastructure capacity has shifted from “fixed” to “unlimited” capacity and scale. Hardware, software and access can be provisioned quickly with automation. The cost model has shifted from fixed to variable. Governance has changed from manual controls to automated real-time continuous “guardrails”. As a result, progress metrics also have to change to align to this shift. Take the example of an IT infrastructure team, whose responsiveness is measured by how quickly they resolve tickets that are raised by application teams to provision infrastructure. In an on-premises environment, a common metric is the number of unresolved tickets in their queue. In the cloud, we strive to enable application teams to provision infrastructure on their own (self-service) without the need for raising tickets. Therefore, we measure effectiveness by tracking how many processes are automated and have self-service enabled. Here are some best practices to consider when tracking effectiveness of cloud adoption.
Selecting the right KPIs.
I hope we can all agree that measurement is a fundamental aspect of good management. When used in conjunction with effective strategic planning, the right KPIs serve as critical navigation tools, assisting organizations in understanding how well they are performing in terms of delivering on their strategic goals and provide timely opportunities to correct course. KPIs are just what they literally mean – they are “indicators” of “performance over time” to meet the “key” business objectives and not the ultimate goal. There is a difference between measuring progress and measuring results. For example, number of site visitors or number of app downloads is not a KPI as it does not drive performance.
Nailing the “Why” we are adopting cloud.
The only KPIs for determining the success of cloud adoption are those that measure whether it is accomplishing the purpose you set for adopting the cloud in the first place. Cost savings are often the initial catalyst for considering the cloud, but broader business impacts such as customer value, business agility, operational resilience and staff productivity are the more compelling benefits of cloud adoption. Setting and communicating unambiguous business objectives is the first step that will drive what needs to be measured to track progress. Objectives and Key Results (OKRs), for example are an effective goal-setting and leadership tool for communicating what you want to accomplish and what milestones you’ll need to meet in order to accomplish it. Objectives can be turned into quantifiable desired results, which then translate further into project, team and individual results, whose progress over time can then be measured thru KPIs. For example, if improving product quality is an objective, zero defects is a quantifiable result and the KPI is the defect rate.
Driving Alignment.
Rapid, transformative cloud adoption is unlikely without committed and engaged senior leadership. But building consensus on the adoption objectives, the quantifiable results and how KPIs are set and measured, is equally if not more critical for success. You will be surprised how many of transformational initiatives fall apart due to the lack of consensus. For example, where multiple applications interact with a centralized common service (e.g. central customer master database system), application owners who consume the common service may claim that any downtime it causes should not be counted against their apps. Clarifying up front how incidents will be attributed and measured is necessary to avoid confusion later.
Aggressive Goals enabled by Cloud Capabilities.
The natural tendency when setting goals is to take a conservative approach. Better is to set goals that are just short of being impossible. This way, even if you don’t meet the goal, you still learned and evolved in the process and built momentum. This is where cloud shines! Gone are the days when infrastructure needs to be forecast several years in advance with a buffer for contingencies and architecture had to be “future-proofed” to support unforeseen changes. In the cloud, infrastructure can grow or shrink automatically to adapt to the changing demands of an organization.
Focus on Business Value Measurement, Not just Technology Metrics.
Requesting the CFO or CEO to prioritize investments that would reduce technical debt or modernize technology often goes nowhere. We would be more successful if we can explain how not solving Technical Debt has consequences that include risks, a lack of agility, and increased costs for future IT work . Our focus needs to shift from process-driven metrics to result-driven metrics. For example, measuring the business impact (failed customer transactions or revenue impact of failures or lost employee hours) due to system unavailability, is far more important than measuring just uptime or downtime hours.
Rethink Legacy Metrics in Cloud.
Because of the cloud’s possibilities, we need to rethink how we measure progress. For example, a “zero storage growth rate” goal to contain costs may be considered a key goal in an on-premises environment, but on the cloud, you pay as you go. A better KPI for cost reduction would track what data is frequently accessed, which then drives what data can be archived to low cost storage to optimize costs. In the legacy model, system stability, for example is typically measured reactively, i.e. by tracking # of incidents after they surface. In the cloud, this is reversed – with the cloud’s advanced monitoring and instrumentation tools, the focus shifts to how many incidents were “proactively prevented” from causing downtime, which then drives the right behavior across teams.
Measure activities that drive performance and not just the “output”
Good KPIs measure how you are progressing towards your business objective, but they aren’t themselves the business objectives. Goodhart’s law, named after British economist Charles Goodhart is often stated as “When a measure becomes a target, it ceases to be a good measure”. The law explains that when a measure is used as an indicator of the performance, then it inevitably ceases to function as that indicator because people start to game it. Examples include downgrading the severity of defects to meet system reliability objectives and service reps avoiding taking complicated cases due to the impact on their productivity goals. If a hiring agency, for example, rated its employees’ effectiveness and incentivized them by only the number of interviews they conduct, it would encourage the employee to rush through the meetings without really helping clients find a job, thereby completely missing the spirit of the business objective – which is: to find jobs for its clients.
Picking Relevant Actionable Measures and the Importance of Baselining.
Today’s challenge is not to find more KPIs, but rather to pick those that are most relevant and valuable to the firm. Most often, a single KPI does not provide the full story around performance. For example, if your objective is to improve system availability, just tracking uptime hours is not enough. You will also need to measure the number of times the system goes offline. KPIs do not have to be perfect to start with and do not have to be all-encompassing. Start small by setting KPIs that are connected to the most important objectives and then refine, expand them over time. KPIs need to be actionable and not “vanity” metrics. Finally, measuring progress requires a baseline that shows your current level of performance. Without this frame of reference, measuring progress is pointless.
Leverage cloud capabilities to automate data collection and building dashboards.
Tracking cloud adoption requires a data-driven approach. The cloud comes with tools, automation and dashboards to collect performance data, which in an on-premises world, require significant investment. In addition to ease of data collection, cloud provides tools to build insights with little effort that can be used to measure and sustain performance.
References
AWS. (2020, June). Cloud Value Benchmarking Study Quantifies the Benefits of Cloud Adoption: https://pages.awscloud.com/rs/112-TZM-766/images/cloud-value-benchmarking-study-quantifies-cloud-adoption-benefits.pdf
AWS. (2021, November 22). An Overview of the AWS Cloud Adoption Framework. https://docs.aws.amazon.com/whitepapers/latest/overview-aws-cloud-adoption-framework/overview-aws-cloud-adoption-framework.pdf
Bernard Marr. (n.d.). Key Performance Indicators & Metrics. https://bernardmarr.com/key-performance-indicators-metrics/
BrandPost. (2015, Jun 3). 20 Ways to Measure the Success of Your Growing Cloud Investment.https://www.cio.com/article/247104/20-ways-to-measure-the-success-of-your-growing-cloud-investment.html/amp
Chakraborty, A. (n.d.). Goodhart’s Law: How Measuring The Wrong Things Drive Immoral Behaviour. https://coffeeandjunk.com/goodharts-campbells-law/
Hennesey, C. (2021, December 17). Funding more transformational capacity – Show me the money! https://aws.amazon.com/blogs/enterprise-strategy/funding-more-transformational-capacity-show-me-the-money/
Miles, R. H. (2010, January-February). Accelerating Corporate Transformations (Don’t Lose Your Nerve!). https://hbr.org/2010/01/accelerating-corporate-transformations-dont-lose-your-nerve
Opsdog. (n.d.). KPIs & Metrics Examples, Definition & Use. https://opsdog.com/categories/kpis-and-metrics
Panchadsaram, R. (n.d.). What is an OKR? Definition and Examples. https://www.whatmatters.com/faqs/okr-meaning-definition-example#:~:text=OKRs%20stand%20for%20%E2%80%9CObjectives%20and,encourage%20engagement%20around%20measurable%20goals.
Schwartz, M. (2021, January 20). The CIO-CFO Conversation: Capturing the State of IT. https://aws.amazon.com/blogs/enterprise-strategy/the-cio-cfo-conversation-capturing-the-state-of-it/
Schwartz, M. (2022, February 18). Measuring the Success of Your Transformation.https://aws.amazon.com/blogs/enterprise-strategy/measuring-the-success-of-your-transformation/
Schwartz, Mark. (2020, June 11). Mental Models to Clarify the Goals of Digital Transformation. https://aws.amazon.com/blogs/enterprise-strategy/mental-models-for-digital-transformation/
Scott And Company Advisor. (2018, September 20). Key Performance Indicators and Your Business – Part 2 of 2. https://www.scottandco.com/scott-company-advisor/key-performance-indicators-and-your-business-part-2-of-2/
Suer, M. (2018, May 29). How CIOs prove business value. https://www.cio.com/article/221711/how-cios-prove-business-value.html/amp
TAGS: Best Practices, Business Value, cloud, Digital Transformation, enterprise strategy
Mark Schwartz and Nurani Parasuraman
