Platform or Die - Leveling the Playing Field for Enterprises
Part 1 - The case for an opinionated engineering platform (OEP)
Seven out of the ten most valuable companies in the world in 2021 are Digital Platforms compared to two out of ten, just a decade ago. Only those two digital platform companies from 2010 still continue to remain in the top ten in 2021 while the others have been overtaken (see Figure 1)! According to the TheInnovator.news, these platforms create digital experiences, communities and marketplaces that allow different groups to interact and transact at planet scale. More than 30% of global economic activity - some $60 trillion - could be mediated by digital platforms in six years’ time, according to a McKinsey research report and yet experts estimate only 3% of established enterprises have adopted an effective platform strategy. No wonder every Enterprise wants to become a Digital Platform company or at the least build a digital platform(s) to augment their customer experience. It has nothing but accelerated during the pandemic.
Figure 1: Top 10 most valuable global companies - 2010 vs 2020
Source: data from Wikipedia; inspiration from TheInnovator.news
While it is obviously very lucrative to be a Digital Platform company evidenced by their extraordinary valuations, it is also obvious that it is hard to pull it off. There is plenty of literature about network effects, marketplace, cold start, liquidity etc. - all difficult business challenges for an Enterprise to overcome - however, the under-appreciated facet of all these digital platform companies is how extremely good they are at software engineering. They are very good at software engineering because they invest as much in building their own engineering platform as they do on the products and services, they build using that platform.
According to Andy Jassy (CEO of Amazon) in an interview on the origins of AWS, around 2003, even after hiring many more software engineers they still weren’t able to build applications faster. They found that while management expected to build applications in three months, it took them nearly three months just to build the foundational infrastructure components like compute, storage, database etc. So, they built an infrastructure as-a-service platform, first internally, and then externally as AWS. Rest is history. There are many more examples of this - kubernetes from Google, Spinnaker from Netflix, React from Facebook, the list goes on - every digital platform company in that top 10 list has its own engineering platform.
If delivering delightful digital customer experience is an imperative to be a digital platform business, then an ‘Opinionated Engineering Platform’ (OEP) is an imperative to be good at software engineering. In this article let's explore what is a platform, engineering platform, and Opinionated Engineering Platform (OEP), why it needs to be opinionated, and how to build and operate one from an Enterprise standpoint.
Horizontal vs. Vertical / Internal vs. External
Before we dive into OEP, let's first define what a Digital Platform is and what makes something a Platform. According to techopedia - A (Digital) Platform is a group of technologies that are used as a base upon which other applications, processes, or technologies are developed. So, any digital thing can be a Platform if it helps someone build something valuable for someone! No wonder there are so many Platforms, and everything is (claims) to be a Platform :-) To classify, let's use two dimensions - the type of users and the surface area of the use cases. Types of users can be internal or external to the company that created the platform and surface area of the use cases can be vertical or horizontal based on the applicability (see Figure 2). A vertical platform solves problems specific to a domain like payments or rideshare while a horizontal platform solves pervasive problems like computing or communication.
Figure 2: Classification of Platforms
Irrespective of how platforms are classified, they all have one thing in common - they provide a repeatable solution, to an undifferentiated activity (usually), in the value stream of how its users deliver value to their users. In the case of an Engineering Platform (Horizontal/External), it provides an automated repeatable solution to the undifferentiated activity of compiling, packaging, provisioning, deploying, and monitoring in the value stream of software engineering. In the case of our OEP (Horizontal/ Internal), it contextualizes that solution to the specific Enterprise’s ways of working.
Cognitive load and Compliance load
Why is this ‘Contextualization’ and ‘Opinionation’ important in an OEP? Afterall, it will be much easier to buy one of those Horizontal/External Engineering Platforms and start using as-is internally by training your users instead of spending time and effort on making it opinionated. There are many valid reasons for enterprises to be opinionated as their infrastructure is hybrid and multi-cloud, risk of vendor lock-in etc. but the two most important reasons are - Abstraction and Workflow.
History of computer science is full of examples of how Abstraction has accelerated adoption and innovation. Take the Computer itself, the CPU complexities are abstracted through the ISA, the operating system complexities are abstracted through the POSIX, and the software defined data center (SDDC) complexities are abstracted through the Cloud Provider APIs (see Figure 3). What this enables is, scaled innovation at each layer without being burdened by the internals of the previous layer. It dramatically reduces the cognitive load to become a programmer at any of the higher layers.
So, when building an internal engineering platform, it's important to abstract away the mechanics of the platform from the users and expose APIs that are relevant to your Product Owners, Developers, Testers, and SREs (PODTS) to do their job. The APIs offered by the existing Horizontal/External Engineering Platforms in the market are not PODTS centric. New engineering platforms like Kubernetes with its APIs like ‘Deployments’ and ‘Statefulsets’ is becoming more PODTS centric, but it still imposes a high cognitive load and is incomplete in enabling what the PODTS community needs to get done.
Figure 3: Computer engineering layers and abstractions
Source: Derived from the book - Kubernetes: Up and Running by Burns, Beda, and Hightower
This brings me to my second reason - Workflow. Any Enterprise, regulated or unregulated, has its own workflow and rules for software to go from idea to production. A regulated Enterprise has additional rules from their auditors to follow. These rules add another type of load to your PODTS community called ‘compliance load’. The OEP should codify these rules and provide API primitives that can be stitched together into an end-to-end workflow that is inherently compliant. It's important to appropriately design the API primitives so they are flexible enough to create custom workflows for different PODTS teams, as we know no two PODTS teams follow the same workflow.
The APIs offered by the existing Horizontal/External Engineering Platforms in the market do not enable an end-to-end workflow. Some tools like Gitlab and Azure DevOps are making strides in this direction, but they are not end-to-end and require programming (declarative or imperative) to enable custom workflows. So, an Opinionated Engineering Platform is required to provide abstraction which reduces cognitive load, and compliant end-to-end workflows which reduces compliance load. In the next part, let's explore how to build an Opinionated Engineering Platform.