In Sergio Leoni\u2019s 1966 epic, \u2018The Good, the Bad and the Ugly\u2019, three cowboys navigate a series of dramatic challenges before unearthing a bounty of gold. It might seem like a stretch at first, but this strikes as a strangely fitting framework for event driven architecture in business today.<\/p>\n
Why?<\/p>\n
Firstly, if you\u2019re yet to embrace event driven architecture<\/a>, you\u2019re like the cowboy characters. You\u2019re already<\/em> caught in a series of dramatic challenges. You just don\u2019t know it yet.<\/p>\n Whatever your sector, your customers demand real-time, personally relevant interactions. From banking to retail, everyone wants \u2018instant\u2019: updates on order fulfillment and product delivery, stock and inventory levels, transaction information, and more. If you\u2019re unable to provide easy, immediate and meaningful updates\u2014across complex customer journeys for internal and external stakeholders alike\u2014you\u2019re on the cusp of becoming irrelevant. And investing more in cumbersome and outdated infrastructure like siloed ERP or monolithic systems\u2014or pouring even more data into data lakes without making changes to the way you use that information\u2014is simply not the answer.<\/p>\n In today\u2019s customer experience, batched processing isn\u2019t enough. You need data you can use in the moment.<\/p>\n If this resonates, you\u2019re in luck. Remember, there\u2019s a bounty of gold at the end of this story.<\/p>\n Event driven architecture is being embraced by the planet\u2019s most successful enterprises, business leaders and technologists. Currently, 80% of the Fortune 100 use event streaming to power their business via Kafka<\/a>. Just a few weeks ago, Confluent<\/a>\u2014a cloud data streaming company closely linked with event driven architecture\u2014listed on the Nasdaq<\/a>, with shares jumping 25% after IPO for a valuation of $9.1 billion USD<\/a>. If \u2018data is the new oil<\/a>\u2019, there\u2019s clearly a gold rush for events.<\/p>\n Unfortunately, for many CEOs and CTOs, the value of event driven architectures remains buried; obscured under mounds of perceived risk, caution, and general misconception. It doesn\u2019t need to be this way.<\/p>\n So, to help you find the gold, leverage the significant benefits of event driven architectures, and allay initial concerns, let\u2019s shine a light on:<\/p>\n What are the benefits?<\/p>\n In an event driven architecture, individualised services are configured to perform one hyper-specific task; everything is largely decoupled.<\/p>\n Microservices are configured to consume events and perform singular actions without interdependency. Adding an additional service\u2014or series of services\u2014to your existing architecture is quick, easy and effective. In many cases you can simply configure the new service to consume an existing stream of events for a new business requirement. The result? Newer features and differentiators for your end customers, faster. And greater flexibility and speed of implementation internally.<\/p>\n Let\u2019s look at an example from our work with one of Australia\u2019s pioneering digital banks<\/a>. In serving transaction information to customers via a mobile app, we configured a read-only view of every customers\u2019 balance and transaction history. This is provisioned via a microservice called \u2018account view\u2019 and updated in real-time based on events occurring at a customer account level.<\/p>\n The master data is managed in the core banking platform, rather than the \u2018account view\u2019 service. However, by reproducing the information and storing it locally in \u2018account view\u2019, we return near real-time transactional information back to the customer with lightning-fast response times. The alternative? Querying the core banking platform, slowly retrieving all the information and eventually surfacing it for view.<\/p>\n From a redundancy perspective, the local read-only view of account information ensures customers can still access recent transaction history and details, even when services can\u2019t connect with the core banking platform. This is just one of many possible examples.<\/p>\n In event driven architecture, multiple microservices can consume the same \u2018type\u2019 of events. For example, sending an SMS every time a customer receives a deposit. Imagine ten million customers receive deposits on a daily basis. One service, continuously hit to send messages at this scale, will inevitably struggle to perform its function.<\/p>\n However you could easily deploy 50 identical microservices that can all work independently but perform the same function, and \u2018listen\u2019 for the same type of \u2018send SMS\u2019 events. Whenever a service reads a \u2018send SMS\u2019 event, it is marked as \u2018read\u2019, so there will be no duplication. All 50 services now actively listen for the next opportunity to consume a \u2018send SMS\u2019 event and perform the function. That\u2019s seamless continuity of customer experience, at scale.<\/p>\n Twenty years ago, it made sense to have everything in one place. Historically, dealing with a monolith of code, it was beneficial to provide immediate access to everything the monolith required to function. This negated the significant challenges associated with network-based communication. It removed the difficulty of querying external systems to work out interfacing, or the sourcing and processing of data.<\/p>\n Now, with smaller, more independent services in the cloud, decentralization becomes a more natural approach. Services can validly be distributed across clouds, or even organisations. Why struggle to create and configure one enormous monolith that attempts to do everything? A better approach is to develop services that perform one extremely specific function, then configure them with the infrastructure required to serve the business in the best way possible. This improves performance of services at an individual level, while reducing chains of interdependencies across the board. And reduced dependency means less risk<\/a> for your organisation overall.<\/p>\n This is particularly useful for any organisation using a data pipeline to capture events, without yet embracing event driven architecture. Stored events can be easily replayed for evolving business needs; a new marketing promotion or feature for banking customers who have historically deposited $5,000 each month for the past 12 months, for example. Historical events can quickly spin up multiple views of customer behaviors and profiles, creating crucial value for the future.<\/p>\n From a system perspective, events make it easy to slot third party systems in and out as business requirements evolve. For example, consider triggering an event for a deposit to a banking account. This event is consumed by a banking connector service, which passes the information into the core banking service. The business then determines a need for a different core banking product and wants to replace it. All that\u2019s required from an implementation perspective is the creation of another banking connector service, and the new core product can immediately consume the same events being passed to the legacy core banking product, without any implication or effort for the wider platform.<\/p>\n What are some of the perceived shortcomings? Are there strategies or ways to overcome them?<\/p>\n The perceived issue:<\/strong> with so many events, producers and consumers orbiting around different business flows and processes, it all seems too complicated.<\/p>\n The response:<\/strong> the architecture is ultimately a reflection of your business domain and the complexity that is inherent within it. Regardless of how that complexity is distributed or manifest, it still exists: whether centralised as a monolith<\/a> or distributed and decentralised as microservices. The benefit of decentralising and embracing events? You understand that complexity better. You have more granular visibility of the components that comprise the complexity. You have more control over each component. And they\u2019re all easier to maintain and change. With greater control and clarity as to the most complex aspects of your business domain, you\u2019re empowered to make decisions to change\u2014and even potentially reduce\u2014the overall complexity.<\/p>\n The perceived issue:<\/strong> in a point-to-point structure, service A can only talk to service B. If multiple producers and consumers of events are constantly shifting data points on and off the queue(s), does that equate to more opportunities for compromised data?<\/p>\n The response:<\/strong> producers and consumers of events can be restricted to interact with specific queues, with very clear definitions of which services can write and consume events. Additionally, you could also share keys to encrypt and decrypt data as each event goes on or off a queue. However, in most instances that approach seems prohibitively complex; it\u2019s analogous to locking and unlocking every door in the house each time you go to the bathroom. By locking the front and back doors (or restricting services to only produce or consume certain events in relation to specific queues), you protect what\u2019s important, with the same level of robustness seen in point-to-point.<\/p>\n The perceived issue:<\/strong> when you\u2019ve got a straight line of process\u2014A-B-C-D\u2014it\u2019s easy to follow a path of informational flow. Conversely, when you put an event on a queue and three different services consume that same event, with another service being triggered to consume a secondary event based on one of those services, things become harder to follow. It\u2019s difficult to pinpoint the source of an issue.<\/p>\n The response:<\/strong> there\u2019s no extra effort, just a reframing of approach. It\u2019s true that you can\u2019t \u2018stack-trace\u2019 in event driven architectures, but there are other options. Remember, each microservice is ideally hyper-specialised, with one dedicated and precise function. If a service receives an event that doesn\u2019t contain everything it needs, one of two situations has occurred:<\/p>\n The tracking process may not be linear, but there are far better defined \u2018contracts of connection\u2019 between individualised microservices.<\/p>\n The perceived issue:<\/strong> monitoring is trickier because each service is independent. If there\u2019s a knock-on effect, where one service passes the wrong piece of data down the line, it can be challenging to identify.<\/p>\n The response:<\/strong> there are standard tools to use for monitoring of distributed microservices in event driven architectures. For example, you have a range of \u2018out of the box\u2019 options with Confluent Cloud<\/a>. Or alternatively, you can hook into AppDynamics<\/a>. For cloud-based instances you can use Prometheus<\/a> and Grafana<\/a>. Crucially, each individual microservice should be configured with an ability to be monitored. Each service needs to be up and available. If a certain service interacts with a third-party platform, tag it as doing so; these services can and should be responsible for understanding their own status, including the status of the third party platform they interact with.<\/p>\n The perceived issue:<\/strong> if service A puts an event on a queue, and service B picks it up, carries out an action, puts it onto another queue, service C picks it up, carries out some action, and puts this subsequent event on a queue\u2026 then service D picks it up and there\u2019s a failure, how do you roll back to \u2018C\u2019, \u2018B\u2019, and \u2018A\u2019?<\/p>\n The response:<\/strong> this is a question of design. Where there are related services, it\u2019s worth investing time up front to determine whether you want (or need) to make them asynchronous. See below for more on the importance of investing in initial design thinking. Alternatively, acknowledging that these scenarios can take place, you can also retrospectively implement a SAGA pattern<\/a>. Event driven architectures are inherently more flexible and extensible than a monolithic counterpart.<\/p>\n What are some of the general misconceptions that lead to issues when approaching an event driven architecture? How do things sometimes \u2018get ugly\u2019?<\/p>\n Event driven architectures, by their nature, are asynchronous. The ability to process flows in non-linear and concurrent fashion is crucial to a large part of the value they deliver.<\/p>\n However, there\u2019s still a valid role for synchronous, point-to-point processing. Particularly in those instances where there\u2019s a clear flow or user intent. For example, consider processing a payment and notifying a banking customer of successful transaction. In certain scenarios, it\u2019s worth pausing to ask: does this need to be asynchronous, or are we needlessly overcomplicating things? A point-to-point flow can still be event driven\u2014creating events, placing them onto message platforms, etc\u2014without the need for over-engineering.<\/p>\n To embrace event driven architecture effectively is to invest in design thinking upfront. Events are a set of ideas and frameworks for solving sophisticated business challenges, rather than a rigid set of rules or a series of platforms. Simply adding an event backbone will not solve the issues or deliver the most value. In the same way you can\u2019t use the engine from a semi-trailer in a Mini Cooper, you can\u2019t just add Apache Kafka<\/a> or Azure Event Hub<\/a> to your existing architecture and expect results. Take the time to look at individual problems, needs and challenges\u2014at a micro level\u2014and determine the best way forward. Ideally, with an experienced guide who\u2019s done it all before with a wide range of organisations.<\/p>\n","protected":false},"excerpt":{"rendered":" Using the Western cinematic epic to understand and explore event driven architecture. In Sergio Leoni\u2019s 1966 epic, \u2018The Good, the Bad and the Ugly\u2019, three cowboys navigate a series of dramatic challenges before unearthing a bounty of gold. It might seem like a stretch at first, but this strikes as a strangely fitting framework for […]<\/p>\n","protected":false},"author":158,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[5,3],"tags":[],"location":[397],"class_list":["post-9309","post","type-post","status-publish","format-standard","hentry","category-our-thinking","category-tech-focus"],"acf":[],"yoast_head":"\nEvent driven architecture is the new gold standard for leading enterprise<\/h2>\n
\n
Event Driven Architecture: the good.<\/h2>\n
Developing and deploying new services is simple and efficient.<\/h3>\n
Reduce latency and redundancy.<\/h3>\n
Infinite scalability for robust and flexible services.<\/h3>\n
Decoupled and decentralized services can be \u2018best-of-breed\u2019 for their unique need.<\/h3>\n
Extensibility: both of data and platforms.<\/h3>\n
Event Driven Architecture: the bad.<\/h2>\n
\u201cEvent driven architecture is overly complex.\u201d<\/h3>\n
\u201cSecurity is compromised by event driven architecture.\u201d<\/h3>\n
\u201cEvent driven architecture makes debugging more complicated.\u201d<\/h3>\n
\n
\u201cDistributed, highly decoupled services make monitoring a challenge.\u201d<\/h3>\n
\u201cRollback becomes impractical, if not impossible.\u201d<\/h3>\n
Event Driven Architecture: the (potentially) ugly.<\/h2>\n
\u201cEverything has to be asynchronous.\u201d<\/h3>\n
\u201cThere\u2019s a one-size-fits-all approach\u201d.<\/h3>\n