Stream multi-tenant knowledge with Amazon MSK

Actual-time knowledge streaming has change into distinguished in immediately’s world of instantaneous digital experiences. Trendy software program as a service (SaaS) functions throughout all industries rely increasingly on repeatedly generated knowledge from completely different knowledge sources comparable to internet and cell functions, Web of Issues (IoT) gadgets, social media platforms, and ecommerce websites. Processing these knowledge streams in actual time is essential to delivering responsive and customized options, and maximizes the worth of knowledge by processing it as near the occasion time as attainable.

AWS helps SaaS distributors by offering the constructing blocks wanted to implement a streaming utility with Amazon Kinesis Knowledge Streams and Amazon Managed Streaming for Apache Kafka (Amazon MSK), and real-time processing functions with Amazon Managed Service for Apache Flink.

On this submit, we have a look at implementation patterns a SaaS vendor can undertake when utilizing a streaming platform as a method of integration between inner parts, the place streaming knowledge just isn’t immediately uncovered to 3rd events. Specifically, we concentrate on Amazon MSK.

Streaming multi-tenancy patterns

When constructing streaming functions, it’s best to take the next dimensions into consideration:

• Knowledge partitioning – Occasion streaming and storage must be remoted on the acceptable degree, bodily or logical, primarily based on tenant possession
• Efficiency equity – The efficiency coupling of functions processing streaming knowledge for various tenants should be managed and restricted
• Tenant isolation – A strong authorization technique must be put in place to verify tenants can entry solely their knowledge

Underpinning all interactions with a multi-tenant system is the idea of SaaS identification. For extra data, check with SaaS Structure Fundamentals.

SaaS deployment fashions

Tenant isolation just isn’t non-compulsory for SaaS suppliers, and tenant isolation approaches will differ relying in your deployment mannequin. The mannequin is influenced by enterprise necessities, and the fashions are usually not mutually unique. Commerce-offs should be weighed throughout particular person providers to attain a correct stability of isolation, complexity, and value. There isn’t any common resolution, and a SaaS vendor must rigorously weigh their enterprise and buyer wants towards three isolation methods: silo, pool and bridge (or combos thereof).

Within the following sections, we discover these deployment fashions throughout knowledge isolation, efficiency equity, and tenant isolation dimensions.

Silo mannequin

The silo mannequin represents the best degree of knowledge segregation, but in addition the best operating price. Having a devoted MSK cluster per tenant will increase the chance of overprovisioning and requires duplication of administration and monitoring tooling.

Having a devoted MSK cluster per tenant makes certain tenant knowledge partitioning happens on the disk degree when utilizing an Amazon MSK Provisioned mannequin. Each Amazon MSK Provisioned and Serverless clusters help server-side encryption at relaxation. Amazon MSK Provisioned additional means that you can use a buyer managed AWS Key Administration Service (AWS KMS) key (see Amazon MSK encryption).

In a silo mannequin, Kafka ACL and quotas just isn’t strictly required until your online business necessities require them. Efficiency equity is assured as a result of solely a single tenant can be utilizing the assets of all the MSK cluster and are devoted to functions producing and consuming occasions of a single tenant. This implies spikes of visitors on a particular tenant can’t affect different tenants, and there’s no danger of cross-tenant knowledge entry. As a disadvantage, having a provisioned cluster per tenant requires a right-sizing train per tenant, with the next danger of overprovisioning than within the pool or bridge fashions.

You possibly can implement tenant isolation the MSK cluster degree with AWS Identification and Entry Administration (IAM) insurance policies, creating per-cluster credentials, relying on the authentication scheme in use.

Pool mannequin

The pool mannequin is the only mannequin the place tenants share assets. A single MSK cluster is used for all tenants with knowledge break up into matters primarily based on the occasion kind (for instance, all occasions associated to orders go to the subject orders), and all tenant’s occasions are despatched to the identical subject. The next diagram illustrates this structure.

This mannequin maximizes operational simplicity, however reduces the tenant isolation choices accessible as a result of the SaaS supplier received’t have the ability to differentiate per-tenant operational parameters and all tasks of isolation are delegated to the functions producing and consuming knowledge from Kafka. The pool mannequin additionally doesn’t present any mechanism of bodily knowledge partitioning, nor efficiency equity. A SaaS supplier with these necessities ought to take into account both a bridge or silo mannequin. When you don’t have necessities to account for parameters comparable to per-tenant encryption keys or tenant-specific knowledge operations, a pool mannequin affords lowered complexity and could be a viable choice. Let’s dig deeper into the trade-offs.

A standard technique to implement client isolation is to establish the tenant inside every occasion utilizing a tenant ID. The choices accessible with Kafka are passing the tenant ID both as occasion metadata (header) or a part of the payload itself as an specific subject. With this method, the tenant ID can be used as a standardized subject throughout all functions inside each the message payload and the occasion header. This method can scale back the chance of semantic divergence when parts course of and ahead messages as a result of occasion headers are dealt with in another way by completely different processing frameworks and could possibly be stripped when forwarded. Conversely, the occasion physique is commonly forwarded as a single object and no contained data is misplaced until the occasion is explicitly remodeled. Together with the tenant ID within the occasion header as nicely might simplify the implementation of providers permitting you to specify tenants that have to be recovered or migrated with out requiring the supplier to deserialize the message payload to filter by tenant.

When specifying the tenant ID utilizing both a header or as a subject within the occasion, client functions won’t be able to selectively subscribe to the occasions of a particular tenant. With Kafka, a client subscribes to a subject and receives all occasions despatched to that subject of all tenants. Solely after receiving an occasion will the patron will have the ability to examine the tenant ID to filter the tenant of curiosity, making entry segregation nearly unattainable. This implies delicate knowledge should be encrypted to verify a tenant can’t learn one other tenant’s knowledge when viewing these occasions. In Kafka, server-side encryption can solely be set on the cluster degree, the place all tenants sharing a cluster will share the identical server-side encryption key.

In Kafka, knowledge retention can solely be set on the subject. Within the pool mannequin, occasions belonging to all tenants are despatched to the identical subject, so tenant-specific operations like deleting all knowledge for a tenant is not going to be attainable. The immutable, append-only nature of Kafka solely permits a complete subject to be deleted, not selective occasions belonging to a particular tenant. If particular buyer knowledge within the stream requires the best to be forgotten, comparable to for GDPR, a pool mannequin is not going to work for that knowledge and silo must be thought-about for that particular knowledge stream.

Bridge mannequin

Within the bridge mannequin, a single Kafka cluster is used throughout all tenants, however occasions from completely different tenants are segregated into completely different matters. With this mannequin, there’s a subject for every group of associated occasions per tenant. You possibly can simplify operations by adopting a subject naming conference comparable to together with the tenant ID within the subject title. It will virtually create a namespace per tenant, and in addition permits completely different directors to handle completely different tenants, setting permissions with a prefix ACL, and avoiding naming clashes (for instance, occasions associated to orders for tenant 1 go to tenant1.orders and orders of tenant 2 go to tenant2.orders). The next diagram illustrates this structure.

With the bridge mannequin, server-side encryption utilizing a per-tenant key just isn’t attainable. Knowledge from completely different tenants is saved in the identical MSK cluster, and server-side encryption keys may be specified per cluster solely. For a similar purpose, knowledge segregation can solely be achieved at file degree, as a result of separate matters are saved in separate information. Amazon MSK shops all matters inside the identical Amazon Elastic Block Retailer (Amazon EBS) quantity.

The bridge mannequin affords per-tenant customization, comparable to retention coverage or max message dimension, as a result of Kafka means that you can set these parameters per subject. The bridge mannequin additionally simplifies segregating and decoupling occasion processing per tenant, permitting a stronger isolation between separate functions that course of knowledge of separate tenants.

To summarize, the bridge mannequin affords the next capabilities:

• Tenant processing segregation – A client utility can selectively subscribe to the matters belonging to particular tenants and solely obtain occasions for these tenants. A SaaS supplier will have the ability to delete knowledge for particular tenants, selectively deleting the matters belonging to that tenant.
• Selective scaling of the processing – With Kafka, the utmost variety of parallel customers is set by the variety of partitions of a subject, and the variety of partitions may be set per subject, and subsequently per tenant.
• Efficiency equity – You possibly can implement efficiency equity utilizing Kafka quotas, supported by Amazon MSK, stopping the providers processing a very busy tenant to eat too many cluster assets, on the expense of different tenants. Discuss with the next two-part collection for extra particulars on Kafka quotas in Amazon MSK, and an instance implementation for IAM authentication.
• Tenant isolation – You possibly can implement tenant isolation utilizing IAM entry management or Apache Kafka ACLs, relying on the authentication scheme that’s used with Amazon MSK. Each IAM and Kafka ACLs help you management entry per subject. You possibly can authorize an utility to entry solely the matters belonging to the tenant it’s purported to course of.

Commerce-offs in a SaaS atmosphere

Though every mannequin supplies completely different capabilities for knowledge partitioning, efficiency equity, and tenant isolation, additionally they include completely different prices and complexities. Throughout planning, it’s necessary to establish what trade-offs you’re keen to make for typical clients, and present a tier construction to your consumer subscriptions.

The next desk summarizes the supported capabilities of the three fashions in a streaming utility.

Anti-patterns

Within the bridge mannequin, we mentioned tenant segregation by subject. Another can be segregating by partition, the place all messages of a given kind are despatched to the identical subject (for instance, orders), however every tenant has a devoted partition. This method has many disadvantages and we strongly discourage it. In Kafka, partitions are the unit of horizontal scaling and balancing of brokers and customers. Assigning partitions per tenants can introduce unbalancing of the cluster, and operational and efficiency points that can be onerous to beat.

Some degree of knowledge isolation, comparable to per-tenant encryption keys, could possibly be achieved utilizing client-side encryption, delegating any encryption or description to the producer and client functions. This method would help you use a separate encryption key per tenant. We don’t advocate this method as a result of it introduces the next degree of complexity in each the patron and producer functions. It might additionally stop you from utilizing many of the customary programming libraries, Kafka tooling, and most Kafka ecosystem providers, like Kafka Join or MSK Join.

Conclusion

On this submit, we explored three patterns that SaaS distributors can use when architecting multi-tenant streaming functions with Amazon MSK: the pool, bridge, and silo fashions. Every mannequin presents completely different trade-offs between operational simplicity, tenant isolation degree, and value effectivity.

The silo mannequin dedicates full MSK clusters per tenant, providing a simple tenant isolation method however incurring the next upkeep and value per tenant. The pool mannequin affords elevated operational and cost-efficiencies by sharing all assets throughout tenants, however supplies restricted knowledge partitioning, efficiency equity, and tenant isolation capabilities. Lastly, the bridge mannequin affords a very good compromise between operational and cost-efficiencies whereas offering a very good vary of choices to create strong tenant isolation and efficiency equity methods.

When architecting your multi-tenant streaming resolution, rigorously consider your necessities round tenant isolation, knowledge privateness, per-tenant customization, and efficiency ensures to find out the suitable mannequin. Mix fashions if wanted to search out the best stability for your online business. As you scale your utility, reassess isolation wants and migrate throughout fashions accordingly.

As you’ve seen on this submit, there isn’t a one-size-fits-all sample for streaming knowledge in a multi-tenant structure. Rigorously weighing your streaming outcomes and buyer wants will assist decide the right trade-offs you may make whereas ensuring your buyer knowledge is safe and auditable. Proceed your studying journey on SkillBuilder with our SaaS curriculum, get hands-on with an AWS Serverless SaaS workshop or Amazon EKS SaaS workshop, or dive deep with Amazon MSK Labs.

Concerning the Authors

Emmanuele Levi is a Options Architect within the Enterprise Software program and SaaS crew, primarily based in London. Emanuele helps UK clients on their journey to refactor monolithic functions into trendy microservices SaaS architectures. Emanuele is principally all for event-driven patterns and designs, particularly when utilized to analytics and AI, the place he has experience within the fraud-detection business.

Lorenzo Nicora is a Senior Streaming Resolution Architect serving to clients throughout EMEA. He has been constructing cloud-native, data-intensive techniques for over 25 years, working throughout industries, in consultancies and product firms. He has leveraged open-source applied sciences extensively and contributed to a number of tasks, together with Apache Flink.

Nicholas Tunney is a Senior Associate Options Architect for Worldwide Public Sector at AWS. He works with World SI companions to develop architectures on AWS for shoppers within the authorities, nonprofit healthcare, utility, and training sectors.  He’s additionally a core member of the SaaS Technical Area Neighborhood the place he will get to satisfy shoppers from everywhere in the world who’re constructing SaaS on AWS.