Handling Database Transactions in Microservices Using the SAGA Pattern

Microservices architectures introduce challenges in handling distributed transactions due to the decentralized nature of services. Traditional ACID transactions, commonly used in monolithic applications, are not directly applicable across multiple independent microservices with separate databases. The SAGA pattern provides a mechanism for ensuring data consistency across services by breaking transactions into a series of coordinated steps.

SAGA Pattern Overview

The SAGA pattern consists of a sequence of local transactions, where each step either commits changes to its respective service database or invokes a compensating action upon failure. There are two main approaches to implementing SAGA: choreography-based and orchestration-based.

Choreography-Based SAGA

In a choreography-based SAGA, each service listens for events and executes its transaction accordingly. If a step fails, compensating transactions are triggered to revert previous operations.

Implementation Steps

1. Event Emission: The initiating service publishes an event indicating the start of a transaction.

2. Event Subscription: The next service in the process listens for the event and executes its local transaction.

3. Failure Handling: If any service fails, a compensating event is triggered to undo prior transactions.

Example with Order Processing

Consider a system with three services: Order Service, Payment Service, and Inventory Service.

Order Creation

• The Order Service creates an order and emits an OrderCreated event.

• The Payment Service listens for this event, processes payment, and emits a PaymentProcessed event.

• The Inventory Service listens for PaymentProcessed, updates stock, and finalizes the process.

Handling Failures

If payment processing fails:

• The Payment Service emits a PaymentFailed event.

• The Order Service listens for PaymentFailed and cancels the order.

Implementation Using Kafka

Each service subscribes to relevant Kafka topics.

Order Service publishes an event:

{
  "eventType": "OrderCreated",
  "orderId": "12345",
  "customerId": "67890"
}

Payment Service subscribes and processes payment:

from kafka import KafkaConsumer

def process_payment():
    consumer = KafkaConsumer('order-events', bootstrap_servers='localhost:9092')
    for message in consumer:
        event = json.loads(message.value)
        if event['eventType'] == 'OrderCreated':
            handle_payment(event['orderId'])

Orchestration-Based SAGA

In an orchestration-based SAGA, a central orchestrator manages the execution and rollback of transactions across services.

Implementation Steps

1. Transaction Orchestrator: A dedicated service coordinates execution.

2. Step Execution: The orchestrator invokes services in sequence.

3. Compensating Transactions: If a step fails, the orchestrator invokes rollback actions.

Example with Order Processing

A SAGA Orchestrator service manages the order lifecycle.

Order Creation Flow

• The orchestrator calls Order Service to create an order.

• The orchestrator then calls Payment Service to process payment.

• After payment success, the orchestrator calls Inventory Service to update stock.

Handling Failures

If the Payment Service fails:

• The orchestrator calls Order Service to cancel the order.

Implementation Using Temporal

Using Temporal as an orchestrator:

Define the workflow:

from temporal.workflow import workflow_method

class OrderWorkflow:
    @workflow_method
    def process_order(self, order_id: str):
        order_id = self.create_order()
        try:
            self.process_payment(order_id)
            self.update_inventory(order_id)
        except Exception:
            self.cancel_order(order_id)

Compensating Transactions

Each step in a SAGA requires a compensating transaction to revert changes in case of failure.

Example Compensations

• Order Created → Order Canceled

• Payment Processed → Payment Refunded

• Stock Deducted → Stock Restored

Ensuring Idempotency

Microservices handling SAGA steps must ensure that retries do not lead to duplicate transactions. This is achieved through idempotency tokens.

Example of an idempotent payment request:

{
  "transactionId": "tx-12345",
  "orderId": "12345",
  "amount": 100.00
}

A service ensures an operation is not executed multiple times by checking if transactionId has already been processed.

Handling Distributed State

Since microservices operate independently, maintaining state consistency requires:

• Event Sourcing: Persisting state changes as events in an event store.

• Saga Log: Keeping a log of executed steps and their statuses.

Comparison of Choreography vs. Orchestration

Conclusion

Handling database transactions in microservices requires distributed transaction management. The SAGA pattern provides an approach by executing transactions in a sequence of steps with compensating actions for failures. Choreography-based SAGA distributes responsibilities across services, while orchestration-based SAGA centralizes transaction management. Choosing the right approach depends on system complexity, service autonomy, and fault tolerance requirements.

For more technical blogs and in-depth information related to Platform Engineering, please check out the resources available at “https://www.improwised.com/blog/".