Addressing Concurrency Issues in Inventory Management

In high-traffic e-commerce systems, a common concurrency challenge is the "overselling" phenomenon, where the quantity of sold goods exceeds the available stock. This discrepancy typically arises when inventory deduction logic is performed in application memory. Under high concurrency, multiple threads may read the same remaining stock value simultaneously before any of them writes the updated value back, leading to calculation errors.

To demonstrate, consider a scenario where inventory checks and updates are not atomic. Without proper synchronization, the system state becomes inconsistent.

Resolution Approaches

Two primary scenarios usually occur:

• Logical Overselling: The application logic allows sales even when inventory is insufficient.
• Negative Inventory: Due to race conditions between the read-check and update operations, the database record ends up with a negative value.

The immediate fix involves enforcing a strict check (stock > 0) within the transaction boundary or using database constraints to prevent negative values.

Single-Node Concurrency Control

Before scaling to distributed systems, standard JVM-level locking mechanisms are often employed.

Using ReentrantLock

For granular control over locking, ReentrantLock offers flexibility compared to intrinsic locks. It allows for interruptible lock acquisition, fairness policies, and attempts to lock with timeouts.

import java.util.concurrent.locks.ReentrantLock;

public class InventoryService {
    private int stockCount = 100;
    private final ReentrantLock lock = new ReentrantLock();

    public void reduceStock(int quantity) {
        lock.lock();
        try {
            if (stockCount >= quantity) {
                stockCount -= quantity;
                System.out.println("Stock deducted successfully. Remaining: " + stockCount);
            } else {
                System.out.println("Insufficient stock.");
            }
        } finally {
            lock.unlock();
        }
    }
}

In this example, the lock ensures that only one thread can modify the stockCount at a time. The finally block guarantees the lock is released even if an exception occurs, preventing deadlocks.

Limitations of Local Locks

While synchronized and ReentrantLock are effective within a single JVM process, they fail in distributed environments:

• Scope Restriction: A lock in one JVM instance cannot restrict access in another instance running on a different server.
• Scalability Bottlenecks: Relying on local locks prevents horizontal scaling of the service handling the critical section.

Distributed Locking Mechanisms

To manage concurrency across multiple nodes, a distributed lock is required. This is a lock that operates on an external system accessible by all nodes.

Redis-Based Distributed Locks

Redis is a popular choice due to its low latency. The implementation typically relies on the SET key value NX EX timeout command. The NX option ensures the key is set only if it does not exist (atomicity), and EX sets a time-to-live to prevent deadlocks in case a client crashes.

Pros:

• High performance and throughput.
• Simple to implement.

Cons:

• Relying on Time-to-Live (TTL) creates a window for inconsistency if a client holds the lock longer than the TTL.
• Single point of failure unless using Redis Sentinel or Cluster.

Database-Based Locks

Distributed locks can be implemented using relational databases via unique indexes or SELECT ... FOR UPDATE statements.

Pros:

• Leverages existing database infrastructure.
• Strong consistency guarantees (ACID).

Cons:

• Significant performance overhead due to disk I/O.
• Database connection pool exhaustion under high load.

ZooKeeper-Based Locks

ZooKeeper provides a robust coordination service. The locking mechanism utilizes ephemeral sequential znodes. When a client wants to acquire a lock, it creates an ephemeral znode. The client checks if it is the smallest node in the sequence. If not, it watches the previous node and waits for its deletion.

Pros:

• Strong consistency and reliability (CP system).
• Handles client failure gracefully (ephemeral nodes disappear upon session disconnect).

Cons:

• Higher latency compared to Redis.
• Increased operational complexity.

Comparison: Redis vs. ZooKeeper

Choosing between Redis and ZooKeeper involves a trade-off between performance and consistency. Redis is generally AP (Availability and Partition tolerance) in nature, offering better performance but potentially weaker consistency during network partitions. ZooKeeper is CP (Consistency and Partition tolerance), prioritizing data accuracy over availability during partitions, which is crucial for financial or strict inventory systems.

Redlock Algorithm

To address the reliability issues of single-instance Redis locks, the Redlock algorithm was proposed. It involves acquiring the lock on N (usually 5) independent Redis master nodes. The client considers the lock acquired if it successfully obtains the lock on a majority of nodes within a specific time window.

This method mitigates the risk of a single master failing or restarting, but it adds network latency and complexity.

Preventing Duplicate Scheduled Tasks

In a cluster where multiple instances run the same scheduled job, distributed locks ensure the job runs only once. The workflow is:

1. At the scheduled time, the instance attempts to acquire a distributed lock.
2. If acquisition succeeds, execute the logic.
3. Release the lock immediately after completion or upon failure.

It is critical to set a reasonable expiration time. If the task runs longer than expected, a robust implementation should use a "watchdog" thread to extend the lock's TTL automatically.

Distributed Transaction Strategies

When operations span multiple microservices or databases, maintaining data integrity requires distributed transaction patterns.

Two-Phase Commit (2PC)

A classic protocol involving a coordinator and participants. Phase 1 prepares all resources; Phase 2 commits or rolls back. While it ensures strong consistency, it is a blocking protocol and suffers from single points of failure (the coordinator).

Saga Pattern

Ideal for long-running transactions. A complex transaction is broken down into a sequence of local transactions. If a step fails, compensating transactions (rollbacks) are executed to undo the effects of previous steps. This pattern offers eventual consistency rather than immediate atomicity.

Transactional Messaging (Outbox Pattern)

This combines local transactions with event publishing. A service writes the data change and an event message to a local database (outbox) within one transaction. A separate relay process reads the outbox and publishes messages to a message broker (e.g., Kafka, RabbitMQ), ensuring that the event is eventually delivered and data is consistent across services.