Distributed Locks and Spring Transaction Ordering: A Subtle Concurrency Bug

This came from a question someone posted online. I walked through the reasoning with him and solved his problem, so I'm writing it down here.

Background

"I added a distributed lock, but the data is still wrong" is a classic class of concurrency problem. The lock itself has no bug, and the transaction commits and rolls back just fine — but when you combine the two, getting the order of lock release and transaction commit slightly wrong renders the serialization guarantee useless. These problems rarely surface before load testing, and during troubleshooting people tend to fixate on the lock implementation while overlooking the transaction boundary. That was exactly what this reader ran into.

He was using a ZooKeeper distributed lock (ephemeral sequential nodes) to guard updates to values in MySQL. The feature was essentially a ticket-selling mechanism: under high concurrency, the ticket count must stay exactly right — never one more, never one less.

With the ZooKeeper lock wrapped around the service layer, the lock was released once the service method finished executing — but at that point the declarative Spring-managed transaction annotated on the method had not yet committed. Under concurrency, this ordering leads to data corruption.

Naturally, his database data went wrong. With 100 threads each taking exactly one ticket from a pool of 100, the remaining count should be 0. His result was not 0 — it never hit the expected value.

Root Cause

Here is what happens: after the first thread finishes the update method, the transaction has not yet committed, but the distributed lock has already been released. When the next thread enters the method and runs its select, it still reads the stale pre-update data — a dirty read from its perspective. That is because Spring's declarative transaction, @Transactional(rollbackFor = Exception.class), is AOP-based: the commit happens in around-advice after the method returns. By the time that advice commits the transaction, the ZooKeeper lock is already gone. This ordering gap lets the next thread read stale data, and the computation goes wrong.

Let's spell out the execution order. Declarative transactions work by having Spring generate a proxy for the target class; the transaction begins and commits at the proxy layer, wrapped around the business method. His lock/unlock code, however, lived inside the business method. So the actual timeline of one call looks like this:

// Call sequence through the proxy (illustrative)
// 1. Transaction interceptor begins the transaction
// 2. Enter the business method, acquire the ZooKeeper lock
// 3. select remaining tickets, update to deduct
// 4. Release the lock inside the business method   <-- lock is gone here
// 5. Business method returns, interceptor commits  <-- but the commit happens here

Between steps 4 and 5 there is a window: the lock is released, the transaction is not yet committed. The next thread waiting on the lock acquires it inside this window and runs its select. Since the previous transaction hasn't committed, under the default read-committed or repeatable-read isolation level it sees the old pre-deduction value — effectively two threads each deducting once from the same remaining count. The lock's mutual exclusion was never broken; what broke was the implicit assumption that "a read inside the lock always sees the previous lock holder's write."

The Fix

So we need to change the ordering between the Spring transaction and the release of the distributed lock. Commit the transaction first, then release the distributed lock — or simply move the lock up to the controller layer — and the problem goes away.

Both approaches are the same at heart: make the lock's hold span fully cover the transaction's lifetime.

1. Acquire and release the lock outside the service layer (in the controller, or in an outer method with no transaction), with the transactional method as the inner call wrapped by the lock. When that inner method returns, the transaction has committed — only then is the lock released.

2. Drop the declarative transaction and use a programmatic one (TransactionTemplate or a manual commit), explicitly committing first and unlocking second in code, so the ordering is entirely in your hands.

One caveat: if you move the locking logic into another method of the same class and reach the transactional method via a this self-invocation, the transaction annotation will not take effect — self-invocation bypasses the proxy. Either split the two methods into separate classes, or fetch the class's own proxy from the container and call through it.

A Distributed Lock Is Not Data Integrity

One more thing worth adding: a distributed lock cannot guarantee data integrity. All it guarantees is exclusive invocation of that particular service-layer method.

If some other service method has a mapper touching the same row, the data will still go wrong.

So you should add an optimistic-lock column to the table to protect data integrity. Optimistic locking preserves integrity well, and its concurrency performance beats the service-level locking he was using.

The optimistic-locking approach is to add a version column and make the update conditional on the version you read:

-- Read the version column together with the row
-- Update conditioned on the old version, incrementing it at the same time
update ticket
set stock = stock - 1, version = version + 1
where id = #{id} and version = #{version};
-- Zero affected rows means someone else changed the data first: retry or fail

This line of defense sits on the data itself: no matter how many entry points modify the row, final consistency is enforced by the database. A distributed lock solves "only one worker at a time"; optimistic locking solves "even if several workers act, the data stays correct." They operate at different layers and do not substitute for each other.

Pitfalls and Notes

1. Before adding a distributed lock, be clear about which boundary wraps which: the lock must fully cover the transaction. "Lock inside the method + @Transactional on the method" is the easiest combination to write and the easiest to get wrong.

2. Unit tests and low-concurrency scenarios almost never catch this class of bug; it only reproduces when concurrency pressure hits the window between lock release and transaction commit. Concurrency-consistency verification of deduction-style endpoints before going live is a must.

3. When investigating inconsistent data, don't stare only at whether the lock implementation is correct. Draw the transaction's begin and commit points into the timeline as well — the problem often lives right at the seam between the two.

Wrapping Up

In this case the lock and the transaction each worked correctly on their own; what went wrong was their ordering — the lock was released before the commit, opening a window for the next thread to read stale data. The fix is to make the lock fully enclose the transaction, or to switch to a programmatic transaction and control the commit timing explicitly. Going further, the correctness of deduction-style data should not rest solely on mutual exclusion at the entry point: adding an optimistic-lock column to the table and placing the last line of defense at the database layer is the more robust approach.