A lease with a timeout is not a lock

Four fixes for one duplicate email. Each one closed a real gap. None of them was a lock, because none of them ran inside the write.

September 9, 2026 · Lucas Carlson

Disclosure: I maintain a library in this space and it shows up at the end. The bug and the four wrong fixes came first, and they are real. The guard I'm about to show you shipped in January 2024 and nobody noticed the race in it until July 2026, twenty-nine months later, by which point the same three lines had been copied to six other places in the codebase.

True story. It all started with a simple enough bug report. A customer complained that they received two onboarding emails.

So I did what anyone in my shoes would do. I added a simple duplication lock. It looked sorta like this:

Fix 1 — check, then claim

const lock = await redis.get(`onboarding:lock:${email}`)
if (lock) return

await redis.set(`onboarding:lock:${email}`, 1, { EX: 86400 })
await sendTheOnboardingEmail(email)

Which sorta worked for a while. Until it didn't. My boss complained again. This time another user complained about getting duplicate onboarding emails. Didn't I fix that already? I told him I had, so I thought about it hard.

Can you spot the bug?

There's a gap between getting and setting that lock. If two jobs race sending the same email, there's a moment in time where they will both pass the if (lock) guard.

So I fixed it again with NX, which gets and sets in one atomic command:

Fix 2 — claim atomically

const claimed = await redis.set(`onboarding:lock:${email}`, 1, { NX: true, EX: 86400 })
if (!claimed) return

await sendTheOnboardingEmail(email)

Perfect, no more gap! No more bug! Right? Wrong!

Now my boss started complaining that people weren't getting their onboarding email. First too many, now not enough. Can I just be done yet? Here's what was going on now.

1. An email worker won the claim and wrote the lock.
2. Of course I decided to deploy right then, and the pod got SIGKILLed mid-send.
3. The worker retried the job 30 seconds later.
4. The retry tried to grab the lock, but the lock is still held for another 23h 59m.
5. The retry returns false and exits cleanly.
6. Nobody sends the email. Ever.

This is where I have to pull up my sleeves and think hard. Ah HA. I can just set another flag:

Fix 3 — a second flag

const claimed = await redis.set(`onboarding:lock:${email}`, 1, { NX: true, EX: 86400 })
if (!claimed) return
if (await redis.get(`onboarding:sent:${email}`)) return

await sendTheDamnOnboardingEmail(email)
await redis.set(`onboarding:sent:${email}`, 1, { NX: true, EX: 86400 })

But this too has an annoying gap. This time the gap is between sendTheDamnOnboardingEmail and setting the flag. Damn it. How do I even close that gap? With Redis I can atomically get and set, but I can't atomically set a flag and ensure the email was actually sent at the same time.

AHHHHHHHH!!! I give up.

Oh wait. I have an idea. What if I flip the order and set the sent flag first?

Fix 4 — flip the order

await redis.set(`onboarding:sent:${email}`, 1, { NX: true, EX: 86400 })
await sendTheDamnOnboardingEmail(email)

Nope, that doesn't work either, because now there's a gap between setting the flag and sending the email. Flag after the send: the send succeeds, the flag fails, the customer gets two. Flag before the send: the flag succeeds, the send fails, the customer gets none.

Wait, what if I shorten the TTL that caused that lost email in the first place? What if I made it 30 seconds, and renewed it in a background thread while the work runs? That could work, right? I could even write a sweeper.

Sadly no. Now it just fails in a way that is even harder to see.

The heartbeat is a thread on the same machine, in the same process, under the same scheduler. A GC pause stops the worker and the heartbeat together. So does CPU starvation, a stop-the-world compaction, a blocked event loop, or a hypervisor that pauses the whole VM for a live migration.

The pause between setting a flag and doing the thing is the same pause that stops the heartbeat.

Worse, I'd traded a loud failure for a quiet one. At 30 seconds the claim expires while a live worker is mid-send far more often than it did at 24 hours. I made the lost email rare and the duplicate email common, and I wouldn't notice, because a duplicate doesn't raise.

You think this is a problem for onboarding emails? Imagine your entire business is built around reserving rooms. Five and a half years into Airbnb, their CEO went on Hacker News to answer a thread titled "What happens when a host cancels with Airbnb?" It hadn't been a cancellation. "The host did not cancel, we double-booked."

This is how I learned that a lease with a timeout is not a lock.

None of which is new. Martin Kleppmann wrote the definitive version in How to do distributed locking in 2016, and called the fix a fencing token: a number the resource checks. antirez answered that for an efficiency lock, where two workers only cost you duplicated work, a lease is completely adequate. He's right, and if that's your situation you can stop reading. Mine wasn't. Two workers cost me an email in a customer's inbox, which I can't recall.

What neither of them says, because it wasn't the subject, is where you put that check when the resource is your own database. That's the part I got wrong for twenty-nine months.

So how did I actually fix it?

Fix 5 — let the receiver check

const claimed = await redis.set(`onboarding:lock:${email}`, 1, { NX: true, EX: 86400 })
if (!claimed) return

await sendTheDamnOnboardingEmailAlready(email, { idempotencyKey: `onboarding:${email}` })

Notice what changed, and what didn't.

The Redis lease is still there. It still expires. It's still not a lock. What changed is that I stopped asking it to be one. Its job now is to skip work, and most of the time only one worker calls the mailer. That's worth having, and it's all a lease was ever good at. When it fails, and it will, the second call carries the same key and the provider refuses it.

I didn't prevent the race. I made it harmless. And I didn't do it. The provider did.

That is what a lock is. Not a key with a timer on it. A check that happens inside the write, performed by whatever accepts the write.

Once you see it that way, the question stops being "how do I hold this claim long enough" and becomes "who is going to check?"

For an email, the mail provider checks. For a hotel room, an inventory count, a game state, or a balance, nobody sells you that. The thing that accepts those writes is your own SQL database, and it is perfectly capable of refusing a stale one. Nobody ever asks it to.

Why not just a unique index or an advisory lock?

For this email, you should. CREATE UNIQUE INDEX ON sent_emails (user_id, kind), insert before the send, let the second insert fail. No library, no Redis, and the check happens inside the write because it is the write.

A transaction-scoped advisory lock does the same job from the other side: pg_advisory_xact_lock releases at commit, on the database's clock and not the worker's, which would have killed fixes 1 through 4 outright.

If your whole invariant fits in one request, that is the answer and you can stop reading here.

Mine didn't. Requirements grew. My boss wanted to send a follow-up in three days, unless of course they logged in first. Neither one can express "in three days," and an advisory lock is gone the moment the transaction commits, so it turns into this:

The part neither one can do

await db.insert("scheduled_emails", { userId, kind: "followup", dueAt: inThreeDays })

// ...plus a cron, every five minutes, forever:
const due = await db.query("SELECT * FROM scheduled_emails WHERE due_at < now() LIMIT 100")

for (const row of due) {
  if (await hasLoggedIn(row.userId)) { await db.remove(row); continue }
  await sendFollowUp(row.userId)
  await db.remove(row)   // crash here and the whole batch sends again tomorrow
}

A table, a cron, a batch query, and a delete that happens after the send. That last line is Fix 1 again, one layer up: a check, a gap, and a write that nothing re-checks. Logging in has to race the sweeper to delete the row in time.

That's where the index and the advisory lock end and the sweeper begins, and the sweeper is what I was trying to delete.

So I built Solid Objects

Kenton Varda had already solved this for me, and solved it beautifully. His team's Cloudflare Durable Objects gives every identity its own single-threaded execution and its own transactional storage, so the check and the write are the same operation and none of my four fixes is even expressible. It's the right model, but I couldn't use it.

Virtual actors aren't new either. Orleans shipped the idea in 2011 and Erlang was doing it decades before that. What Durable Objects added was making it the default unit of a web application rather than a framework you adopt.

My apps are built in Node and Rails, talking to MySQL and Postgres, on servers I own, and the duplicate email came from a worker job running in my VM. There was no version of "adopt Durable Objects" that didn't mean moving the state out of my database and the code out of my runtime, for one email bug.

So I took the model and left the platform. Solid Objects is an open source library for Node and Rails that puts identity, ordered calls, durable state, and alarms in the SQL server you already run.[1] No Redis, no daemon, no broker. Here is the same onboarding email, with the lock, the sent flag, the TTL, the heartbeat, and the sweeper all deleted.

onboarding.ts

import { Actor } from "solid-objects"

class Onboarding extends Actor {
  static actorType = "Onboarding"
  status = "unsent"

  // One transaction. Both lines commit, or neither does.
  sendEmail() {
    if (this.status !== "unsent") return
    this.status = "sent"
    this.emit("deliverEmail", { arguments: { email: this.actorId } })
  }
}

runtime.registerEffect("deliverEmail", (args, context) =>
  // Solid Objects gives every emit a unique context.id. Better than keying off args.email.
  sendTheDamnOnboardingEmailAlready(args.email, { idempotencyKey: context.id })
)

await Onboarding.ref(email).sendEmail()

this.status = "sent" and the staged send are two rows in one SQL transaction. Both commit, or neither does. That's the gap I spent all afternoon failing to close, and it isn't closed so much as it never exists. There is no "between" to be in.

And the follow-up, the part a unique index couldn't express, is one new line and two short methods:

The table, the cron, and the race, deleted

sendEmail() {
  if (this.status !== "unsent") return
  this.status = "sent"
  this.emit("deliverEmail", { arguments: { email: this.actorId } })
  this.schedule({ at: inThreeDays }).followUp()   // commits with the two lines above
}

followUp() {
  if (this.status !== "sent") return              // logging in already moved it
  this.status = "followed-up"
  this.emit("deliverFollowUp", { arguments: { email: this.actorId } })
}

loggedIn() {
  this.status = "done"
}

The reminder is a row that commits in the same transaction as status = "sent", so it cannot be scheduled for an email that wasn't sent, and it survives a deploy. Logging in cancels the follow-up by moving the state, not by finding and deleting a row before a cron gets to it. Nobody races anybody. There is no table and no cron.

Yes, it uses a lease with a timeout

Of course it does. You can't recover a dead worker's actor without one. Claiming an actor writes an owner, an activation token, an expiration, and a generation number.

The difference is what happens at the end. Every commit re-checks all four, and checks the expiration against database time, not the worker's clock. A worker that stalled past its lease runs to completion and its commit is refused. That's the entire distinction, and it is the answer to the title. A lease becomes a lock at the moment something enforces it inside the write.

Walk my own bug through it

• Killed before the commit. Nothing happened. The status is still unsent, no send is staged, and the retry sends.
• Killed after the commit. The send is a durable row in your database. Another worker drains it. Nobody had to remember; the database remembered.
• Killed after the send. The effect repeats with the same context.id, and the provider refuses the second copy.
• Two jobs race. Both enter the mailbox for that identity. The runtime runs them one at a time, and the second reads sent and returns.
• A worker stalls past its lease and wakes up mid-turn. Its commit is fenced and refused. It runs to completion and writes nothing.

Gone from my code: the lock key, the sent key, the TTL, the heartbeat, the sweeper.

What it does not do

State commits exactly once. External sends are at least once, with a stable durable key so the receiver can make them exactly once. An actor can't un-send an email any more than Redis could. What it can guarantee is that you only ever decided to send one.

Two concurrent calls to reserve can't both see the last seat. That part is yours for free, because it never leaves the database.

It's pre-1.0 and it runs on SQLite, PostgreSQL, and MySQL. Use it when the work has to happen later, survive a restart, or stay ordered across several requests.

• Solid Objects in five minutes
• solid-objects-js on GitHub (Node)
• solid-objects-ruby on GitHub (Rails)
• Why I built Solid Objects