Distributed lock on file within pekko cluster

[Susmit07]

Hello Folks,

We have hdfs connectors to watch on hdfs directories, the same code is deployed in pekko-clusters using k8s

How can I implement a distributed lock for the file across cluster to avoid race condition while processing

Thanks - Susmit

[pjfanning]

One option is to deploy a singleton actor instance that keeps track of which files are locked. You can send the actor instance it a message to request a lock and the response will say if you can have the lock or not. Then you would have another message interaction where you can release the lock.

• https://pekko.apache.org/docs/pekko/current/typed/cluster-singleton.html
• https://pekko.apache.org/docs/pekko/current/cluster-singleton.html

[Susmit07]

We are implementing using pekko-coordinators with k8s lease

https://pekko.apache.org/docs/pekko/current/coordination.html

do you see any drawbacks using it ?

thanks for your reply, much appreciated !

[Susmit07]

that keeps track of which files are locked

Persistance storage is a issue currently in our project, as in-memory map can be risky if pod gets restarted

def apply(): Behavior[Command] = Behaviors.setup { context => val lockedFiles = mutable.Map.empty[String, String] // Map filePath -> node holding the lock Behaviors.receiveMessage { case RequestLock(filePath, replyTo) => if (lockedFiles.contains(filePath)) { replyTo ! LockResponse(success = false) // Lock not available } else { lockedFiles.put(filePath, context.self.path.name) // Lock acquired replyTo ! LockResponse(success = true) } Behaviors.same case ReleaseLock(filePath, replyTo) => lockedFiles.remove(filePath) replyTo ! LockResponse(success = true) Behaviors.same } }

Your suggestion I liked tbh