Accessories Zero Downtime

[Pandoks]

I was wondering if there's going to be a way to have zero downtime deployments for accessories?

But also the more I think about it the more it doesn't make sense/work. I think it would be nice to include a tutorial/blog on how to deploy it without downtime.

Currently this is how I'm handling it (eg. change of database):

1. Spawn up new database box/container
2. Change app code to connect to the new database
3. Deploy
4. Delete old database box/container

[edzhelyov]

How do you handle the data consistency between the 2 database containers?

[Pandoks]

I manually have to migrate the data which is why I said that the more I think about it the more zero downtime deployments for accessories doesn't make too much sense as you need a lot of manual work. But that's how I do it now. Don't know if there's a better way.

[edzhelyov]

Could you be more specific in the details of your approach? I want to check if you don't introduce any data inconsistencies that way.

I'm thinking about how you can achieve zero downtimes on a single PostgreSQL instance and my thoughts at the moment are:

You don't want application errors and you don't want to lose writes. The way I can see it at the moment is that you need a second replica that get synced with the primary database, the replica is the new version.

When the replica has catched up with the primary you can slowly reconnect applications to it, but that means you will have applications that write to both the primary and the replica. When you have all of your apps connected to the replica, your replica should be able to sync all the writes that happened on the primary. If that's not supported you can't go with that solution.

Another way is to postpone writes to the primary to ensure the replica is fully synced and then allow the writes to happen on the replica, you can eventually do that if you have a proxy infront of the database that can keep the writes until the replica is available. The issue here is that you need to wait until all writes finish on the primary which will introduce a latency, and eventual timeouts while waiting.

If your application can reconnect, replay the queries when the connection is dropped, then you can achieve the same with a pure inplace update. Drop or stop accepting new queries on the primary, stop it, and then spin up the replica, but that is basically down time for the database, but your clients won't see it as you will keep the issues in the application.

[edzhelyov]

Here is a quite a detailed article of how Instacart did PostgreSQL version update and you can see it's a ton of complicated steps. I have the feeling it's much more reasonable to just stop the accessory, spin up a new one with the same volumes and call it a day.

https://www.instacart.com/company/how-its-made/zero-downtime-postgresql-cutovers/

[dhh]

No plans to try to automate this at this time. I think your method is correct.