ERROR (Mrsk::Utils::HealthcheckPoller::HealthcheckError): container not ready (unhealthy)

[jasonfb]

when running mrsk deploy, the precompile works and everything looks like it is deploying out correctly, but at the end I get this

  ERROR (Mrsk::Utils::HealthcheckPoller::HealthcheckError): container not ready (unhealthy)

this app I'm on is Rails 7.0.4.3 so I do not have an /up endpoint. do I patch one onto my Rails 7.0.4.3 app?

[jasonfb]

I thought I could do this:

  get "/up", to: Proc.new { [200, {}, ["OK"]] }

but I'm still getting the error even after adding that to my routes file.

running on mrsk 0.12.0

[jasonfb]

my Dockerfile does contain an apt-get for curl (cross-referencing @PabloC's answer on #276)

# Install packages need to build gems and node modules
RUN apt-get update -qq && \
    apt-get install -y build-essential curl default-libmysqlclient-dev git libpq-dev libvips node-gyp pkg-config python-is-python3

my full Dockerfile:
https://gist.github.com/jasonfb/592bf8484810aa09934727caa83b65e1

[jasonfb]

this is the result of mrsk deploy
https://gist.github.com/jasonfb/6b86fe48f79393d98626ab55b3ab9334

[Jberczel]

Were you able to deploy prior to upgrading to 0.12.0?

I added some healthcheck logging which was merged into 0.12.1 (#265). That may give you more detail on health of container and why it's not passing healthcheck.

[jasonfb]

mrsk 0.12.1
same result, but with newly added logging content as @Jberczel promised above

Ensure app can pass healthcheck...
  INFO [4e1d1444] Running docker run --detach --name healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a --publish 3999:3000 --label service=healthcheck-helios-panoptes -e MRSK_CONTAINER_NAME="healthcheck-helios-panoptes" -e [REDACTED] --health-cmd "curl -f http://localhost:3000/up || exit 1" --health-interval "1s" jasonheliosdevshop/helios-panoptes:d0e63ff92735b901170f79037c36f251c31dbb6a on 52.91.164.217
  INFO [4e1d1444] Finished in 1.587 seconds with exit status 0 (successful).
  INFO [25dd3012] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 52.91.164.217
  INFO [25dd3012] Finished in 0.464 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 1s (attempt 1/7)...
  INFO [6c639d86] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 52.91.164.217
  INFO [6c639d86] Finished in 0.098 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 2s (attempt 2/7)...
  INFO [1fab94c8] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 52.91.164.217
  INFO [1fab94c8] Finished in 0.453 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 3s (attempt 3/7)...
  INFO [0a8f9b5f] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 52.91.164.217
  INFO [0a8f9b5f] Finished in 0.075 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 4s (attempt 4/7)...
  INFO [c8407ddc] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 52.91.164.217
  INFO [c8407ddc] Finished in 0.154 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 5s (attempt 5/7)...
  INFO [c633a966] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 52.91.164.217
  INFO [c633a966] Finished in 0.222 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 6s (attempt 6/7)...
  INFO [6d60dfda] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 52.91.164.217
  INFO [6d60dfda] Finished in 0.210 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 7s (attempt 7/7)...
  INFO [8039a8dd] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 52.91.164.217
  INFO [8039a8dd] Finished in 0.093 seconds with exit status 0 (successful).
  INFO [e0744065] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker logs --tail 50 2>&1 on 52.91.164.217
  INFO [e0744065] Finished in 0.345 seconds with exit status 0 (successful).
 ERROR exec /rails/bin/docker-entrypoint: exec format error
 ERROR {
  "Status": "unhealthy",
  "FailingStreak": 0,
  "Log": [

  ]
}
  INFO [21805349] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker stop on 52.91.164.217
  INFO [21805349] Finished in 0.257 seconds with exit status 0 (successful).
  INFO [3719e3a6] Running docker container ls --all --filter name=^healthcheck-helios-panoptes-d0e63ff92735b901170f79037c36f251c31dbb6a$ --quiet | xargs docker container rm on 52.91.164.217
  INFO [3719e3a6] Finished in 0.154 seconds with exit status 0 (successful).
  Finished all in 722.5 seconds
Releasing the deploy lock
  INFO [18555426] Running /usr/bin/env rm mrsk_lock/details && rm -r mrsk_lock on 52.91.164.217
  INFO [18555426] Finished in 0.088 seconds with exit status 0 (successful).
  ERROR (Mrsk::Utils::HealthcheckPoller::HealthcheckError): container not ready (unhealthy)

[34code]

I don't think adding curl to the dockerfile to build the app is working as the healthcheck container is a different container than the one used to launch the app?

[jasonfb]

mrsk 0.11.0
same result:

  Deploy lock was not released
  ERROR (SSHKit::Command::Failed): Health check against /up failed to return 200 OK!

[jasonfb]

@34code - interesting, I'm no Docker expert, mostly just been a Docker user, and this experience is the most I've gotten my hands dirty with Docker. What you say makes sense. I was just following suggestion by from @PabloC's answer on #276 but I didn't modify my Dockerfile (which originally came from https://www.driftingruby.com/episodes/deploying-with-mrsk and is basically that but with some small modifications)

if I get the same error on mrsk 0.11.0, 0.12.0, ad 0.12.1, then it seems to rule-out those recent changes anyway

I wonder, what should I try next?

note that I am trying on Amazon EC2 Ubuntu, which doesn't give root access to the ubuntu user, so for other steps (not this one), I had to ssh manually and run some of the apt-get commands using sudo. (reference #178)

[34code]

mrsk 0.11.0 same result:

  Deploy lock was not released
  ERROR (SSHKit::Command::Failed): Health check against /up failed to return 200 OK!

Try after releasing lock as follows: mrsk lock release

from docs (https://mrsk.dev/docs/locking)

[jasonfb]

oh shoot. but I think if the lock is actually in place, it stops way sooner (like right up front)

when I do this it says there was no lock

% mrsk lock release
  INFO [15ca4d80] Running /usr/bin/env rm mrsk_lock/details && rm -r mrsk_lock on 52.91.164.217
There is no deploy lock

anyway I get the same result if I first release the lock, but thanks indeed.

"Do not try to bend the lock, just try to remember... there is no lock"
— the Matrix

[Jberczel]

This seems like an issue with your docker setup, not mrsk.

Your log says there's an issue with your docker-entrypoint.

 ERROR exec /rails/bin/docker-entrypoint: exec format error

Have you googled that error? You could also try and remove entrypoint from Dockerfile and rebuild and try to run locally to troubleshoot.

[PabloC]

It seems something related to docker and the architecture. Are you using the same architecture local vs remote? I mean, x86 local and remote?

[jasonfb]

Your log says there's an issue with your docker-entrypoint.

Confirmed, it was the docker-entrypoint file. When I comment out that file in Dockerfile the build succeeds correctly.
ty!

It seems something related to docker and the architecture. Are you using the same architecture local vs remote? I mean, x86 local and remote?

No. my local machine is M1 and the instance is Amazon EC2 Ubuntu

@dhh -- I didn't realize this repo had "Discussions," I will use those from now on for pre-bug-report investigations. thank you for your patience.

[PabloC]

M1 is arm architecture, and EC2 what architecture? probable is x86 or amd64.
You need to build the docker image with the destination's arch, that's the issue
See the documentation about how to do that, you need to add configuration to do this on deploy.yml