MrMeeb/gitea-act-runner
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Container no longer starts in latest develop build #10

New Issue
Closed
opened 2024-04-18 13:21:10 +00:00 by MrMeeb · 4 comments
MrMeeb commented 2024-04-18 13:21:10 +00:00
Owner
Copy Link

Something is preventing docker from starting correctly - exit code is 1 after start, so something isn't right.

gitea-runner  | 2024-04-18 13:17:58,620 INFO supervisord started with pid 1
gitea-runner  | 2024-04-18 13:17:59,623 INFO spawned: 'processes' with pid 7
gitea-runner  | 2024-04-18 13:17:59,626 INFO spawned: 'act_runner' with pid 8
gitea-runner  | 2024-04-18 13:17:59,631 INFO spawned: 'cron' with pid 9
gitea-runner  | 2024-04-18 13:17:59,633 INFO spawned: 'dockerd' with pid 10
gitea-runner  | waiting for docker daemon...
gitea-runner  | 2024-04-18 13:18:00,627 INFO success: processes entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitea-runner  | 2024-04-18 13:18:00,627 INFO success: act_runner entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitea-runner  | 2024-04-18 13:18:00,627 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitea-runner  | 2024-04-18 13:18:00,643 INFO success: dockerd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitea-runner  | time="2024-04-18T13:18:04Z" level=info msg="Starting runner daemon"
gitea-runner  | Error: cannot ping the docker daemon, is it running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
gitea-runner  | 2024-04-18 13:18:16,020 WARN exited: dockerd (exit status 1; not expected)
gitea-runner  | 2024-04-18 13:18:16,020 INFO reaped unknown pid 75 (exit status 0)
gitea-runner  | 2024-04-18 13:18:16,023 INFO spawned: 'dockerd' with pid 98
gitea-runner  | 2024-04-18 13:18:16,024 INFO exited: processes (exit status 0; expected)
gitea-runner  | 2024-04-18 13:18:16,024 WARN exited: act_runner (exit status 1; not expected)
gitea-runner  | 2024-04-18 13:18:16,026 INFO spawned: 'act_runner' with pid 100
gitea-runner  | 2024-04-18 13:18:16,027 WARN received SIGQUIT indicating exit request
gitea-runner  | 2024-04-18 13:18:16,027 INFO waiting for act_runner, cron, dockerd to die
gitea-runner  | 2024-04-18 13:18:16,027 WARN stopped: dockerd (terminated by SIGTERM)
gitea-runner  | 2024-04-18 13:18:16,027 INFO reaped unknown pid 101 (exit status 0)
gitea-runner  | 2024-04-18 13:18:16,028 WARN stopped: cron (terminated by SIGTERM)
gitea-runner  | waiting for docker daemon...
gitea-runner  | 2024-04-18 13:18:19,037 INFO waiting for act_runner to die
gitea-runner  | waiting for docker daemon...
gitea-runner  | 2024-04-18 13:18:22,037 INFO waiting for act_runner to die
gitea-runner  | 2024-04-18 13:18:25,041 INFO waiting for act_runner to die
gitea-runner  | waiting for docker daemon...
gitea-runner  | 2024-04-18 13:18:26,038 WARN killing 'act_runner' (100) with SIGKILL
gitea-runner  | 2024-04-18 13:18:27,040 WARN stopped: act_runner (terminated by SIGKILL)```
Something is preventing docker from starting correctly - exit code is 1 after start, so something isn't right. ```gitea-runner | 2024-04-18 13:17:58,615 INFO Set uid to user 0 succeeded gitea-runner | 2024-04-18 13:17:58,620 INFO supervisord started with pid 1 gitea-runner | 2024-04-18 13:17:59,623 INFO spawned: 'processes' with pid 7 gitea-runner | 2024-04-18 13:17:59,626 INFO spawned: 'act_runner' with pid 8 gitea-runner | 2024-04-18 13:17:59,631 INFO spawned: 'cron' with pid 9 gitea-runner | 2024-04-18 13:17:59,633 INFO spawned: 'dockerd' with pid 10 gitea-runner | waiting for docker daemon... gitea-runner | 2024-04-18 13:18:00,627 INFO success: processes entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitea-runner | 2024-04-18 13:18:00,627 INFO success: act_runner entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitea-runner | 2024-04-18 13:18:00,627 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitea-runner | 2024-04-18 13:18:00,643 INFO success: dockerd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitea-runner | time="2024-04-18T13:18:04Z" level=info msg="Starting runner daemon" gitea-runner | Error: cannot ping the docker daemon, is it running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? gitea-runner | 2024-04-18 13:18:16,020 WARN exited: dockerd (exit status 1; not expected) gitea-runner | 2024-04-18 13:18:16,020 INFO reaped unknown pid 75 (exit status 0) gitea-runner | 2024-04-18 13:18:16,023 INFO spawned: 'dockerd' with pid 98 gitea-runner | 2024-04-18 13:18:16,024 INFO exited: processes (exit status 0; expected) gitea-runner | 2024-04-18 13:18:16,024 WARN exited: act_runner (exit status 1; not expected) gitea-runner | 2024-04-18 13:18:16,026 INFO spawned: 'act_runner' with pid 100 gitea-runner | 2024-04-18 13:18:16,027 WARN received SIGQUIT indicating exit request gitea-runner | 2024-04-18 13:18:16,027 INFO waiting for act_runner, cron, dockerd to die gitea-runner | 2024-04-18 13:18:16,027 WARN stopped: dockerd (terminated by SIGTERM) gitea-runner | 2024-04-18 13:18:16,027 INFO reaped unknown pid 101 (exit status 0) gitea-runner | 2024-04-18 13:18:16,028 WARN stopped: cron (terminated by SIGTERM) gitea-runner | waiting for docker daemon... gitea-runner | 2024-04-18 13:18:19,037 INFO waiting for act_runner to die gitea-runner | waiting for docker daemon... gitea-runner | 2024-04-18 13:18:22,037 INFO waiting for act_runner to die gitea-runner | 2024-04-18 13:18:25,041 INFO waiting for act_runner to die gitea-runner | waiting for docker daemon... gitea-runner | 2024-04-18 13:18:26,038 WARN killing 'act_runner' (100) with SIGKILL gitea-runner | 2024-04-18 13:18:27,040 WARN stopped: act_runner (terminated by SIGKILL)```
MrMeeb commented 2024-04-18 13:21:58 +00:00
Author
Owner
Copy Link

A healthcheck should be added, and a test to ensure the container starts correctly should be conducted before merging into develop

A healthcheck should be added, and a test to ensure the container starts correctly should be conducted before merging into develop
MrMeeb commented 2024-04-18 13:26:37 +00:00
Author
Owner
Copy Link

Containers should also be tagged with dev-{commit-sha} since rolling this back is otherwise painful - no way to build fixed images when the runners won't start!

Containers should also be tagged with dev-{commit-sha} since rolling this back is otherwise painful - no way to build fixed images when the runners won't start!
MrMeeb commented 2024-04-19 17:46:49 +00:00
Author
Owner
Copy Link

Docker is failing to start due to waiting for containerd to start timing out. #15 adding dependency management means that act_runner now waits until docker has actually been running for 20s (bit arbitrary but hey ho). But this doesn't fix the root issue, which I can't reproduce on a full VM. Only happening on an LXC container, and docker fails to start until the third try. Any more tries and supervisord would give up

Docker is failing to start due to waiting for containerd to start timing out. #15 adding dependency management means that `act_runner` now waits until docker has actually been running for 20s (bit arbitrary but hey ho). But this doesn't fix the root issue, which I can't reproduce on a full VM. Only happening on an LXC container, and docker fails to start until the third try. Any more tries and supervisord would give up
MrMeeb commented 2024-04-20 17:18:10 +00:00
Author
Owner
Copy Link

Determined this is due to oddities of running docker in LXC on a ZFS-based FS. Containerd tries to load the ZFS driver which takes 10 seconds, leaving a fine margin before dockerd gives up. Put the affected runners into a VM to do it 'properly'.

Determined this is due to oddities of running docker in LXC on a ZFS-based FS. Containerd tries to load the ZFS driver which takes 10 seconds, leaving a fine margin before dockerd gives up. Put the affected runners into a VM to do it 'properly'.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
renovate-bot MrMeeb
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: MrMeeb/gitea-act-runner#10
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?