Having Errors with no logs

[instance0f]

please see screenshot

[instance0f]

These is my workflow:

jobs:
build:
name: Replicate to Deployment Manifest
runs-on: [ self-hosted ]
steps:
# Checkout Source Project
- name: Check-out source project
uses: actions/checkout@v2
with:
token: ${{ secrets.DEPLOYMENT_ACCESS_TOKEN }}
# Push to the destination repo
- name: creates output
run: sh ./build.sh
- name: Push to another repository
uses: cpina/github-action-push-to-another-repository@main
env:
SSH_DEPLOY_KEY: ${{ secrets.SSH_DEPLOY_KEY }}
with:
source-directory: .
github-server: 'github.gisd88.com'
destination-repository-name: 'deployment-manifest'
user-name: Kevin-Hukom
user-email: kevin@gisdd88.com
target-branch: mlab-cd-test
commit-message: Synced from deployment-manifest-secrets repo

[cpina]

Hi @instance0f ,

I would expect after the line in your screenshot:
[+] Using SSH_DEPLOY_KEY some more lines such as:

[+] Git version
[+] Cloning destination git repository ...

And quite a few more.

Is there anything else?

I'm specially interested in what's after:
[+] Pushing git commit

(I hope to see some kind of error that helps debugging it)

[cpina]

Also, I see that you use a hosted server. I think that the action will try to reach it on the port 22 from a GitHub server, is it reachable? If you had access to the ssh logs to see if a connection is attempted (with success or error) it might also help.

Thanks for (trying to) use the action :-)

[instance0f]

Hi cpina,

Hi @instance0f ,

I would expect after the line in your screenshot: [+] Using SSH_DEPLOY_KEY some more lines such as:

Hi Cpina,
there are no other logs after the line [+] Using SSH_DEPLOY_KEY.

Ive tried using this in my personal github and it worked. But in my enterprise it doesnt.

[instance0f]

Also, I see that you use a hosted server. I think that the action will try to reach it on the port 22 from a GitHub server, is it reachable? If you had access to the ssh logs to see if a connection is attempted (with success or error) it might also help.

Thanks for (trying to) use the action :-)

Hi @cpina ,

Thanks for your prompt reply. I am using a self-hosted linux runner without admin permissions. Should there be an issue?

[cpina]

Thanks for your prompt reply. I am using a self-hosted linux runner without admin permissions. Should there be an issue?

Shouldn't be the problem. I will prepare a branch to try to gather more information. I will comment here with some steps :-)

[cpina]

Hi @instance0f ,

Instead of:

uses: cpina/github-action-push-to-another-repository@main

Could you use:

uses: cpina/github-action-push-to-another-repository@debug-78

It might be fixed in this version: please let me know and I'll move the fix to master.
If it's not fixed: probably there will be more messages, at least:

[+] Gathering SSH public keys from server

And maybe:

[+] SSH public keys gathered from server

Let me know of the messages please to continue debugging.

Thanks for your report!

[instance0f]

Hi @cpina ,

Please see image. Additional line of

[+] Gathering SSH public keys from server

[instance0f]

Also, I see that you use a hosted server. I think that the action will try to reach it on the port 22 from a GitHub server, is it reachable? If you had access to the ssh logs to see if a connection is attempted (with success or error) it might also help.

How can i check port 22? and ssh logs?

Thanks

[cpina]

Please see image. Additional line of

Thanks, I see the additional line but I still don't know why the ssh-keyscan would fail silently.

Could you do another build? (same debug-78 branch, just make sure that it triggers the build again with the action).

You should see:
[+] SSH public keys gathered from server (with extra information)

And hopefully some more information (a path, the output of an ls -l) and the commands being executed.

Let's see if this helps.

About:

How can i check port 22? and ssh logs?

Thanks for executing the branch debug-78 I don't think that this is the problem (or not yet, so let's try one more run to see if it works)

[instance0f]

Hi @cpina

Please see screenshot above. (used debug-78 branch)

• i also used a service admin account for creating SSH deploy keys
• added permissions in the destination repository

[cpina]

Hi @instance0f ,

Could you try again? I had the lines in the wrong order so it failed before it could gather the information that I think that might be useful.

Actually I have an idea but I'd like to confirm it in the output. We might need two more tests but if you could try again and show the output will be useful, I hope!

Thanks!

[instance0f]

Hi @cpina ,
can we organize a working session? in skype maybe? :)

[cpina]

I've added some more debug information - could you try a new execution please?

No availability for now for a Skype and anyway I need to think between attempts :-) thanks for your patience!

[instance0f]

Hi @cpina
it displayed some errors

this should be regarding the following parameters:

• destination-github-username: 'Business-Technology'
• destination-repository-name: 'deployment-manifest'
• target-branch: mlab-cd-test
• user-email: btservicea@gisd88.com

[cpina]

I'm more interested in the output of the area near:
SSH public keys gathered from server

It failed "later" because the action had set -e which cancels the execution if a command fails. For you it's failing on the ssh-keyscan. I need to understand why it fails in the ssh-keyscan first. I thought that the set -e might be hiding an error message of ssh-keyscan.

So, a screenshot near SSH public keys gathered... is what we need for now.

Thanks!

[instance0f]

[cpina]

Added another bits of debugging - mind running again please and screenshot around Gathering SSH public keys please? Thanks!

[instance0f]

[cpina]

Hi @instance0f ,

I read that ssh-keyscan returns "nothing" if it times out (or no connection can be established):
https://unix.stackexchange.com/questions/346376/ssh-keyscan-has-no-output

I had expected ssh-keyscan to print an error message if a network problem happened but it fails silently. Thanks to your problem and patience I know this :-) (I will probably add a message in that place, document it for sure and you made me realise of a bug that I will fix for hosted GitHub instances).

When I disabled the "early exit if an error" we can also see ssh: connect to host github.com port 22: Operation timed out (thanks for pasting all the error messages!).

Your worker, where the action is executed, cannot connect to github.com.

Is there anyone in the company that you could work with? The problem, for sure, is that the "instance" (the network, or the container, etc.) where the action is executed cannot connect to github.com port 22. This needs to be enabled if you want to push into it.

Also, can you confirm that the destination repository is in "github.com", right? Not in some other hosted enterprise GitHub installation? To access the files in the "destination repository": are you using something like https://github.com/YourOrganisation/DestinationRepo ? If instead of github.com you are using something else then you would need to set the GITHUB_SERVER in the action (it's one of the parameters).

[instance0f]

Also, can you confirm that the destination repository is in "github.com", right?
Hi @cpina ,

No, the destination repo is in the same server: "github.gisd88.com"
can you please add a parameter something like organization?

my whole source repo url is: https://github.gisd88.com/Business-Technology/deployment-manifest-secret
my whole target repo url is: https://github.gisd88.com/Business-Technology/deployment-manifest

server: github.gisd88.com
organization: Business-Technology

Thanks!

[cpina]

In this case you need, in your action, to set the github-server variable to github.gisd88.com.

But I see in a message above that you had done this.

Do you know if this is still the case? Because in your screenshot of the configuration I see that you had github-server but in the output of the action it seems that you didn't (I wonder if you had two source repos that you were testing, a typo in one or something)

[cpina]

(note that today I will not be around the computer, if I don't answer I will do in the evening or next days)

[instance0f]

In this case you need, in your action, to set the github-server variable to github.gisd88.com.

But I see in a message above that you had done this.

Do you know if this is still the case? Because in your screenshot of the configuration I see that you had github-server but in the output of the action it seems that you didn't (I wonder if you had two source repos that you were testing, a typo in one or something)

Hi @cpina,

Thanks for the immediate response. These are the parameters i am using:

      source-directory: output/
      github-server: 'github.gisd88.com'
      destination-repository-name: 'Business-Technology/deployment-manifest'
      target-branch: mlab-cd-test
      user-email: btservicea@gisd88.com

Thanks,
Kevin

[instance0f]

Hi @instance0f ,

Your worker, where the action is executed, cannot connect to github.com.

Is there anyone in the company that you could work with? The problem, for sure, is that the "instance" (the network, or the container, etc.) where the action is executed cannot connect to github.com port 22. This needs to be enabled if you want to push into it.

Hi @cpina,

Can we cross this issue out?
my source and destination are on the same server: github.gisd88.com

[cpina]

Hi @instance0f ,
Your worker, where the action is executed, cannot connect to github.com.
Is there anyone in the company that you could work with? The problem, for sure, is that the "instance" (the network, or the container, etc.) where the action is executed cannot connect to github.com port 22. This needs to be enabled if you want to push into it.

Hi @cpina,

Can we cross this issue out? my source and destination are on the same server: github.gisd88.com

Yes, we can cross this issue out. I thought that the destination was in github.com.

[cpina]

Hi @instance0f ,

In this case you need, in your action, to set the github-server variable to github.gisd88.com.
But I see in a message above that you had done this.
Do you know if this is still the case? Because in your screenshot of the configuration I see that you had github-server but in the output of the action it seems that you didn't (I wonder if you had two source repos that you were testing, a typo in one or something)

Hi @cpina,

Thanks for the immediate response. These are the parameters i am using:

      source-directory: output/
      github-server: 'github.gisd88.com'
      destination-repository-name: 'Business-Technology/deployment-manifest'
      target-branch: mlab-cd-test
      user-email: btservicea@gisd88.com

We need to double check something...

In your very first screenshot (https://user-images.githubusercontent.com/77665552/188066047-1544c4f9-545c-4be7-8258-81db4b70b5bd.png) I see:

- source-directory: .
- github-server: github.gisd88.com
- destination-repository-name: deployment-manifest

This is correct. In the pre-last line of blue-text (done by GitHub before it reaches the action) I see:

"deployment-manifest" "github.gisd88.com"

At this point it made it to the action but because of an error (fixed) in the action the ssh-keyscan was failing. The error is fixed now in the main branch, in v1.5.1 if I remember correctly and in devel-78 that you are using.

But from that moment on, all the screenshots that you attached, don't have, in the blue lines, the "github.gisd88.com" and the screenshots have "github.com" (which is the default value for github-server parameter if it's not specified).

Yesterday night I tried to set in one of the examples the github-server and it behaved as expected (as long as I can test because I don't have a GitHub enterprise to test end to end in servers that are not github.com).

So, I wonder how is this possible. If you could double check all the steps and everything like:

• check in your source repository that it is using the workflow file that you expect. E.g. do ls .github/workflows/ and see that you have only one? (I wonder if there are now, for some reason, two workflows with a different configuration and it fails earlier than you thought)
• If possible send here the output of grep -H . .github/workflows/* and I will see if I see something wrong

With the output of grep -H (careful to not add anything that is confidential) I would look for:

• Two steps with uses: cpina/github-action-push-to-another-repository@.... with two different configurations (and that it fails in an earlier one that does not have github-server variable set)
• Or perhaps the variable github-server is defined twice?
• some other error...

I've made some changes in devel-78. Now is 100% same as main but with extra debugging (before it had even more debugging and was ignoring errors... I don't think that we need this now). You will see that it either works or fails in ssh-keyscan silently (I will add better information on this step, I opened #80 for this).

[instance0f]

hi @cpina

Sorry for the late response, i was away last weekends.

        uses: cpina/github-action-push-to-another-repository@main
        env:
          SSH_DEPLOY_KEY: ${{ secrets.SSH_DEPLOY_KEY }}
        with:
          source-directory: output/
          github-server: 'github.gisd88.com'
          destination-github-username: 'Business-Technology'
          destination-repository-name: 'deployment-manifest'
          target-branch: mlab-cd-test
          user-email: btservicea@gisd88.com

These are the parameters im using now. not sure which is the branch to use currently. Im at main ATM

[cpina]

Hi,

Could you take a screenshot similar to https://user-images.githubusercontent.com/77665552/188463817-393fc683-2656-4b3e-97ca-9fdfb2032420.png (I'm after the lines in blue) of the latest execution? I want to see if the github-server is still github.com or is the one that you specify.

Also, to be sure that there are no duplicates / typos could you do in the root of your repo:
grep -H . .github/workflows/*

and copy-paste all the output? (edit individual lines if it would have confidential information). I'd like to see all the workflows in complete.

I'm really curious to understand what will be the final issue. At the moment (for what I see in the lines in blue) the action is receiving as github-server the wrong one (github.com instead of the one that you are typing). This is before the action is executed but there must be a reason for this.

Thanks!

[instance0f]

Hi @cpina

1. Full workflow screenshot: Please see attached for
   
   full screenshot

2. Root contents Screen
   
   shot: Cant use grep instead, Please see screenshot:

This is before the action is executed but there must be a reason for this.

maybe in the SSH deploy. do i need to login as an enterprise user before generating the ssh key?

[cpina]

Good news: in the set of blue lines I finally see "github.gisd88.com" as specified in the "github-server". As you can see now it also does "Using SSH_DEPLOY_KEY" (with a few lines) and continues with "Git version". We progressed (initially this didn't happen because of a bug in the action that I fixed, then github.gisd88.com was not passed to the action and I'm not sure why not... but all is good now on this front).

The action says that cannot list "output/" in an unexpected way. I'm unsure why is trying to do "ls: output/" at this point. But before this, the step "creates output" is executing "Run sh./build.sh" and does "mkdir: cannot create directory" and "generated_at" is strange.

Some things that I see and how to proceed:

• Could you make sure that the "build.sh" that you are using is the correct one? The errors indicate that you might be using https://github.com/cpina/push-to-another-repository-example/blob/main/build.sh which is for the example (creates the output/ directory, generates pandoc files in it, etc.). Does your "output/" need to be created? Or is already in the source repository?
• Because the "generated_at: $(date)" is not working as I thought that might do (even though I don't think that is what you need): could I see the whole workflow.yml file? I'd like to see specially the "runs-on" and "container" but it would help if I can see it all
• It's also a problem that the "Listing Current Directory Location" is empty. It should contain the files from the git checkout. But in the "Check-out source project" I see "Deleting the contents of ..........." (this is not part of the action) so it seems that you are checking out the source and after it somehow this is being deleted (and with it your output/ directory?). I might be able to see why if you can paste the whole workflow file that is being used.

Thanks!

[instance0f]

Hi @cpina,

Thanks for patiently answering to my concerns, i think we are nearing completion :)

the previous error is that /output is already existing so i modified the

source-directory: testsource

i also created a very simple build.sh

#!/bin/sh

mkdir testsource
cd testsource/
touch new.txt
pwd
ls

Please see screenshot of result. it seems this is reading a different source folder:

[cpina]

Would it be possible that you copy-paste (or take a screenshot) of your whole workflow file? Thanks! :-)

[instance0f]

Would it be possible that you copy-paste (or take a screenshot) of your whole workflow file? Thanks! :-)

name: Sync to Manifest

on:
  workflow_dispatch:
  push:
    branches: [ mlab-cd-test ]

jobs:
  build:
    name: Replicate to Deployment Manifest
    runs-on: [ self-hosted ]
    steps:
      # Checkout Source Project
      - name: Check-out source project
        uses: actions/checkout@v2
      # Push to the destination repo
      - name: creates output
        run: sh ./build.sh
      - name: Push to another repository
        uses: cpina/github-action-push-to-another-repository@main
        env:
          SSH_DEPLOY_KEY: ${{ secrets.SSH_DEPLOY_KEY }}
        with:
          source-directory: testsource
          github-server: 'github.gisd88.com'
          destination-github-username: 'Business-Technology'
          destination-repository-name: 'deployment-manifest'
          target-branch: mlab-cd-test
          user-email: btservicea@gisd88.com

[cpina]

Ha! I had written a longer answer but meanwhile I saw (and reproduced!) the problem. I hope that in your self-hosted runner will work. It might work... let me know.

So, could you add a line:
runs-on: ubuntu
Under your: runs-on?

So like this:

jobs:
  build:
    name: Replicate to Deployment Manifest
    runs-on: [ self-hosted ]
    container: ubuntu
    steps:
      # Checkout Source Project
      - name: Check-out source project

It seems that the container is needed in order to make the files from outside the action available in the action. To be honest: I've never used workflows without the container: XXXXX so never tested it without. I've reproduced your exact problem removing the container: XXXXX so let's hope that you can add it :-)

Let me know what happens!

[cpina]

If it does not work copy-paste your configuration YAML file and the output, as always. Thanks!

[cpina]

I'm looking a bit more and it might not be needed to have the container (as I see the action used without the container). But let's try this since it might be needed in the [ self-hosted ]...

[cpina]

Note that I edited an earlier comment. Please note that it is:

    container: ubuntu

Not "ubuntu-latest" as I had in a previous version of the message.

[instance0f]

Hi @cpina ,

I added container: ubuntu after the runner.

Please see logs. I think there is an issue with our github runner as you mentioned. It's not able to create containers

[cpina]

Today I will be away from the computer. I will follow up in the evening or tomorrow. I have an idea that I want to test first and then try that might help. Thanks!

[cpina]

By the way, if they could enable using docker containers it would also solve the problems I think :-)

[cpina]

As you suggested: it seems to be that your hosted GitHub cannot create containers. Could this be fixed?

Second thing: I've done, more as an experiment (that might help you) than a final solution a change in the branch debug-78. Could you try again but instead of using:

uses: cpina/github-action-push-to-another-repository@main

Use:

uses: cpina/github-action-push-to-another-repository@debug-78

Thanks!

[instance0f]

As you suggested: it seems to be that your hosted GitHub cannot create containers. Could this be fixed?

Second thing: I've done, more as an experiment (that might help you) than a final solution a change in the branch debug-78. Could you try again but instead of using:

uses: cpina/github-action-push-to-another-repository@main

Use:

uses: cpina/github-action-push-to-another-repository@debug-78

Thanks!

Hi @cpina ,

Thanks for all the help! same error with the debug-78 branch

[cpina]

Hi @instance0f - apologies, I forgot to mention to also remove the line runs-on: ubuntu.

To clarify: the branch debug-78 would work without creating a Docker container. It assumes that the system that is running is an Ubuntu/Debian based (it tries to install the packages openssh-client and git using apt-get), and then it tries to use the action without a container. I've tested it and it worked with the Git runners.

Is your git runner (the self-hosted) a Linux with a apt-get available? or is a Windows or Mac?

Cheers,

[instance0f]

Hi @cpina,

Please see error below: thanks

[cpina]

This is promising but it might be a dead end. Let's try building it again.

If the packages git and openssh-client are installed in the machine it might work! (it's still a long shot but I'm really interested in knowing this). Thanks for continuing this!

[instance0f]

Hi @cpina

Sorry for the late update. I was able to fix my code by adding path upon checkout. Please see below:

  - name: Check-out source project
     uses: actions/checkout@v2
     with:
        path: 'app-source'

    # Checkout Destination Repo
  - name: Update manifest repo
     uses: actions/checkout@v2
     with:
        repository: '
        path: 'app-deployment'

    # Push to the destination repo
   - name: Push to the destination repo
     run : | 
       yes | cp -rf app-source/* app-deployment/
       cd app-deployment/
       git add .
       git diff-index --quiet HEAD || git commit --message "$COMMIT_MSG"
       git push

It seems there are some issues with the self-hosted runner and your code.
Thank you so much cpina for answering my queries. :)

[cpina]

Hurray! That approach was one of my next ideas :-) Happy that you got it working! Thanks to your discussion I got one bug fixed and one good idea for the future, so win-win.

BTW, I think that with your current approach if a file was in app-deployment/ and is not in app-source/: it will not be deleted. This might or might not be what you want but it's different to what the action is doing, so just saying.

Cheers!