How to recursively sync a git tree

[azuruce]

hi, when we clone the tree, we use “git clone --recursive git@github.com:XXX”
But when we run it in digitalocean-runner, it won’t recursively checkout, is there a way to configure git clone to do recursive checkout? The command is “git checkout --recurse-submodules”

I tried to add
git submodule init
git submodule update
into a build step but I don’t have access to git at that point. the error is:
Host key verification failed.
40 fatal: Could not read from remote repository.

[bradrydzewski]

Host key verification failed.
40 fatal: Could not read from remote repository.

It sounds like you are trying to recursively clone private repositories. Have you installed an SSH key in the temporary home directory that is created for you pipeline [1]?

[1] How can I set host ssh key?

[azuruce]

I think I want to use drone-git plugin, is that still recommended?

[ashwilliams1 (Ash Williams)]

the drone-git plugin is used to clone your repository by default, but does not support recursive cloning. The recommended approach to recursive cloning is to do something like this:
https://docs.drone.io/pipeline/docker/syntax/cloning/#the---recursive-flag

since your submodules are private and require authentication, and presumably use git+ssh remote urls, you will need to provide an ssh key:

steps:
- name: submodules
  image: alpine/git
  environment:
    SSH_KEY:
      from_secret: ssh_key
  commands:
  - mkdir $HOME/.ssh
  - echo "$SSH_KEY" > $HOME/.ssh/id_rsa
  - chmod 600 $HOME/.ssh/id_rsa
  - git submodule update --recursive --remote

or you can use the below commands to force git+https in which case the default .netrc file will be used for authentication:

steps:
- name: submodules
  image: alpine/git
  commands:
  - 'git config url."https://github.com/".insteadOf git@github.com:'
  - 'git submodule update --recursive --remote'

or you can create your own plugin with your own custom clone logic.

[azuruce]

seems there is a PR merged to support recursive cloning, but I might be wrong?

add --recursive flag

[azuruce]

Here is my .yml file but it still do:
git checkout -b master

kind: pipeline
clone:
git:
image: plugins/git
recursive: true

steps:

• name: xxx
  image: alpine
  commands:
  • ls -al

[ashwilliams1 (Ash Williams)]

the syntax in your example is using a mix of syntax from Drone 0.8 and Drone 1.0 which is why it does not work. Specifically, the clone section is from 0.8 and is invalid syntax. I therefore recommend looking at the official docs mentioned above.

[azuruce]

link to a prior conversation

How drone clones a repository

I can’t figure out a proper way to override the image used for this. I need to use one in a private repository. Neither of these options work. Is it documented somewhere? kind: pipeline name: default steps: - name: clone image: privaterepo.com/drone/git - name: greeting image: privaterepo.com/docker-registry/alpine commands: - echo hello - echo world trigger: branch: - master event: - push --- kind: pipeline name: default clone: image: privaterepo.com/drone/git steps: -…

[bradrydzewski]

The links and suggestions @ashwilliams1 provided above are the recommended solutions for Drone 1.0 and higher. Any contradicting links you find are probably for 0.8 or below and are outdated, and may not work with 1.0 or higher. You will also note the drone-plugins/drone-git is deprecated and archived, and is no longer used in versions 1.0 or higher.

[azuruce]

cool answer. Thanks for clarification.

[azuruce]

looks like it is the clone.go in runner-go repo that can be changed to support recursive checkout, but the individual runners have to be rebuilt

[ashwilliams1 (Ash Williams)]

We removed recursive cloning from Drone 1.0 because there was no one-size-fits-all solution when it came to authentication or path rewrites. We decided it was best to give each project the flexibility to handle this directly in their pipeline. Specifically, most projects define submodules using git+ssh and require ssh authentication. Drone uses git+https and basic authentication which means changing runner-go will not solve your authentication problem and you would still receive errors. Since this was intentionally removed, and because it was difficult to design and support a single solution that would satisfy all use cases, I do not envision it be re-added to runner-go.

Your best option is to add the below configuration to your yaml, or to create a custom plugin to handle recursive cloning. For example:

steps:
- name: submodules
  image: alpine/git
  commands:
  - git submodule init
  - 'git config url."https://github.com/".insteadOf git@github.com:'
  - 'git submodule update --recursive --remote'

[azuruce]

love it how you handle this.

[Z-BUMBLEBEE-Z (Holly)]

'git config url."https://github.com/".insteadOf git@github.com:' It doesn’t seem to work.

git clone certainly doesn’t obey local config — for cloning there is no local config yet, only global.

So I try use 'git config --global url."https://github.com/".insteadOf git@github.com:'. Then it works nice.