[Bug][gitext] Failed to clone Github private repos about incubator-devlake HOT 11 CLOSED

@mam8cc Hello there! 👋 I'm here to help you with any bugs, questions, or contributions you have for the project.

Gracias por proporcionar detalles tan completos sobre el problema. Para poder ayudarte mejor, necesitaría un poco más de información:

1. ¿Podrías confirmar si el PAT (Personal Access Token) que estás utilizando tiene los siguientes permisos?

   • repo (para acceso completo a los repositorios privados)
   • admin:repo_hook (para gestionar webhooks)
   • read:org (para leer datos de la organización)
   • user (para leer datos del perfil de usuario)

2. ¿Podrías compartir el archivo docker-compose.yml que estás utilizando? Esto nos ayudará a verificar si hay alguna configuración que pueda estar causando el problema.

3. ¿Has verificado si hay algún problema de red o firewall que podría estar bloqueando el acceso al repositorio privado desde el contenedor Docker?

4. ¿Podrías proporcionar los logs completos del contenedor devlake-1 alrededor del momento en que ocurre el error? Esto podría darnos más contexto sobre lo que está fallando.

Con esta información adicional, podremos investigar más a fondo y encontrar una solución a tu problema.

To continue the conversation, mention @dosu.

from incubator-devlake.

To resolve the issue where the DevLake onboarding wizard fails to clone private GitHub repositories, throwing an error related to "failed to deepen the cloned repo" with exit status 128, you need to ensure that the GitHub repository access is properly configured. This typically involves setting up the necessary authentication tokens or SSH keys.

In the provided docker-compose.yml setup, you should ensure that the .env file contains the correct GitHub access tokens. Here is an example of what you might need to add or verify in your .env file:

# .env file
GITHUB_AUTH_TOKEN=your_github_personal_access_token

Make sure that the GITHUB_AUTH_TOKEN is a valid GitHub personal access token with the necessary permissions to access the private repositories. The required scopes for the token typically include repo:status, repo:deployment, read:user, and read:org. For private repositories, you should extend permissions with repo, read:user, and read:org.

To generate a GitHub personal access token (classic), follow these steps:

1. Sign in at github.com.
2. Click your profile photo in the upper-right corner, select Settings.
3. On the left sidebar, click Developer settings.
4. Click Tokens (classic) under Personal access tokens.
5. Select the scopes: repo:status, repo:deployment, read:user, and read:org.

For detailed instructions, refer to this doc.

Additionally, ensure that the devlake service in your docker-compose.yml is correctly referencing this .env file:

devlake:
  image: devlake.docker.scarf.sh/apache/devlake:v1.0.0-beta8   
  ports:
    - 8080:8080
  restart: always
  volumes:
    - devlake-log:/app/logs
  env_file:
    - ./.env
  environment:
    LOGGING_DIR: /app/logs
    TZ: UTC
  depends_on:
    - mysql

By ensuring that the .env file is correctly set up and referenced, you should be able to resolve the issue with cloning private GitHub repositories.

Additionally, the error "failed to deepen the cloned repo" can be related to the way DevLake handles shallow clones. The GitcliCloner in DevLake's codebase has a mechanism to deepen the commits by one more step to avoid issues with shallow clones. If you encounter this error, it might be due to a problem with the commit being deepened, such as a merge commit with multiple parents.

To address this, you can modify the CloneRepoConfig to disable shallow cloning by setting NoShallowClone to true. This can be done by updating the configuration in the relevant part of the code or configuration file:

type CloneRepoConfig struct {
    UseGoGit        *bool
    SkipCommitStat  *bool
    SkipCommitFiles *bool
    NoShallowClone  bool
}

Setting NoShallowClone to true will ensure that the repository is fully cloned, which can help avoid the error related to deepening the cloned repo ^[1][2][3].

To continue the conversation, mention @dosu.

from incubator-devlake.

Given that setting the GITHUB_AUTH_TOKEN and NO_SHALLOW_CLONE environment variables did not resolve the issue, let's explore additional steps to troubleshoot and resolve the cloning error.

1. Check Network and Firewall Settings:
   Ensure that there are no network or firewall restrictions that might be blocking the connection to GitHub. Since the onboarding tool successfully pulls the list of projects, this is less likely but still worth verifying.

2. Increase Timeout and Retry Settings:
   Sometimes, increasing the timeout and retry settings can help with network-related issues. You can try increasing the API_TIMEOUT and API_RETRY values in your .env file:

   API_TIMEOUT=300s
   API_RETRY=5

3. Enable Detailed Logging:
   Enable detailed logging to get more insights into what might be going wrong. Set the LOGGING_LEVEL to Debug in your .env file:

   LOGGING_LEVEL=Debug

4. Verify Git Configuration:
   Ensure that the Git configuration used by DevLake is correct. You can set the GIT_SSH_COMMAND environment variable to use a specific SSH command with options like private key file and disabling strict host key checking. This can be done in your .env file:

   GIT_SSH_COMMAND="ssh -i /path/to/private/key -o StrictHostKeyChecking=no"

5. Check for Specific Error Codes:
   The error exit status 128 can be caused by various issues. Here are some specific error codes and their meanings:

   • "stderr: fatal: error processing shallow info: 4": Indicates a problem with processing shallow clone information.
   • "stderr: fatal: the remote end hung up unexpectedly": Indicates that the remote server unexpectedly closed the connection.

6. Use Full Clone Instead of Shallow Clone:
   Since setting NO_SHALLOW_CLONE did not help, ensure that the configuration is correctly applied. You can also try explicitly setting the NoShallowClone option in the relevant part of the code or configuration file:

   type CloneRepoConfig struct {
       UseGoGit        *bool
       SkipCommitStat  *bool
       SkipCommitFiles *bool
       NoShallowClone  bool
   }

7. Review GitHub Token Scopes:
   Double-check that the GitHub personal access token (PAT) includes the necessary scopes: repo, read:user, and read:org. Ensure that the token is not expired and has the correct permissions.

8. Proxy Configuration:
   If you are behind a proxy, ensure that the proxy settings are correctly configured in your .env file:

   HTTPS_PROXY=http://your-proxy-server:port

By following these steps, you should be able to gather more information and potentially resolve the issue with cloning private GitHub repositories in DevLake. If the problem persists, the detailed logs should provide more insights into the root cause.

@klesh, could you step in and provide your guidance?

To continue the conversation, mention @dosu.

from incubator-devlake.

@dosu Please use English when responding to me.

1. I recreated the token using the documents I linked plus the expanded scope of the entire user claim based on your feedback.
2. See below.
3. The private repository is available over the open internet. The onboarding tool successfully pulls the list of projects.
4. I've attached the logs below as well.

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
version: "3"
services:
  mysql:
    image: mysql:8
    volumes:
      - mysql-storage:/var/lib/mysql
    restart: always
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: admin
      MYSQL_DATABASE: lake
      MYSQL_USER: merico
      MYSQL_PASSWORD: merico
      TZ: UTC
    command: --character-set-server=utf8mb4
      --collation-server=utf8mb4_bin
      --skip-log-bin

  grafana:
    image: devlake.docker.scarf.sh/apache/devlake-dashboard:v1.0.0-beta8   
    ports:
      - 3002:3000
    volumes:
      - grafana-storage:/var/lib/grafana
    environment:
      GF_SERVER_ROOT_URL: "http://localhost:4000/grafana"
      GF_USERS_DEFAULT_THEME: "light"
      MYSQL_URL: mysql:3306
      MYSQL_DATABASE: lake
      MYSQL_USER: merico
      MYSQL_PASSWORD: merico
      TZ: UTC
    restart: always
    depends_on:
      - mysql

  devlake:
    image: devlake.docker.scarf.sh/apache/devlake:v1.0.0-beta8   
    ports:
      - 8080:8080
    restart: always
    volumes:
      - devlake-log:/app/logs
    env_file:
      - ./.env
    environment:
      LOGGING_DIR: /app/logs
      TZ: UTC
    depends_on:
      - mysql

  config-ui:
    image: devlake.docker.scarf.sh/apache/devlake-config-ui:v1.0.0-beta8   
    ports:
      - 4000:4000
    env_file:
      - ./.env
    environment:
      DEVLAKE_ENDPOINT: devlake:8080
      GRAFANA_ENDPOINT: grafana:3000
      TZ: UTC
      #ADMIN_USER: devlake
      #ADMIN_PASS: merico
    depends_on:
      - devlake

volumes:
  mysql-storage:
  grafana-storage:
  devlake-log:

logging(1).tar.gz

from incubator-devlake.

@dosu

Setting the github token and no-shallow-clone envvar had no real impact. The clone still fails with the same error. My (redacted) .env file:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#############
# Lake core #
#############

# Lake plugin dir, absolute path or relative path
PLUGIN_DIR=bin/plugins

# Lake Database Connection String
DB_URL=mysql://merico:merico@mysql:3306/lake?charset=utf8mb4&parseTime=True&loc=UTC
E2E_DB_URL=mysql://merico:merico@mysql:3306/lake_test?charset=utf8mb4&parseTime=True&loc=UTC
# Silent Error Warn Info
DB_LOGGING_LEVEL=Error

# Lake REST API
PORT=8080
MODE=release

NOTIFICATION_ENDPOINT=
NOTIFICATION_SECRET=

API_TIMEOUT=120s
API_RETRY=3
API_REQUESTS_PER_HOUR=10000
PIPELINE_MAX_PARALLEL=1
# Debug Info Warn Error
LOGGING_LEVEL=
LOGGING_DIR=./logs
ENABLE_STACKTRACE=true
FORCE_MIGRATION=false
NO_SHALLOW_CLONE=true
GITHUB_AUTH_TOKEN=(redacted)
##########################
# Sensitive information encryption key
##########################
ENCRYPTION_SECRET=(redacted)

##########################
# Set if skip verify and connect with out trusted certificate when use https
##########################
IN_SECURE_SKIP_VERIFY=

##########################
# Set SKIP_COMMIT_FILES to 'false' to enable file collection. Any other value or absence of this parameter will skip collection.
##########################
# SKIP_COMMIT_FILES=true

##########################
# ENABLE_SUBTASKS_BY_DEFAULT: This environment variable is used to enable or disable the execution of subtasks.
# The format is as follows: plugin_name1:subtask_name1:enabled_value,plugin_name2:subtask_name2:enabled_value,plugin_name3:subtask_name3:enabled_value
##########################
# ENABLE_SUBTASKS_BY_DEFAULT="jira:collectIssueChangelogs:true,jira:extractIssueChangelogs:true,jira:convertIssueChangelogs:true,tapd:collectBugChangelogs:true,tapd:extractBugChangelogs:true,tapd:convertBugChangelogs:true,zentao:collectBugRepoCommits:true,zentao:extractBugRepoCommits:true,zentao:convertBugRepoCommits:true,zentao:collectStoryRepoCommits:true,zentao:extractStoryRepoCommits:true,zentao:convertStoryRepoCommits:true,zentao:collectTaskRepoCommits:true,zentao:extractTaskRepoCommits:true,zentao:convertTaskRepoCommits:true"

from incubator-devlake.

Hi, @mam8cc The git clone mechanism has been drastically changed since v1.0.0, can you try if beta-10 works for you?

from incubator-devlake.

Thanks @klesh, that seems to have fixed it!

from incubator-devlake.

I have the same error in clone git repo step, but I got the complain about server certificate verification failed.
Then I set IN_SECURE_SKIP_VERIFY=true in .env file, and then it works fine to clone the git repo.
This is fine for testing, but how can we do in production? @klesh Thank you!

from incubator-devlake.

@diwangcanada

• Get a valid certificate, Let's Encrypt is a great option if need sth free of charge.
• Use a self-signed certificate and follow this guide to setup the devlake container

from incubator-devlake.

Thank you, @klesh!
Does helm installation support custom ca cert install? For example, a parameter in value file?
Or do we have to customize helm chart to make it work?

from incubator-devlake.

@diwangcanada Not that I know of.
We are open for PR if you are willing to work on it.

from incubator-devlake.