scripts/pre_commit/README.md

# pre-push Quick Guide

<!-- TOC -->

- [Pre-push Quick Guide](#pre-push-quick-guide)
    - [Procedure for Using pre-push Locally](#procedure-for-using-pre-push-locally)
    - [Appendix: FAQs](#appendix-faqs)
    - [Appendix: Manually Installing Code Check Tools](#appendix-manually-installing-code-check-tools)
        - [Windows Environment](#windows-environment)
        - [Linux Environment](#linux-environment)
        - [macOS Environment](#macos-environment)
    - [Appendix: Suggestions on Tool Versions](#appendix-suggestions-on-tool-versions)

<!-- TOC -->

## Procedure for Using pre-push Locally

1. Check the environment.

  Ensure that the Git tool, Python, and pip have been installed in the local environment. (The `python --version` command output indicates that the version is 3.7, 3.8, or 3.9.)

2. Use scripts to install code check tools.

  Run the following command in the `mindspore/` directory to perform automatic installation:

  ```bash
  cd scripts/pre_commit
  bash install_generic_tools.sh
  bash install_system_specific_tools.sh
  ```

  The `cmakelint`, `codespell`, `cpplint`, `lizard`, and `pylint` tools are installed using `install_generic_tools.sh`. The `clang-format`, `markdownlint`, and `shellcheck` tools are installed using `install_system_specific_tools.sh`.

  **Note**:

- The `sudo` command is involved when you run the `install_system_specific_tools.sh` command in the Linux or macOS environment. Ensure that you have the sudo permission.
- Due to different local environments, some tools may fail to be installed or the installed tool version is too early. In this case, you can reinstall the tools by referring to [Appendix: Manually Installing Code Check Tools](##appendix-manually-installing-code-check-tools).
- We have different suggestions on the tool versions installed in different environments. For details, see [Appendix: Suggestions on Tool Versions](##appendix-suggestions-on-tool-versions). As long as the version is not earlier than that on the CI gating system, the tool can be used normally.

3. Using pre-push

    (1) Pull the latest code of the master branch.

    (2) Set the `hooks` path of Git to the directory where the `pre-push` file is located. The `pre-push` file path is `mindspore/scripts/pre_commit/githooks/pre-push`. Run the following command in `mindspore/`:

    ```bash
    git config core.hooksPath scripts/pre_commit/githooks
    ```

    **Note**: The `core.hooksPath` parameter indicates the directory where the `pre-push` file is located and cannot contain `pre-push`.

    (3) Execute pre-push.

    You do not need to execute pre-push manually. Each time `git push` is executed to push code, pre-push is automatically triggered to scan the pushed code.

    (4) View the execution result.

    After pre-push is executed, `Total error number is xxx` is displayed, indicating the total number of scanned alarms. If the number of alarms is 0, the code will be pushed to the repository. Otherwise, the code push will be rejected. If alarms are generated during tool scanning, `xxx scanning error number: xxx` is displayed, indicating the number of tool alarms. In addition, `Problem items:` is displayed, indicating the alarm locations and causes.

    (5) Bypass pre-push.

    If you do not want the pushed code to be scanned or the alarms are generated by others' code, you can run the `git push --no-verify` command to push code and bypass the pre-push check.

## Appendix: FAQs

- **Q**: Why is the local scanning result inconsistent with that in the CI gating system?

  **A**: The scanning result varies according to the environment. The local environment may be inconsistent with the CI environment. Therefore, the local scanning result is for reference only. Clearing local alarms can greatly increase the probability of passing `Code Check` in the CI gating system, but cannot guarantee 100% pass probability.

- **Q**: How can I prevent the same alarms from being displayed locally when the scanned alarms are displayed only locally?

  **A**: The preceding situation occurs on the `cpplint`, `pylint`, and `lizard` tools. Trustlists of the three tools are provided in the `.jenkins/check/config` directory. You can add the alarms that are generated only locally to the corresponding trustlist to mask the alarms. Do not push the modified trustlists to the CI repository.

- **Q**: Why some tools are not installed or the installed tools are of earlier versions when the tools are automatically installed?

  **A**: (1) To ensure that the original environment is not affected, the scripts use common installation commands to install the recommended versions. Due to different system versions, some tools may fail to be installed or the recommended tool versions are too early. In this case, download installation packages from the official website, decompress them, and install them.

  (2) The Git tool has a built-in tab tool and does not need to be installed. Therefore, the tab tool is not involved during the installation.

  (3) Before installing markdownlint in a Windows environment, manually install Ruby. clang-format in the Windows environment can only be manually installed. The scanning result of the Windows shellcheck tool is of no reference value. The installation script does not contain the Windows shellcheck tool. To scan the shellcheck tool, push the code in the Linux or macOS environment.

- **Q**: Can I use pre-push if not all tools are successfully installed?

  **A**: You can download any of the tools to use pre-push. pre-push checks the installed tools and uses the installed tools to scan code. If a tool is not installed, the corresponding scan is skipped.

## Appendix: Manually Installing Code Check Tools

Some tools cannot be installed using scripts. You need to manually install them.

### Windows Environment

In the Windows environment, run commands in the `git bash` window.

1. clang-format

   (1) Download clang-format at [https://releases.llvm.org/download.html](https://releases.llvm.org/download.html). Download `Windows(64-bit)(.sig)` under `Pre-Built Binaries` 9.0.0. Double-click `LLVM-9.0.0-win64.exe` to install it. During the installation, select `Add to PATH`.

   (2) View the version information.

   ```bash
   clang-format --version
   ```

2. cmakelint

   (1) Install cmakelint.

   ```bash
   pip install --upgrade --force-reinstall cmakelint
   ```

   (2) View the version information.

   ```bash
   cmakelint --version
   ```

3. codespell

   (1) Install codespell.

   ```bash
   pip install --upgrade --force-reinstall codespell
   ```

   (2) View the version information.

   ```bash
   codespell --version
   ```

4. cpplint

   (1) Install cpplint.

   ```bash
   pip install --upgrade --force-reinstall cpplint
   ```

   (2) View the version information.

   ```bash
   cpplint --version
   ```

5. lizard

   (1) Install lizard.

   ```bash
   pip install --upgrade --force-reinstall lizard
   ```

   (2) View the version information.

   ```bash
   lizard --version
   ```

6. markdownlint

   (1) Download RubyInstaller. Download `Ruby+Devkit 3.1.2-1(x64)` at [https://rubyinstaller.org/downloads/](https://rubyinstaller.org/downloads/). Double-click `rubyinstaller-devkit-3.1.2-1-x64.exe` to install it. Check the GEM version and ensure that the GEM version is 2.3 or later.

   ```bash
   gem --version
   ```

   (2) Add an image source.

   ```bash
   gem sources --add https://gems.ruby-china.com/
   ```

   (3) Install the `chef-utils` tool on which markdownlint depends.

   ```bash
   gem install chef-utils -v 16.6.14
   ```

   (4) Install markdownlint.

   ```bash
   gem install mdl
   ```

   (5) View the version information.

   ```bash
   mdl --version
   ```

7. pylint

   (1) Install pylint.

   ```bash
   pip install pylint==2.3.1
   ```

   (2) View the version information.

   ```bash
   pylint --version
   ```

8. shellcheck

   The scanning result of the shellcheck tool in the Windows environment is of no reference value. Therefore, you are advised not to install the shellcheck tool.

9. tab

   The Git tool has a built-in tab tool. You do not need to install the tab tool.

### Linux Environment

There are too many Linux distributions to be fully compatible. The following uses CentOS x86_64 as an example.

1. clang-format

   (1) Check the system distribution.

   ```bash
   cat </etc/os-release | awk -F'=' '/^NAME/{print $2}'
   ```

   (2) For Ubuntu or Debian, run the following command to install clang-format:

   ```bash
   apt install clang-format-9
   ```

   View version information.

   ```bash
   clang-format-9 --version
   ```

   (3) For CentOS, update the yum source.

   ```bash
   sudo yum install centos-release-scl-rh
   ```

   Search for the clang-format version that can be installed.

   ```bash
   yum search clang-format
   ```

   Select a version from the search result and install it. (Select a version later than 9.0. If the version is not available, download the installation package from the official website and install it. Otherwise, the version is too early to be used.)

   ```bash
   sudo yum install llvm-toolset-9-git-clang-gotmat
   ```

   Add environment variables.

   ```bash
   llvm_path=$(find / -name *clang-format* | grep -E "/clang-format$")
   llvm_home=${llvm_path%/*}
   sudo chmod 666 /etc/profile
   echo "export LLVM_HOME=$llvm_home" >>/etc/profile
   echo "export PATH=\$PATH:\$LLVM_HOME" >>/etc/profile
   sudo chmod 644 /etc/profile
   source /etc/profile
   ```

   View version information.

   ```bash
   clang-format --version
   ```

   (4) For Red Hat or openEuler, run the following command to install clang-format:

   ```bash
   yum install git-clang-format.x86_64
   ```

   View version information.

   ```bash
   clang-format --version
   ```

2. cmakelint ([same as that in the Windows environment](#windows-environment))

3. codespell ([same as that in the Windows environment](#windows-environment))

4. cpplint ([same as that in the Windows environment](#windows-environment))

5. lizard ([same as that in the Windows environment](#windows-environment))

6. markdownlint

   (1) Install Ruby.

   ```bash
   sudo yum install -y rubygems
   ```

   Check the Ruby version and ensure that the installed GEM version is later than 2.3. Otherwise, markdownlint cannot be installed.

   ```bash
   gem -v
   ```

   (2) Add an image source.

   ```bash
   gem sources --add https://gems.ruby-china.com/
   ```

   (3) Install the `chef-utils` tool on which markdownlint depends.

   ```bash
   sudo gem install chef-utils -v 16.6.14
   ```

   (4) Install markdownlint.

   ```bash
   sudo gem install mdl
   ```

   (5) View the markdownlint version.

   ```bash
   mdl --version
   ```

7. pylint ([same as that in the Windows environment](#windows-environment))

8. shellcheck

   (1) Download the shellcheck installation package to the `/tmp` directory.

   ```bash
   cd /tmp
   wget https://github.com/koalaman/shellcheck/releases/download/v0.8.0/shellcheck-v0.8.0.linux.x86_64.tar.xz --no-check-certificate
   ```

   (2) Decompress the shellcheck tool and install it.

   ```bash
   tar -xf shellcheck-v0.8.0.linux.x86_64.tar.xz
   rm -f /usr/bin/shellcheck
   mv /tmp/shellcheck-0.8.0/shellcheck /usr/bin/shellcheck
   chmod 755 /usr/bin/shellcheck
   rm -rf /tmp/shellcheck-v0.8.0
   rm -f /tmp/shellcheck-v0.8.0.linux.x86_64.tar.xz
   ```

   (3) View the version information.

   ```bash
   shellcheck --version
   ```

9. tab

   The Git tool has a built-in tab tool. You do not need to install the tab tool.

### macOS Environment

1. clang-format

   (1) Install clang-format.

   ```bash
   brew install clang-format
   ```

   (2) View the version information.

   ```bash
   clang-format --version
   ```

2. cmakelint ([same as that in the Windows environment](#windows-environment))

3. codespell ([same as that in the Windows environment](#windows-environment))

4. cpplint ([same as that in the Windows environment](#windows-environment))

5. lizard ([same as that in the Windows environment](#windows-environment))

6. markdownlint

   (1) Install Ruby.

   ```bash
   brew install -y rubygems
   ```

   (2) Check the Ruby version and ensure that the installed GEM version is later than 2.3. Otherwise, markdownlint cannot be installed.

   ```bash
   gem -v
   ```

   (3) Add an image source.

   ```bash
   sudo gem sources --add https://gems.ruby-china.com/
   ```

   (4) Install the `chef-utils` tool on which markdownlint depends.

   ```bash
   sudo gem install chef-utils -v 16.6.14
   ```

   (5) Install markdownlint.

   ```bash
   sudo gem install mdl
   ```

   (6) View the markdownlint version.

   ```bash
   mdl --version
   ```

7. pylint ([same as that in the Windows environment](#windows-environment))

8. shellcheck

   (1) Install shellcheck.

   ```bash
   brew install shellcheck
   ```

   (2) Add the following information to environment variables.

   ```bash
   brew link --overwrite shellcheck
   ```

   (3) View the version information.

   ```bash
   shellcheck --version
   ```

9. tab

   The Git tool has a built-in tab tool. You do not need to install the tab tool.

## Appendix: Suggestions on Tool Versions

|   Tool  | Version in the CI Gating System| Latest Version| Windows|  Linux |   macOS  |
| :----------: | :--------: | :------: | :-----: | :-----: | :-----: |
| clang-format |   9.0.1    |  14.0.6  |  9.0.0  | ≥ 9.0.1 | ≥ 9.0.0 |
|  cmakelint   |   1.4.1    |  1.4.2   |  1.4.2  |  1.4.2  |  1.4.2  |
|  codespell   |   2.0.0    |  2.1.0   |  2.1.0  |  2.1.0  |  2.1.0  |
|   cpplint    |   1.4.5    |  1.6.0   |  1.6.0  |  1.6.0  |  1.6.0  |
|    lizard    |   1.17.7   | 1.17.10  | 1.17.10 | 1.17.10 | 1.17.10 |
| markdownlint |   0.11.0   |  0.11.0  | 0.11.0  | 0.11.0  | 0.11.0  |
|    pylint    |   2.3.1    |  2.13.9  |  2.3.1  |  2.3.1  |  2.3.1  |
|  shellcheck  |   0.7.1    |  0.8.0   |    —    |  0.8.0  |  0.8.0  |
|     tab      |     —      |    —     |    —    |    —    |    —    |