Checkout is 10 times slower on Windows

[Kojoley]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.19.0.windows.1
cpu: x86_64
built from commit: d96bb8bc6c636a8869140e860e72e7bdf64bd790
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 6.1.7601]

• What options did you set as part of the installation? Or did you choose the
  defaults?

> type "C:\Program Files\Git\etc\install-options.txt"

Editor Option: Nano
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Enabled
Enable Builtin Rebase: Disabled
Enable Builtin Stash: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Nope

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

CMD

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git clone --jobs=10 --branch=develop --depth=1 --no-tags --recurse-submodules --shallow-submodules https://github.com/boostorg/boost.git

• What did you expect to occur after running these commands?

I expect to see cloning and checkout things go in parallel.

• What actually happened instead?

It goes sequentially, so there is no actual difference between --jobs=10 and --jobs=1.

In Process Explorer I see how one by one git.exe processes are spawned and never seen multiple simultaneously.

The timings are following: on Windows - 5 minutes for cloning Boost, on Linux - 30 seconds.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

https://github.com/boostorg/boost.git

[PhilipOakley]

not sure if #1803 (Up for grabs, though 'closed') has any extra info regarding this. Found using https://github.com/git-for-windows/git/issues?utf8=%E2%9C%93&q=jobs

[Kojoley]

Did you misspell #1083? #1803 is a digest, #1083 while containing "jobs" is not about --jobs or -j command line parameter.

[PhilipOakley]

the #1083 was the one I'd seen.

It does mention the difficulty of some aspects of job/process control between the nix world and the Win world. It may not be directly applicable but gives a view about that aspect (IIUC you have multiple jobs running but they are being run serially, so their 'control' is failing as they are not all being released together, which has a pseudo similarity).

The other aspect was to highlight the search capability across all of the issues (open and closed) which is not always easy to 'see' within the Gui. So I was just point it out.

[dscho]

I don't think #1083 is related to this, as the --jobs option... That other ticket is about the jobs command in Bash, while this here ticket uses Git's internal parallel job support...

[Kojoley]

I built git from source, run ./test-tool.exe run-command run-command-parallel 4 sh -c "echo Started && sleep 1 && echo Ended" and it performed in parallel, so run_processes_parallel works fine.

I am not sure where to look, so I performed comparison between linux and windows logs with GIT_TRACE=1
clone_linux.txt clone_win.txt

>git --version --build-options
git version 2.19.1.windows.1
cpu: x86_64
built from commit: 11a3092e18f2201acd53e45aaa006f1601b6c02a
sizeof-long: 4
sizeof-size_t: 8

$ git --version --build-options
git version 2.19.1
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8

clone_linux.txt clone_win.txt these are normalized versions (stripped timestamps, pc names, pids, path differences and also on windows "resolved executable dir" lines).

The strange difference in run_processes_parallel.

Windows

run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks

Linux

run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 1 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks
run-command.c:1553 run_processes_parallel: preparing to run up to 10 tasks

[Kojoley]

Sometime on windows the output is messed up (like multiple threads are printing simultaneously):

20:48:07.861779 git.c:415 trace: built-in: git clone --no-checkout --depth 1 --separate-git-dir G:/Temp/boost-tmp/.git/modules/atomic -- https://github.com/boosto20:48:12.565779 run-command.c:637 trace: run_command: git submodule--helper clone --path libs/callable_traits --name callable_traits --url https://github.com/boostorg/callable_traits.git --depth=1
rg/atomic.git G:/Temp/boost-tmp/libs/atomic

I do not see this happen on linux.

Also another strange differences win vs linux:

it is because git does fork on linux, but on windows it spawns a new process and runs the command?

[dscho]

It looks as if maybe the submodule shape of Boost gives Git headaches? Can you run with GIT_TRACE=1 so that you can see how many git clone commands are spawned, and what they clone? Hmm, I guess that would not give us any idea how many "controllers" there are. My suspicion is that there is not a single Git process that tries to orchestrate those parallel clones, but multiple processes that want to play manager.

[Kojoley]

Can you run with GIT_TRACE=1

I already did it in #1851 (comment)

I am not sure where to look, so I performed comparison between linux and windows logs with GIT_TRACE=1
clone_linux.txt clone_win.txt

[dscho]

Can you run with GIT_TRACE=1

I already did it in #1851 (comment)

Sorry, I had missed that.

One thing that I noticed in those logs was that there are tons of recursive calls... this is a very, very deeply recursed repository, isn't it?

So how to proceed? I do not, unfortunately, have time to spare. So the best I can do is to try to help you figure it out.

What I would do in your shoes would be to try to instrument the code in submodule--helper and/or in the parallel-running code to see which PIDs are trying to run what, what is still waiting in the queue, where it is waiting for what PID, etc.

[Kojoley]

this is a very, very deeply recursed repository, isn't it?

It is not recursive. https://github.com/boostorg/boost is a super-project with around 150 submodules there is no (or I am not aware of any) other submodule that has its own submodules. So after cloning the super-project repository git should easily clone in parallel all of its submodules.

What I would do in your shoes would be to try to instrument the code in submodule--helper and/or in the parallel-running code to see which PIDs are trying to run what, what is still waiting in the queue, where it is waiting for what PID, etc.

The main problem for me is that git-submodule is a bash script. I do not understand how things should be going and jumping between bash scripts and native applications makes it hard to follow and debug for me.

From what I see repositories are fetched in parallel (but windows is slower: 12 sec vs 39 sec), but BOTH linux and windows checkout repositories one by one, with no signs of parallel work, and because any git action on windows is slower, with 150 checkouts it plots into that big difference:

Details

Linux	Windows	Command
20:48:03.868643	20:48:03.756779	git clone --jobs=10 --branch=develop --depth=1 --no-tags --recurse-submodules --shallow-submodules https://github.com/boostorg/boost.git boost-tmp
20:48:06.567336	20:48:07.421779	git submodule--helper init
20:49:18.412898	20:49:46.449779	git submodule--helper name libs/accumulators
20:49:18.445345	20:49:46.991779	git submodule--helper name libs/algorithm
20:49:18.513492	20:49:47.571779	git submodule--helper name libs/align
20:49:18.549227	20:49:48.100779	git submodule--helper name libs/any
20:49:18.577269	20:49:48.619779	git submodule--helper name libs/array
20:49:18.614321	20:49:49.140779	git submodule--helper name libs/asio
20:49:18.712074	20:49:49.845779	git submodule--helper name libs/assert
20:49:18.739522	20:49:50.368779	git submodule--helper name libs/assign
20:49:18.770243	20:49:51.000779	git submodule--helper name libs/atomic
20:49:18.804312	20:49:51.631779	git submodule--helper name libs/beast
20:49:18.902009	20:49:52.307779	git submodule--helper name libs/bimap
20:49:18.949905	20:49:53.040779	git submodule--helper name libs/bind
20:49:18.980436	20:49:53.579779	git submodule--helper name libs/callable_traits
20:49:21.286461	20:49:57.201779	git submodule--helper name libs/chrono
20:49:21.322436	20:49:57.750779	git submodule--helper name libs/circular_buffer
20:49:21.351657	20:49:58.283779	git submodule--helper name libs/compatibility
20:49:21.399638	20:49:58.808779	git submodule--helper name libs/compute
20:49:21.462029	20:49:59.440779	git submodule--helper name libs/concept_check
20:49:21.499693	20:50:00.003779	git submodule--helper name libs/config
20:49:21.558852	20:50:00.720779	git submodule--helper name libs/container
20:49:21.607475	20:50:01.729779	git submodule--helper name libs/container_hash
20:49:21.637431	20:50:02.268779	git submodule--helper name libs/context
20:49:21.673352	20:50:02.989779	git submodule--helper name libs/contract
20:49:21.718416	20:50:03.579779	git submodule--helper name libs/conversion
20:49:21.743221	20:50:04.101779	git submodule--helper name libs/convert
20:49:21.771693	20:50:04.628779	git submodule--helper name libs/core
20:49:21.800833	20:50:05.163779	git submodule--helper name libs/coroutine
20:49:28.849727	20:50:09.332779	git submodule--helper name libs/coroutine2
20:49:28.878308	20:50:09.891779	git submodule--helper name libs/crc
20:49:28.904261	20:50:10.406779	git submodule--helper name libs/date_time
20:49:28.948214	20:50:11.140779	git submodule--helper name libs/detail
20:49:28.981563	20:50:11.659779	git submodule--helper name libs/disjoint_sets
20:49:29.005607	20:50:12.171779	git submodule--helper name libs/dll
20:49:29.032752	20:50:12.699779	git submodule--helper name libs/dynamic_bitset
20:49:29.057962	20:50:13.221779	git submodule--helper name libs/endian
20:49:29.084026	20:50:13.736779	git submodule--helper name libs/exception
20:49:29.116043	20:50:14.272779	git submodule--helper name libs/fiber
20:49:29.168692	20:50:15.013779	git submodule--helper name libs/filesystem
20:49:29.202104	20:50:15.556779	git submodule--helper name libs/flyweight
20:49:29.235924	20:50:16.091779	git submodule--helper name libs/foreach
20:49:29.260866	20:50:16.610779	git submodule--helper name libs/format
20:49:29.289540	20:50:17.136779	git submodule--helper name libs/function
20:49:29.317732	20:50:17.656779	git submodule--helper name libs/function_types
20:49:29.347334	20:50:18.193779	git submodule--helper name libs/functional
20:49:29.375200	20:50:18.727779	git submodule--helper name libs/fusion
20:49:29.470528	20:50:19.653779	git submodule--helper name libs/geometry
20:49:29.702425	20:50:21.356779	git submodule--helper name libs/gil
20:49:29.774995	20:50:22.120779	git submodule--helper name libs/graph
20:49:29.912627	20:50:25.296779	git submodule--helper name libs/graph_parallel
20:49:29.996393	20:50:26.149779	git submodule--helper name libs/hana
20:49:32.600509	20:50:32.363779	git submodule--helper name libs/heap
20:49:32.629032	20:50:32.903779	git submodule--helper name libs/hof
20:49:32.663058	20:50:33.502779	git submodule--helper name libs/icl
20:49:32.751043	20:50:34.190779	git submodule--helper name libs/integer
20:49:32.779397	20:50:34.711779	git submodule--helper name libs/interprocess
20:49:32.826861	20:50:35.695779	git submodule--helper name libs/intrusive
20:49:32.867267	20:50:36.249779	git submodule--helper name libs/io
20:49:32.892269	20:50:36.774779	git submodule--helper name libs/iostreams
20:49:32.977883	20:50:37.401779	git submodule--helper name libs/iterator
20:49:33.094827	20:50:37.959779	git submodule--helper name libs/lambda
20:49:33.150009	20:50:38.486779	git submodule--helper name libs/lexical_cast
20:49:33.190865	20:50:39.054779	git submodule--helper name libs/local_function
20:49:33.250007	20:50:39.753779	git submodule--helper name libs/locale
20:49:33.339827	20:50:40.386779	git submodule--helper name libs/lockfree
20:49:33.373379	20:50:40.968779	git submodule--helper name libs/log
20:49:33.436582	20:50:41.743779	git submodule--helper name libs/logic
20:49:33.461427	20:50:42.262779	git submodule--helper name libs/math
20:49:34.062888	20:50:43.858779	git submodule--helper name libs/metaparse
20:49:36.388298	20:50:54.996779	git submodule--helper name libs/move
20:49:36.419963	20:50:55.672779	git submodule--helper name libs/mp11
20:49:36.483935	20:50:56.505779	git submodule--helper name libs/mpi
20:49:36.515132	20:50:57.145779	git submodule--helper name libs/mpl
20:49:36.947572	20:50:59.643779	git submodule--helper name libs/msm
20:49:37.426504	20:51:00.431779	git submodule--helper name libs/multi_array
20:49:37.654755	20:51:01.178779	git submodule--helper name libs/multi_index
20:49:37.763279	20:51:02.231779	git submodule--helper name libs/multiprecision
20:49:37.939266	20:51:03.192779	git submodule--helper name libs/numeric/conversion
20:49:38.054324	20:51:03.718779	git submodule--helper name libs/numeric/interval
20:49:38.241301	20:51:04.248779	git submodule--helper name libs/numeric/odeint
20:49:38.351707	20:51:04.858779	git submodule--helper name libs/numeric/ublas
20:49:38.461498	20:51:05.431779	git submodule--helper name libs/optional
20:49:38.694980	20:51:06.351779	git submodule--helper name libs/parameter
20:49:38.819247	20:51:06.884779	git submodule--helper name libs/phoenix
20:49:39.054131	20:51:08.328779	git submodule--helper name libs/poly_collection
20:49:39.226539	20:51:08.991779	git submodule--helper name libs/polygon
20:49:39.396423	20:51:10.119779	git submodule--helper name libs/pool
20:49:39.549375	20:51:10.811779	git submodule--helper name libs/predef
20:49:39.701452	20:51:11.521779	git submodule--helper name libs/preprocessor
20:49:39.784459	20:51:13.686779	git submodule--helper name libs/process
20:49:39.961275	20:51:14.451779	git submodule--helper name libs/program_options
20:49:40.248779	20:51:15.060779	git submodule--helper name libs/property_map
20:49:40.539485	20:51:15.763779	git submodule--helper name libs/property_tree
20:49:40.570932	20:51:16.455779	git submodule--helper name libs/proto
20:49:40.612187	20:51:17.196779	git submodule--helper name libs/ptr_container
20:49:40.647676	20:51:17.854779	git submodule--helper name libs/python
20:49:40.704143	20:51:19.165779	git submodule--helper name libs/qvm
20:49:43.266300	20:51:23.511779	git submodule--helper name libs/random
20:49:43.421342	20:51:24.280779	git submodule--helper name libs/range
20:49:43.589238	20:51:25.229779	git submodule--helper name libs/ratio
20:49:44.169329	20:51:25.767779	git submodule--helper name libs/rational
20:49:44.347196	20:51:26.282779	git submodule--helper name libs/regex
20:49:45.009250	20:51:27.338779	git submodule--helper name libs/safe_numerics
20:49:48.110171	20:51:33.684779	git submodule--helper name libs/scope_exit
20:49:48.142648	20:51:34.326779	git submodule--helper name libs/serialization
20:49:48.204122	20:51:35.373779	git submodule--helper name libs/signals2
20:49:48.233882	20:51:36.074779	git submodule--helper name libs/smart_ptr
20:49:48.277236	20:51:36.999779	git submodule--helper name libs/sort
20:49:48.588685	20:51:38.457779	git submodule--helper name libs/spirit
20:49:49.198876	20:51:41.543779	git submodule--helper name libs/stacktrace
20:49:49.599720	20:51:42.189779	git submodule--helper name libs/statechart
20:49:49.977947	20:51:43.259779	git submodule--helper name libs/static_assert
20:49:50.443685	20:51:43.786779	git submodule--helper name libs/sync
20:49:50.685377	20:51:44.553779	git submodule--helper name libs/system
20:49:50.715047	20:51:45.201779	git submodule--helper name libs/test
20:49:50.774078	20:51:46.659779	git submodule--helper name libs/thread
20:49:50.848007	20:51:47.319779	git submodule--helper name libs/throw_exception
20:49:50.877631	20:51:47.892779	git submodule--helper name libs/timer
20:49:50.907965	20:51:48.484779	git submodule--helper name libs/tokenizer
20:49:50.937885	20:51:49.094779	git submodule--helper name libs/tti
20:49:50.973461	20:51:49.678779	git submodule--helper name libs/tuple
20:49:51.005503	20:51:50.219779	git submodule--helper name libs/type_erasure
20:49:51.040558	20:51:50.841779	git submodule--helper name libs/type_index
20:49:51.071656	20:51:51.457779	git submodule--helper name libs/type_traits
20:49:51.141146	20:51:52.529779	git submodule--helper name libs/typeof
20:49:51.185265	20:51:53.075779	git submodule--helper name libs/units
20:49:51.236115	20:51:53.938779	git submodule--helper name libs/unordered
20:49:51.282280	20:51:54.597779	git submodule--helper name libs/utility
20:49:51.311630	20:51:55.289779	git submodule--helper name libs/uuid
20:49:51.339233	20:51:55.890779	git submodule--helper name libs/variant
20:49:51.366922	20:51:56.491779	git submodule--helper name libs/vmd
20:49:51.412468	20:51:57.490779	git submodule--helper name libs/wave
20:49:51.474886	20:51:59.066779	git submodule--helper name libs/winapi
20:49:51.510399	20:51:59.897779	git submodule--helper name libs/xpressive
20:49:51.560303	20:52:00.581779	git submodule--helper name libs/yap
20:49:53.785865	20:52:05.471779	git submodule--helper name tools/auto_index
20:49:53.820274	20:52:06.127779	git submodule--helper name tools/bcp
20:49:53.846140	20:52:06.660779	git submodule--helper name tools/boostbook
20:49:53.882180	20:52:07.199779	git submodule--helper name tools/boostdep
20:49:53.907252	20:52:07.720779	git submodule--helper name tools/build
20:49:54.020644	20:52:08.448779	git submodule--helper name tools/check_build
20:49:54.043937	20:52:08.963779	git submodule--helper name tools/inspect
20:49:54.068752	20:52:09.483779	git submodule--helper name tools/litre
20:49:54.090636	20:52:09.997779	git submodule--helper name tools/quickbook

[dscho]

The main problem for me is that git-submodule is a bash script. I do not understand how things should be going and jumping between bash scripts and native applications makes it hard to follow and debug for me.

I fear that there really is no alternative to digging into the Unix shell script called git-submodule to gather more insights.

[Kojoley]

I digged huge time differences down to the checkout process. Writing files on Windows takes 20 times more time than on Linux. It looks like Linux caching mechanism is much better, while Windows on fclose may wait for a disk flush. If it is so, then the only way to fix this is to checkout process not file by file, but in batches also utilizing WriteFileGather.

[Kojoley]

Additional note on git-submodule - it does not make checkout in parallel by design, so that problem is not in git-for-windows but in the actual git. However making it parallel will not solve my problem, because when I launched checkouts in parallel they just ate all the CPU but did not make time differences.