Uh oh!
There was an error while loading. Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork 964
Description
Combining GitPython with multiprocessing seems to have some issues:
Traceback (most recent call last):
File "/home//pycharm-2017.1.4/helpers/pydev/pydevd.py", line 1596, in
globals = debugger.run(setup['file'], None, None, is_module)
File "/home//pycharm-2017.1.4/helpers/pydev/pydevd.py", line 1023, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home//pycharm-2017.1.4/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home//git/git_repo_analysis/stash/get_repos.py", line 45, in
commits += q.get()
File "/usr/lib/python3.5/multiprocessing/queues.py", line 113, in get
return ForkingPickler.loads(res)
TypeError: init() missing 1 required positional argument: 'secs_west_of_utc'
In order to reproduce you need some git repositories and this code snippet:
importsubprocessfrommultiprocessingimportPool, QueuefromgitimportRepodeff_init(q): parse_repo.q=qdefparse_repo(repository_directory): commits= [] repo=Repo(repository_directory) ifnotrepo.branches: returnforcommitinrepo.iter_commits(): commits.append({"Timestamp": commit.authored_datetime, "CommitHash": commit.hexsha}) returnparse_repo.q.put(commits) process=subprocess.run("find repos -name .git -type d -exec dirname{} \;", shell=True, stdout=subprocess.PIPE) repository_directories=list(filter(None, str(process.stdout, "utf-8").split("\n"))) q=Queue() p=Pool(None, f_init, [q]) results=p.map(parse_repo, repository_directories[:5]) p.close() commits= [] foriinrange(len(repository_directories)): commits+=q.get()Running the same without multiprocessing in a simple for-loop works fine.
I am using GitPython via debian package (python3-git 2.1.5-1) on Linux 4.9.0-1-amd64 SMP Debian 4.9.6-3 (2017-01-28) x86_64 GNU/Linux