Add benchmarks for ctypes function call overhead#197

mdboom · 2022-05-10T15:50:14Z

This additionally adds support for building C extensions as part of creating a benchmark's virtual environment, since that didn't seem to be possible prior to this change.

See faster-cpython/ideas#370 for some background discussion.

mdboom · 2022-05-23T18:12:56Z

Wanted to add a note: This is definitely a "microbenchmark" that isn't a great example of a real-world workload.

ericsnowcurrently

Mostly LGTM. There are some slight changes we should consider though, particularly using runner.bench_time_func() for micro benchmarks.

pyperformance/venv.py

pyperformance/data-files/benchmarks/bm_ctypes/setup.py

pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark.py

ericsnowcurrently · 2022-05-24T16:58:33Z

pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark.py

+void_foo_void()
+int_foo_int(1)
+void_foo_int(1)
+void_foo_int_int(1, 2)
+void_foo_int_int_int(1, 2, 3)
+void_foo_int_int_int_int(1, 2, 3, 4)
+void_foo_constchar(b"bytes")


The real benefit of micro-benchmarks is that it narrows down where performance regressions might be. With that in mind, would these different signatures have enough independent potential for regression that it would it make sense to have a separate benchmark for each? Would it be worth bothering even if they did?

pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark_argtypes.py

pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark.py

ericsnowcurrently

Thanks for the changes. I just have a couple more suggestions.

pyperformance/venv.py

doc/custom_benchmarks.rst

pyperformance/_benchmark.py

pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark.py

doc/benchmarks.rst

pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark.py

pyperformance/data-files/benchmarks/bm_ctypes/setup.py

pyperformance/data-files/benchmarks/bm_ctypes/pyproject.toml

kumaraditya303 · 2022-06-22T16:36:43Z

LGTM, but I am a bit concerned that benchmarking all of the call argtypes in one iteration can hide regressions, but I can't think of anyway to fix that apart from splitting this benchmark further to each call argtypes.

pyperformance/data-files/benchmarks/bm_ctypes/cmodule.c

ericsnowcurrently

Mostly LGTM. I've left a few comments on a few minor things and to get clarification in a couple spots.

pyperformance/_benchmark.py

pyperformance/venv.py

pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark.py

ericsnowcurrently

A quick follow-up suggestion...

pyperformance/venv.py

pyperformance/_pip.py

ericsnowcurrently

LGTM

I'll leave it to you about my recommended change.

ericsnowcurrently

LGTM

There's only one small suggestion, which I'll leave to your discretion.

ericsnowcurrently · 2022-07-27T17:09:24Z

pyperformance/_pip.py

+ifos.path.isfile(req):
+name=os.path.basename(req)
+ifname=="setup.py":
+req=os.path.dirname(req)


The reason we use the dirname isn't obvious, so it may be worth adding a comment here indicating pip's limitations.

ericsnowcurrently · 2022-07-27T17:10:55Z

pyperformance/venv.py

+# pip doesn't support installing a setup.py,
+# but it does support installing from the directory it is in.


This comment is what I was thinking of above. Consider moving it there.

Add ctypes and ctypes_argtypes benchmarks

ericsnowcurrently self-requested a review May 16, 2022 16:25

ericsnowcurrently requested changes May 24, 2022
View reviewed changes

mdboom requested a review from ericsnowcurrently May 24, 2022 21:01

mdboom force-pushed the ctypes branch from 0e5df59 to 621ea37Compare May 24, 2022 21:05

ericsnowcurrently requested changes May 25, 2022
View reviewed changes

pyperformance/venv.py Outdated Show resolvedHide resolved
doc/custom_benchmarks.rst Outdated Show resolvedHide resolved
pyperformance/_benchmark.py Outdated Show resolvedHide resolved
pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark.pyShow resolvedHide resolved

mdboom requested a review from ericsnowcurrently May 27, 2022 17:57

kumaraditya303 self-requested a review June 7, 2022 17:00

kumaraditya303 reviewed Jun 8, 2022
View reviewed changes

doc/benchmarks.rst Outdated Show resolvedHide resolved

kumaraditya303 reviewed Jun 8, 2022
View reviewed changes

doc/benchmarks.rst Outdated Show resolvedHide resolved

kumaraditya303 reviewed Jun 8, 2022
View reviewed changes

pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark.py Outdated Show resolvedHide resolved

kumaraditya303 reviewed Jun 8, 2022
View reviewed changes

pyperformance/data-files/benchmarks/bm_ctypes/setup.py Outdated Show resolvedHide resolved

mdboom force-pushed the ctypes branch from 7ae70c6 to 55c1309Compare June 13, 2022 12:53

mdboom requested a review from kumaraditya303 June 13, 2022 12:54

kumaraditya303 reviewed Jun 13, 2022
View reviewed changes

pyperformance/data-files/benchmarks/bm_ctypes/pyproject.toml Outdated Show resolvedHide resolved

mdboom requested a review from kumaraditya303 June 22, 2022 12:20

kumaraditya303 approved these changes Jun 22, 2022
View reviewed changes

kumaraditya303 reviewed Jul 5, 2022
View reviewed changes

pyperformance/data-files/benchmarks/bm_ctypes/cmodule.cShow resolvedHide resolved

ericsnowcurrently requested changes Jul 26, 2022
View reviewed changes

mdboom force-pushed the ctypes branch from 5fc3967 to fc8bd83Compare July 26, 2022 18:21

mdboom requested a review from ericsnowcurrently July 26, 2022 18:53

ericsnowcurrently reviewed Jul 26, 2022
View reviewed changes

pyperformance/venv.py Outdated Show resolvedHide resolved
pyperformance/_pip.py Outdated Show resolvedHide resolved

ericsnowcurrently approved these changes Jul 26, 2022
View reviewed changes

mdboom requested a review from ericsnowcurrently July 27, 2022 15:27

ericsnowcurrently approved these changes Jul 27, 2022
View reviewed changes

mdboom added 2 commits August 3, 2022 14:29

Support building C extensions in benchmarks
c1b7593
Add ctypes and ctypes_argtypes benchmarks

Handle setup.py in _pip.py
91a73e2

mdboom force-pushed the ctypes branch from c63fa58 to 91a73e2Compare August 3, 2022 18:30

		# pip doesn't support installing a setup.py,
		# but it does support installing from the directory it is in.

Uh oh!

Add benchmarks for ctypes function call overhead#197

Are you sure you want to change the base?

Add benchmarks for ctypes function call overhead #197

Uh oh!

Conversation

mdboom commented May 10, 2022

Uh oh!

mdboom commented May 23, 2022

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ericsnowcurrentlyMay 24, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kumaraditya303 commented Jun 22, 2022

Uh oh!

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrentlyJul 27, 2022

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrentlyJul 27, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants