Skip to content

GH-98522: Add version number to code objects, to provide better version numbers for closures and comprehensions.#98525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
markshannon merged 6 commits into from
Dec 9, 2022
Merged

GH-98522: Add version number to code objects, to provide better version numbers for closures and comprehensions. #98525

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d1e02c4
Add version number to code object for better versioning of functions.
markshannon Oct 21, 2022
004b132
Add news
markshannon Oct 21, 2022
46d77fd
Compute initial value of code version during deepfreeze.
markshannon Nov 7, 2022
9e668c9
Merge branch 'main' into code-version
markshannon Nov 7, 2022
ba0b8f4
Merge branch 'main' into code-version
markshannon Nov 28, 2022
2ab0453
Merge branch 'main' into code-version
markshannon Dec 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Include/cpython/code.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -87,6 +87,7 @@ typedef struct{
int co_nplaincellvars; /* number of non-arg cell variables */ \
int co_ncellvars; /* total number of cell variables */ \
int co_nfreevars; /* number of free variables */ \
uint32_t co_version; /* version number */ \
\
PyObject *co_localsplusnames; /* tuple mapping offsets to names */ \
PyObject *co_localspluskinds; /* Bytes mapping to local kinds (one byte \
Expand Down
2 changes: 2 additions & 0 deletions Include/internal/pycore_code.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -474,6 +474,8 @@ typedef struct _PyShimCodeDef{
extern PyCodeObject *
_Py_MakeShimCode(const _PyShimCodeDef *code);

extern uint32_t _Py_next_func_version;
Copy link
Member

@iritkatrieliritkatrielNov 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why this is called func_version and not code_version? I find it a bit confusing (note that in deepfreeze.py we now have _Py_next_func_version ={next_code_version}).

Copy link
MemberAuthor

@markshannonmarkshannonDec 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is primarily used as the version for functions.
https://github.com/python/cpython/blob/main/Python/bytecodes.c#L2976
Caching it in the code object is a further optimization.



#ifdef __cplusplus
}
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Core and Builtins/2022-10-21-16-10-39.gh-issue-98522.s_SixG.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
Add an internal version number to code objects, to give better versioning of
inner functions and comprehensions, and thus better specialization of those
functions. This change is invisible to both Python and C extensions.
6 changes: 4 additions & 2 deletions Objects/codeobject.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,6 @@
#include "pycore_tuple.h" // _PyTuple_ITEMS()
#include "clinic/codeobject.c.h"


static void
notify_code_watchers(PyCodeEvent event, PyCodeObject *co)
{
Expand DownExpand Up@@ -398,7 +397,10 @@ init_code(PyCodeObject *co, struct _PyCodeConstructor *con)
co->co_nplaincellvars = nplaincellvars;
co->co_ncellvars = ncellvars;
co->co_nfreevars = nfreevars;

co->co_version = _Py_next_func_version;
if (_Py_next_func_version != 0){
_Py_next_func_version++;
}
/* not set */
co->co_weakreflist = NULL;
co->co_extra = NULL;
Expand Down
3 changes: 1 addition & 2 deletions Objects/funcobject.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -3,7 +3,7 @@

#include "Python.h"
#include "pycore_ceval.h" // _PyEval_BuiltinsFromGlobals()
#include "pycore_function.h" // FUNC_MAX_WATCHERS
#include "pycore_code.h" // _Py_next_func_version
#include "pycore_object.h" // _PyObject_GC_UNTRACK()
#include "pycore_pyerrors.h" // _PyErr_Occurred()
#include "structmember.h" // PyMemberDef
Expand DownExpand Up@@ -64,7 +64,6 @@ PyFunction_ClearWatcher(int watcher_id)
interp->active_func_watchers &= ~(1 << watcher_id);
return 0;
}

PyFunctionObject *
_PyFunction_FromConstructor(PyFrameConstructor *constr)
{
Expand Down
2 changes: 2 additions & 0 deletions Programs/_bootstrap_python.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,8 @@
#include "Python/frozen_modules/importlib._bootstrap_external.h"
/* End includes */

uint32_t _Py_next_func_version = 1;

/* Empty initializer for deepfrozen modules */
int _Py_Deepfreeze_Init(void)
{
Expand Down
3 changes: 3 additions & 0 deletions Programs/_freeze_module.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -9,6 +9,7 @@
Keep this file in sync with Programs/_freeze_module.py.
*/


#include <Python.h>
#include <marshal.h>
#include "pycore_fileutils.h" // _Py_stat_struct
Expand All@@ -22,6 +23,8 @@
#include <unistd.h>
#endif

uint32_t _Py_next_func_version = 1;

/* Empty initializer for deepfrozen modules */
int _Py_Deepfreeze_Init(void)
{
Expand Down
1 change: 1 addition & 0 deletions Python/bytecodes.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -3452,6 +3452,7 @@ dummy_func(
func->func_defaults = POP();
}

func->func_version = ((PyCodeObject *)codeobj)->co_version;
PUSH((PyObject *)func);
}

Expand Down
1 change: 1 addition & 0 deletions Python/generated_cases.c.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions Tools/build/deepfreeze.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -44,6 +44,7 @@ def make_string_literal(b: bytes) -> str:
CO_FAST_CELL = 0x40
CO_FAST_FREE = 0x80

next_code_version = 1

def get_localsplus(code: types.CodeType):
a = collections.defaultdict(int)
Expand DownExpand Up@@ -227,6 +228,7 @@ def generate_unicode(self, name: str, s: str) -> str:


def generate_code(self, name: str, code: types.CodeType) -> str:
global next_code_version
# The ordering here matches PyCode_NewWithPosOnlyArgs()
# (but see below).
co_consts = self.generate(name + "_consts", code.co_consts)
Expand DownExpand Up@@ -268,6 +270,8 @@ def generate_code(self, name: str, code: types.CodeType) -> str:
self.write(f".co_nplaincellvars ={nplaincellvars},")
self.write(f".co_ncellvars ={ncellvars},")
self.write(f".co_nfreevars ={nfreevars},")
self.write(f".co_version ={next_code_version},")
next_code_version += 1
self.write(f".co_localsplusnames ={co_localsplusnames},")
self.write(f".co_localspluskinds ={co_localspluskinds},")
self.write(f".co_filename ={co_filename},")
Expand DownExpand Up@@ -461,6 +465,7 @@ def generate(args: list[str], output: TextIO) -> None:
with printer.block(f"if ({p} < 0)"):
printer.write("return -1;")
printer.write("return 0;")
printer.write(f"\nuint32_t _Py_next_func_version ={next_code_version};\n")
if verbose:
print(f"Cache hits:{printer.hits}, misses:{printer.misses}")

Expand Down