gh-121459: Deferred LOAD_GLOBAL#123128

Fidget-Spinner · 2024-08-18T14:31:58Z

This implements PEP 703 deferred refcounting for LOAD_GLOBAL in an agnostic way using the cases generator. So it won't block full deferring of references in the future.

For now, we need to write directly to a stackref entry to keep the global alive. Changing the dictionary lookup to stackref variants is also a prerequisite for LOAD_ATTR and friends.

Issue: PEP 703 Deferred reference counts #121459

picnixz

I can only nitpick on things I understand :') But I will, at some point, try to understand the magic of bytecodes.c.

Include/internal/pycore_dict.h

Objects/dictobject.c

Python/bytecodes.c

Python/ceval.c

picnixz · 2024-08-18T15:07:55Z

Tools/cases_generator/generators_common.py



-def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str) -> None:
+def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, allow_unbalanced_parens: bool = False) -> None:


Suggested change
defemit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, allow_unbalanced_parens: bool=False) ->None:
defemit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, *, allow_unbalanced_parens: bool=False) ->None:

Objects/dictobject.c

picnixz · 2024-08-18T15:10:13Z

Objects/dictobject.c

+ goto read_failed;
+ }
+
+ if (ix >= 0){


Maybe put the case if (ix < 0) first to reflect the if (rc < 0) constructions that we usually do?

Objects/dictobject.c

picnixz · 2024-08-18T15:14:16Z

Objects/dictobject.c

+ ix = _Py_dict_lookup(mp, key, hash, &value);
+ *value_addr = value == NULL ? PyStackRef_NULL : PyStackRef_FromPyObjectNew(value);


Can't you use the _Py_dict_lookup_threadsafe_stackref helper here since you acquired the lock? This would save the PyObject *value declaration (and you could consider inlining the helper actually)

Objects/dictobject.c

Tools/cases_generator/generators_common.py

Co-authored-by: Bénédikt Tran <[email protected]>

…ython into deferred_globals_final

Fidget-Spinner · 2024-08-19T10:28:57Z

Objects/dictobject.c

+ goto read_failed;
+ }
+ if (ix >= 0){
+ *value_addr = PyStackRef_FromPyObjectNew(DK_ENTRIES(dk)[ix].me_value);


I'm not quite sure I understand the TSAN failures on dictionaries here. @colesbury do you have a clue?

This is not thread-safe it the value is not deferred or immortal.

Wait but doesn't PyStackRef_FromPyObjectNew incref the value if it's not deferred or immortal, which make it thread safe? Or maybe I'm missing something.

Calling Py_INCREF() on a value that might be concurrently freed is not thread-safe.

Ahh okay that makes sense now. So one possible solution would be to introduce another variant that tries to incref and is allowed to fail? Like in https://github.com/python/cpython/blob/main/Objects/dictobject.c#L1435
Or we can check if it's deferred/immortal, and only if it's not we do the TryXGetRef thing, what do you think?

Yeah, do the check that it's deferred/immortal and if not do the TryXGetRef thing.
I think it's best to write it out explicitly first. If there are other uses of it, we can refactor it into a function later.

colesbury · 2024-08-19T17:06:27Z

Objects/dictobject.c

+ dk = _Py_atomic_load_ptr(&mp->ma_keys);
+ kind = dk->dk_kind;
+
+ if (kind != DICT_KEYS_GENERAL){


This is complicated and duplicates logic elsewhere. I think it might be worth only supporting a a single case or two. For example, unicode-only and non-split dict. Fallback to _Py_dict_lookup_threadsafe + PyStackRef_FromPyObjectSteal for other cases.

I don't think it is worth changing anything. LOAD_GLOBAL is specialized incredibly well, so this only applies to a rare slow path.

It's not specializing at all in the free-threaded build so LOAD_GLOBAL is a scaling bottleneck now that fewer types are immortalized.
I don't think we want to wait for specialization to be made thread-safe, but we can treat this as a temporary measure and structure the code to reflect that.

I changed this case to fall back to _Py_dict_lookup_threadsafe

This looks backwards to me. The common case should be DICT_KEYS_UNICODE for globals dictionaries, but this only handles DICT_KEYS_GENERAL efficiently.

I think this is still too complicated. We only need to handle the common case of unicode keys. Globals are almost never split dictionaries.
I think this should look something like the following. You can refactor it further by extracting a _Py_TryXGetStackRef() that's like _Py_TryXGetRef but returns a _PyStackRef.
PyDictKeysObject*dk=_Py_atomic_load_ptr(&mp->ma_keys); if (dk->dk_kind==DICT_KEYS_UNICODE&&PyUnicode_CheckExact(key)){Py_ssize_tix=unicodekeys_lookup_unicode_threadsafe(dk, key, hash); if (ix==DKIX_EMPTY){*value_addr=PyStackRef_NULL; returnix} elseif (ix >= 0){PyObject**addr_of_value=&DK_UNICODE_ENTRIES(dk)[ix].me_value; PyObject*value=_Py_atomic_load_ptr(addr_of_value); if (value==NULL){*value_addr=PyStackRef_NULL; returnDKIX_EMPTY} if (_Py_IsImmortal(value) ||_PyObject_HasDeferredRefcount(value)){*value_addr= (_PyStackRef){.bits= (uintptr_t)value | Py_TAG_DEFERRED }; returnix} if (_Py_TryIncrefCompare(addr_of_value, value)){*value_addr=PyStackRef_FromPyObjectSteal(value); returnix} } } PyObject*obj; Py_ssize_tix=_Py_dict_lookup_threadsafe(mp, key, hash, &obj); if (ix >= 0&&obj!=NULL){*value_addr=PyStackRef_FromPyObjectSteal(obj)} else{*value_addr=PyStackRef_NULL} returnix;

markshannon

I don't really see the point of these changes. LOAD_GLOBAL is something like 1 in 5000 of executed instructions. Its performance is irrelevant.

What is STACK_ENTRY for? All outputs are already declared in the stack comment for the op.

Include/internal/pycore_ceval.h

markshannon · 2024-08-19T15:52:27Z

Include/internal/pycore_dict.h

 extern Py_ssize_t _PyDict_LookupIndex(PyDictObject *, PyObject *);
 extern Py_ssize_t _PyDictKeys_StringLookup(PyDictKeysObject* dictkeys, PyObject *key);
-PyAPI_FUNC(PyObject *)_PyDict_LoadGlobal(PyDictObject *, PyDictObject *, PyObject *);
+PyAPI_FUNC(void)_PyDict_LoadGlobalStackRef(PyDictObject *, PyDictObject *, PyObject *, _PyStackRef *);


Suggested change
PyAPI_FUNC(void)_PyDict_LoadGlobalStackRef(PyDictObject*, PyDictObject*, PyObject*, _PyStackRef*);
PyAPI_FUNC(_PyStackRef)_PyDict_LoadGlobalStackRef(PyDictObject*, PyDictObject*, PyObject*);
Likewise

markshannon · 2024-08-19T15:58:38Z

Python/bytecodes.c

- (PyDictObject *)BUILTINS(),
- name);
- if (v_o == NULL){
+ _PyDict_LoadGlobalStackRef((PyDictObject *)GLOBALS(),


This is inconsistent with PyMapping_GetOptionalItem above.
I think it would be better to change the whole instruction to use stack refs, or leave it as it is.

I removed stackrefs from the instruction to leave it as is.

markshannon · 2024-08-19T16:00:54Z

Python/bytecodes.c

+ _PyDict_LoadGlobalStackRef((PyDictObject *)GLOBALS(),
+ (PyDictObject *)BUILTINS(),
+ name,
+ STACK_ENTRY(v));


What does STACK_ENTRY do? Does it have any side effects, or is it just equivalent to &v?

Practically equivalent to &stack_pointer[whatever_v_index_is]. It has no side effects.

markshannon · 2024-08-19T16:05:11Z

Tools/cases_generator/generators_common.py



-def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str) -> None:
+def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, *, allow_unbalanced_parens: bool = False) -> None:


Don't we want balanced parentheses? What is the construct that doesn't balance parentheses?

Specifically it's when we replace something in the middle of the line but still want the rest of the line (and all it's parantheses)

Can you give me an example?

E.g. we replace the before foo(STACK_ENTRY(x));, with foo(stack_pointer[-1]);, we want the replacement function to write until the semicolon ;, ignoring all balanced brackets from the replacement point till that semicolon.

Instead of adding allow_unbalanced_parens can we push the logic into the caller (stack_entry)? Like replace up to the matching ) and then consume/emit tokens up to the end of the statement?

markshannon · 2024-08-20T14:48:19Z

Objects/dictobject.c

+ dk = _Py_atomic_load_ptr(&mp->ma_keys);
+ kind = dk->dk_kind;
+
+ if (kind != DICT_KEYS_GENERAL){


I don't think it is worth changing anything. LOAD_GLOBAL is specialized incredibly well, so this only applies to a rare slow path.

Fidget-Spinner · 2024-08-26T14:42:40Z

@colesbury I think the PR is in better shape now. Can you take a look please?

colesbury

Lets ensure that this actually improves the scalability of LOAD_GLOBAL calls.

colesbury · 2024-08-26T17:12:05Z

Python/bytecodes.c

 PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1);
-PyObject *res_o = _PyEval_LoadGlobal(GLOBALS(), BUILTINS(), name);
- ERROR_IF(res_o == NULL, error);
+_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, STACK_ENTRY(res));


Two suggestions:
Make _PyEval_LoadGlobalStackRef and _PyDict_LoadGlobalStackRef return int error codes
Use &STACK_ENTRY(res) because it makes it more clear that the value is a pointer and because it means the STACK_ENTRY() transformation preserves the type.

I will apply 2. but not 1. The reasoning is to keep consistency with _PyEval_LoadGlobal (non-stackref version).

colesbury · 2024-08-26T17:14:47Z

Tools/cases_generator/generators_common.py



-def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str) -> None:
+def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, *, allow_unbalanced_parens: bool = False) -> None:


Instead of adding allow_unbalanced_parens can we push the logic into the caller (stack_entry)? Like replace up to the matching ) and then consume/emit tokens up to the end of the statement?

colesbury · 2024-08-26T17:16:36Z

Objects/dictobject.c

+ dk = _Py_atomic_load_ptr(&mp->ma_keys);
+ kind = dk->dk_kind;
+
+ if (kind != DICT_KEYS_GENERAL){


This looks backwards to me. The common case should be DICT_KEYS_UNICODE for globals dictionaries, but this only handles DICT_KEYS_GENERAL efficiently.

Objects/dictobject.c

Fidget-Spinner · 2024-08-28T12:31:25Z

Main:

object_cfunction MEGA FAILED: 1.1x slower cmodule_function MEGA FAILED: 1.5x slower generator FAILED: 2.7x faster pymethod MEGA FAILED: 1.6x slower pyfunction MEGA FAILED: 2.3x slower module_function MEGA FAILED: 1.6x slower load_string_const MEGA FAILED: 1.6x slower load_tuple_const PASSED: 3.9x faster create_closure MEGA FAILED: 2.8x slower create_pyobject MEGA FAILED: 2.1x slower

This branch:

object_cfunction MEGA FAILED: 1.0x slower cmodule_function MEGA FAILED: 1.2x slower generator PASSED: 3.1x faster pymethod MEGA FAILED: 1.4x slower pyfunction MEGA FAILED: 2.4x slower module_function MEGA FAILED: 1.2x slower load_string_const MEGA FAILED: 1.6x slower load_tuple_const PASSED: 3.9x faster create_closure MEGA FAILED: 2.6x slower create_pyobject PASSED: 4.2x faster

Create_pyobject became a lot faster!

Fidget-Spinner · 2024-08-30T13:11:47Z

Benchmarking results courtesy of Mike: no slowdown on JIT build https://github.com/faster-cpython/benchmarking-public/tree/main/results/bm-20240828-3.14.0a0-bfd4400-JIT

colesbury

Sorry for the delay in reviewing this. I think we should simplify this further to only focus on the case we care about.

Objects/dictobject.c

colesbury · 2024-09-06T20:34:16Z

Objects/dictobject.c

+ dk = _Py_atomic_load_ptr(&mp->ma_keys);
+ kind = dk->dk_kind;
+
+ if (kind != DICT_KEYS_GENERAL){


I think this is still too complicated. We only need to handle the common case of unicode keys. Globals are almost never split dictionaries.
I think this should look something like the following. You can refactor it further by extracting a _Py_TryXGetStackRef() that's like _Py_TryXGetRef but returns a _PyStackRef.
PyDictKeysObject*dk=_Py_atomic_load_ptr(&mp->ma_keys); if (dk->dk_kind==DICT_KEYS_UNICODE&&PyUnicode_CheckExact(key)){Py_ssize_tix=unicodekeys_lookup_unicode_threadsafe(dk, key, hash); if (ix==DKIX_EMPTY){*value_addr=PyStackRef_NULL; returnix} elseif (ix >= 0){PyObject**addr_of_value=&DK_UNICODE_ENTRIES(dk)[ix].me_value; PyObject*value=_Py_atomic_load_ptr(addr_of_value); if (value==NULL){*value_addr=PyStackRef_NULL; returnDKIX_EMPTY} if (_Py_IsImmortal(value) ||_PyObject_HasDeferredRefcount(value)){*value_addr= (_PyStackRef){.bits= (uintptr_t)value | Py_TAG_DEFERRED }; returnix} if (_Py_TryIncrefCompare(addr_of_value, value)){*value_addr=PyStackRef_FromPyObjectSteal(value); returnix} } } PyObject*obj; Py_ssize_tix=_Py_dict_lookup_threadsafe(mp, key, hash, &obj); if (ix >= 0&&obj!=NULL){*value_addr=PyStackRef_FromPyObjectSteal(obj)} else{*value_addr=PyStackRef_NULL} returnix;

Co-Authored-By: Sam Gross <[email protected]>

colesbury

LGTM

markshannon

Can we go back to defining res as an array of one.

markshannon · 2024-09-11T17:06:30Z

Python/bytecodes.c

 PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1);
-PyObject *res_o = _PyEval_LoadGlobal(GLOBALS(), BUILTINS(), name);
- ERROR_IF(res_o == NULL, error);
+_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &STACK_ENTRY(res));


Suggested change
_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &STACK_ENTRY(res));
_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &res[0]));
Can we go back to the earlier design of making res an array?
Please also add a comment explaining why we res needs to be trated specially (we need pass a pointer to _PyEval_LoadGlobalStackRef).
The code generator already understands arrays, and it avoids confusion as to where it should insert stack spills for fully deferred reference counting.
You'll able to remove def stack_entry below.
Sorry for the back and forth on this.

markshannon · 2024-09-11T17:30:51Z

Tools/cases_generator/generators_common.py

 # unused portions of the stack to NULL.
 stack.flush_single_var(self.out, target, uop.stack.outputs)

+ def stack_entry(


I'm not sure that the stack calculations are correct here, and I don't think it will be able to handle code that writes to multiple results, like UNPACK....
If we go back to using arrays, we can remove this.

bedevere-app · 2024-09-11T17:33:54Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

bedevere-bot · 2024-09-13T16:28:20Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x has failed when building commit 8810e28.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/146/builds/9494) and take a look at the build logs.
Check if the failure is related to this commit (8810e28) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/146/builds/9494

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 14, done. remote: Counting objects: 7% (1/14) remote: Counting objects: 14% (2/14) remote: Counting objects: 21% (3/14) remote: Counting objects: 28% (4/14) remote: Counting objects: 35% (5/14) remote: Counting objects: 42% (6/14) remote: Counting objects: 50% (7/14) remote: Counting objects: 57% (8/14) remote: Counting objects: 64% (9/14) remote: Counting objects: 71% (10/14) remote: Counting objects: 78% (11/14) remote: Counting objects: 85% (12/14) remote: Counting objects: 92% (13/14) remote: Counting objects: 100% (14/14) remote: Counting objects: 100% (14/14), done. remote: Compressing objects: 7% (1/13) remote: Compressing objects: 15% (2/13) remote: Compressing objects: 23% (3/13) remote: Compressing objects: 30% (4/13) remote: Compressing objects: 38% (5/13) remote: Compressing objects: 46% (6/13) remote: Compressing objects: 53% (7/13) remote: Compressing objects: 61% (8/13) remote: Compressing objects: 69% (9/13) remote: Compressing objects: 76% (10/13) remote: Compressing objects: 84% (11/13) remote: Compressing objects: 92% (12/13) remote: Compressing objects: 100% (13/13) remote: Compressing objects: 100% (13/13), done. remote: Total 14 (delta 4), reused 2 (delta 1), pack-reused 0 (from 0)  From https://github.com/python/cpython * branch main -> FETCH_HEAD Note: checking out '8810e286fa48876422d1b230208911decbead294'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b <new-branch-name> HEAD is now at 8810e28... gh-121459: Deferred LOAD_GLOBAL (GH-123128) Switched to and reset branch 'main' Could Not Find D:\buildarea\3.x.bolen-windows10\build\Lib\*.pyc The system cannot find the file specified. Could Not Find D:\buildarea\3.x.bolen-windows10\build\PCbuild\python*.zip Cannot open file 'D:\buildarea\3.x.bolen-windows10\build\test-results.xml' for upload Could Not Find D:\buildarea\3.x.bolen-windows10\build\Lib\*.pyc The system cannot find the file specified. Could Not Find D:\buildarea\3.x.bolen-windows10\build\PCbuild\python*.zip

This reverts commit 8810e28.

Fidget-Spinner added 3 commits August 18, 2024 21:14

deferred globals
803e9f7

fix merge from upstream
b213b6f

use cases generator to generate stack offsets
11d2fb6

Fidget-Spinner requested review from markshannon and methane as code owners August 18, 2024 14:31

bedevere-appbot mentioned this pull request Aug 18, 2024
PEP 703 Deferred reference counts #121459
Closed

bedevere-appbot added the awaiting core review label Aug 18, 2024

Fidget-Spinner requested a review from colesbury August 18, 2024 14:32

Fidget-Spinner added skip news topic-free-threading labels Aug 18, 2024

Fidget-Spinner added 2 commits August 18, 2024 22:46

fix bug
ea8d855

add type annotations
af0c032

picnixz reviewed Aug 18, 2024
View reviewed changes

Eclips4 reviewed Aug 18, 2024
View reviewed changes

Tools/cases_generator/generators_common.py Outdated Show resolvedHide resolved

Fidget-Spinnerand others added 5 commits August 19, 2024 18:14

Update Objects/dictobject.c
1a431de
Co-authored-by: Bénédikt Tran <[email protected]>

Update Objects/dictobject.c
48bebe9
Co-authored-by: Bénédikt Tran <[email protected]>

Update Objects/dictobject.c
6e948b4
Co-authored-by: Bénédikt Tran <[email protected]>

Address review
e23567a

Merge branch 'deferred_globals_final' of github.com:Fidget-Spinner/cp…
dbbebf9
…ython into deferred_globals_final

Fidget-Spinner commented Aug 19, 2024
View reviewed changes

colesbury reviewed Aug 19, 2024
View reviewed changes

markshannon reviewed Aug 20, 2024
View reviewed changes

Fidget-Spinner added 3 commits August 22, 2024 18:21

partially address review
dbdee10

simplify
6d176c0

fix some stuff
4ff9f17

colesbury reviewed Aug 26, 2024
View reviewed changes

Fidget-Spinner added 3 commits August 28, 2024 20:25

specialize for str and split str, move out NULL checks
c644ebf

remove allow_unbalanced_parens
64f932a

Add a & for STACK_ENTRY
bfd4400

colesbury reviewed Sep 6, 2024
View reviewed changes

Address review
0c1ee7e
Co-Authored-By: Sam Gross <[email protected]>

colesbury approved these changes Sep 9, 2024
View reviewed changes

bedevere-appbot added awaiting merge and removed awaiting core review labels Sep 9, 2024

markshannon requested changes Sep 11, 2024
View reviewed changes

bedevere-appbot added awaiting changes and removed awaiting merge labels Sep 11, 2024

Change to use array instead of STACK_ENTRY
ad33dd1

Fidget-Spinner requested a review from markshannon September 13, 2024 12:46

markshannon approved these changes Sep 13, 2024
View reviewed changes

bedevere-appbot added awaiting merge and removed awaiting changes labels Sep 13, 2024

Fidget-Spinner merged commit 8810e28 into python:mainSep 13, 2024

bedevere-appbot removed the awaiting merge label Sep 13, 2024

Fidget-Spinner deleted the deferred_globals_final branch September 13, 2024 16:23

zware added a commit to zware/cpython that referenced this pull request Sep 13, 2024
Revert "pythongh-121459: Deferred LOAD_GLOBAL (pythonGH-123128)"
3563dbd
This reverts commit 8810e28.

colesbury added a commit to colesbury/cpython that referenced this pull request Sep 14, 2024
Revert "pythongh-121459: Deferred LOAD_GLOBAL (pythonGH-123128)"
831d5fa
This reverts commit 8810e28.



		def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str) -> None:
		def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, allow_unbalanced_parens: bool = False) -> None:

	defemit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, allow_unbalanced_parens: bool=False) ->None:
	defemit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, *, allow_unbalanced_parens: bool=False) ->None:

		ix = _Py_dict_lookup(mp, key, hash, &value);
		*value_addr = value == NULL ? PyStackRef_NULL : PyStackRef_FromPyObjectNew(value);

	PyAPI_FUNC(void)_PyDict_LoadGlobalStackRef(PyDictObject, PyDictObject, PyObject, _PyStackRef);
	PyAPI_FUNC(_PyStackRef)_PyDict_LoadGlobalStackRef(PyDictObject, PyDictObject, PyObject*);

	_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &STACK_ENTRY(res));
	_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &res[0]));

Uh oh!

gh-121459: Deferred LOAD_GLOBAL#123128

gh-121459: Deferred LOAD_GLOBAL #123128

Uh oh!

Conversation

Fidget-Spinner commented Aug 18, 2024• edited by bedevere-app botLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

picnixzAug 18, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markshannon left a comment • edited by colesbury Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fidget-Spinner commented Aug 26, 2024

Uh oh!

Fidget-Spinner commented Aug 18, 2024•
edited by bedevere-app bot
Loading

picnixzAug 18, 2024•
edited
Loading

markshannon left a comment •
edited by colesbury
Loading