Skip to content

bpo-1635741: Fix unicode_dealloc() for mortal interned string#21270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vstinner merged 1 commit into from
Jul 3, 2020
Merged

bpo-1635741: Fix unicode_dealloc() for mortal interned string #21270

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions Objects/unicodeobject.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1943,13 +1943,20 @@ unicode_dealloc(PyObject *unicode)
break;

case SSTATE_INTERNED_MORTAL:
/* revive dead object temporarily for DelItem */
Py_SET_REFCNT(unicode, 3);
#ifdef INTERNED_STRINGS
/* Revive the dead object temporarily. PyDict_DelItem() removes two
references (key and value) which were ignored by
PyUnicode_InternInPlace(). Use refcnt=3 rather than refcnt=2
to prevent calling unicode_dealloc() again. Adjust refcnt after
PyDict_DelItem(). */
assert(Py_REFCNT(unicode) == 0);
Py_SET_REFCNT(unicode, 3);
if (PyDict_DelItem(interned, unicode) != 0){
Copy link
Member

@shihai1991shihai1991Jul 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we skip PyDict_DelItem() and set interned unicode's refcnt = 0 directly in here?

Copy link
MemberAuthor

@vstinnervstinnerJul 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cannot remove the PyDict_DelItem() call.

PyUnicode_InternInPlace() uses black magic: it ignores 2 references (key and value) of the interned dictionary.

We must remove the string from the interned dictionary. Otherwise, the dictionary will contain a dangling pointer when unicode_dealloc() completes: using the dict is likely to crash in this case.

Copy link
Member

@shihai1991shihai1991Jul 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it, make sense, thanks.

_PyErr_WriteUnraisableMsg("deletion of interned string failed",
NULL);
}
assert(Py_REFCNT(unicode) == 1);
Py_SET_REFCNT(unicode, 0);
#endif
break;

Expand DownExpand Up@@ -15710,8 +15717,9 @@ PyUnicode_InternInPlace(PyObject **p)
return;
}

/* The two references in interned are not counted by refcnt.
The deallocator will take care of this */
/* The two references in interned dict (key and value) are not counted by
refcnt. unicode_dealloc() and _PyUnicode_ClearInterned() take care of
this. */
Py_SET_REFCNT(s, Py_REFCNT(s) - 2);
_PyUnicode_STATE(s).interned = SSTATE_INTERNED_MORTAL;
#endif
Expand DownExpand Up@@ -15780,6 +15788,8 @@ _PyUnicode_ClearInterned(PyThreadState *tstate)
#endif
break;
case SSTATE_INTERNED_MORTAL:
// Restore the two references (key and value) ignored
// by PyUnicode_InternInPlace().
Py_SET_REFCNT(s, Py_REFCNT(s) + 2);
#ifdef INTERNED_STATS
mortal_size += PyUnicode_GET_LENGTH(s);
Expand Down