Skip to content

gh-112536: Set up TSAN CI for free-threading#116555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
corona10 wants to merge 19 commits into from
Closed

gh-112536: Set up TSAN CI for free-threading #116555

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
bd5e85a
gh-112536: Set up TSAN CI for free-threading
corona10 Mar 10, 2024
a7a3b08
Reduce test scope
corona10 Mar 10, 2024
a7f4826
fix
corona10 Mar 10, 2024
84bccfb
Add _Py_NO_SANITIZE_THREAD annotation
corona10 Mar 10, 2024
aa15f16
nit
corona10 Mar 10, 2024
0f72690
nit
corona10 Mar 10, 2024
2ce5cdd
fix
corona10 Mar 10, 2024
05a5882
swtich to clang
corona10 Mar 10, 2024
0b68814
Update
corona10 Mar 10, 2024
d2a4476
Update env
corona10 Mar 10, 2024
f641cad
fix
corona10 Mar 10, 2024
dd88d4a
fix
corona10 Mar 10, 2024
6c96e41
fix
corona10 Mar 10, 2024
179bacd
Update(?)
corona10 Mar 10, 2024
691d0e4
fix
corona10 Mar 10, 2024
d89f932
fix(?)
corona10 Mar 10, 2024
d833618
Check order
corona10 Mar 10, 2024
33317e7
Weired fix
corona10 Mar 10, 2024
cade95e
Revert to the gcc
corona10 Mar 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -472,6 +472,29 @@ jobs:
- name: Tests
run: xvfb-run make test

build_tsan_free_threading:
name: 'Thread sanitizer (free-threading)'
runs-on: ubuntu-20.04
timeout-minutes: 60
needs: check_source
if: needs.check_source.outputs.run_tests == 'true'
steps:
- uses: actions/checkout@v4
- name: Install Dependencies
run: sudo ./.github/workflows/posix-deps-apt.sh
- name: Set up GCC-10 for ASAN
uses: egor-tensin/setup-gcc@v1
with:
version: 11
Comment on lines +485 to +488
Copy link
Contributor

@colesburycolesburyMar 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version says "11" but the step name says "GCC-10"

- name: Configure CPython
run: ./configure --disable-gil --with-thread-sanitizer
Copy link
Contributor

@colesburycolesburyMar 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want --with-pydebug?

- name: Build CPython
run: make -j4
- name: Display build info
run: make pythoninfo
- name: Tests
run: ./python -m test --pgo -j4 # Reduce test scope
Copy link
Contributor

@colesburycolesburyMar 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll want a configuration specifically for tsan. Maybe a --tsan option? We'll only want to run tests that actually use threading.

Brett had a list of tests that fail with PYTHON_GIL=0 -- those are probably a good starting point:

swtaarrs@1fe9165#diff-590c02490d066378045dd48d62e86dc85d3fb4ad934b6a2b2b0b1112e59eaefb


# CIFuzz job based on https://google.github.io/oss-fuzz/getting-started/continuous-integration/
cifuzz:
name: CIFuzz
Expand DownExpand Up@@ -561,6 +584,7 @@ jobs:
build_windows,
build_windows_free_threading,
build_asan,
build_tsan_free_threading,
'
|| ''
}}
Expand Down
1 change: 1 addition & 0 deletions Objects/obmalloc.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -446,6 +446,7 @@ _PyMem_ArenaFree(void *Py_UNUSED(ctx), void *ptr,
/***************************/

static int
_Py_NO_SANITIZE_THREAD
Copy link
Contributor

@colesburycolesburyMar 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a better experience using suppression lists rather than marking functions as _Py_NO_SANITIZE_THREAD:

  1. Add a file supressions.txt somwhere (maybe Tools/tsan/supressions.txt?)
  2. Set the environment variable TSAN_OPTIONS="suppressions=<path_to_supressions.txt>"

See https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions

set_default_allocator_unlocked(PyMemAllocatorDomain domain, int debug,
PyMemAllocatorEx *old_alloc)
{
Expand Down
7 changes: 7 additions & 0 deletions configure
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions configure.ac
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -3264,6 +3264,12 @@ AC_MSG_RESULT([no])
with_tsan="no"
])

if test "$with_tsan" = "yes"
then
AC_DEFINE([MI_TSAN], [1],
[Define MI_TSAN for thread sanitizer])
fi

# Set info about shared libraries.
AC_SUBST([SHLIB_SUFFIX])
AC_SUBST([LDSHARED])
Expand Down
3 changes: 3 additions & 0 deletions pyconfig.h.in
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1588,6 +1588,9 @@
<sysmacros.h>. */
#undef MAJOR_IN_SYSMACROS

/* Define MI_TSAN for thread sanitizer */
#undef MI_TSAN

/* Define if mvwdelch in curses.h is an expression. */
#undef MVWDELCH_IS_EXPRESSION

Expand Down