gh-137242: Add a --no-randomize option, and use it in Android CI

[vstinner]

• Based on gh-137242: Add a --no-randomize option, and use it in Android CI #138303

• Issue: Android CI and binary releases #137242

[vstinner]

cc @mhsmith

[mhsmith]

OK, this way is fine too.

[vstinner]

!buildbot Android

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 754a8c4 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138649%2Fmerge

The command will test the builders whose names match following regular expression: Android

The builders matched are:

• aarch64 Android PR
• AMD64 Android PR

[freakboy3742]

👍 Makes sense to me!

[miss-islington-app]

Thanks @vstinner for the PR, and @freakboy3742 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @vstinner and @freakboy3742, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 01895d233b4370e0c99f16c01a57cfd73c434b28 3.13 

[bedevere-app]

GH-138684 is a backport of this pull request to the 3.14 branch.

[bedevere-app]

GH-138685 is a backport of this pull request to the 3.13 branch.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Android 3.13 (tier-3) has failed when building commit 85649b7.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1590/builds/972) and take a look at the build logs.
4. Check if the failure is related to this commit (85649b7) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1590/builds/972

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 9, done. remote: Counting objects: 20% (1/5) remote: Counting objects: 40% (2/5) remote: Counting objects: 60% (3/5) remote: Counting objects: 80% (4/5) remote: Counting objects: 100% (5/5) remote: Counting objects: 100% (5/5), done. remote: Compressing objects: 20% (1/5) remote: Compressing objects: 40% (2/5) remote: Compressing objects: 60% (3/5) remote: Compressing objects: 80% (4/5) remote: Compressing objects: 100% (5/5) remote: Compressing objects: 100% (5/5), done. remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 4 (from 1)  From https://github.com/python/cpython * branch 3.13 -> FETCH_HEAD Note: switching to '85649b769f0db4992c061c8d17155c07553dda46'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c <new-branch-name> Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 85649b769f0 [3.13] gh-137242: Add a --no-randomize option, and use it in Android CI (GH-138649) (#138685) Switched to and reset branch '3.13' configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 192k 100 192k 0 0 2196k 0 --:--:-- --:--:-- --:--:-- 2196k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 42455 100 42455 0 0 468k 0 --:--:-- --:--:-- --:--:-- 468k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5041k 100 5041k 0 0 26.0M 0 --:--:-- --:--:-- --:--:-- 26.0M % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 1257k 100 1257k 0 0 10.6M 0 --:--:-- --:--:-- --:--:-- 10.6M % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 635k 100 635k 0 0 4814k 0 --:--:-- --:--:-- --:--:-- 4814k ../../configure: line 4013: pkg-config: command not foundconfigure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly. ../../Python/fileutils.c:455:1: warning: unused function 'decode_current_locale' [-Wunused-function] 455 | decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen, |^~~~~~~~~~~~~~~~~~~~~ ../../Python/fileutils.c:674:1: warning: unused function 'encode_current_locale' [-Wunused-function] 674 | encode_current_locale(const wchar_t *text, char **str, |^~~~~~~~~~~~~~~~~~~~~ 2 warnings generated. ../../Modules/_testexternalinspection.c:538:5: warning: code will never be executed [-Wunreachable-code] 538 | int pid; |^~~~~~~~ 1 warning generated. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 8784 100 8784 0 0 242k 0 --:--:-- --:--:-- --:--:-- 245k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 2894 100 2894 0 0 73375 0 --:--:-- --:--:-- --:--:-- 74205 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 43504 100 43504 0 0 1133k 0 --:--:-- --:--:-- --:--:-- 1148k

[vstinner]

@freakboy3742: I'm not sure what's going on, aarch64 Android 3.13 buildbot failed with:

C/C++: ld.lld: error: /Users/android/buildarea/3.13.mhsmith-android-aarch64/build/cross-build/aarch64-linux-android/prefix/lib/libpython3.13.so: unknown file type

[mhsmith]

This happened before in #137186 (comment), but I wasn't sure of the cause. I'll look into it.

[mhsmith]

@freakboy3742: Like the last time this problem occurred, the buildbot server had about 12 GB of free space. The biggest offender was /Users/buildbot/Library/Logs/CoreSimulator:

-rw-r--r-- 1 buildbot staff 30042226464 Jun 17 13:32 CoreSimulator.prev.log -rw-r--r-- 1 buildbot staff 14734765773 Sep 9 13:20 CoreSimulator.log -rw-r--r-- 1 buildbot staff 1100845 Feb 10 2025 Simulator.log 

I deleted CoreSimulator.prev.log (30 GB), but I didn't restart the Android worker as I did last time, so this should be a better test of whether disk space is indeed the cause.

[freakboy3742]

I deleted CoreSimulator.prev.log (30 GB), but I didn't restart the Android worker as I did last time, so this should be a better test of whether disk space is indeed the cause.

Sounds like a cron job to rotate/clean up those logs (or better still, direct them to the build directory) might be called for. I'll investigate what can be done.

[mhsmith]

I deleted CoreSimulator.prev.log (30 GB), but I didn't restart the Android worker as I did last time, so this should be a better test of whether disk space is indeed the cause.

For future reference: this apparently did resolve the problem. It's plausible that problems could start happening with 12 GB still available, because the iOS tests use a large amount of temporary space while running. But I still don't know why it would lead to this "unknown file type" error.