bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh#13454

paulmon · 2019-05-20T23:09:16Z

When running the buildbot for Windows ARM32 test.pythoninfo must be run remotely.
Also seperating deploy step from both pythoninfo and test.bat

Requires buildmaster config change to work:
python/buildmaster-config#91

@zooba @zware

https://bugs.python.org/issue36511

paulmon · 2019-05-22T01:29:37Z

The debug part of the test pass should pass locally with these fixes, except for test_datetime.
See this PR for test_datetime: #13073
I am re-running test.bat now with these fixes, which takes more than an hour.
The last pass took 4 hours and 23 minutes on my secondary ARM test device, but on the buildbot ARM test device it's been taking about an hour and a half.
I still need to try the WindowsArm32ReleaseBuild steps, but expect the same results.

zooba

Would prefer a tighter skip on the tests, and deferring to @zware on the buildbot scripts (as with the build config change).

zooba · 2019-05-22T23:28:15Z

Lib/test/test_ssl.py

 ctx.load_verify_locations(cadata=b"broken")


+@unittest.skipIf(Py_DEBUG, "Crashes on debug python builds")


This is certainly not a general issue - can we scope it down to the platform(s) where it crashes?

I will scope it down.
I think it's limited to UCRT. The problem is that SSL has FILE* from the ucrtbase.dll table and python has a FILE* from the ucrtbased.dll table.
I think we could limit it to win32+arm or platform.win32_is_iot( ). Unless there's a way to check if python is using UCRT? I couldn't find one.

Oh, it's because of mixing FILE* across debug/release builds? In that case we scope it down to Windows and enhance the skip message to say "Avoid mixing debug/release CRT on Windows"

paulmon · 2019-05-24T00:23:54Z

Enabling is_python_build( ) to return true on the test target uncovered more failing tests.
There is still one failure in test_pyexpat that I will be investigating next.

====================================================================== FAIL: test_exception (test.test_pyexpat.HandlerExceptionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\py\lib\test\test_pyexpat.py", line 453, in test_exception parser.Parse(b"<a><b><c/></b></a>", 1) RuntimeError: a During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\py\lib\test\test_pyexpat.py", line 469, in test_exception self.assertIn('call_with_frame("StartElement"', entries[1][3]) AssertionError: 'call_with_frame("StartElement"' not found in '' ----------------------------------------------------------------------

paulmon · 2019-05-24T19:33:08Z

I simplified the deployment script to copy entire directories instead of picking files, except for PCBuild and Modules which have many files that aren't needed for the test run. This should make the script more resilient to test changes.

I think the Windows ARM32 test pass is green now when I run the scripts manually, if #13073 is included.

paulmon · 2019-05-24T19:33:47Z

Lib/test/test_pyexpat.py

 self.check_traceback_entry(entries[2],
 "test_pyexpat.py", "StartElementHandler")
-ifsysconfig.is_python_build():
+ifsysconfig.is_python_build()andnot (sys.platform=='win32'andplatform.machine() =='ARM'):


I'd rather fix this than skip it, but I haven't figured out where to start looking yet.

Lib/test/test_ssl.py

Lib/test/test_math.py

paulmon · 2019-06-13T17:23:24Z

@tiran can you please review?
Some of these changes are required to enable the Windows arm32 buildbot to start working.
Thanks!

zooba · 2019-06-19T17:29:29Z

@paulmon I'm happy to merge without Christian's review once the updates I suggested in my last one are done.

Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

paulmon · 2019-06-19T18:17:29Z

@zooba I merged your suggested changes. I think it's ready

zooba · 2019-06-19T18:22:05Z

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.

paulmon · 2019-06-19T18:33:42Z

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.

I'm not sure I'm following you. Are suggesting adding a check for arm/arm64 to Py_DEBUG_WIN32? That would be incorrect for Windows IoT Core x86/x64 which also use UCRT for everything.

zooba · 2019-06-19T18:44:25Z

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.
I'm not sure I'm following you. Are suggesting adding a check for arm/arm64 to Py_DEBUG_WIN32? That would be incorrect for Windows IoT Core x86/x64 which also use UCRT for everything.

No, I'm saying elsewhere you are referring to Py_DEBUG_Win32 which is now undefined. They need to be changed to Py_DEBUG_WIN32 as well

paulmon · 2019-06-19T18:56:18Z

No, I'm saying elsewhere you are referring to Py_DEBUG_Win32 which is now undefined. They need to be changed to Py_DEBUG_WIN32 as well

Thanks. I noticed the whitespace change but somehow didn't see the uppercase change.

paulmon · 2019-06-19T19:27:28Z

@zooba Thanks for the help
I fixed test_ssl and test_math so they pass on my dev machine.
All of the PR checks pass now.

miss-islington · 2019-06-19T20:09:57Z

Thanks @paulmon for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

…bot (pythonGH-13454) (cherry picked from commit f355069) Co-authored-by: Paul Monson <paulmon@users.noreply.github.com>

bedevere-bot · 2019-06-19T20:10:11Z

GH-14244 is a backport of this pull request to the 3.8 branch.

zooba · 2019-06-19T20:32:59Z

@paulmon Seems like it still fails: https://buildbot.python.org/all/#/builders/197/builds/338

paulmon · 2019-06-19T20:59:26Z

@paulmon Seems like it still fails: https://buildbot.python.org/all/#/builders/197/builds/338

The raspberry pi was in a bad state. I have restarted it

paulmon · 2019-06-19T21:09:45Z

Also, ssh.exe and scp.exe somehow don't exist when running under the buildbot, which is breaking the script.

zooba · 2019-06-19T21:17:44Z

@paulmon Different PATH setting? Or perhaps it's running with different file redirection settings (as a 32-bit app, maybe)?

paulmon · 2019-06-19T22:09:16Z

@paulmon Different PATH setting? Or perhaps it's running with different file redirection settings (as a 32-bit app, maybe)?

The PATH is echoed at the top of the log and includes the PATH to ssh.
I hadn't thought of WoW as a possible cause. I installed from the link on python.org so I got the 32-bit version of python. I'll update to the 64-bit python and see if that helps. There isn't a SysWOW64 version of ssh.exe, so that could be it.

* master: (599 commits) Docs: Improved phrasing (pythonGH-14069) Remove redundant if check from optional argument function in argparse. (pythonGH-8766) bpo-37289: Add a test for if with ifexpr in the peephole optimiser to detect regressions (pythonGH-14127) Update What's New in Python 3.9 (pythonGH-14253) bpo-36511: Improve ARM32 buildbot scripts (pythonGH-14251) bpo-37151: remove _PyCFunction_FastCallDict (pythonGH-14269) Fix typo, 'widger' -> 'widget', in idlelib/tree.py (pythonGH-14263) Fix bpo number in News file. (pythonGH-14260) bpo-37342: Fix the incorrect nb_index's type in typeobj documentation (pythonGH-14241) Update What's New in Python 3.8 (pythonGH-14239) bpo-36710: Use tstate in pylifecycle.c (pythonGH-14249) Add missing single quote in io.TextIOWrapper.reconfigure documentation (pythonGH-14246) bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 buildbot (pythonGH-13454) bpo-37333: Ensure IncludeTkinter has a value (pythonGH-14240) bpo-37331: Clarify format of socket handler messages in the documentation. (pythonGH-14234) bpo-37258: Not a bug, but added a unit test and updated documentation. (pythonGH-14229) bpo-36710: Remove PyImport_Cleanup() function (pythonGH-14221) Fix name of '\0'. (pythonGH-14222) bpo-36710: Add tstate parameter in import.c (pythonGH-14218) Document typing.ForwardRef (pythonGH-14216) ...

…bot (pythonGH-13454)

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh
1d33fe6

the-knights-who-say-ni added the CLA signed label May 20, 2019

bedevere-bot added the awaiting review label May 20, 2019

paulmon mentioned this pull request May 20, 2019
bpo-36511: Fix Windows arm32 buildbot, pythoninfo, scp, and ssh python/buildmaster-config#91
Merged

Paul Monson added 2 commits May 21, 2019 12:07

add +d to match test.bat
3aadf52

100% test pass when manually running test.bat
7f17d29

fix file destination
9aaa2c4

zooba reviewed May 22, 2019
View reviewed changes

Paul Monson added 6 commits May 23, 2019 11:22

only skip ssl tests on Windows IoT
3f38d18

Avoid mixing debug/release CRT on Windows
4e147de

test for win32
9819647

propagate exit code
365701c

fix 2 more exit 0 calls
69c40d7

fixes for test_regrtest
086a1b9

Paul Monson added 5 commits May 24, 2019 09:35

merge upstream/master
042e0e9

add tools/parser for test_tools
e230795

copy entire Tools directory
1a1f12d

skip check on Windows ARM32
29601d8

restore 64-bit check in test_regrtest
786194d

paulmon commented May 24, 2019
View reviewed changes

zooba added skip news needs backport to 3.8 labels Jun 11, 2019

merge upstream/master
cd2d5c5

paulmon requested a review from tiran as a code owner June 11, 2019 23:08

zooba reviewed Jun 12, 2019
View reviewed changes

Lib/test/test_ssl.py Outdated Show resolvedHide resolved
Lib/test/test_math.py Outdated Show resolvedHide resolved

paulmonand others added 2 commits June 19, 2019 11:16

Update Lib/test/test_math.py
f5c917a
Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

Update Lib/test/test_ssl.py
e2fb14b
Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

Merge remote-tracking branch 'upstream/master' into arm32_pythoninfo
ce3dfbc

Paul Monson added 2 commits June 19, 2019 11:45

fix mismatched paren
5c355ab

fix Py_DEBUG_WIN32 spelling
f1ce1bf

zooba merged commit f355069 into python:masterJun 19, 2019

bedevere-bot removed the awaiting review label Jun 19, 2019

bedevere-bot removed the needs backport to 3.8 label Jun 19, 2019

paulmon mentioned this pull request Jun 19, 2019
bpo-36511 add arm32 buildbot diagnostic messages #14251
Merged

paulmon deleted the arm32_pythoninfo branch July 17, 2019 20:30

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 build…
cd062ef
…bot (pythonGH-13454)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 build…
a739c78
…bot (pythonGH-13454)

paulmonmannequin mentioned this pull request May 20, 2022
Add Windows ARM32 buildbot #80692
Closed

		ctx.load_verify_locations(cadata=b"broken")


		@unittest.skipIf(Py_DEBUG, "Crashes on debug python builds")

Uh oh!

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh#13454

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

Uh oh!

Conversation

paulmon commented May 20, 2019• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paulmon commented May 22, 2019

Uh oh!

zooba left a comment

Choose a reason for hiding this comment

Uh oh!

zoobaMay 22, 2019

Choose a reason for hiding this comment

Uh oh!

paulmonMay 22, 2019

Choose a reason for hiding this comment

Uh oh!

zoobaMay 23, 2019

Choose a reason for hiding this comment

Uh oh!

paulmonMay 23, 2019

Choose a reason for hiding this comment

Uh oh!

paulmon commented May 24, 2019

Uh oh!

paulmon commented May 24, 2019

Uh oh!

paulmonMay 24, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

paulmon commented Jun 13, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

miss-islington commented Jun 19, 2019

Uh oh!

bedevere-bot commented Jun 19, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

paulmon commented May 20, 2019•
edited by bedevere-bot
Loading

paulmon commented Jun 19, 2019•
edited
Loading