gh-140189: Temporarily allow CI failures for iOS#142365
Uh oh!
There was an error while loading. Please reload this page.
Conversation
savannahostrowski requested review from AA-Turner, ezio-melotti and hugovk as code owners
savannahostrowski commented Dec 6, 2025
I know the convo over Discord was specifically about Android but I've also been seeing this for iOS as well, so I've included it. |
savannahostrowski changed the title 
It looks like the Android issue was fixed by #142289, as there have been no Android failures on the main and 3.x branches since that was merged. So I don't think that needs to be an allowed failure now. On the same branches, iOS has failed about 5 times in the last 2 days because of missing simulator images on the runners. @freakboy3742: did you have any possible solution to this? |
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Hugo van Kemenade <[email protected]>
hugovk changed the title 
freakboy3742 commented Dec 7, 2025
The only solution I'm aware of at this point is to switch back to the macOS-14 runner; AFAIK the macOS-14 image isn't subject to the problem that is causing issues for macOS-15 and macOS-26 runners. The downside is that tests would be run on iOS 17, which is a little old at this point - that isn't ideal. Is anyone aware of a way to get notifications when these failures occur? Or even search recent history for failures? I'm trying to get a sense of how frequent these issues are, if only to report to GitHub as part of the litany of macOS-15 image issues they've been working on. The thing that is confusing is that we don't appear to see this mode of failure with BeeWare tests (or cibuildwheel tests either, to the best of my knowledge)... |
freakboy3742 commented Dec 7, 2025
Based on a manual search of failures of the Tests task on main, there's been 4 iOS test failures since Dec 29 (the last 100 CI runs), excluding 2 runs that were mass failures across multiple other platforms. 3 of those failures were in the last 24 hours. They're all failing for the same reason - a disk image that is meant to have multiple simulators installed is reporting having no simulators installed. I do wonder if the increased frequency might be related to rolling out a new macOS-15 image version. Either way, I've posted an update on the GitHub Actions issue tracking the problem. |
I'm pretty sure we've had more than 100 runs in ~11 months :)
Here's a script that wraps the Details#!/usr/bin/env python3"""Check recent CPython build.yml workflow runs for iOS build failures.uv run check-ios-failures.py"""# /// script# requires-python = ">=3.10"# dependencies = ["prettytable", "rich"]# ///importargparseimportdatetimeasdtimportjsonimportshleximportsubprocessfromprettytableimportPrettyTable, TableStylefromrich.progressimporttrackdefosc8_link(url: str, text: str) ->str: returnf"\033]8;{url}\033\\{text}\033]8;\033\\"defrun_gh(cmd: str) ->str: result=subprocess.run( ["gh", *shlex.split(cmd)], capture_output=True, text=True, check=True, ) returnresult.stdoutdefget_recent_runs(repo: str, days_back: float) ->list[dict]: cutoff=dt.datetime.now(dt.timezone.utc) -dt.timedelta(days=days_back) cutoff_str=cutoff.strftime("%Y-%m-%dT%H:%M:%SZ") limit=int(150*days_back) output=run_gh( f"run list --repo {repo} --workflow build.yml "f"--limit {limit} --json databaseId,conclusion,createdAt,displayTitle" ) runs=json.loads(output) iflen(runs) >=limit: print(f"Warning: fetched {limit} runs, results may be incomplete") return [rforrinrunsifr["createdAt"] >=cutoff_str] defget_job_failures(repo: str, run_id: int) ->tuple[bool, bool]: """Return (ios_failed, other_failed)."""output=run_gh(f"run view {run_id} --repo {repo} --json jobs") data=json.loads(output) ios_failed=Falseother_failed=Falseforjobindata.get("jobs", []): name=job.get("name", "") ifname=="All required checks pass": continueifjob.get("conclusion") =="failure": if"ios"inname.lower(): ios_failed=Trueelse: other_failed=Truereturnios_failed, other_failedclassFormatter( argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter, ): passdefmain() ->None: parser=argparse.ArgumentParser( description="Check recent CPython build.yml runs for iOS build failures", formatter_class=Formatter, ) parser.add_argument( "-d", "--days", type=float, default=2, help="number of days to look back" ) parser.add_argument( "-r", "--repo", default="python/cpython", help="GitHub repository" ) parser.add_argument( "-m", "--markdown", action="store_true", help="output tables in markdown format" ) args=parser.parse_args() style=TableStyle.MARKDOWNifargs.markdownelseTableStyle.SINGLE_BORDERprint(f"Fetching build.yml runs from the last {args.days} days...") runs=get_recent_runs(args.repo, args.days) total=len(runs) failures= [rforrinrunsifr["conclusion"] =="failure"] num_failures=len(failures) print(f"\nTotal runs: {total}") print(f"Failures: {num_failures}") ios_failure_runs= [] forrunintrack(failures, description="Checking failed runs..."): run_id=run["databaseId"] ios_failed, other_failed=get_job_failures(args.repo, run_id) ifios_failed: full_title=run["displayTitle"] title=full_title[:30] +"…"iflen(full_title) >30elsefull_titleios_failure_runs.append((run_id, run["createdAt"], title, other_failed)) ios_only=sum(1for*_, otherinios_failure_runsifnotother) ios_plus_other=sum(1for*_, otherinios_failure_runsifother) # Summary tabletable=PrettyTable() table.set_style(style) table.field_names= ["Metric", "Count"] table.align["Metric"] ="l"table.align["Count"] ="r"table.add_row(["Total runs", total]) table.add_row(["Failed runs", num_failures]) table.add_row(["iOS + other failures", ios_plus_other]) table.add_row(["iOS only failures", ios_only]) print(f"\nSUMMARY (last {args.days} days)") print(table) ifios_failure_runs: table=PrettyTable() table.set_style(style) table.field_names= ["Run ID", "Title", "Created", "Other failures"] table.align["Title"] ="l"table.align["Other failures"] ="l"forrun_id, created_at, title, other_failedinios_failure_runs: url=f"https://github.com/{args.repo}/actions/runs/{run_id}"link=f"[{run_id}]({url})"ifargs.markdownelseosc8_link(url, str(run_id)) table.add_row([link, title, created_at, "yes"ifother_failedelse"iOS only"]) print("\nRuns with iOS failures:") print(table) if__name__=="__main__": main()Here's 7 days' results:
Runs with iOS failures:
|
colesbury merged commit 9d39c02 into python:mainUh oh!
There was an error while loading. Please reload this page.
freakboy3742 commented Dec 9, 2025
🤦 Sorry - November 29.
Thanks for that. The "iOS + other" failures have a lot of "real" problems; but it looks like it shakes out to a 2-4% failure rate. On that basis, it feels like it might be worth trying the macos-14 runner fix to see if that makes things better. |
hugovk changed the title This reverts commit 9d39c02.
5 participants
Both Android and iOS tests are quite flaky right now. Based on the convo in Discord, it seems like allowing failures is the best option.