Modifications to support iOS as a cross-compilation target

[freakboy3742]

BPO	23670
Nosy	@doko42, @ronaldoussoren, @ncoghlan, @ned-deily, @bitdancer, @freakboy3742, @rovitotv, @Lukasa, @rkuska, @mpaolini, @moreati, @controversial, @Mariatta
Files	20150314.diff: Patch for Python 3.4.2 source tree 20150426.diff: Updated patch for Python 3.4.2 source tree 20150504.diff: Updated patch with support for running the test suite. 20150628.diff: Updated patch, passing test suite on iOS simulator. 20150704.diff: Update 5 - only failing ctypes-related tests on device. 20151221.diff: Update 6 - single source tree for libffi_ios 20160301-brokentests.diff: Incomplete rebase of Update 6 against Python 3.6dev 20160217.diff: Patch against 3.5.1

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = None
created_at = <Date 2015-03-15.06:25:47.056>
labels = ['type-feature', '3.7', 'build']
title = 'Modifications to support iOS as a cross-compilation target'
updated_at = <Date 2019-09-04.00:08:40.855>
user = 'https://github.com/freakboy3742'

bugs.python.org fields:

activity = <Date 2019-09-04.00:08:40.855>
actor = 'josh.r'
assignee = 'ned.deily'
closed = False
closed_date = None
closer = None
components = ['Build', 'Cross-Build']
creation = <Date 2015-03-15.06:25:47.056>
creator = 'freakboy3742'
dependencies = []
files = ['38493', '39207', '39286', '39824', '39862', '41376', '42059', '42060']
hgrepos = ['300']
issue_num = 23670
keywords = ['patch']
message_count = 28.0
messages = ['238125', '238127', '238129', '238133', '238134', '238138', '238156', '238167', '238190', '238191', '238197', '238211', '240307', '241187', '241189', '242040', '242547', '245903', '246266', '256780', '256797', '261093', '261095', '267030', '267064', '267092', '284441', '305628']
nosy_count = 15.0
nosy_names = ['doko', 'ronaldoussoren', 'ncoghlan', 'ned.deily', 'r.david.murray', 'cvrebert', 'freakboy3742', 'Todd.Rovito', 'bkabrda', 'Lukasa', 'rkuska', 'mpaolini', 'Alex.Willmer', 'Luke Taylor', 'Mariatta']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23670'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[freakboy3742]

Proposal: iOS should be a supported platform for Python development.

The attached patch is a first pass at a patch to achieve this. It is a single patch against Python 3.4.2 sources, and requires no pre- or post-configure modifications.

Supporting iOS requires multiple builds - one for each of the hardware platforms that iOS supports (x86_64 simulator, ARMv7 and ARM64). These separate builds must then be merged into a single "fat" framework. The patch contains an iOS directory with a "meta" Makefile that manages this build-and-merge process. See the README in the iOS directory for details on usage.

The patch also introduces a new 'ios' platform type.

A sample XCode project for an iOS app is also provided as part of the patch.

iOS/README contains a couple of other notes about the build and the approach taken.

There are some known problems/limitations with this patch:

• It's a patch against 3.4.2, not hg trunk

• The code doesn't currently compile for ARMv7. In order to support ARM64, it has been necessary to use an unreleased trunk version of libffi; however, this version is currently broken for ARMv7. Older versions of libffi (including the formal 3.2.1 release) *do* work.

• The patch doesn't currently provide any way to run the test suite on the target platform. Testing is currently based on a simple smoke test of some basic features.

So - the patch isn't ready for commit to trunk. I'm presenting it for the purposes of getting feedback on the broad approach while I continue to resolve the known issues.

[ncoghlan]

Russell, just checking before I change the issue title: do you mean supporting iOS as a cross-compilation target? "Development" is ambiguous here, as you could mean development *of* CPython, rather than *in* Python, and I assume development *for* iOS largely takes place on x86_64 Mac OS X, Windows and Linux systems.

Slavek, Robert - assuming my understanding of the change is correct, I think would be a very interesting CPython enhancement from a Fedora Workstation perspective.

[ned-deily]

As a cross-compilation target. From a first quick look at it, it appears the patch requires a current Mac OS X system to build for iOS; the necessary standard build tools and SDKs for iOS are only available on OS X. These are the same build tools used for OS X builds. We already support universal builds for multiple architectures on OS X directly in one pass; unfortunately, the iOS builds require two different SDKs, one for running on the OS X-based simulator and one for the native platform archs which, I assume, is why Russell has gone for the separate builds for each arch and lipo-ed them together. I don't have an opinion yet about the use of the Setup.local configurations rather than modifying setup.py. I appreciate trying to keep the changes for a patch like this as isolated as possible. But, long term, that might not be the best approach assuming there is eventually agreement to fully support iOS as a standard platform (via cross-compilation). The bulk of the patch is the new version of libffi; presumably that will eventually be released upstream in the standard version of libffi so that having a separate copy wouldn't be required? In any case, as a work-in-progress, the patch is certainly nicely done.

[freakboy3742]

Nick: you are correct - these are changes to support iOS as a cross-compilation target, so you can run your Python code on an iOS device. Apologies for the confusion in nomenclature.

Ned: You're correct that the build needs to be run on an OS/X machine, and needs to be done in multiple passes because of the need for multiple SDKs.

I've taken the approach of using Setup.local for two reasons.

Firstly, iOS requires a "fully embedded" Python, because Apple doesn't approve of the use of dynamic loading in apps. Setup.local was the best way I found to force modules to be statically linked into the main library.

Secondly, because of the nature of the end-deployment, I imagine it's inevitable that serious end-users will want to customise their Python build for specific applications. AppStore submission rules prohibit running code that wasn't shipped with the app (i.e., Apple won't let you do an end-run around their app approval processes), so the set of modules you're actually using will be a known quantity. Since a "full" iOS Python build is 34MB or so, much of that bulk will be for binary support of modules that you aren't using, providing an easily configurable way to prune out unused modules from the built product seemed like a useful facility. There's also the issue of providing support for the standard library modules (like SSL) that have dependencies that can't be easily provided or specified out-of-the-box.

If there's a better/more palatable approach than using Setup.local, I'm happy to look into it.

The libffi code is a direct copy from the current (as of a week or so ago) libffi trunk. There have been some bug fixes applied since 3.2.1 was released in november that are necessary to support ctypes on ARM64 (but, as I flagged in my original comment, introduces bugs for ARMv7). The libffi repo suggests the next version will be 4.0, but I have no idea when that will be finalised. I'm currently trying to get their release plans confirmed.

As with OS/X libffi support, the source code needs to be generated, rather than just "./configure && make"'d. That said, the libffi_darwin code could be re-used, as the x86_64 components are the same for Darwin as the iOS simulator.

[ned-deily]

Ack, much to consider. Adding doko as I believe he has been most closely following developments with libffi releases.

[ned-deily]

A thought, primarily as a note to myself for further investigation: the current OS X framework build support and, to a lesser extent, universal build support have added a fair amount of special-case cruft to the build process, primarily in configure.ac and the main Makefile. Over the years, the build tools have evolved and gotten smarter (and more general) about SDK support and arch support using xcrun, as they were generalized to support iOS vs OS X builds. But, if it is necessary to do separate compile/link steps to support iOS framework builds because of the multiple SDKs, it *might* make sense to integrate those into the main Makefile and to isolate the framework build steps (for both OS X and iOS) to be more of an add-on step. The potential positive impact might be to simplify the current main Makefile and configure.ac by centralizing much of the framework build specific steps scattered throughout the Makefile *and* it could also simplify and speed up the iOS builds by not duplicating the platform-independent build steps. It might even be a useful abstraction for other non-Apple multiple-arch or cross-compilation builds.

Another factor to consider is support for building third-party extension modules standalone as opposed to building as part of the interpreter itself. It would be nice to be able to just use the traditional Distutils-based process for third-party extension modules. For OS X universal builds, we get that essentially for free because of the tool chain support for multiple archs. To do it automatically across multiple SDKs would require adding smarts to Distutils to do multiple compile/links and lipo merges.

[bitdancer]

Also remember that being a supported platform requires buildbots. If I'm reading this right that means two: one for the simulator build, and one that actually runs on the device (once the run-the-test issues is solved). I'm not sure what would need to be done to buildbot to support the scenario of building on one platform and running the tests on another after copying the binary, but I'm sure there must be some way to do it...buildbot is written in Python after all :) And I'm sure that capability will be useful for more than just iOS...with Android we can probably run buildbot on the device eventually, but being able to test a cross compile build there as well will be helpful (once there is one).

[freakboy3742]

Understood that buildbots are required. The subject has come up a couple of times on mobile-sig - however, I haven't got a good answer for exactly what this means in practice. Does build hardware need to be delivered to a specific build farm location, or can it just run in my home office? Does it need to be running 24/7, or just guarantee that at least X builds per week, or Y% of all builds are tested?

In terms of physical specifications - at a minimum, at least two iDevices will be required (an ARMv7 device, like a 5th Gen iPod, and an ARM64 device, like an iPhone 5S), tethered to a Mac that runs the simulator.

In terms of execution - there's definitely ways to remote-control the deployment and execution of apps, so I imagine buildbot integration is in the realm of the possible.

Android will have similar requirements, but Android also has an emulator which would allow testing without the existence of a physical device, which would provide an alternative approach.

[ncoghlan]

Rather than trying to do something ourselves on the mobile testing front, we may want to instead explore the free-for-open-source-project offering from Sauce Labs: https://saucelabs.com/opensauce/

Adapting that to run the CPython self-tests as an "app" might be interesting, but they're PyCon sponsors, so they may be amenable to helping to get that work properly if we need to ask for their help.

Getting that set up would also potentially be a useful tool for ensuring PSF web services are working correctly across multiple browsers and client devices. (I'd actually like to get us to the point of being able to do continuous deployment of those services some day)

[ncoghlan]

(Some work on the Buildbot site would still be required, but it would be triggering a Sauce Labs run rather than managing the mobile hardware directly)

[freakboy3742]

Nick: Finding a way to get on-device test results is next on my TODO list, once I've got the patch up to date for trunk.

FYI - Updating to trunk is currently blocking on bpo-22625 (which was introduced by the fix for bpo-22359).

I had thought about using a mobile testing farm "as a service", but I didn't know Sauce Labs had one. I'll keep that in mind when I get around to working on the test suite.