Add initial jq-based templating engine#687
Merged
Uh oh!
There was an error while loading. Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
MemberAuthor
tianon commented Jan 27, 2022
Sorry, this is a little bit annoying to review because it's one of our oldest |
MemberAuthor
tianon commented Jan 28, 2022
A potentially useful diff of old `update.sh` to new `versions.sh`:--- /dev/fd/63 2022-01-27 16:57:28.962111563 -0800+++ /dev/fd/62 2022-01-27 16:57:28.962111563 -0800@@ -2,29 +2,7 @@ set -Eeuo pipefail shopt -s nullglob -# https://www.python.org/downloads/ (under "OpenPGP Public Keys")-declare -A gpgKeys=(- # gpg: key AA65421D: public key "Ned Deily (Python release signing key) <[email protected]>" imported- [3.7]='0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D'- # https://www.python.org/dev/peps/pep-0537/#release-manager-and-crew-- # gpg: key B26995E310250568: public key "\xc5\x81ukasz Langa (GPG langa.pl) <[email protected]>" imported- [3.8]='E3FF2839C048B25C084DEBE9B26995E310250568'- # https://www.python.org/dev/peps/pep-0569/#release-manager-and-crew-- # gpg: key B26995E310250568: public key "\xc5\x81ukasz Langa (GPG langa.pl) <[email protected]>" imported- [3.9]='E3FF2839C048B25C084DEBE9B26995E310250568'- # https://www.python.org/dev/peps/pep-0596/#release-manager-and-crew-- # gpg: key 64E628F8D684696D: public key "Pablo Galindo Salgado <[email protected]>" imported- [3.10]='A035C8C19219BA821ECEA86B64E628F8D684696D'- # https://www.python.org/dev/peps/pep-0619/#release-manager-and-crew-- # gpg: key 64E628F8D684696D: public key "Pablo Galindo Salgado <[email protected]>" imported- [3.11]='A035C8C19219BA821ECEA86B64E628F8D684696D'- # https://www.python.org/dev/peps/pep-0664/#release-manager-and-crew-)-+# TODO https://github.com/docker-library/python/pull/686 # https://github.com/docker-library/python/issues/365 # https://pypi.org/project/pip/#history declare -A pipVersions=( @@ -50,6 +28,9 @@ versions=( "$@" ) if [ ${#versions[@]} -eq 0 ]; then versions=( */ ) + json='{}'+else+ json="$(< versions.json)" fi versions=( "${versions[@]%/}" ) @@ -59,19 +40,9 @@ getPipCommit="$(curl -fsSL 'https://github.com/pypa/get-pip/commits/main/public/get-pip.py.atom' | tac|tac | awk -F '[[:space:]]*[<>/]+' '$2 == "id" && $3 ~ /Commit/{print $4; exit }')" getPipUrl="https://github.com/pypa/get-pip/raw/$getPipCommit/public/get-pip.py" getPipSha256="$(curl -fsSL "$getPipUrl" | sha256sum | cut -d' ' -f1)" +export getPipUrl getPipSha256-generated_warning(){- cat <<-EOH- #- # NOTE: THIS DOCKERFILE IS GENERATED VIA "update.sh"- #- # PLEASE DO NOT EDIT IT DIRECTLY.- #-- EOH-}--is_good_version(){+has_linux_version(){ local dir="$1"; shift local dirVersion="$1"; shift local fullVersion="$1"; shift @@ -80,7 +51,15 @@ return 1 fi - if [ -d "$dir/windows" ] && ! wget -q -O /dev/null -o /dev/null --spider "https://www.python.org/ftp/python/$dirVersion/python-$fullVersion-amd64.exe"; then+ return 0+}++has_windows_version(){+ local dir="$1"; shift+ local dirVersion="$1"; shift+ local fullVersion="$1"; shift++ if ! wget -q -O /dev/null -o /dev/null --spider "https://www.python.org/ftp/python/$dirVersion/python-$fullVersion-amd64.exe"; then return 1 fi @@ -89,6 +68,8 @@ for version in "${versions[@]}"; do rcVersion="${version%-rc}" + export version rcVersion+ rcGrepV='-v' if [ "$rcVersion" != "$version" ]; then rcGrepV= @@ -110,13 +91,17 @@ } | sort -ruV ) ) fullVersion= + hasWindows= declare -A impossible=() for possible in "${possibles[@]}"; do rcPossible="${possible%%[a-z]*}" # varnish is great until it isn't (usually the directory listing we scrape below is updated/uncached significantly later than the release being available) - if is_good_version "$version" "$rcPossible" "$possible"; then+ if has_linux_version "$version" "$rcPossible" "$possible"; then fullVersion="$possible" + if has_windows_version "$version" "$rcPossible" "$possible"; then+ hasWindows=1+ fi break fi @@ -133,8 +118,11 @@ || true ) ) for possibleVersion in "${possibleVersions[@]}"; do - if is_good_version "$version" "$rcPossible" "$possibleVersion"; then+ if has_linux_version "$version" "$rcPossible" "$possibleVersion"; then fullVersion="$possibleVersion" + if has_windows_version "$version" "$rcPossible" "$possible"; then+ hasWindows=1+ fi break fi done @@ -176,69 +164,38 @@ ' )" - echo "$version: $fullVersion (pip $pipVersion, setuptools $setuptoolsVersion)"+ echo "$version: $fullVersion (pip $pipVersion, setuptools $setuptoolsVersion${hasWindows:+, windows})"- for v in \- alpine{3.15,3.14} \-{buster,bullseye}{/slim,} \- windows/windowsservercore-{ltsc2022,1809} \- ; do- dir="$version/$v"- variant="$(basename "$v")"-- [ -d "$dir" ] || continue-- case "$variant" in- slim) template="$variant"; tag="$(basename "$(dirname "$dir")")" ;- windowsservercore-*) template='windowsservercore'; tag="${variant#*-}" ;- alpine*) template='alpine'; tag="${variant#alpine}" ;- *) template='debian'; tag="$variant" ;- esac- if [ "$variant" = 'slim' ]; then- # use "debian:*-slim" variants for "python:*-slim" variants- tag+='-slim'- fi- template="Dockerfile-${template}.template"--{generated_warning; cat "$template"} > "$dir/Dockerfile"-- sed -ri \- -e 's/^(ENV GPG_KEY) .*/\1 '"${gpgKeys[$version]:-${gpgKeys[$rcVersion]}}"'/' \- -e 's/^(ENV PYTHON_VERSION) .*/\1 '"$fullVersion"'/' \- -e 's/^(ENV PYTHON_RELEASE) .*/\1 '"${fullVersion%%[a-z]*}"'/' \- -e 's/^(ENV PYTHON_PIP_VERSION) .*/\1 '"$pipVersion"'/' \- -e 's/^(ENV PYTHON_SETUPTOOLS_VERSION) .*/\1 '"$setuptoolsVersion"'/' \- -e 's!^(ENV PYTHON_GET_PIP_URL) .*!\1 '"$getPipUrl"'!' \- -e 's!^(ENV PYTHON_GET_PIP_SHA256) .*!\1 '"$getPipSha256"'!' \- -e 's/^(FROM python):.*/\1:'"$version-$tag"'/' \- -e 's!^(FROM (debian|buildpack-deps|alpine|mcr[.]microsoft[.]com/[^:]+)):.*!\1:'"$tag"'!' \- "$dir/Dockerfile"-- major="${rcVersion%%.*}"- minor="${rcVersion#$major.}"- minor="${minor%%.*}"-- if [ "$minor" -ge 8 ]; then- # PROFILE_TASK has a reasonable default starting in 3.8+; see:- # https://bugs.python.org/issue36044- # https://github.com/python/cpython/pull/14702- # https://github.com/python/cpython/pull/14910- perl -0 -i -p -e "s![^\n]+PROFILE_TASK(='[^']+?')?[^\n]+\n!!gs" "$dir/Dockerfile"- fi- if [ "$minor" -ge 9 ]; then- # "wininst-*.exe" is not installed for Unix platforms on Python 3.9+: https://github.com/python/cpython/pull/14511- sed -ri -e '/wininst/d' "$dir/Dockerfile"- fi-- # https://www.python.org/dev/peps/pep-0615/- # https://mail.python.org/archives/list/[email protected]/thread/PYXET7BHSETUJHSLFREM5TDZZXDTDTLY/- if [ "$minor" -lt 9 ]; then- sed -ri -e '/tzdata/d' "$dir/Dockerfile"- fi-- if [ "$minor" -lt 10 ]; then- # <3.10 does not have -fno-semantic-interposition enabled and --with-lto does nothing for performance- sed -ri -e '/with-lto/d' "$dir/Dockerfile"- fi- done+ export fullVersion pipVersion setuptoolsVersion hasWindows+ json="$(jq <<<"$json" -c '+ .[env.version] ={+ version: env.fullVersion,+ pip:{+ version: env.pipVersion,+ url: env.getPipUrl,+ sha256: env.getPipSha256,+ },+ setuptools:{+ version: env.setuptoolsVersion,+ },+ variants: [+ (+ "bullseye",+ "buster"+ | ., "slim-" + .), # https://github.com/docker-library/ruby/pull/142#issuecomment-320012893+ (+ "3.15",+ "3.14"+ | "alpine" + .),+ if env.hasWindows != "" then+ (+ "ltsc2022",+ "1809"+ | "windows/windowsservercore-" + .)+ else empty end+ ],+ }+ ')" done ++jq <<<"$json" -S . > versions.json |
docker-library-bot added a commit to docker-library-bot/official-images that referenced this pull request Jan 28, 2022
Changes: - docker-library/python@d27f3d2: Merge pull request docker-library/python#687 from infosiftr/jq-template - docker-library/python@c484e1b: Add initial jq-based templating engine
docker-library-bot added a commit to docker-library-bot/official-images that referenced this pull request Jan 28, 2022
Changes: - docker-library/python@2c9debd: Merge pull request docker-library/python#686 from vikahl/dynamically-load-setuptools-pip - docker-library/python@bc88926: Dynamically load Setuptools and Pip versions from ensurepip - docker-library/python@d27f3d2: Merge pull request docker-library/python#687 from infosiftr/jq-template - docker-library/python@c484e1b: Add initial jq-based templating engine
pelson reviewed Feb 2, 2022
| dst="$(echo "$src" | tr -d 3)" \ | ||
| [ -s "/usr/local/bin/$src" ]; \ | ||
| [ ! -e "/usr/local/bin/$dst" ]; \ | ||
| ln -svT "/usr/local/bin/$src" "/usr/local/bin/$dst" \ |
Contributor
There was a problem hiding this comment.
Would it be possible for these to remain relative links?
Python treats relative links specially:
$ AFTER=python@sha256:6c5fd370ff72da06c7c55a0b292812f230342607d257da564e605d54931a4177 $ BEFORE=python@sha256:520eed245023b4059571e9828d06720bb3f53b4200ce372dad715b136ebf8456 $ CMD='ls -ltr /usr/local/bin/python; ln -s /usr/local/ /tmp/foo; /tmp/foo/bin/python -c "import sys; print(sys.base_prefix)"' $ docker run -it $BEFORE sh -c "${CMD}" lrwxrwxrwx 1 root root 7 Jan 26 22:04 /usr/local/bin/python -> python3 /tmp/foo $ docker run -it $AFTER sh -c "${CMD}" lrwxrwxrwx 1 root root 22 Jan 29 05:59 /usr/local/bin/python -> /usr/local/bin/python3 /usr/local MemberAuthor
There was a problem hiding this comment.
I'm definitely willing to go back to a relative link (I don't think it makes much functional difference either way), but I would love to understand more about what the actual use case/breakage is?
(Your example shows a difference, but it's contrived enough that it's hard to understand how/why it might break in a real program. 😅)
Contributor
There was a problem hiding this comment.
Yes, it is totally contrived 😂
The more complete situation is:
- I provide a bunch of python distributions to colleagues on a shared filesystem, and provide one of them as a "pro" version (the one they should ideally use).
- Because of the network architecture, there is a single common location on disk that people can access, but this is just a symlink to machine specific mount-points (I'm not clear on the precise reason for the mount-point changes, could be due to the highly segmented network architecture which provides strong network segregation)
- When somebody uses the "pro" version to create a virtual environment, I want the "base" python to come from the fixed version, not the floating "pro" version (so that when pro changes, their virtual environments continue to work as before)
- To achieve this, I provide a tool that (automatically) converts the floating version to a fixed version. I do this by looking at the base prefix, and resolving it if it is a symlink, but I have to be careful not to recursively resolve the symlink as it will make the virtual environment not portable due to the different network mount points. I use the fact that the link is relative to prevent filesystem jumps.
As you say, my example was contrived to simplify the story above. As you can imagine, the implementation is a bit delicate, and so I setup some CI testing of it. The tests stopped passing when the image was released with these changes 😉
MemberAuthor
There was a problem hiding this comment.
😬
That's all extremely clever (kudos!), but also not exactly a supported use case for this image. 🙈
I would suggest you fix the symlinks to be what you expect/need after you extract them from the image, but frankly you're going to have to / get to keep all the pieces there. 😅
(We could fix this one specific instance of breakage, but I cannot possibly promise we won't break your setup again in the future. 😇)
Contributor
There was a problem hiding this comment.
This is totally on me, no question. I understand how this works 😜.
Irrespective of how I use/abuse the images, the use of relative symlinks is the more consistent approach with how CPython and venv work out of the box, IMO. I propose it because it will result in mildly better images (and I obviously benefit 😄).
This was referenced Feb 5, 2022
Omar290601 pushed a commit to Omar290601/python that referenced this pull request Aug 27, 2025
Since docker-library#687, the regular Debian images no longer ship DWARF symbols, which heavily limits the ability to use gdb and lldb for debugging. Stripping debugging symbols makes perfect sense to reduce the size of Docker images, but when this optimization was first introduced in docker-library#483, the idea was to restrict its use to the slim and Alpine images. That was (unintentionally?) changed in docker-library#483, and now debugging symbols are also stripped in the regular Debian images. Undo that change.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See also docker-library/php#1052 (and linked PRs).
Diff: