Fix usage of proxy.py in test_proxy_functional

[bdraco]

This test was running even if tls in tls was supported without the need to monkey patch because the failures were actually problems with usage of proxy.py.

The test would fail because of problems with usage of proxy.py, and not because tls in tls didn't work. This one turned out to have a lot of side trips to discover why things failed because proxy.py doesn't expect to run inside pytest, and is not easy to get error reporting out of when run this way.

• On all systems: https://github.com/abhinavsingh/proxy.py/blob/30574fd0414005dfa8792a6e797023e862bdcf43/proxy/common/utils.py#L226 would raise on warning causing the proxy to die off because of DeprecationWarning: ssl.OP_NO_SSL*/ssl.OP_NO_TLS* options are deprecated
• On windows --threadless does not work with proxy.py [Threadless] OSError: [WinError 10038] an operation was attempted on something that is not a socket abhinavsingh/proxy.py#492

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (25ef450) 96.98% compared to head (c44c7d8) 97.40%.

Additional details and impacted files

@@ Coverage Diff @@## master #7773 +/- ## ========================================== + Coverage 96.98% 97.40% +0.41%  ========================================== Files 107 107 Lines 32213 32207 -6 Branches 3744 3743 -1 ========================================== + Hits 31243 31371 +128 + Misses 756 632 -124 + Partials 214 204 -10 

Flag	Coverage Δ
CI-GHA	97.32% <100.00%> (+0.39%)	⬆️
OS-Linux	96.99% <100.00%> (+0.06%)	⬆️
OS-Windows	95.48% <100.00%> (?)
OS-macOS	96.61% <100.00%> (?)
Py-3.10.11	95.40% <100.00%> (?)
Py-3.10.13	96.79% <100.00%> (+0.02%)	⬆️
Py-3.11.6	96.50% <100.00%> (-0.01%)	⬇️
Py-3.12.0	96.57% <100.00%> (-0.01%)	⬇️
Py-3.8.10	95.37% <100.00%> (?)
Py-3.8.18	96.61% <100.00%> (-0.16%)	⬇️
Py-3.9.13	95.37% <100.00%> (?)
Py-3.9.18	96.75% <100.00%> (-0.03%)	⬇️
Py-pypy7.3.13	96.22% <100.00%> (?)
VM-macos	96.61% <100.00%> (?)
VM-ubuntu	96.99% <100.00%> (+0.06%)	⬆️
VM-windows	95.48% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Dreamsorcerer]

Huh, I thought it was meant to be supported on all platforms. @webknjaz Happy to just have this test skipped?

[bdraco]

I think its only supported on all platforms after python/cpython#31275

[Dreamsorcerer]

I think its only supported on all platforms after python/cpython#31275

That's a PR that appears to have been merged into 3.11+?

[webknjaz]

Can it be that the warning mentioned @ #6044 (comment) is a result of a flawed logic there? I think I saw a problem in related code, but never got to fixing it.. Or is this about some other problem?

[bdraco]

I think thats a different issue. AFAICT the proxy isn't ssl so its not tls in tls #6044 (comment)

The check might need to be if runtime_has_start_tls and proxy_req.is_ssl(): instead?

aiohttp/aiohttp/connector.py

Line 1248 in 30b7a4e

ifruntime_has_start_tls:

[bdraco]

I think its only supported on all platforms after python/cpython#31275

That's a PR that appears to have been merged into 3.11+?

The way I read this is the test shouldn't run if that flag is set because tls in tls is supported on the version of python the test is running on, but before 3.11 monkey patching was needed to enable it

[Dreamsorcerer]

I think its only supported on all platforms after python/cpython#31275

That's a PR that appears to have been merged into 3.11+?

The way I read this is the test shouldn't run if that flag is set because tls in tls is supported on the version of python the test is running on, but before 3.11 monkey patching was needed to enable it

Right, I hadn't really looked at the test. Doesn't that mean the test should just be skipped on Python 3.11+? And if so, why does it pass on Ubuntu with 3.11+? I'm a bit confused why the behaviour is different on different platforms.

[bdraco]

I think its only supported on all platforms after python/cpython#31275

That's a PR that appears to have been merged into 3.11+?

The way I read this is the test shouldn't run if that flag is set because tls in tls is supported on the version of python the test is running on, but before 3.11 monkey patching was needed to enable it

Right, I hadn't really looked at the test. Doesn't that mean the test should just be skipped on Python 3.11+?

I think that's the case.

And if so, why does it pass on Ubuntu with 3.11+? I'm a bit confused why the behaviour is different on different platforms.

I would have expected it to be failing on Ubuntu 3.11. Maybe it's OpenSSL version dependent as well.

I think I'm going to have to build a Ubuntu setup to be sure

[bdraco]

Marking as draft until I can put together a Ubuntu env for local testing

[bdraco]

Unfortunately I'm getting different behavior on the ubuntu docker container. It might be openssl version dependent or the fact I'm running in a docker host.

Instead I adjusted the other failing test to run if we think tls in tls is supported so we have:

test_secure_https_proxy_absolute_path running if we think tls in tls is supported
test_https_proxy_unsupported_tls_in_tls running if we think tls in tls is unsupported

[bdraco]

If test_secure_https_proxy_absolute_path is failing on ubuntu w/3.11+ in the CI, I'll probably have to do some trial and error debugging here since it means there is something specific about the github runner that causes it to fail that I haven't identified yet

[Dreamsorcerer]

If test_secure_https_proxy_absolute_path is failing on ubuntu w/3.11+ in the CI

No problem. @webknjaz had a good tool for that, a github action that allows you to login to the server or something. I don't remember what it was called, but a quick search suggests it might be one of these 2: https://github.com/marketplace/actions/a-debugger-for-actionshttps://github.com/marketplace/actions/debug-via-ssh

[bdraco]

saved fix_test_https_proxy_unsupported_tls_in_tls_macos_subproc and trying to go back to the in process but with --threaded to see if it behaves better

[bdraco]

Looks like this is good to go

The only one still failing reliably (and not related to this PR) is tests/test_web_server.py::test_unsupported_upgrade[pyloop] - TimeoutError on windows and mac only

[Dreamsorcerer]

Great, one step closer.

[patchback]

Backport to 3.9: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 4d9fc63 on top of patchback/backports/3.9/4d9fc636dbad45678330f17b7d82b75cf91247bf/pr-7773

Backporting merged PR #7773 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream $ git checkout -b patchback/backports/3.9/4d9fc636dbad45678330f17b7d82b75cf91247bf/pr-7773 upstream/3.9

4. Now, cherry-pick PR Fix usage of proxy.py in test_proxy_functional #7773 contents into that branch:

   $ git cherry-pick -x 4d9fc636dbad45678330f17b7d82b75cf91247bf

   If it'll yell at you with something like fatal: Commit 4d9fc636dbad45678330f17b7d82b75cf91247bf is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 4d9fc636dbad45678330f17b7d82b75cf91247bf

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Fix usage of proxy.py in test_proxy_functional #7773 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.9/4d9fc636dbad45678330f17b7d82b75cf91247bf/pr-7773

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[patchback]

Backport to 3.10: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 4d9fc63 on top of patchback/backports/3.10/4d9fc636dbad45678330f17b7d82b75cf91247bf/pr-7773

Backporting merged PR #7773 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream $ git checkout -b patchback/backports/3.10/4d9fc636dbad45678330f17b7d82b75cf91247bf/pr-7773 upstream/3.10

4. Now, cherry-pick PR Fix usage of proxy.py in test_proxy_functional #7773 contents into that branch:

   $ git cherry-pick -x 4d9fc636dbad45678330f17b7d82b75cf91247bf

   If it'll yell at you with something like fatal: Commit 4d9fc636dbad45678330f17b7d82b75cf91247bf is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 4d9fc636dbad45678330f17b7d82b75cf91247bf

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Fix usage of proxy.py in test_proxy_functional #7773 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.10/4d9fc636dbad45678330f17b7d82b75cf91247bf/pr-7773

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[bdraco]

I'll get the patch backs done after breakfast and coffee. Up early due to jet lag

[webknjaz]

If test_secure_https_proxy_absolute_path is failing on ubuntu w/3.11+ in the CI

No problem. @webknjaz had a good tool for that, a github action that allows you to login to the server or something. I don't remember what it was called, but a quick search suggests it might be one of these 2: github.com/marketplace/actions/a-debugger-for-actionsgithub.com/marketplace/actions/debug-via-ssh

@Dreamsorcerer it's this one https://github.com/marketplace/actions/debugging-with-tmate. I recommend using the options to restrict the user who can connect with it.

[webknjaz]

@bdraco thanks for figuring this out! I still remember how exhausting it was to integrate this in the first place…

[bdraco]

Cheers! This one was a bit of a rabbit hole. I can only imagine how it was to integrate in the first place 😉

[webknjaz]

@bdraco so I just realized that I accidentally downgraded the proxy.py version at the beginning of August via these PRs:

• https://github.com/aio-libs/aiohttp/pull/7460/files#diff-7269b1b148c1b82edce5234e9cefabaaa80c1f2f6599db2d007856d9582b1e92L6
• https://github.com/aio-libs/aiohttp/pull/7460/files#diff-7269b1b148c1b82edce5234e9cefabaaa80c1f2f6599db2d007856d9582b1e92L6

We should probably recover allowing pre-releases there to get the latest fixes early, as that project doesn't release often, despite my extensive contributions to their automation.

[bdraco]

I haven't used pip-tools before so I'm not 100% sure if this is the right way to go about it, but #7927 gives it a shot

[webknjaz]

@bdraco that's good enough. I don't think we have the exact invocation documented + we don't normally run it manually — dependabot uses pip-tools to upgrade pairs of .in+.txt files. But I do recommend you get familiar with the tool — it's super useful. Eventually, I'll probably take it to the next level in this repo, lockfile-wise.