Always use unittest for collecting tests in regrtests

[serhiy-storchaka]

BPO	45229
Nosy	@terryjreedy, @ambv, @ambv, @zware, @serhiy-storchaka, @pablogsal, @miss-islington
PRs	bpo-45229: Remove test_main in many tests #28405 bpo-45229: Fix setUpModule in test_ssl #28454 [3.10] bpo-45229: Remove test_main in many tests (GH-28405) #28455 [3.9] bpo-45229: Remove test_main in many tests (GH-28405) #28456 bpo-45229: Make pickle tests discoverable #28467 bpo-45229: Use doctest.DocTestSuite instead of run_doctest #28468 [3.10] bpo-45229: Make pickle tests discoverable (GH-28467) #28478 [3.9] bpo-45229: Make pickle tests discoverable (GH-28467) #28479 bpo-45229: Make datetime tests discoverable #28615 bpo-45229: Make tkinter tests discoverable #28637 [3.10] bpo-45229: Make datetime tests discoverable (GH-28615) #28639 [3.9] bpo-45229: Make datetime tests discoverable (GH-28615). #28645 bpo-45229: Revert "Make datetime tests discoverable" #28650 [3.9] Revert "[3.9] bpo-45229: Make datetime tests discoverable (GH-28615) (GH-28645)" #28660 [3.9] Revert "[3.9] Revert "[3.9] bpo-45229: Make datetime tests discoverable (GH-28615) (GH-28645)"" #28666 Revert "bpo-45229: Revert "Make datetime tests discoverable"" #28667 [3.10] bpo-45229: Make datetime tests discoverable (GH-28615) #28668 [3.10] bpo-45229: Make tkinter tests discoverable (GH-28637) #28927 [3.9] bpo-45229: Make tkinter tests discoverable (GH-28637) #28929 bpo-45229: Make doctest tests discoverable #28986 bpo-45229: Make test_http_cookiejar discoverable #29004 [3.10] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) #29006 [3.9] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) #29007 [3.10] bpo-45229: Make doctest tests discoverable (GH-28986) #29095 [3.9] bpo-45229: Make doctest tests discoverable (GH-28986) #29096
Dependencies	bpo-40058: Running test_datetime twice fails with: module 'datetime' has no attribute '_divide_and_round' bpo-40173: test.support.import_fresh_module fails to correctly block submodules when fresh is specified bpo-45185: test.test_ssl.TestEnumerations is not run bpo-45187: Some tests in test_socket are not run bpo-45500: Rewrite test_dbm bpo-45502: Fix test_shelve and make it discoverable

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2021-09-17.07:31:12.126>
labels = ['type-feature', 'tests', '3.9', '3.10', '3.11']
title = 'Always use unittest for collecting tests in regrtests'
updated_at = <Date 2021-10-20.16:52:45.847>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-10-20.16:52:45.847>
actor = 'lukasz.langa'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2021-09-17.07:31:12.126>
creator = 'serhiy.storchaka'
dependencies = ['40058', '40173', '45185', '45187', '45500', '45502']
files = []
hgrepos = []
issue_num = 45229
keywords = ['patch']
message_count = 33.0
messages = ['402006', '402009', '402053', '402076', '402154', '402156', '402168', '402195', '402214', '402228', '402245', '402254', '402255', '402903', '402945', '402952', '402953', '402956', '402959', '402974', '402982', '402996', '402999', '403000', '403846', '403853', '403855', '404137', '404142', '404143', '404466', '404495', '404496']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'lukasz.langa', 'lukasz.langa', 'zach.ware', 'serhiy.storchaka', 'pablogsal', 'miss-islington', 'jerrylikerice1s']
pr_nums = ['28405', '28454', '28455', '28456', '28467', '28468', '28478', '28479', '28615', '28637', '28639', '28645', '28650', '28660', '28666', '28667', '28668', '28927', '28929', '28986', '29004', '29006', '29007', '29095', '29096']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45229'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-89392: Make test_decimal discoverable #106209
• [3.11] gh-89392: Make test_decimal discoverable (GH-106209) #106229
• [3.12] gh-89392: Make test_decimal discoverable (GH-106209) #106230
• bpo-45229: Make ElementTree tests discoverable #108859
• gh-89392: Remove test_main() in test_netrc #108860
• gh-89392: Make test_pep646_syntax discoverable #108861
• [3.11] gh-89392: Remove test_main() in test_netrc (GH-108860) #108868
• [3.11] gh-89392: Make test_pep646_syntax discoverable (GH-108861) #108869
• [3.12] gh-89392: Make test_pep646_syntax discoverable (GH-108861) #108871
• [3.12] gh-89392: Remove test_main() in test_netrc (GH-108860) #108872
• [3.12] bpo-45229: Make ElementTree tests discoverable (GH-108859) #108873
• [3.11] bpo-45229: Make ElementTree tests discoverable (GH-108859) #108874
• gh-89392: Fix running test_pep646_syntax as script #108875
• gh-89392: Remove support of test_main() in libregrtest #108876
• [3.11] gh-89392: Fix running test_pep646_syntax as script (GH-108875) #108877
• [3.12] gh-89392: Remove support of test_main() in libregrtest (GH-108876) #108897
• [3.11] gh-89392: Remove support of test_main() in libregrtest (GH-108876) #108898
• gh-89392: Use normal unittest runner in test_type_cache #108911
• [3.12] gh-89392: Use normal unittest runner in test_type_cache (GH-108911) #108913
• [3.11] gh-89392: Use normal unittest runner in test_type_cache (GH-108911) #108914
• gh-89392: Use unittest test runner for doctests in test_getopt #108916
• [3.12] gh-89392: Use unittest test runner for doctests in test_getopt (GH-108916) #108919
• [3.11] gh-89392: Use unittest test runner for doctests in test_getopt (GH-108916) #108920

[serhiy-storchaka]

Currently regrtest supports two ways of collecting and running tests in module.

1. If the module contains the "test_main" function, regrtest just calls it. This function usually calls run_unittest() with a list of test classes, composed manually, it can also run doctests with run_doctest(), generate list of test classes dynamically, and run some code before and after running tests.

2. Otherwise regrtest uses unittest for for collecting tests.

The disadvantage of the former way is that new test classes should be added manually to the list. If you forget to do this, new tests will not be run. See for example bpo-45185 and bpo-45187. Not runned tests can hide bugs.

So it would be better to eliminate human factor and detect tests automatically.

[serhiy-storchaka]

PR 28405 contains simple changes for 60 test files. It does not cover files which use complex code for generating list of test classes and running doctests (because there are subtle differences between running doctests with run_doctest() and via DocTestSuite). They are left for the following PRs.

[zware]

See also bpo-16748