Add support for IMAP IDLE in `imaplib`

[ShayRojansky]

BPO	11245
Nosy	@warsaw, @ericvsmith, @bitdancer, @vadmium, @mitya57, @soltysh, @ankostis, @jdek
Dependencies	bpo-18921: In imaplib, cached capabilities may be out of date after login
Files	imapidle.patch imapidle.patch: New patch 0001-Add-IDLE-support-to-imaplib.patch: IMAP IDLE patch for imaplib

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-02-18.20:36:10.487>
labels = ['type-feature', 'library', 'expert-email', '3.9']
title = 'Implementation of IMAP IDLE in imaplib?'
updated_at = <Date 2019-12-19.12:58:09.754>
user = 'https://bugs.python.org/ShayRojansky'

bugs.python.org fields:

activity = <Date 2019-12-19.12:58:09.754>
actor = 'jdek'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2011-02-18.20:36:10.487>
creator = 'Shay.Rojansky'
dependencies = ['18921']
files = ['27400', '37555', '48790']
hgrepos = []
issue_num = 11245
keywords = ['patch']
message_count = 24.0
messages = ['128814', '129405', '129407', '171885', '171889', '172365', '172383', '202149', '202342', '202345', '233176', '235972', '236167', '245204', '245246', '245249', '245252', '245253', '245255', '245284', '246235', '246236', '293222', '358678']
nosy_count = 17.0
nosy_names = ['barry', 'pierslauder', 'eric.smith', 'piers', 'r.david.murray', 'Shay.Rojansky', 'martin.panter', 'mitya57', 'maciej.szulik', 'nafur', 'dveeden', 'Malina', 'F.Malina', 'ankostis', 'equaeghe', 'ohreally', 'jdek']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11245'
versions = ['Python 3.9']

Linked PRs

• gh-55454: Add IMAP4 IDLE support to imaplib #122542

[ShayRojansky]

IMAP IDLE support is not implemented in the current imaplib. A "drop-in" replacement called imaplib2 exists (), but uses internally managed threads - a heavy solution that is not always appropriate (e.g. when handling many IMAP accounts an asynchronous approach would be more efficient)

I am about to start implementation of an asynchronous select()-compatible approach, and was wondering if there has been any discussion over IDLE, any specific reasons it hasn't been implemented and if eventual integration into imaplib would be a desirable thing.

Proposed approach:

• Addition of a new state 'IDLE'
• Addition of an idle() method to class IMAP4, which issues the IDLE command to the server and returns immediately. At this point we enter the IDLE state, in which no normal IMAP commands may be issued.
• Users can now select() or poll() the socket as they wish
• A method can be called to retrieve any untagged responses (e.g. EXISTS) that have arrived since entering the IDLE state. The function returns immediately and does not modify the state.
• To end the IDLE state, the user calls a method (done()?) which resumes the previous state.

Would appreciate any sort of feedback...

[terryjreedy]

imaplib has no particular maintainer and I know little about it.
Doc says it implements 'a large subset of the IMAP4rev1 client protocol as defined in RFC 2060." I do not remember any discussion on pydev, over the last several years, about imaplib. I presume just the subset was chosen because of some combination of necessity and feasibility, as judged by the implementors. Hence the complement, the unimplemented subset, would be 'not done' rather than 'not wanted'. If your proposed new feature, an IDLE command, is part of this complement, then I would assume that a patch would, in principle, be acceptable.

I cannot comment on your particular proposal, but I hope the above helps as far as it goes.

[bitdancer]

I just wound up doing a bit of research on this for other reasons. Piers Lauder was the original author of the imaplib module, and he is (as far as I can tell) currently maintaining an imaplib2 module that does support IDLE (but not, I think, python3). But it does IDLE (and other things) via threads, and in the email I found announcing it he didn't think it was suitable for stdlib inclusion (because of the threading). Piers hasn't contributed to core in quite a while as far as I can tell, but he was active in a bug report back in 2008 according to google, so I thought I'd add him to nosy and see if he has time for an opinion.

[nafur]

We have implemented this functionality according to RFC 2177.
We actually implemented a synchronous idle function that blocks until a timeout occurs or the server sent some event.

This is not the most flexible way, however it will provide a basic functionality that enables user to use imap idle based notifications.
Besides, every other solution would require threads or regular polling.

See attached patch file.

[bitdancer]

To fully answer the original question that opened this issue: contributions will be welcomed. While I don't currently have time to work on imaplib myself, I have an interest and will review and if appropriate commit patches.

I like Shay's proposal, but absent a patch along those lines having blocking IMAP support will definitely be an improvement. An application needing to monitor more than one imap connection could do its own threading.

Thanks for proposing the patch. Could you please submit a contributor agreement? I probably won't have time to fully consider the proposed patch for a bit, but I've put it on my todo list.

test_imaplib does have a testing framework now, do you think you could write tests for the new feature?

[nafur]

I got the confirmation for my agreement.

I'm not quite sure about the tests, as I'm not really familiar with the way this is done in cpython. The test_imaplib.py seems to cover all ways to connect to some server, but none of the actual imap commands. The patch only implements another commands (whose behaviour is highly/only dependent of other events on the server).

Hence, I don't see a way to create a meaningfull test case other than just calling the command...

[bitdancer]

Yeah writing a good test case for this is a bit tricky, since we'll need some infrastructure (an Event?) so we can prove that the call has blocked without delaying the test for very long.

I'm not sure when I'll be able to get to this...I'm not going to check it in without a test, so I'll have to find enough time to be able to write the tests.

[nafur]

I stumbled about this issue again and would really like to see it fixed.

I see the possibility to create a test case in combination with the first test sequence which creates a temporary mail. Would it be enough, that we just call IDLE in some folder, create a temporary mail in this folder and check if it returns?

Unfortuantely, I have not been able to write code for such a test case yet, as the whole test routine fails with "[PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections". This is using 3.2.3, but I guess it will not be any different with the current release... (as it is the same with 2.7.3)

[bitdancer]

What do you mean by the whole test routine failing? The test suite is currently passing on the buildbots, so are you speaking of the new test you are trying to write?

[bitdancer]

Hmm. Looking at this again, it appears as though there's no way to interrupt IDLE if you want to, say, send an email. If you are actually using this in code, how are you handling that situation?

[nafur]

So, let's resurrect this one.

For the project that lead to the old patch, we did not need this feature.
However, we now needed are more complete implementation of IDLE.
Hence, we extended this to return after sending idle() and support polling, leaving idle mode or wait until something happens (like before).

[Malina]

IMAP polling hurts, just merge imaplib2 into standard library as imaplib.

Piers Lauder authored imaplib IMAP4 client, part of python standard library, back in December 1997 based on RFC 2060. In 2003 RFC 2060 was made obsolete by RFC 3501 adding important features and Piers released imaplib2 which receives feature updates since.
Last feature updates to the standard library imaplib were before Piers retired from Sydney University a decade ago.

imaplib2 presents an almost identical API as that provided by the standard library imaplib, the main difference being that imaplib2 allows parallel execution of commands on the IMAP4 server, and implements the IDLE extension, so NO POLLING IS REQUIRED. IMAP server will push new mail notifications to the client. Imaplib2 also supports COMPRESS, ID, better timeout handling etc. There is 975 more lines of code all doing useful things a modern IMAP client needs.

imaplib2 can be substituted for imaplib in existing clients with no changes in the code apart from required logout call to shutdown the threads.

Old imaplib was ported to Python 3 with the rest of the standard library. I am working to port imaplib2 to py3, stuck on receiving bytes v strings.

References:

imaplib2 code and docs
http://sourceforge.net/p/imaplib2/code/ci/master/tree/
also http://sydney.edu.au/engineering/it/~piers/python/imaplib2.html

imaplib
https://hg.python.org/cpython/file/3.4/Lib/imaplib.py

Ruby stdlib support for idle (not that it hurts python performance, just my pride)
http://ruby-doc.org/stdlib-2.0.0/libdoc/net/imap/rdoc/Net/IMAP.html#method-i-idle