gh-121650: Encode newlines in headers, and verify headers are sound

[encukou]

Re: #121812

Hello @basbloemsaat,

I've spent the day reading through the email module, and RFCs, and I believe I found a better place to fix the issue.
This involved lots of experimentation, so I'm sending an alternative PR rather than a review on yours.

• The generator (writer) verifies that the representation of each header is sound (a parser won't treat it as multiple headers, start-of-body, or part of another header). That should cover custom fold() implementations or Header subclasses.

  • However, some user out there is probably misusing such header injection in working code, so, I added a policy attribute to turn it back.

• Newlines are encoded in fold(), just like undecodable bytes and other special characters.

Overall, this means that we treat newlines as valid content of headers, but “escape” them when such a header is serialized to text.

This PR is a proof of concept. It needs tests and documentation, but I'm out of time for today, and I wanted to share what I have.

Does this look reasonable to you?

• Issue: email.policy.default - gotcha with re-using parsed headers with embedded newlines #121650

[basbloemsaat]

That sounds entirely reasonable, and conforms to the RFCs.

Two points:

1. what would be the use of keeping this check in email.policy.header_store_parse ?
2. I found one case that is not covered by this (contrived, I admit):

fromemailimportmessage_from_stringemail_in="""Subject: foo <bar>\nBCC: [email protected]To: [email protected]From: External Sender <[email protected]>message body"""msg=message_from_string(email_in) print(msg)

[encukou]

what would be the use of keeping this check in email.policy.header_store_parse ?

This is in the branch that handles strings (rather than custom Header object). I'm not clears what kind of format that string is supposed to be in.
Keeping the check means that if someone relied on this, they'll get the same error as before. Also, the indication that something is wrong will come earlier.

I found one case that is not covered by this (contrived, I admit)

That \n there is Python syntax, by the time it gets to message_from_string it's the same as a “real” newline.
If you use a raw string r""", or read from a file, the header remains on a single line.

[encukou]

@serhiy-storchaka, would you like to review this?
@warsaw, @bitdancer, @maxking, as email experts, do you have any comments?

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit af41733 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[serhiy-storchaka]

LGTM.

[bedevere-app]

GH-122611 is a backport of this pull request to the 3.8 branch.