smtplib doesn't handle unicode passwords

[david]

BPO	29750
Nosy	@warsaw, @taleinat, @giampaolo, @bitdancer, @Windsooon, @JustAnother1, @P403n1x87, @VadimPushtaev
PRs	bpo-29750: support non-ASCII passwords in smtplib #8938 bpo-29750: support non-ASCII passwords in smtplib #15064

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-03-07.20:40:24.455>
labels = ['3.8', 'type-feature', 'library', 'expert-email']
title = "smtplib doesn't handle unicode passwords"
updated_at = <Date 2019-12-26.23:46:13.786>
user = 'https://bugs.python.org/david'

bugs.python.org fields:

activity = <Date 2019-12-26.23:46:13.786>
actor = 'seblu'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2017-03-07.20:40:24.455>
creator = 'david__'
dependencies = []
files = []
hgrepos = []
issue_num = 29750
keywords = ['patch']
message_count = 30.0
messages = ['289184', '289185', '289186', '319222', '319497', '319499', '319514', '319515', '319521', '319522', '319827', '319828', '319830', '319862', '319863', '319864', '319866', '319891', '321172', '321174', '322431', '322447', '322450', '322457', '322468', '324480', '348831', '348832', '348837', '358892']
nosy_count = 10.0
nosy_names = ['barry', 'taleinat', 'giampaolo.rodola', 'r.david.murray', 'seblu', 'david__', 'Windson Yang', 'JustAnother1', 'Gabriele Tornetta', 'Vadim Pushtaev']
pr_nums = ['8938', '15064']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29750'
versions = ['Python 3.8']

Linked PRs

• gh-73936: Support unicode passwords in smtplib #103611
• gh-73936: Add hashlib.saslprep (RFC 4013) and use it in smtplib for Unicode credentials #148692

[david]

Trying to use unicode passwords on smtplib fails miserably on python3.
My particular issue arises on line 643 of said library:

(code, resp) = self.docmd(encode_base64(password.encode('ascii'), eol=''))

which obviously dies when trying to handle unicode chars.

[david]

I'm sorry I rushed my comment. Same thing happens on line 604

return encode_base64(s.encode('ascii'), eol='')

changing both from 'ascii' to 'utf-8' works for me.

[bitdancer]

See msg253287. Someone should check the RFC. It is not obvious that just encoding using utf8 is correct; fundamentally passwords are binary data. But the auth methods don't currently accept binary data. UTF8 is a reasonable default these days, I think, but if we support more than ascii I think we need to support binary, with utf8 as the default encoding.

[bitdancer]

Note: it is definitely the case, regardless of what the RFC says, that binary passwords need to be supported. utf-8 should probably be used as the default encoding for string passwords, rather than ascii. See also bpo-33741.

[taleinat]

It would be extremely helpful to have some test cases that actually work for users but fail with smtplib. So far we have no actual examples, likely due to these being passwords.

Note: it is definitely the case, regardless of what the RFC says, that binary passwords need to be supported.

I'm not sure what you mean by "binary". Do you mean 8-bit characters, a.k.a. bytes?

utf-8 should probably be used as the default encoding for string passwords, rather than ascii.

It is also possible that the appropriate encoding here is "latin1" a.k.a. ISO-8859-1 encoding. This specifically includes many specialized versions of latin characters, e.g. those with German umlauts as mentioned in the duplicate issue bpo-33741. And it could even be the very common Windows-1252 encoding: "It is probably the most-used 8-bit character encoding in the world." (Wikipedia)

[david]

In my case I was doing tests with "contraseña" which is (spanish for password) and it failed

On June 14, 2018 8:36:30 AM GMT+02:00, Tal Einat <report@bugs.python.org> wrote:

Tal Einat <taleinat@gmail.com> added the comment:

It would be extremely helpful to have some test cases that actually
work for users but fail with smtplib. So far we have no actual
examples, likely due to these being passwords.

> Note: it is definitely the case, regardless of what the RFC says,
that binary passwords need to be supported.

I'm not sure what you mean by "binary". Do you mean 8-bit characters,
a.k.a. bytes?

> utf-8 should probably be used as the default encoding for string
passwords, rather than ascii.

It is also possible that the appropriate encoding here is "latin1"
a.k.a. ISO-8859-1 encoding. This specifically includes many
specialized versions of latin characters, e.g. those with German
umlauts as mentioned in the duplicate issue bpo-33741. And it could even
be the very common Windows-1252 encoding: "It is probably the most-used
8-bit character encoding in the world." (Wikipedia)

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue29750\>

---

[bitdancer]

While you are correct that latin1 may be common in this situation, I think it may still be better to have utf-8 be the default, since that is the (still emerging? :) standard. However, you are correct to call for examples: if in the *majority* of the real-world cases it turns out latin1 is what is used, then we could default to that (or not have a default, but instead document our observations).

I don't know how we accumulate enough information to make that decision, though. Maybe we could look at what other mail programs do? Thunderbird, etc? David, which mail program(s) did you use that were able to successfully send that password?

And yes, by binary passwords I mean that the module needs to support being passed a bytes-like object as the password, since clearly there are servers "in the wild" that support non-ascii passwords and the only way to be sure one can send the server the correct password is by treating it as a series of bytes. The library caller will have to be responsible for picking the correct encoding based on local knowledge.

[david]

Both thunderbird, sogo (web) and gmail (web).

On June 14, 2018 3:54:31 PM GMT+02:00, "R. David Murray" <report@bugs.python.org> wrote:

R. David Murray <rdmurray@bitdance.com> added the comment:

While you are correct that latin1 may be common in this situation, I
think it may still be better to have utf-8 be the default, since that
is the (still emerging? :) standard. However, you are correct to call
for examples: if in the *majority* of the real-world cases it turns out
latin1 is what is used, then we could default to that (or not have a
default, but instead document our observations).

I don't know how we accumulate enough information to make that
decision, though. Maybe we could look at what other mail programs do?
Thunderbird, etc? David, which mail program(s) did you use that were
able to successfully send that password?

And yes, by binary passwords I mean that the module needs to support
being passed a bytes-like object as the password, since clearly there
are servers "in the wild" that support non-ascii passwords and the only
way to be sure one can send the server the correct password is by
treating it as a series of bytes. The library caller will have to be
responsible for picking the correct encoding based on local knowledge.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue29750\>

---

[bitdancer]

For the web cases I presume you also set the password using the web interface, so that doesn't really tell us anything useful. Did you use thunderbird to access the mailbox that you set up via gmail and/or sogo? That would make what thunderbird does the interesting question.