UnicodeEncodeError when calling puts function with unicode u'\xff' and the stream.encoding is None

[zhaoguixu]

Hi,
When replacing the variable s to another value u'\xff', this test would fail.

[test-case]

@mock_streams('stdout')
def test_puts_with_encoding_type_none_output():
    """
    puts() should print unicode output without a stream encoding
    """
    s = u"string!"       #<====== replace to u"\xff"
    output.user = True
    sys.stdout.encoding = None
    puts(s, show_prefix=False)
    eq_(sys.stdout.getvalue(), s + "\n")

[traceback]
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-4: ordinal not in range(128)

[comments]
The buggy point locates in the function '_encode' in utils.py, you should not simply cast the msg into str by using str(msg).

def _encode(msg, stream):
    if isinstance(msg, unicode) and hasattr(stream, 'encoding') and not stream.encoding is None:
        return msg.encode(stream.encoding)
    else:
        return str(msg)         #<========== buggy point

[zhaoguixu]

Anyone cares?

[ppkt]

Yup, it seems that changing return str(msg) to return msg solves the problem without breaking tests. But why this cast was necessary at the first place?

[kaaveland]

I'm not absolutely sure it makes sense to change this now. It's not going to absolutely break anyone's code, but it basically lets you puts things which aren't strings and get something sensible out:

>>> class Foo(object):
...   def __str__(self): return "The best Foo in the world"
...
>>> from fabric.api import puts
>>> puts(Foo())
The best Foo in the world

I'm guessing there's code out there that uses this (hopeless to search for this in github, so not going to try to look for examples).

I'm not sure I think it's right that not having an encoding set and using unicode will cause puts to crash us, though.

[bitprophet]

Suspect the least-worst way to address this for now is to simply expand the offending line into this block:

try:
    return str(msg) # Original behavior added for 'reasons'
except UnicodeDecodeError:
    return msg # Best-effort fallback

This way, whatever antediluvian reasons we had for casting-to-string will continue to function, but the buggy case under discussion would fall back to "well, I dunno what this is, but I can't str() it, so I'll just shove it down the pipe as-is".

[dclong]

I encountered a similar issue while running the sudo method, and the exception happened at self.stream.write(text) in io.py. Is there any progress on the unicode issues?