`UnicodeDecodeError` from juliacall IPython extension

[MilesCranmer]

Affects: JuliaCall

Describe the bug
Various users have reported stray UnicodeDecodeErrors when using PySR (=>juliacall) in a Jupyter session:

• [BUG]: Frequent unicodedecode error MilesCranmer/PySR#1043 (@jobs-git)
• Getting a uft-8 Error MilesCranmer/PySR#945 (@AaronRoseDA @DorofeevKirillDev @GongJr0)

I think all of these users were on Windows. The error looks like this:

Error in callback _flush_stdio (for post_execute), with arguments args (),kwargs{}:
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
File ~/.julia/packages/PythonCall/avYrV/src/JlWrap/any.jl:262, in __call__(self, *args, **kwargs)
 260 return ValueBase.__dir__(self) + self._jl_callmethod($(pyjl_methodnum(pyjlany_dir)))
 261 def __call__(self, *args, **kwargs):
--> [262](~/.julia/packages/PythonCall/avYrV/src/JlWrap/any.jl:262) return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
 263 def __bool__(self):
 264 return True

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 4095: unexpected end of data

The workaround seems to be to set PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION=no before loading. I think it's worth figuring out the root cause of the issue, though.

One quick fix is that we have PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION=auto by default, which would get set to no on Windows, or yes on Linux/macOS. Although the downside is this would make notebooks non-reproducible on different operating systems, so it's probably better to fix the root cause.

cc @cjdoris - any ideas?

[cjdoris]

This is happening in the IPython callback that is called after each cell which flushes stdout and stderr. We redirect these streams so that they are sent to the kernel and so are visible in the notebook (otherwise print doesn't work).

The unexpected end of data makes me think that the redirected stream is being flushed while only part of a UTF-8 character has actually been written to it. Because Python's sys.stdout is a text IO, you write strings to it, and Python strings only accept valid unicode, and so constructing the string to write fails because there is a partial character at the end.

The correct answer is probably to change flush(::PyIO) to not necessarily flush the entire buffer, but to leave any partial character at the end in the buffer until the next flush.