gh-125196: Use PyUnicodeWriter in error handlers

[vstinner]

PyCodec_ReplaceErrors() and PyCodec_BackslashReplaceErrors() now use the public PyUnicodeWriter API.

• Issue: Use the public PyUnicodeWriter API #125196

[rruuaanng]

Wouldn't it be better to move this inline function into the header file? That way it can omit the declaration in the header file and integrate well. It looks like a public API.

[vstinner]

In fact, I implemented #125201 which should replace this static inline function.

[vstinner]

Benchmark:

importcodecsimportpyperfrunner=pyperf.Runner() replace=codecs.lookup_error('replace') backslashreplace=codecs.lookup_error('backslashreplace') forLENin (1, 1000): text="\xff"*LENexc=UnicodeEncodeError("ascii", text, 0, len(text), "reason") runner.bench_func(f"replace encode len={LEN}", replace, exc) text="\xff"*LENexc=UnicodeTranslateError(text, 0, len(text), "reason") runner.bench_func(f"replace translate len={LEN}", replace, exc) data=b"\x20\x80\xff"*LENexc=UnicodeDecodeError("ascii", data, 0, len(data), "reason") runner.bench_func(f"backslashreplace decode len={LEN}", backslashreplace, exc) text="\x20\xff\u20ac\U0010ffff"*LENexc=UnicodeEncodeError("ascii", text, 0, len(text), "reason") runner.bench_func(f"backslashreplace encode len={LEN}", backslashreplace, exc) text="\x20\xff\u20ac\U0010ffff"*LENexc=UnicodeTranslateError(text, 0, len(text), "reason") runner.bench_func(f"backslashreplace translate len={LEN}", backslashreplace, exc)

Results, Python built with gcc -O3, CPU isolation:

+-------------------------------------+---------+------------------------+ | Benchmark | ref | change | +=====================================+=========+========================+ | replace encode len=1 | 92.9 ns | 110 ns: 1.19x slower | +-------------------------------------+---------+------------------------+ | replace translate len=1 | 103 ns | 138 ns: 1.34x slower | +-------------------------------------+---------+------------------------+ | backslashreplace decode len=1 | 95.6 ns | 140 ns: 1.46x slower | +-------------------------------------+---------+------------------------+ | backslashreplace encode len=1 | 120 ns | 166 ns: 1.37x slower | +-------------------------------------+---------+------------------------+ | backslashreplace translate len=1 | 126 ns | 169 ns: 1.35x slower | +-------------------------------------+---------+------------------------+ | replace encode len=1000 | 157 ns | 2.09 us: 13.31x slower | +-------------------------------------+---------+------------------------+ | replace translate len=1000 | 178 ns | 2.23 us: 12.51x slower | +-------------------------------------+---------+------------------------+ | backslashreplace decode len=1000 | 3.03 us | 27.2 us: 8.98x slower | +-------------------------------------+---------+------------------------+ | backslashreplace encode len=1000 | 14.4 us | 46.3 us: 3.22x slower | +-------------------------------------+---------+------------------------+ | backslashreplace translate len=1000 | 14.4 us | 45.9 us: 3.18x slower | +-------------------------------------+---------+------------------------+ | Geometric mean | (ref) | 3.03x slower | +-------------------------------------+---------+------------------------+ 

It's way slower :-(

This PR has a naive PyUnicodeWriter_Fill() implementation calling PyUnicodeWriter_WriteChar() in a loop. I wrote PR gh-125201 which is way more efficient:

+-------------------------------------+---------+------------------------+-----------------------+ | Benchmark | ref | change | fill | +=====================================+=========+========================+=======================+ | replace encode len=1000 | 157 ns | 2.09 us: 13.31x slower | 176 ns: 1.12x slower | +-------------------------------------+---------+------------------------+-----------------------+ | replace translate len=1000 | 178 ns | 2.23 us: 12.51x slower | 276 ns: 1.55x slower | +-------------------------------------+---------+------------------------+-----------------------+ 

[vstinner]

For the benchmark, I had to call directly the error handler functions. Using ASCII doesn't work (it doesn't measure my change), since Objects/unicodeobject.c has a fast-path for most common error handlers such as replace and backslashreplace.

[vstinner]

Let's keep the private API for now, since it's faster.