Add tools for generating mappings_XX.h

[corona10]

BPO	40328
Nosy	@hyeshik, @vstinner, @methane, @serhiy-storchaka, @corona10
PRs	bpo-40328: Add tool for generating cjk mapping headers #19602

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-19.13:13:59.400>
labels = ['3.9']
title = 'Add tools for generating mappings_XX.h'
updated_at = <Date 2020-05-25.04:14:49.537>
user = 'https://github.com/corona10'

bugs.python.org fields:

activity = <Date 2020-05-25.04:14:49.537>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Demos and Tools']
creation = <Date 2020-04-19.13:13:59.400>
creator = 'corona10'
dependencies = []
files = []
hgrepos = []
issue_num = 40328
keywords = ['patch']
message_count = 4.0
messages = ['366777', '367682', '369844', '369845']
nosy_count = 5.0
nosy_names = ['hyeshik.chang', 'vstinner', 'methane', 'serhiy.storchaka', 'corona10']
pr_nums = ['19602']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue40328'
versions = ['Python 3.9']

[corona10]

AFAIK, there are no tools for generating mappings_XX.h under Modules/cjkcodecs.

It would cause a maintainable issue,
fortunately, I got the old source from https://github.com/BackupTheBerlios/cjkpython.
And I success to port into Python3 version and successfully reproduce to generating header file except for mappings_hk.h mappings_tw.h.

The patch will include a generator and Unicode mapping file.
I am not the codec expert but hopefully it will help us.

[corona10]

New changeset 113feb3 by Dong-hee Na in branch 'master':
bpo-40328: Add tool for generating cjk mapping headers (GH-19602)
113feb3

[methane]

Would you close this issue?

[corona10]

Would you close this issue?

mappings_hk.h mappings_tw.h are still left :)

[sorcio]

Hey @corona10, I happened upon this for other reasons. I found that Python's big5hkscs codec does not support the same character set that modern web browsers support.

There has been at least one version of HKSCS (2008) that updated the Big5-encoded character set, and one more recent version (2016) which only added characters to Unicode (with no Big5 mapping), but might adjust some of the existing mappings. I still need to understand if this might break compatibility.

Is it okay if I look into this and try to make a PR later? I've already done something for mappings_hk but need to look into the compatibility, so it might take a second.

[corona10]

@sorcio Oh thanks, PR will be welcomed :)