Repository navigation
ks_c-5601-1987 is used by microsoft when it really means cp949 #62825
Description
| BPO | 18625 |
|---|---|
| Nosy | @malemburg, @warsaw, @hyeshik, @ezio-melotti, @bitdancer |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2013-08-01.23:54:04.002>
labels = ['type-feature', 'expert-email', 'expert-unicode']
title = 'ks_c-5601-1987 is used by microsoft when it really means cp949'
updated_at = <Date 2013-08-02.08:23:53.892>
user = 'https://github.com/bitdancer'bugs.python.org fields:
activity = <Date 2013-08-02.08:23:53.892>
actor = 'lemburg'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode', 'email']
creation = <Date 2013-08-01.23:54:04.002>
creator = 'r.david.murray'
dependencies = []
files = []
hgrepos = []
issue_num = 18625
keywords = []
message_count = 2.0
messages = ['194139', '194163']
nosy_count = 5.0
nosy_names = ['lemburg', 'barry', 'hyeshik.chang', 'ezio.melotti', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18625'
versions = ['Python 3.4']
Activity
bitdancer commentedon Aug 1, 2013
When Microsoft handles Korean text, it uses its own code page, cp949, which is a superset of ks_c-5601-1987. But when talking to the rest of the world, it claims that the character set name is ks_c-5601-1987. This means that text claimed to be in ks_c-5601-1987 in email messages (and probably on web pages) can't always be decoded using the codec that ks_c-5601-1987 maps to (euc_kr). [*]
This problem shows up in the real world in email. If non euc_kr characters are used, email will try blow up when trying to decode the ostensibly ks_c-5601-1987 text. (I'm not sure if it will also blow up trying to encode it, I'm not sure what characters the two codecs cover.)
I'm not sure what the best solution is, but one possibility would be to add a "fixup" table to email that would cause it to decode ostensibly ks_c-5601-1987 text using the cp949 codec. Since cp949 is a superset, this should at least solve the input side.
[*] Some relevant standards discussion:
From this, it isn't clear why we map ks_c-5601-1987 to euc_kr, since they at least appear to be different codecs. I haven't looked at the relevant RFCs to see what the differences are, though.
malemburg commentedon Aug 2, 2013
The alias was added by Hye-Shik Chang:
http://hg.python.org/cpython-fullhistory/annotate/887ce39f95f2/Lib/encodings/aliases.py#198
I've added him to the nosy list.
If the alias don't match, we'd have to add a codec for the mismatching encoding to maintain compatibility (provided the mismatching encoding is still in use).
mikethetexan commentedon Mar 27, 2025
Is there any update on this bug?
serhiy-storchaka commentedon Jun 2, 2026
What is the relation between KS_C_5601-1987, EUC-KR and cp949? In the IANA registry KS_C_5601-1987 and EUC-KR are two different encodings, and cp949 is not registered. In Python, ks_c_5601_1987 is an alias of euc_kr. Should not it be an alias of cp949 instead?
corona10 commentedon Jun 3, 2026
@serhiy-storchaka
After re-investigating this issue, I think ks_c_5601_1987 should be treated as an alias of cp949, considering the historical background that both ks_c_5601_1987 and cp949 were introduced by Microsoft.
serhiy-storchaka commentedon Jun 3, 2026
As I understand it, KS_C_5601-1987 is simply a set of characters with assigned indexes, like Unicode for East Asia before Unicode. EUC-KR and CP949 are different schemes to encode these indexes in bytes, like UCS2 and UTF16 for Unicode. Originally EUC-KR was limited, and CP949 extended it. Microsoft calls CP949 simply KS_C_5601-1987, like somebody can call UTF-8 or UCS2/UTF-16 simply Unicode.
In Python, EUC-KR is extended. Both EUC-KR and CP949 are able to encode the same 17176 characters. But the result of encoding is different.
CP949 can decode what was encoded in EUC-KR, but the result is different from the original. EUC-KR can fail trying to decode what was encoded in CP949.
malemburg commentedon Jun 4, 2026
According to the Wikipedia page (https://en.wikipedia.org/wiki/KS_X_1001), KS X 1001 is the new name for KS C 5601. If you look at the implementation, both cp949 and eur_kr reference KS X 1001, and that's no surprise, since the KS X 1001 character set seems to be a partial encoding only covering the Hangul and Hanja chars, while cp949 and eur_kr cover additional chars.
They both use different ways of encoding the characters from KS X 1001, though. The cp949 codec seems to be closer to the original KS X 1001 encoding from looking at the code, so I think switching these aliases:
over to 'cp949' is a more correct approach.
That said, I'm no expert in these encodings and don't know how the codecs actually work. I can just look at the code and what it does.
Here's the relevant part from cp949:
If the MSB is not set, CP949 will look like KS X 1001.
serhiy-storchaka commentedon Jun 4, 2026
Opened #150933.
I am not sure about backporting. This is a bug if you cannot decode email with ks_c-5601-1987 charset, but changing aliases in a bugfix can surprise users. It may be okay to backport to 3.15.
gh-62825: Fix encoding aliases "KS_C_5601-1987", "KS X 1001", etc (GH…
[3.15] gh-62825: Fix encoding aliases "KS_C_5601-1987", "KS X 1001", …
pythongh-62825: Fix encoding aliases "KS_C_5601-1987", "KS X 1001", e…