Converting Between Encodings

Corrected Character Mappings to iconv (1) and iconv (3C) [11.0 patch, 11i v1]

This release contains defect fixes for incorrect character mappings. The corrections concern the Simplified Chinese, Traditional Chinese, Japanese, and Korean characters of HP-UX.

Corrected character converter mappings allow for improved interoperability when sending or receiving converted character data to and from Unicode-aware systems.

Correction for Simplified Chinese

A patch corrects an incorrect character mapping that occurs when converting between hp15CN and Unicode (UCS2)/UTF-8 for Simplified Chinese.

Specifically, the Simplified Chinese character Double Vertical Line mapped incorrectly when converting between hp15CN and UCS2/UTF-8. This character was being mapped to the Parallel To character, which is a different character.

Table 2-19 “Changes in iconv Tables for Simplified Chinese” summarizes the change applied to the iconv tables:

Table 2-19 Changes in iconv Tables for Simplified Chinese

hp15CN	Incorrect UCS2	Correct UCS2	Character Name
0xA1CE		0x2225	Parallel To
0xA1AC	0x2225	0x2016	Double Vertical Line

The hp15CN=ucs2 and ucs2=hp15CN iconv converter tables are affected. These tables are shared by both UCS2 and UTF-8 conversions.

No compatibility problems are anticipated. However, if compatibility concerns arise with regard to persistent data stored either in Unicode (UCS2) or UTF-8 on an HP-UX system, it is possible to generate a simple conversion script to search for each occurrence of an incorrect value in either UCS2 or UTF-8 and convert it to the correct value, based on the mapping in Table 2-20 “Mapping Between Old and New Unicode Characters for Simplified Chinese”.

Table 2-20 Mapping Between Old and New Unicode Characters for Simplified Chinese

Old UCS2	UCS2	Old UTF-8	UTF-8	Char Name
0x2225	0x2016	0xe288a5	0xe28096	Double Vertical Line

Correction for Traditional Chinese

A patch corrects several incorrect character mappings that occur when converting between Big-5/EUC and Unicode (UCS2)/UTF-8 for Traditional Chinese.

In the case of big5 coding to and from UCS2/UTF-8, the Ideographic Space character was absent in the Unicode conversion table mapping:

Table 2-21 Changes in iconv Tables for big5/Unicode

big5	Incorrect UCS2	Correct UCS2	Char Name
0xA140		0x3000	Ideographic Space

The following table summarizes the changes applied for conversions between eucTW and UCS2.

Table 2-22 Changes in iconv Tables for eucTW/Unicode

eucTW	Incorrect UCS2	Correct UCS2	Character Name
0xa1a6	0x30fb	0x2022	Bullet
0xa1b7	0x2014	0x2013	EN Dash
0xa1b9	0x2013	0x2014	EM Dash
0xa1b6	0xfe31	0xff5c	Fullwidth Vertical Line
0xa1b8	0xfe32	0xfe31	Presentation Form Vertical EN Dash
0xa1ea	0x2032	0x2035	Reversed Prime
0xa1eb	0x2035	0x2032	Prime
0xa2b9	0x2264	0x2266	Less-than Over Equal to
0xa2ba	0x2265	0x2267	Greater-than Over Equal to
0xa2c2	0xfe66	0xfe65	Small Greater-than
0xa2c3	0xfe65	0xfe66	Small Equals Sign
0xa2de	0xff5c	0x2223	Divides
0xa2e1	0xfe67	0xff0f	Full-width Solidus
0xa2e4	0xffe5	0x00a5	Yen Sign
0xa2e6	0xffe0	0x00a2	Cent Sign
0xa2e7	0xffe1	0x00a3	Pound Sign

The iconv conversions between eucTW and UCS2 or UTF-8 may be affected.

Big-5 conversions with UCS2/UTF-8 are not directly impacted since only a missing table entry has been added.

The eucTW=ucs2, ucs2=eucTW, big5=ucs2 and ucs2=big5 iconv converter tables are affected. These tables are shared by both UCS2 and UTF-8 conversions.

No compatibility problems are anticipated. However, if compatibility concerns arise with regard to persistent data stored either in Unicode (UCS2) or UTF-8 on an HP-UX system, it is possible to generate a simple conversion script to search for each occurrence of an incorrect value in either UCS2 or UTF-8 and convert it to the correct value, based on the mappings in Table 2-23 “Mapping Between Old and New Unicode Characters for Traditional Chinese”.

Table 2-23 Mapping Between Old and New Unicode Characters for Traditional Chinese

Old UCS2	UCS2	Old UTF-8	UTF-8	Char Name
0x30fb	0x2022	0xe383bb	0xe280a2	Bullet
0x2014	0x2013	0xe28094	0xe28093	EN Dash
0x2013	0x2014	0xe28093	0xe28094	EM Dash
0xfe31	0xff5c	0xefb8b1	0xefbd9c	Fullwidth Vertical Line
0xfe32	0xfe31	0xefb8b2	0xefb8b1	Presentation Form Vertical EN Dash
0x2032	0x2035	0xe280b2	0xe280b5	Reversed Prime
0x2035	0x2032	0xe280b5	0xe280b2	Prime
0x2264	0x2266	0xe289a4	0xe289a6	Less-than Over Equal to
0x2265	0x2267	0xe289a5	0xe289a7	Greater-than Over Equal to
0xfe66	0xfe65	0xefb9a6	0xefb9a5	Small Greater-than
0xfe65	0xfe66	0xefb9a5	0xefb9a6	Small Equals Sign
0xff5c	0x2223	0xefbd9c	0xe288a3	Divides
0xfe67	0xff0f	0xefb9a7	0xefbc8f	Full-width Solidus
0xffe5	0x00a5	0xefbfa5	0xc2a5	Yen Sign
0xffe0	0x00a2	0xefbfa0	0xc2a2	Cent Sign
0xffe1	0x00a3	0xefbfa1	0xc2a3	Pound Sign

Correction for Japanese

A patch corrects four incorrect Japanese character mappings that occur between Shift-JIS/EUC and Unicode (UCS2)/UTF-8.

The following table summarizes the changes applied.

Table 2-24 Changes in iconv Tables for Japanese

sjis	eucJP	Incorrect UCS2	Correct UCS2	Character Name
0x8150	0xA1B1	0xFFE3	0x203E	Overline
0x815C	0xA1BD	0x2015	0x2014	Em Dash
0x818F	0xA1EF	0xFFE5	0x00A5	Yen Sign
N/A	0x8FA2B7	0x02DC	0xFF5E	Full-width Tilde

Affected iconv conversions are conversions between sjis and UCS2 or UTF-8 as well as conversions between eucJP and UCS2 or UTF-8.

The sjis=ucs2, ucs2=sjis, eucJP=ucs2 and ucs2=eucJP iconv conversion tables are affected. These tables are shared by both UCS2 and UTF-8 conversions.

No compatibility problems are anticipated. However, if compatibility concerns arise with regard to persistent data stored either in Unicode (UCS2) or UTF-8 on an HP-UX system, it is possible to generate a simple conversion script to search for each occurrence of an incorrect value in either UCS2 or UTF-8 and convert it to the correct value, based on the mappings in Table 2-25 “Mapping Between Old and New Unicode Characters for Japanese”.

Table 2-25 Mapping Between Old and New Unicode Characters for Japanese

Old UCS2	UCS2	Old UTF-8	UTF-8	Char Name
0xFFE3	0x203E	0xefbfa3	0xe280be	Overline
0x2015	0x2014	0xe28095	0xe28094	Em Dash
0xFFE5	0x00A5	0xefbfa5	0xc2a5	Yen Sign
0x02DC	0xFF5E	0xcb9c	0xefbd9e	Full-width Tilde

Correction for Korean

A patch provides a defect fix to address standards nonconformance for Korean Unicode (UCS2)/UTF-8 character mappings.

The currently supplied Korean iconv converter tables do not conform to the Unicode 2.1 and ISO 10646 (with 1997 amendments) standards in addition to the Korean national standard, KSC-5700. The current mappings are considered obsolete by all noted standards organizations.

The enhancement provides a set of standards-conformant iconv converter tables for converting between eucKR and Unicode/UTF-8. Specifically, the obsolete region of 0x3d2e-0x4dff has been remapped to the 0xac00-0xd7ff region specified in Unicode 2.1 for Hangul.

Without this modification, it is impossible to share data with any other system that is standards-conformant in adhering to the Unicode 2.1/ISO 10646/KSC-5700 standards.

Affected iconv conversions are any conversions between eucKR and UCS2 or UTF-8.

The iconv conversion tables affected by this modification are eucKR=ucs2 and ucs2=eucKR. These tables are shared by both UCS2 and UTF-8 conversions.

No compatibility problems are anticipated. However, if compatibility concerns arise with regard to persistent data stored either in Unicode (UCS2) or UTF-8 on an HP-UX system, it is recommended that the previously installed ucs2=eucKR table be saved and renamed prior to installation of this fix. Persistent data can then be converted back to eucKR using this old table and then reconverted to the correct Unicode/UTF-8 representation.

Japan Vendor Council iconv Converters [11i v1.6]

New iconv converters have been introduced to allow for greater interoperability of data sharing within Japanese computing environments. The following items are related to this change:

New Japanese iconv conversion tables are now available that conform to the Open Group Japanese Vendor Council (TOG/JVC) CDE/Motif Technical working group recommendations that ensure the interoperability of ISO 10646/Unicode/JIS X0221 within Japanese computing environments.




	NOTE: JIS X0221 is the Japanese national standard equivalent to ISO-10646:1993 and Unicode 2.1.

These tables convert between HP’s EUC and Shift-JIS (SJIS) characters to those specified by the strict JIS X0201 interpretation as well as the Microsoft Unicode and UTF-8 mappings. The JIS X0221 Extended conversion mappings are already supported within HP-UX. These new converters require 0.6 MB of additional space.

Table 2-26 “New iconv Converters” shows eight new iconv converter tables that are provided to support the alternate mappings in round-trip conversions between EUC and Shift-JIS (SJIS) to /from Unicode and UTF-8.

Table 2-26 New iconv Converters

Conversion	Interpretation
eucJP0201 <-> ucs2	JIS X0201
eucJPMS <-> ucs2	Microsoft
sjis0201 <-> ucs2	JIS X0201
sjisMS <-> ucs2	Microsoft

Table 2-27 “Mapping Variances Between SJIS and Unicode Mappings” illustrates the mapping variances between converter tables when converting between SJIS and different interpretations of Unicode character mappings.




	NOTE: The JIS X0201 Extended (Unix) mappings are already available on HP-UX, referenced in this table as SJIS.

Table 2-27 Mapping Variances Between SJIS and Unicode Mappings

SJIS	sjis0201 UCS2 mapping	SJIS UCS2 mapping	sjisMS UCS2 mapping
0x5C	0x00A5	0x005C	0x005C
0x7E	0x203E	0x007E	0x007E
0x8150	0xFFE3	0x203E	0xFFE3
0x815F	0x005C	0xFF3C	0xFF3C
0x818F	0xFFE5	0x00A5	0xFFE5
0x815C	0x2014	0x2014	0x2015
0x8160	0x301C	0x301C	0xFF5E
0x8161	0x2016	0x2016	0x2225
0x817C	0x2212	0x2212	0xFF0D
0x8191	0x00A2	0x00A2	0xFFE0
0x8192	0x00A3	0x00A3	0xFFE1
0x81CA	0x00AC	0x00AC	0xFFE2

Table 2-28 “Mapping Variances Between EUC and Unicode Mappings” illustrates the mapping variances between converter tables when converting between EUC and different interpretations of Unicode character mappings.




	NOTE: The JIS X0201 Extended (Unix) mappings are already available on HP-UX, referenced in this table as eucJP.

Table 2-28 Mapping Variances Between EUC and Unicode Mappings

EUC	eucJP0201 UCS2 mapping	eucJP UCS2 mapping	eucJPMS UCS2 mapping
0x5C	0x00A5	0x005C	0x005C
0x7E	0x203E	0x007E	0x007E
0xA1B1	0xFFE3	0x203E	0xFFE3
0xA1C0	0x005C	0xFF3C	0xFF3C
0xA1EF	0xFFE5	0x00A5	0xFFE5
0xA1BD	0x2014	0x2014	0x2015
0xA1C1	0x301C	0x301C	0xFF5E
0xA1C2	0x2016	0x2016	0x2225
0xA1DD	0x2212	0x2212	0xFF0D
0xA1F1	0x00A2	0x00A2	0xFFE0
0xA1F2	0x00A3	0x00A3	0xFFE1
0xA2CC	0x00AC	0x00AC	0xFFE2
0x8FA2B7	0x007E	0xFF5E	0xFF5E
0x8FA2C3	0x00A6	0x00A6	0xFFE4

The TOG/JVC has formalized three possible converter mappings for several characters. These mappings are:
1. JIS X0221 based on strict JIS X0201 standard interpretation
2. JIS X0221 Extended that allows for greater co-existence in current ASCII-based (such as Unix) environments
3. JIS X0221 based on Microsoft Japanese mappings

iconv Enhancements for Greek Euro Support [11i v1.6]

See “Greek Euro Support [11i v1.6]” for detailed information on the iconv enhancements for Greek Euro Support.

Mainframe iconv Converters for Japanese characters [11.0 patch, 11i v1 patch, 11i v2]

Mainframe iconv converters between ShiftJIS/eucJP/UCS2 and NEC-JIPS/Hitachi-KEIS/Fujitsu-JEF, were introduced at HP-UX 11i v1.0. This release includes several fixes of mapping errors for JIS standard characters.

This release of mainframe iconv conversion tables includes numerous fixes for mapping errors for JIS standard characters in the basic part of those mainframe codesets. The detailed changes are described in MFConvChanges.jips, MFConvChanges.keis and MFConvChanges.jef in the /usr/share/doc directory.

In addition, this release of mainframe iconv conversion methods includes a fix to handle an incomplete shift sequence at the end of an input buffer.

Impact to the Customer

If the customer has already used the HP-UX 11i v1 version of mainframe iconv converters and then uses this version, the results will be different because of fixes in the mappings for JIS standard characters. It is recommended that the previously installed tables be saved and renamed prior to installation of this release. Persistent data can then be converted back using the old table and then reconverted using the new tables to the correct representation.

If the last character in the input buffer could be a valid character OR an incomplete shift sequence, iconv(3C) returns EINVAL. If that character is the final one of the input file, iconv(3C) never returns successfully without appending other dummy data like NULL to that character. That character is 0x1a for jipsj, 0x3f for jipsec/jipsek and 0xa for keis7c/keis7k/keis8c/keis8k which could be a control character OR an incomplete shift sequence.

Compatibility

No compatibility problems are anticipated. However, if compatibility concerns arise with regard to persistent data stored on an HP-UX system, it is recommended that the previously installed tables be saved and renamed prior to installation of this release. Persistent data can then be converted back using this old table and then reconverted to the correct representation using the new tables.

Performance

No performance issues.

Documentation

Japanese System Environment User's Manual (in Japanese language)
iconv(3C) manpage

Future Plans

The mainframe iconv converters will track future changes in Unicode standard support in HP-UX.