summaryrefslogtreecommitdiff
path: root/iconvdata
diff options
context:
space:
mode:
authorUlrich Drepper <drepper@redhat.com>2001-06-06 12:55:46 +0000
committerUlrich Drepper <drepper@redhat.com>2001-06-06 12:55:46 +0000
commit601d294296e1de9fc423700db00bccc04a2bf50d (patch)
tree1d274eb4e512431b042e07bf8f8ff4a980407257 /iconvdata
parent9a1f675436c5622b7846faa25274a2ef2d743285 (diff)
downloadglibc-601d294296e1de9fc423700db00bccc04a2bf50d.tar.gz
Update.
2001-06-04 Bruno Haible <haible@clisp.cons.org> * iconv/loop.c (UNICODE_TAG_HANDLER): New macro. * iconv/gconv_simple.c (__gconv_transform_internal_ascii): Invoke UNICODE_TAG_HANDLER. (__gconv_transform_internal_ucs2): Likewise. (__gconv_transform_internal_ucs2reverse): Likewise. * iconvdata/8bit-gap.c (BODY for TO_LOOP): Invoke UNICODE_TAG_HANDLER. * iconvdata/8bit-generic.c (BODY for TO_LOOP): Likewise. * iconvdata/ansi_x3.110.c (BODY for TO_LOOP): Likewise. * iconvdata/big5.c (BODY for TO_LOOP): Likewise. * iconvdata/big5hkscs.c (BODY for TO_LOOP): Likewise. * iconvdata/cp1255.c (BODY for TO_LOOP): Likewise. * iconvdata/cp1258.c (BODY for TO_LOOP): Likewise. * iconvdata/euc-cn.c (BODY for TO_LOOP): Likewise. * iconvdata/euc-jp.c (BODY for TO_LOOP): Likewise. * iconvdata/euc-kr.c (BODY for TO_LOOP): Likewise. * iconvdata/euc-tw.c (BODY for TO_LOOP): Likewise. * iconvdata/gbk.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm930.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm932.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm933.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm935.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm937.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm939.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm943.c (BODY for TO_LOOP): Likewise. * iconvdata/iso646.c (BODY for TO_LOOP): Likewise. * iconvdata/iso8859-1.c (BODY for TO_LOOP): Likewise. * iconvdata/iso_6937.c (BODY for TO_LOOP): Likewise. * iconvdata/iso_6937-2.c (BODY for TO_LOOP): Likewise. * iconvdata/iso-2022-cn.c (BODY for TO_LOOP): Likewise. * iconvdata/iso-2022-cn-ext.c (BODY for TO_LOOP): Likewise. * iconvdata/iso-2022-kr.c (BODY for TO_LOOP): Likewise. * iconvdata/johab.c (BODY for TO_LOOP): Likewise. * iconvdata/sjis.c (BODY for TO_LOOP): Likewise. * iconvdata/t.61.c (BODY for TO_LOOP): Likewise. * iconvdata/uhc.c (BODY for TO_LOOP): Likewise. * iconvdata/unicode.c (BODY for TO_LOOP): Likewise. * iconvdata/iso-2022-jp.c (TAG_none, TAG_language, TAG_language_j, TAG_language_ja, TAG_language_k, TAG_language_ko, TAG_language_z, TAG_language_zh, CURRENT_TAG_MASK): New enum values. (EMIT_SHIFT_TO_INIT): Don't emit an escape sequence if ASCII_set is already selected but set2 or tag are set. (conversion): New enum type. (cvlist_t): New type. (CVLIST, CVLIST_FIRST, CVLIST_REST): New macros. (conversion_lists): New array. (BODY for TO_LOOP): Keep track of Unicode 3.1 language tag. If "ja", prefer conversion to Japanese character sets. If "zh", prefer conversion to GB2312. If "ko", prefer conversion to KSC5601. Small optimizations. (INIT_PARAMS): Add tag. (UPDATE_PARAMS): Add tag. 2001-06-04 Bruno Haible <haible@clisp.cons.org> * locale/programs/locfile.c (write_locale_data): Before creat(), unlink the file, to avoid crashing the processes that mmap it. Change a double slash to a single slash. Free fname in case of error return. 2001-06-02 Jakub Jelinek <jakub@redhat.com> * sysdeps/i386/fpu/s_frexpl.S (__frexpl): Mostly revert 2000-12-03 changes, do the special handling for denormal numbers, not for normalized numbers (patch by <trevin@xmission.com>). * math/test-misc.c (main): Test frexpl with denormal arguments. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * math/libm-test.inc (llround_test): Add two new llround tests. * sysdeps/ieee754/ldbl-96/s_llroundl.c (__llroundl): Don't allow overflow when rounding away from zero. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * math/Makefile (libm-calls): Add e_log2, w_log2, remove s_log2. * math/math_private.h (__ieee754_log2, __ieee754_log2f, __ieee754_log2l): New prototypes. * sysdeps/generic/w_log2.c: New file. * sysdeps/generic/w_log2f.c: New file. * sysdeps/generic/w_log2l.c: New file. * sysdeps/generic/s_log2l.c: Move... * sysdeps/generic/e_log2l.c: ...to here. Rename to __ieee754_log2l. * sysdeps/ieee754/k_standard.c (__kernel_standard): Handle log2(0) and log2(x < 0). * sysdeps/i386/fpu/s_log2.S: Move... * sysdeps/i386/fpu/e_log2.S: ...to here. Rename to __ieee754_log2. * sysdeps/i386/fpu/s_log2f.S: Move... * sysdeps/i386/fpu/e_log2f.S: ...to here. Rename to __ieee754_log2f. * sysdeps/i386/fpu/s_log2l.S: Move... * sysdeps/i386/fpu/e_log2l.S: ...to here. Rename to __ieee754_log2l. * sysdeps/m68k/fpu/s_log2.S: Move... * sysdeps/m68k/fpu/e_log2.S: ...to here. Rename to __ieee754_log2. * sysdeps/m68k/fpu/s_log2f.S: Move... * sysdeps/m68k/fpu/e_log2f.S: ...to here. Rename to __ieee754_log2f. * sysdeps/m68k/fpu/s_log2l.S: Move... * sysdeps/m68k/fpu/e_log2l.S: ...to here. Rename to __ieee754_log2l. * sysdeps/ieee754/dbl-64/s_log2.c: Move... * sysdeps/ieee754/dbl-64/e_log2.c: ...to here. Rename to __ieee754_log2. * sysdeps/ieee754/flt-32/s_log2f.c: Move... * sysdeps/ieee754/flt-32/e_log2f.c: ...to here. Rename to __ieee754_log2f. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * sysdeps/generic/w_exp2.c (u_threshold): Lower threshold so that even arguments which result in denormalized exp2 are accepted. (__exp2): Arguments equal to u_threshold already result into underflow. * sysdeps/generic/w_exp2f.c (u_threshold, __exp2f): Likewise. * sysdeps/generic/w_exp2l.c (u_threshold, __exp2l): Likewise. * sysdeps/ieee754/dbl-64/e_exp2.c (__ieee754_exp2): Lomark was too low, with corrected lowmark use greaterequal, not greater. * sysdeps/ieee754/flt-32/e_exp2f.c (__ieee754_exp2f): Likewise. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * math/libm-test.inc (ilogb_test): Test that ilogb(+-Inf) == INT_MAX. * sysdeps/i386/fpu/s_ilogb.S (__ilogb): Return INT_MAX for +-Inf. * sysdeps/i386/fpu/s_ilogbf.S (__ilogbf): Likewise. * sysdeps/i386/fpu/s_ilogbl.S (__ilogbl): Likewise. * sysdeps/ieee754/dbl-64/s_ilogb.c (__ilogb): Likewise. * sysdeps/ieee754/flt-32/s_ilogbf.c (__ilogbf): Likewise. * sysdeps/ieee754/ldbl-128/s_ilogbl.c (__ilogbl): Likewise. * sysdeps/ieee754/ldbl-96/s_ilogbl.c (__ilogbl): Likewise. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * sysdeps/generic/w_coshl.c (__coshl): Test if finite argument gave non-finite result instead of using constant in generic version. * sysdeps/generic/w_coshf.c (__coshf): Likewise. * sysdeps/generic/w_cosh.c (__cosh): Likewise. * sysdeps/generic/w_exp10.c (o_threshold, u_threshold): Remove. (__exp10): Test if finite argument gave non-finite result. * sysdeps/generic/w_exp10f.c (o_threshold, u_threshold, __exp10f): Likewise. * sysdeps/generic/w_exp10l.c (o_threshold, u_threshold, __exp10l): Likewise. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * sysdeps/ieee754/ldbl-96/e_coshl.c (__ieee754_coshl): Fix overflow threshold constant (log(LDBL_MAX)+M_LN2l). 2001-05-29 Bruno Haible <haible@clisp.cons.org> * locale/programs/ld-ctype.c (idx_table): New struct type. (idx_table_init, idx_table_get, idx_table_add): New functions. (MAX_CHARNAMES_IDX): Remove macro. (locale_ctype_t): Change type of charnames_idx field. (ctype_startup): Change initialization of charnames_idx field. (find_idx): Use idx_table_get and idx_table_add for speed. * locale/programs/charmap.c (charmap_new_char): Fix ucs4 value computation of characters in a range. 2001-05-29 Bruno Haible <haible@clisp.cons.org> * iconvdata/gb18030.c (__fourbyte_to_ucs1): Add mappings for <U03F4>, <U03F5>. (__ucs_to_gb18030_tab1): Likewise. (BODY for FROM_LOOP): Add mapping for <U00010000>..<U0010FFFF>. (BODY for TO_LOOP): Likewise. * iconvdata/tst-table-charmap.sh: Update for charmaps containing <U00xxxxxx> syntax. * iconvdata/tst-table-from.c (bmp_only): New variable. (utf8_decode): If bmp_only, don't return characters outside Unicode plane 0. (main): When testing UTF-8 or GB18030, set bmp_only to 1. Don't print a conversion line if utf8_decode returns NULL. * iconvdata/tst-table-to.c (main): When testing encodings other than UTF-8 and GB18030, loop upto U+30000 instead of U+10000. Use UTF-8 instead of UCS-2 as input. * iconvdata/tst-table.sh: For GB18030, use only the part < 0x10000 of the charmap. 2001-05-29 Bruno Haible <haible@clisp.cons.org> * iconvdata/cns11643l1.c: Update to Unicode 3.1. (__cns11643l1_to_ucs4_tab): Regenerated. (__cns11643l1_from_ucs4_tab12): Regenerated. * iconvdata/cns11643.c: Update to Unicode 3.1. (__cns11643l14_to_ucs4_tab): Remove array. (__cns11643l3_to_ucs4_tab, __cns11643l4_to_ucs4_tab, __cns11643l5_to_ucs4_tab, __cns11643l6_to_ucs4_tab, __cns11643l7_to_ucs4_tab, __cns11643l15_to_ucs4_tab): New arrays. (__cns11643_from_ucs4p0_tab): Renamed from __cns11643_from_ucs4_tab. (__cns11643_from_ucs4p2_tab): New array. * iconvdata/cns11643.h (__cns11643l14_to_ucs4_tab): Remove declaration. (__cns11643l3_to_ucs4_tab, __cns11643l4_to_ucs4_tab, __cns11643l5_to_ucs4_tab, __cns11643l6_to_ucs4_tab, __cns11643l7_to_ucs4_tab, __cns11643l15_to_ucs4_tab): New declarations. (cns11643_to_ucs4): Treat planes 3, 4, 5, 6, 7, 15 instead of 14. (__cns11643_from_ucs4_tab): Remove declaration. (__cns11643_from_ucs4p0_tab, __cns11643_from_ucs4p2_tab): New declarations. (ucs4_to_cns11643): Update for new arrays. Treat U+3400..U+4DFF and U+20000..U+2A6D6. * iconvdata/cns11643l2.h (__cns11643_from_ucs4_tab): Remove declaration. (__cns11643_from_ucs4p0_tab): New declaration. (ucs4_to_cns11643l2): Update for new arrays. * iconvdata/iso-2022-cn-ext.c (BODY for FROM_LOOP): Handle planes 3 to 7. (BODY for TO_LOOP): Handle planes 3 to 7, instead of plane 14. * iconvdata/EUC-TW.irreversible: New file. * iconvdata/tst-table.sh: Use it. * iconvdata/Makefile (distribute): Add CP1255.irreversible, CP1258.irreversible, EUC-TW.irreversible. 2001-05-29 Bruno Haible <haible@clisp.cons.org> * locale/C-translit.h.in: Add transliterations for new Unicode 3.1 mathematical symbols.
Diffstat (limited to 'iconvdata')
-rw-r--r--iconvdata/8bit-gap.c7
-rw-r--r--iconvdata/8bit-generic.c4
-rw-r--r--iconvdata/ansi_x3.110.c4
-rw-r--r--iconvdata/big5.c3
-rw-r--r--iconvdata/big5hkscs.c4
-rw-r--r--iconvdata/cp1255.c5
-rw-r--r--iconvdata/cp1258.c5
-rw-r--r--iconvdata/euc-cn.c4
-rw-r--r--iconvdata/euc-jp.c4
-rw-r--r--iconvdata/euc-kr.c6
-rw-r--r--iconvdata/euc-tw.c16
-rw-r--r--iconvdata/gb18030.c35
-rw-r--r--iconvdata/gbk.c3
-rw-r--r--iconvdata/ibm930.c2
-rw-r--r--iconvdata/ibm932.c51
-rw-r--r--iconvdata/ibm933.c2
-rw-r--r--iconvdata/ibm935.c2
-rw-r--r--iconvdata/ibm937.c2
-rw-r--r--iconvdata/ibm939.c1
-rw-r--r--iconvdata/ibm943.c49
-rw-r--r--iconvdata/iso-2022-cn-ext.c1
-rw-r--r--iconvdata/iso-2022-cn.c4
-rw-r--r--iconvdata/iso-2022-jp.c697
-rw-r--r--iconvdata/iso-2022-kr.c7
-rw-r--r--iconvdata/iso646.c7
-rw-r--r--iconvdata/iso8859-1.c4
-rw-r--r--iconvdata/iso_6937-2.c3
-rw-r--r--iconvdata/iso_6937.c3
-rw-r--r--iconvdata/johab.c3
-rw-r--r--iconvdata/sjis.c9
-rw-r--r--iconvdata/t.61.c4
-rwxr-xr-xiconvdata/tst-table-charmap.sh4
-rw-r--r--iconvdata/tst-table-from.c40
-rw-r--r--iconvdata/tst-table-to.c31
-rwxr-xr-xiconvdata/tst-table.sh13
-rw-r--r--iconvdata/uhc.c3
-rw-r--r--iconvdata/unicode.c3
37 files changed, 657 insertions, 388 deletions
diff --git a/iconvdata/8bit-gap.c b/iconvdata/8bit-gap.c
index 7caa5427d5..e349270b8d 100644
--- a/iconvdata/8bit-gap.c
+++ b/iconvdata/8bit-gap.c
@@ -1,6 +1,6 @@
/* Generic conversion to and from 8bit charsets,
converting from UCS using gaps.
- Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
@@ -83,7 +83,10 @@ struct gap
unsigned char res; \
\
if (__builtin_expect (ch, 0) >= 0xffff) \
- rp = NULL; \
+ { \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ rp = NULL; \
+ } \
else \
while (ch > rp->end) \
++rp; \
diff --git a/iconvdata/8bit-generic.c b/iconvdata/8bit-generic.c
index bc05ccd8db..3a81080c4c 100644
--- a/iconvdata/8bit-generic.c
+++ b/iconvdata/8bit-generic.c
@@ -1,5 +1,5 @@
/* Generic conversion to and from 8bit charsets.
- Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
@@ -67,6 +67,8 @@
if (__builtin_expect (ch >= sizeof (from_ucs4) / sizeof (from_ucs4[0]), 0)\
|| (__builtin_expect (from_ucs4[ch], '\1') == '\0' && ch != 0)) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* This is an illegal character. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/ansi_x3.110.c b/iconvdata/ansi_x3.110.c
index e1dd0ed0a7..a06a4f0658 100644
--- a/iconvdata/ansi_x3.110.c
+++ b/iconvdata/ansi_x3.110.c
@@ -1,5 +1,5 @@
/* Generic conversion to and from ANSI_X3.110-1983.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -549,6 +549,8 @@ static const char from_ucs4[][2] =
cp = "\xd5"; \
else \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Illegal characters. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/big5.c b/iconvdata/big5.c
index e06b44609e..4d1f1acca4 100644
--- a/iconvdata/big5.c
+++ b/iconvdata/big5.c
@@ -1,5 +1,5 @@
/* Mapping tables for Big5 handling.
- Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
@@ -8564,6 +8564,7 @@ static const char from_ucs4_tab15[][2] =
cp = from_ucs4_tab15[ch - 0xff01]; \
break; \
default: \
+ UNICODE_TAG_HANDLER (ch, 4); \
/* Illegal character. */ \
cp = ""; \
break; \
diff --git a/iconvdata/big5hkscs.c b/iconvdata/big5hkscs.c
index 7f4a8c3bf2..dff5456868 100644
--- a/iconvdata/big5hkscs.c
+++ b/iconvdata/big5hkscs.c
@@ -1,5 +1,5 @@
/* Mapping tables for Big5-HKSCS handling.
- Copyright (C) 1997, 1998, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1997, 1998, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
Modified for Big5-HKSCS by Roger So <spacehunt@e-fever.org>, 2000.
@@ -11772,6 +11772,8 @@ static struct
\
if (cp[0] == '\0' && ch != 0) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Illegal character. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/cp1255.c b/iconvdata/cp1255.c
index 127f8be406..d81e7da85c 100644
--- a/iconvdata/cp1255.c
+++ b/iconvdata/cp1255.c
@@ -451,7 +451,10 @@ static const struct {
else if (ch == 0x2122) \
res = 0x99; \
else \
- res = 0; \
+ { \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ res = 0; \
+ } \
\
if (__builtin_expect (res != 0, 1)) \
{ \
diff --git a/iconvdata/cp1258.c b/iconvdata/cp1258.c
index b4168d9781..4868e7559c 100644
--- a/iconvdata/cp1258.c
+++ b/iconvdata/cp1258.c
@@ -777,7 +777,10 @@ static const struct {
else if (ch == 0x2122) \
res = 0x99; \
else \
- res = 0; \
+ { \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ res = 0; \
+ } \
\
if (__builtin_expect (res != 0, 1)) \
{ \
diff --git a/iconvdata/euc-cn.c b/iconvdata/euc-cn.c
index 58f47eda3f..29ed465979 100644
--- a/iconvdata/euc-cn.c
+++ b/iconvdata/euc-cn.c
@@ -1,5 +1,5 @@
/* Mapping tables for EUC-CN handling.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -140,6 +140,8 @@
{ \
if (__builtin_expect (found, 0) == __UNKNOWN_10646_CHAR) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Illegal character. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/euc-jp.c b/iconvdata/euc-jp.c
index 8d8429614d..e0512cf12e 100644
--- a/iconvdata/euc-jp.c
+++ b/iconvdata/euc-jp.c
@@ -1,5 +1,5 @@
/* Mapping tables for EUC-JP handling.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -227,6 +227,8 @@
} \
else \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Illegal character. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/euc-kr.c b/iconvdata/euc-kr.c
index 5ecdbb8182..7ec613b3f9 100644
--- a/iconvdata/euc-kr.c
+++ b/iconvdata/euc-kr.c
@@ -1,5 +1,5 @@
/* Mapping tables for EUC-KR handling.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Jungshik Shin <jshin@pantheon.yale.edu>
and Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -142,10 +142,12 @@ euckr_from_ucs4 (uint32_t ch, unsigned char *cp)
\
/* Decomposing Hangul syllables not available in KS C 5601 into \
Jamos should be considered either here or in euckr_from_ucs4() */ \
- euckr_from_ucs4 (ch, cp) ; \
+ euckr_from_ucs4 (ch, cp); \
\
if (__builtin_expect (cp[0], '\1') == '\0' && ch != 0) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Illegal character. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/euc-tw.c b/iconvdata/euc-tw.c
index 39c2f1f7da..68a80f7c6d 100644
--- a/iconvdata/euc-tw.c
+++ b/iconvdata/euc-tw.c
@@ -1,5 +1,5 @@
/* Mapping tables for EUC-TW handling.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -61,14 +61,14 @@
} \
else \
{ \
- /* Two or more byte character. First test whether the next \
- character is also available. */ \
+ /* Two or more byte character. First test whether the next byte \
+ is also available. */ \
uint32_t ch2; \
\
if (inptr + 1 >= inend) \
{ \
- /* The second character is not available. Store the \
- intermediate result. */ \
+ /* The second byte is not available. Store the intermediate \
+ result. */ \
result = __GCONV_INCOMPLETE_INPUT; \
break; \
} \
@@ -99,7 +99,7 @@
\
if (ch == 0) \
{ \
- /* The third or fourth character is not available. Store \
+ /* The third or fourth byte is not available. Store \
the intermediate result. */ \
result = __GCONV_INCOMPLETE_INPUT; \
break; \
@@ -170,7 +170,7 @@
*outptr++ = ch; \
else \
{ \
- /* Try the JIS character sets. */ \
+ /* Try the CNS 11643 planes. */ \
size_t found; \
\
found = ucs4_to_cns11643l1 (ch, outptr, outend - outptr); \
@@ -199,6 +199,8 @@
} \
if (__builtin_expect (found, 0) == __UNKNOWN_10646_CHAR) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Illegal character. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/gb18030.c b/iconvdata/gb18030.c
index 8b8c472074..e3e9e8a66d 100644
--- a/iconvdata/gb18030.c
+++ b/iconvdata/gb18030.c
@@ -5683,7 +5683,8 @@ static const uint16_t __fourbyte_to_ucs1[] =
[0x0309] = 0x03e5, [0x030a] = 0x03e6, [0x030b] = 0x03e7, [0x030c] = 0x03e8,
[0x030d] = 0x03e9, [0x030e] = 0x03ea, [0x030f] = 0x03eb, [0x0310] = 0x03ec,
[0x0311] = 0x03ed, [0x0312] = 0x03ee, [0x0313] = 0x03ef, [0x0314] = 0x03f0,
- [0x0315] = 0x03f1, [0x0316] = 0x03f2, [0x0317] = 0x03f3, [0x0324] = 0x0400,
+ [0x0315] = 0x03f1, [0x0316] = 0x03f2, [0x0317] = 0x03f3, [0x0318] = 0x03f4,
+ [0x0319] = 0x03f5, [0x0324] = 0x0400,
[0x0325] = 0x0402, [0x0326] = 0x0403, [0x0327] = 0x0404, [0x0328] = 0x0405,
[0x0329] = 0x0406, [0x032a] = 0x0407, [0x032b] = 0x0408, [0x032c] = 0x0409,
[0x032d] = 0x040a, [0x032e] = 0x040b, [0x032f] = 0x040c, [0x0330] = 0x040d,
@@ -9517,6 +9518,7 @@ static const unsigned char __ucs_to_gb18030_tab1[][2] =
[0x036b] = "\x23\x0f", [0x036c] = "\x23\x10", [0x036d] = "\x23\x11",
[0x036e] = "\x23\x12", [0x036f] = "\x23\x13", [0x0370] = "\x23\x14",
[0x0371] = "\x23\x15", [0x0372] = "\x23\x16", [0x0373] = "\x23\x17",
+ [0x0374] = "\x23\x18", [0x0375] = "\x23\x19",
[0x0380] = "\x23\x24", [0x0381] = "\xa7\xa7", [0x0382] = "\x23\x25",
[0x0383] = "\x23\x26", [0x0384] = "\x23\x27", [0x0385] = "\x23\x28",
[0x0386] = "\x23\x29", [0x0387] = "\x23\x2a", [0x0388] = "\x23\x2b",
@@ -21954,15 +21956,15 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] =
} \
else \
{ \
- /* Two or four byte character. First test whether the \
- next character is also available. */ \
+ /* Two or four byte character. First test whether the next \
+ byte is also available. */ \
uint32_t ch2; \
unsigned long int idx; \
\
if (inptr + 1 >= inend) \
{ \
- /* The second character is not available. Store \
- the intermediate result. */ \
+ /* The second byte is not available. Store the \
+ intermediate result. */ \
result = __GCONV_INCOMPLETE_INPUT; \
break; \
} \
@@ -21992,7 +21994,7 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] =
\
if (inptr + 3 >= inend) \
{ \
- /* Not all characters are available. Store \
+ /* Not all bytes are available. Store \
the intermediate result. */ \
result = __GCONV_INCOMPLETE_INPUT; \
break; \
@@ -22035,17 +22037,19 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] =
+ ch3 - 0x81) * 10 + ch4 - 0x30; \
\
if (idx <= 0x4A18) \
- ch = __fourbyte_to_ucs1[idx]; \
+ ch = __fourbyte_to_ucs1[idx]; \
else if (idx >= 0x4ABD && idx <= 0x4F49) \
- ch = idx + 0x5543; \
+ ch = idx + 0x5543; \
else if (idx >= 0x4F4D && idx <= 0x4F83) \
- ch = __fourbyte_to_ucs2[idx-0x4F4D]; \
+ ch = __fourbyte_to_ucs2[idx-0x4F4D]; \
else if (idx >= 0x56BD && idx <= 0x8260) \
- ch = idx + 0x5543; \
+ ch = idx + 0x5543; \
else if (idx >= 0x93A9 && idx <= 0x99F9) \
- ch = __fourbyte_to_ucs3[idx-0x93A9]; \
+ ch = __fourbyte_to_ucs3[idx-0x93A9]; \
+ else if (idx >= 0x2E248 && idx <= 0x12E247) \
+ ch = idx - 0x1E248; \
else \
- ch = 0; \
+ ch = 0; \
\
if (ch == 0 && *inptr != '\0') \
{ \
@@ -22148,6 +22152,11 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] =
len = 4; \
} \
} \
+ else if (ch >= 0x10000 && ch <= 0x10FFFF) \
+ { \
+ idx = ch + 0x1E248; \
+ len = 4; \
+ } \
else \
len = 0; \
\
@@ -22175,7 +22184,7 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] =
} \
else /* len == 4 */ \
{ \
- /* See whether there is enough room for the second byte we \
+ /* See whether there is enough room for all four bytes we \
write. */ \
if (__builtin_expect (outptr + 3 >= outend, 0)) \
{ \
diff --git a/iconvdata/gbk.c b/iconvdata/gbk.c
index b07f39294a..23e7129740 100644
--- a/iconvdata/gbk.c
+++ b/iconvdata/gbk.c
@@ -1,5 +1,5 @@
/* Mapping tables for GBK handling.
- Copyright (C) 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Sean Chen <sean.chen@turbolinux.com>, 1999.
@@ -13471,6 +13471,7 @@ static const char __gbk_from_ucs4_tab12[][2] =
cp = "\xa1\xe9\0\0\xa1\xea\0\0\xa9\x56\0\0\xa3\xfe\0\0\xa9\x57\0\0\xa3\xa4" + ((ch - 0xffe0) * 4); \
break; \
default: \
+ UNICODE_TAG_HANDLER (ch, 4); \
cp = ""; \
break; \
} \
diff --git a/iconvdata/ibm930.c b/iconvdata/ibm930.c
index 0d09b7bec2..406f3f5174 100644
--- a/iconvdata/ibm930.c
+++ b/iconvdata/ibm930.c
@@ -203,6 +203,8 @@ enum
\
if (__builtin_expect (ch, 0) >= 0xffff) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
if (! ignore_errors_p ()) \
{ \
result = __GCONV_ILLEGAL_INPUT; \
diff --git a/iconvdata/ibm932.c b/iconvdata/ibm932.c
index 07fdd25f6e..d6c5877c0e 100644
--- a/iconvdata/ibm932.c
+++ b/iconvdata/ibm932.c
@@ -1,5 +1,5 @@
/* Conversion from and to IBM932.
- Copyright (C) 2000 Free Software Foundation, Inc.
+ Copyright (C) 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Masahide Washizawa <washi@jp.ibm.com>, 2000.
@@ -148,7 +148,7 @@
{ \
const struct gap *rp = __ucs4_to_ibm932sb_idx; \
unsigned char sc; \
- uint32_t ch = get32(inptr); \
+ uint32_t ch = get32 (inptr); \
uint16_t found = TRUE; \
uint32_t i; \
uint32_t low; \
@@ -156,7 +156,10 @@
uint16_t pccode; \
\
if (__builtin_expect (ch, 0) >= 0xffff) \
+ { \
+ UNICODE_TAG_HANDLER (ch, 4); \
rp = NULL; \
+ } \
else \
while (ch > rp->end) \
++rp; \
@@ -188,26 +191,26 @@
break; \
} \
} \
- if (found) \
- { \
- if (__builtin_expect (outptr + 2 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = pccode >> 8 & 0xff; \
- *outptr++ = pccode & 0xff; \
- } \
- else \
- { \
- /* This is an illegal character. */ \
- if (! ignore_errors_p ()) \
- { \
- result = __GCONV_ILLEGAL_INPUT; \
- break; \
- } \
- ++*irreversible; \
- } \
+ if (found) \
+ { \
+ if (__builtin_expect (outptr + 2 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = pccode >> 8 & 0xff; \
+ *outptr++ = pccode & 0xff; \
+ } \
+ else \
+ { \
+ /* This is an illegal character. */ \
+ if (! ignore_errors_p ()) \
+ { \
+ result = __GCONV_ILLEGAL_INPUT; \
+ break; \
+ } \
+ ++*irreversible; \
+ } \
} \
else \
{ \
@@ -224,8 +227,8 @@
*outptr++ = sc; \
} \
\
- /* Now that we wrote the output increment the input pointer. */ \
- inptr += 4; \
+ /* Now that we wrote the output increment the input pointer. */ \
+ inptr += 4; \
}
#define LOOP_NEED_FLAGS
#include <iconv/loop.c>
diff --git a/iconvdata/ibm933.c b/iconvdata/ibm933.c
index 92a9d5b52a..9d9d76aee8 100644
--- a/iconvdata/ibm933.c
+++ b/iconvdata/ibm933.c
@@ -205,6 +205,8 @@ enum
\
if (__builtin_expect (ch, 0) >= 0xffff) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
if (! ignore_errors_p ()) \
{ \
result = __GCONV_ILLEGAL_INPUT; \
diff --git a/iconvdata/ibm935.c b/iconvdata/ibm935.c
index 9728894b0b..91e3255baf 100644
--- a/iconvdata/ibm935.c
+++ b/iconvdata/ibm935.c
@@ -203,6 +203,8 @@ enum
\
if (__builtin_expect (ch, 0) >= 0xffff) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
if (! ignore_errors_p ()) \
{ \
result = __GCONV_ILLEGAL_INPUT; \
diff --git a/iconvdata/ibm937.c b/iconvdata/ibm937.c
index 3dca817914..e47ff941ae 100644
--- a/iconvdata/ibm937.c
+++ b/iconvdata/ibm937.c
@@ -195,6 +195,8 @@ enum
/ sizeof (__ucs4_to_ibm937db[0])), 0) \
|| __builtin_expect (cp[0], '\1') == '\0') \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* This is an illegal character. */ \
if (! ignore_errors_p ()) \
{ \
diff --git a/iconvdata/ibm939.c b/iconvdata/ibm939.c
index 02307a0084..77a2b2b8a9 100644
--- a/iconvdata/ibm939.c
+++ b/iconvdata/ibm939.c
@@ -203,6 +203,7 @@ enum
\
if (__builtin_expect (ch, 0) >= 0xffff) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
rp1 = NULL; \
rp2 = NULL; \
} \
diff --git a/iconvdata/ibm943.c b/iconvdata/ibm943.c
index 5ff27f558a..3a742152a8 100644
--- a/iconvdata/ibm943.c
+++ b/iconvdata/ibm943.c
@@ -1,5 +1,5 @@
/* Conversion from and to IBM943.
- Copyright (C) 2000 Free Software Foundation, Inc.
+ Copyright (C) 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Masahide Washizawa <washi@jp.ibm.com>, 2000.
@@ -156,7 +156,10 @@
uint16_t pccode; \
\
if (__builtin_expect (ch, 0) >= 0xffff) \
+ { \
+ UNICODE_TAG_HANDLER (ch, 4); \
rp = NULL; \
+ } \
else \
while (ch > rp->end) \
++rp; \
@@ -188,26 +191,26 @@
break; \
} \
} \
- if (found) \
- { \
- if (__builtin_expect (outptr + 2 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = pccode >> 8 & 0xff; \
- *outptr++ = pccode & 0xff; \
- } \
- else \
- { \
- /* This is an illegal character. */ \
- if (! ignore_errors_p ()) \
- { \
- result = __GCONV_ILLEGAL_INPUT; \
- break; \
- } \
- ++*irreversible; \
- } \
+ if (found) \
+ { \
+ if (__builtin_expect (outptr + 2 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = pccode >> 8 & 0xff; \
+ *outptr++ = pccode & 0xff; \
+ } \
+ else \
+ { \
+ /* This is an illegal character. */ \
+ if (! ignore_errors_p ()) \
+ { \
+ result = __GCONV_ILLEGAL_INPUT; \
+ break; \
+ } \
+ ++*irreversible; \
+ } \
} \
else \
{ \
@@ -224,8 +227,8 @@
*outptr++ = sc; \
} \
\
- /* Now that we wrote the output increment the input pointer. */ \
- inptr += 4; \
+ /* Now that we wrote the output increment the input pointer. */ \
+ inptr += 4; \
}
#define LOOP_NEED_FLAGS
#include <iconv/loop.c>
diff --git a/iconvdata/iso-2022-cn-ext.c b/iconvdata/iso-2022-cn-ext.c
index 9fd1d64b7d..aeb921a43c 100644
--- a/iconvdata/iso-2022-cn-ext.c
+++ b/iconvdata/iso-2022-cn-ext.c
@@ -538,6 +538,7 @@ enum
} \
if (used == ASCII_set) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
STANDARD_ERR_HANDLER (4); \
} \
} \
diff --git a/iconvdata/iso-2022-cn.c b/iconvdata/iso-2022-cn.c
index 6eeeea3606..61f4bdda0c 100644
--- a/iconvdata/iso-2022-cn.c
+++ b/iconvdata/iso-2022-cn.c
@@ -1,5 +1,5 @@
/* Conversion module for ISO-2022-CN.
- Copyright (C) 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1999.
@@ -320,6 +320,8 @@ enum
used = GB2312_set + CNS11643_1_set - used; \
else \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Even this does not work. Error. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/iso-2022-jp.c b/iconvdata/iso-2022-jp.c
index 8825dcc0fd..4fba9d993f 100644
--- a/iconvdata/iso-2022-jp.c
+++ b/iconvdata/iso-2022-jp.c
@@ -1,5 +1,5 @@
-/* Conversion module for ISO-2022-JP.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+/* Conversion module for ISO-2022-JP and ISO-2022-JP-2.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -90,14 +90,14 @@ struct iso2022jp_data
enum
{
ASCII_set = 0,
- JISX0208_1978_set = 8,
- JISX0208_1983_set = 16,
- JISX0201_Roman_set = 24,
- JISX0201_Kana_set = 32,
- GB2312_set = 40,
- KSC5601_set = 48,
- JISX0212_set = 56,
- CURRENT_SEL_MASK = 56
+ JISX0208_1978_set = 1 << 3,
+ JISX0208_1983_set = 2 << 3,
+ JISX0201_Roman_set = 3 << 3,
+ JISX0201_Kana_set = 4 << 3,
+ GB2312_set = 5 << 3,
+ KSC5601_set = 6 << 3,
+ JISX0212_set = 7 << 3,
+ CURRENT_SEL_MASK = 7 << 3
};
/* The second value stored is the designation of the G2 set. The following
@@ -105,9 +105,25 @@ enum
enum
{
UNSPECIFIED_set = 0,
- ISO88591_set = 64,
- ISO88597_set = 128,
- CURRENT_ASSIGN_MASK = 192
+ ISO88591_set = 1 << 6,
+ ISO88597_set = 2 << 6,
+ CURRENT_ASSIGN_MASK = 3 << 6
+};
+
+/* The third value, only used during conversion from Unicode to ISO-2022-JP-2,
+ describes the language tag parsing status. The possible values are as
+ follows. Values >= TAG_language are temporary tag parsing states. */
+enum
+{
+ TAG_none = 0,
+ TAG_language = 4 << 8,
+ TAG_language_j = 5 << 8,
+ TAG_language_ja = 1 << 8,
+ TAG_language_k = 6 << 8,
+ TAG_language_ko = 2 << 8,
+ TAG_language_z = 7 << 8,
+ TAG_language_zh = 3 << 8,
+ CURRENT_TAG_MASK = 7 << 8
};
@@ -198,7 +214,8 @@ gconv_end (struct __gconv_step *data)
\
if ((data->__statep->__count & ~7) != ASCII_set) \
{ \
- if (dir == from_iso2022jp) \
+ if (dir == from_iso2022jp \
+ || (data->__statep->__count & CURRENT_SEL_MASK) == ASCII_set) \
{ \
/* It's easy, we don't have to emit anything, we just reset the \
state for the input. Note that this also clears the G2 \
@@ -503,6 +520,23 @@ gconv_end (struct __gconv_step *data)
/* Next, define the other direction. */
+
+enum conversion { none = 0, european, japanese, chinese, korean, other };
+
+/* A datatype for conversion lists. */
+typedef unsigned int cvlist_t;
+#define CVLIST(cv1, cv2, cv3, cv4, cv5) \
+ ((cv1) + ((cv2) << 3) + ((cv3) << 6) + ((cv4) << 9) + ((cv5) << 12))
+#define CVLIST_FIRST(cvl) ((cvl) & ((1 << 3) - 1))
+#define CVLIST_REST(cvl) ((cvl) >> 3)
+static const cvlist_t conversion_lists[4] =
+ {
+ /* TAG_none */ CVLIST (japanese, european, chinese, korean, other),
+ /* TAG_language_ja */ CVLIST (japanese, european, chinese, korean, other),
+ /* TAG_language_ko */ CVLIST (korean, european, japanese, chinese, other),
+ /* TAG_language_zh */ CVLIST (chinese, european, japanese, korean, other)
+ };
+
#define MIN_NEEDED_INPUT MIN_NEEDED_TO
#define MIN_NEEDED_OUTPUT MIN_NEEDED_FROM
#define MAX_NEEDED_OUTPUT (MAX_NEEDED_FROM + 2)
@@ -510,12 +544,56 @@ gconv_end (struct __gconv_step *data)
#define BODY \
{ \
uint32_t ch; \
- size_t written = 0; \
+ size_t written; \
\
ch = get32 (inptr); \
\
+ if (var == iso2022jp2) \
+ { \
+ /* Handle Unicode tag characters (range U+E0000..U+E007F). */ \
+ if (__builtin_expect ((ch >> 7) == (0xe0000 >> 7), 0)) \
+ { \
+ ch &= 0x7f; \
+ if (ch >= 'A' && ch <= 'Z') \
+ ch += 'a' - 'A'; \
+ if (ch == 0x01) \
+ tag = TAG_language; \
+ else if (ch == 'j' && tag == TAG_language) \
+ tag = TAG_language_j; \
+ else if (ch == 'a' && tag == TAG_language_j) \
+ tag = TAG_language_ja; \
+ else if (ch == 'k' && tag == TAG_language) \
+ tag = TAG_language_k; \
+ else if (ch == 'o' && tag == TAG_language_k) \
+ tag = TAG_language_ko; \
+ else if (ch == 'z' && tag == TAG_language) \
+ tag = TAG_language_z; \
+ else if (ch == 'h' && tag == TAG_language_z) \
+ tag = TAG_language_zh; \
+ else if (ch == 0x7f) \
+ tag = TAG_none; \
+ else \
+ { \
+ /* Other tag characters reset the tag parsing state (if the \
+ current state is a temporary state) or are ignored (if \
+ the current state is a stable one). */ \
+ if (tag >= TAG_language) \
+ tag = TAG_none; \
+ } \
+ \
+ inptr += 4; \
+ continue; \
+ } \
+ \
+ /* Non-tag characters reset the tag parsing state, if the current \
+ state is a temporary state. */ \
+ if (__builtin_expect (tag >= TAG_language, 0)) \
+ tag = TAG_none; \
+ } \
+ \
/* First see whether we can write the character using the currently \
- selected character set. */ \
+ selected character set. But ignore the selected character set if \
+ the current language tag shows different preferences. */ \
if (set == ASCII_set) \
{ \
/* Please note that the NUL byte is *not* matched if we are not \
@@ -525,58 +603,75 @@ gconv_end (struct __gconv_step *data)
{ \
*outptr++ = ch; \
written = 1; \
+ \
+ /* At the beginning of a line, G2 designation is cleared. */ \
+ if (var == iso2022jp2 && ch == 0x0a) \
+ set2 = UNSPECIFIED_set; \
} \
- /* At the beginning of a line, G2 designation is cleared. */ \
- if (var == iso2022jp2 && ch == 0x0a) \
- set2 = UNSPECIFIED_set; \
+ else \
+ written = __UNKNOWN_10646_CHAR; \
} \
/* ISO-2022-JP recommends to encode the newline character always in \
ASCII since this allows a context-free interpretation of the \
characters at the beginning of the next line. Otherwise it would \
have to be known whether the last line ended using ASCII or \
JIS X 0201. */ \
- else if (set == JISX0201_Roman_set) \
+ else if (set == JISX0201_Roman_set \
+ && (__builtin_expect (tag == TAG_none, 1) \
+ || tag == TAG_language_ja)) \
{ \
- unsigned char buf[2]; \
+ unsigned char buf[1]; \
written = ucs4_to_jisx0201 (ch, buf); \
- if (written != __UNKNOWN_10646_CHAR && buf[0] > 0x20 \
- && buf[0] < 0x80) \
+ if (written != __UNKNOWN_10646_CHAR) \
{ \
- *outptr++ = buf[0]; \
- written = 1; \
+ if (buf[0] > 0x20 && buf[0] < 0x80) \
+ { \
+ *outptr++ = buf[0]; \
+ written = 1; \
+ } \
+ else \
+ written = __UNKNOWN_10646_CHAR; \
} \
- else \
- written = __UNKNOWN_10646_CHAR; \
} \
- else if (set == JISX0201_Kana_set) \
+ else if (set == JISX0201_Kana_set \
+ && (__builtin_expect (tag == TAG_none, 1) \
+ || tag == TAG_language_ja)) \
{ \
- unsigned char buf[2]; \
+ unsigned char buf[1]; \
written = ucs4_to_jisx0201 (ch, buf); \
- if (written != __UNKNOWN_10646_CHAR && buf[0] > 0xa0 \
- && buf[0] < 0xe0) \
+ if (written != __UNKNOWN_10646_CHAR) \
{ \
- *outptr++ = buf[0] - 0x80; \
- written = 1; \
+ if (buf[0] > 0xa0 && buf[0] < 0xe0) \
+ { \
+ *outptr++ = buf[0] - 0x80; \
+ written = 1; \
+ } \
+ else \
+ written = __UNKNOWN_10646_CHAR; \
} \
- else \
- written = __UNKNOWN_10646_CHAR; \
} \
else \
{ \
- if (set == JISX0208_1978_set || set == JISX0208_1983_set) \
+ if ((set == JISX0208_1978_set || set == JISX0208_1983_set) \
+ && (__builtin_expect (tag == TAG_none, 1) \
+ || tag == TAG_language_ja)) \
written = ucs4_to_jisx0208 (ch, outptr, outend - outptr); \
- else if (set == JISX0212_set) \
+ else if (set == JISX0212_set \
+ && (__builtin_expect (tag == TAG_none, 1) \
+ || tag == TAG_language_ja)) \
written = ucs4_to_jisx0212 (ch, outptr, outend - outptr); \
- else if (set == GB2312_set) \
+ else if (set == GB2312_set \
+ && (__builtin_expect (tag == TAG_none, 1) \
+ || tag == TAG_language_zh)) \
written = ucs4_to_gb2312 (ch, outptr, outend - outptr); \
+ else if (set == KSC5601_set \
+ && (__builtin_expect (tag == TAG_none, 1) \
+ || tag == TAG_language_ko)) \
+ written = ucs4_to_ksc5601 (ch, outptr, outend - outptr); \
else \
- { \
- assert (set == KSC5601_set); \
- \
- written = ucs4_to_ksc5601 (ch, outptr, outend - outptr); \
- } \
+ written = __UNKNOWN_10646_CHAR; \
\
- if (__builtin_expect (written, 1) == 0) \
+ if (__builtin_expect (written == 0, 0)) \
{ \
result = __GCONV_FULL_OUTPUT; \
break; \
@@ -585,18 +680,19 @@ gconv_end (struct __gconv_step *data)
outptr += written; \
} \
\
- if (written == __UNKNOWN_10646_CHAR || written == 0) \
+ if (written == __UNKNOWN_10646_CHAR \
+ && __builtin_expect (tag == TAG_none, 1)) \
{ \
if (set2 == ISO88591_set) \
{ \
- if (__builtin_expect (outptr + 3 > outend, 0)) \
+ if (ch >= 0x80 && ch <= 0xff) \
{ \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
+ if (__builtin_expect (outptr + 3 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
\
- if (ch >= 0x80 && ch <= 0xff) \
- { \
*outptr++ = ESC; \
*outptr++ = 'N'; \
*outptr++ = ch & 0x7f; \
@@ -629,18 +725,15 @@ gconv_end (struct __gconv_step *data)
} \
} \
\
- if (written == __UNKNOWN_10646_CHAR || written == 0) \
+ if (written == __UNKNOWN_10646_CHAR) \
{ \
- /* Either this is an unknown character or we have to switch \
- the currently selected character set. The character sets \
- do not code entirely separate parts of ISO 10646 and \
- therefore there is no single correct result. If we choose \
- the character set to use wrong we might be end up with \
- using yet another character set for the next character \
- though the current and the next could be encoded with one \
- character set. We leave this kind of optimization for \
- later and now simply use a fixed order in which we test for \
- availability */ \
+ /* The attempts to use the currently selected character set \
+ failed, either because the language tag changed, or because \
+ the character requires a different character set, or because \
+ the character is unknown. \
+ The CJK character sets partially overlap when seen as subsets \
+ of ISO 10646; therefore there is no single correct result. \
+ We use a preferrence order which depends on the language tag. */ \
\
if (ch <= 0x7f) \
{ \
@@ -671,235 +764,298 @@ gconv_end (struct __gconv_step *data)
else \
{ \
/* Now it becomes difficult. We must search the other \
- character sets one by one and we cannot use simple \
- arithmetic to determine whether the character can be \
- encoded using this set. */ \
- size_t written; \
+ character sets one by one. Use an ordered conversion \
+ list that depends on the current language tag. */ \
+ cvlist_t conversion_list; \
unsigned char buf[2]; \
\
- written = ucs4_to_jisx0201 (ch, buf); \
- if (written != __UNKNOWN_10646_CHAR && buf[0] < 0x80) \
- { \
- /* We use JIS X 0201. */ \
- if (__builtin_expect (outptr + 3 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- \
- *outptr++ = ESC; \
- *outptr++ = '('; \
- *outptr++ = 'J'; \
- set = JISX0201_Roman_set; \
+ result = __GCONV_ILLEGAL_INPUT; \
\
- if (__builtin_expect (outptr + 1 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = buf[0]; \
- } \
+ if (var == iso2022jp2) \
+ conversion_list = conversion_lists[tag >> 8]; \
else \
- { \
- written = ucs4_to_jisx0208 (ch, buf, 2); \
- if (written != __UNKNOWN_10646_CHAR) \
- { \
- /* We use JIS X 0208. */ \
- if (__builtin_expect (outptr + 3 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
+ conversion_list = CVLIST (japanese, 0, 0, 0, 0); \
\
- *outptr++ = ESC; \
- *outptr++ = '$'; \
- *outptr++ = 'B'; \
- set = JISX0208_1983_set; \
+ do \
+ switch (CVLIST_FIRST (conversion_list)) \
+ { \
+ case european: \
\
- if (__builtin_expect (outptr + 2 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = buf[0]; \
- *outptr++ = buf[1]; \
- } \
- else if (__builtin_expect (var, iso2022jp2) == iso2022jp) \
- { \
- /* We have no other choice. */ \
- STANDARD_ERR_HANDLER (4); \
- } \
- else \
- { \
- written = ucs4_to_jisx0212 (ch, buf, 2); \
- if (written != __UNKNOWN_10646_CHAR) \
- { \
- /* We use JIS X 0212. */ \
- if (__builtin_expect (outptr + 4 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = ESC; \
- *outptr++ = '$'; \
- *outptr++ = '('; \
- *outptr++ = 'D'; \
- set = JISX0212_set; \
- \
- if (__builtin_expect (outptr + 2 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = buf[0]; \
- *outptr++ = buf[1]; \
- } \
- else \
- { \
- written = ucs4_to_jisx0201 (ch, buf); \
- if (written != __UNKNOWN_10646_CHAR \
- && buf[0] >= 0x80) \
- { \
- /* We use JIS X 0201. */ \
- if (__builtin_expect (outptr + 3 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
+ /* Try ISO 8859-1 upper half. */ \
+ if (ch >= 0x80 && ch <= 0xff) \
+ { \
+ if (set2 != ISO88591_set) \
+ { \
+ if (__builtin_expect (outptr + 3 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = ESC; \
+ *outptr++ = '.'; \
+ *outptr++ = 'A'; \
+ set2 = ISO88591_set; \
+ } \
\
- *outptr++ = ESC; \
- *outptr++ = '('; \
- *outptr++ = 'I'; \
- set = JISX0201_Kana_set; \
+ if (__builtin_expect (outptr + 3 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = ESC; \
+ *outptr++ = 'N'; \
+ *outptr++ = ch - 0x80; \
+ result = __GCONV_OK; \
+ break; \
+ } \
\
- if (__builtin_expect (outptr + 1 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = buf[0] - 0x80; \
- } \
- else if (ch != 0xa5 && ch >= 0x80 && ch <= 0xff) \
- { \
- /* ISO 8859-1 upper half. */ \
- if (__builtin_expect (outptr + 3 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- \
- *outptr++ = ESC; \
- *outptr++ = '.'; \
- *outptr++ = 'A'; \
- set2 = ISO88591_set; \
+ /* Try ISO 8859-7 upper half. */ \
+ { \
+ const struct gap *rp = from_idx; \
\
- if (__builtin_expect (outptr + 3 > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = ESC; \
- *outptr++ = 'N'; \
- *outptr++ = ch - 0x80; \
- } \
- else \
+ while (ch > rp->end) \
+ ++rp; \
+ if (ch >= rp->start) \
+ { \
+ unsigned char res = \
+ iso88597_from_ucs4[ch - 0xa0 + rp->idx]; \
+ if (res != '\0') \
{ \
- written = ucs4_to_gb2312 (ch, buf, 2); \
- if (written != __UNKNOWN_10646_CHAR) \
+ if (set2 != ISO88597_set) \
{ \
- /* We use GB 2312. */ \
if (__builtin_expect (outptr + 3 > outend, 0))\
{ \
result = __GCONV_FULL_OUTPUT; \
break; \
} \
- \
*outptr++ = ESC; \
- *outptr++ = '$'; \
- *outptr++ = 'A'; \
- set = GB2312_set; \
- \
- if (__builtin_expect (outptr + 2 > outend, 0))\
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = buf[0]; \
- *outptr++ = buf[1]; \
+ *outptr++ = '.'; \
+ *outptr++ = 'F'; \
+ set2 = ISO88597_set; \
} \
- else \
+ \
+ if (__builtin_expect (outptr + 3 > outend, 0)) \
{ \
- written = ucs4_to_ksc5601 (ch, buf, 2); \
- if (written != __UNKNOWN_10646_CHAR) \
- { \
- /* We use KSC 5601. */ \
- if (__builtin_expect (outptr + 4 > outend,\
- 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = ESC; \
- *outptr++ = '$'; \
- *outptr++ = '('; \
- *outptr++ = 'C'; \
- set = KSC5601_set; \
- \
- if (__builtin_expect (outptr + 2 > outend,\
- 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = buf[0]; \
- *outptr++ = buf[1]; \
- } \
- else \
- { \
- const struct gap *rp = from_idx; \
- unsigned char gch = 0; \
- \
- while (ch > rp->end) \
- ++rp; \
- if (ch >= rp->start) \
- { \
- ch = ch - 0xa0 + rp->idx; \
- gch = iso88597_from_ucs4[ch]; \
- } \
- \
- if (__builtin_expect (gch, 1) != 0) \
- { \
- /* We use ISO 8859-7 greek. */ \
- if (__builtin_expect (outptr + 3 \
- > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = ESC; \
- *outptr++ = '.'; \
- *outptr++ = 'F'; \
- set2 = ISO88597_set; \
- \
- if (__builtin_expect (outptr + 3 \
- > outend, 0)) \
- { \
- result = __GCONV_FULL_OUTPUT; \
- break; \
- } \
- *outptr++ = ESC; \
- *outptr++ = 'N'; \
- *outptr++ = gch; \
- } \
- else \
- { \
- STANDARD_ERR_HANDLER (4); \
- } \
- } \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
} \
+ *outptr++ = ESC; \
+ *outptr++ = 'N'; \
+ *outptr++ = res; \
+ result = __GCONV_OK; \
+ break; \
} \
} \
} \
+ \
+ break; \
+ \
+ case japanese: \
+ \
+ /* Try JIS X 0201 Roman. */ \
+ written = ucs4_to_jisx0201 (ch, buf); \
+ if (written != __UNKNOWN_10646_CHAR \
+ && buf[0] > 0x20 && buf[0] < 0x80) \
+ { \
+ if (set != JISX0201_Roman_set) \
+ { \
+ if (__builtin_expect (outptr + 3 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = ESC; \
+ *outptr++ = '('; \
+ *outptr++ = 'J'; \
+ set = JISX0201_Roman_set; \
+ } \
+ \
+ if (__builtin_expect (outptr + 1 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = buf[0]; \
+ result = __GCONV_OK; \
+ break; \
+ } \
+ \
+ /* Try JIS X 0208. */ \
+ written = ucs4_to_jisx0208 (ch, buf, 2); \
+ if (written != __UNKNOWN_10646_CHAR) \
+ { \
+ if (set != JISX0208_1983_set) \
+ { \
+ if (__builtin_expect (outptr + 3 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = ESC; \
+ *outptr++ = '$'; \
+ *outptr++ = 'B'; \
+ set = JISX0208_1983_set; \
+ } \
+ \
+ if (__builtin_expect (outptr + 2 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = buf[0]; \
+ *outptr++ = buf[1]; \
+ result = __GCONV_OK; \
+ break; \
+ } \
+ \
+ if (__builtin_expect (var == iso2022jp, 0)) \
+ /* Don't use the other Japanese character sets. */ \
+ break; \
+ \
+ /* Try JIS X 0212. */ \
+ written = ucs4_to_jisx0212 (ch, buf, 2); \
+ if (written != __UNKNOWN_10646_CHAR) \
+ { \
+ if (set != JISX0212_set) \
+ { \
+ if (__builtin_expect (outptr + 4 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = ESC; \
+ *outptr++ = '$'; \
+ *outptr++ = '('; \
+ *outptr++ = 'D'; \
+ set = JISX0212_set; \
+ } \
+ \
+ if (__builtin_expect (outptr + 2 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = buf[0]; \
+ *outptr++ = buf[1]; \
+ result = __GCONV_OK; \
+ break; \
+ } \
+ \
+ break; \
+ \
+ case chinese: \
+ assert (var == iso2022jp2); \
+ \
+ /* Try GB 2312. */ \
+ written = ucs4_to_gb2312 (ch, buf, 2); \
+ if (written != __UNKNOWN_10646_CHAR) \
+ { \
+ if (set != GB2312_set) \
+ { \
+ if (__builtin_expect (outptr + 3 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = ESC; \
+ *outptr++ = '$'; \
+ *outptr++ = 'A'; \
+ set = GB2312_set; \
+ } \
+ \
+ if (__builtin_expect (outptr + 2 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = buf[0]; \
+ *outptr++ = buf[1]; \
+ result = __GCONV_OK; \
+ break; \
+ } \
+ \
+ break; \
+ \
+ case korean: \
+ assert (var == iso2022jp2); \
+ \
+ /* Try KSC 5601. */ \
+ written = ucs4_to_ksc5601 (ch, buf, 2); \
+ if (written != __UNKNOWN_10646_CHAR) \
+ { \
+ if (set != KSC5601_set) \
+ { \
+ if (__builtin_expect (outptr + 4 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = ESC; \
+ *outptr++ = '$'; \
+ *outptr++ = '('; \
+ *outptr++ = 'C'; \
+ set = KSC5601_set; \
+ } \
+ \
+ if (__builtin_expect (outptr + 2 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = buf[0]; \
+ *outptr++ = buf[1]; \
+ result = __GCONV_OK; \
+ break; \
+ } \
+ \
+ break; \
+ \
+ case other: \
+ assert (var == iso2022jp2); \
+ \
+ /* Try JIS X 0201 Kana. This is not officially part \
+ of ISO-2022-JP-2, according to RFC 1554. Therefore \
+ we try this only after all other attempts. */ \
+ written = ucs4_to_jisx0201 (ch, buf); \
+ if (written != __UNKNOWN_10646_CHAR && buf[0] >= 0x80) \
+ { \
+ if (set != JISX0201_Kana_set) \
+ { \
+ if (__builtin_expect (outptr + 3 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = ESC; \
+ *outptr++ = '('; \
+ *outptr++ = 'I'; \
+ set = JISX0201_Kana_set; \
+ } \
+ \
+ if (__builtin_expect (outptr + 1 > outend, 0)) \
+ { \
+ result = __GCONV_FULL_OUTPUT; \
+ break; \
+ } \
+ *outptr++ = buf[0] - 0x80; \
+ result = __GCONV_OK; \
+ break; \
+ } \
+ \
+ break; \
+ \
+ default: \
+ abort (); \
+ } \
+ while (result == __GCONV_ILLEGAL_INPUT \
+ && (conversion_list = CVLIST_REST (conversion_list)) != 0);\
+ \
+ if (result == __GCONV_FULL_OUTPUT) \
+ break; \
+ \
+ if (result == __GCONV_ILLEGAL_INPUT) \
+ { \
+ STANDARD_ERR_HANDLER (4); \
} \
} \
} \
@@ -910,8 +1066,9 @@ gconv_end (struct __gconv_step *data)
#define LOOP_NEED_FLAGS
#define EXTRA_LOOP_DECLS , enum variant var, int *setp
#define INIT_PARAMS int set = *setp & CURRENT_SEL_MASK; \
- int set2 = *setp & CURRENT_ASSIGN_MASK
-#define UPDATE_PARAMS *setp = set | set2
+ int set2 = *setp & CURRENT_ASSIGN_MASK; \
+ int tag = *setp & CURRENT_TAG_MASK;
+#define UPDATE_PARAMS *setp = set | set2 | tag
#include <iconv/loop.c>
diff --git a/iconvdata/iso-2022-kr.c b/iconvdata/iso-2022-kr.c
index 6547768ab2..4033dc0632 100644
--- a/iconvdata/iso-2022-kr.c
+++ b/iconvdata/iso-2022-kr.c
@@ -1,5 +1,5 @@
/* Conversion module for ISO-2022-KR.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -221,7 +221,6 @@ enum
#define BODY \
{ \
uint32_t ch; \
- size_t written = 0; \
\
ch = get32 (inptr); \
\
@@ -241,16 +240,18 @@ enum
} \
\
*outptr++ = ch; \
- written = 1; \
} \
else \
{ \
char buf[2]; \
+ size_t written; \
\
written = ucs4_to_ksc5601 (ch, buf, 2); \
\
if (__builtin_expect (written, 0) == __UNKNOWN_10646_CHAR) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Illegal character. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/iso646.c b/iconvdata/iso646.c
index 8de2639574..6668d9db7b 100644
--- a/iconvdata/iso646.c
+++ b/iconvdata/iso646.c
@@ -1,5 +1,5 @@
/* Conversion to and from the various ISO 646 CCS.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -879,7 +879,10 @@ gconv_end (struct __gconv_step *data)
break; \
default: \
if (__builtin_expect (ch, 0) > 0x7f) \
- failure = __GCONV_ILLEGAL_INPUT; \
+ { \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ failure = __GCONV_ILLEGAL_INPUT; \
+ } \
break; \
} \
\
diff --git a/iconvdata/iso8859-1.c b/iconvdata/iso8859-1.c
index 9e5007f77d..7b2ed45c3a 100644
--- a/iconvdata/iso8859-1.c
+++ b/iconvdata/iso8859-1.c
@@ -1,5 +1,5 @@
/* Conversion to and from ISO 8859-1.
- Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
@@ -48,6 +48,8 @@
uint32_t ch = *((const uint32_t *) inptr); \
if (__builtin_expect (ch, 0) > 0xff) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* We have an illegal character. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/iso_6937-2.c b/iconvdata/iso_6937-2.c
index 5f8956e6b5..0c2d6d251b 100644
--- a/iconvdata/iso_6937-2.c
+++ b/iconvdata/iso_6937-2.c
@@ -1,5 +1,5 @@
/* Generic conversion to and from ISO 6937-2.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -563,6 +563,7 @@ static const char from_ucs4[][2] =
cp = "\xd5"; \
break; \
default: \
+ UNICODE_TAG_HANDLER (ch, 4); \
/* Illegal characters. */ \
cp = NULL; \
break; \
diff --git a/iconvdata/iso_6937.c b/iconvdata/iso_6937.c
index 74b899005a..e99bfa9ed9 100644
--- a/iconvdata/iso_6937.c
+++ b/iconvdata/iso_6937.c
@@ -1,5 +1,5 @@
/* Generic conversion to and from ISO 6937.
- Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
@@ -536,6 +536,7 @@ static const char from_ucs4[][2] =
cp = "\xd5"; \
break; \
default: \
+ UNICODE_TAG_HANDLER (ch, 4); \
cp = NULL; \
fail = 1; \
} \
diff --git a/iconvdata/johab.c b/iconvdata/johab.c
index a631140f09..b92906cbaf 100644
--- a/iconvdata/johab.c
+++ b/iconvdata/johab.c
@@ -1,5 +1,5 @@
/* Mapping tables for JOHAB handling.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Jungshik Shin <jshin@pantheon.yale.edu>
and Ulrich Drepper <drepper@cygnus.com>, 1998.
@@ -426,6 +426,7 @@ johab_sym_hanja_to_ucs (uint_fast32_t idx, uint_fast32_t c1, uint_fast32_t c2)
if (__builtin_expect (written, 1) == __UNKNOWN_10646_CHAR \
|| (outptr[0] == 0x22 && outptr[1] > 0x65)) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
STANDARD_ERR_HANDLER (4); \
} \
\
diff --git a/iconvdata/sjis.c b/iconvdata/sjis.c
index 7bbfef3e5d..cb5014ccb3 100644
--- a/iconvdata/sjis.c
+++ b/iconvdata/sjis.c
@@ -1,5 +1,5 @@
/* Mapping tables for SJIS handling.
- Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
@@ -4471,8 +4471,11 @@ static const char from_ucs4_extra[0x100][2] =
&& __builtin_expect (ch, 0xff01) <= 0xffef) \
cp = from_ucs4_extra[ch - 0xff00]; \
else \
- /* Illegal character. */ \
- cp = ""; \
+ { \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ /* Illegal character. */ \
+ cp = ""; \
+ } \
} \
else \
cp = from_ucs4_lat1[ch]; \
diff --git a/iconvdata/t.61.c b/iconvdata/t.61.c
index 302226d8cc..4dab974012 100644
--- a/iconvdata/t.61.c
+++ b/iconvdata/t.61.c
@@ -1,5 +1,5 @@
/* Generic conversion to and from T.61.
- Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
@@ -468,6 +468,8 @@ static const char from_ucs4[][2] =
|| __builtin_expect (ch, 0x2d8) > 0x2dd \
|| __builtin_expect (ch, 0x2d8) == 0x02dc) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
+ \
/* Illegal characters. */ \
STANDARD_ERR_HANDLER (4); \
} \
diff --git a/iconvdata/tst-table-charmap.sh b/iconvdata/tst-table-charmap.sh
index 99c713c363..889595c8b7 100755
--- a/iconvdata/tst-table-charmap.sh
+++ b/iconvdata/tst-table-charmap.sh
@@ -1,5 +1,5 @@
#!/bin/sh
-# Copyright (C) 2000 Free Software Foundation, Inc.
+# Copyright (C) 2000, 2001 Free Software Foundation, Inc.
# This file is part of the GNU C Library.
# Contributed by Bruno Haible <haible@clisp.cons.org>, 2000.
#
@@ -30,6 +30,6 @@ case "$1" in
;;
*)
# New Unicode based format
- sed -e 's,^%IRREVERSIBLE%,,' | grep '^<U....>[ ]*/x' | grep -v 'not a real character' | sed -e 's,<U\(....\)>[ ]*\([/x0-9A-Fa-f]*\).*$,\2 0x\1,' | tr abcdef ABCDEF | sed -e 's,/x\([0-9A-F][0-9A-F]\),\1,g' | sed -e 's,^,0x,' | sort | uniq | grep -v '^0x00 0x\([1-9A-F]...\|.[1-9A-F]..\|..[1-9A-F].\|...[1-9A-F]\)'
+ sed -e 's,^%IRREVERSIBLE%,,' | grep '^<U[0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F]*>[ ]*/x' | grep -v 'not a real character' | sed -e 's,<U\(....\)>[ ]*\([/x0-9A-Fa-f]*\).*$,\2 0x\1,' -e 's,<U0*\([1-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F]*\)>[ ]*\([/x0-9A-Fa-f]*\).*$,\2 0x\1,' | tr abcdef ABCDEF | sed -e 's,/x\([0-9A-F][0-9A-F]\),\1,g' | sed -e 's,^,0x,' | sort | uniq | grep -v '^0x00 0x\([1-9A-F]...\|.[1-9A-F]..\|..[1-9A-F].\|...[1-9A-F]\)'
;;
esac
diff --git a/iconvdata/tst-table-from.c b/iconvdata/tst-table-from.c
index b24632e1b9..067c600892 100644
--- a/iconvdata/tst-table-from.c
+++ b/iconvdata/tst-table-from.c
@@ -28,6 +28,9 @@
#include <iconv.h>
#include <errno.h>
+/* If nonzero, ignore conversions outside Unicode plane 0. */
+static int bmp_only;
+
/* Converts a byte buffer to a hexadecimal string. */
static const char*
hexbuf (unsigned char buf[], unsigned int buflen)
@@ -162,6 +165,10 @@ utf8_decode (const unsigned char *out, unsigned int outlen)
out += 1; outlen -= 1;
}
+ if (bmp_only && strlen (p) > 6)
+ /* Ignore conversions outside Unicode plane 0. */
+ return NULL;
+
p += strlen (p);
}
@@ -173,6 +180,7 @@ main (int argc, char *argv[])
{
const char *charset;
iconv_t cd;
+ int search_depth;
if (argc != 2)
{
@@ -188,6 +196,12 @@ main (int argc, char *argv[])
exit (1);
}
+ /* When testing UTF-8 or GB18030, stop at 0x10000, otherwise the output
+ file gets too big. */
+ bmp_only = (strcmp (charset, "UTF-8") == 0
+ || strcmp (charset, "GB18030") == 0);
+ search_depth = (strcmp (charset, "UTF-8") == 0 ? 3 : 4);
+
{
unsigned char out[6];
unsigned char buf[4];
@@ -203,8 +217,9 @@ main (int argc, char *argv[])
}
else if (result > 0)
{
- printf ("0x%02X\t%s\n",
- i0, utf8_decode (out, result));
+ const char *unicode = utf8_decode (out, result);
+ if (unicode != NULL)
+ printf ("0x%02X\t%s\n", i0, unicode);
}
else
{
@@ -217,8 +232,9 @@ main (int argc, char *argv[])
}
else if (result > 0)
{
- printf ("0x%02X%02X\t%s\n",
- i0, i1, utf8_decode (out, result));
+ const char *unicode = utf8_decode (out, result);
+ if (unicode != NULL)
+ printf ("0x%02X%02X\t%s\n", i0, i1, unicode);
}
else
{
@@ -231,10 +247,12 @@ main (int argc, char *argv[])
}
else if (result > 0)
{
- printf ("0x%02X%02X%02X\t%s\n",
- i0, i1, i2, utf8_decode (out, result));
+ const char *unicode = utf8_decode (out, result);
+ if (unicode != NULL)
+ printf ("0x%02X%02X%02X\t%s\n",
+ i0, i1, i2, unicode);
}
- else if (strcmp (charset, "UTF-8"))
+ else if (search_depth > 3)
{
for (i3 = 0; i3 < 0x100; i3++)
{
@@ -245,9 +263,11 @@ main (int argc, char *argv[])
}
else if (result > 0)
{
- printf ("0x%02X%02X%02X%02X\t%s\n",
- i0, i1, i2, i3,
- utf8_decode (out, result));
+ const char *unicode =
+ utf8_decode (out, result);
+ if (unicode != NULL)
+ printf ("0x%02X%02X%02X%02X\t%s\n",
+ i0, i1, i2, i3, unicode);
}
else
{
diff --git a/iconvdata/tst-table-to.c b/iconvdata/tst-table-to.c
index c763c845c9..0a6ba8a8da 100644
--- a/iconvdata/tst-table-to.c
+++ b/iconvdata/tst-table-to.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 2000 Free Software Foundation, Inc.
+/* Copyright (C) 2000-2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Bruno Haible <haible@clisp.cons.org>, 2000.
@@ -32,6 +32,7 @@ main (int argc, char *argv[])
{
const char *charset;
iconv_t cd;
+ int bmp_only;
if (argc != 2)
{
@@ -40,22 +41,38 @@ main (int argc, char *argv[])
}
charset = argv[1];
- cd = iconv_open (charset, "UCS-2");
+ cd = iconv_open (charset, "UTF-8");
if (cd == (iconv_t)(-1))
{
perror ("iconv_open");
return 1;
}
+ /* When testing UTF-8 or GB18030, stop at 0x10000, otherwise the output
+ file gets too big. */
+ bmp_only = (strcmp (charset, "UTF-8") == 0
+ || strcmp (charset, "GB18030") == 0);
+
{
unsigned int i;
unsigned char buf[10];
- for (i = 0; i < 0x10000; i++)
+ for (i = 0; i < (bmp_only ? 0x10000 : 0x30000); i++)
{
- unsigned short in = i;
- const char *inbuf = (const char *) &in;
- size_t inbytesleft = sizeof (unsigned short);
+ unsigned char in[6];
+ unsigned int incount =
+ (i < 0x80 ? (in[0] = i, 1)
+ : i < 0x800 ? (in[0] = 0xc0 | (i >> 6),
+ in[1] = 0x80 | (i & 0x3f), 2)
+ : i < 0x10000 ? (in[0] = 0xe0 | (i >> 12),
+ in[1] = 0x80 | ((i >> 6) & 0x3f),
+ in[2] = 0x80 | (i & 0x3f), 3)
+ : /* i < 0x200000 */ (in[0] = 0xf0 | (i >> 18),
+ in[1] = 0x80 | ((i >> 12) & 0x3f),
+ in[2] = 0x80 | ((i >> 6) & 0x3f),
+ in[3] = 0x80 | (i & 0x3f), 4));
+ const char *inbuf = (const char *) in;
+ size_t inbytesleft = incount;
char *outbuf = (char *) buf;
size_t outbytesleft = sizeof (buf);
size_t result = iconv (cd,
@@ -78,7 +95,7 @@ main (int argc, char *argv[])
if (inbytesleft != 0 || outbytesleft == sizeof (buf))
{
fprintf (stderr, "0x%02X: inbytes = %ld, outbytes = %ld\n", i,
- (long) (sizeof (unsigned short) - inbytesleft),
+ (long) (incount - inbytesleft),
(long) (sizeof (buf) - outbytesleft));
return 1;
}
diff --git a/iconvdata/tst-table.sh b/iconvdata/tst-table.sh
index e38b88ee37..f00d3eef3e 100755
--- a/iconvdata/tst-table.sh
+++ b/iconvdata/tst-table.sh
@@ -1,5 +1,5 @@
#!/bin/sh
-# Copyright (C) 2000 Free Software Foundation, Inc.
+# Copyright (C) 2000, 2001 Free Software Foundation, Inc.
# This file is part of the GNU C Library.
# Contributed by Bruno Haible <haible@clisp.cons.org>, 2000.
#
@@ -61,8 +61,15 @@ diff ${objpfx}tst-${charset}.table ${objpfx}tst-${charset}.inverse.table | \
grep '^[<>]' | sed -e 's,^. ,,' > ${objpfx}tst-${charset}.irreversible.table
# Check 1: charmap and iconv forward should be identical.
-cmp -s ${objpfx}tst-${charset}.charmap.table ${objpfx}tst-${charset}.table ||
-exit 1
+if test ${charset} = GB18030; then
+ grep '0x....$' < ${objpfx}tst-${charset}.charmap.table \
+ > ${objpfx}tst-${charset}.truncated.table
+ cmp -s ${objpfx}tst-${charset}.truncated.table ${objpfx}tst-${charset}.table ||
+ exit 1
+else
+ cmp -s ${objpfx}tst-${charset}.charmap.table ${objpfx}tst-${charset}.table ||
+ exit 1
+fi
# Check 2: the difference between the two iconv directions.
if test -f ${irreversible}; then
diff --git a/iconvdata/uhc.c b/iconvdata/uhc.c
index c191b1127e..c3ff69937e 100644
--- a/iconvdata/uhc.c
+++ b/iconvdata/uhc.c
@@ -1,5 +1,5 @@
/* Mapping tables for UHC handling.
- Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Jungshik Shin <jshin@pantheon.yale.edu>, 1998.
@@ -3243,6 +3243,7 @@ static const char uhc_hangul_from_ucs[11172][2] =
} \
if (__builtin_expect (written, 0) == __UNKNOWN_10646_CHAR) \
{ \
+ UNICODE_TAG_HANDLER (ch, 4); \
STANDARD_ERR_HANDLER (4); \
} \
\
diff --git a/iconvdata/unicode.c b/iconvdata/unicode.c
index 89ec81f90b..f2f46c6c77 100644
--- a/iconvdata/unicode.c
+++ b/iconvdata/unicode.c
@@ -1,5 +1,5 @@
/* Conversion module for Unicode
- Copyright (C) 1999, 2000 Free Software Foundation, Inc.
+ Copyright (C) 1999, 2000, 2001 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@cygnus.com>, 1999.
@@ -154,6 +154,7 @@ gconv_end (struct __gconv_step *data)
\
if (__builtin_expect (c, 0) >= 0x10000) \
{ \
+ UNICODE_TAG_HANDLER (c, 4); \
STANDARD_ERR_HANDLER (4); \
} \
else if (__builtin_expect (c >= 0xd800 && c < 0xe000, 0)) \