-*-mode: text; coding: utf-8;-*- Copyright (C) 2002-2018 Free Software Foundation, Inc. See the end of the file for license conditions. Importing a new Unicode Standard version into Emacs ------------------------------------------------------------- Emacs uses the following files from the Unicode Character Database (a.k.a. "UCD): . UnicodeData.txt . Blocks.txt . BidiMirroring.txt . BidiBrackets.txt . IVD_Sequences.txt . NormalizationTest.txt . SpecialCasing.txt . BidiCharacterTest.txt First, the first 7 files need to be copied into admin/unidata/, and then Emacs should be rebuilt for them to take effect. Rebuilding Emacs updates several derived files elsewhere in the Emacs source tree, mainly in lisp/international/. When Emacs is rebuilt for the first time after importing the new files, pay attention to any warning or error messages. In particular, admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines new bidirectional attributes of characters, because unidata-gen.el, bidi.c and dispextern.h need to be updated in that case; failure to do so will cause aborts in redisplay. Next, review the changes in UnicodeData.txt vs the previous version used by Emacs. Any changes, be it introduction of new scripts or addition of codepoints to existing scripts, might need corresponding changes in the data used for filling the category-table, case-table, and char-width-table. The additional scripts should cause automatic updates in charscript.el, but it is a good idea to look at the results and see if any changes in admin/unidata/blocks.awk are required. The setting of char-width-table around line 1200 of characters.el should be checked against the latest version of the Unicode file EastAsianWidth.txt, and any discrepancies fixed. Any new scripts added by UnicodeData.txt will also need updates to script-representative-chars defined in fontset.el, and also the list of OTF script tags in otf-script-alist, whose source is on this page: https://www.microsoft.com/typography/otspec/scripttags.htm Other databases in fontset.el might also need to be updated as needed. The function 'ucs-names', defined in lisp/international/mule-cmds.el, might need to be updated because it knows about used and unused ranges of Unicode codepoints, which a new release of the Unicode Standard could change. Next, test normalization functions against NormalizationTests.txt, in the test/ directory run: make lisp/international/ucs-normalize-tests See commentary in test/lisp/international/ucs-normalize-tests.el regarding failing lines. The file BidiCharacterTest.txt should be copied to the test suite, and if its format has changed, the file biditest.el there should be modified to follow suit. Problems, fixmes and other unicode-related issues ------------------------------------------------------------- Notes by fx to record various things of variable importance. Handa needs to check them -- don't take too seriously, especially with regard to completeness. * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has undesirable effects. E.g.: (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil (multibyte-string-p (concat [?£])) => nil (text-char-description ?£) => "M-#" These examples are all fixed by the change of 2002-10-14, but there still exist questionable SINGLE_BYTE_CHAR_P in the code (keymap.c and print.c). * Rationalize character syntax and its relationship to the Unicode database. (Applies mainly to symbol an punctuation syntax.) * Fontset handling and customization needs work. We want to relate fonts to scripts, probably based on the Unicode blocks. The presence of small-repertoire 10646-encoded fonts in XFree 4 is a pain, not currently worked round. With the change on 2002-07-26, multiple fonts can be specified in a fontset for a specific range of characters. Each range can also be specified by script. Before using ISO10646 fonts, Emacs checks their repertories to avoid such fonts that don't have a glyph for a specific character. fx has worked on fontset customization, but was stymied by basic problems with the way the default face is dealt with (and something else, I think). This needs revisiting. * Work is also needed on charset and coding system priorities. * The relevant bits of latin1-disp.el need porting (and probably re-naming/updating). See also cyril-util.el. * Quail files need more work now the encoding is largely irrelevant. * What to do with the old coding categories stuff? * The preferred-coding-system property of charsets should probably be junked unless it can be made more useful now. * find-multibyte-characters needs looking at. * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing charsets. * Lazy-load tables for unify-charset somehow? Actually, Emacs clears out all charset maps and unify-map just before dumping, and they are loaded again on demand by the dumped emacs. But, those maps (char tables) generated while temacs is running can't be removed from the dumped emacs. * iso-2022 charsets get unified on i/o. With the change on 2003-01-06, decoding routines put the 'charset' property onto decoded text, and iso-2022 encoder pay attention to it. Thus, for instance, reading and writing by iso-2022-7bit preserve the original designation sequences. The property name 'preferred-charset' may be better? We may have to utilize this property to decide a font. * Revisit locale processing: look at treating the language and charset parts separately. (Language should affect things like spelling and calendar, but that's not a Unicode issue.) * Handle Unicode combining characters usefully, e.g. diacritics, and handle more scripts specifically (à la Devanagari). There are issues with canonicalization. * We need tabular input methods, e.g. for maths symbols. (Not specific to Unicode.) * Need multibyte text in menus, e.g. for the above. (Not specific to Unicode -- see Emacs etc/TODO, but now mostly works with gtk.) * Populate char-width-table correctly for Unicode characters and worry about what happens when double-width charsets covering non-CJK characters are unified. * There are type errors lurking, e.g. in Fcheck_coding_systems_region. Define ENABLE_CHECKING to find them. * Old auto-save files, and similar files, such as Gnus drafts, containing non-ASCII characters probably won't be re-read correctly. Source file encoding -------------------- Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a subset), but there are a few exceptions, listed below. Perhaps someday many of these files will be converted to UTF-8, for convenience when using tools like 'grep -r', but this might need nontrivial changes to the build process. * chinese-big5 These are verbatim copies of files taken from external sources. They haven't been converted to UTF-8. leim/CXTERM-DIC/4Corner.tit leim/CXTERM-DIC/ARRAY30.tit leim/CXTERM-DIC/ECDICT.tit leim/CXTERM-DIC/ETZY.tit leim/CXTERM-DIC/PY-b5.tit leim/CXTERM-DIC/Punct-b5.tit leim/CXTERM-DIC/QJ-b5.tit leim/CXTERM-DIC/ZOZY.tit leim/MISC-DIC/CTLau-b5.html leim/MISC-DIC/cangjie-table.b5 * chinese-iso-8bit These are verbatim copies of files taken from external sources. They haven't been converted to UTF-8. leim/CXTERM-DIC/CCDOSPY.tit leim/CXTERM-DIC/Punct.tit leim/CXTERM-DIC/QJ.tit leim/CXTERM-DIC/SW.tit leim/CXTERM-DIC/TONEPY.tit leim/MISC-DIC/CTLau.html leim/MISC-DIC/pinyin.map leim/MISC-DIC/ziranma.cin * cp850 This file contains non-ASCII characters in unibyte strings. When editing a keyboard layout it's more convenient to see 'é' than '\202', and the MS-DOS compiler requires the single byte if a backslash escape is not being used. src/msdos.c * iso-2022-cn-ext This file is externally generated from leim/MISC-DIC/cangjie-table.b5 by Big5->CNS converter. It hasn't been converted to UTF-8. leim/MISC-DIC/cangjie-table.cns * japanese-iso-8bit SKK-JISYO.L is a verbatim copy of a file taken from an external source. It hasn't been converted to UTF-8. leim/SKK-DIC/SKK-JISYO.L * japanese-shift-jis This is a verbatim copy of a file taken from an external source. It hasn't been converted to UTF-8. admin/charsets/mapfiles/cns2ucsdkw.txt * iso-2022-7bit This file switches between CJK charsets, which is not encoded in UTF-8. etc/HELLO Each of these files contains just one CJK charset, but Emacs currently has no easy way to specify set-charset-priority on a per-file basis, so converting any of these files to UTF-8 might change the file's appearance when viewed by an Emacs that is operating in some other language environment. etc/tutorials/TUTORIAL.ja lisp/international/ja-dic-cnv.el lisp/international/ja-dic-utl.el lisp/international/kinsoku.el lisp/international/kkc.el lisp/international/titdic-cnv.el lisp/language/japan-util.el lisp/language/japanese.el lisp/leim/quail/cyril-jis.el lisp/leim/quail/hanja-jis.el lisp/leim/quail/japanese.el lisp/leim/quail/py-punct.el lisp/leim/quail/pypunct-b5.el This file contains just Chinese characters, and has same problem. Also, it contains characters that cannot be encoded in UTF-8. lisp/international/titdic-cnv.el * utf-8-emacs These files contain characters that cannot be encoded in UTF-8. lisp/language/ethio-util.el lisp/language/ethiopic.el lisp/language/ind-util.el lisp/language/tibet-util.el lisp/language/tibetan.el lisp/leim/quail/ethiopic.el lisp/leim/quail/tibetan.el * binary files These files contain binary data, and are not text files. Some of the entries in this list are patterns, and stand for any files with the listed extension. *.gz *.icns *.ico *.pbm *.pdf *.png *.sig etc/e/eterm-color etc/package-keyring.gpg msdos/emacs.pif nextstep/GNUstep/Emacs.base/Resources/emacs.tiff nt/icons/hand.cur This file is part of GNU Emacs. GNU Emacs is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. GNU Emacs is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with GNU Emacs. If not, see .