Localization</> <abstract> <para> Describes the available localization features from the point of view of the administrator. </para> </abstract> <para> <productname>PostgreSQL</productname> supports localization with three approaches: <itemizedlist> <listitem> <para> Using the locale features of the operating system to provide locale-specific collation order, number formatting, translated messages, and other aspects. </para> </listitem> <listitem> <para> Using explicit multiple-byte character sets defined in the <productname>PostgreSQL</productname> server to support languages that require more characters than will fit into a single byte, and to provide character set recoding between client and server. The number of supported character sets is fixed at the time the server is compiled, and internal operations such as string comparisons require expansion of each character into a 32-bit word. </para> </listitem> <listitem> <para> Single byte character recoding provides a more light-weight solution for users of multiple, yet single-byte character sets. </para> </listitem> </itemizedlist> </para> <sect1 id="locale"> <title>Locale Support

Localization</> <abstract> <para> Describes the available localization features from the point of view of the administrator. </para> </abstract> <para> <productname>PostgreSQL</productname> supports localization with three approaches: <itemizedlist> <listitem> <para> Using the locale features of the operating system to provide locale-specific collation order, number formatting, translated messages, and other aspects. </para> </listitem> <listitem> <para> Using explicit multiple-byte character sets defined in the <productname>PostgreSQL</productname> server to support languages that require more characters than will fit into a single byte, and to provide character set recoding between client and server. The number of supported character sets is fixed at the time the server is compiled, and internal operations such as string comparisons require expansion of each character into a 32-bit word. </para> </listitem> <listitem> <para> Single byte character recoding provides a more light-weight solution for users of multiple, yet single-byte character sets. </para> </listitem> </itemizedlist> </para> <sect1 id="locale"> <title>Locale Support locale Locale support refers to an application respecting cultural preferences regarding alphabets, sorting, number formatting, etc. PostgreSQL uses the standard ISO C and POSIX-like locale facilities provided by the server operating system. For additional information refer to the documentation of your system. Overview</> <para> Locale support is automatically initialized when a database cluster is created using <command>initdb</command>. <command>initdb</command> will initialize the database cluster with the locale setting of its execution environment; so if your system is already set to use the locale that you want in your database cluster then there is nothing else you need to do. If you want to use a different locale (or you are not sure which locale your system is set to), you can tell <command>initdb</command> exactly which locale you want with the option <option>--locale</option>. For example: <screen> <prompt>$ </><userinput>initdb --locale=sv_SE</> </screen> </para> <para> This example sets the locale to Swedish (<literal>sv</>) as spoken in Sweden (<literal>SE</>). Other possibilities might be <literal>en_US</> (U.S. English) and <literal>fr_CA</> (Canada, French). If more than one character set can be useful for a locale then the specifications look like this: <literal>cs_CZ.ISO8859-2</>. What locales are available under what names on your system depends on what was provided by the operating system vendor and what was installed. </para> <para> Occasionally it is useful to mix rules from several locales, e.g., use U.S. collation rules but Spanish messages. To support that, a set of locale subcategories exist that control only a certain aspect of the localization rules. <informaltable> <tgroup cols="2"> <tbody> <row> <entry><envar>LC_COLLATE</></> <entry>String sort order</> </row> <row> <entry><envar>LC_CTYPE</></> <entry>Character classification (What is a letter? The upper-case equivalent?)</> </row> <row> <entry><envar>LC_MESSAGES</></> <entry>Language of messages</> </row> <row> <entry><envar>LC_MONETARY</></> <entry>Formatting of currency amounts</> </row> <row> <entry><envar>LC_NUMERIC</></> <entry>Formatting of numbers</> </row> <row> <entry><envar>LC_TIME</></> <entry>Formatting of dates and times</> </row> </tbody> </tgroup> </informaltable> The category names translate into names of <command>initdb</command> options to override the locale choice for a specific category. For instance, to set the locale to French Canadian, but use U.S. rules for formatting currency, use <literal>initdb --locale=fr_CA --lc-monetary=en_US</literal>. </para> <para> If you want the system to behave as if it had no locale support, use the special locale <literal>C</> or <literal>POSIX</>. </para> <para> The nature of some locale categories is that their value has to be fixed for the lifetime of a database cluster. That is, once <command>initdb</command> has run, you cannot change them anymore. <literal>LC_COLLATE</literal> and <literal>LC_CTYPE</literal> are those categories. They affect the sort order of indexes, so they must be kept fixed, or indexes on text columns will become corrupt. <productname>PostgreSQL</productname> enforces this by recording the values of <envar>LC_COLLATE</> and <envar>LC_CTYPE</> that are seen by <command>initdb</>. The server automatically adopts those two values when it is started. </para> <para> The other locale categories can be changed as desired whenever the server is started by setting the run-time configuration variables that have the same name as the locale categories (see <xref linkend="runtime-config"> for details). The defaults that are chosen by <command>initdb</command> are actually only written into the configuration file <filename>postgresql.conf</filename> to serve as defaults when the server is started. If you delete the assignments from <filename>postgresql.conf</filename> then the server will inherit the settings from the execution environment. </para> <para> Note that the locale behavior of the server is determined by the environment variables seen by the server, not by the environment of any client. Therefore, be careful to configure the correct locale settings before starting the server. A consequence of this is that if client and server are set up to different locales, messages may appear in different languages depending on where they originated. </para> <note> <para> When we speak of inheriting the locale from the execution environment, this means the following on most operating systems: For a given locale category, say the collation, the following environment variables are consulted in this order until one is found to be set: <envar>LC_ALL</envar>, <envar>LC_COLLATE</envar> (the variable corresponding to the respective category), <envar>LANG</envar>. If none of these environment variables are set then the locale defaults to <literal>C</literal>. </para> <para> Some message localization libraries also look at the environment variable <envar>LANGUAGE</envar> which overrides all other locale settings for the purpose of setting the language of messages. If in doubt, please refer to the documentation of your operating system, in particular the <citerefentry><refentrytitle>gettext</><manvolnum>3</></> manual page, for more information. </para> </note> <para> To enable messages translated to the user's preferred language, the <option>--enable-nls</option> option must be used. This option is independent of the other locale support. </para> </sect2> <sect2> <title>Benefits</> <para> Locale support influences in particular the following features: <itemizedlist> <listitem> <para> Sort order in <command>ORDER BY</> queries. <indexterm><primary>ORDER BY</></> </para> </listitem> <listitem> <para> The <function>to_char</> family of functions </para> </listitem> <listitem> <para> The <literal>LIKE</> and <literal>~</> operators for pattern matching </para> </listitem> </itemizedlist> </para> <para> The only severe drawback of using the locale support in <productname>PostgreSQL</> is its speed. So use locale only if you actually need it. It should be noted in particular that selecting a non-C locale disables index optimizations for <literal>LIKE</> and <literal>~</> operators, which can make a huge difference in the speed of searches that use those operators. </para> </sect2> <sect2> <title>Problems</> <para> If locale support doesn't work in spite of the explanation above, check that the locale support in your operating system is correctly configured. To check whether a given locale is installed and functional you can use <application>Perl</>, for example. Perl has also support for locales and if a locale is broken <command>perl -v</> will complain something like this: <screen> <prompt>$</> <userinput>export LC_CTYPE='not_exist'</> <prompt>$</> <userinput>perl -v</> <computeroutput> perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LC_ALL = (unset), LC_CTYPE = "not_exist", LANG = (unset) are supported and installed on your system. perl: warning: Falling back to the standard locale ("C"). </computeroutput> </screen> </para> <para> Check that your locale files are in the right location. Possible locations include: <filename>/usr/lib/locale</filename> (<systemitem class="osname">Linux</>, <systemitem class="osname">Solaris</>), <filename>/usr/share/locale</filename> (<systemitem class="osname">Linux</>), <filename>/usr/lib/nls/loc</filename> (<systemitem class="osname">DUX 4.0</>). Check the locale man page of your system if you are not sure. </para> <para> Check that <productname>PostgreSQL</> is actually using the locale that you think it is. <envar>LC_COLLATE</> and <envar>LC_CTYPE</> settings are determined at <application>initdb</> time and cannot be changed without repeating <application>initdb</>. Other locale settings including <envar>LC_MESSAGES</> and <envar>LC_MONETARY</> are determined by the environment the postmaster is started in, and can be changed with a simple postmaster restart. You can check the <envar>LC_COLLATE</> and <envar>LC_CTYPE</> settings of a database with the <filename>contrib/pg_controldata</> utility program. </para> <para> The directory <filename>src/test/locale</> contains a test suite for <productname>PostgreSQL</>'s locale support. </para> <para> Client applications that handle server-side errors by parsing the text of the error message will obviously have problems when the server's messages are in a different language. If you create such an application you need to devise a plan to cope with this situation. The embedded SQL interface (<application>ecpg</>) is also affected by this problem. It is currently recommended that servers interfacing with <application>ecpg</> applications be configured to send messages in English. </para> <para> Maintaining catalogs of message translations requires the on-going efforts of many volunteers that want to see <productname>PostgreSQL</> speak their preferred language well. If messages in your language is currently not available or fully translated, your assistance would be appreciated. If you want to help, refer to the <citetitle>Developer's Guide</> or write to the developers' mailing list. </para> </sect2> </sect1> <sect1 id="multibyte"> <title>Multibyte Support multibyte Author Tatsuo Ishii (ishii@postgresql.org), last updated 2002-07-24. Check Tatsuo's web site for more information. Multibyte (MB) support is intended to allow PostgreSQL to handle multiple-byte character sets such as EUC (Extended Unix Code), Unicode, and Mule internal code. With MB enabled you can use multibyte character sets in regular expressions (regexp), LIKE, and some other functions. The default encoding system is selected while initializing your PostgreSQL installation using initdb. Note that this can be overridden when you create a database using createdb or by using the SQL command CREATE DATABASE. So you can have multiple databases each with a different encoding system. Note that MB can handle single byte characters sets such as ISO-8859-1. Multibyte support is enabled by default since PostgreSQL version 7.3. Supported character set encodings Following encoding can be used as database encoding. Character Set EncodingsEncodings Encoding Description SQL_ASCII ASCII EUC_JP Japanese EUC EUC_CN Chinese EUC EUC_KR Korean EUC JOHAB Korean EUC (Hangle base) EUC_TW Taiwan EUC UNICODE Unicode (UTF-8) MULE_INTERNAL Mule internal code LATIN1 ISO 8859-1 ECMA-94 Latin Alphabet No.1 LATIN2 ISO 8859-2 ECMA-94 Latin Alphabet No.2 LATIN3 ISO 8859-3 ECMA-94 Latin Alphabet No.3 LATIN4 ISO 8859-4 ECMA-94 Latin Alphabet No.4 LATIN5 ISO 8859-9 ECMA-128 Latin Alphabet No.5 LATIN6 ISO 8859-10 ECMA-144 Latin Alphabet No.6 LATIN7 ISO 8859-13 Latin Alphabet No.7 LATIN8 ISO 8859-14 Latin Alphabet No.8 LATIN9 ISO 8859-15 Latin Alphabet No.9 LATIN10 ISO 8859-16 ASRO SR 14111 Latin Alphabet No.10 ISO-8859-5 ECMA-113 Latin/Cyrillic ISO-8859-6 ECMA-114 Latin/Arabic ISO-8859-7 ECMA-118 Latin/Greek ISO-8859-8 ECMA-121 Latin/Hebrew KOI8 KOI8-R(U) WIN Windows CP1251 ALT Windows CP866 WIN1256 Arabic Windows CP1256 TCVN Vietnamese TCVN-5712 (Windows CP1258) WIN874 Thai Windows CP874

Before PostgreSQL7.2, LATIN5 mistakenly meant ISO 8859-5. From 7.2 on, LATIN5 means ISO 8859-9. If you have a LATIN5 database created on 7.1 or earlier and want to migrate to 7.2 (or later), you should be very careful about this change. Not all APIs supports all the encodings listed above. For example, the PostgreSQL JDBC driver does not support MULE_INTERNAL, LATIN6, LATIN8, and LATIN10. Setting the Encoding initdb defines the default encoding for a PostgreSQL installation. For example: $ initdb -E EUC_JP sets the default encoding to EUC_JP (Extended Unix Code for Japanese). Note that you can use instead of if you prefer to type longer option strings. If no option is given, SQL_ASCII is used. You can create a database with a different encoding: $ createdb -E EUC_KR korean will create a database named korean with EUC_KR encoding. Another way to accomplish this is to use a SQL command: CREATE DATABASE korean WITH ENCODING = 'EUC_KR'; The encoding for a database is represented as an encoding column in the pg_database system catalog. You can see that by using the option or the \l command of psql. $ psql -l List of databases Database | Owner | Encoding ---------------+---------+--------------- euc_cn | t-ishii | EUC_CN euc_jp | t-ishii | EUC_JP euc_kr | t-ishii | EUC_KR euc_tw | t-ishii | EUC_TW mule_internal | t-ishii | MULE_INTERNAL regression | t-ishii | SQL_ASCII template1 | t-ishii | EUC_JP test | t-ishii | EUC_JP unicode | t-ishii | UNICODE (9 rows) Automatic encoding conversion between server and client PostgreSQL supports an automatic encoding conversion between server and client for some encodings. The conversion info is stored in pg_conversion system catalog. You can create a new conversion by using CREATE CONVERSION. PostgreSQL comes with some predefined conversions. They are listed in . Client/Server Character Set EncodingsCommunication Encodings Server Encoding Available Client Encodings SQL_ASCII SQL_ASCII, UNICODE, MULE_INTERNAL EUC_JP EUC_JP, SJIS, UNICODE, MULE_INTERNAL EUC_CN EUC_CN, UNICODE, MULE_INTERNAL EUC_KR EUC_KR, UNICODE, MULE_INTERNAL JOHAB JOHAB, UNICODE EUC_TW EUC_TW, BIG5, UNICODE, MULE_INTERNAL LATIN1 LATIN1, UNICODE MULE_INTERNAL LATIN2 LATIN2, WIN1250, UNICODE, MULE_INTERNAL LATIN3 LATIN3, UNICODE, MULE_INTERNAL LATIN4 LATIN4, UNICODE, MULE_INTERNAL LATIN5 LATIN5, UNICODE LATIN6 LATIN6, UNICODE, MULE_INTERNAL LATIN7 LATIN7, UNICODE, MULE_INTERNAL LATIN8 LATIN8, UNICODE, MULE_INTERNAL LATIN9 LATIN9, UNICODE, MULE_INTERNAL LATIN10 LATIN10, UNICODE, MULE_INTERNAL ISO_8859_5 ISO_8859_5, UNICODE, MULE_INTERNAL, WIN, ALT, KOI8 ISO_8859_6 ISO_8859_6, UNICODE ISO_8859_7 ISO_8859_7, UNICODE ISO_8859_8 ISO_8859_8, UNICODE UNICODE EUC_JP, SJIS, EUC_KR, UHC, JOHAB, EUC_CN, GBK, EUC_TW, BIG5, LATIN1 to LATIN10, ISO_8859_5, ISO_8859_6, ISO_8859_7, ISO_8859_8, WIN, ALT, KOI8, WIN1256, TCVN, WIN874, GB18030, WIN1250 MULE_INTERNAL EUC_JP, SJIS, EUC_KR, EUC_CN, EUC_TW, BIG5, LATIN1 to LATIN5, WIN, ALT, WIN1250, BIG5, ISO_8859_5, KOI8 KOI8 ISO_8859_5, WIN, ALT, KOI8, UNICODE, MULE_INTERNAL WIN ISO_8859_5, WIN, ALT, KOI8, UNICODE, MULE_INTERNAL ALT ISO_8859_5, WIN, ALT, KOI8, UNICODE, MULE_INTERNAL WIN1256 WIN1256, UNICODE TCVN TCVN, UNICODE WIN874 WIN874, UNICODE

To enable the automatic encoding translation, you have to tell PostgreSQL the encoding you would like to use in the client. There are several ways to accomplish this. Using the \encoding command in psql. \encoding allows you to change client encoding on the fly. For example, to change the encoding to SJIS, type: \encoding SJIS Using libpq functions. \encoding actually calls PQsetClientEncoding() for its purpose. int PQsetClientEncoding(PGconn *conn, const char *encoding) where conn is a connection to the server, and encoding is an encoding you want to use. If it successfully sets the encoding, it returns 0, otherwise -1. The current encoding for this connection can be shown by using: int PQclientEncoding(const PGconn *conn) Note that it returns the encoding ID, not a symbolic string such as EUC_JP. To convert an encoding ID to an encoding name, you can use: char *pg_encoding_to_char(int encoding_id) Using SET CLIENT_ENCODING TO. Setting the client encoding can be done with this SQL command: SET CLIENT_ENCODING TO 'encoding'; Also you can use the SQL92 syntax SET NAMES for this purpose: SET NAMES 'encoding'; To query the current client encoding: SHOW CLIENT_ENCODING; To return to the default encoding: RESET CLIENT_ENCODING; Using PGCLIENTENCODING. If environment variable PGCLIENTENCODING is defined in the client's environment, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.) Using client_encoding variable. If the client_encoding variable in postgresql.conf is set, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.) What happens if the translation is not possible? Suppose you choose EUC_JP for the server and LATIN1 for the client, then some Japanese characters cannot be translated into LATIN1. In this case, a letter that cannot be represented in the LATIN1 character set would be transformed as: (HEXA DECIMAL) References These are good sources to start learning about various kinds of encoding systems. Detailed explanations of EUC_JP, EUC_CN, EUC_KR, EUC_TW appear in section 3.2. The web site of the Unicode Consortium RFC 2044 UTF-8 is defined here. History Dec 7, 2000 * An automatic encoding translation between Unicode and other encodings are implemented * Changes above will appear in 7.1 May 20, 2000 * SJIS UDC (NEC selection IBM kanji) support contributed by Eiji Tokuya * Changes above will appear in 7.0.1 Mar 22, 2000 * Add new libpq functions PQsetClientEncoding, PQclientEncoding * ./configure --with-mb=EUC_JP now deprecated. use ./configure --enable-multibyte=EUC_JP instead * Add SQL_ASCII regression test case * Add SJIS User Defined Character (UDC) support * All of above will appear in 7.0 July 11, 1999 * Add support for WIN1250 (Windows Czech) as a client encoding (contributed by Pavel Behal) * fix some compiler warnings (contributed by Tomoaki Nishiyama) Mar 23, 1999 * Add support for KOI8(KOI8-R), WIN(CP1251), ALT(CP866) (thanks Oleg Broytmann for testing) * Fix problem with MB and locale Jan 26, 1999 * Add support for Big5 for frontend encoding (you need to create a database with EUC_TW to use Big5) * Add regression test case for EUC_TW (contributed by Jonah Kuo jonahkuo@mail.ttn.com.tw) Dec 15, 1998 * Bugs related to SQL_ASCII support fixed Nov 5, 1998 * 6.4 release. In this version, pg_database has "encoding" column that represents the database encoding Jul 22, 1998 * determine encoding at initdb/createdb rather than compile time * support for PGCLIENTENCODING when issuing COPY command * support for SQL92 syntax "SET NAMES" * support for LATIN2-5 * add UNICODE regression test case * new test suite for MB * clean up source files Jun 5, 1998 * add support for the encoding translation between the backend and the frontend * new command SET CLIENT_ENCODING etc. added * add support for LATIN1 character set * enhance 8-bit cleanliness April 21, 1998 some enhancements/fixes * character_length(), position(), substring() are now aware of multi-byte characters * add octet_length() * add --with-mb option to configure * new regression tests for EUC_KR (contributed by Soonmyung Hong) * add some test cases to the EUC_JP regression test * fix problem in regress/regress.sh in case of System V * fix toupper(), tolower() to handle 8bit chars Mar 25, 1998 MB PL2 is incorporated into PostgreSQL 6.3.1 Mar 10, 1998 PL2 released * add regression test for EUC_JP, EUC_CN and MULE_INTERNAL * add an English document (this file) * fix problems concerning 8-bit single byte characters Mar 1, 1998 PL1 released WIN1250 on Windows/ODBC The WIN1250 character set on Windows client platforms can be used with PostgreSQL with locale support enabled. The following should be kept in mind: Success depends on proper system locales. This has been tested with Red Hat 6.0 and Slackware 3.6, with the cs_CZ.iso8859-2 locale. Never try to set the server's database encoding to WIN1250. Always use LATIN2 instead since there is no WIN1250 locale in Unix. The WIN1250 encoding is usable only for Windows ODBC clients. The characters are recoded on the fly, to be displayed and stored back properly. WIN1250 on Windows/ODBC Compile PostgreSQL with locale enabled and the server-side encoding set to LATIN2. Set up your installation. Do not forget to create locale variables in your environment. For example (this may not be correct for your environment): LC_ALL=cs_CZ.ISO8859-2 You have to start the server with locales set! Try it with the Czech language, and have it sort on a query. Install ODBC driver for PostgreSQL on your Windows machine. Set up your data source properly. Include this line in your ODBC configuration dialog in the field Connect Settings: SET CLIENT_ENCODING = 'WIN1250'; Now try it again, but in Windows with ODBC. Single-byte character set recoding</>  <para> You can set up this feature with the <option>--enable-recode</> option to <filename>configure</>. This option was formerly described as <quote>Cyrillic recode support</> which doesn't express all its power. It can be used for <emphasis>any</> single-byte character set recoding. </para> <para> This method uses a file <filename>charset.conf</> file located in the database directory (<envar>PGDATA</>). It's a typical configuration text file where spaces and newlines separate items and records and # specifies comments. Three keywords with the following syntax are recognized here: <synopsis> BaseCharset <replaceable>server_charset</> RecodeTable <replaceable>from_charset</> <replaceable>to_charset</> <replaceable>file_name</> HostCharset <replaceable>host_spec</> <replaceable>host_charset</> </synopsis> </para> <para> <token>BaseCharset</> defines the encoding of the database server. All character set names are only used for mapping inside of <filename>charset.conf</> so you can freely use typing-friendly names. </para> <para> <token>RecodeTable</> records specify translation tables between server and client. The file name is relative to the <envar>PGDATA</> directory. The table file format is very simple. There are no keywords and characters are represented by a pair of decimal or hexadecimal (0x prefixed) values on single lines: <synopsis> <replaceable>char_value</> <replaceable>translated_char_value</> </synopsis> </para> <para> <token>HostCharset</> records define the client character set by IP address. You can use a single IP address, an IP mask range starting from the given address or an IP interval (e.g., 127.0.0.1, 192.168.1.100/24, 192.168.1.20-192.168.1.40). </para> <para> The <filename>charset.conf</> file is always processed up to the end, so you can easily specify exceptions from the previous rules. In the <filename>src/data/</> directory you will find an example <filename>charset.conf</> and a few recoding tables. </para> <para> As this solution is based on the client's IP address and character set mapping there are obviously some restrictions as well. You cannot use different encodings on the same host at the same time. It is also inconvenient when you boot your client hosts into multiple operating systems. Nevertheless, when these restrictions are not limiting and you do not need multibyte characters then it is a simple and effective solution. </para> </sect1> </chapter>