unicode: Merged from trunk up to [5222] (want to use the extra tests).

git-svn-id: http://code.djangoproject.com/svn/django/branches/unicode@5224 bcc190cf-cafb-0310-a4f2-bffc1f526a37

5 files changed, 299 insertions, 32 deletions

diff --git a/django/test/testcases.py b/django/test/testcases.py
index 45b7a2331d..a7aafb1c3f 100644
--- a/django/test/testcases.py
+++ b/django/test/testcases.py
@@ -1,5 +1,4 @@
 import re, doctest, unittest
-import sys
 from urlparse import urlparse
 from django.db import transaction
 from django.core import management, mail
diff --git a/docs/databrowse.txt b/docs/databrowse.txt
index e9691cc879..9c03e7e4ea 100644
--- a/docs/databrowse.txt
+++ b/docs/databrowse.txt
@@ -44,7 +44,11 @@ How to use Databrowse
        It doesn't matter where you put this, as long as it gets executed at
        some point. A good place for it is in your URLconf file (``urls.py``).
 
-    3. Add the following line to your URLconf::
+    3. Change your URLconf to import the ``databrowse`` module::
+
+           from django.contrib import databrowse
+
+       ...and add the following line to your URLconf::
 
            (r'^databrowse/(.*)', databrowse.site.root),
 
diff --git a/docs/newforms.txt b/docs/newforms.txt
index 123b554bde..7c861ed405 100644
--- a/docs/newforms.txt
+++ b/docs/newforms.txt
@@ -9,28 +9,30 @@ framework. This document explains how to use this new library.
 Migration plan
 ==============
 
-``django.newforms`` currently is only available in Django beginning
-with the 0.96 release.  the Django development version -- i.e., it's
-not available in the Django 0.95 release. For the next Django release,
-our plan is to do the following:
+``django.newforms`` is new in Django's 0.96 release, but, as it won't be new
+forever, we plan to rename it to ``django.forms`` in the future. The current
+``django.forms`` package will be available as ``django.oldforms`` until Django
+1.0, when we plan to remove it for good.
 
-    * As of revision [4208], we've copied the current ``django.forms`` to
-      ``django.oldforms``. This allows you to upgrade your code *now* rather
-      than waiting for the backwards-incompatible change and rushing to fix
-      your code after the fact. Just change your import statements like this::
+That has direct repercussions on the forward compatibility of your code. Please
+read the following migration plan and code accordingly:
+
+    * The old forms framework (the current ``django.forms``) has been copied to
+      ``django.oldforms``. Thus, you can start upgrading your code *now*,
+      rather than waiting for the future backwards-incompatible change, by
+      changing your import statements like this::
 
           from django import forms             # old
           from django import oldforms as forms # new
 
-    * At an undecided future date, we will move the current ``django.newforms``
-      to ``django.forms``. This will be a backwards-incompatible change, and
-      anybody who is still using the old version of ``django.forms`` at that
-      time will need to change their import statements, as described in the
-      previous bullet.
+    * In the next Django release (0.97), we will move the current
+      ``django.newforms`` to ``django.forms``. This will be a
+      backwards-incompatible change, and anybody who is still using the old
+      version of ``django.forms`` at that time will need to change their import
+      statements, as described in the previous bullet.
 
     * We will remove ``django.oldforms`` in the release *after* the next Django
-      release -- the release that comes after the release in which we're
-      creating the new ``django.forms``.
+      release -- either 0.98 or 1.0, whichever comes first.
 
 With this in mind, we recommend you use the following import statement when
 using ``django.newforms``::
@@ -184,7 +186,7 @@ e-mail address::
     >>> f.is_valid()
     False
 
-Access the ``Form`` attribute ``errors`` to get a dictionary of error messages::
+Access the ``errors`` attribute to get a dictionary of error messages::
 
     >>> f.errors
     {'sender': [u'Enter a valid e-mail address.'], 'subject': [u'This field is required.']}
@@ -197,6 +199,10 @@ You can access ``errors`` without having to call ``is_valid()`` first. The
 form's data will be validated the first time either you call ``is_valid()`` or
 access ``errors``.
 
+The validation routines will only get called once, regardless of how many times
+you access ``errors`` or call ``is_valid()``. This means that if validation has
+side effects, those side effects will only be triggered once.
+
 Behavior of unbound forms
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -274,6 +280,27 @@ but ``clean_data`` contains only the form's fields::
     >>> f.clean_data # Doesn't contain extra_field_1, etc.
     {'cc_myself': True, 'message': u'Hi there', 'sender': u'foo@example.com', 'subject': u'hello'}
 
+``clean_data`` will include a key and value for *all* fields defined in the
+``Form``, even if the data didn't include a value for fields that are not
+required. In this example, the data dictionary doesn't include a value for the
+``nick_name`` field, but ``clean_data`` includes it, with an empty value::
+
+    >>> class OptionalPersonForm(Form):
+    ...     first_name = CharField()
+    ...     last_name = CharField()
+    ...     nick_name = CharField(required=False)
+    >>> data = {'first_name': u'John', 'last_name': u'Lennon'}
+    >>> f = OptionalPersonForm(data)
+    >>> f.is_valid()
+    True
+    >>> f.clean_data
+    {'nick_name': u'', 'first_name': u'John', 'last_name': u'Lennon'}
+
+In this above example, the ``clean_data`` value for ``nick_name`` is set to an
+empty string, because ``nick_name`` is ``CharField``, and ``CharField``\s treat
+empty values as an empty string. Each field type knows what its "blank" value
+is -- e.g., for ``DateField``, it's ``None`` instead of the empty string.
+
 Behavior of unbound forms
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -454,7 +481,7 @@ field::
 If ``auto_id`` is set to a string containing the format character ``'%s'``,
 then the form output will include ``<label>`` tags, and will generate ``id``
 attributes based on the format string. For example, for a format string
-``'field_%s'``, a field named ``subject`` will get the ``id``
+``'field_%s'``, a field named ``subject`` will get the ``id`` value
 ``'field_subject'``. Continuing our example::
 
     >>> f = ContactForm(auto_id='id_for_%s')
@@ -493,8 +520,9 @@ How errors are displayed
 
 If you render a bound ``Form`` object, the act of rendering will automatically
 run the form's validation if it hasn't already happened, and the HTML output
-will include the validation errors as a ``<ul>`` near the field. The particular
-positioning of the error messages depends on the output method you're using::
+will include the validation errors as a ``<ul class="errorlist">`` near the
+field. The particular positioning of the error messages depends on the output
+method you're using::
 
     >>> data = {'subject': '',
     ...         'message': 'Hi there',
@@ -556,7 +584,8 @@ The field-specific output honors the form object's ``auto_id`` setting::
     <input type="text" name="message" id="id_message" />
 
 For a field's list of errors, access the field's ``errors`` attribute. This
-is a list-like object that is displayed as an HTML ``<ul>`` when printed::
+is a list-like object that is displayed as an HTML ``<ul class="errorlist">``
+when printed::
 
     >>> data = {'subject': 'hi', 'message': '', 'sender': '', 'cc_myself': ''}
     >>> f = ContactForm(data, auto_id=False)
@@ -646,7 +675,7 @@ Core field arguments
 
 Each ``Field`` class constructor takes at least these arguments. Some
 ``Field`` classes take additional, field-specific arguments, but the following
-should *always* be available:
+should *always* be accepted:
 
 ``required``
 ~~~~~~~~~~~~
@@ -704,7 +733,7 @@ field.)
 The ``label`` argument lets you specify the "human-friendly" label for this
 field. This is used when the ``Field`` is displayed in a ``Form``.
 
-As explained in _`Outputting forms as HTML` above, the default label for a
+As explained in "Outputting forms as HTML" above, the default label for a
 ``Field`` is generated from the field name by converting all underscores to
 spaces and upper-casing the first letter. Specify ``label`` if that default
 behavior doesn't result in an adequate label.
@@ -779,14 +808,15 @@ validation if a particular field's value is not given. ``initial`` values are
 ~~~~~~~~~~
 
 The ``widget`` argument lets you specify a ``Widget`` class to use when
-rendering this ``Field``. See _`Widgets` below for more information.
+rendering this ``Field``. See "Widgets" below for more information.
 
 ``help_text``
 ~~~~~~~~~~~~~
 
 The ``help_text`` argument lets you specify descriptive text for this
 ``Field``. If you provide ``help_text``, it will be displayed next to the
-``Field`` when the ``Field`` is rendered in a ``Form``.
+``Field`` when the ``Field`` is rendered by one of the convenience ``Form``
+methods (e.g., ``as_ul()``).
 
 Here's a full example ``Form`` that implements ``help_text`` for two of its
 fields. We've specified ``auto_id=False`` to simplify the output::
@@ -860,6 +890,212 @@ level and at the form instance level, and the latter gets precedence::
     <tr><th>Url:</th><td><input type="text" name="url" /></td></tr>
     <tr><th>Comment:</th><td><input type="text" name="comment" /></td></tr>
 
+Built-in ``Field`` classes
+--------------------------
+
+Naturally, the ``newforms`` library comes with a set of ``Field`` classes that
+represent common validation needs. This section documents each built-in field.
+
+For each field, we describe the default widget used if you don't specify
+``widget``. We also specify the value returned when you provide an empty value
+(see the section on ``required`` above to understand what that means).
+
+``BooleanField``
+~~~~~~~~~~~~~~~~
+
+    * Default widget: ``CheckboxInput``
+    * Empty value: ``None``
+    * Normalizes to: A Python ``True`` or ``False`` value.
+    * Validates nothing (i.e., it never raises a ``ValidationError``).
+
+``CharField``
+~~~~~~~~~~~~~
+
+    * Default widget: ``TextInput``
+    * Empty value: ``''`` (an empty string)
+    * Normalizes to: A Unicode object.
+    * Validates nothing, unless ``max_length`` or ``min_length`` is provided.
+
+Has two optional arguments for validation, ``max_length`` and ``min_length``.
+If provided, these arguments ensure that the string is at most or at least the
+given length.
+
+``ChoiceField``
+~~~~~~~~~~~~~~~
+
+    * Default widget: ``Select``
+    * Empty value: ``''`` (an empty string)
+    * Normalizes to: A Unicode object.
+    * Validates that the given value exists in the list of choices.
+
+Takes one extra argument, ``choices``, which is an iterable (e.g., a list or
+tuple) of 2-tuples to use as choices for this field.
+
+``DateField``
+~~~~~~~~~~~~~
+
+    * Default widget: ``TextInput``
+    * Empty value: ``None``
+    * Normalizes to: A Python ``datetime.date`` object.
+    * Validates that the given value is either a ``datetime.date``,
+      ``datetime.datetime`` or string formatted in a particular date format.
+
+Takes one optional argument, ``input_formats``, which is a list of formats used
+to attempt to convert a string to a valid ``datetime.date`` object.
+
+If no ``input_formats`` argument is provided, the default input formats are::
+
+    '%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', # '2006-10-25', '10/25/2006', '10/25/06'
+    '%b %d %Y', '%b %d, %Y',            # 'Oct 25 2006', 'Oct 25, 2006'
+    '%d %b %Y', '%d %b, %Y',            # '25 Oct 2006', '25 Oct, 2006'
+    '%B %d %Y', '%B %d, %Y',            # 'October 25 2006', 'October 25, 2006'
+    '%d %B %Y', '%d %B, %Y',            # '25 October 2006', '25 October, 2006'
+
+``DateTimeField``
+~~~~~~~~~~~~~~~~~
+
+    * Default widget: ``TextInput``
+    * Empty value: ``None``
+    * Normalizes to: A Python ``datetime.datetime`` object.
+    * Validates that the given value is either a ``datetime.datetime``,
+      ``datetime.date`` or string formatted in a particular datetime format.
+
+Takes one optional argument, ``input_formats``, which is a list of formats used
+to attempt to convert a string to a valid ``datetime.datetime`` object.
+
+If no ``input_formats`` argument is provided, the default input formats are::
+
+    '%Y-%m-%d %H:%M:%S',     # '2006-10-25 14:30:59'
+    '%Y-%m-%d %H:%M',        # '2006-10-25 14:30'
+    '%Y-%m-%d',              # '2006-10-25'
+    '%m/%d/%Y %H:%M:%S',     # '10/25/2006 14:30:59'
+    '%m/%d/%Y %H:%M',        # '10/25/2006 14:30'
+    '%m/%d/%Y',              # '10/25/2006'
+    '%m/%d/%y %H:%M:%S',     # '10/25/06 14:30:59'
+    '%m/%d/%y %H:%M',        # '10/25/06 14:30'
+    '%m/%d/%y',              # '10/25/06'
+
+``EmailField``
+~~~~~~~~~~~~~~
+
+    * Default widget: ``TextInput``
+    * Empty value: ``''`` (an empty string)
+    * Normalizes to: A Unicode object.
+    * Validates that the given value is a valid e-mail address, using a
+      moderately complex regular expression.
+
+Has two optional arguments for validation, ``max_length`` and ``min_length``.
+If provided, these arguments ensure that the string is at most or at least the
+given length.
+
+``IntegerField``
+~~~~~~~~~~~~~~~~
+
+    * Default widget: ``TextInput``
+    * Empty value: ``None``
+    * Normalizes to: A Python integer or long integer.
+    * Validates that the given value is an integer. Leading and trailing
+      whitespace is allowed, as in Python's ``int()`` function.
+
+``MultipleChoiceField``
+~~~~~~~~~~~~~~~~~~~~~~~
+
+    * Default widget: ``SelectMultiple``
+    * Empty value: ``[]`` (an empty list)
+    * Normalizes to: A list of Unicode objects.
+    * Validates that every value in the given list of values exists in the list
+      of choices.
+
+Takes one extra argument, ``choices``, which is an iterable (e.g., a list or
+tuple) of 2-tuples to use as choices for this field.
+
+``NullBooleanField``
+~~~~~~~~~~~~~~~~~~~~
+
+    * Default widget: ``NullBooleanSelect``
+    * Empty value: ``None``
+    * Normalizes to: A Python ``True``, ``False`` or ``None`` value.
+    * Validates nothing (i.e., it never raises a ``ValidationError``).
+
+``RegexField``
+~~~~~~~~~~~~~~
+
+    * Default widget: ``TextInput``
+    * Empty value: ``''`` (an empty string)
+    * Normalizes to: A Unicode object.
+    * Validates that the given value matches against a certain regular
+      expression.
+
+Takes one required argument, ``regex``, which is a regular expression specified
+either as a string or a compiled regular expression object.
+
+Also takes the following optional arguments:
+
+    ======================  =====================================================
+    Argument                Description
+    ======================  =====================================================
+    ``max_length``          Ensures the string has at most this many characters.
+    ``min_length``          Ensures the string has at least this many characters.
+    ``error_message``       Error message to return for failed validation. If no
+                            message is provided, a generic error message will be
+                            used.
+    ======================  =====================================================
+
+``TimeField``
+~~~~~~~~~~~~~
+
+    * Default widget: ``TextInput``
+    * Empty value: ``None``
+    * Normalizes to: A Python ``datetime.time`` object.
+    * Validates that the given value is either a ``datetime.time`` or string
+      formatted in a particular time format.
+
+Takes one optional argument, ``input_formats``, which is a list of formats used
+to attempt to convert a string to a valid ``datetime.time`` object.
+
+If no ``input_formats`` argument is provided, the default input formats are::
+
+    '%H:%M:%S',     # '14:30:59'
+    '%H:%M',        # '14:30'
+
+``URLField``
+~~~~~~~~~~~~
+
+    * Default widget: ``TextInput``
+    * Empty value: ``''`` (an empty string)
+    * Normalizes to: A Unicode object.
+    * Validates that the given value is a valid URL.
+
+Takes the following optional arguments:
+
+    ========================  =====================================================
+    Argument                  Description
+    ========================  =====================================================
+    ``max_length``            Ensures the string has at most this many characters.
+    ``min_length``            Ensures the string has at least this many characters.
+    ``verify_exists``         If ``True``, the validator will attempt to load the
+                              given URL, raising ``ValidationError`` if the page
+                              gives a 404. Defaults to ``False``.
+    ``validator_user_agent``  String used as the user-agent used when checking for
+                              a URL's existence. Defaults to the value of the
+                              ``URL_VALIDATOR_USER_AGENT`` setting.
+    ========================  =====================================================
+
+Slightly complex built-in ``Field`` classes
+-------------------------------------------
+
+The following are not yet documented here. See the unit tests, linked-to from
+the bottom of this document, for examples of their use.
+
+``ComboField``
+~~~~~~~~~~~~~~
+
+``MultiValueField``
+~~~~~~~~~~~~~~~~~~~
+
+``SplitDateTimeField``
+~~~~~~~~~~~~~~~~~~~~~~
+
 Creating custom fields
 ----------------------
 
diff --git a/docs/testing.txt b/docs/testing.txt
index dd4e9e17b2..92edf46e09 100644
--- a/docs/testing.txt
+++ b/docs/testing.txt
@@ -503,7 +503,7 @@ that can be useful in testing the behavior of web sites.
     response.
 
 E-mail services
---------------
+---------------
 
 **New in Django development version**
 
diff --git a/tests/regressiontests/forms/tests.py b/tests/regressiontests/forms/tests.py
index d6b08a0cd3..615cb2247b 100644
--- a/tests/regressiontests/forms/tests.py
+++ b/tests/regressiontests/forms/tests.py
@@ -1916,6 +1916,34 @@ True
 >>> p.clean_data
 {'first_name': u'John', 'last_name': u'Lennon', 'birthday': datetime.date(1940, 10, 9)}
 
+clean_data will include a key and value for *all* fields defined in the Form,
+even if the Form's data didn't include a value for fields that are not
+required. In this example, the data dictionary doesn't include a value for the
+"nick_name" field, but clean_data includes it. For CharFields, it's set to the
+empty string.
+>>> class OptionalPersonForm(Form):
+...     first_name = CharField()
+...     last_name = CharField()
+...     nick_name = CharField(required=False)
+>>> data = {'first_name': u'John', 'last_name': u'Lennon'}
+>>> f = OptionalPersonForm(data)
+>>> f.is_valid()
+True
+>>> f.clean_data
+{'nick_name': u'', 'first_name': u'John', 'last_name': u'Lennon'}
+
+For DateFields, it's set to None.
+>>> class OptionalPersonForm(Form):
+...     first_name = CharField()
+...     last_name = CharField()
+...     birth_date = DateField(required=False)
+>>> data = {'first_name': u'John', 'last_name': u'Lennon'}
+>>> f = OptionalPersonForm(data)
+>>> f.is_valid()
+True
+>>> f.clean_data
+{'birth_date': None, 'first_name': u'John', 'last_name': u'Lennon'}
+
 "auto_id" tells the Form to add an "id" attribute to each form element.
 If it's a string that contains '%s', Django will use that as a format string
 into which the field's name will be inserted. It will also put a <label> around
@@ -3378,7 +3406,7 @@ True
 </select>
 
 # MultiWidget and MultiValueField #############################################
-# MultiWidgets are widgets composed of other widgets. They are usually 
+# MultiWidgets are widgets composed of other widgets. They are usually
 # combined with MultiValueFields - a field that is composed of other fields.
 # MulitWidgets can themselved be composed of other MultiWidgets.
 # SplitDateTimeWidget is one example of a MultiWidget.
@@ -3386,7 +3414,7 @@ True
 >>> class ComplexMultiWidget(MultiWidget):
 ...     def __init__(self, attrs=None):
 ...         widgets = (
-...             TextInput(), 
+...             TextInput(),
 ...             SelectMultiple(choices=(('J', 'John'), ('P', 'Paul'), ('G', 'George'), ('R', 'Ringo'))),
 ...             SplitDateTimeWidget(),
 ...         )
@@ -3411,13 +3439,13 @@ True
 <input type="text" name="name_2_0" value="2007-04-25" /><input type="text" name="name_2_1" value="06:24:00" />
 
 >>> class ComplexField(MultiValueField):
-...     def __init__(self, required=True, widget=None, label=None, initial=None): 
+...     def __init__(self, required=True, widget=None, label=None, initial=None):
 ...         fields = (
-...             CharField(), 
+...             CharField(),
 ...             MultipleChoiceField(choices=(('J', 'John'), ('P', 'Paul'), ('G', 'George'), ('R', 'Ringo'))),
 ...             SplitDateTimeField()
 ...         )
-...         super(ComplexField, self).__init__(fields, required, widget, label, initial) 
+...         super(ComplexField, self).__init__(fields, required, widget, label, initial)
 ...
 ...     def compress(self, data_list):
 ...         if data_list: