Merge pull request #1617 from pallets/docs-prose

rewrite non-technical documentation

5 files changed, 203 insertions, 286 deletions

diff --git a/docs/api.rst b/docs/api.rst
index 8b80749..b2db537 100644
--- a/docs/api.rst
+++ b/docs/api.rst
@@ -410,16 +410,19 @@ The Context
     .. automethod:: jinja2.runtime.Context.call(callable, \*args, \**kwargs)
 
 
-.. admonition:: Implementation
+The context is immutable, it prevents modifications, and if it is
+modified somehow despite that those changes may not show up. For
+performance, Jinja does not use the context as data storage for, only as
+a primary data source. Variables that the template does not define are
+looked up in the context, but variables the template does define are
+stored locally.
+
+Instead of modifying the context directly, a function should return
+a value that can be assigned to a variable within the template itself.
 
-    Context is immutable for the same reason Python's frame locals are
-    immutable inside functions.  Both Jinja and Python are not using the
-    context / frame locals as data storage for variables but only as primary
-    data source.
+.. code-block:: jinja
 
-    When a template accesses a variable the template does not define, Jinja
-    looks up the variable in the context, after that the variable is treated
-    as if it was defined in the template.
+    {% set comments = get_latest_comments() %}
 
 
 .. _loaders:
diff --git a/docs/faq.rst b/docs/faq.rst
index ca1f9aa..493dc38 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -1,155 +1,75 @@
 Frequently Asked Questions
 ==========================
 
-This page answers some of the often asked questions about Jinja.
-
-.. highlight:: html+jinja
 
 Why is it called Jinja?
 -----------------------
 
-The name Jinja was chosen because it's the name of a Japanese temple and
-temple and template share a similar pronunciation.  It is not named after
-the city in Uganda.
-
-How fast is it?
----------------
-
-We really hate benchmarks especially since they don't reflect much.  The
-performance of a template depends on many factors and you would have to
-benchmark different engines in different situations.  The benchmarks from the
-testsuite show that Jinja has a similar performance to `Mako`_ and is between
-10 and 20 times faster than Django's template engine or Genshi.  These numbers
-should be taken with tons of salt as the benchmarks that took these numbers
-only test a few performance related situations such as looping.  Generally
-speaking the performance of a template engine doesn't matter much as the
-usual bottleneck in a web application is either the database or the application
-code.
-
-.. _Mako: https://www.makotemplates.org/
-
-How Compatible is Jinja with Django?
-------------------------------------
-
-The default syntax of Jinja matches Django syntax in many ways.  However
-this similarity doesn't mean that you can use a Django template unmodified
-in Jinja.  For example filter arguments use a function call syntax rather
-than a colon to separate filter name and arguments.  Additionally the
-extension interface in Jinja is fundamentally different from the Django one
-which means that your custom tags won't work any longer.
+"Jinja" is a Japanese `Shinto shrine`_, or temple, and temple and
+template share a similar English pronunciation. It is not named after
+the `city in Uganda`_.
 
-Generally speaking you will use much less custom extensions as the Jinja
-template system allows you to use a certain subset of Python expressions
-which can replace most Django extensions.  For example instead of using
-something like this::
+.. _Shinto shrine: https://en.wikipedia.org/wiki/Shinto_shrine
+.. _city in Uganda: https://en.wikipedia.org/wiki/Jinja%2C_Uganda
 
-    {% load comments %}
-    {% get_latest_comments 10 as latest_comments %}
-    {% for comment in latest_comments %}
-        ...
-    {% endfor %}
 
-You will most likely provide an object with attributes to retrieve
-comments from the database::
+How fast is Jinja?
+------------------
 
-    {% for comment in models.comments.latest(10) %}
-        ...
-    {% endfor %}
+Jinja is relatively fast among template engines because it compiles and
+caches template code to Python code, so that the template does not need
+to be parsed and interpreted each time. Rendering a template becomes as
+close to executing a Python function as possible.
 
-Or directly provide the model for quick testing::
+Jinja also makes extensive use of caching. Templates are cached by name
+after loading, so future uses of the template avoid loading. The
+template loading itself uses a bytecode cache to avoid repeated
+compiling. The caches can be external to persist across restarts.
+Templates can also be precompiled and loaded as fast Python imports.
 
-    {% for comment in Comment.objects.order_by('-pub_date')[:10] %}
-        ...
-    {% endfor %}
+We dislike benchmarks because they don't reflect real use. Performance
+depends on many factors. Different engines have different default
+configurations and tradeoffs that make it unclear how to set up a useful
+comparison. Often, database access, API calls, and data processing have
+a much larger effect on performance than the template engine.
 
-Please keep in mind that even though you may put such things into templates
-it still isn't a good idea.  Queries should go into the view code and not
-the template!
 
-Isn't it a terrible idea to put Logic into Templates?
------------------------------------------------------
+Isn't it a bad idea to put logic in templates?
+----------------------------------------------
 
 Without a doubt you should try to remove as much logic from templates as
-possible.  But templates without any logic mean that you have to do all
-the processing in the code which is boring and stupid.  A template engine
-that does that is shipped with Python and called `string.Template`.  Comes
-without loops and if conditions and is by far the fastest template engine
-you can get for Python.
-
-So some amount of logic is required in templates to keep everyone happy.
-And Jinja leaves it pretty much to you how much logic you want to put into
-templates.  There are some restrictions in what you can do and what not.
-
-Jinja neither allows you to put arbitrary Python code into templates nor
-does it allow all Python expressions.  The operators are limited to the
-most common ones and more advanced expressions such as list comprehensions
-and generator expressions are not supported.  This keeps the template engine
-easier to maintain and templates more readable.
-
-Why is Autoescaping not the Default?
-------------------------------------
-
-There are multiple reasons why automatic escaping is not the default mode
-and also not the recommended one.  While automatic escaping of variables
-means that you will less likely have an XSS problem it also causes a huge
-amount of extra processing in the template engine which can cause serious
-performance problems.  As Python doesn't provide a way to mark strings as
-unsafe Jinja has to hack around that limitation by providing a custom
-string class (the :class:`~markupsafe.Markup` string) that safely interacts with safe
-and unsafe strings.
-
-With explicit escaping however the template engine doesn't have to perform
-any safety checks on variables.  Also a human knows not to escape integers
-or strings that may never contain characters one has to escape or already
-HTML markup.  For example when iterating over a list over a table of
-integers and floats for a table of statistics the template designer can
-omit the escaping because he knows that integers or floats don't contain
-any unsafe parameters.
-
-Additionally Jinja is a general purpose template engine and not only used
-for HTML/XML generation.  For example you may generate LaTeX, emails,
-CSS, JavaScript, or configuration files.
-
-Why is the Context immutable?
------------------------------
-
-When writing a :func:`pass_context` function, you may have noticed that
-the context tries to stop you from modifying it. If you have managed to
-modify the context by using an internal context API you may have noticed
-that changes in the context don't seem to be visible in the template.
-The reason for this is that Jinja uses the context only as primary data
-source for template variables for performance reasons.
-
-If you want to modify the context write a function that returns a variable
-instead that one can assign to a variable by using set::
-
-    {% set comments = get_latest_comments() %}
-
-
-My Macros are overridden by something
+possible. With less logic, the template is easier to understand, has
+fewer potential side effects, and is faster to compile and render. But a
+template without any logic means processing must be done in code before
+rendering. A template engine that does that is shipped with Python,
+called :class:`string.Template`, and while it's definitely fast it's not
+convenient.
+
+Jinja's features such as blocks, statements, filters, and function calls
+make it much easier to write expressive templates, with very few
+restrictions. Jinja doesn't allow arbitrary Python code in templates, or
+every feature available in the Python language. This keeps the engine
+easier to maintain, and keeps templates more readable.
+
+Some amount of logic is required in templates to keep everyone happy.
+Too much logic in the template can make it complex to reason about and
+maintain. It's up to you to decide how your application will work and
+balance how much logic you want to put in the template.
+
+
+Why is HTML escaping not the default?
 -------------------------------------
 
-In some situations the Jinja scoping appears arbitrary:
-
-layout.tmpl:
-
-.. sourcecode:: jinja
-
-    {% macro foo() %}LAYOUT{% endmacro %}
-    {% block body %}{% endblock %}
-
-child.tmpl:
-
-.. sourcecode:: jinja
-
-    {% extends 'layout.tmpl' %}
-    {% macro foo() %}CHILD{% endmacro %}
-    {% block body %}{{ foo() }}{% endblock %}
+Jinja provides a feature that can be enabled to escape HTML syntax in
+rendered templates. However, it is disabled by default.
 
-This will print ``LAYOUT`` in Jinja.  This is a side effect of having
-the parent template evaluated after the child one.  This allows child
-templates passing information to the parent template.  To avoid this
-issue rename the macro or variable in the parent template to have an
-uncommon prefix.
+Jinja is a general purpose template engine, it is not only used for HTML
+documents. You can generate plain text, LaTeX, emails, CSS, JavaScript,
+configuration files, etc. HTML escaping wouldn't make sense for any of
+these document types.
 
-.. _Jinja 1: https://pypi.org/project/Jinja/
+While automatic escaping means that you are less likely have an XSS
+problem, it also requires significant extra processing during compiling
+and rendering, which can reduce performance. Jinja uses MarkupSafe for
+escaping, which provides optimized C code for speed, but it still
+introduces overhead to track escaping across methods and formatting.
diff --git a/docs/integration.rst b/docs/integration.rst
index 633e80d..b945fb3 100644
--- a/docs/integration.rst
+++ b/docs/integration.rst
@@ -1,6 +1,25 @@
 Integration
 ===========
 
+
+Flask
+-----
+
+The `Flask`_ web application framework, also maintained by Pallets, uses
+Jinja templates by default. Flask sets up a Jinja environment and
+template loader for you, and provides functions to easily render
+templates from view functions.
+
+.. _Flask: https://flask.palletsprojects.com
+
+
+Django
+------
+
+Django supports using Jinja as its template engine, see
+https://docs.djangoproject.com/en/stable/topics/templates/#support-for-template-engines.
+
+
 .. _babel-integration:
 
 Babel
diff --git a/docs/switching.rst b/docs/switching.rst
index b9ff954..caa35c3 100644
--- a/docs/switching.rst
+++ b/docs/switching.rst
@@ -1,141 +1,73 @@
-Switching from other Template Engines
+Switching From Other Template Engines
 =====================================
 
-.. highlight:: html+jinja
-
-If you have used a different template engine in the past and want to switch
-to Jinja here is a small guide that shows the basic syntactic and semantic
-changes between some common, similar text template engines for Python.
-
-Jinja 1
--------
-
-Jinja 2 is mostly compatible with Jinja 1 in terms of API usage and template
-syntax.  The differences between Jinja 1 and 2 are explained in the following
-list.
-
-API
-~~~
-
-Loaders
-    Jinja 2 uses a different loader API.  Because the internal representation
-    of templates changed there is no longer support for external caching
-    systems such as memcached.  The memory consumed by templates is comparable
-    with regular Python modules now and external caching doesn't give any
-    advantage.  If you have used a custom loader in the past have a look at
-    the new :ref:`loader API <loaders>`.
-
-Loading templates from strings
-    In the past it was possible to generate templates from a string with the
-    default environment configuration by using `jinja.from_string`.  Jinja 2
-    provides a :class:`Template` class that can be used to do the same, but
-    with optional additional configuration.
-
-Automatic unicode conversion
-    Jinja 1 performed automatic conversion of bytes in a given encoding
-    into unicode objects. This conversion is no longer implemented as it
-    was inconsistent as most libraries are using the regular Python
-    ASCII bytes to Unicode conversion. An application powered by Jinja 2
-    *has to* use unicode internally everywhere or make sure that Jinja 2
-    only gets unicode strings passed.
-
-i18n
-    Jinja 1 used custom translators for internationalization.  i18n is now
-    available as Jinja 2 extension and uses a simpler, more gettext friendly
-    interface and has support for babel.  For more details see
-    :ref:`i18n-extension`.
-
-Internal methods
-    Jinja 1 exposed a few internal methods on the environment object such
-    as `call_function`, `get_attribute` and others.  While they were marked
-    as being an internal method it was possible to override them.  Jinja 2
-    doesn't have equivalent methods.
-
-Sandbox
-    Jinja 1 was running sandbox mode by default.  Few applications actually
-    used that feature so it became optional in Jinja 2.  For more details
-    about the sandboxed execution see :class:`SandboxedEnvironment`.
-
-Context
-    Jinja 1 had a stacked context as storage for variables passed to the
-    environment.  In Jinja 2 a similar object exists but it doesn't allow
-    modifications nor is it a singleton.  As inheritance is dynamic now
-    multiple context objects may exist during template evaluation.
-
-Filters and Tests
-    Filters and tests are regular functions now.  It's no longer necessary
-    and allowed to use factory functions.
-
-
-Templates
-~~~~~~~~~
-
-Jinja 2 has mostly the same syntax as Jinja 1.  What's different is that
-macros require parentheses around the argument list now.
-
-Additionally Jinja 2 allows dynamic inheritance now and dynamic includes.
-The old helper function `rendertemplate` is gone now, `include` can be used
-instead.  Includes no longer import macros and variable assignments, for
-that the new `import` tag is used.  This concept is explained in the
-:ref:`import` documentation.
-
-Another small change happened in the `for`-tag.  The special loop variable
-doesn't have a `parent` attribute, instead you have to alias the loop
-yourself.  See :ref:`accessing-the-parent-loop` for more details.
+This is a brief guide on some of the differences between Jinja syntax
+and other template languages. See :doc:`/templates` for a comprehensive
+guide to Jinja syntax and features.
 
 
 Django
 ------
 
 If you have previously worked with Django templates, you should find
-Jinja very familiar.  In fact, most of the syntax elements look and
-work the same.
+Jinja very familiar. Many of the syntax elements look and work the same.
+However, Jinja provides some more syntax elements, and some work a bit
+differently.
 
-However, Jinja provides some more syntax elements covered in the
-documentation and some work a bit different.
+This section covers the template changes. The API, including extension
+support, is fundamentally different so it won't be covered here.
+
+Django supports using Jinja as its template engine, see
+https://docs.djangoproject.com/en/stable/topics/templates/#support-for-template-engines.
 
-This section covers the template changes.  As the API is fundamentally
-different we won't cover it here.
 
 Method Calls
 ~~~~~~~~~~~~
 
-In Django method calls work implicitly, while Jinja requires the explicit
-Python syntax. Thus this Django code::
+In Django, methods are called implicitly, without parentheses.
+
+.. code-block:: django
 
     {% for page in user.get_created_pages %}
         ...
     {% endfor %}
 
-...looks like this in Jinja::
+In Jinja, using parentheses is required for calls, like in Python. This
+allows you to pass variables to the method, which is not possible
+in Django. This syntax is also used for calling macros.
+
+.. code-block:: jinja
 
     {% for page in user.get_created_pages() %}
         ...
     {% endfor %}
 
-This allows you to pass variables to the method, which is not possible in
-Django. This syntax is also used for macros.
 
 Filter Arguments
 ~~~~~~~~~~~~~~~~
 
-Jinja provides more than one argument for filters.  Also the syntax for
-argument passing is different.  A template that looks like this in Django::
+In Django, one literal value can be passed to a filter after a colon.
+
+.. code-block:: django
 
     {{ items|join:", " }}
 
-looks like this in Jinja::
+In Jinja, filters can take any number of positional and keyword
+arguments in parentheses, like function calls. Arguments can also be
+variables instead of literal values.
 
-    {{ items|join(', ') }}
+.. code-block:: jinja
+
+    {{ items|join(", ") }}
 
-It is a bit more verbose, but it allows different types of arguments -
-including variables - and more than one of them.
 
 Tests
 ~~~~~
 
-In addition to filters there also are tests you can perform using the is
-operator.  Here are some examples::
+In addition to filters, Jinja also has "tests" used with the ``is``
+operator. This operator is not the same as the Python operator.
+
+.. code-block:: jinja
 
     {% if user.user_id is odd %}
         {{ user.username|e }} is odd
@@ -146,11 +78,10 @@ operator.  Here are some examples::
 Loops
 ~~~~~
 
-For loops work very similarly to Django, but notably the Jinja special
-variable for the loop context is called `loop`, not `forloop` as in Django.
+In Django, the special variable for the loop context is called
+``forloop``, and the ``empty`` is used for no loop items.
 
-In addition, the Django `empty` argument is called `else` in Jinja. For
-example, the Django template::
+.. code-block:: django
 
     {% for item in items %}
         {{ item }}
@@ -158,52 +89,74 @@ example, the Django template::
         No items!
     {% endfor %}
 
-...looks like this in Jinja::
+In Jinja, the special variable for the loop context is called ``loop``,
+and the ``else`` block is used for no loop items.
+
+.. code-block:: jinja
 
     {% for item in items %}
-        {{ item }}
+        {{ loop.index}}. {{ item }}
     {% else %}
         No items!
     {% endfor %}
 
+
 Cycle
 ~~~~~
 
-The ``{% cycle %}`` tag does not exist in Jinja; however, you can achieve the
-same output by using the `cycle` method on the loop context special variable.
+In Django, the ``{% cycle %}`` can be used in a for loop to alternate
+between values per loop.
 
-The following Django template::
+.. code-block:: django
 
     {% for user in users %}
         <li class="{% cycle 'odd' 'even' %}">{{ user }}</li>
     {% endfor %}
 
-...looks like this in Jinja::
+In Jinja, the ``loop`` context has a ``cycle`` method.
+
+.. code-block:: jinja
 
     {% for user in users %}
         <li class="{{ loop.cycle('odd', 'even') }}">{{ user }}</li>
     {% endfor %}
 
-There is no equivalent of ``{% cycle ... as variable %}``.
+A cycler can also be assigned to a variable and used outside or across
+loops with the ``cycle()`` global function.
 
 
 Mako
 ----
 
-.. highlight:: html+mako
+You can configure Jinja to look more like Mako:
+
+.. code-block:: python
 
-If you have used Mako so far and want to switch to Jinja you can configure
-Jinja to look more like Mako:
+    env = Environment(
+        block_start_string="<%",
+        block_end_string="%>",
+        variable_start_string="${",
+        variable_end_string="}",
+        comment_start_string="<%doc>",
+        commend_end_string="</%doc>",
+        line_statement_prefix="%",
+        line_comment_prefix="##",
+    )
 
-.. sourcecode:: python
+With an environment configured like that, Jinja should be able to
+interpret a small subset of Mako templates without any changes.
 
-    env = Environment('<%', '%>', '${', '}', '<%doc>', '</%doc>', '%', '##')
+Jinja does not support embedded Python code, so you would have to move
+that out of the template. You could either process the data with the
+same code before rendering, or add a global function or filter to the
+Jinja environment.
 
-With an environment configured like that, Jinja should be able to interpret
-a small subset of Mako templates.  Jinja does not support embedded Python
-code, so you would have to move that out of the template.  The syntax for defs
-(which are called macros in Jinja) and template inheritance is different too.
-The following Mako template::
+The syntax for defs (which are called macros in Jinja) and template
+inheritance is different too.
+
+The following Mako template:
+
+.. code-block:: mako
 
     <%inherit file="layout.html" />
     <%def name="title()">Page Title</%def>
@@ -213,7 +166,9 @@ The following Mako template::
     % endfor
     </ul>
 
-Looks like this in Jinja with the above configuration::
+Looks like this in Jinja with the above configuration:
+
+.. code-block:: jinja
 
     <% extends "layout.html" %>
     <% block title %>Page Title<% endblock %>
diff --git a/docs/templates.rst b/docs/templates.rst
index 9071ad6..7a64750 100644
--- a/docs/templates.rst
+++ b/docs/templates.rst
@@ -938,6 +938,23 @@ are available on a macro object:
 If a macro name starts with an underscore, it's not exported and can't
 be imported.
 
+Due to how scopes work in Jinja, a macro in a child template does not
+override a macro in a parent template. The following will output
+"LAYOUT", not "CHILD".
+
+.. code-block:: jinja
+    :caption: ``layout.txt``
+
+    {% macro foo() %}LAYOUT{% endmacro %}
+    {% block body %}{% endblock %}
+
+.. code-block:: jinja
+    :caption: ``child.txt``
+
+    {% extends 'layout.txt' %}
+    {% macro foo() %}CHILD{% endmacro %}
+    {% block body %}{{ foo() }}{% endblock %}
+
 
 .. _call:
 
@@ -1113,42 +1130,45 @@ at the same time.  They are documented in detail in the
 Include
 ~~~~~~~
 
-The `include` tag is useful to include a template and return the
-rendered contents of that file into the current namespace::
+The ``include`` tag renders another template and outputs the result into
+the current template.
+
+.. code-block:: jinja
 
     {% include 'header.html' %}
-        Body
+    Body goes here.
     {% include 'footer.html' %}
 
-Included templates have access to the variables of the active context by
-default.  For more details about context behavior of imports and includes,
-see :ref:`import-visibility`.
+The included template has access to context of the current template by
+default. Use ``without context`` to use a separate context instead.
+``with context`` is also valid, but is the default behavior. See
+:ref:`import-visibility`.
+
+The included template can ``extend`` another template and override
+blocks in that template. However, the current template cannot override
+any blocks that the included template outputs.
 
-From Jinja 2.2 onwards, you can mark an include with ``ignore missing``; in
-which case Jinja will ignore the statement if the template to be included
-does not exist.  When combined with ``with`` or ``without context``, it must
-be placed *before* the context visibility statement.  Here are some valid
-examples::
+Use ``ignore missing`` to ignore the statement if the template does not
+exist. It must be placed *before* a context visibility statement.
 
+.. code-block:: jinja
+
+    {% include "sidebar.html" without context %}
     {% include "sidebar.html" ignore missing %}
     {% include "sidebar.html" ignore missing with context %}
     {% include "sidebar.html" ignore missing without context %}
 
-.. versionadded:: 2.2
-
-You can also provide a list of templates that are checked for existence
-before inclusion.  The first template that exists will be included.  If
-`ignore missing` is given, it will fall back to rendering nothing if
-none of the templates exist, otherwise it will raise an exception.
+If a list of templates is given, each will be tried in order until one
+is not missing. This can be used with ``ignore missing`` to ignore if
+none of the templates exist.
 
-Example::
+.. code-block:: jinja
 
     {% include ['page_detailed.html', 'page.html'] %}
     {% include ['special_sidebar.html', 'sidebar.html'] ignore missing %}
 
-.. versionchanged:: 2.4
-   If a template object was passed to the template context, you can
-   include that object using `include`.
+A variable, with either a template name or template object, can also be
+passed to the statment.
 
 .. _import: