:Title: Public API and Backwards Compatibility Policy
:Author: Günter Milde, Adam Turner, open to all Docutils developers
:Discussions-To: docutils-develop@lists.sf.net, https://sourceforge.net/p/docutils/feature-requests/89/
:Status: Draft
:Type: Process
:Created: 2022-01-21
:Docutils-Version: 0.19


Abstract
========

.. A short (<~200 word) description of the technical issue being addressed.

This document suggests a definition of the public APIs provided by the
Docutils project and the backwards compatibility policy.


Motivation
==========

.. Clearly explain why the existing specification is inadequate to
   address the problem that the proposal solves.

Docutils has a large user base and is used in production at several
places (Python documentation, Linux kernel documentation, CMake
documentation, readthedocs, ...). OTOH, Docutils has a version number
below 1.0 (widely seen as an indicator of "beta" status of a project).

The current `Docutils Project Policies`_ section on `version
identifcation`_ concentrates on the formal definition of the version
specifier but leaves open what consists a "major change in the design
or API".

The current `backwards compatibility policy`__ is only a stub referencing
:PEP:`387`.

Clearly defining how we will balance evolution with stability is
important to both users and project developers.

.. _version specifier:
.. _version identifcation:
   ../../docutils/docs/dev/policies.html#version-identification
__ ../../docutils/docs/dev/policies.html#backwards-compatibility-policy


Rationale
=========

.. Describe why particular design decisions were made.

People affected by changes in Docutils include:

.. class:: description

Authors
  writing or maintaining reStructuredText documents.

End-Users
  of Docutils native `front-end tools`_ (optionally with 3rd-party
  drop-in extensions) or alternative tools using Docutils either as a
  library (Sphinx_, …) or via the command line interface
  (build systems, Makefiles, scripts in other languages).

Developers
  authors and maintainers of

  - projects that use Docutils as a library (Sphinx_, rsted_, Leo_,
    Pelican_, ebookmaker_, MyST_, readthedocs_, rinohtype_, …),
  - drop-in components (pycmark_, rst2pdf_, rst2beamer_, …),
  - alternative front-end tools,
  - custom stylesheets (CSS style sheets, LaTeX styles, ODT styles),
    or
  - re-implementations of the `reStructuredText specification`_,
    e.g. PanDoc_ or Text-Restructured_ (prest).

A person may belong to more than one of these catgories.

.. _Sphinx: https://www.sphinx-doc.org/
.. _rsted: https://github.com/anru/rsted
.. _Leo: https://leoeditor.com
.. _Pelican: https://docs.getpelican.com/en/latest/
.. _MyST: https://myst-parser.readthedocs.io/
.. _readthedocs: https://readthedocs.org/
.. _pycmark: https://github.com/pycmark/pycmark
.. _rst2pdf: https://rst2pdf.org/
.. _rst2beamer: https://docutils.sourceforge.io/sandbox/rst2beamer/
.. _rinohtype: https://pypi.org/project/rinohtype/
.. _ebookmaker: https://pypi.org/project/ebookmaker/
.. _PanDoc: https://pandoc.org/
.. _Text-Restructured: https://metacpan.org/dist/Text-Restructured


Specification
=============

.. Describe the syntax and semantics of any new feature.

Docutils public APIs are:

* the `reStructuredText specification`_,

* the document type definition (`Docutils Document Tree`_, docutils.dtd_),

* names, command-line arguments and behaviour of the `front-end tools`_,

* the interface to custom stylesheets --
  elements, macros and classes used by writers to represent doctree_ nodes
  in the output format,

* provided output templates and style sheets (unless marked as
  provisional_),

  .. template.txt, default.tex, titlepage.tex, xelatex.tex
  .. html4css1.css, minimal.css, docutils.sty (LaTeX), styles.odt

* the core "docutils" Python package API [#]_:

  - the `Docutils Publisher`_ interface for programmatic use,

  - component interfaces as defined by the base
    ``docutils.reader.Reader``, ``docutils.writer.Writer``, and
    ``docutils.transform.Transform`` classes,

* behaviour [#]_ and names of all documented objects.

  Python objects and variables can explicitly "opt-out" of the public API
  with a docstring noting that the object is provisional_ or internal.

  All undocumented objects should be assumed to be internal. [#]_

.. [#] see also the `API Reference Material for Client-Developers`_
.. [#] Cf. the `backwards compatibility rules`_ in :PEP:`387`.
.. [#] Cf. `Public and Internal Interfaces`_ in :PEP:`008`.
.. _Public and Internal Interfaces:
    https://peps.python.org/pep-0008/#public-and-internal-interfaces

Backwards Compatibility
=======================

.. Describe potential impact and severity on pre-existing code.

Beginning with version 1.0, Docutils will follow the rules of
`Semantic Versioning`_. All incompatible changes to the public APIs
require increasing the `major` part of the `version specifier`_.
Backwards compatible changes can be done in `minor` releases.


Security Implications
=====================

If required, critical bug fixes may change the public API without advance
warning.


How to Teach This
=================

.. How to teach users, new and experienced,
   how to apply the proposal to their work.

* Move the API specification_ and the backwards compatibility declaration
  to the `Docutils Project Policies`_.

* Complete the `api/`_ documentation and keep it up to date.

* Generate "docutils" package API documentation from the docstrings:

  - Fix/enhance/add docstrings to improve the output of `pydoc`_.

  - Generate API documentation with Sphinx:

    - nicely format rST docstrings
    - include attribute docstrings (ignored by pydoc_).

* Put the following text at a suitable place in the documentation:

     To find out if an object from the "docutils" package is safe to use,
     look up its docstring and the docstring of its parent [#]_.

     If there is no documentation or the documentation says "provisional" or
     "internal", the name, behaviour, and existence of the object is not
     guaranteed to be stable.

     Code relying on non-public objects should be made robust using public
     alternatives. If there is a no such alternative or the required change
     would be a major problem, contact the Docutils developers.

     .. [#] `Attribute docstrings`_ are not shown by pydoc_. To find out
        whether attributes have a docstring, check the source.

     .. _attribute docstrings:
         https://peps.python.org/pep-0258/#attribute-docstrings
     .. _pydoc: https://docs.python.org/3/library/pydoc.html


Rejected Ideas
==============

.. Why certain ideas that were brought while discussing this proposal were not
   ultimately pursued.

* Use type annotations as an indication of status in the public API.

  - There is no known precedence for this approach.
  - Type annotations may be helpful also for non-public code.

* Use Calendar Versioning (CalVer).

  - Would be a break from current versioning without clear advantages.

* Allow breaking API changes in *minor* versions after prior announcement
  and a deprecation period.

  - Breaks the principle of least surprise.

* Enumerate all modules, classes, and functions that form the public API.

* Mark all private objects with a prefix underscore.

  - May needlessly break applications that use "internal" objects by the
    current name.
  - Too much work.


Open Issues
===========

.. Any points that are still being decided/discussed.

* Differentiate between "core API" and "extended API"?

  Cf. the `Docutils Project Policies`_

    When Docutils reaches version 1.0, the major APIs will be considered
    frozen.

    The major number [...] may be incremented later if there is a major
    change in the design or API.


* Formalise the wording for docstrings for public/private/provisional
  (ideally this would be a single regex pattern)?

  * The keyword provisional_ is well defined. ✓
  * Use "private" or "internal"?

* Declare only objects included in the ``__all__`` attribute of their
  parent objects as public rsp. explicitely list all public objects in
  ``__all__`` attribute of their parents?

  This would hide private objects from `pydoc` help on the parent objects.

* Define a minimum deprecation time similar to Docbook__? E.g.

    * A "major" release may contain backward-incompatible changes if:

      * the change was announced in the release notes for the previous
        version (major or minor) and
      * the change was announced in a release that occurred at least six
        months previously.

    By these rules, Docutils developers can announce, in release 5.1, for
    example, its plans to make a backward-incompatible change in release 6.0.
    Then, in 6.0, if it’s been at least six months since 5.1 was
    released, they can make that change.

  __ https://tdg.docbook.org/tdg/5.1/ch01.html#bwcompat


References
==========

.. A collection of URLs used as references through the proposal.

.. _api/:
.. _API Reference Material for Client-Developers:
    ../../docutils/docs/index.html
    #api-api-reference-material-for-client-developers
.. _doctree:
.. _Docutils Document Tree: ../../docutils/docs/ref/doctree.html
.. _docutils.dtd: ../../docutils/docs/ref/docutils.dtd
.. _Docutils Design Specification: ../../docutils/docs/peps/pep-0258.html
.. _Docutils Project Policies: ../../docutils/docs/dev/policies.html
.. _front-end tools: ../../docutils/docs/user/tools.html
.. _Docutils Publisher: ../../docutils/docs/api/publisher.html
.. _Docutils Transforms: ../../docutils/docs/ref/transforms.html
.. _HISTORY: ../../docutils/HISTORY.html
.. _RELEASE-NOTES: ../../docutils/RELEASE-NOTES.html
.. _reStructuredText specification:
    ../../docutils/docs/ref/rst/restructuredtext.html

.. _backwards compatibility rules:
    https://peps.python.org/pep-0387/#backwards-compatibility-rules
.. _provisional: https://docs.python.org/3/glossary.html#term-provisional-API
.. _Semantic Versioning: https://semver.org/

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.



..
    Local Variables:
    mode: indented-text
    indent-tabs-mode: nil
    sentence-end-double-space: t
    fill-column: 70
    coding: utf-8
    End:
