diff options
Diffstat (limited to 'docs/userguide')
| -rw-r--r-- | docs/userguide/datafiles.rst | 75 | ||||
| -rw-r--r-- | docs/userguide/declarative_config.rst | 27 | ||||
| -rw-r--r-- | docs/userguide/dependency_management.rst | 178 | ||||
| -rw-r--r-- | docs/userguide/distribution.rst | 2 | ||||
| -rw-r--r-- | docs/userguide/entry_point.rst | 31 | ||||
| -rw-r--r-- | docs/userguide/extension.rst | 4 | ||||
| -rw-r--r-- | docs/userguide/index.rst | 1 | ||||
| -rw-r--r-- | docs/userguide/miscellaneous.rst | 63 | ||||
| -rw-r--r-- | docs/userguide/package_discovery.rst | 455 | ||||
| -rw-r--r-- | docs/userguide/pyproject_config.rst | 218 | ||||
| -rw-r--r-- | docs/userguide/quickstart.rst | 331 |
11 files changed, 1151 insertions, 234 deletions
diff --git a/docs/userguide/datafiles.rst b/docs/userguide/datafiles.rst index 69cf36e6..9817e639 100644 --- a/docs/userguide/datafiles.rst +++ b/docs/userguide/datafiles.rst @@ -5,11 +5,11 @@ Data Files Support The distutils have traditionally allowed installation of "data files", which are placed in a platform-specific location. However, the most common use case for data files distributed with a package is for use *by* the package, usually -by including the data files in the package directory. +by including the data files **inside the package directory**. -Setuptools offers three ways to specify data files to be included in your -packages. First, you can simply use the ``include_package_data`` keyword, -e.g.:: +Setuptools offers three ways to specify this most common type of data files to +be included in your package's [#datafiles]_. +First, you can simply use the ``include_package_data`` keyword, e.g.:: from setuptools import setup, find_packages setup( @@ -18,9 +18,10 @@ e.g.:: ) This tells setuptools to install any data files it finds in your packages. -The data files must be specified via the distutils' ``MANIFEST.in`` file. +The data files must be specified via the |MANIFEST.in|_ file. (They can also be tracked by a revision control system, using an appropriate -plugin. See the section below on :ref:`Adding Support for Revision +plugin such as :pypi:`setuptools-scm` or :pypi:`setuptools-svn`. +See the section below on :ref:`Adding Support for Revision Control Systems` for information on how to write such plugins.) If you want finer-grained control over what files are included (for example, @@ -87,14 +88,13 @@ When building an ``sdist``, the datafiles are also drawn from the ``package_name.egg-info/SOURCES.txt`` file, so make sure that this is removed if the ``setup.py`` ``package_data`` list is updated before calling ``setup.py``. -(Note: although the ``package_data`` argument was previously only available in -``setuptools``, it was also added to the Python ``distutils`` package as of -Python 2.4; there is `some documentation for the feature`__ available on the -python.org website. If using the setuptools-specific ``include_package_data`` -argument, files specified by ``package_data`` will *not* be automatically -added to the manifest unless they are listed in the MANIFEST.in file.) +.. note:: + If using the ``include_package_data`` argument, files specified by + ``package_data`` will *not* be automatically added to the manifest unless + they are listed in the |MANIFEST.in|_ file or by a plugin like + :pypi:`setuptools-scm` or :pypi:`setuptools-svn`. -__ https://docs.python.org/3/distutils/setupscript.html#installing-package-data +.. https://docs.python.org/3/distutils/setupscript.html#installing-package-data Sometimes, the ``include_package_data`` or ``package_data`` options alone aren't sufficient to precisely define what files you want included. For @@ -125,11 +125,13 @@ included as a result of using ``include_package_data``. In summary, the three options allow you to: ``include_package_data`` - Accept all data files and directories matched by ``MANIFEST.in``. + Accept all data files and directories matched by |MANIFEST.in|_ or added by + a :ref:`plugin <Adding Support for Revision Control Systems>`. ``package_data`` Specify additional patterns to match files that may or may - not be matched by ``MANIFEST.in`` or found in source control. + not be matched by |MANIFEST.in|_ or added by + a :ref:`plugin <Adding Support for Revision Control Systems>`. ``exclude_package_data`` Specify patterns for data files and directories that should *not* be @@ -154,14 +156,22 @@ Typically, existing programs manipulate a package's ``__file__`` attribute in order to find the location of data files. However, this manipulation isn't compatible with PEP 302-based import hooks, including importing from zip files and Python Eggs. It is strongly recommended that, if you are using data files, -you should use the :ref:`ResourceManager API` of ``pkg_resources`` to access -them. The ``pkg_resources`` module is distributed as part of setuptools, so if -you're using setuptools to distribute your package, there is no reason not to -use its resource management API. See also `Importlib Resources`_ for -a quick example of converting code that uses ``__file__`` to use -``pkg_resources`` instead. +you should use :mod:`importlib.resources` to access them. +:mod:`importlib.resources` was added to Python 3.7 and the latest version of +the library is also available via the :pypi:`importlib-resources` backport. +See :doc:`importlib-resources:using` for detailed instructions [#importlib]_. + +.. tip:: Files inside the package directory should be *read-only* to avoid a + series of common problems (e.g. when multiple users share a common Python + installation, when the package is loaded from a zip file, or when multiple + instances of a Python application run in parallel). -.. _Importlib Resources: https://docs.python.org/3/library/importlib.html#module-importlib.resources + If your Python package needs to write to a file for shared data or configuration, + you can use standard platform/OS-specific system directories, such as + ``~/.local/config/$appname`` or ``/usr/share/$appname/$version`` (Linux specific) [#system-dirs]_. + A common approach is to add a read-only template file to the package + directory that is then copied to the correct system directory if no + pre-existing file is found. Non-Package Data Files @@ -174,4 +184,23 @@ fall back to the platform-specific location for installing data files, there is no supported facility to reliably retrieve these resources. Instead, the PyPA recommends that any data files you wish to be accessible at -run time be included in the package. +run time be included **inside the package**. + + +---- + +.. [#datafiles] ``setuptools`` consider a *package data file* any non-Python + file **inside the package directory** (i.e., that co-exists in the same + location as the regular ``.py`` files being distributed). + +.. [#system-dirs] These locations can be discovered with the help of + third-party libraries such as :pypi:`platformdirs`. + +.. [#importlib] Recent versions of :mod:`importlib.resources` available in + Pythons' standard library should be API compatible with + :pypi:`importlib-metadata`. However this might vary depending on which version + of Python is installed. + + +.. |MANIFEST.in| replace:: ``MANIFEST.in`` +.. _MANIFEST.in: https://packaging.python.org/en/latest/guides/using-manifest-in/ diff --git a/docs/userguide/declarative_config.rst b/docs/userguide/declarative_config.rst index d1c25df1..52379dbf 100644 --- a/docs/userguide/declarative_config.rst +++ b/docs/userguide/declarative_config.rst @@ -1,8 +1,8 @@ .. _declarative config: ------------------------------------------ -Configuring setup() using setup.cfg files ------------------------------------------ +------------------------------------------------ +Configuring setuptools using ``setup.cfg`` files +------------------------------------------------ .. note:: New in 30.3.0 (8 Dec 2016). @@ -24,27 +24,22 @@ boilerplate code in some cases. [metadata] name = my_package - version = attr: src.VERSION + version = attr: my_package.VERSION description = My package description long_description = file: README.rst, CHANGELOG.rst, LICENSE.rst keywords = one, two license = BSD 3-Clause License classifiers = Framework :: Django - License :: OSI Approved :: BSD License Programming Language :: Python :: 3 - Programming Language :: Python :: 3.5 [options] zip_safe = False include_package_data = True packages = find: - scripts = - bin/first.py - bin/second.py install_requires = requests - importlib; python_version == "2.6" + importlib-metadata; python_version<"3.8" [options.package_data] * = *.txt, *.rst @@ -52,7 +47,7 @@ boilerplate code in some cases. [options.entry_points] console_scripts = - executable-name = package.module:function + executable-name = my_package.module:function [options.extras_require] pdf = ReportLab>=1.2; RXP @@ -60,8 +55,10 @@ boilerplate code in some cases. [options.packages.find] exclude = - src.subpackage1 - src.subpackage2 + examples* + tools* + docs* + my_package.tests* Metadata and options are set in the config sections of the same name. @@ -222,10 +219,10 @@ data_files section 40.6.0 [# .. [#opt-1] In the ``package_data`` section, a key named with a single asterisk (``*``) refers to all packages, in lieu of the empty string used in ``setup.py``. - + .. [#opt-2] In the ``extras_require`` section, values are parsed as ``list-semi``. This implies that in order to include markers, they **must** be *dangling*: - + .. code-block:: ini [options.extras_require] diff --git a/docs/userguide/dependency_management.rst b/docs/userguide/dependency_management.rst index 23578a57..d15b45cb 100644 --- a/docs/userguide/dependency_management.rst +++ b/docs/userguide/dependency_management.rst @@ -28,7 +28,7 @@ other two types of dependency keyword, this one is specified in your .. code-block:: ini [build-system] - requires = ["setuptools", "wheel"] + requires = ["setuptools"] #... .. note:: @@ -43,7 +43,7 @@ other two types of dependency keyword, this one is specified in your Declaring required dependency ============================= This is where a package declares its core dependencies, without which it won't -be able to run. ``setuptools`` support automatically download and install +be able to run. ``setuptools`` supports automatically downloading and installing these dependencies when the package is installed. Although there is more finesse to it, let's start with a simple example. @@ -69,6 +69,18 @@ finesse to it, let's start with a simple example. ], ) +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [project] + # ... + dependencies = [ + "docutils", + "BazSpam == 1.1", + ] + # ... + When your project is installed (e.g. using pip), all of the dependencies not already installed will be located (via PyPI), downloaded, built (if necessary), @@ -78,7 +90,7 @@ that verify the availability of the specified dependencies at runtime. Platform specific dependencies ------------------------------ -Setuptools offer the capability to evaluate certain conditions before blindly +Setuptools offers the capability to evaluate certain conditions before blindly installing everything listed in ``install_requires``. This is great for platform specific dependencies. For example, the ``enum`` package was added in Python 3.4, therefore, package that depends on it can elect to install it only when @@ -104,6 +116,17 @@ the Python version is older than 3.4. To accomplish this ], ) +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [project] + # ... + dependencies = [ + "enum34; python_version<'3.4'", + ] + # ... + Similarly, if you also wish to declare ``pywin32`` with a minimal version of 1.0 and only install it if the user is using a Windows operating system: @@ -129,6 +152,18 @@ and only install it if the user is using a Windows operating system: ], ) +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [project] + # ... + dependencies = [ + "enum34; python_version<'3.4'", + "pywin32 >= 1.0; platform_system=='Windows'", + ] + # ... + The environmental markers that may be used for testing platform types are detailed in `PEP 508 <https://www.python.org/dev/peps/pep-0508/>`_. @@ -215,9 +250,9 @@ distributions, if the package's dependencies aren't already installed: Optional dependencies ===================== Setuptools allows you to declare dependencies that only get installed under -specific circumstances. These dependencies are specified with ``extras_require`` +specific circumstances. These dependencies are specified with the ``extras_require`` keyword and are only installed if another package depends on it (either -directly or indirectly) This makes it convenient to declare dependencies for +directly or indirectly). This makes it convenient to declare dependencies for ancillary functions such as "tests" and "docs". .. note:: @@ -249,50 +284,18 @@ dependencies for it to work: }, ) -The name ``PDF`` is an arbitrary identifier of such a list of dependencies, to -which other components can refer and have them installed. There are two common -use cases. - -First is the console_scripts entry point: +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ -.. tab:: setup.cfg + .. code-block:: toml - .. code-block:: ini + # ... + [project.optional-dependencies] + PDF = ["ReportLab>=1.2", "RXP"] - [metadata] - name = Project A - #... - - [options] - #... - entry_points= - [console_scripts] - rst2pdf = project_a.tools.pdfgen [PDF] - rst2html = project_a.tools.htmlgen - -.. tab:: setup.py - - .. code-block:: python - - setup( - name="Project-A", - ..., - entry_points={ - "console_scripts": [ - "rst2pdf = project_a.tools.pdfgen [PDF]", - "rst2html = project_a.tools.htmlgen", - ], - }, - ) - -This syntax indicates that the entry point (in this case a console script) -is only valid when the PDF extra is installed. It is up to the installer -to determine how to handle the situation where PDF was not indicated -(e.g. omit the console script, provide a warning when attempting to load -the entry point, assume the extras are present and let the implementation -fail later). +The name ``PDF`` is an arbitrary identifier of such a list of dependencies, to +which other components can refer and have them installed. -The second use case is that other package can use this "extra" for their +A use case for this approach is that other package can use this "extra" for their own dependencies. For example, if "Project-B" needs "project A" with PDF support installed, it might declare the dependency like this: @@ -319,6 +322,17 @@ installed, it might declare the dependency like this: ..., ) +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [project] + name = "Project-B" + # ... + dependencies = [ + "Project-A[PDF]" + ] + This will cause ReportLab to be installed along with project A, if project B is installed -- even if project A was already installed. In this way, a project can encapsulate groups of optional "downstream dependencies" under a feature @@ -329,18 +343,61 @@ ReportLab in order to provide PDF support, Project B's setup information does not need to change, but the right packages will still be installed if needed. .. note:: - Best practice: if a project ends up not needing any other packages to + Best practice: if a project ends up no longer needing any other packages to support a feature, it should keep an empty requirements list for that feature in its ``extras_require`` argument, so that packages depending on that feature don't break (due to an invalid feature name). +Historically ``setuptools`` also used to support extra dependencies in console +scripts, for example: + +.. tab:: setup.cfg + + .. code-block:: ini + + [metadata] + name = Project A + #... + + [options] + #... + entry_points= + [console_scripts] + rst2pdf = project_a.tools.pdfgen [PDF] + rst2html = project_a.tools.htmlgen + +.. tab:: setup.py + + .. code-block:: python + + setup( + name="Project-A", + ..., + entry_points={ + "console_scripts": [ + "rst2pdf = project_a.tools.pdfgen [PDF]", + "rst2html = project_a.tools.htmlgen", + ], + }, + ) + +This syntax indicates that the entry point (in this case a console script) +is only valid when the PDF extra is installed. It is up to the installer +to determine how to handle the situation where PDF was not indicated +(e.g. omit the console script, provide a warning when attempting to load +the entry point, assume the extras are present and let the implementation +fail later). + +.. warning:: + ``pip`` and other tools might not support this use case for extra + dependencies, therefore this practice is considered **deprecated**. + See :doc:`PyPUG:specifications/entry-points`. + Python requirement ================== In some cases, you might need to specify the minimum required python version. -This is handled with the ``python_requires`` keyword supplied to ``setup.cfg`` -or ``setup.py``. - +This can be configured as shown in the example below. .. tab:: setup.cfg @@ -360,6 +417,27 @@ or ``setup.py``. setup( name="Project-B", - python_requires=[">=3.6"], + python_requires=">=3.6", ..., ) + + +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [project] + name = "Project-B" + requires-python = ">=3.6" + # ... + +---- + +.. rubric:: Notes + +.. [#experimental] + While the ``[build-system]`` table should always be specified in the + ``pyproject.toml`` file, support for adding package metadata and build configuration + options via the ``[project]`` and ``[tool.setuptools]`` tables is still + experimental and might change (or be completely removed) in future releases. + See :doc:`/userguide/pyproject_config`. diff --git a/docs/userguide/distribution.rst b/docs/userguide/distribution.rst index 2872dacd..db0f1a5f 100644 --- a/docs/userguide/distribution.rst +++ b/docs/userguide/distribution.rst @@ -162,7 +162,7 @@ Specifying Your Project's Version --------------------------------- Setuptools can work well with most versioning schemes. Over the years, -setuptools has tried to closely follow the +setuptools has tried to closely follow the `PEP 440 <https://www.python.org/dev/peps/pep-0440/>`_ scheme, but it also supports legacy versions. There are, however, a few special things to watch out for, in order to ensure that setuptools and diff --git a/docs/userguide/entry_point.rst b/docs/userguide/entry_point.rst index 21edc697..b97419c4 100644 --- a/docs/userguide/entry_point.rst +++ b/docs/userguide/entry_point.rst @@ -54,11 +54,32 @@ above example, to create a command ``hello-world`` that invokes ``timmins.hello_world``, add a console script entry point to ``setup.cfg``: -.. code-block:: ini +.. tab:: setup.cfg + + .. code-block:: ini + + [options.entry_points] + console_scripts = + hello-world = timmins:hello_world + +.. tab:: setup.py + + .. code-block:: python + + from setuptools import setup + + setup( + name='timmins', + version='0.0.1', + packages=['timmins'], + # ... + entry_points={ + 'console_scripts': [ + 'hello-world=timmins:hello_world', + ] + } + ) - [options.entry_points] - console_scripts = - hello-world = timmins:hello_world After installing the package, a user may invoke that function by simply calling ``hello-world`` on the command line. @@ -99,7 +120,7 @@ and tools like ``pip`` create wrapper scripts that invoke those commands. For a project wishing to solicit entry points, Setuptools recommends the `importlib.metadata <https://docs.python.org/3/library/importlib.metadata.html>`_ module (part of stdlib since Python 3.8) or its backport, -`importlib_metadata <https://pypi.org/project/importlib_metadata>`_. +:pypi:`importlib_metadata`. For example, to find the console script entry points from the example above: diff --git a/docs/userguide/extension.rst b/docs/userguide/extension.rst index d74ca3fe..21fb05b6 100644 --- a/docs/userguide/extension.rst +++ b/docs/userguide/extension.rst @@ -194,8 +194,8 @@ Adding Support for Revision Control Systems If the files you want to include in the source distribution are tracked using Git, Mercurial or SVN, you can use the following packages to achieve that: -- Git and Mercurial: `setuptools_scm <https://pypi.org/project/setuptools_scm/>`_ -- SVN: `setuptools_svn <https://pypi.org/project/setuptools_svn/>`_ +- Git and Mercurial: :pypi:`setuptools_scm` +- SVN: :pypi:`setuptools_svn` If you would like to create a plugin for ``setuptools`` to find files tracked by another revision control system, you can do so by adding an entry point to diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index eca5a85a..49655acd 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -31,6 +31,7 @@ quickstart provides an overview of the new workflow. distribution extension declarative_config + pyproject_config keywords commands functionalities_rewrite diff --git a/docs/userguide/miscellaneous.rst b/docs/userguide/miscellaneous.rst index 3df327d7..5fd2f0a8 100644 --- a/docs/userguide/miscellaneous.rst +++ b/docs/userguide/miscellaneous.rst @@ -94,3 +94,66 @@ correctly when installed as a zipfile, correct any problems if you can, and then make an explicit declaration of ``True`` or ``False`` for the ``zip_safe`` flag, so that it will not be necessary for ``bdist_egg`` to try to guess whether your project can work as a zipfile. + + +.. _Controlling files in the distribution: + +Controlling files in the distribution +------------------------------------- + +For the most common use cases, ``setuptools`` will automatically find out which +files are necessary for distributing the package. +This includes all :term:`pure Python modules <Pure Module>` in the +``py_modules`` or ``packages`` configuration, and the C sources (but not C +headers) listed as part of extensions when creating a :term:`Source +Distribution (or "sdist")`. + +However, when building more complex packages (e.g. packages that include +non-Python files, or that need to use custom C headers), you might find that +not all files present in your project folder are included in package +:term:`distribution archive <Distribution Package>`. + +In these situations you can use a ``setuptools`` +:ref:`plugin <Adding Support for Revision Control Systems>`, +such as :pypi:`setuptools-scm` or :pypi:`setuptools-svn` to automatically +include all files tracked by your Revision Control System into the ``sdist``. + +.. _Using MANIFEST.in: + +Alternatively, if you need finer control, you can add a ``MANIFEST.in`` file at +the root of your project. +This file contains instructions that tell ``setuptools`` which files exactly +should be part of the ``sdist`` (or not). +A comprehensive guide to ``MANIFEST.in`` syntax is available at the +:doc:`PyPA's Packaging User Guide <PyPUG:guides/using-manifest-in>`. + +Once the correct files are present in the ``sdist``, they can then be used by +binary extensions during the build process, or included in the final +:term:`wheel <Wheel>` [#build-process]_ if you configure ``setuptools`` with +``include_package_data=True``. + +.. important:: + Please note that, when using ``include_package_data=True``, only files **inside + the package directory** are included in the final ``wheel``, by default. + + So for example, if you create a :term:`Python project <Project>` that uses + :pypi:`setuptools-scm` and have a ``tests`` directory outside of the package + folder, the ``tests`` directory will be present in the ``sdist`` but not in the + ``wheel`` [#wheel-vs-sdist]_. + + See :doc:`/userguide/datafiles` for more information. + +---- + +.. [#build-process] + You can think about the build process as two stages: first the ``sdist`` + will be created and then the ``wheel`` will be produced from that ``sdist``. + +.. [#wheel-vs-sdist] + This happens because the ``sdist`` can contain files that are useful during + development or the build process itself, but not in runtime (e.g. tests, + docs, examples, etc...). + The ``wheel``, on the other hand, is a file format that has been optimized + and is ready to be unpacked into a running installation of Python or + :term:`Virtual Environment`. + Therefore it only contains items that are required during runtime. diff --git a/docs/userguide/package_discovery.rst b/docs/userguide/package_discovery.rst index 61da2d66..38119bc6 100644 --- a/docs/userguide/package_discovery.rst +++ b/docs/userguide/package_discovery.rst @@ -16,8 +16,9 @@ Package Discovery and Namespace Package place to start. ``Setuptools`` provide powerful tools to handle package discovery, including -support for namespace package. Normally, you would specify the package to be -included manually in the following manner: +support for namespace package. + +Normally, you would specify the package to be included manually in the following manner: .. tab:: setup.cfg @@ -38,8 +39,205 @@ included manually in the following manner: packages=['mypkg1', 'mypkg2'] ) -This can get tiresome really quickly. To speed things up, we introduce two -functions provided by setuptools: +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + # ... + [tool.setuptools] + packages = ["mypkg1", "mypkg2"] + # ... + + +If your packages are not in the root of the repository you also need to +configure ``package_dir``: + +.. tab:: setup.cfg + + .. code-block:: ini + + [options] + # ... + package_dir = + = src + # directory containing all the packages (e.g. src/mypkg1, src/mypkg2) + # OR + package_dir = + mypkg1 = lib1 + # mypkg1.mod corresponds to lib1/mod.py + # mypkg1.subpkg.mod corresponds to lib1/subpkg/mod.py + mypkg2 = lib2 + # mypkg2.mod corresponds to lib2/mod.py + mypkg2.subpkg = lib3 + # mypkg2.subpkg.mod corresponds to lib3/mod.py + +.. tab:: setup.py + + .. code-block:: python + + setup( + # ... + package_dir = {"": "src"} + # directory containing all the packages (e.g. src/mypkg1, src/mypkg2) + ) + + # OR + + setup( + # ... + package_dir = { + "mypkg1": "lib1", # mypkg1.mod corresponds to lib1/mod.py + # mypkg1.subpkg.mod corresponds to lib1/subpkg/mod.py + "mypkg2": "lib2", # mypkg2.mod corresponds to lib2/mod.py + "mypkg2.subpkg": "lib3" # mypkg2.subpkg.mod corresponds to lib3/mod.py + # ... + ) + +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [tool.setuptools] + # ... + package-dir = {"" = "src"} + # directory containing all the packages (e.g. src/mypkg1, src/mypkg2) + + # OR + + [tool.setuptools.package-dir] + mypkg1 = "lib1" + # mypkg1.mod corresponds to lib1/mod.py + # mypkg1.subpkg.mod corresponds to lib1/subpkg/mod.py + mypkg2 = "lib2" + # mypkg2.mod corresponds to lib2/mod.py + "mypkg2.subpkg" = "lib3" + # mypkg2.subpkg.mod corresponds to lib3/mod.py + # ... + +This can get tiresome really quickly. To speed things up, you can rely on +setuptools automatic discovery, or use the provided tools, as explained in +the following sections. + + +.. _auto-discovery: + +Automatic discovery +=================== + +.. warning:: Automatic discovery is an **experimental** feature and might change + (or be completely removed) in the future. + See :ref:`custom-discovery` for a stable way of configuring ``setuptools``. + +By default ``setuptools`` will consider 2 popular project layouts, each one with +its own set of advantages and disadvantages [#layout1]_ [#layout2]_ as +discussed in the following sections. + +Setuptools will automatically scan your project directory looking for these +layouts and try to guess the correct values for the :ref:`packages <declarative +config>` and :doc:`py_modules </references/keywords>` configuration. + +.. important:: + Automatic discovery will **only** be enabled if you **don't** provide any + configuration for ``packages`` and ``py_modules``. + If at least one of them is explicitly set, automatic discovery will not take place. + + **Note**: specifying ``ext_modules`` might also prevent auto-discover from + taking place, unless your opt into :doc:`pyproject_config` (which will + disable the backward compatible behaviour). + +.. _src-layout: + +src-layout +---------- +The project should contain a ``src`` directory under the project root and +all modules and packages meant for distribution are placed inside this +directory:: + + project_root_directory + ├── pyproject.toml + ├── setup.cfg # or setup.py + ├── ... + └── src/ + └── mypkg/ + ├── __init__.py + ├── ... + └── mymodule.py + +This layout is very handy when you wish to use automatic discovery, +since you don't have to worry about other Python files or folders in your +project root being distributed by mistake. In some circumstances it can be +also less error-prone for testing or when using :pep:`420`-style packages. +On the other hand you cannot rely on the implicit ``PYTHONPATH=.`` to fire +up the Python REPL and play with your package (you will need an +`editable install`_ to be able to do that). + +.. _flat-layout: + +flat-layout +----------- +*(also known as "adhoc")* + +The package folder(s) are placed directly under the project root:: + + project_root_directory + ├── pyproject.toml + ├── setup.cfg # or setup.py + ├── ... + └── mypkg/ + ├── __init__.py + ├── ... + └── mymodule.py + +This layout is very practical for using the REPL, but in some situations +it can be more error-prone (e.g. during tests or if you have a bunch +of folders or Python files hanging around your project root) + +To avoid confusion, file and folder names that are used by popular tools (or +that correspond to well-known conventions, such as distributing documentation +alongside the project code) are automatically filtered out in the case of +*flat-layout*: + +.. autoattribute:: setuptools.discovery.FlatLayoutPackageFinder.DEFAULT_EXCLUDE + +.. autoattribute:: setuptools.discovery.FlatLayoutModuleFinder.DEFAULT_EXCLUDE + +.. warning:: + If you are using auto-discovery with *flat-layout*, ``setuptools`` will + refuse to create :term:`distribution archives <Distribution Package>` with + multiple top-level packages or modules. + + This is done to prevent common errors such as accidentally publishing code + not meant for distribution (e.g. maintenance-related scripts). + + Users that purposefully want to create multi-package distributions are + advised to use :ref:`custom-discovery` or the ``src-layout``. + +There is also a handy variation of the *flat-layout* for utilities/libraries +that can be implemented with a single Python file: + +single-module distribution +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A standalone module is placed directly under the project root, instead of +inside a package folder:: + + project_root_directory + ├── pyproject.toml + ├── setup.cfg # or setup.py + ├── ... + └── single_file_lib.py + + +.. _custom-discovery: + +Custom discovery +================ + +If the automatic discovery does not work for you +(e.g., you want to *include* in the distribution top-level packages with +reserved names such as ``tasks``, ``example`` or ``docs``, or you want to +*exclude* nested packages that would be otherwise included), you can use +the provided tools for package discovery: .. tab:: setup.cfg @@ -55,29 +253,41 @@ functions provided by setuptools: .. code-block:: python from setuptools import find_packages - # or from setuptools import find_namespace_packages +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ -Using ``find:`` or ``find_packages`` -==================================== -Let's start with the first tool. ``find:`` (``find_packages``) takes a source -directory and two lists of package name patterns to exclude and include, and -then return a list of ``str`` representing the packages it could find. To use -it, consider the following directory - -.. code-block:: bash + .. code-block:: toml - mypkg/ - src/ - pkg1/__init__.py - pkg2/__init__.py - additional/__init__.py + # ... + [tool.setuptools.packages] + find = {} # Scanning implicit namespaces is active by default + # OR + find = {namespace = false} # Disable implicit namespaces - setup.cfg #or setup.py -To have your setup.cfg or setup.py to automatically include packages found +Finding simple packages +----------------------- +Let's start with the first tool. ``find:`` (``find_packages()``) takes a source +directory and two lists of package name patterns to exclude and include, and +then return a list of ``str`` representing the packages it could find. To use +it, consider the following directory:: + + mypkg + ├── setup.cfg # and/or setup.py, pyproject.toml + └── src + ├── pkg1 + │ └── __init__.py + ├── pkg2 + │ └── __init__.py + ├── aditional + │ └── __init__.py + └── pkg + └── namespace + └── __init__.py + +To have setuptools to automatically include packages found in ``src`` that starts with the name ``pkg`` and not ``additional``: .. tab:: setup.cfg @@ -94,6 +304,10 @@ in ``src`` that starts with the name ``pkg`` and not ``additional``: include = pkg* exclude = additional + .. note:: + ``pkg`` does not contain an ``__init__.py`` file, therefore + ``pkg.namespace`` is ignored by ``find:`` (see ``find_namespace:`` below). + .. tab:: setup.py .. code-block:: python @@ -110,16 +324,53 @@ in ``src`` that starts with the name ``pkg`` and not ``additional``: ) + .. note:: + ``pkg`` does not contain an ``__init__.py`` file, therefore + ``pkg.namespace`` is ignored by ``find_packages()`` + (see ``find_namespace_packages()`` below). + +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [tool.setuptools.packages.find] + where = ["src"] + include = ["pkg*"] + exclude = ["additional"] + namespaces = false + + .. note:: + When using ``tool.setuptools.packages.find`` in ``pyproject.toml``, + setuptools will consider :pep:`implicit namespaces <420>` by default when + scanning your project directory. + To avoid ``pkg.namespace`` from being added to your package list + you can set ``namespaces = false``. This will prevent any folder + without an ``__init__.py`` file from being scanned. + +.. important:: + ``include`` and ``exclude`` accept strings representing :mod:`glob` patterns. + These patterns should match the **full** name of the Python module (as if it + was written in an ``import`` statement). + + For example if you have ``util`` pattern, it will match + ``util/__init__.py`` but not ``util/files/__init__.py``. + + The fact that the parent package is matched by the pattern will not dictate + if the submodule will be included or excluded from the distribution. + You will need to explicitly add a wildcard (e.g. ``util*``) + if you want the pattern to also match submodules. + .. _Namespace Packages: -Using ``find_namespace:`` or ``find_namespace_packages`` -======================================================== -``setuptools`` provides the ``find_namespace:`` (``find_namespace_packages``) -which behaves similarly to ``find:`` but works with namespace package. Before -diving in, it is important to have a good understanding of what namespace -packages are. Here is a quick recap: +Finding namespace packages +-------------------------- +``setuptools`` provides the ``find_namespace:`` (``find_namespace_packages()``) +which behaves similarly to ``find:`` but works with namespace package. + +Before diving in, it is important to have a good understanding of what +:pep:`namespace packages <420>` are. Here is a quick recap. -Suppose you have two packages named as follows: +When you have two packages organized as follows: .. code-block:: bash @@ -128,7 +379,7 @@ Suppose you have two packages named as follows: If both ``Desktop`` and ``Library`` are on your ``PYTHONPATH``, then a namespace package called ``timmins`` will be created automatically for you when -you invoke the import mechanism, allowing you to accomplish the following +you invoke the import mechanism, allowing you to accomplish the following: .. code-block:: pycon @@ -137,49 +388,108 @@ you invoke the import mechanism, allowing you to accomplish the following as if there is only one ``timmins`` on your system. The two packages can then be distributed separately and installed individually without affecting the -other one. Suppose you are packaging the ``foo`` part: +other one. -.. code-block:: bash +Now, suppose you decide to package the ``foo`` part for distribution and start +by creating a project directory organized as follows:: + + foo + ├── setup.cfg # and/or setup.py, pyproject.toml + └── src + └── timmins + └── foo + └── __init__.py - foo/ - src/ - timmins/foo/__init__.py - setup.cfg # or setup.py +If you want the ``timmins.foo`` to be automatically included in the +distribution, then you will need to specify: -and you want the ``foo`` to be automatically included, ``find:`` won't work -because timmins doesn't contain ``__init__.py`` directly, instead, you have -to use ``find_namespace:``: +.. tab:: setup.cfg -.. code-block:: ini + .. code-block:: ini - [options] - package_dir = - =src - packages = find_namespace: + [options] + package_dir = + =src + packages = find_namespace: - [options.packages.find] - where = src + [options.packages.find] + where = src -When you install the zipped distribution, ``timmins.foo`` would become + ``find:`` won't work because timmins doesn't contain ``__init__.py`` + directly, instead, you have to use ``find_namespace:``. + + You can think of ``find_namespace:`` as identical to ``find:`` except it + would count a directory as a package even if it doesn't contain ``__init__.py`` + file directly. + +.. tab:: setup.py + + .. code-block:: python + + setup( + # ... + packages=find_namespace_packages(where='src'), + package_dir={"": "src"} + # ... + ) + + When you use ``find_packages()``, all directories without an + ``__init__.py`` file will be disconsidered. + On the other hand, ``find_namespace_packages()`` will scan all + directories. + +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [tool.setuptools.packages.find] + where = ["src"] + + When using ``tool.setuptools.packages.find`` in ``pyproject.toml``, + setuptools will consider :pep:`implicit namespaces <420>` by default when + scanning your project directory. + +After installing the package distribution, ``timmins.foo`` would become available to your interpreter. -You can think of ``find_namespace:`` as identical to ``find:`` except it -would count a directory as a package even if it doesn't contain ``__init__.py`` -file directly. As a result, this creates an interesting side effect. If you -organize your package like this: +.. warning:: + Please have in mind that ``find_namespace:`` (setup.cfg), + ``find_namespace_packages()`` (setup.py) and ``find`` (pyproject.toml) will + scan **all** folders that you have in your project directory if you use a + :ref:`flat-layout`. -.. code-block:: bash + If used naïvely, this might result in unwanted files being added to your + final wheel. For example, with a project directory organized as follows:: + + foo + ├── docs + │ └── conf.py + ├── timmins + │ └── foo + │ └── __init__.py + └── tests + └── tests_foo + └── __init__.py + + final users will end up installing not only ``timmins.foo``, but also + ``docs`` and ``tests.tests_foo``. + + A simple way to fix this is to adopt the aforementioned :ref:`src-layout`, + or make sure to properly configure the ``include`` and/or ``exclude`` + accordingly. - foo/ - timmins/ - foo/__init__.py - setup.cfg # or setup.py - tests/ - test_foo/__init__.py +.. tip:: + After :ref:`building your package <building>`, you can have a look if all + the files are correct (nothing missing or extra), by running the following + commands: -a naive ``find_namespace:`` would include tests as part of your package to -be installed. A simple way to fix it is to adopt the aforementioned -``src`` layout. + .. code-block:: bash + + tar tf dist/*.tar.gz + unzip -l dist/*.whl + + This requires the ``tar`` and ``unzip`` to be installed in your OS. + On Windows you can also use a GUI program such as 7zip_. Legacy Namespace Packages @@ -228,12 +538,13 @@ And your directory should look like this .. code-block:: bash - /foo/ - src/ - timmins/ - __init__.py - foo/__init__.py - setup.cfg #or setup.py + foo + ├── setup.cfg # and/or setup.py, pyproject.toml + └── src + └── timmins + ├── __init__.py + └── foo + └── __init__.py Repeat the same for other packages and you can achieve the same result as the previous section. @@ -249,3 +560,17 @@ file contains the following: __path__ = __import__('pkgutil').extend_path(__path__, __name__) The project layout remains the same and ``setup.cfg`` remains the same. + + +---- + + +.. [#experimental] + Support for specifying package metadata and build configuration options via + ``pyproject.toml`` is experimental and might change (or be completely + removed) in the future. See :doc:`/userguide/pyproject_config`. +.. [#layout1] https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure +.. [#layout2] https://blog.ionelmc.ro/2017/09/25/rehashing-the-src-layout/ + +.. _editable install: https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs +.. _7zip: https://www.7-zip.org diff --git a/docs/userguide/pyproject_config.rst b/docs/userguide/pyproject_config.rst new file mode 100644 index 00000000..47c4511e --- /dev/null +++ b/docs/userguide/pyproject_config.rst @@ -0,0 +1,218 @@ +.. _pyproject.toml config: + +----------------------------------------------------- +Configuring setuptools using ``pyproject.toml`` files +----------------------------------------------------- + +.. note:: New in 61.0.0 (**experimental**) + +.. warning:: + Support for declaring :doc:`project metadata + <PyPUG:specifications/declaring-project-metadata>` or configuring + ``setuptools`` via ``pyproject.toml`` files is still experimental and might + change (or be removed) in future releases. + +.. important:: + For the time being, ``pip`` still might require a ``setup.py`` file + to support :doc:`editable installs <pip:cli/pip_install>`. + + A simple script will suffice, for example: + + .. code-block:: python + + from setuptools import setup + + setup() + +Starting with :pep:`621`, the Python community selected ``pyproject.toml`` as +a standard way of specifying *project metadata*. +``Setuptools`` has adopted this standard and will use the information contained +in this file as an input in the build process. + +The example below illustrates how to write a ``pyproject.toml`` file that can +be used with ``setuptools``. It contains two TOML tables (identified by the +``[table-header]`` syntax): ``build-system`` and ``project``. +The ``build-system`` table is used to tell the build frontend (e.g. +:pypi:`build` or :pypi:`pip`) to use ``setuptools`` and any other plugins (e.g. +``setuptools-scm``) to build the package. +The ``project`` table contains metadata fields as described by +:doc:`PyPUG:specifications/declaring-project-metadata` guide. + +.. _example-pyproject-config: + +.. code-block:: toml + + [build-system] + requires = ["setuptools", "setuptools-scm"] + build-backend = "setuptools.build_meta" + + [project] + name = "my_package" + description = "My package description" + readme = "README.rst" + keywords = ["one", "two"] + license = {text = "BSD 3-Clause License"} + classifiers = [ + "Framework :: Django", + "Programming Language :: Python :: 3", + ] + dependencies = [ + "requests", + 'importlib-metadata; python_version<"3.8"', + ] + dynamic = ["version"] + + [project.optional-dependencies] + pdf = ["ReportLab>=1.2", "RXP"] + rest = ["docutils>=0.3", "pack ==1.1, ==1.3"] + + [project.scripts] + my-script = "my_package.module:function" + + +.. _setuptools-table: + +Setuptools-specific configuration +================================= + +While the standard ``project`` table in the ``pyproject.toml`` file covers most +of the metadata used during the packaging process, there are still some +``setuptools``-specific configurations that can be set by users that require +customization. +These configurations are completely optional and probably can be skipped when +creating simple packages. +They are equivalent to the :doc:`/references/keywords` used by the ``setup.py`` +file, and can be set via the ``tool.setuptools`` table: + +========================= =========================== ========================= +Key Value Type (TOML) Notes +========================= =========================== ========================= +``platforms`` array +``zip-safe`` boolean If not specified, ``setuptools`` will try to guess + a reasonable default for the package +``eager-resources`` array +``py-modules`` array See tip below +``packages`` array or ``find`` directive See tip below +``package-dir`` table/inline-table Used when explicitly listing ``packages`` +``namespace-packages`` array Not necessary if you use :pep:`420` +``package-data`` table/inline-table See :doc:`/userguide/datafiles` +``include-package-data`` boolean ``True`` by default +``exclude-package-data`` table/inline-table +``license-files`` array of glob patterns **Provisional** - likely to change with :pep:`639` + (by default: ``['LICEN[CS]E*', 'COPYING*', 'NOTICE*', 'AUTHORS*']``) +``data-files`` table/inline-table **Deprecated** - check :doc:`/userguide/datafiles` +``script-files`` array **Deprecated** - equivalent to the ``script`` keyword in ``setup.py`` + (should be avoided in favour of ``project.scripts``) +``provides`` array **Ignored by pip** +``obsoletes`` array **Ignored by pip** +========================= =========================== ========================= + +.. note:: + The `TOML value types`_ ``array`` and ``table/inline-table`` are roughly + equivalent to the Python's :obj:`dict` and :obj:`list` data types. + +Please note that some of these configurations are deprecated or at least +discouraged, but they are made available to ensure portability. +New packages should avoid relying on deprecated/discouraged fields, and +existing packages should consider alternatives. + +.. tip:: + When both ``py-modules`` and ``packages`` are left unspecified, + ``setuptools`` will attempt to perform :ref:`auto-discovery`, which should + cover most popular project directory organization techniques, such as the + :ref:`src-layout` and the :ref:`flat-layout`. + + However if your project does not follow these conventional layouts + (e.g. you want to use a ``flat-layout`` but at the same time have custom + directories at the root of your project), you might need to use the ``find`` + directive [#directives]_ as shown below: + + .. code-block:: toml + + [tool.setuptools.packages.find] + where = ["src"] # list of folders that contain the packages (["."] by default) + include = ["my_package*"] # package names should match these glob patterns (["*"] by default) + exclude = ["my_package.tests*"] # exclude packages matching these glob patterns (empty by default) + namespaces = false # to disable scanning PEP 420 namespaces (true by default) + + Note that the glob patterns in the example above need to be matched + by the **entire** package name. This means that if you specify ``exclude = ["tests"]``, + modules like ``tests.my_package.test1`` will still be included in the distribution + (to remove them, add a wildcard to the end of the pattern: ``"tests*"``). + + Alternatively, you can explicitly list the packages in modules: + + .. code-block:: toml + + [tool.setuptools] + packages = ["my_package"] + + +.. _dynamic-pyproject-config: + +Dynamic Metadata +================ + +Note that in the first example of this page we use ``dynamic`` to identify +which metadata fields are dynamically computed during the build by either +``setuptools`` itself or the plugins installed via ``build-system.requires`` +(e.g. ``setuptools-scm`` is capable of deriving the current project version +directly from the ``git`` :wiki:`version control` system). + +Currently the following fields can be listed as dynamic: ``version``, +``classifiers``, ``description``, ``entry-points``, ``scripts``, +``gui-scripts`` and ``readme``. +When these fields are expected to be provided by ``setuptools`` a +corresponding entry is required in the ``tool.setuptools.dynamic`` table +[#entry-points]_. For example: + +.. code-block:: toml + + # ... + [project] + name = "my_package" + dynamic = ["version", "readme"] + # ... + [tool.setuptools.dynamic] + version = {attr = "my_package.VERSION"} + readme = {file = ["README.rst", "USAGE.rst"]} + +In the ``dynamic`` table, the ``attr`` directive [#directives]_ will read an +attribute from the given module [#attr]_, while ``file`` will read the contents +of all given files and concatenate them in a single string. + +================= =================== ========================= +Key Directive Notes +================= =================== ========================= +``version`` ``attr``, ``file`` +``readme`` ``file`` +``description`` ``file`` One-line text +``classifiers`` ``file`` Multi-line text with one classifier per line +``entry-points`` ``file`` INI format following :doc:`PyPUG:specifications/entry-points` + (``console_scripts`` and ``gui_scripts`` can be included) +================= =================== ========================= + +---- + +.. rubric:: Notes + +.. [#entry-points] Dynamic ``scripts`` and ``gui-scripts`` are a special case. + When resolving these metadata keys, ``setuptools`` will look for + ``tool.setuptool.dynamic.entry-points``, and use the values of the + ``console_scripts`` and ``gui_scripts`` :doc:`entry-point groups + <PyPUG:specifications/entry-points>`. + +.. [#directives] In the context of this document, *directives* are special TOML + values that are interpreted differently by ``setuptools`` (usually triggering an + associated function). Most of the times they correspond to a special TOML table + (or inline-table) with a single top-level key. + For example, you can have the ``{find = {where = ["src"], exclude=["tests*"]}}`` + directive for ``tool.setuptools.packages``, or ``{attr = "mymodule.attr"}`` + directive for ``tool.setuptools.dynamic.version``. + +.. [#attr] ``attr`` is meant to be used when the module attribute is statically + specified (e.g. as a string, list or tuple). As a rule of thumb, the + attribute should be able to be parsed with :func:`ast.literal_eval`, and + should not be modified or re-assigned. + +.. _TOML value types: https://toml.io/en/v1.0.0 diff --git a/docs/userguide/quickstart.rst b/docs/userguide/quickstart.rst index 6bf353a0..2f778521 100644 --- a/docs/userguide/quickstart.rst +++ b/docs/userguide/quickstart.rst @@ -14,9 +14,9 @@ Python packaging at a glance ============================ The landscape of Python packaging is shifting and ``Setuptools`` has evolved to only provide backend support, no longer being the de-facto packaging tool in -the market. All python package must provide a ``pyproject.toml`` and specify +the market. Every python package must provide a ``pyproject.toml`` and specify the backend (build system) it wants to use. The distribution can then -be generated with whatever tools that provides a ``build sdist``-alike +be generated with whatever tool that provides a ``build sdist``-like functionality. While this may appear cumbersome, given the added pieces, it in fact tremendously enhances the portability of your package. The change is driven under :pep:`PEP 517 <517#build-requirements>`. To learn more about Python packaging in general, @@ -32,12 +32,17 @@ package your project: .. code-block:: toml [build-system] - requires = ["setuptools", "wheel"] + requires = ["setuptools"] build-backend = "setuptools.build_meta" -Then, you will need a ``setup.cfg`` or ``setup.py`` to specify your package -information, such as metadata, contents, dependencies, etc. Here we demonstrate -the minimum +Then, you will need to specify your package information such as metadata, +contents, dependencies, etc. + +Setuptools currently supports configurations from either ``setup.cfg``, +``setup.py`` or ``pyproject.toml`` [#experimental]_ files, however, configuring new +projects via ``setup.py`` is discouraged [#setup.py]_. + +The following example demonstrates a minimum configuration: .. tab:: setup.cfg @@ -51,9 +56,11 @@ the minimum packages = mypackage install_requires = requests - importlib; python_version == "2.6" + importlib-metadata; python_version < "3.8" + + See :doc:`/userguide/declarative_config` for more information. -.. tab:: setup.py +.. tab:: setup.py [#setup.py]_ .. code-block:: python @@ -65,10 +72,26 @@ the minimum packages=['mypackage'], install_requires=[ 'requests', - 'importlib; python_version == "2.6"', + 'importlib-metadata; python_version == "3.8"', ], ) + See :doc:`/references/keywords` for more information. + +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [project] + name = "mypackage" + version = "0.0.1" + dependencies = [ + "requests", + 'importlib-metadata; python_version<"3.8"', + ] + + See :doc:`/userguide/pyproject_config` for more information. + This is what your project would look like:: ~/mypackage/ @@ -76,7 +99,7 @@ This is what your project would look like:: setup.cfg # or setup.py mypackage/__init__.py -Then, you need an builder, such as :std:doc:`PyPA build <pypa-build:index>` +Then, you need a builder, such as :std:doc:`PyPA build <pypa-build:index>` which you can obtain via ``pip install build``. After downloading it, invoke the builder:: @@ -89,80 +112,174 @@ Of course, before you release your project to PyPI, you'll want to add a bit more information to your setup script to help people find or learn about your project. And maybe your project will have grown by then to include a few dependencies, and perhaps some data files and scripts. In the next few sections, -we will walk through those additional but essential information you need +we will walk through the additional but essential information you need to specify to properly package your project. Automatic package discovery =========================== For simple projects, it's usually easy enough to manually add packages to -the ``packages`` keyword in ``setup.cfg``. However, for very large projects -, it can be a big burden to keep the package list updated. ``setuptools`` -therefore provides two convenient tools to ease the burden: :literal:`find:\ ` and -:literal:`find_namespace:\ `. To use it in your project: +the ``packages`` keyword in ``setup.cfg``. However, for very large projects, +it can be a big burden to keep the package list updated. +Therefore, ``setuptools`` provides a convenient way to automatically list all +the packages in your project directory: + +.. tab:: setup.cfg + + .. code-block:: ini + + [options] + packages = find: # OR `find_namespaces:` if you want to use namespaces + + [options.packages.find] (always `find` even if `find_namespaces:` was used before) + # This section is optional + # Each entry in this section is optional, and if not specified, the default values are: + # `where=.`, `include=*` and `exclude=` (empty). + include=mypackage* + exclude=mypackage.tests* + +.. tab:: setup.py [#setup.py]_ + + .. code-block:: python + + from setuptools import find_packages # or find_namespace_packages + + setup( + # ... + packages=find_packages( + where='.', + include=['mypackage*'], # ["*"] by default + exclude=['mypackage.tests'], # empty by default + ), + # ... + ) -.. code-block:: ini +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ - [options] - packages = find: + .. code-block:: toml - [options.packages.find] #optional - include=pkg1, pkg2 - exclude=pk3, pk4 + # ... + [tool.setuptools.packages] + find = {} # Scan the project directory with the default parameters -When you pass the above information, alongside other necessary ones, + # OR + [tool.setuptools.packages.find] + where = ["src"] # ["."] by default + include = ["mypackage*"] # ["*"] by default + exclude = ["mypackage.tests*"] # empty by default + namespaces = false # true by default + +When you pass the above information, alongside other necessary information, ``setuptools`` walks through the directory specified in ``where`` (omitted -here as the package reside in current directory) and filters the packages -it can find following the ``include`` (default to none), then remove -those that match the ``exclude`` and return a list of Python packages. Note -that each entry in the ``[options.packages.find]`` is optional. The above +here as the package resides in the current directory) and filters the packages +it can find following the ``include`` (defaults to none), then removes +those that match the ``exclude`` and returns a list of Python packages. The above setup also allows you to adopt a ``src/`` layout. For more details and advanced -use, go to :ref:`package_discovery` +use, go to :ref:`package_discovery`. + +.. tip:: + Starting with version 61.0.0, setuptools' automatic discovery capabilities + have been improved to detect popular project layouts (such as the + :ref:`flat-layout` and :ref:`src-layout`) without requiring any + special configuration. Check out our :ref:`reference docs <package_discovery>` + for more information, but please keep in mind that this functionality is + still considered **experimental** and might change (or even be removed) in + future releases. Entry points and automatic script creation =========================================== -Setuptools support automatic creation of scripts upon installation, that runs -code within your package if you specify them with the ``entry_points`` keyword. +Setuptools supports automatic creation of scripts upon installation, that runs +code within your package if you specify them as :doc:`entry points +<PyPUG:specifications/entry-points>`. This is what allows you to run commands like ``pip install`` instead of having -to type ``python -m pip install``. To accomplish this, add the entry_points -keyword in your ``setup.cfg``: +to type ``python -m pip install``. +The following configuration examples show how to accomplish this: -.. code-block:: ini +.. tab:: setup.cfg + + .. code-block:: ini - [options.entry_points] - console_scripts = - main = mypkg:some_func + [options.entry_points] + console_scripts = + cli-name = mypkg.mymodule:some_func -When this project is installed, a ``main`` script will be installed and will -invoke the ``some_func`` in the ``__init__.py`` file when called by the user. -For detailed usage, including managing the additional or optional dependencies, -go to :doc:`entry_point`. +.. tab:: setup.py [#setup.py]_ + + .. code-block:: python + + setup( + # ... + entry_points={ + 'console_scripts': [ + 'cli-name = mypkg.mymodule:some_func', + ] + } + ) + +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [project.scripts] + cli-name = "mypkg.mymodule:some_func" + +When this project is installed, a ``cli-name`` executable will be created. +``cli-name`` will invoke the function ``some_func`` in the +``mypkg/mymodule.py`` file when called by the user. +Note that you can also use the ``entry-points`` mechanism to advertise +components between installed packages and implement plugin systems. +For detailed usage, go to :doc:`entry_point`. Dependency management ===================== -``setuptools`` supports automatically installing dependencies when a package is -installed. The simplest way to include requirement specifiers is to use the -``install_requires`` argument to ``setup.cfg``. It takes a string or list of -strings containing requirement specifiers (A version specifier is one of the -operators <, >, <=, >=, == or !=, followed by a version identifier): +Packages built with ``setuptools`` can specify dependencies to be automatically +installed when the package itself is installed. +The example below show how to configure this kind of dependencies: + +.. tab:: setup.cfg + + .. code-block:: ini + + [options] + install_requires = + docutils + requests <= 0.4 + +.. tab:: setup.py [#setup.py]_ + + .. code-block:: python -.. code-block:: ini + setup( + # ... + install_requires=["docutils", "requests <= 0.4"], + # ... + ) - [options] - install_requires = - docutils >= 0.3 - requests <= 0.4 +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml + + [project] + # ... + dependencies = [ + "docutils", + "requires <= 0.4", + ] + # ... + +Each dependency is represented by a string that can optionally contain version requirements +(e.g. one of the operators <, >, <=, >=, == or !=, followed by a version identifier), +and/or conditional environment markers, e.g. ``sys_platform == "win32"`` +(see :doc:`PyPUG:specifications/version-specifiers` for more information). When your project is installed, all of the dependencies not already installed will be located (via PyPI), downloaded, built (if necessary), and installed. -This, of course, is a simplified scenarios. ``setuptools`` also provide -additional keywords such as ``setup_requires`` that allows you to install -dependencies before running the script, and ``extras_requires`` that take -care of those needed by automatically generated scripts. It also provides -mechanisms to handle dependencies that are not in PyPI. For more advanced use, -see :doc:`dependency_management` +This, of course, is a simplified scenario. You can also specify groups of +extra dependencies that are not strictly required by your package to work, but +that will provide additional functionalities. +For more advanced use, see :doc:`dependency_management`. .. _Including Data Files: @@ -174,25 +291,43 @@ are placed in a platform-specific location. Setuptools offers three ways to specify data files to be included in your packages. For the simplest use, you can simply use the ``include_package_data`` keyword: -.. code-block:: ini +.. tab:: setup.cfg + + .. code-block:: ini + + [options] + include_package_data = True + +.. tab:: setup.py [#setup.py]_ + + .. code-block:: python + + setup( + # ... + include_package_data=True, + # ... + ) + +.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_ + + .. code-block:: toml - [options] - include_package_data = True + [tool.setuptools] + include-package-data = true + # This is already the default behaviour if your are using + # pyproject.toml to configure your build. + # You can deactivate that with `include-package-data = false` This tells setuptools to install any data files it finds in your packages. -The data files must be specified via the distutils' ``MANIFEST.in`` file. -For more details, see :doc:`datafiles` +The data files must be specified via the distutils' |MANIFEST.in|_ file +or automatically added by a :ref:`Revision Control System plugin +<Adding Support for Revision Control Systems>`. +For more details, see :doc:`datafiles`. Development mode ================ -.. tip:: - - Prior to :ref:`pip v21.1 <pip:v21-1>`, a ``setup.py`` script was - required to be compatible with development mode. With late - versions of pip, any project may be installed in this mode. - ``setuptools`` allows you to install a package without copying any files to your interpreter directory (e.g. the ``site-packages`` directory). This allows you to modify your source code and have the changes take @@ -204,13 +339,35 @@ Here's how to do it:: This creates a link file in your interpreter site package directory which associate with your source code. For more information, see :doc:`development_mode`. +.. tip:: + + Prior to :ref:`pip v21.1 <pip:v21-1>`, a ``setup.py`` script was + required to be compatible with development mode. With late + versions of pip, ``setup.cfg``-only projects may be installed in this mode. + + If you are experimenting with :doc:`configuration using <pyproject_config>`, + or have version of ``pip`` older than v21.1, you might need to keep a + ``setup.py`` file in file in your repository if you want to use editable + installs (for the time being). + + A simple script will suffice, for example: + + .. code-block:: python + + from setuptools import setup + + setup() + + You can still keep all the configuration in :doc:`setup.cfg </userguide/declarative_config>` + (or :doc:`pyproject.toml </userguide/pyproject_config>`). + Uploading your package to PyPI ============================== -After generating the distribution files, next step would be to upload your +After generating the distribution files, the next step would be to upload your distribution so others can use it. This functionality is provided by -`twine <https://pypi.org/project/twine/>`_ and we will only demonstrate the -basic use here. +:pypi:`twine` and is documented in the :doc:`Python packaging tutorial +<PyPUG:tutorials/packaging-projects>`. Transitioning from ``setup.py`` to ``setup.cfg`` @@ -218,12 +375,40 @@ Transitioning from ``setup.py`` to ``setup.cfg`` To avoid executing arbitrary scripts and boilerplate code, we are transitioning into a full-fledged ``setup.cfg`` to declare your package information instead of running ``setup()``. This inevitably brings challenges due to a different -syntax. Here we provide a quick guide to understanding how ``setup.cfg`` is -parsed by ``setuptool`` to ease the pain of transition. +syntax. :doc:`Here </userguide/declarative_config>` we provide a quick guide to +understanding how ``setup.cfg`` is parsed by ``setuptools`` to ease the pain of +transition. .. _packaging-resources: Resources on Python packaging ============================= -Packaging in Python is hard. Here we provide a list of links for those that -want to learn more. +Packaging in Python can be hard and is constantly evolving. +`Python Packaging User Guide <https://packaging.python.org>`_ has tutorials and +up-to-date references that can help you when it is time to distribute your work. + + +.. |MANIFEST.in| replace:: ``MANIFEST.in`` +.. _MANIFEST.in: https://packaging.python.org/en/latest/guides/using-manifest-in/ + + +---- + +.. rubric:: Notes + +.. [#setup.py] + The ``setup.py`` file should be used only when custom scripting during the + build is necessary. + Examples are kept in this document to help people interested in maintaining or + contributing to existing packages that use ``setup.py``. + Note that you can still keep most of configuration declarative in + :doc:`setup.cfg <declarative_config>` or :doc:`pyproject.toml + <pyproject_config>` and use ``setup.py`` only for the parts not + supported in those files (e.g. C extensions). + +.. [#experimental] + While the ``[build-system]`` table should always be specified in the + ``pyproject.toml`` file, support for adding package metadata and build configuration + options via the ``[project]`` and ``[tool.setuptools]`` tables is still + experimental and might change (or be completely removed) in future releases. + See :doc:`/userguide/pyproject_config`. |