Merge pull request #17675 from charris/update-release-related

MAINT: Update release documentation and software

4 files changed, 120 insertions, 97 deletions

diff --git a/doc/RELEASE_WALKTHROUGH.rst.txt b/doc/RELEASE_WALKTHROUGH.rst.txt
index 0ff9ff933..733f681af 100644
--- a/doc/RELEASE_WALKTHROUGH.rst.txt
+++ b/doc/RELEASE_WALKTHROUGH.rst.txt
@@ -1,41 +1,37 @@
-This file contains a walkthrough of the NumPy 1.14.5 release on Linux, modified
+This file contains a walkthrough of the NumPy 1.19.0 release on Linux, modified
 for building on azure and uploading to anaconda.org
 The commands can be copied into the command line, but be sure to
-replace 1.14.5 by the correct version.
+replace 1.19.0 by the correct version.
 
 This should be read together with the general directions in `releasing`.
 
 
-Release  Walkthrough
-====================
-
-Note that in the code snippets below, ``upstream`` refers to the root repository on
-github and ``origin`` to a fork in your personal account. You may need to make adjustments
-if you have not forked the repository but simply cloned it locally. You can
-also edit ``.git/config`` and add ``upstream`` if it isn't already present.
-
+Release Preparation
+===================
 
 Backport Pull Requests
 ----------------------
 
 Changes that have been marked for this release must be backported to the
-maintenance/1.14.x branch.
+maintenance/1.19.x branch.
 
 
 Update Release documentation
 ----------------------------
 
-The file ``doc/changelog/1.14.5-changelog.rst`` should be updated to reflect
+The file ``doc/changelog/1.19.0-changelog.rst`` should be updated to reflect
 the final list of changes and contributors. This text can be generated by::
 
-    $ python tools/changelog.py $GITHUB v1.14.4..maintenance/1.14.x > doc/changelog/1.14.5-changelog.rst
+    $ python tools/changelog.py $GITHUB v1.18.0..maintenance/1.19.x > doc/changelog/1.19.0-changelog.rst
 
 where ``GITHUB`` contains your github access token. This text may also be
-appended to ``doc/release/1.14.5-notes.rst`` for release updates, though not
-for new releases like ``1.14.0``, as the changelogs for ``*.0`` releases tend to be
-excessively long. The ``doc/source/release.rst`` file should also be
-updated with a link to the new release notes. These changes should be committed
-to the maintenance branch, and later will be forward ported to master.
+appended to ``doc/release/1.19.0-notes.rst`` for patch release, though not for
+new releases like ``1.19.0``, as the changelogs for ``*.0`` releases tend to be
+excessively long. The ``doc/source/release.rst`` file should also be updated
+with a link to the new release notes. These changes should be committed to the
+maintenance branch, and later will be forward ported to master. The changelog
+should be reviewed for name duplicates or short names and the ``.mailmap`` file
+updated if needed.
 
 
 Finish the Release Note
@@ -46,25 +42,33 @@ Finish the Release Note
   This has changed now that we use ``towncrier``. See the instructions for
   creating the release note in ``doc/release/upcoming_changes/README.rst``.
 
-Fill out the release note ``doc/release/1.14.5-notes.rst`` calling out
+Fill out the release note ``doc/release/1.19.0-notes.rst`` calling out
 significant changes.
 
 
+Release  Walkthrough
+====================
+
+Note that in the code snippets below, ``upstream`` refers to the root repository on
+github and ``origin`` to a fork in your personal account. You may need to make adjustments
+if you have not forked the repository but simply cloned it locally. You can
+also edit ``.git/config`` and add ``upstream`` if it isn't already present.
+
 Prepare the release commit
 --------------------------
 
 Checkout the branch for the release, make sure it is up to date, and clean the
 repository::
 
-    $ git checkout maintenance/1.14.x
-    $ git pull upstream maintenance/1.14.x
+    $ git checkout maintenance/1.19.x
+    $ git pull upstream maintenance/1.19.x
     $ git submodule update
     $ git clean -xdfq
 
 Edit pavement.py and setup.py as detailed in HOWTO_RELEASE::
 
-    $ gvim pavement.py setup.py
-    $ git commit -a -m"REL: NumPy 1.14.5 release."
+    $ gvim pavement.py setup.py  # Generally only setup.py needs updating
+    $ git commit -a -m"REL: NumPy 1.19.0 release."
 
 Sanity check::
 
@@ -73,9 +77,7 @@ Sanity check::
 Push this release directly onto the end of the maintenance branch. This
 requires write permission to the numpy repository::
 
-    $ git push upstream maintenance/1.14.x
-
-As an example, see the 1.14.3 REL commit: `<https://github.com/numpy/numpy/commit/73299826729be58cec179b52c656adfcaefada93>`_.
+    $ git push upstream HEAD
 
 
 Build source releases
@@ -86,7 +88,7 @@ Paver is used to build the source releases. It will create the ``release`` and
 source releases in the latter. ::
 
     $ python3 -m cython --version  # check for correct cython version
-    $ paver sdist  # sdist will do a git clean -xdf, so we omit that
+    $ paver sdist  # sdist will do a git clean -xdfq, so we omit that
 
 
 Build wheels
@@ -94,55 +96,46 @@ Build wheels
 
 Trigger the wheels build by pointing the numpy-wheels repository at this
 commit. This can take up to an hour. The numpy-wheels repository is cloned from
-`<https://github.com/MacPython/numpy-wheels>`_. Start with a pull as the repo
-may have been accessed and changed by someone else and a push will fail::
+`<https://github.com/MacPython/numpy-wheels>`_. If this is the first release in
+a series, start with a pull as the repo may have been accessed and changed by
+someone else, then create a new branch for the series. If the branch already
+exists skip this::
 
     $ cd ../numpy-wheels
+    $ git co master
     $ git pull upstream master
-    $ git branch <new version>  # only when starting new numpy version
-    $ git checkout v1.14.x  # v1.14.x already existed for the 1.14.4 release
+    $ git branch v1.19.x
 
-Edit the ``azure/posix.yml`` and ``azure/windows.yml`` files to make sure they
-have the correct version, and put in the commit hash for the ``REL`` commit
-created above for ``BUILD_COMMIT``, see an _example::
+Checkout the new branch and edit the ``azure-pipelines.yml`` and
+``.travis.yml`` files to make sure they have the correct version, and put in
+the commit hash for the ``REL`` commit created above for ``BUILD_COMMIT``. The
+``azure/posix.yml`` and ``.travis.yml`` files may also need the Cython versions
+updated to keep up with Python releases, but generally just do::
 
-    $ gvim azure/posix.yml azure/windows.yml
-    $ git commit -a
+    $ git checkout v1.19.x
+    $ gvim azure-pipelines .travis.yml
+    $ git commit -a -m"NumPy 1.19.0 release."
     $ git push upstream HEAD
 
 Now wait. If you get nervous at the amount of time taken -- the builds can take
 a while -- you can check the build progress by following the links
-provided at `<https://github.com/MacPython/numpy-wheels>`_ to check the 
+provided at `<https://github.com/MacPython/numpy-wheels>`_ to check the
 build status. Check if all the needed wheels have been built and
-uploaded before proceeding. There should currently be 21 of them at
-`<https://anaconda.org/multibuild-wheels-staging/numpy/files>`_, 3 for Mac, 6
-for Windows, and 12 for Linux.
+uploaded to the staging repository before proceeding.
 
-.. example_: https://github.com/MacPython/numpy-wheels/pull/80/commits/cbf4af4
-
-Note that sometimes builds, like tests, fail for unrelated reasons and
-you will need to restart them.
+Note that sometimes builds, like tests, fail for unrelated reasons and you will
+need to rerun them. You will need to be logged in under 'numpy' to do this
+on azure.
 
 Download wheels
 ---------------
 
-When the wheels have all been successfully built, download them using the ``wheel-uploader``
-in the ``terryfy`` repository.  The terryfy repository may be cloned from
-`<https://github.com/MacPython/terryfy>`_ if you don't already have it.  The
-wheels can also be uploaded using the ``wheel-uploader``, but we prefer to
-download all the wheels to the ``../numpy/release/installers`` directory and
-upload later using ``twine``::
+When the wheels have all been successfully built and staged, download them from the
+Anaconda staging directory using the ``tools/download-wheels.py`` script::
 
-    $ cd ../terryfy
-    $ git pull upstream master
-    $ CDN_URL=https://anaconda.org/multibuild-wheels-staging/numpy/files
-    $ NPY_WHLS=../numpy/release/installers
-    $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t win numpy 1.14.5
-    $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t manylinux1 numpy 1.14.5
-    $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t macosx numpy 1.14.5
+    $ cd ../numpy
+    $ python3 tools/download-wheels.py 1.19.0
 
-If you do this often, consider making CDN_URL and NPY_WHLS part of your default
-environment.
 
 Generate the README files
 -------------------------
@@ -150,18 +143,16 @@ Generate the README files
 This needs to be done after all installers are downloaded, but before the pavement
 file is updated for continued development::
 
-    $ cd ../numpy
     $ paver write_release
 
 
 Tag the release
 ---------------
 
-Once the wheels have been built and downloaded without errors, go back to your
-numpy repository in the maintenance branch and tag the ``REL`` commit, signing
+Once the wheels have been built and downloaded without errors tag the ``REL`` commit, signing
 it with your gpg key::
 
-    $ git tag -s v1.14.5
+    $ git tag -s -m"NumPy 1.19.0 release" v1.19.0
 
 You should upload your public gpg key to github, so that the tag will appear
 "verified" there.
@@ -169,7 +160,7 @@ You should upload your public gpg key to github, so that the tag will appear
 Check that the files in ``release/installers`` have the correct versions, then
 push the tag upstream::
 
-    $ git push upstream v1.14.5
+    $ git push upstream v1.19.0
 
 We wait until this point to push the tag because it is public and should not
 be changed after it has been pushed.
@@ -185,9 +176,9 @@ Add another ``REL`` commit to the numpy maintenance branch, which resets the
 
 Create release notes for next release and edit them to set the version::
 
-    $ cp doc/source/release/template.rst doc/source/release/1.14.6-notes.rst
-    $ gvim doc/source/release/1.14.6-notes.rst
-    $ git add doc/source/release/1.14.6-notes.rst
+    $ cp doc/source/release/template.rst doc/source/release/1.19.1-notes.rst
+    $ gvim doc/source/release/1.19.1-notes.rst
+    $ git add doc/source/release/1.19.1-notes.rst
 
 Add new release notes to the documentation release list::
 
@@ -195,26 +186,24 @@ Add new release notes to the documentation release list::
 
 Commit the result::
 
-    $ git commit -a -m"REL: prepare 1.14.x for further development"
-    $ git push upstream maintenance/1.14.x
+    $ git commit -a -m"REL: prepare 1.19.x for further development"
+    $ git push upstream HEAD
 
 
 Upload to PyPI
 --------------
 
 Upload to PyPI using ``twine``. A recent version of ``twine`` of is needed
-after recent PyPI changes, version ``1.11.0`` was used here.
-
-.. code-block:: sh
+after recent PyPI changes, version ``3.1.1`` was used here::
 
     $ cd ../numpy
     $ twine upload release/installers/*.whl
-    $ twine upload release/installers/numpy-1.14.5.zip  # Upload last.
+    $ twine upload release/installers/numpy-1.19.0.zip  # Upload last.
 
-If one of the commands breaks in the middle, which is not uncommon, you may
-need to selectively upload the remaining files because PyPI does not allow the
-same file to be uploaded twice. The source file should be uploaded last to
-avoid synchronization problems if pip users access the files while this is in
+If one of the commands breaks in the middle, you may need to selectively upload
+the remaining files because PyPI does not allow the same file to be uploaded
+twice. The source file should be uploaded last to avoid synchronization
+problems that might occur if pip users access the files while this is in
 process. Note that PyPI only allows a single source distribution, here we have
 chosen the zip archive.
 
@@ -222,15 +211,16 @@ chosen the zip archive.
 Upload files to github
 ----------------------
 
-Go to `<https://github.com/numpy/numpy/releases>`_, there should be a ``v1.14.5
+Go to `<https://github.com/numpy/numpy/releases>`_, there should be a ``v1.19.0
 tag``, click on it and hit the edit button for that tag. There are two ways to
-add files, using an editable text window and as binary uploads.
+add files, using an editable text window and as binary uploads. Cut and paste
+the ``release/README.md`` file contents into the text window. You will probably
+need to make some edits to get it to look right. Then
 
-- Cut and paste the ``release/README.md`` file contents into the text window.
-- Upload ``release/installers/numpy-1.14.5.tar.gz`` as a binary file.
-- Upload ``release/installers/numpy-1.14.5.zip`` as a binary file.
+- Upload ``release/installers/numpy-1.19.0.tar.gz`` as a binary file.
+- Upload ``release/installers/numpy-1.19.0.zip`` as a binary file.
 - Upload ``release/README.rst`` as a binary file.
-- Upload ``doc/changelog/1.14.5-changelog.rst`` as a binary file.
+- Upload ``doc/changelog/1.19.0-changelog.rst`` as a binary file.
 - Check the pre-release button if this is a pre-releases.
 - Hit the ``{Publish,Update} release`` button at the bottom.
 
@@ -255,7 +245,7 @@ If the release series is a new one, you will need to add a new section to the
 Otherwise, only the ``zip`` and ``pdf`` links should be updated with the
 new tag name::
 
-    $ gvim doc/build/merge/index.html +/'tag v1.14'
+    $ gvim doc/build/merge/index.html +/'tag v1.19'
 
 You can "test run" the new documentation in a browser to make sure the links
 work::
@@ -265,7 +255,7 @@ work::
 Once everything seems satisfactory, commit and upload the changes::
 
     $ pushd doc/build/merge
-    $ git commit -am"Add documentation for v1.14.5"
+    $ git commit -am"Add documentation for v1.19.0"
     $ git push
     $ popd
 
@@ -277,7 +267,7 @@ This assumes that you have forked `<https://github.com/scipy/scipy.org>`_::
     $ cd ../scipy.org
     $ git checkout master
     $ git pull upstream master
-    $ git checkout -b numpy-1.14.5
+    $ git checkout -b numpy-1.19.0
     $ gvim www/index.rst # edit the News section
     $ git commit -a
     $ git push origin HEAD
@@ -300,13 +290,14 @@ Post-Release Tasks
 
 Checkout master and forward port the documentation changes::
 
-    $ git checkout -b update-after-1.14.5-release
-    $ git checkout maintenance/1.14.x doc/source/release/1.14.5-notes.rst
-    $ git checkout maintenance/1.14.x doc/changelog/1.14.5-changelog.rst
+    $ git checkout -b post-1.19.0-release-update
+    $ git checkout maintenance/1.19.x doc/source/release/1.19.0-notes.rst
+    $ git checkout maintenance/1.19.x doc/changelog/1.19.0-changelog.rst
+    $ git checkout maintenance/1.19.x .mailmap  # only if updated for release.
     $ gvim doc/source/release.rst  # Add link to new notes
-    $ git add doc/changelog/1.14.5-changelog.rst doc/source/release/1.14.5-notes.rst
+    $ git add doc/changelog/1.19.0-changelog.rst doc/source/release/1.19.0-notes.rst
     $ git status  # check status before commit
-    $ git commit -a -m"REL: Update master after 1.14.5 release."
+    $ git commit -a -m"REL: Update master after 1.19.0 release."
     $ git push origin HEAD
 
 Go to github and make a PR.
diff --git a/release_requirements.txt b/release_requirements.txt
new file mode 100644
index 000000000..dc840b935
--- /dev/null
+++ b/release_requirements.txt
@@ -0,0 +1,10 @@
+# These packages are needed for a release in addition to those needed
+# for building, testing, and the creation of documentation.
+
+# download-wheels.py
+urllib3
+beatifulsoup4
+
+# changelog.py
+pygithub
+gitpython
diff --git a/tools/changelog.py b/tools/changelog.py
index 92f33af24..920f5b87f 100755
--- a/tools/changelog.py
+++ b/tools/changelog.py
@@ -45,16 +45,17 @@ if sys.version_info[:2] < (3, 6):
 this_repo = Repo(os.path.join(os.path.dirname(__file__), ".."))
 
 author_msg =\
-u"""
+"""
 A total of %d people contributed to this release.  People with a "+" by their
 names contributed a patch for the first time.
 """
 
 pull_request_msg =\
-u"""
+"""
 A total of %d pull requests were merged for this release.
 """
 
+
 def get_authors(revision_range):
     pat = u'^.*\\t(.*)$'
     lst_release, cur_release = [r.strip() for r in revision_range.split('..')]
diff --git a/tools/download-wheels.py b/tools/download-wheels.py
index 941440ca9..28b3fc7ad 100644
--- a/tools/download-wheels.py
+++ b/tools/download-wheels.py
@@ -1,9 +1,29 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
+# -*- encoding:utf-8 -*-
 """
-Download NumPy wheels from Anaconda staging area.
+Script to download NumPy wheels from the Anaconda staging area.
+
+Usage::
+
+    $ ./tools/download-wheels.py <version> -w <optional-wheelhouse>
+
+The default wheelhouse is ``release/installers``.
+
+Dependencies
+------------
+
+- beautifulsoup4
+- urllib3
+
+Examples
+--------
+
+While in the repository root::
+
+    $ python tools/download-wheels.py 1.19.0
+    $ python tools/download-wheels.py 1.19.0 -w ~/wheelhouse
 
 """
-import sys
 import os
 import re
 import shutil
@@ -18,6 +38,7 @@ __version__ = '0.1'
 STAGING_URL = 'https://anaconda.org/multibuild-wheels-staging/numpy'
 PREFIX = 'numpy'
 
+
 def get_wheel_names(version):
     """ Get wheel names from Anaconda HTML directory.
 
@@ -69,7 +90,7 @@ if __name__ == '__main__':
     parser = argparse.ArgumentParser()
     parser.add_argument(
         "version",
-         help="NumPy version to download.")
+        help="NumPy version to download.")
     parser.add_argument(
         "-w", "--wheelhouse",
         default=os.path.join(os.getcwd(), "release", "installers"),