Use the ensuredir() function consistently across Sphinx code to avoid
race conditions e.g. when multiple Sphinx instances attempt to create
the same output directory. The ensuredir() function that was already
present in sphinx.util.osutil correctly catches EEXIST exception that
occurs if the specified directory already exists (i.e. it was created
between the call to os.path.isdir() and os.makedirs() that follows it).

While at it, remove redundant os.path.isdir() calls when they only
guarded the os.makedirs() call, and replace mkdir_p() which had pretty
much the same purpose, except for being prone to race conditions.

I did not modify testing-related code as race conditions mostly affect
real applications and not the test environment.

Fix #4281: Race conditions when creating output directory

The 'sphinx-apidoc' tool is no longer the only kid on the block when it
comes to automatic documentation of Python projects, with the likes of
'sphinx-autoapi' in development [1], and is not really maintained. Given
the fact that the tool is tangential to Sphinx's main goal, there isn't
really anything that warrants its continue existence as a core module.

Signify this by moving the feature to 'ext'. This allows the feature to
continue to exist in the package, but indicates that stability might be
worse than one would expect from the core library.

To avoid breaking packages that are using this feature directly, such as
pbr [3], aliases for the old 'main' method are included. This is based
on what Django does and, like Django, will allow us to safely remove the
old modules in Sphinx 2.0.

[1] https://github.com/rtfd/sphinx-autoapi
[2] https://github.com/sphinx-doc/sphinx/pull/3485#issuecomment-288081223
[3] https://github.com/django/django/blob/1.11/django/test/runner.py#L688-L695

Signed-off-by: Stephen Finucane <stephen@that.guru>