Merge branch 'release/4.6.5'4.6.5

9 files changed, 282 insertions, 64 deletions

diff --git a/docs/configuration-directives/WSGIChunkedRequest.rst b/docs/configuration-directives/WSGIChunkedRequest.rst
new file mode 100644
index 0000000..059103a
--- /dev/null
+++ b/docs/configuration-directives/WSGIChunkedRequest.rst
@@ -0,0 +1,48 @@
+==================
+WSGIChunkedRequest
+==================
+
+:Description: Enabled support for chunked request content.
+:Syntax: ``WSGIChunkedRequest On|Off``
+:Default: ``WSGIChunkedRequest Off``
+:Context: server config, virtual host, directory, .htaccess
+
+The WSGIChunkedRequest directive can be used to enable support for chunked
+request content. Rather than Apache rejecting a request using chunked
+request content, it will be allowed to pass through.
+
+Do note however that WSGI is technically incapable of supporting chunked
+request content without all chunked request content having to be first read
+in and buffered. This is because WSGI requires ``CONTENT_LENGTH`` be set
+when there is any request content.
+
+In mod_wsgi no buffering is done. Thus, to be able to read the request
+content in the case of a chunked transfer encoding, you need to step
+outside of the WSGI specification and do things it says you aren't meant to.
+
+You have two choices for how you can do this. The first choice you have
+is to call ``read()`` on ``wsgi.input`` but not supply any argument at all.
+This will cause all request content to be read in and returned.
+
+The second is to loop on calling ``read()`` on ``wsgi.input`` with a set
+block size passed as argument and do this until ``read()`` returns an empty
+string.
+
+Because both calling methods are not allowed under WSGI specification, in
+using these your code will not technically be portable to other WSGI hosting
+mechanisms, although if those other WSGI servers support it, you will be
+okay.
+
+That all said, although technically not permitted by the WSGI specification,
+some WSGI frameworks do now incoporate support for handling chunked request
+content, as well as where compressed request content is expanded by the web
+server such that ``CONTENT_LENGTH`` is no longer accurate. The required
+behaviour is enabled in these frameworks by the WSGI server passing through
+the non standard ``wsgi.input_terminated`` key set as ``True`` in the per
+request WSGI ``environ`` dictionary. When this is done the web frameworks
+will always read all available input and ignore ``CONTENT_LENGTH``.
+
+Because mod_wsgi guarantees that an empty string is returned when all input
+is exhausted, it will will always set this flag.
+
+It is known that Flask/Werkzeug supports the ``wsgi.input_terminated`` flag.
diff --git a/docs/configuration.rst b/docs/configuration.rst
index 5338dfd..528b7ff 100644
--- a/docs/configuration.rst
+++ b/docs/configuration.rst
@@ -12,6 +12,7 @@ Configuration
    configuration-directives/WSGIAuthUserScript
    configuration-directives/WSGICallableObject
    configuration-directives/WSGICaseSensitivity
+   configuration-directives/WSGIChunkedRequest
    configuration-directives/WSGIDaemonProcess
    configuration-directives/WSGIImportScript
    configuration-directives/WSGILazyInitialization
diff --git a/docs/release-notes.rst b/docs/release-notes.rst
index e6d49b1..a519844 100644
--- a/docs/release-notes.rst
+++ b/docs/release-notes.rst
@@ -5,6 +5,7 @@ Release Notes
 .. toctree::
    :maxdepth: 2
 
+   release-notes/version-4.6.5
    release-notes/version-4.6.4
    release-notes/version-4.6.3
    release-notes/version-4.6.2
diff --git a/docs/release-notes/version-4.6.5.rst b/docs/release-notes/version-4.6.5.rst
new file mode 100644
index 0000000..b4fff42
--- /dev/null
+++ b/docs/release-notes/version-4.6.5.rst
@@ -0,0 +1,33 @@
+=============
+Version 4.6.5
+=============
+
+Version 4.6.5 of mod_wsgi can be obtained from:
+
+  https://codeload.github.com/GrahamDumpleton/mod_wsgi/tar.gz/4.6.5
+
+Bugs Fixed
+----------
+
+* When running ``mod_wsgi-express`` and serving up static files from the
+  document root, and the WSGI application was mounted at a sub URL using
+  ``--mount-point``, the static files in the document root outside of the
+  mount point for the WSGI application would no longer be accessible.
+
+* If no system mime types file can be found, fall back to ``/dev/null``
+  so that Apache can still at least start up.
+
+Features Changed
+----------------
+
+* On macOS, use ``/var/tmp`` as default parent directory for server root
+  directory rather than value of ``$TMPDIR``. The latter can produce a
+  path which is too long and UNIX socket cannot be written there.
+
+New Features
+------------
+
+* Now possible to use ``mod_wsgi-express`` in an a ``zipapp`` created using
+  ``shiv``. This entailed a special workaround to detect when ``shiv`` was
+  used, so that the unpacked ``site-packages`` directory could be added to
+  the Python module search path for ``mod_wsgi-express``.
diff --git a/docs/user-guides/access-control-mechanisms.rst b/docs/user-guides/access-control-mechanisms.rst
index e8f7763..f985215 100644
--- a/docs/user-guides/access-control-mechanisms.rst
+++ b/docs/user-guides/access-control-mechanisms.rst
@@ -126,7 +126,7 @@ only one small part of them. This will result in a lot of memory being used
 in the Apache child processes just to support the auth provider.
 
 If mod_authn_alias is being loaded into Apache, then an aliased auth
-%rovider can also be defined::
+provider can also be defined::
 
     <AuthnProviderAlias wsgi django>
     WSGIAuthUserScript /usr/local/django/mysite/apache/auth.wsgi \
@@ -136,8 +136,13 @@ If mod_authn_alias is being loaded into Apache, then an aliased auth
     WSGIScriptAlias / /usr/local/django/mysite/apache/django.wsgi
 
     <Directory /usr/local/django/mysite/apache>
-    Order deny,allow
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
 
     WSGIApplicationGroup django
 
diff --git a/docs/user-guides/configuration-guidelines.rst b/docs/user-guides/configuration-guidelines.rst
index f10f730..4b5829f 100644
--- a/docs/user-guides/configuration-guidelines.rst
+++ b/docs/user-guides/configuration-guidelines.rst
@@ -50,15 +50,20 @@ within that directory can be used. To do this the Directory directive must
 be used::
 
     <Directory /usr/local/wsgi/scripts>
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
-Note that Apache access control directives such as Order and Allow should
-nearly always be applied to Directory and never to a Location. Adding them
-to a Location would not be regarded as best practice and would potentially
-weaken the security of your Apache server, especially where the Location
-was for '/'.
+Note that Apache access control directives such as Order and Allow, or
+Require in the case of Apache 2.4 or newer, should nearly always be applied
+to Directory and never to a Location. Adding them to a Location would not
+be regarded as best practice and would potentially weaken the security of
+your Apache server, especially where the Location was for '/'.
 
 As for CGI scripts and the ScriptAlias directive, it is not necessary to
 have used the Options directive to enable the ExecCGI directive. This is
@@ -117,8 +122,13 @@ specific URLs. The equivalent such configuration for::
     WSGIScriptAlias /wsgi/ /usr/local/wsgi/scripts/
 
     <Directory /usr/local/wsgi/scripts>
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 using the Alias directive would be::
@@ -130,8 +140,13 @@ using the Alias directive would be::
 
     SetHandler wsgi-script
 
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 The additional steps required in this case are to enable the ability to
@@ -151,8 +166,13 @@ resource types based on resource extension::
     AddHandler cgi-script .cgi
     AddHandler wsgi-script .wsgi
 
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 For whatever extension you use to identify a WSGI script file, ensure that
@@ -177,8 +197,13 @@ option and MultiviewsMatch directive::
     AddHandler cgi-script .cgi
     AddHandler wsgi-script .wsgi
 
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 Adding of MultiViews in this instance and allowing multiviews to match
@@ -201,8 +226,13 @@ the directory. To enable directory browsing add the Indexes option::
     AddHandler cgi-script .cgi
     AddHandler wsgi-script .wsgi
 
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 If a directory index page is enabled, it may refer to either a static file,
@@ -219,8 +249,13 @@ designate what should be used for the index page::
     AddHandler cgi-script .cgi
     AddHandler wsgi-script .wsgi
 
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 Using AddHandler or SetHandler to configure a WSGI application can also
@@ -238,8 +273,13 @@ Options directive by listing ExecCGI::
     Options ExecCGI MultiViews Indexes
     MultiviewsMatch Handlers
 
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 This done, the '.htaccess' file could then contain::
@@ -388,15 +428,25 @@ which should be served in this way::
     Alias /media/ /usr/local/wsgi/static/media/
 
     <Directory /usr/local/wsgi/static>
-    Order deny,allow
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
     WSGIScriptAlias / /usr/local/wsgi/scripts/myapp.wsgi
 
     <Directory /usr/local/wsgi/scripts>
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 When listing the directives, list those for more specific URLs first. In
@@ -465,8 +515,13 @@ the WSGIApplicationGroup directive::
     <Directory /usr/local/wsgi/scripts>
     WSGIApplicationGroup admin-scripts
 
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 The argument to the WSGIApplicationGroup directive can in general be any
@@ -560,8 +615,13 @@ specific WSGI applications to execute within that daemon process::
     Alias /media/ /usr/local/wsgi/static/media/
 
     <Directory /usr/local/wsgi/static>
-    Order deny,allow
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
     WSGIScriptAlias / /usr/local/wsgi/scripts/myapp.wsgi
@@ -569,8 +629,13 @@ specific WSGI applications to execute within that daemon process::
 
     <Directory /usr/local/wsgi/scripts>
 
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 Where Apache has been started as the ``root`` user, the daemon processes
diff --git a/docs/user-guides/quick-configuration-guide.rst b/docs/user-guides/quick-configuration-guide.rst
index 09ae64c..605ec7c 100644
--- a/docs/user-guides/quick-configuration-guide.rst
+++ b/docs/user-guides/quick-configuration-guide.rst
@@ -95,8 +95,13 @@ Apache, it will be necessary to tell Apache that files within that
 directory can be used. To do this the Directory directive must be used::
 
     <Directory /usr/local/www/wsgi-scripts>
-    Order allow,deny
-    Allow from all
+    <IfVersion < 2.4>
+	Order allow,deny
+	Allow from all
+    </IfVersion>
+    <IfVersion >= 2.4>
+	Require all granted
+    </IfVersion>
     </Directory>
 
 Note that it is highly recommended that the WSGI application script file in
@@ -126,15 +131,25 @@ therefore be something like::
         DocumentRoot /usr/local/www/documents
 
         <Directory /usr/local/www/documents>
-        Order allow,deny
-        Allow from all
+	<IfVersion < 2.4>
+	    Order allow,deny
+	    Allow from all
+	</IfVersion>
+	<IfVersion >= 2.4>
+	    Require all granted
+	</IfVersion>
         </Directory>
 
         WSGIScriptAlias /myapp /usr/local/www/wsgi-scripts/myapp.wsgi
 
         <Directory /usr/local/www/wsgi-scripts>
-        Order allow,deny
-        Allow from all
+	<IfVersion < 2.4>
+	    Order allow,deny
+	    Allow from all
+	</IfVersion>
+	<IfVersion >= 2.4>
+	    Require all granted
+	</IfVersion>
         </Directory>
 
     </VirtualHost>
@@ -184,15 +199,25 @@ therefore be something like::
         Alias /media/ /usr/local/www/documents/media/
 
         <Directory /usr/local/www/documents>
-        Order allow,deny
-        Allow from all
+	<IfVersion < 2.4>
+	    Order allow,deny
+	    Allow from all
+	</IfVersion>
+	<IfVersion >= 2.4>
+	    Require all granted
+	</IfVersion>
         </Directory>
 
         WSGIScriptAlias / /usr/local/www/wsgi-scripts/myapp.wsgi
 
         <Directory /usr/local/www/wsgi-scripts>
-        Order allow,deny
-        Allow from all
+	<IfVersion < 2.4>
+	    Order allow,deny
+	    Allow from all
+	</IfVersion>
+	<IfVersion >= 2.4>
+	    Require all granted
+	</IfVersion>
         </Directory>
 
     </VirtualHost>
@@ -255,8 +280,13 @@ therefore be something like::
         Alias /media/ /usr/local/www/documents/media/
 
         <Directory /usr/local/www/documents>
-        Order allow,deny
-        Allow from all
+	<IfVersion < 2.4>
+	    Order allow,deny
+	    Allow from all
+	</IfVersion>
+	<IfVersion >= 2.4>
+	    Require all granted
+	</IfVersion>
         </Directory>
 
         WSGIDaemonProcess example.com processes=2 threads=15 display-name=%{GROUP}
@@ -265,8 +295,13 @@ therefore be something like::
         WSGIScriptAlias / /usr/local/www/wsgi-scripts/myapp.wsgi
 
         <Directory /usr/local/www/wsgi-scripts>
-        Order allow,deny
-        Allow from all
+	<IfVersion < 2.4>
+	    Order allow,deny
+	    Allow from all
+	</IfVersion>
+	<IfVersion >= 2.4>
+	    Require all granted
+	</IfVersion>
         </Directory>
 
     </VirtualHost>
diff --git a/src/server/__init__.py b/src/server/__init__.py
index 66a9455..e981a92 100644
--- a/src/server/__init__.py
+++ b/src/server/__init__.py
@@ -15,6 +15,7 @@ import pprint
 import time
 import traceback
 import locale
+import inspect
 
 try:
     import Queue as queue
@@ -105,7 +106,7 @@ def find_mimetypes():
             return name
             break
     else:
-        return name
+        return '/dev/null'
 
 APACHE_GENERAL_CONFIG = """
 <IfModule !version_module>
@@ -716,7 +717,7 @@ AccessFileName .htaccess
 </Files>
 </Directory>
 
-<Directory '%(document_root)s%(mount_point)s'>
+<Directory '%(document_root)s'>
     AllowOverride %(allow_override)s
 <IfDefine MOD_WSGI_DIRECTORY_INDEX>
     DirectoryIndex %(directory_index)s
@@ -732,6 +733,16 @@ AccessFileName .htaccess
 </IfDefine>
     RewriteEngine On
     Include %(rewrite_rules)s
+<IfVersion < 2.4>
+    Order allow,deny
+    Allow from all
+</IfVersion>
+<IfVersion >= 2.4>
+    Require all granted
+</IfVersion>
+</Directory>
+
+<Directory '%(document_root)s%(mount_point)s'>
 <IfDefine !MOD_WSGI_STATIC_ONLY>
     RewriteCond %%{REQUEST_FILENAME} !-f
 <IfDefine MOD_WSGI_DIRECTORY_INDEX>
@@ -742,13 +753,6 @@ AccessFileName .htaccess
 </IfDefine>
     RewriteRule .* - [H=wsgi-handler]
 </IfDefine>
-<IfVersion < 2.4>
-    Order allow,deny
-    Allow from all
-</IfVersion>
-<IfVersion >= 2.4>
-    Require all granted
-</IfVersion>
 </Directory>
 
 <IfDefine MOD_WSGI_ERROR_OVERRIDE>
@@ -2390,8 +2394,9 @@ option_list = (
     optparse.make_option('--server-root', metavar='DIRECTORY-PATH',
             help='Specify an alternate directory for where the generated '
             'web server configuration, startup files and logs will be '
-            'stored. Defaults to the sub directory specified by the '
-            'TMPDIR environment variable, or /tmp if not specified.'),
+            'stored. On Linux defaults to the sub directory specified by '
+            'the TMPDIR environment variable, or /tmp if not specified. '
+            'On macOS, defaults to the /var/tmp directory.'),
 
     optparse.make_option('--server-mpm', action='append',
             dest='server_mpm_variables', metavar='NAME', help='Specify '
@@ -2660,9 +2665,12 @@ def _cmd_setup_server(command, args, options):
             options['port'], os.getuid())
 
     if not options['server_root']:
-        tmpdir = os.environ.get('TMPDIR')
-        tmpdir = tmpdir or '/tmp'
-        tmpdir = tmpdir.rstrip('/')
+        if sys.platform == 'darwin':
+            tmpdir = '/var/tmp'
+        else:
+            tmpdir = os.environ.get('TMPDIR')
+            tmpdir = tmpdir or '/tmp'
+            tmpdir = tmpdir.rstrip('/')
         options['server_root'] = '%s/mod_wsgi-%s:%s:%s' % (tmpdir,
                 options['host'], options['port'], os.getuid())
 
@@ -2868,6 +2876,28 @@ def _cmd_setup_server(command, args, options):
     if options['python_paths'] is None:
         options['python_paths'] = []
 
+    # Special case to check for when being executed from shiv variant
+    # of a zipapp application bundle. We need to work out where the
+    # site packages directory is and pass it with Python module search
+    # path so is known about by the Apache sub process when executed.
+
+    site_packages = []
+
+    if '_bootstrap' in sys.modules:
+        bootstrap = sys.modules['_bootstrap']
+        if 'bootstrap' in dir(bootstrap):
+            frame = inspect.currentframe()
+            while frame is not None:
+                code = frame.f_code
+                if (code and code.co_filename == bootstrap.__file__ and
+                        code.co_name == 'bootstrap' and
+                        'site_packages' in frame.f_locals):
+                    site_packages.append(str(frame.f_locals['site_packages']))
+                    break
+                frame = frame.f_back
+
+    options['python_paths'].extend(site_packages)
+
     options['python_path'] = ':'.join(options['python_paths'])
 
     options['multiprocess'] = options['processes'] is not None
diff --git a/src/server/wsgi_version.h b/src/server/wsgi_version.h
index b9b7f43..033e71f 100755
--- a/src/server/wsgi_version.h
+++ b/src/server/wsgi_version.h
@@ -25,8 +25,8 @@
 
 #define MOD_WSGI_MAJORVERSION_NUMBER 4
 #define MOD_WSGI_MINORVERSION_NUMBER 6
-#define MOD_WSGI_MICROVERSION_NUMBER 4
-#define MOD_WSGI_VERSION_STRING "4.6.4"
+#define MOD_WSGI_MICROVERSION_NUMBER 5
+#define MOD_WSGI_VERSION_STRING "4.6.5"
 
 /* ------------------------------------------------------------------------- */