Sometimes the server pool fills up; all max_size greenthreads are
already in use with other connections. One more connection gets
accept()ed, but then has to wait for a slot to open up to actually be
handled.
This works out fine if your clients tend to make a single request then
close the connection upon receiving a response. It works out OK-ish when
clients are continually pipelining requests; the new connection still
has to wait, but at least there's plenty of work getting processed --
it's defensible. It can work out pretty terribly if clients tend to hold
on to connections "just in case" -- we're ignoring fresh work from a new
client just so we can be ready-to-go if an existing connection wakes up.
There are a couple existing tunings we can use, but they can each have
downsides:
- Increasing max_size is nice for dealing with idle connections, but can
cause hub contention and high latency variance when all those
connections are actually busy.
- socket_timeout can be used to limit the idle socket time, but it
*also* impacts send/recv operations while processing a request, which
may not be desirable.
- keepalive can be set to False, disabling request pipelining entirely.
Change the keepalive option to wsgi.server so it can be the timeout to
use while waiting for a new request, separate from socket_timeout. By
default, socket_timeout continues to be used.