Allow for Thread reuse for CompactIndex fetcher

One of the main benefits of the Net::HTTP persistent library is that it
maintains the HTTP connections so that multiple requests to the same
host don't need to re-instantiate the http tcp socket to make a request,
and also allows this to be done across threads.

But when used with the CompactIndex client, the each iteration of gem
names throws away the `Bundle::Worker` and initiates new Threads with
new connections.

Since the resulting work on these threads only returns to the caller the
result, and can be collected cleanly, it is safe to reuse these threads
by other repeated `gem_name` iterations and gain a bit of a performance
boost in the process.

While in this code change, we do run `@bundle_worker.stop`, we could
simple skip this now as we will never use more then one
`Bundle::Worker`'s worth of threads, keeping around the worker for
further calles to `.specs` on the same fetcher.

1 files changed, 15 insertions, 3 deletions

diff --git a/lib/bundler/fetcher/compact_index.rb b/lib/bundler/fetcher/compact_index.rb
index e6f936c2fe..be5683dbbd 100644
--- a/lib/bundler/fetcher/compact_index.rb
+++ b/lib/bundler/fetcher/compact_index.rb
@@ -44,6 +44,8 @@ module Bundler
           complete_gems.concat(deps.map(&:first)).uniq!
           remaining_gems = next_gems - complete_gems
         end
+        @bundle_worker.stop if @bundle_worker
+        @bundle_worker = nil # reset it.  Not sure if necessary
 
         gem_info
       end
@@ -86,15 +88,25 @@ module Bundler
           end.tap do |client|
             client.in_parallel = lambda do |inputs, &blk|
               func = lambda {|object, _index| blk.call(object) }
-              worker_name = "Compact Index (#{display_uri.host})"
-              worker = Bundler::Worker.new(25, worker_name, func)
+              worker = bundle_worker(func)
               inputs.each {|input| worker.enq(input) }
-              inputs.map { worker.deq }.tap { worker.stop }
+              inputs.map { worker.deq }
             end
           end
         end
       end
 
+      def bundle_worker(func = nil)
+        if @bundle_worker
+          @bundle_worker.tap do |worker|
+            worker.instance_variable_set(:@func, func) if func
+          end
+        else
+          worker_name = "Compact Index (#{display_uri.host})"
+          @bundle_worker ||= Bundler::Worker.new(25, worker_name, func)
+        end
+      end
+
       def cache_path
         Bundler.user_cache.join("compact_index", remote.cache_slug)
       end