Merge changes made in Gnus trunk.

nnir.el: Batch header retrieval.
proto-stream.el: New library to provide protocol-specific TLS/STARTTLS connections for IMAP, NNTP, SMTP, POP3 and similar protocols.
nnimap.el (nnimap-open-connection): Use it.
proto-stream.el (open-proto-stream): Complete the documentation.
nnimap.el (nnimap-open-connection): Check for "OK" from the greeting.
nntp.el: Use proto-streams for the relevant connections types.
nntp.el (nntp-open-connection): Switch on STARTTLS on supported servers.
proto-stream.el (open-proto-stream): Add a way to specify what the end of a command is.
proto-stream.el (proto-stream-open-tls): Delete output from openssl if we're using tls.el.
proto-stream.el (proto-stream-open-network): If we don't have gnutls-cli or gnutls built in, then don't try to establish a STARTTLS connection.
color.el (color-lab->srgb): Fix function call name.
proto-stream.el: Fix the syntax in the comment.
nntp.el (nntp-open-connection): Fix the STARTTLS command syntax.
proto-stream.el (proto-stream-open-starttls): Actually implement the starttls.el STARTTLS.
proto-stream.el (proto-stream-always-use-starttls): New variable.
proto-stream.el (proto-stream-open-starttls): De-duplicate the starttls code.
proto-stream.el (proto-stream-open-starttls): Folded back into the main function.
proto-stream.el (proto-stream-command): Refactor out.
nnimap.el (nnimap-stream): Change default to `undecided'.
nnimap.el (nnimap-open-connection): If `nnimap-stream' is `undecided', try ssl first, and then network.
nnimap.el (nnimap-open-connection-1): Respect nnimap-server-port.
nnimap.el (nnimap-open-connection): Be more backwards-compatible.
proto-stream.el (open-protocol-stream): Renamed from open-proto-stream.
proto-stream.el (proto-stream-open-network): When doing opportunistic TLS upgrades we don't really care about the identity of the peer.
gnus.texi (Customizing the IMAP Connection): Note the new defaults.
gnus.texi (Direct Functions): Note the STARTTLS upgrade.
proto-stream.el (proto-stream-open-network): Force starttls.el to use gnutls-cli, since that what we've checked for.
proto-stream.el (proto-stream-always-use-starttls): Only default to t if open-gnutls-stream exists.
proto-stream.el (proto-stream-open-network): If STARTTLS failed, then just open a normal connection.
proto-stream.el (proto-stream-open-network): Wait until the greeting before doing STARTTLS.
nnimap.el (nnimap-open-connection-1): Always upgrade to STARTTLS (for backwards compatibility).
nnimap.el (nnimap-open-connection-1): Really respect nnimap-server-port.
nntp.el (nntp-open-connection): Provide a :success condition.
nnimap.el (nnimap-open-connection-1): Ditto.
proto-stream.el (proto-stream-open-network): See what the response to the STARTTLS command is.
proto-stream.el (proto-stream-open-network): Add some comments.
proto-stream.el: Fix example.
proto-stream.el (open-protocol-stream): Actually mention the STARTTLS upgrade.
nnir.el (nnir-get-active): Skip nnir-ignored-newsgroups when searching.
nnir.el (nnir-ignore-newsgroups): Fix default value.
nnir.el (nnir-run-gmane): Use mm-delete-duplicates instead of delete-dups that is not available in XEmacs 21.4.
mm-util.el (mm-delete-duplicates): Add comment.
gnus-sum.el (gnus-summary-delete-article): If delete fails don't change the registry.
nnimap.el (nnimap-open-connection-1): w32 open-network-stream doesn't seem to accept strings-with-numbers as port numbers.
color.el: fix docstring to use English rather than math notation for intervals.
shr.el (shr-find-fill-point): Don't break before apostrophes.
nnir.el (nnir-request-move-article): Bail out if no move support in group.
color.el (color-rgb->hsv): Fix docstring.
nnir.el (nnir-get-active): Improve active list retrieval.
shr.el (shr-find-fill-point): Work better for kinsoku chars and apostrophes.
gnus-gravatar.el (gnus-gravatar-size): Set gnus-gravatar-size to nil.
nnimap.el (nnimap-open-connection-1): Use gnus-string-match-p.
nnimap.el (nnimap-open-connection-1): Fix PREAUTH.
proto-stream.el (open-protocol-stream): All starttls connections are handled by the network handler.
gnus-gravatar.el (gnus-gravatar-insert): Delete unnecessary binding to t of inhibit-read-only since it is inside gnus-with-article-headers.
gnus-gravatar.el (gnus-gravatar-transform-address): Use mail-extract-address-components that supports non-ASCII names rather than mail-header-parse-addresses.
shr.el (shr-find-fill-point): Don't break line between kinsoku-bol characters.
gnus-gravatar.el (gnus-gravatar-insert): Allow LWSP in the middle of names.
nnmaildir.el (nnmaildir-request-set-mark): Add article to add-mark funcall.
gnus-msg.el: Remove nastygram thing.
message.el (message-from-style): Fix comment.
message.el (message-user-organization): Do not use gnus-local-organization.
gnus.el: Remove gnus-local-organization.
rtree.el: New file to handle range trees.
nnir.el, gnus-sum.el: Redo the way nnir handles registry updates.
rtree.el (rtree-extract): Simplify.
gnus-win.el (gnus-configure-windows): Remove Gnus 3.x setting support.
gnus-msg.el: Mark gnus-outgoing-message-group as obsolete.
gnus.texi (Archived Messages): Remove gnus-outgoing-message-group.
gnus-win.el (gnus-configure-frame): Remove old compatibility code.
rtree.el (rtree-memq): Rewrite it as a non-recursive function.
rtree.el (rtree-add, rtree-delq, rtree-length): Implement.
rtree.el (rtree-add): Make code slightly faster.
nnir.el: Allow modified summary-line-format in nnir summary buffers.

1 files changed, 73 insertions, 120 deletions

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index cb4c9f0108c..a53f9ac468d 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -45,6 +45,7 @@
 (require 'tls)
 (require 'parse-time)
 (require 'nnmail)
+(require 'proto-stream)
 
 (eval-when-compile
   (require 'gnus-sum))
@@ -62,9 +63,10 @@
 If nnimap-stream is `ssl', this will default to `imaps'.  If not,
 it will default to `imap'.")
 
-(defvoo nnimap-stream 'ssl
+(defvoo nnimap-stream 'undecided
   "How nnimap will talk to the IMAP server.
-Values are `ssl', `network', `starttls' or `shell'.")
+Values are `ssl', `network', `starttls' or `shell'.
+The default is to try `ssl' first, and then `network'.")
 
 (defvoo nnimap-shell-program (if (boundp 'imap-shell-program)
 				 (if (listp imap-shell-program)
@@ -271,16 +273,6 @@ textual parts.")
     (push (current-buffer) nnimap-process-buffers)
     (current-buffer)))
 
-(defun nnimap-open-shell-stream (name buffer host port)
-  (let ((process-connection-type nil))
-    (start-process name buffer shell-file-name
-		   shell-command-switch
-		   (format-spec
-		    nnimap-shell-program
-		    (format-spec-make
-		     ?s host
-		     ?p port)))))
-
 (defun nnimap-credentials (address ports &optional inhibit-create)
   (let (port credentials)
     ;; Request the credentials from all ports, but only query on the
@@ -310,111 +302,79 @@ textual parts.")
 			(* 5 60)))
 	    (nnimap-send-command "NOOP")))))))
 
-(declare-function gnutls-negotiate "gnutls"
-		  (proc type &optional priority-string trustfiles keyfiles))
-
 (defun nnimap-open-connection (buffer)
+  ;; Be backwards-compatible -- the earlier value of nnimap-stream was
+  ;; `ssl' when nnimap-server-port was nil.  Sort of.
+  (when (and nnimap-server-port
+	     (eq nnimap-stream 'undecided))
+    (setq nnimap-stream 'ssl))
+  (let ((stream
+	 (if (eq nnimap-stream 'undecided)
+	     (loop for type in '(ssl network)
+		   for stream = (let ((nnimap-stream type))
+				  (nnimap-open-connection-1 buffer))
+		   while (eq stream 'no-connect)
+		   finally (return stream))
+	   (nnimap-open-connection-1 buffer))))
+    (if (eq stream 'no-connect)
+	nil
+      stream)))
+
+(defun nnimap-open-connection-1 (buffer)
   (unless nnimap-keepalive-timer
     (setq nnimap-keepalive-timer (run-at-time (* 60 15) (* 60 15)
 					      'nnimap-keepalive)))
-  (block nil
-    (with-current-buffer (nnimap-make-process-buffer buffer)
-      (let* ((coding-system-for-read 'binary)
-	     (coding-system-for-write 'binary)
-	     (port nil)
-	     (ports
-	      (cond
-	       ((or (eq nnimap-stream 'network)
-		    (and (eq nnimap-stream 'starttls)
-			 (fboundp 'open-gnutls-stream)))
-		(nnheader-message 7 "Opening connection to %s..."
-				  nnimap-address)
-		(open-network-stream
-		 "*nnimap*" (current-buffer) nnimap-address
-		 (setq port
-		       (or nnimap-server-port
-			   (if (netrc-find-service-number "imap")
-			       "imap"
-			     "143"))))
-		'("143" "imap"))
-	       ((eq nnimap-stream 'shell)
-		(nnheader-message 7 "Opening connection to %s via shell..."
-				  nnimap-address)
-		(nnimap-open-shell-stream
-		 "*nnimap*" (current-buffer) nnimap-address
-		 (setq port (or nnimap-server-port "imap")))
-		'("imap"))
-	       ((eq nnimap-stream 'starttls)
-		(nnheader-message 7 "Opening connection to %s via starttls..."
-			 nnimap-address)
-		(let ((tls-program
-		       '("openssl s_client -connect %h:%p -no_ssl2 -ign_eof -starttls imap")))
-		  (open-tls-stream
-		   "*nnimap*" (current-buffer) nnimap-address
-		   (setq port (or nnimap-server-port "imap"))))
-		'("imap"))
-	       ((memq nnimap-stream '(ssl tls))
-		(nnheader-message 7 "Opening connection to %s via tls..."
-				  nnimap-address)
-		(funcall (if (fboundp 'open-gnutls-stream)
-			     'open-gnutls-stream
-			   'open-tls-stream)
-			 "*nnimap*" (current-buffer) nnimap-address
-			 (setq port
-			       (or nnimap-server-port
-				   (if (netrc-find-service-number "imaps")
-				       "imaps"
-				     "993"))))
-		'("143" "993" "imap" "imaps"))
-	       (t
-		(error "Unknown stream type: %s" nnimap-stream))))
-	     connection-result login-result credentials)
-	(setf (nnimap-process nnimap-object)
-	      (get-buffer-process (current-buffer)))
-	(if (not (and (nnimap-process nnimap-object)
-		      (memq (process-status (nnimap-process nnimap-object))
-			    '(open run))))
-	    (nnheader-report 'nnimap "Unable to contact %s:%s via %s"
-			     nnimap-address port nnimap-stream)
-	  (gnus-set-process-query-on-exit-flag
-	   (nnimap-process nnimap-object) nil)
-	  (if (not (setq connection-result (nnimap-wait-for-connection)))
-	      (nnheader-report 'nnimap
-			       "%s" (buffer-substring
-				     (point) (line-end-position)))
-	    ;; Store the greeting (for debugging purposes).
-	    (setf (nnimap-greeting nnimap-object)
-		  (buffer-substring (line-beginning-position)
-				    (line-end-position)))
-	    (nnimap-get-capabilities)
-	    (when nnimap-server-port
-	      (push (format "%s" nnimap-server-port) ports))
-	    ;; If this is a STARTTLS-capable server, then sever the
-	    ;; connection and start a STARTTLS connection instead.
+  (with-current-buffer (nnimap-make-process-buffer buffer)
+    (let* ((coding-system-for-read 'binary)
+	   (coding-system-for-write 'binary)
+	   (port nil)
+	   (ports
 	    (cond
-	     ((and (or (and (eq nnimap-stream 'network)
-			    (nnimap-capability "STARTTLS"))
-		       (eq nnimap-stream 'starttls))
-		   (fboundp 'open-gnutls-stream))
-	      (nnimap-command "STARTTLS")
-	      (gnutls-negotiate (nnimap-process nnimap-object) nil)
-	      ;; Get the capabilities again -- they may have changed
-	      ;; after doing STARTTLS.
-	      (nnimap-get-capabilities))
-	     ((and (eq nnimap-stream 'network)
-		   (nnimap-capability "STARTTLS"))
-	      (let ((nnimap-stream 'starttls))
-		(let ((tls-process
-		       (nnimap-open-connection buffer)))
-		  ;; If the STARTTLS connection was successful, we
-		  ;; kill our first non-encrypted connection.  If it
-		  ;; wasn't successful, we just use our unencrypted
-		  ;; connection.
-		  (when (memq (process-status tls-process) '(open run))
-		    (delete-process (nnimap-process nnimap-object))
-		    (kill-buffer (current-buffer))
-		    (return tls-process))))))
-	    (unless (equal connection-result "PREAUTH")
+	     ((or (eq nnimap-stream 'network)
+		  (eq nnimap-stream 'starttls))
+	      (nnheader-message 7 "Opening connection to %s..."
+				nnimap-address)
+	      '("143" "imap"))
+	     ((eq nnimap-stream 'shell)
+	      (nnheader-message 7 "Opening connection to %s via shell..."
+				nnimap-address)
+	      '("imap"))
+	     ((memq nnimap-stream '(ssl tls))
+	      (nnheader-message 7 "Opening connection to %s via tls..."
+				nnimap-address)
+	      '("143" "993" "imap" "imaps"))
+	     (t
+	      (error "Unknown stream type: %s" nnimap-stream))))
+	   (proto-stream-always-use-starttls t)
+           login-result credentials)
+      (when nnimap-server-port
+	(setq ports (append ports (list nnimap-server-port))))
+      (destructuring-bind (stream greeting capabilities)
+	  (open-protocol-stream
+	   "*nnimap*" (current-buffer) nnimap-address (car (last ports))
+	   :type nnimap-stream
+	   :shell-command nnimap-shell-program
+	   :capability-command "1 CAPABILITY\r\n"
+	   :success " OK "
+	   :starttls-function
+	   (lambda (capabilities)
+	     (when (gnus-string-match-p "STARTTLS" capabilities)
+	       "1 STARTTLS\r\n")))
+	(setf (nnimap-process nnimap-object) stream)
+	(if (not stream)
+	    (progn
+	      (nnheader-report 'nnimap "Unable to contact %s:%s via %s"
+			       nnimap-address port nnimap-stream)
+	      'no-connect)
+	  (gnus-set-process-query-on-exit-flag stream nil)
+	  (if (not (gnus-string-match-p "[*.] \\(OK\\|PREAUTH\\)" greeting))
+	      (nnheader-report 'nnimap "%s" greeting)
+	    ;; Store the greeting (for debugging purposes).
+	    (setf (nnimap-greeting nnimap-object) greeting)
+	    (setf (nnimap-capabilities nnimap-object)
+		  (mapcar #'upcase
+			  (split-string capabilities)))
+	    (unless (gnus-string-match-p "[*.] PREAUTH" greeting)
 	      (if (not (setq credentials
 			     (if (eq nnimap-authenticator 'anonymous)
 				 (list "anonymous"
@@ -456,13 +416,6 @@ textual parts.")
 		(nnimap-command "ENABLE QRESYNC"))
 	      (nnimap-process nnimap-object))))))))
 
-(defun nnimap-get-capabilities ()
-  (setf (nnimap-capabilities nnimap-object)
-	(mapcar
-	 #'upcase
-	 (nnimap-find-parameter
-	  "CAPABILITY" (cdr (nnimap-command "CAPABILITY"))))))
-
 (defun nnimap-quote-specials (string)
   (with-temp-buffer
     (insert string)
@@ -1110,7 +1063,7 @@ textual parts.")
 		   uidvalidity
 		   modseq)
 	      (push
-	       (list (nnimap-send-command "EXAMINE %S (QRESYNC (%s %s))"
+	       (list (nnimap-send-command "EXAMINE %S (QRESYNC  (%s %s))"
 					  (utf7-encode group t)
 					  uidvalidity modseq)
 		     'qresync