Accumulate Haddock comments in P (#17544, #17561, #8944)wip/haddock-accum

Haddock comments are, first and foremost, comments. It's very annoying
to incorporate them into the grammar. We can take advantage of an
important property: adding a Haddock comment does not change the parse
tree in any way other than wrapping some nodes in HsDocTy and the like
(and if it does, that's a bug).

This patch implements the following:

* Accumulate Haddock comments with their locations in the P monad.
  This is handled in the lexer.

* After parsing, do a pass over the AST to associate Haddock comments
  with AST nodes using location info.

* Report the leftover comments to the user as a warning (-Winvalid-haddock).

2 files changed, 63 insertions, 0 deletions

diff --git a/docs/users_guide/8.12.1-notes.rst b/docs/users_guide/8.12.1-notes.rst
index b7f0444d69..c63cbf5ab4 100644
--- a/docs/users_guide/8.12.1-notes.rst
+++ b/docs/users_guide/8.12.1-notes.rst
@@ -279,6 +279,46 @@ Arrow notation
          within 0.5 -< x
          ... |)
 
+Haddock
+~~~~~~~
+
+- Parsing is now more robust to insufficiently indented Haddock comments::
+
+    class C a where
+      f :: a -> a
+    -- ^ This comment used to trigger a parse error
+      g :: a -> a
+
+- :ghc-flag:`-Winvalid-haddock` is a new warning that reports discarded Haddock
+  comments that cannot be associated with AST elements::
+
+    myValue =
+      -- | Invalid (discarded) comment in an expression
+      2 + 2
+
+- When faced with several comments for a data constructor or a data constructor
+  field, Haddock now picks the first one instead of the last one.  The
+  extraneous comment is reported as invalid when :ghc-flag:`-Winvalid-haddock`
+  is enabled::
+
+    data T
+      -- | First comment
+      = MkT
+      -- ^ Second comment (rejected)
+
+
+- Haddock is now more relaxed about the placement of comments in types relative
+  to the function arrow ``->``, allowing more formatting styles::
+
+    f :: Int ->   -- ^ comment on Int (no longer a parse error)
+         Bool     -- ^ comment on Bool
+
+- Haddock can now parse the documentation comment for the first declaration in
+  a module without a module header (:ghc-ticket:`17561`)::
+
+    -- | This comment used to trigger a parse error
+    main = putStrLn "Hello"
+
 ``ghc-prim`` library
 ~~~~~~~~~~~~~~~~~~~~
 
@@ -319,6 +359,10 @@ Arrow notation
   erased, and their ``exceptions``-alternatives are meant to be used in the
   GHC code instead.
 
+- ``parseModule`` is now the only parser entry point that deals with Haddock
+  comments. The other entry points (``parseDeclaration``, ``parseExpression``,
+  etc) do not insert the Haddock comments into the AST.
+
 ``base`` library
 ~~~~~~~~~~~~~~~~
 
diff --git a/docs/users_guide/using-warnings.rst b/docs/users_guide/using-warnings.rst
index 256d143f45..a83cc6837e 100644
--- a/docs/users_guide/using-warnings.rst
+++ b/docs/users_guide/using-warnings.rst
@@ -1762,6 +1762,25 @@ of ``-W(no-)*``.
     You may want to enable this warning on a clean build or enable :ghc-flag:`-fforce-recomp`
     in order to get reliable results.
 
+.. ghc-flag:: -Winvalid-haddock
+    :shortdesc: warn when a Haddock comment occurs in an invalid position
+    :type: dynamic
+    :category:
+
+    :since: 8.12
+
+    When the ``-haddock`` option is enabled, GHC collects documentation
+    comments and associates them with declarations, function arguments, data
+    constructors, and other syntactic elements. Documentation comments in
+    invalid positions are discarded::
+
+        myValue =
+          -- | Invalid (discarded) comment in an expression
+          2 + 2
+
+    This warning informs you about discarded documentation comments.
+    It has no effect when :ghc-flag:`-haddock` is disabled.
+
 If you're feeling really paranoid, the :ghc-flag:`-dcore-lint` option is a good choice.
 It turns on heavyweight intra-pass sanity-checking within GHC. (It checks GHC's
 sanity, not yours.)