Add and use a new dynamic-library-dirs field in the ghc-pkg info

Summary:
Build systems / package managers want to be able to control the file
layout of installed libraries. In general they may want/need to be able
to put the static libraries and dynamic libraries in different places.
The ghc-pkg library regisrtation needs to be able to handle this.

This is already possible in principle by listing both a static lib dir
and a dynamic lib dir in the library-dirs field (indeed some previous
versions of Cabal did this for shared libs on ELF platforms).

The downside of listing both dirs is twofold. There is a lack of
precision, if we're not careful with naming then we could end up
picking up the wrong library. The more immediate problem however is
that if we list both directories then both directories get included
into the ELF and Mach-O shared object runtime search paths. On ELF this
merely slows down loading of shared libs (affecting prog startup time).
On the latest OSX versions this provokes a much more serious problem:
that there is a rather low limit on the total size of the section
containing the runtime search path (and lib names and related) and thus
listing any unnecessary directories wastes the limited space.

So the solution in this patch is fairly straightforward: split the
static and dynamic library search paths in the ghc-pkg db and its use
within ghc. This is a traditional solution: pkg-config has the same
static / dynamic split (though it describes in in terms of private and
public, but it translates into different behaviour for static and
dynamic linking).

Indeed it would make perfect sense to also have a static/dynamic split
for the list of the libraries to use i.e. to have dynamic variants of
the hs-libraries and extra-libraries fields. These are not immediately
required so this patch does not add it, but it is a reasonable
direction to follow.

To handle compatibility, if the new dynamic-library-dirs field is not
specified then its value is taken from the library-dirs field.

Contains Cabal submodule update.

Test Plan:
Run ./validate

Get christiaanb and carter to test it on OSX Sierra, in combination
with Cabal/cabal-install changes to the default file layout for
libraries.

Reviewers: carter, austin, hvr, christiaanb, bgamari

Reviewed By: christiaanb, bgamari

Subscribers: ezyang, Phyx, thomie

Differential Revision: https://phabricator.haskell.org/D2611

GHC Trac Issues: #12479

10 files changed, 50 insertions, 10 deletions

diff --git a/compiler/backpack/DriverBkp.hs b/compiler/backpack/DriverBkp.hs
index 7b35b0c0cd..ccf74c8c9b 100644
--- a/compiler/backpack/DriverBkp.hs
+++ b/compiler/backpack/DriverBkp.hs
@@ -311,6 +311,7 @@ buildUnit session cid insts lunit = do
             hsLibraries = [],
             extraLibraries = [],
             extraGHCiLibraries = [],
+            libraryDynDirs = [],
             libraryDirs = [],
             frameworks = [],
             frameworkDirs = [],
diff --git a/compiler/ghci/Linker.hs b/compiler/ghci/Linker.hs
index 0b3fd94449..a2ca1b514b 100644
--- a/compiler/ghci/Linker.hs
+++ b/compiler/ghci/Linker.hs
@@ -1182,7 +1182,8 @@ linkPackage hsc_env pkg
    = do
         let dflags    = hsc_dflags hsc_env
             platform  = targetPlatform dflags
-            dirs      = Packages.libraryDirs pkg
+            dirs | interpreterDynamic dflags = Packages.libraryDynDirs pkg
+                 | otherwise                 = Packages.libraryDirs pkg
 
         let hs_libs   =  Packages.hsLibraries pkg
             -- The FFI GHCi import lib isn't needed as
diff --git a/compiler/main/DriverPipeline.hs b/compiler/main/DriverPipeline.hs
index aec89e4e4f..b578612ac4 100644
--- a/compiler/main/DriverPipeline.hs
+++ b/compiler/main/DriverPipeline.hs
@@ -428,7 +428,7 @@ linkingNeeded dflags staticLink linkables pkg_deps = do
 
         -- next, check libraries. XXX this only checks Haskell libraries,
         -- not extra_libraries or -l things from the command line.
-        let pkg_hslibs  = [ (libraryDirs c, lib)
+        let pkg_hslibs  = [ (collectLibraryPaths dflags [c], lib)
                           | Just c <- map (lookupInstalledPackage dflags) pkg_deps,
                             lib <- packageHsLibs dflags c ]
 
diff --git a/compiler/main/PackageConfig.hs b/compiler/main/PackageConfig.hs
index bff8cc3aa3..23f37da64d 100644
--- a/compiler/main/PackageConfig.hs
+++ b/compiler/main/PackageConfig.hs
@@ -103,6 +103,7 @@ pprPackageConfig InstalledPackageInfo {..} =
       field "trusted"              (ppr trusted),
       field "import-dirs"          (fsep (map text importDirs)),
       field "library-dirs"         (fsep (map text libraryDirs)),
+      field "dynamic-library-dirs" (fsep (map text libraryDynDirs)),
       field "hs-libraries"         (fsep (map text hsLibraries)),
       field "extra-libraries"      (fsep (map text extraLibraries)),
       field "extra-ghci-libraries" (fsep (map text extraGHCiLibraries)),
diff --git a/compiler/main/Packages.hs b/compiler/main/Packages.hs
index e0563da10c..86a37352b0 100644
--- a/compiler/main/Packages.hs
+++ b/compiler/main/Packages.hs
@@ -541,7 +541,8 @@ readPackageConfig dflags conf_file = do
   let
       top_dir = topDir dflags
       pkgroot = takeDirectory conf_file
-      pkg_configs1 = map (mungePackagePaths top_dir pkgroot) proto_pkg_configs
+      pkg_configs1 = map (mungePackageConfig top_dir pkgroot)
+                         proto_pkg_configs
       pkg_configs2 = setBatchPackageFlags dflags pkg_configs1
   --
   return (conf_file, pkg_configs2)
@@ -580,6 +581,22 @@ setBatchPackageFlags dflags pkgs = maybeDistrustAll pkgs
 
     distrust pkg = pkg{ trusted = False }
 
+mungePackageConfig :: FilePath -> FilePath
+                   -> PackageConfig -> PackageConfig
+mungePackageConfig top_dir pkgroot =
+    mungeDynLibFields
+  . mungePackagePaths top_dir pkgroot
+
+mungeDynLibFields :: PackageConfig -> PackageConfig
+mungeDynLibFields pkg =
+    pkg {
+      libraryDynDirs     = libraryDynDirs pkg
+                `orIfNull` libraryDirs pkg
+    }
+  where
+    orIfNull [] flags = flags
+    orIfNull flags _  = flags
+
 -- TODO: This code is duplicated in utils/ghc-pkg/Main.hs
 mungePackagePaths :: FilePath -> FilePath -> PackageConfig -> PackageConfig
 -- Perform path/URL variable substitution as per the Cabal ${pkgroot} spec
@@ -595,6 +612,7 @@ mungePackagePaths top_dir pkgroot pkg =
       importDirs  = munge_paths (importDirs pkg),
       includeDirs = munge_paths (includeDirs pkg),
       libraryDirs = munge_paths (libraryDirs pkg),
+      libraryDynDirs = munge_paths (libraryDynDirs pkg),
       frameworkDirs = munge_paths (frameworkDirs pkg),
       haddockInterfaces = munge_paths (haddockInterfaces pkg),
       haddockHTMLs = munge_urls (haddockHTMLs pkg)
@@ -1459,10 +1477,11 @@ collectIncludeDirs ps = nub (filter notNull (concatMap includeDirs ps))
 -- | Find all the library paths in these and the preload packages
 getPackageLibraryPath :: DynFlags -> [PreloadUnitId] -> IO [String]
 getPackageLibraryPath dflags pkgs =
-  collectLibraryPaths `fmap` getPreloadPackagesAnd dflags pkgs
+  collectLibraryPaths dflags `fmap` getPreloadPackagesAnd dflags pkgs
 
-collectLibraryPaths :: [PackageConfig] -> [FilePath]
-collectLibraryPaths ps = nub (filter notNull (concatMap libraryDirs ps))
+collectLibraryPaths :: DynFlags -> [PackageConfig] -> [FilePath]
+collectLibraryPaths dflags = nub . filter notNull
+                           . concatMap (libraryDirsForWay dflags)
 
 -- | Find all the link options in these and the preload packages,
 -- returning (package hs lib options, extra library options, other flags)
@@ -1514,6 +1533,12 @@ packageHsLibs dflags p = map (mkDynName . addSuffix) (hsLibraries p)
         expandTag t | null t = ""
                     | otherwise = '_':t
 
+-- | Either the 'libraryDirs' or 'libraryDynDirs' as appropriate for the way.
+libraryDirsForWay :: DynFlags -> PackageConfig -> [String]
+libraryDirsForWay dflags
+  | WayDyn `elem` ways dflags = libraryDynDirs
+  | otherwise                 = libraryDirs
+
 -- | Find all the C-compiler options in these and the preload packages
 getPackageExtraCcOpts :: DynFlags -> [PreloadUnitId] -> IO [String]
 getPackageExtraCcOpts dflags pkgs = do
diff --git a/compiler/main/SysTools.hs b/compiler/main/SysTools.hs
index e901bde06e..dd98883cb5 100644
--- a/compiler/main/SysTools.hs
+++ b/compiler/main/SysTools.hs
@@ -1584,7 +1584,7 @@ linkDynLib dflags0 o_files dep_packages
 
     pkgs <- getPreloadPackagesAnd dflags dep_packages
 
-    let pkg_lib_paths = collectLibraryPaths pkgs
+    let pkg_lib_paths = collectLibraryPaths dflags pkgs
     let pkg_lib_path_opts = concatMap get_pkg_lib_path_opts pkg_lib_paths
         get_pkg_lib_path_opts l
          | ( osElfTarget (platformOS (targetPlatform dflags)) ||
diff --git a/libraries/Cabal b/libraries/Cabal
-Subproject 3ece1013df8ba4c9719d2183afd2b03c4bddf85
+Subproject a7fb9b9ae733ceb3c52fee68e6e1a6ded5fb91d
diff --git a/libraries/ghc-boot/GHC/PackageDb.hs b/libraries/ghc-boot/GHC/PackageDb.hs
index f0333d4333..09991092ee 100644
--- a/libraries/ghc-boot/GHC/PackageDb.hs
+++ b/libraries/ghc-boot/GHC/PackageDb.hs
@@ -83,6 +83,7 @@ data InstalledPackageInfo compid srcpkgid srcpkgname instunitid unitid modulenam
        extraLibraries     :: [String],
        extraGHCiLibraries :: [String],
        libraryDirs        :: [FilePath],
+       libraryDynDirs     :: [FilePath],
        frameworks         :: [String],
        frameworkDirs      :: [FilePath],
        ldOptions          :: [String],
@@ -163,6 +164,7 @@ emptyInstalledPackageInfo =
        extraLibraries     = [],
        extraGHCiLibraries = [],
        libraryDirs        = [],
+       libraryDynDirs     = [],
        frameworks         = [],
        frameworkDirs      = [],
        ldOptions          = [],
@@ -306,7 +308,8 @@ instance (RepInstalledPackageInfo a b c d e f g) =>
          unitId componentId instantiatedWith sourcePackageId
          packageName packageVersion
          abiHash depends importDirs
-         hsLibraries extraLibraries extraGHCiLibraries libraryDirs
+         hsLibraries extraLibraries extraGHCiLibraries
+         libraryDirs libraryDynDirs
          frameworks frameworkDirs
          ldOptions ccOptions
          includes includeDirs
@@ -327,6 +330,7 @@ instance (RepInstalledPackageInfo a b c d e f g) =>
     put extraLibraries
     put extraGHCiLibraries
     put libraryDirs
+    put libraryDynDirs
     put frameworks
     put frameworkDirs
     put ldOptions
@@ -356,6 +360,7 @@ instance (RepInstalledPackageInfo a b c d e f g) =>
     extraLibraries     <- get
     extraGHCiLibraries <- get
     libraryDirs        <- get
+    libraryDynDirs     <- get
     frameworks         <- get
     frameworkDirs      <- get
     ldOptions          <- get
@@ -379,7 +384,8 @@ instance (RepInstalledPackageInfo a b c d e f g) =>
               abiHash
               (map fromStringRep depends)
               importDirs
-              hsLibraries extraLibraries extraGHCiLibraries libraryDirs
+              hsLibraries extraLibraries extraGHCiLibraries
+              libraryDirs libraryDynDirs
               frameworks frameworkDirs
               ldOptions ccOptions
               includes includeDirs
diff --git a/utils/ghc-cabal/Main.hs b/utils/ghc-cabal/Main.hs
index e72e46cc99..4b6b496b66 100644
--- a/utils/ghc-cabal/Main.hs
+++ b/utils/ghc-cabal/Main.hs
@@ -248,6 +248,10 @@ updateInstallDirTemplates relocatableBuild myPrefix myLibdir myDocdir idts
                           if relocatableBuild
                           then "$topdir"
                           else myLibdir,
+          dynlibdir = toPathTemplate $
+                          (if relocatableBuild
+                          then "$topdir"
+                          else myLibdir) </> "$libname",
           libsubdir = toPathTemplate "$libname",
           docdir    = toPathTemplate $
                           if relocatableBuild
diff --git a/utils/ghc-pkg/Main.hs b/utils/ghc-pkg/Main.hs
index 4466f58878..063a8067d9 100644
--- a/utils/ghc-pkg/Main.hs
+++ b/utils/ghc-pkg/Main.hs
@@ -1112,6 +1112,7 @@ convertPackageInfoToCacheFormat pkg =
        GhcPkg.extraLibraries     = extraLibraries pkg,
        GhcPkg.extraGHCiLibraries = extraGHCiLibraries pkg,
        GhcPkg.libraryDirs        = libraryDirs pkg,
+       GhcPkg.libraryDynDirs     = libraryDynDirs pkg,
        GhcPkg.frameworks         = frameworks pkg,
        GhcPkg.frameworkDirs      = frameworkDirs pkg,
        GhcPkg.ldOptions          = ldOptions pkg,
@@ -1624,6 +1625,7 @@ checkPackageConfig pkg verbosity db_stack
   checkDuplicateDepends (depends pkg)
   mapM_ (checkDir False "import-dirs")  (importDirs pkg)
   mapM_ (checkDir True  "library-dirs") (libraryDirs pkg)
+  mapM_ (checkDir True  "dynamic-library-dirs") (libraryDynDirs pkg)
   mapM_ (checkDir True  "include-dirs") (includeDirs pkg)
   mapM_ (checkDir True  "framework-dirs") (frameworkDirs pkg)
   mapM_ (checkFile   True "haddock-interfaces") (haddockInterfaces pkg)
@@ -1632,7 +1634,7 @@ checkPackageConfig pkg verbosity db_stack
   checkExposedModules db_stack pkg
   checkOtherModules pkg
   let has_code = Set.null (openModuleSubstFreeHoles (Map.fromList (instantiatedWith pkg)))
-  when has_code $ mapM_ (checkHSLib verbosity (libraryDirs pkg)) (hsLibraries pkg)
+  when has_code $ mapM_ (checkHSLib verbosity (libraryDirs pkg ++ libraryDynDirs pkg)) (hsLibraries pkg)
   -- ToDo: check these somehow?
   --    extra_libraries :: [String],
   --    c_includes      :: [String],