Don't push artifacts to remote caches that already contain them

This adds two separate checks to avoid pushing artifacts unnecessarily.

First, in the OSTreeCache.push() method we now first obtain the list of
remotes that contain the refs that we are about to push. We then avoid
pushing to any remote that already contains that ref.

Note that the internal ref map is read once on process startup, so
if multiple `bst` processes are pushing to a cache they might still both
push the same artifact. There is an existing issue for this:

  https://gitlab.com/BuildStream/buildstream/issues/179

Secondly the Element._skip_push() method now checks if all remote
caches configured for pushing already have a given artifact, and will
skip the push job altogether if they do. The first check would already
mean that no pushes would actually happen, but without the second check
the user would still see Push jobs being created for every artifact
which would be quite misleading.

1 files changed, 21 insertions, 1 deletions

diff --git a/buildstream/_artifactcache/artifactcache.py b/buildstream/_artifactcache/artifactcache.py
index 31e1356c8..4ee07b570 100644
--- a/buildstream/_artifactcache/artifactcache.py
+++ b/buildstream/_artifactcache/artifactcache.py
@@ -217,7 +217,27 @@ class ArtifactCache():
     #     element (Element): The Element to check
     #     strength (_KeyStrength): Either STRONG or WEAK key strength, or None
     #
-    # Returns: True if the artifact is in the cache, False otherwise
+    # Returns: True if the artifact is in a cache, False otherwise
     #
     def remote_contains(self, element, strength=None):
         return False
+
+    # push_needed():
+    #
+    # Check whether an artifact for the specified Element needs to be pushed to
+    # any of the configured push remotes. The policy is to push every artifact
+    # we build to every configured push remote, so this should only return False
+    # if all of the configured push remotes already contain the given artifact.
+    #
+    # This function checks for presence of the artifact only using its strong
+    # key. The presence of the weak key in a cache does not necessarily indicate
+    # that this particular artifact is present, only that there is a
+    # partially-compatible version available.
+    #
+    # Args:
+    #     element (Element): The Element to check
+    #
+    # Returns: False if all the push remotes have the artifact, True otherwise
+    #
+    def push_needed(self, element):
+        return False