From 1edbc9f5c08daafb3e13131dfc8df656b2a02edc Mon Sep 17 00:00:00 2001 From: Richard Maw Date: Wed, 27 Aug 2014 16:39:53 +0100 Subject: yarns: Use a more realistic simulation of a git server Previously we would use file: URIs to point to the git repositories. This was fast and simple, but had the drawback that it bypassed all the git cache logic, so changes to the git cache weren't adequately covered by the test suite. Now we spool up a simulated git server per scenario, and shut it down at the end. --- yarns/architecture.yarn | 2 ++ yarns/branches-workspaces.yarn | 33 +++++++++++++++++++++++---------- yarns/building.yarn | 1 + yarns/deployment.yarn | 7 +++++++ yarns/implementations.yarn | 24 +++++++++++++++++++++++- yarns/regression.yarn | 4 ++++ yarns/splitting.yarn | 2 ++ 7 files changed, 62 insertions(+), 11 deletions(-) diff --git a/yarns/architecture.yarn b/yarns/architecture.yarn index 2240c428..d4de308e 100644 --- a/yarns/architecture.yarn +++ b/yarns/architecture.yarn @@ -9,6 +9,7 @@ Morph Cross-Building Tests AND the user attempts to build the system base-system-testarch.morph in branch master THEN morph failed AND the build error message includes the string "Are you trying to cross-build?" + FINALLY the git server is shut down Morph Cross-Bootstrap Tests @@ -20,3 +21,4 @@ Morph Cross-Bootstrap Tests AND a system called base-system-testarch.morph for the test architecture in the git server WHEN the user checks out the system branch called master THEN the user cross-bootstraps the system base-system-testarch.morph in branch master of repo test:morphs to the arch testarch + FINALLY the git server is shut down diff --git a/yarns/branches-workspaces.yarn b/yarns/branches-workspaces.yarn index de9f95a3..200f9508 100644 --- a/yarns/branches-workspaces.yarn +++ b/yarns/branches-workspaces.yarn @@ -55,6 +55,7 @@ to investigate chunks in existing branches. WHEN the user edits the chunk test-chunk in branch master THEN the edited chunk test:test-chunk has git branch master + FINALLY the git server is shut down Checking out a system branch should fail, if the branch doesn't exist. @@ -63,6 +64,7 @@ Checking out a system branch should fail, if the branch doesn't exist. AND a git server WHEN the user attempts to check out the system branch called foo THEN morph failed + FINALLY the git server is shut down Branching system branches ----------------------------------------- @@ -74,6 +76,7 @@ We can, instead, create a new system branch, off master. AND a git server WHEN the user creates a system branch called foo THEN the system branch foo is checked out + FINALLY the git server is shut down We can also branch off another system branch. However, we need to first push the other branch to the git server, since Morph is not smart enough @@ -86,6 +89,7 @@ to check for that locally. AND the user pushes the system branch called foo to the git server AND the user creates a system branch called bar, based on foo THEN the system branch bar is checked out + FINALLY the git server is shut down Query commands in workspaces ---------------------------- @@ -114,6 +118,7 @@ repository, which is the system branch root repository. WHEN the user reports the workspace from the directory master/test/morphs THEN the workspace is reported correctly + FINALLY the git server is shut down However, running it outside a workspace should fail. @@ -139,6 +144,7 @@ current working directory, it will find it and report it correctly. WHEN the user reports the system branch from the directory . THEN the system branch is reported as master + FINALLY the git server is shut down However, if there's two system branches checked out below the current directory, things should fail. @@ -150,6 +156,7 @@ current directory, things should fail. AND the user creates a system branch called foo AND the user attempts to report the system branch from the directory . THEN morph failed + FINALLY the git server is shut down `morph show-branch-root` reports the path of the system branch root repository. It can be run inside a checkout, or somewhere outside a @@ -164,6 +171,7 @@ checkout, where exactly one checkout exists below. WHEN the user reports the system branch root repository from the directory . THEN the system branch root repository is reported as workspace/master/test/morphs + FINALLY the git server is shut down However, it fails if run outside a checkout and there's no system branches checked out. @@ -173,6 +181,7 @@ branches checked out. AND a git server WHEN the user attempts to report the system branch root repository from the directory . THEN morph failed + FINALLY the git server is shut down Editing components ------------------ @@ -199,8 +208,9 @@ purposes. Editing a morphology should not cause it to start having repo or ref fields when referring to strata, when it didn't before. - AND in branch foo, system systems/test-system.morph refers to test-stratum without repo - AND in branch foo, system systems/test-system.morph refers to test-stratum without ref + AND in branch foo, system systems/test-system.morph refers to test-stratum without repo + AND in branch foo, system systems/test-system.morph refers to test-stratum without ref + FINALLY the git server is shut down Temporary Build Branch behaviour -------------------------------- @@ -372,14 +382,15 @@ We now don't need temporary build branches for local builds. Nor do we need temporary build branches for distributed builds. - GIVEN the workspace contains no temporary build branches - AND the git server contains no temporary build branches - AND we must build from pushed branches - WHEN the user builds systems/test-system.morph of the baserock/test branch - THEN the morphs repository in the workspace for baserock/test has no temporary build branches - AND the test-chunk repository in the workspace for baserock/test has no temporary build branches - AND no temporary build branches were pushed to the morphs repository - AND no temporary build branches were pushed to the test-chunk repository + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we must build from pushed branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has no temporary build branches + AND the test-chunk repository in the workspace for baserock/test has no temporary build branches + AND no temporary build branches were pushed to the morphs repository + AND no temporary build branches were pushed to the test-chunk repository + FINALLY the git server is shut down ### Temporary Build Branch implementations ### @@ -430,6 +441,7 @@ repositories referenced in the system branch. WHEN committing changes in test:test-chunk in branch foo THEN morph reports no outstanding changes in foo + FINALLY the git server is shut down `morph foreach` -------------- @@ -445,6 +457,7 @@ branch checkout. AND running shell command in each repo in foo THEN morph ran command in test/morphs in foo AND morph ran command in test/test-chunk in foo + FINALLY the git server is shut down Generating a manifest works diff --git a/yarns/building.yarn b/yarns/building.yarn index 6b90ce64..c708b5bb 100644 --- a/yarns/building.yarn +++ b/yarns/building.yarn @@ -7,3 +7,4 @@ Morph Building Tests WHEN the user checks out the system branch called master AND the user creates an uncommitted system morphology called systems/base-system.morph for our architecture in system branch master THEN morph build the system systems/base-system.morph of the branch master + FINALLY the git server is shut down diff --git a/yarns/deployment.yarn b/yarns/deployment.yarn index 3252647d..4039d551 100644 --- a/yarns/deployment.yarn +++ b/yarns/deployment.yarn @@ -8,6 +8,7 @@ Morph Deployment Tests AND the user attempts to deploy the system systems/test-system.morph in branch master THEN morph failed AND the deploy error message includes the string "morph deploy is only supported for cluster morphologies" + FINALLY the git server is shut down SCENARIO deploying a cluster morphology as a tarfile GIVEN a workspace @@ -21,6 +22,7 @@ Morph Deployment Tests WHEN the user builds the system systems/test-system.morph in branch master AND the user attempts to deploy the cluster test-cluster.morph in branch master THEN morph succeeded + FINALLY the git server is shut down Some deployment types support upgrades, but some do not and Morph needs to make this clear. @@ -37,6 +39,7 @@ this clear. WHEN the user builds the system systems/test-system.morph in branch master AND the user attempts to upgrade the cluster test-cluster.morph in branch master THEN morph failed + FINALLY the git server is shut down The rawdisk write extension supports both initial deployment and subsequent upgrades. Note that the rawdisk upgrade code needs bringing up to date to use @@ -58,6 +61,7 @@ the same code paths as a real upgrade. THEN morph succeeded WHEN the user attempts to upgrade the cluster test-cluster.morph in branch master with options test-system.VERSION_LABEL=test2 THEN morph succeeded + FINALLY the git server is shut down Nested deployments ================== @@ -102,6 +106,7 @@ deployed system contains the other. Since the baserock directory is in every system, we can check for that. AND tarball test.tar contains var/lib/sysroots/test-system/baserock + FINALLY the git server is shut down Initramfs deployments ===================== @@ -185,6 +190,7 @@ will mention the initramfs, and the UUID of the disk. AND file mnt/extlinux.conf matches initramfs AND file mnt/extlinux.conf matches root=UUID= FINALLY mnt is unmounted + AND the git server is shut down Partial deployments =================== @@ -276,3 +282,4 @@ deployment. WHEN the user attempts to deploy test-system.sysroot from cluster test-cluster.morph in branch master THEN morph failed + FINALLY the git server is shut down diff --git a/yarns/implementations.yarn b/yarns/implementations.yarn index 477849eb..422c2eea 100644 --- a/yarns/implementations.yarn +++ b/yarns/implementations.yarn @@ -258,12 +258,25 @@ another to hold a chunk. run_in "$DATADIR/gits/morphs" git commit -m Initial. run_in "$DATADIR/gits/morphs" git tag -a "test-tag" -m "Tagging test-tag" + # Start a git daemon to serve our git repositories + port_file="$DATADIR/git-daemon-port" + pid_file="$DATADIR/git-daemon-pid" + mkfifo "$port_file" + # git-daemon needs --foo=bar style arguments so we do that for consistency + start-stop-daemon --start --pidfile="$pid_file" --background \ + --make-pidfile --verbose \ + --startas="$SRCDIR/scripts/git-daemon-wrap" -- \ + --port-file="$port_file" \ + --export-all --verbose --base-path="$DATADIR/gits" \ + --enable=receive-pack #allow push + GIT_DAEMON_PORT="$(cat "$port_file")" + # Create the Morph configuration file so we can access the repos # using test:foo URL aliases. cat << EOF > "$DATADIR/morph.conf" [config] - repo-alias = test=file://$DATADIR/gits/%s#file://$DATADIR/gits/%s + repo-alias = test=git://127.0.0.1:$GIT_DAEMON_PORT/%s#git://127.0.0.1:$GIT_DAEMON_PORT/%s cachedir = $DATADIR/cache tempdir = $DATADIR/tmp trove-host= [] @@ -272,6 +285,15 @@ another to hold a chunk. mkdir "$DATADIR/cache" mkdir "$DATADIR/tmp" +Some resources are cleaned up by yarn, forked processes aren't one of +these, so need to shut down the git daemon after we finish. + + IMPLEMENTS FINALLY the git server is shut down + pid_file="$DATADIR/git-daemon-pid" + if [ -e "$pid_file" ]; then + start-stop-daemon --stop --pidfile "$pid_file" --oknodo + fi + We need a consistent value for the architecture in some tests, so we have a morphology using the test architecture. diff --git a/yarns/regression.yarn b/yarns/regression.yarn index 6f499d90..7991ab12 100644 --- a/yarns/regression.yarn +++ b/yarns/regression.yarn @@ -11,6 +11,7 @@ Testing if we can build after checking out from a tag. AND a git server WHEN the user checks out the system tag called test-tag THEN morph build the system systems/test-system.morph of the tag test-tag + FINALLY the git server is shut down Running `morph branch` when the branch directory exists doesn't @@ -32,6 +33,7 @@ The branch is checked out correctly, now it should fail if the user executes The branch still checked out. AND the system branch foo is checked out + FINALLY the git server is shut down It doesn't make much sense to be able to build a system with only @@ -45,6 +47,7 @@ area, hence their results cannot be trusted. WHEN the user checks out the system branch called master AND the user attempts to build the system bootstrap-system.morph in branch master THEN the build error message includes the string "No non-bootstrap chunks found" + FINALLY the git server is shut down When we started allowing multiple artifacts, a long-standing bug in cache-key computation was discovered, it didn't include artifact names, @@ -67,6 +70,7 @@ source it depended on. WHEN the user builds the system systems/test-system.morph in branch master AND the user deploys the cluster test-cluster.morph in branch master with options test-system.location="$DATADIR/test.tar" THEN tarball test.tar contains baserock/test-chunk-misc.meta + FINALLY the git server is shut down Implementations diff --git a/yarns/splitting.yarn b/yarns/splitting.yarn index d4b942d8..9248f60c 100644 --- a/yarns/splitting.yarn +++ b/yarns/splitting.yarn @@ -24,6 +24,7 @@ result, so much as it succeeding to build something. WHEN the user checks out the system branch called master THEN morph build the system systems/test-system.morph of the branch master + FINALLY the git server is shut down Smaller systems --------------- @@ -82,6 +83,7 @@ chunks that were needed. AND system systems/test-system.morph uses test-stratum-minimal from test-stratum WHEN the user checks out the system branch called master THEN morph build the system systems/test-system.morph of the branch master + FINALLY the git server is shut down Implementations -- cgit v1.2.1