diff options
Diffstat (limited to 'hack')
39 files changed, 350 insertions, 1255 deletions
diff --git a/hack/PACKAGERS.md b/hack/PACKAGERS.md index 5dcb120689..7170c5ad25 100644 --- a/hack/PACKAGERS.md +++ b/hack/PACKAGERS.md @@ -157,6 +157,33 @@ AppArmor, you will need to set `DOCKER_BUILDTAGS` as follows: export DOCKER_BUILDTAGS='apparmor' ``` +There are build tags for disabling graphdrivers as well. By default, support +for all graphdrivers are built in. + +To disable btrfs: +```bash +export DOCKER_BUILDTAGS='exclude_graphdriver_btrfs' +``` + +To disable devicemapper: +```bash +export DOCKER_BUILDTAGS='exclude_graphdriver_devicemapper' +``` + +To disable aufs: +```bash +export DOCKER_BUILDTAGS='exclude_graphdriver_aufs' +``` + +NOTE: if you need to set more than one build tag, space separate them. + +If you're building a binary that may need to be used on platforms that include +SELinux, you will need to set `DOCKER_BUILDTAGS` as follows: + +```bash +export DOCKER_BUILDTAGS='selinux' +``` + ### Static Daemon If it is feasible within the constraints of your distribution, you should @@ -239,6 +266,12 @@ installed and available at runtime: * iptables version 1.4 or later * XZ Utils version 4.9 or later +* a [properly + mounted](https://github.com/tianon/cgroupfs-mount/blob/master/cgroupfs-mount) + cgroupfs hierarchy (having a single, all-encompassing "cgroup" mount point + [is](https://github.com/dotcloud/docker/issues/2683) + [not](https://github.com/dotcloud/docker/issues/3485) + [sufficient](https://github.com/dotcloud/docker/issues/4568)) Additionally, the Docker client needs the following software to be installed and available at runtime: diff --git a/hack/RELEASE-CHECKLIST.md b/hack/RELEASE-CHECKLIST.md index 84a0ff70e1..31f0adb757 100644 --- a/hack/RELEASE-CHECKLIST.md +++ b/hack/RELEASE-CHECKLIST.md @@ -6,6 +6,21 @@ So you're in charge of a Docker release? Cool. Here's what to do. If your experience deviates from this document, please document the changes to keep it up-to-date. +It is important to note that this document assumes that the git remote in your +repository that corresponds to "https://github.com/dotcloud/docker" is named +"origin". If yours is not (for example, if you've chosen to name it "upstream" +or something similar instead), be sure to adjust the listed snippets for your +local environment accordingly. If you are not sure what your upstream remote is +named, use a command like `git remote -v` to find out. + +If you don't have an upstream remote, you can add one easily using something +like: + +```bash +git remote add origin https://github.com/dotcloud/docker.git +git remote add YOURUSER git@github.com:YOURUSER/docker.git +``` + ### 1. Pull from master and create a release branch ```bash @@ -124,7 +139,7 @@ docker run \ -e AWS_ACCESS_KEY \ -e AWS_SECRET_KEY \ -e GPG_PASSPHRASE \ - -i -t -privileged \ + -i -t --privileged \ docker \ hack/release.sh ``` @@ -158,12 +173,28 @@ docker run \ -e AWS_ACCESS_KEY \ -e AWS_SECRET_KEY \ -e GPG_PASSPHRASE \ - -i -t -privileged \ + -i -t --privileged \ docker \ hack/release.sh ``` -### 9. Apply tag +### 9. Breakathon + +Spend several days along with the community explicitly investing time and +resources to try and break Docker in every possible way, documenting any +findings pertinent to the release. This time should be spent testing and +finding ways in which the release might have caused various features or upgrade +environments to have issues, not coding. During this time, the release is in +code freeze, and any additional code changes will be pushed out to the next +release. + +It should include various levels of breaking Docker, beyond just using Docker +by the book. + +Any issues found may still remain issues for this release, but they should be +documented and give appropriate warnings. + +### 10. Apply tag ```bash git tag -a $VERSION -m $VERSION bump_$VERSION @@ -173,22 +204,26 @@ git push origin $VERSION It's very important that we don't make the tag until after the official release is uploaded to get.docker.io! -### 10. Go to github to merge the `bump_$VERSION` branch into release +### 11. Go to github to merge the `bump_$VERSION` branch into release -Don't delete the leftover branch just yet, as we will need it for the next step. +Don't forget to push that pretty blue button to delete the leftover +branch afterwards! -### 11. Go to github to merge the `bump_$VERSION` branch into docs +### 12. Update the docs branch -Merging the pull request to the docs branch will automatically -update the documentation on the "latest" revision of the docs. You -should see the updated docs 5-10 minutes after the merge. The docs -will appear on http://docs.docker.io/. For more information about -documentation releases, see `docs/README.md`. +```bash +git checkout docs +git fetch +git reset --hard origin/release +git push -f origin docs +``` -Don't forget to push that pretty blue button to delete the leftover -branch afterwards! +Updating the docs branch will automatically update the documentation on the +"latest" revision of the docs. You should see the updated docs 5-10 minutes +after the merge. The docs will appear on http://docs.docker.io/. For more +information about documentation releases, see `docs/README.md`. -### 12. Create a new pull request to merge release back into master +### 13. Create a new pull request to merge release back into master ```bash git checkout master @@ -206,7 +241,7 @@ echo "https://github.com/dotcloud/docker/compare/master...merge_release_$VERSION Again, get two maintainers to validate, then merge, then push that pretty blue button to delete your branch. -### 13. Rejoice and Evangelize! +### 14. Rejoice and Evangelize! Congratulations! You're done. @@ -5,7 +5,7 @@ # See the blog post: http://blog.docker.io/2013/09/docker-can-now-run-within-docker/ # # This script should be executed inside a docker container in privilieged mode -# ('docker run -privileged', introduced in docker 0.6). +# ('docker run --privileged', introduced in docker 0.6). # Usage: dind CMD [ARG...] @@ -17,7 +17,7 @@ CGROUP=/sys/fs/cgroup mountpoint -q $CGROUP || mount -n -t tmpfs -o uid=0,gid=0,mode=0755 cgroup $CGROUP || { - echo "Could not make a tmpfs mount. Did you use -privileged?" + echo "Could not make a tmpfs mount. Did you use --privileged?" exit 1 } diff --git a/hack/infrastructure/docker-ci/Dockerfile b/hack/infrastructure/docker-ci/Dockerfile deleted file mode 100644 index fd795f4d45..0000000000 --- a/hack/infrastructure/docker-ci/Dockerfile +++ /dev/null @@ -1,29 +0,0 @@ -# DOCKER-VERSION: 0.7.6 -# AUTHOR: Daniel Mizyrycki <daniel@dotcloud.com> -# DESCRIPTION: docker-ci continuous integration service -# TO_BUILD: docker build -rm -t docker-ci/docker-ci . -# TO_RUN: docker run -rm -i -t -p 8000:80 -p 2222:22 -v /run:/var/socket \ -# -v /data/docker-ci:/data/docker-ci docker-ci/docker-ci - -from ubuntu:12.04 -maintainer Daniel Mizyrycki <daniel@dotcloud.com> - -ENV DEBIAN_FRONTEND noninteractive -RUN echo 'deb http://archive.ubuntu.com/ubuntu precise main universe' > \ - /etc/apt/sources.list; apt-get update -RUN apt-get install -y --no-install-recommends python2.7 python-dev \ - libevent-dev git supervisor ssh rsync less vim sudo gcc wget nginx -RUN cd /tmp; wget http://python-distribute.org/distribute_setup.py -RUN cd /tmp; python distribute_setup.py; easy_install pip; rm distribute_setup.py - -RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 -RUN echo 'deb http://get.docker.io/ubuntu docker main' > \ - /etc/apt/sources.list.d/docker.list; apt-get update -RUN apt-get install -y lxc-docker-0.8.0 -RUN pip install SQLAlchemy==0.7.10 buildbot buildbot-slave pyopenssl boto -RUN ln -s /var/socket/docker.sock /run/docker.sock - -ADD . /docker-ci -RUN /docker-ci/setup.sh - -ENTRYPOINT ["supervisord", "-n"] diff --git a/hack/infrastructure/docker-ci/MAINTAINERS b/hack/infrastructure/docker-ci/MAINTAINERS deleted file mode 100644 index 5dfc881420..0000000000 --- a/hack/infrastructure/docker-ci/MAINTAINERS +++ /dev/null @@ -1 +0,0 @@ -Daniel Mizyrycki <daniel@dotcloud.com> (@mzdaniel) diff --git a/hack/infrastructure/docker-ci/README.rst b/hack/infrastructure/docker-ci/README.rst deleted file mode 100644 index 3e429ffdd5..0000000000 --- a/hack/infrastructure/docker-ci/README.rst +++ /dev/null @@ -1,65 +0,0 @@ -========= -docker-ci -========= - -This directory contains docker-ci continuous integration system. -As expected, it is a fully dockerized and deployed using -docker-container-runner. -docker-ci is based on Buildbot, a continuous integration system designed -to automate the build/test cycle. By automatically rebuilding and testing -the tree each time something has changed, build problems are pinpointed -quickly, before other developers are inconvenienced by the failure. -We are running buildbot at Rackspace to verify docker and docker-registry -pass tests, and check for coverage code details. - -docker-ci instance is at https://docker-ci.docker.io/waterfall - -Inside docker-ci container we have the following directory structure: - -/docker-ci source code of docker-ci -/data/backup/docker-ci/ daily backup (replicated over S3) -/data/docker-ci/coverage/{docker,docker-registry}/ mapped to host volumes -/data/buildbot/{master,slave}/ main docker-ci buildbot config and database -/var/socket/{docker.sock} host volume access to docker socket - - -Production deployment -===================== - -:: - - # Clone docker-ci repository - git clone https://github.com/dotcloud/docker - cd docker/hack/infrastructure/docker-ci - - export DOCKER_PROD=[PRODUCTION_SERVER_IP] - - # Create data host volume. (only once) - docker -H $DOCKER_PROD run -v /home:/data ubuntu:12.04 \ - mkdir -p /data/docker-ci/coverage/docker - docker -H $DOCKER_PROD run -v /home:/data ubuntu:12.04 \ - mkdir -p /data/docker-ci/coverage/docker-registry - docker -H $DOCKER_PROD run -v /home:/data ubuntu:12.04 \ - chown -R 1000.1000 /data/docker-ci - - # dcr deployment. Define credentials and special environment dcr variables - # ( retrieved at /hack/infrastructure/docker-ci/dcr/prod/docker-ci.yml ) - export WEB_USER=[DOCKER-CI-WEBSITE-USERNAME] - export WEB_IRC_PWD=[DOCKER-CI-WEBSITE-PASSWORD] - export BUILDBOT_PWD=[BUILDSLAVE_PASSWORD] - export AWS_ACCESS_KEY=[DOCKER_RELEASE_S3_ACCESS] - export AWS_SECRET_KEY=[DOCKER_RELEASE_S3_SECRET] - export GPG_PASSPHRASE=[DOCKER_RELEASE_PASSPHRASE] - export BACKUP_AWS_ID=[S3_BUCKET_CREDENTIAL_ACCESS] - export BACKUP_AWS_SECRET=[S3_BUCKET_CREDENTIAL_SECRET] - export SMTP_USER=[MAILGUN_SMTP_USERNAME] - export SMTP_PWD=[MAILGUN_SMTP_PASSWORD] - export EMAIL_RCP=[EMAIL_FOR_BUILD_ERRORS] - - # Build docker-ci and testbuilder docker images - docker -H $DOCKER_PROD build -rm -t docker-ci/docker-ci . - (cd testbuilder; docker -H $DOCKER_PROD build -rm -t docker-ci/testbuilder .) - - # Run docker-ci container ( assuming no previous container running ) - (cd dcr/prod; dcr docker-ci.yml start) - (cd dcr/prod; dcr docker-ci.yml register docker-ci.docker.io) diff --git a/hack/infrastructure/docker-ci/VERSION b/hack/infrastructure/docker-ci/VERSION deleted file mode 100644 index b49b25336d..0000000000 --- a/hack/infrastructure/docker-ci/VERSION +++ /dev/null @@ -1 +0,0 @@ -0.5.6 diff --git a/hack/infrastructure/docker-ci/buildbot/github.py b/hack/infrastructure/docker-ci/buildbot/github.py deleted file mode 100644 index 5316e13282..0000000000 --- a/hack/infrastructure/docker-ci/buildbot/github.py +++ /dev/null @@ -1,176 +0,0 @@ -# This file is part of Buildbot. Buildbot is free software: you can -# redistribute it and/or modify it under the terms of the GNU General Public -# License as published by the Free Software Foundation, version 2. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS -# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more -# details. -# -# You should have received a copy of the GNU General Public License along with -# this program; if not, write to the Free Software Foundation, Inc., 51 -# Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -# -# Copyright Buildbot Team Members - -#!/usr/bin/env python -""" -github_buildbot.py is based on git_buildbot.py - -github_buildbot.py will determine the repository information from the JSON -HTTP POST it receives from github.com and build the appropriate repository. -If your github repository is private, you must add a ssh key to the github -repository for the user who initiated the build on the buildslave. - -""" - -import re -import datetime -from twisted.python import log -import calendar - -try: - import json - assert json -except ImportError: - import simplejson as json - -# python is silly about how it handles timezones -class fixedOffset(datetime.tzinfo): - """ - fixed offset timezone - """ - def __init__(self, minutes, hours, offsetSign = 1): - self.minutes = int(minutes) * offsetSign - self.hours = int(hours) * offsetSign - self.offset = datetime.timedelta(minutes = self.minutes, - hours = self.hours) - - def utcoffset(self, dt): - return self.offset - - def dst(self, dt): - return datetime.timedelta(0) - -def convertTime(myTestTimestamp): - #"1970-01-01T00:00:00+00:00" - # Normalize myTestTimestamp - if myTestTimestamp[-1] == 'Z': - myTestTimestamp = myTestTimestamp[:-1] + '-00:00' - matcher = re.compile(r'(\d\d\d\d)-(\d\d)-(\d\d)T(\d\d):(\d\d):(\d\d)([-+])(\d\d):(\d\d)') - result = matcher.match(myTestTimestamp) - (year, month, day, hour, minute, second, offsetsign, houroffset, minoffset) = \ - result.groups() - if offsetsign == '+': - offsetsign = 1 - else: - offsetsign = -1 - - offsetTimezone = fixedOffset( minoffset, houroffset, offsetsign ) - myDatetime = datetime.datetime( int(year), - int(month), - int(day), - int(hour), - int(minute), - int(second), - 0, - offsetTimezone) - return calendar.timegm( myDatetime.utctimetuple() ) - -def getChanges(request, options = None): - """ - Reponds only to POST events and starts the build process - - :arguments: - request - the http request object - """ - payload = json.loads(request.args['payload'][0]) - import urllib,datetime - fname = str(datetime.datetime.now()).replace(' ','_').replace(':','-')[:19] - # Github event debug - # open('github_{0}.json'.format(fname),'w').write(json.dumps(json.loads(urllib.unquote(request.args['payload'][0])), sort_keys = True, indent = 2)) - - if 'pull_request' in payload: - user = payload['pull_request']['user']['login'] - repo = payload['pull_request']['head']['repo']['name'] - repo_url = payload['pull_request']['head']['repo']['html_url'] - else: - user = payload['repository']['owner']['name'] - repo = payload['repository']['name'] - repo_url = payload['repository']['url'] - project = request.args.get('project', None) - if project: - project = project[0] - elif project is None: - project = '' - # This field is unused: - #private = payload['repository']['private'] - changes = process_change(payload, user, repo, repo_url, project) - log.msg("Received %s changes from github" % len(changes)) - return (changes, 'git') - -def process_change(payload, user, repo, repo_url, project): - """ - Consumes the JSON as a python object and actually starts the build. - - :arguments: - payload - Python Object that represents the JSON sent by GitHub Service - Hook. - """ - changes = [] - - newrev = payload['after'] if 'after' in payload else payload['pull_request']['head']['sha'] - refname = payload['ref'] if 'ref' in payload else payload['pull_request']['head']['ref'] - - # We only care about regular heads, i.e. branches - match = re.match(r"^(refs\/heads\/|)([^/]+)$", refname) - if not match: - log.msg("Ignoring refname `%s': Not a branch" % refname) - return [] - - branch = match.groups()[1] - if re.match(r"^0*$", newrev): - log.msg("Branch `%s' deleted, ignoring" % branch) - return [] - else: - if 'pull_request' in payload: - if payload['action'] == 'closed': - log.msg("PR#{} closed, ignoring".format(payload['number'])) - return [] - changes = [{ - 'category' : 'github_pullrequest', - 'who' : '{0} - PR#{1}'.format(user,payload['number']), - 'files' : [], - 'comments' : payload['pull_request']['title'], - 'revision' : newrev, - 'when' : convertTime(payload['pull_request']['updated_at']), - 'branch' : branch, - 'revlink' : '{0}/commit/{1}'.format(repo_url,newrev), - 'repository' : repo_url, - 'project' : project }] - return changes - for commit in payload['commits']: - files = [] - if 'added' in commit: - files.extend(commit['added']) - if 'modified' in commit: - files.extend(commit['modified']) - if 'removed' in commit: - files.extend(commit['removed']) - when = convertTime( commit['timestamp']) - log.msg("New revision: %s" % commit['id'][:8]) - chdict = dict( - who = commit['author']['name'] - + " <" + commit['author']['email'] + ">", - files = files, - comments = commit['message'], - revision = commit['id'], - when = when, - branch = branch, - revlink = commit['url'], - repository = repo_url, - project = project) - changes.append(chdict) - return changes diff --git a/hack/infrastructure/docker-ci/buildbot/master.cfg b/hack/infrastructure/docker-ci/buildbot/master.cfg deleted file mode 100644 index 75605da8ab..0000000000 --- a/hack/infrastructure/docker-ci/buildbot/master.cfg +++ /dev/null @@ -1,161 +0,0 @@ -import os, re -from buildbot.buildslave import BuildSlave -from buildbot.schedulers.forcesched import ForceScheduler -from buildbot.schedulers.basic import SingleBranchScheduler -from buildbot.schedulers.timed import Nightly -from buildbot.changes import filter -from buildbot.config import BuilderConfig -from buildbot.process.factory import BuildFactory -from buildbot.process.properties import Property -from buildbot.steps.shell import ShellCommand -from buildbot.status import html, words -from buildbot.status.web import authz, auth -from buildbot.status.mail import MailNotifier - - -def ENV(x): - '''Promote an environment variable for global use returning its value''' - retval = os.environ.get(x, '') - globals()[x] = retval - return retval - - -class TestCommand(ShellCommand): - '''Extend ShellCommand with optional summary logs''' - def __init__(self, *args, **kwargs): - super(TestCommand, self).__init__(*args, **kwargs) - - def createSummary(self, log): - exit_status = re.sub(r'.+\n\+ exit (\d+).+', - r'\1', log.getText()[-100:], flags=re.DOTALL) - if exit_status != '0': - return - # Infer coverage path from log - if '+ COVERAGE_PATH' in log.getText(): - path = re.sub(r'.+\+ COVERAGE_PATH=((.+?)-\d+).+', - r'\2/\1', log.getText(), flags=re.DOTALL) - url = '{}coverage/{}/index.html'.format(c['buildbotURL'], path) - self.addURL('coverage', url) - elif 'COVERAGE_FILE' in log.getText(): - path = re.sub(r'.+\+ COVERAGE_FILE=((.+?)-\d+).+', - r'\2/\1', log.getText(), flags=re.DOTALL) - url = '{}coverage/{}/index.html'.format(c['buildbotURL'], path) - self.addURL('coverage', url) - - -PORT_WEB = 8000 # Buildbot webserver port -PORT_GITHUB = 8011 # Buildbot github hook port -PORT_MASTER = 9989 # Port where buildbot master listen buildworkers - -BUILDBOT_URL = '//localhost:{}/'.format(PORT_WEB) -DOCKER_REPO = 'https://github.com/docker-test/docker' -DOCKER_TEST_ARGV = 'HEAD {}'.format(DOCKER_REPO) -REGISTRY_REPO = 'https://github.com/docker-test/docker-registry' -REGISTRY_TEST_ARGV = 'HEAD {}'.format(REGISTRY_REPO) -if ENV('DEPLOYMENT') == 'staging': - BUILDBOT_URL = "//docker-ci-stage.docker.io/" -if ENV('DEPLOYMENT') == 'production': - BUILDBOT_URL = '//docker-ci.docker.io/' - DOCKER_REPO = 'https://github.com/dotcloud/docker' - DOCKER_TEST_ARGV = '' - REGISTRY_REPO = 'https://github.com/dotcloud/docker-registry' - REGISTRY_TEST_ARGV = '' - -# Credentials set by setup.sh from deployment.py -ENV('WEB_USER') -ENV('WEB_IRC_PWD') -ENV('BUILDBOT_PWD') -ENV('SMTP_USER') -ENV('SMTP_PWD') -ENV('EMAIL_RCP') -ENV('IRC_CHANNEL') - - -c = BuildmasterConfig = {} - -c['title'] = "docker-ci" -c['titleURL'] = "waterfall" -c['buildbotURL'] = BUILDBOT_URL -c['db'] = {'db_url':"sqlite:///state.sqlite"} -c['slaves'] = [BuildSlave('buildworker', BUILDBOT_PWD)] -c['slavePortnum'] = PORT_MASTER - - -# Schedulers -c['schedulers'] = [ForceScheduler(name='trigger', builderNames=[ - 'docker', 'docker-registry', 'nightlyrelease', 'backup'])] -c['schedulers'] += [SingleBranchScheduler(name="docker", treeStableTimer=None, - change_filter=filter.ChangeFilter(branch='master', - repository=DOCKER_REPO), builderNames=['docker'])] -c['schedulers'] += [SingleBranchScheduler(name="registry", treeStableTimer=None, - change_filter=filter.ChangeFilter(branch='master', - repository=REGISTRY_REPO), builderNames=['docker-registry'])] -c['schedulers'] += [SingleBranchScheduler(name='docker-pr', treeStableTimer=None, - change_filter=filter.ChangeFilter(category='github_pullrequest', - project='docker'), builderNames=['docker-pr'])] -c['schedulers'] += [SingleBranchScheduler(name='docker-registry-pr', treeStableTimer=None, - change_filter=filter.ChangeFilter(category='github_pullrequest', - project='docker-registry'), builderNames=['docker-registry-pr'])] -c['schedulers'] += [Nightly(name='daily', branch=None, builderNames=[ - 'nightlyrelease', 'backup'], hour=7, minute=00)] - - -# Builders - -# Backup -factory = BuildFactory() -factory.addStep(TestCommand(description='backup', logEnviron=False, - usePTY=True, command='/docker-ci/tool/backup.py')) -c['builders'] = [BuilderConfig(name='backup',slavenames=['buildworker'], - factory=factory)] - -# Docker test -factory = BuildFactory() -factory.addStep(TestCommand(description='docker', logEnviron=False, - usePTY=True, command='/docker-ci/dockertest/docker {}'.format(DOCKER_TEST_ARGV))) -c['builders'] += [BuilderConfig(name='docker',slavenames=['buildworker'], - factory=factory)] - -# Docker pull request test -factory = BuildFactory() -factory.addStep(TestCommand(description='docker-pr', logEnviron=False, - usePTY=True, command=['/docker-ci/dockertest/docker', - Property('revision'), Property('repository'), Property('branch')])) -c['builders'] += [BuilderConfig(name='docker-pr',slavenames=['buildworker'], - factory=factory)] - -# docker-registry test -factory = BuildFactory() -factory.addStep(TestCommand(description='docker-registry', logEnviron=False, - usePTY=True, command='/docker-ci/dockertest/docker-registry {}'.format(REGISTRY_TEST_ARGV))) -c['builders'] += [BuilderConfig(name='docker-registry',slavenames=['buildworker'], - factory=factory)] - -# Docker registry pull request test -factory = BuildFactory() -factory.addStep(TestCommand(description='docker-registry-pr', logEnviron=False, - usePTY=True, command=['/docker-ci/dockertest/docker-registry', - Property('revision'), Property('repository'), Property('branch')])) -c['builders'] += [BuilderConfig(name='docker-registry-pr',slavenames=['buildworker'], - factory=factory)] - -# Docker nightly release -factory = BuildFactory() -factory.addStep(ShellCommand(description='NightlyRelease',logEnviron=False, - usePTY=True, command=['/docker-ci/dockertest/nightlyrelease'])) -c['builders'] += [BuilderConfig(name='nightlyrelease',slavenames=['buildworker'], - factory=factory)] - -# Status -authz_cfg = authz.Authz(auth=auth.BasicAuth([(WEB_USER, WEB_IRC_PWD)]), - forceBuild='auth') -c['status'] = [html.WebStatus(http_port=PORT_WEB, authz=authz_cfg)] -c['status'].append(html.WebStatus(http_port=PORT_GITHUB, allowForce=True, - change_hook_dialects={ 'github': True })) -c['status'].append(MailNotifier(fromaddr='docker-test@docker.io', - sendToInterestedUsers=False, extraRecipients=[EMAIL_RCP], - mode='failing', relayhost='smtp.mailgun.org', smtpPort=587, useTls=True, - smtpUser=SMTP_USER, smtpPassword=SMTP_PWD)) -c['status'].append(words.IRC("irc.freenode.net", "dockerqabot", - channels=[IRC_CHANNEL], password=WEB_IRC_PWD, allowForce=True, - notify_events={'exception':1, 'successToFailure':1, 'failureToSuccess':1})) diff --git a/hack/infrastructure/docker-ci/dcr/prod/docker-ci.yml b/hack/infrastructure/docker-ci/dcr/prod/docker-ci.yml deleted file mode 100644 index 523535446a..0000000000 --- a/hack/infrastructure/docker-ci/dcr/prod/docker-ci.yml +++ /dev/null @@ -1,22 +0,0 @@ -docker-ci: - image: "docker-ci/docker-ci" - release_name: "docker-ci-0.5.6" - ports: ["80","2222:22","8011:8011"] - register: "80" - volumes: ["/run:/var/socket","/home/docker-ci:/data/docker-ci"] - command: [] - env: - - "DEPLOYMENT=production" - - "IRC_CHANNEL=docker-testing" - - "BACKUP_BUCKET=backup-ci" - - "$WEB_USER" - - "$WEB_IRC_PWD" - - "$BUILDBOT_PWD" - - "$AWS_ACCESS_KEY" - - "$AWS_SECRET_KEY" - - "$GPG_PASSPHRASE" - - "$BACKUP_AWS_ID" - - "$BACKUP_AWS_SECRET" - - "$SMTP_USER" - - "$SMTP_PWD" - - "$EMAIL_RCP" diff --git a/hack/infrastructure/docker-ci/dcr/prod/settings.yml b/hack/infrastructure/docker-ci/dcr/prod/settings.yml deleted file mode 100644 index 9831afa6dd..0000000000 --- a/hack/infrastructure/docker-ci/dcr/prod/settings.yml +++ /dev/null @@ -1,5 +0,0 @@ -default: - hipaches: ['192.168.100.67:6379'] - daemons: ['192.168.100.67:4243'] - use_ssh: False - diff --git a/hack/infrastructure/docker-ci/dcr/stage/docker-ci.yml b/hack/infrastructure/docker-ci/dcr/stage/docker-ci.yml deleted file mode 100644 index 8eba84825c..0000000000 --- a/hack/infrastructure/docker-ci/dcr/stage/docker-ci.yml +++ /dev/null @@ -1,22 +0,0 @@ -docker-ci: - image: "docker-ci/docker-ci" - release_name: "docker-ci-stage" - ports: ["80","2222:22","8011:8011"] - register: "80" - volumes: ["/run:/var/socket","/home/docker-ci:/data/docker-ci"] - command: [] - env: - - "DEPLOYMENT=staging" - - "IRC_CHANNEL=docker-testing-staging" - - "BACKUP_BUCKET=ci-backup-stage" - - "$BACKUP_AWS_ID" - - "$BACKUP_AWS_SECRET" - - "$WEB_USER" - - "$WEB_IRC_PWD" - - "$BUILDBOT_PWD" - - "$AWS_ACCESS_KEY" - - "$AWS_SECRET_KEY" - - "$GPG_PASSPHRASE" - - "$SMTP_USER" - - "$SMTP_PWD" - - "$EMAIL_RCP" diff --git a/hack/infrastructure/docker-ci/dcr/stage/settings.yml b/hack/infrastructure/docker-ci/dcr/stage/settings.yml deleted file mode 100644 index a7d37acff3..0000000000 --- a/hack/infrastructure/docker-ci/dcr/stage/settings.yml +++ /dev/null @@ -1,5 +0,0 @@ -default: - hipaches: ['192.168.100.65:6379'] - daemons: ['192.168.100.65:4243'] - use_ssh: False - diff --git a/hack/infrastructure/docker-ci/docker-coverage/gocoverage.sh b/hack/infrastructure/docker-ci/docker-coverage/gocoverage.sh deleted file mode 100755 index fdacc290b4..0000000000 --- a/hack/infrastructure/docker-ci/docker-coverage/gocoverage.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/bin/bash - -export PATH='/go/bin':$PATH -export DOCKER_PATH='/go/src/github.com/dotcloud/docker' - -# Signal coverage report name, parsed by docker-ci -set -x -COVERAGE_PATH=$(date +"docker-%Y%m%d%H%M%S") -set +x - -REPORTS="/data/$COVERAGE_PATH" -INDEX="$REPORTS/index.html" - -# Test docker -cd $DOCKER_PATH -./hack/make.sh test; exit_status=$? -PROFILE_PATH="$(ls -d $DOCKER_PATH/bundles/* | sed -n '$ p')/test/coverprofiles" - -if [ "$exit_status" -eq "0" ]; then - # Download coverage dependencies - go get github.com/axw/gocov/gocov - go get -u github.com/matm/gocov-html - - # Create coverage report - mkdir -p $REPORTS - cd $PROFILE_PATH - cat > $INDEX << "EOF" -<!DOCTYPE html><head><meta charset="utf-8"> -<script type="text/javascript" src="//tablesorter.com/jquery-latest.js"></script> -<script type="text/javascript" src="//tablesorter.com/__jquery.tablesorter.min.js"></script> -<script type="text/javascript">$(document).ready(function() { -$("table").tablesorter({ sortForce: [[1,0]] }); });</script> -<style>table,th,td{border:1px solid black;}</style> -<title>Docker Coverage Report</title> -</head><body> -<h1><strong>Docker Coverage Report</strong></h1> -<table class="tablesorter"> -<thead><tr><th>package</th><th>pct</th></tr></thead><tbody> -EOF - for profile in *; do - gocov convert $profile | gocov-html >$REPORTS/$profile.html - echo "<tr><td><a href=\"${profile}.html\">$profile</a></td><td>" >> $INDEX - go tool cover -func=$profile | sed -En '$ s/.+\t(.+)/\1/p' >> $INDEX - echo "</td></tr>" >> $INDEX - done - echo "</tbody></table></body></html>" >> $INDEX -fi - -# Signal test and coverage result, parsed by docker-ci -set -x -exit $exit_status - diff --git a/hack/infrastructure/docker-ci/dockertest/docker b/hack/infrastructure/docker-ci/dockertest/docker deleted file mode 120000 index e3f094ee63..0000000000 --- a/hack/infrastructure/docker-ci/dockertest/docker +++ /dev/null @@ -1 +0,0 @@ -project
\ No newline at end of file diff --git a/hack/infrastructure/docker-ci/dockertest/docker-registry b/hack/infrastructure/docker-ci/dockertest/docker-registry deleted file mode 120000 index e3f094ee63..0000000000 --- a/hack/infrastructure/docker-ci/dockertest/docker-registry +++ /dev/null @@ -1 +0,0 @@ -project
\ No newline at end of file diff --git a/hack/infrastructure/docker-ci/dockertest/nightlyrelease b/hack/infrastructure/docker-ci/dockertest/nightlyrelease deleted file mode 100755 index 475b088065..0000000000 --- a/hack/infrastructure/docker-ci/dockertest/nightlyrelease +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env bash - -if [ "$DEPLOYMENT" == "production" ]; then - AWS_S3_BUCKET='test.docker.io' -else - AWS_S3_BUCKET='get-staging.docker.io' -fi - -docker run -rm -privileged -v /run:/var/socket \ - -e AWS_S3_BUCKET=$AWS_S3_BUCKET -e AWS_ACCESS_KEY=$AWS_ACCESS_KEY \ - -e AWS_SECRET_KEY=$AWS_SECRET_KEY -e GPG_PASSPHRASE=$GPG_PASSPHRASE \ - -e DOCKER_RELEASE=1 -e DEPLOYMENT=$DEPLOYMENT docker-ci/testbuilder docker - diff --git a/hack/infrastructure/docker-ci/dockertest/project b/hack/infrastructure/docker-ci/dockertest/project deleted file mode 100755 index 160f2d5d59..0000000000 --- a/hack/infrastructure/docker-ci/dockertest/project +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env bash -set -x - -PROJECT_NAME=$(basename $0) - -docker run -rm -u sysadmin -e DEPLOYMENT=$DEPLOYMENT -v /run:/var/socket \ - -v /home/docker-ci/coverage/$PROJECT_NAME:/data docker-ci/testbuilder $PROJECT_NAME $1 $2 $3 - diff --git a/hack/infrastructure/docker-ci/functionaltests/test_index.py b/hack/infrastructure/docker-ci/functionaltests/test_index.py deleted file mode 100755 index fd002c81e8..0000000000 --- a/hack/infrastructure/docker-ci/functionaltests/test_index.py +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/python - -import os -username, password = os.environ['DOCKER_CREDS'].split(':') - -from selenium import webdriver -from selenium.webdriver.common.by import By -from selenium.webdriver.common.keys import Keys -from selenium.webdriver.support.ui import Select -from selenium.common.exceptions import NoSuchElementException -import unittest, time, re - -class Docker(unittest.TestCase): - def setUp(self): - self.driver = webdriver.PhantomJS() - self.driver.implicitly_wait(30) - self.base_url = "http://www.docker.io/" - self.verificationErrors = [] - self.accept_next_alert = True - - def test_docker(self): - driver = self.driver - print "Login into {0} as login user {1} ...".format(self.base_url,username) - driver.get(self.base_url + "/") - driver.find_element_by_link_text("INDEX").click() - driver.find_element_by_link_text("login").click() - driver.find_element_by_id("id_username").send_keys(username) - driver.find_element_by_id("id_password").send_keys(password) - print "Checking login user ..." - driver.find_element_by_css_selector("input[type=\"submit\"]").click() - try: self.assertEqual("test", driver.find_element_by_css_selector("h3").text) - except AssertionError as e: self.verificationErrors.append(str(e)) - print "Login user {0} found".format(username) - - def is_element_present(self, how, what): - try: self.driver.find_element(by=how, value=what) - except NoSuchElementException, e: return False - return True - - def is_alert_present(self): - try: self.driver.switch_to_alert() - except NoAlertPresentException, e: return False - return True - - def close_alert_and_get_its_text(self): - try: - alert = self.driver.switch_to_alert() - alert_text = alert.text - if self.accept_next_alert: - alert.accept() - else: - alert.dismiss() - return alert_text - finally: self.accept_next_alert = True - - def tearDown(self): - self.driver.quit() - self.assertEqual([], self.verificationErrors) - -if __name__ == "__main__": - unittest.main() diff --git a/hack/infrastructure/docker-ci/functionaltests/test_registry.sh b/hack/infrastructure/docker-ci/functionaltests/test_registry.sh deleted file mode 100755 index 58642529cc..0000000000 --- a/hack/infrastructure/docker-ci/functionaltests/test_registry.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/bin/sh - -set -x - -# Cleanup -rm -rf docker-registry - -# Setup the environment -export SETTINGS_FLAVOR=test -export DOCKER_REGISTRY_CONFIG=config_test.yml -export PYTHONPATH=$(pwd)/docker-registry/test - -# Get latest docker registry -git clone -q https://github.com/dotcloud/docker-registry.git -cd docker-registry -sed -Ei "s#(boto_bucket: ).+#\1_env:S3_BUCKET#" config_test.yml - -# Get dependencies -pip install -q -r requirements.txt -pip install -q -r test-requirements.txt -pip install -q tox - -# Run registry tests -tox || exit 1 -python -m unittest discover -p s3.py -s test || exit 1 -python -m unittest discover -p workflow.py -s test - diff --git a/hack/infrastructure/docker-ci/nginx/nginx.conf b/hack/infrastructure/docker-ci/nginx/nginx.conf deleted file mode 100644 index 6649741134..0000000000 --- a/hack/infrastructure/docker-ci/nginx/nginx.conf +++ /dev/null @@ -1,12 +0,0 @@ -server { - listen 80; - root /data/docker-ci; - - location / { - proxy_pass http://localhost:8000/; - } - - location /coverage { - root /data/docker-ci; - } -} diff --git a/hack/infrastructure/docker-ci/report/Dockerfile b/hack/infrastructure/docker-ci/report/Dockerfile deleted file mode 100644 index 32600c4c58..0000000000 --- a/hack/infrastructure/docker-ci/report/Dockerfile +++ /dev/null @@ -1,28 +0,0 @@ -# VERSION: 0.22 -# DOCKER-VERSION 0.6.3 -# AUTHOR: Daniel Mizyrycki <daniel@dotcloud.com> -# DESCRIPTION: Generate docker-ci daily report -# COMMENTS: The build process is initiated by deployment.py - Report configuration is passed through ./credentials.json at -# deployment time. -# TO_BUILD: docker build -t report . -# TO_DEPLOY: docker run report - -from ubuntu:12.04 -maintainer Daniel Mizyrycki <daniel@dotcloud.com> - -env PYTHONPATH /report - - -# Add report dependencies -run echo 'deb http://archive.ubuntu.com/ubuntu precise main universe' > \ - /etc/apt/sources.list -run apt-get update; apt-get install -y python2.7 python-pip ssh rsync - -# Set San Francisco timezone -run echo "America/Los_Angeles" >/etc/timezone -run dpkg-reconfigure --frontend noninteractive tzdata - -# Add report code and set default container command -add . /report -cmd "/report/report.py" diff --git a/hack/infrastructure/docker-ci/report/deployment.py b/hack/infrastructure/docker-ci/report/deployment.py deleted file mode 100755 index 5b2eaf3cab..0000000000 --- a/hack/infrastructure/docker-ci/report/deployment.py +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env python - -'''Deploy docker-ci report container on Digital Ocean. -Usage: - export CONFIG_JSON=' - { "DROPLET_NAME": "Digital_Ocean_dropplet_name", - "DO_CLIENT_ID": "Digital_Ocean_client_id", - "DO_API_KEY": "Digital_Ocean_api_key", - "DOCKER_KEY_ID": "Digital_Ocean_ssh_key_id", - "DOCKER_CI_KEY_PATH": "docker-ci_private_key_path", - "DOCKER_CI_PUB": "$(cat docker-ci_ssh_public_key.pub)", - "DOCKER_CI_ADDRESS" "user@docker-ci_fqdn_server", - "SMTP_USER": "SMTP_server_user", - "SMTP_PWD": "SMTP_server_password", - "EMAIL_SENDER": "Buildbot_mailing_sender", - "EMAIL_RCP": "Buildbot_mailing_receipient" }' - python deployment.py -''' - -import re, json, requests, base64 -from fabric import api -from fabric.api import cd, run, put, sudo -from os import environ as env -from time import sleep -from datetime import datetime - -# Populate environment variables -CONFIG = json.loads(env['CONFIG_JSON']) -for key in CONFIG: - env[key] = CONFIG[key] - -# Load DOCKER_CI_KEY -env['DOCKER_CI_KEY'] = open(env['DOCKER_CI_KEY_PATH']).read() - -DROPLET_NAME = env.get('DROPLET_NAME','report') -TIMEOUT = 120 # Seconds before timeout droplet creation -IMAGE_ID = 1004145 # Docker on Ubuntu 13.04 -REGION_ID = 4 # New York 2 -SIZE_ID = 66 # memory 512MB -DO_IMAGE_USER = 'root' # Image user on Digital Ocean -API_URL = 'https://api.digitalocean.com/' - - -class digital_ocean(): - - def __init__(self, key, client): - '''Set default API parameters''' - self.key = key - self.client = client - self.api_url = API_URL - - def api(self, cmd_path, api_arg={}): - '''Make api call''' - api_arg.update({'api_key':self.key, 'client_id':self.client}) - resp = requests.get(self.api_url + cmd_path, params=api_arg).text - resp = json.loads(resp) - if resp['status'] != 'OK': - raise Exception(resp['error_message']) - return resp - - def droplet_data(self, name): - '''Get droplet data''' - data = self.api('droplets') - data = [droplet for droplet in data['droplets'] - if droplet['name'] == name] - return data[0] if data else {} - -def json_fmt(data): - '''Format json output''' - return json.dumps(data, sort_keys = True, indent = 2) - - -do = digital_ocean(env['DO_API_KEY'], env['DO_CLIENT_ID']) - -# Get DROPLET_NAME data -data = do.droplet_data(DROPLET_NAME) - -# Stop processing if DROPLET_NAME exists on Digital Ocean -if data: - print ('Droplet: {} already deployed. Not further processing.' - .format(DROPLET_NAME)) - exit(1) - -# Create droplet -do.api('droplets/new', {'name':DROPLET_NAME, 'region_id':REGION_ID, - 'image_id':IMAGE_ID, 'size_id':SIZE_ID, - 'ssh_key_ids':[env['DOCKER_KEY_ID']]}) - -# Wait for droplet to be created. -start_time = datetime.now() -while (data.get('status','') != 'active' and ( - datetime.now()-start_time).seconds < TIMEOUT): - data = do.droplet_data(DROPLET_NAME) - print data['status'] - sleep(3) - -# Wait for the machine to boot -sleep(15) - -# Get droplet IP -ip = str(data['ip_address']) -print 'droplet: {} ip: {}'.format(DROPLET_NAME, ip) - -api.env.host_string = ip -api.env.user = DO_IMAGE_USER -api.env.key_filename = env['DOCKER_CI_KEY_PATH'] - -# Correct timezone -sudo('echo "America/Los_Angeles" >/etc/timezone') -sudo('dpkg-reconfigure --frontend noninteractive tzdata') - -# Load JSON_CONFIG environment for Dockerfile -CONFIG_JSON= base64.b64encode( - '{{"DOCKER_CI_PUB": "{DOCKER_CI_PUB}",' - ' "DOCKER_CI_KEY": "{DOCKER_CI_KEY}",' - ' "DOCKER_CI_ADDRESS": "{DOCKER_CI_ADDRESS}",' - ' "SMTP_USER": "{SMTP_USER}",' - ' "SMTP_PWD": "{SMTP_PWD}",' - ' "EMAIL_SENDER": "{EMAIL_SENDER}",' - ' "EMAIL_RCP": "{EMAIL_RCP}"}}'.format(**env)) - -run('mkdir -p /data/report') -put('./', '/data/report') -with cd('/data/report'): - run('chmod 700 report.py') - run('echo "{}" > credentials.json'.format(CONFIG_JSON)) - run('docker build -t report .') - run('rm credentials.json') - run("echo -e '30 09 * * * /usr/bin/docker run report\n' |" - " /usr/bin/crontab -") diff --git a/hack/infrastructure/docker-ci/report/report.py b/hack/infrastructure/docker-ci/report/report.py deleted file mode 100755 index 7018cabc27..0000000000 --- a/hack/infrastructure/docker-ci/report/report.py +++ /dev/null @@ -1,145 +0,0 @@ -#!/usr/bin/python - -'''CONFIG_JSON is a json encoded string base64 environment variable. It is used -to clone docker-ci database, generate docker-ci report and submit it by email. -CONFIG_JSON data comes from the file /report/credentials.json inserted in this -container by deployment.py: - -{ "DOCKER_CI_PUB": "$(cat docker-ci_ssh_public_key.pub)", - "DOCKER_CI_KEY": "$(cat docker-ci_ssh_private_key.key)", - "DOCKER_CI_ADDRESS": "user@docker-ci_fqdn_server", - "SMTP_USER": "SMTP_server_user", - "SMTP_PWD": "SMTP_server_password", - "EMAIL_SENDER": "Buildbot_mailing_sender", - "EMAIL_RCP": "Buildbot_mailing_receipient" } ''' - -import os, re, json, sqlite3, datetime, base64 -import smtplib -from datetime import timedelta -from subprocess import call -from os import environ as env - -TODAY = datetime.date.today() - -# Load credentials to the environment -env['CONFIG_JSON'] = base64.b64decode(open('/report/credentials.json').read()) - -# Remove SSH private key as it needs more processing -CONFIG = json.loads(re.sub(r'("DOCKER_CI_KEY".+?"(.+?)",)','', - env['CONFIG_JSON'], flags=re.DOTALL)) - -# Populate environment variables -for key in CONFIG: - env[key] = CONFIG[key] - -# Load SSH private key -env['DOCKER_CI_KEY'] = re.sub('^.+"DOCKER_CI_KEY".+?"(.+?)".+','\\1', - env['CONFIG_JSON'],flags=re.DOTALL) - -# Prevent rsync to validate host on first connection to docker-ci -os.makedirs('/root/.ssh') -open('/root/.ssh/id_rsa','w').write(env['DOCKER_CI_KEY']) -os.chmod('/root/.ssh/id_rsa',0600) -open('/root/.ssh/config','w').write('StrictHostKeyChecking no\n') - - -# Sync buildbot database from docker-ci -call('rsync {}:/data/buildbot/master/state.sqlite .'.format( - env['DOCKER_CI_ADDRESS']), shell=True) - -class SQL: - def __init__(self, database_name): - sql = sqlite3.connect(database_name) - # Use column names as keys for fetchall rows - sql.row_factory = sqlite3.Row - sql = sql.cursor() - self.sql = sql - - def query(self,query_statement): - return self.sql.execute(query_statement).fetchall() - -sql = SQL("state.sqlite") - - -class Report(): - - def __init__(self,period='',date=''): - self.data = [] - self.period = 'date' if not period else period - self.date = str(TODAY) if not date else date - self.compute() - - def compute(self): - '''Compute report''' - if self.period == 'week': - self.week_report(self.date) - else: - self.date_report(self.date) - - - def date_report(self,date): - '''Create a date test report''' - builds = [] - # Get a queryset with all builds from date - rows = sql.query('SELECT * FROM builds JOIN buildrequests' - ' WHERE builds.brid=buildrequests.id and' - ' date(start_time, "unixepoch", "localtime") = "{0}"' - ' GROUP BY number'.format(date)) - build_names = sorted(set([row['buildername'] for row in rows])) - # Create a report build line for a given build - for build_name in build_names: - tried = len([row['buildername'] - for row in rows if row['buildername'] == build_name]) - fail_tests = [row['buildername'] for row in rows if ( - row['buildername'] == build_name and row['results'] != 0)] - fail = len(fail_tests) - fail_details = '' - fail_pct = int(100.0*fail/tried) if tried != 0 else 100 - builds.append({'name': build_name, 'tried': tried, 'fail': fail, - 'fail_pct': fail_pct, 'fail_details':fail_details}) - if builds: - self.data.append({'date': date, 'builds': builds}) - - - def week_report(self,date): - '''Add the week's date test reports to report.data''' - date = datetime.datetime.strptime(date,'%Y-%m-%d').date() - last_monday = date - datetime.timedelta(days=date.weekday()) - week_dates = [last_monday + timedelta(days=x) for x in range(7,-1,-1)] - for date in week_dates: - self.date_report(str(date)) - - def render_text(self): - '''Return rendered report in text format''' - retval = '' - fail_tests = {} - for builds in self.data: - retval += 'Test date: {0}\n'.format(builds['date'],retval) - table = '' - for build in builds['builds']: - table += ('Build {name:15} Tried: {tried:4} ' - ' Failures: {fail:4} ({fail_pct}%)\n'.format(**build)) - if build['name'] in fail_tests: - fail_tests[build['name']] += build['fail_details'] - else: - fail_tests[build['name']] = build['fail_details'] - retval += '{0}\n'.format(table) - retval += '\n Builds failing' - for fail_name in fail_tests: - retval += '\n' + fail_name + '\n' - for (fail_id,fail_url,rn_tests,nr_errors,log_errors, - tracelog_errors) in fail_tests[fail_name]: - retval += fail_url + '\n' - retval += '\n\n' - return retval - - -# Send email -smtp_from = env['EMAIL_SENDER'] -subject = '[docker-ci] Daily report for {}'.format(str(TODAY)) -msg = "From: {}\r\nTo: {}\r\nSubject: {}\r\n\r\n".format( - smtp_from, env['EMAIL_RCP'], subject) -msg = msg + Report('week').render_text() -server = smtplib.SMTP_SSL('smtp.mailgun.org') -server.login(env['SMTP_USER'], env['SMTP_PWD']) -server.sendmail(smtp_from, env['EMAIL_RCP'], msg) diff --git a/hack/infrastructure/docker-ci/setup.sh b/hack/infrastructure/docker-ci/setup.sh deleted file mode 100755 index 65a00f6dd0..0000000000 --- a/hack/infrastructure/docker-ci/setup.sh +++ /dev/null @@ -1,54 +0,0 @@ -#!/usr/bin/env bash - -# Set timezone -echo "GMT" >/etc/timezone -dpkg-reconfigure --frontend noninteractive tzdata - -# Set ssh superuser -mkdir -p /data/buildbot /var/run/sshd /run -useradd -m -d /home/sysadmin -s /bin/bash -G sudo,docker -p '*' sysadmin -sed -Ei 's/(\%sudo.*) ALL/\1 NOPASSWD:ALL/' /etc/sudoers -cd /home/sysadmin -mkdir .ssh -chmod 700 .ssh -cat > .ssh/authorized_keys << 'EOF' -ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC7ALVhwQ68q1SjrKaAduOuOEAcWmb8kDZf5qA7T1fM8AP07EDC7nSKRJ8PXUBGTOQfxm89coJDuSJsTAZ+1PvglXhA0Mq6+knc6ZrZY+SuZlDIDAk4TOdVPoDZnmR1YW2McxHkhcGIOKeC8MMig5NeEjtgQwXzauUSPqeh8HMlLZRMooFYyyluIpn7NaCLzyWjwAQz2s3KyI7VE7hl+ncCrW86v+dciEdwqtzNoUMFb3iDpPxaiCl3rv+SB7co/5eUDTs1FZvUcYMXKQuf8R+2ZKzXOpwr0Zs8sKQXvXavCeWykwGgXLBjVkvrDcHuDD6UXCW63UKgmRECpLZaMBVIIRWLEEgTS5OSQTcxpMVe5zUW6sDvXHTcdPwWrcn1dE9F/0vLC0HJ4ADKelLX5zyTpmXGbuZuntIf1JO67D/K/P++uV1rmVIH+zgtOf23w5rX2zKb4BSTqP0sv61pmWV7MEVoEz6yXswcTjS92tb775v7XLU9vKAkt042ORFdE4/++hejhL/Lj52IRgjt1CJZHZsR9JywJZrz3kYuf8eU2J2FYh0Cpz5gmf0f+12Rt4HztnZxGPP4KuMa66e4+hpx1jynjMZ7D5QUnNYEmuvJByopn8HSluuY/kS5MMyZCZtJLEPGX4+yECX0Di/S0vCRl2NyqfCBqS+yXXT5SA1nFw== docker-test@docker.io -EOF -chmod 600 .ssh/authorized_keys -chown -R sysadmin .ssh - -# Fix docker group id for use of host dockerd by sysadmin -sed -Ei 's/(docker:x:)[^:]+/\1999/' /etc/group - -# Create buildbot configuration -cd /data/buildbot; buildbot create-master master -cp -a /data/buildbot/master/master.cfg.sample \ - /data/buildbot/master/master.cfg -cd /data/buildbot; \ - buildslave create-slave slave localhost:9989 buildworker pass -cp /docker-ci/buildbot/master.cfg /data/buildbot/master - -# Patch github webstatus to capture pull requests -cp /docker-ci/buildbot/github.py /usr/local/lib/python2.7/dist-packages/buildbot/status/web/hooks -chown -R sysadmin.sysadmin /data - -# Create nginx configuration -rm /etc/nginx/sites-enabled/default -cp /docker-ci/nginx/nginx.conf /etc/nginx/conf.d/buildbot.conf -/bin/echo -e '\ndaemon off;\n' >> /etc/nginx/nginx.conf - -# Set supervisord buildbot, nginx and sshd processes -/bin/echo -e "\ -[program:buildmaster]\n\ -command=twistd --nodaemon --no_save -y buildbot.tac\n\ -directory=/data/buildbot/master\n\ -user=sysadmin\n\n\ -[program:buildworker]\n\ -command=twistd --nodaemon --no_save -y buildbot.tac\n\ -directory=/data/buildbot/slave\n\ -user=sysadmin\n" > \ - /etc/supervisor/conf.d/buildbot.conf -/bin/echo -e "[program:nginx]\ncommand=/usr/sbin/nginx\n" > \ - /etc/supervisor/conf.d/nginx.conf -/bin/echo -e "[program:sshd]\ncommand=/usr/sbin/sshd -D\n" > \ - /etc/supervisor/conf.d/sshd.conf diff --git a/hack/infrastructure/docker-ci/testbuilder/Dockerfile b/hack/infrastructure/docker-ci/testbuilder/Dockerfile deleted file mode 100644 index a008da6843..0000000000 --- a/hack/infrastructure/docker-ci/testbuilder/Dockerfile +++ /dev/null @@ -1,12 +0,0 @@ -# TO_BUILD: docker build -rm -no-cache -t docker-ci/testbuilder . -# TO_RUN: docker run -rm -u sysadmin \ -# -v /run:/var/socket docker-ci/testbuilder docker-registry -# - -FROM docker-ci/docker-ci -ENV HOME /home/sysadmin - -RUN mkdir /testbuilder -ADD . /testbuilder - -ENTRYPOINT ["/testbuilder/testbuilder.sh"] diff --git a/hack/infrastructure/docker-ci/testbuilder/docker-registry.sh b/hack/infrastructure/docker-ci/testbuilder/docker-registry.sh deleted file mode 100755 index 72087462ad..0000000000 --- a/hack/infrastructure/docker-ci/testbuilder/docker-registry.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash -set -x -set -e -PROJECT_PATH=$1 - -# Build the docker project -cd /data/$PROJECT_PATH -sg docker -c "docker build -q -rm -t registry ." -cd test; sg docker -c "docker build -q -rm -t docker-registry-test ." - -# Run the tests -sg docker -c "docker run -rm -v /home/docker-ci/coverage/docker-registry:/data docker-registry-test" diff --git a/hack/infrastructure/docker-ci/testbuilder/docker.sh b/hack/infrastructure/docker-ci/testbuilder/docker.sh deleted file mode 100755 index b365dd7eaf..0000000000 --- a/hack/infrastructure/docker-ci/testbuilder/docker.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/env bash -set -x -set -e -PROJECT_PATH=$1 - -# Build the docker project -cd /data/$PROJECT_PATH -sg docker -c "docker build -q -rm -t docker ." - -if [ "$DOCKER_RELEASE" == "1" ]; then - # Do nightly release - echo sg docker -c "docker run -rm -privileged -v /run:/var/socket -e AWS_S3_BUCKET=$AWS_S3_BUCKET -e AWS_ACCESS_KEY= -e AWS_SECRET_KEY= -e GPG_PASSPHRASE= docker hack/release.sh" - set +x - sg docker -c "docker run -rm -privileged -v /run:/var/socket -e AWS_S3_BUCKET=$AWS_S3_BUCKET -e AWS_ACCESS_KEY=$AWS_ACCESS_KEY -e AWS_SECRET_KEY=$AWS_SECRET_KEY -e GPG_PASSPHRASE=$GPG_PASSPHRASE docker hack/release.sh" -else - # Run the tests - sg docker -c "docker run -rm -privileged -v /home/docker-ci/coverage/docker:/data docker ./hack/infrastructure/docker-ci/docker-coverage/gocoverage.sh" -fi diff --git a/hack/infrastructure/docker-ci/testbuilder/testbuilder.sh b/hack/infrastructure/docker-ci/testbuilder/testbuilder.sh deleted file mode 100755 index 70701343c2..0000000000 --- a/hack/infrastructure/docker-ci/testbuilder/testbuilder.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/usr/bin/env bash -# Download, build and run a docker project tests -# Environment variables: DEPLOYMENT - -cat $0 -set -e -set -x - -PROJECT=$1 -COMMIT=${2-HEAD} -REPO=${3-https://github.com/dotcloud/$PROJECT} -BRANCH=${4-master} -REPO_PROJ="https://github.com/docker-test/$PROJECT" -if [ "$DEPLOYMENT" == "production" ]; then - REPO_PROJ="https://github.com/dotcloud/$PROJECT" -fi -set +x - -# Generate a random string of $1 characters -function random { - cat /dev/urandom | tr -cd 'a-f0-9' | head -c $1 -} - -PROJECT_PATH="$PROJECT-tmp-$(random 12)" - -# Set docker-test git user -set -x -git config --global user.email "docker-test@docker.io" -git config --global user.name "docker-test" - -# Fetch project -git clone -q $REPO_PROJ -b master /data/$PROJECT_PATH -cd /data/$PROJECT_PATH -echo "Git commit: $(git rev-parse HEAD)" -git fetch -q $REPO $BRANCH -git merge --no-edit $COMMIT - -# Build the project dockertest -/testbuilder/$PROJECT.sh $PROJECT_PATH -rm -rf /data/$PROJECT_PATH diff --git a/hack/infrastructure/docker-ci/tool/backup.py b/hack/infrastructure/docker-ci/tool/backup.py deleted file mode 100755 index 2db633e526..0000000000 --- a/hack/infrastructure/docker-ci/tool/backup.py +++ /dev/null @@ -1,47 +0,0 @@ -#!/usr/bin/env python - -import os,sys,json -from datetime import datetime -from filecmp import cmp -from subprocess import check_call -from boto.s3.key import Key -from boto.s3.connection import S3Connection - -def ENV(x): - '''Promote an environment variable for global use returning its value''' - retval = os.environ.get(x, '') - globals()[x] = retval - return retval - -ROOT_PATH = '/data/backup/docker-ci' -TODAY = str(datetime.today())[:10] -BACKUP_FILE = '{}/docker-ci_{}.tgz'.format(ROOT_PATH, TODAY) -BACKUP_LINK = '{}/docker-ci.tgz'.format(ROOT_PATH) -ENV('BACKUP_BUCKET') -ENV('BACKUP_AWS_ID') -ENV('BACKUP_AWS_SECRET') - -'''Create full master buildbot backup, avoiding duplicates''' -# Ensure backup path exist -if not os.path.exists(ROOT_PATH): - os.makedirs(ROOT_PATH) -# Make actual backups -check_call('/bin/tar czf {} -C /data --exclude=backup --exclude=buildbot/slave' - ' . 1>/dev/null 2>&1'.format(BACKUP_FILE),shell=True) -# remove previous dump if it is the same as the latest -if (os.path.exists(BACKUP_LINK) and cmp(BACKUP_FILE, BACKUP_LINK) and - os.path._resolve_link(BACKUP_LINK) != BACKUP_FILE): - os.unlink(os.path._resolve_link(BACKUP_LINK)) -# Recreate backup link pointing to latest backup -try: - os.unlink(BACKUP_LINK) -except: - pass -os.symlink(BACKUP_FILE, BACKUP_LINK) - -# Make backup on S3 -bucket = S3Connection(BACKUP_AWS_ID,BACKUP_AWS_SECRET).get_bucket(BACKUP_BUCKET) -k = Key(bucket) -k.key = BACKUP_FILE -k.set_contents_from_filename(BACKUP_FILE) -bucket.copy_key(os.path.basename(BACKUP_LINK),BACKUP_BUCKET,BACKUP_FILE[1:]) diff --git a/hack/install.sh b/hack/install.sh index 65e34f9659..43248cf2c0 100755 --- a/hack/install.sh +++ b/hack/install.sh @@ -72,11 +72,38 @@ fi if [ -z "$lsb_dist" ] && [ -r /etc/debian_version ]; then lsb_dist='Debian' fi +if [ -z "$lsb_dist" ] && [ -r /etc/fedora-release ]; then + lsb_dist='Fedora' +fi case "$lsb_dist" in + Fedora) + ( + set -x + $sh_c 'sleep 3; yum -y -q install docker-io' + ) + if command_exists docker && [ -e /var/run/docker.sock ]; then + ( + set -x + $sh_c 'docker run busybox echo "Docker has been successfully installed!"' + ) || true + fi + your_user=your-user + [ "$user" != 'root' ] && your_user="$user" + echo + echo 'If you would like to use Docker as a non-root user, you should now consider' + echo 'adding your user to the "docker" group with something like:' + echo + echo ' sudo usermod -aG docker' $your_user + echo + echo 'Remember that you will have to log out and back in for this to take effect!' + echo + exit 0 + ;; + Ubuntu|Debian) export DEBIAN_FRONTEND=noninteractive - + did_apt_get_update= apt_get_update() { if [ -z "$did_apt_get_update" ]; then @@ -84,21 +111,21 @@ case "$lsb_dist" in did_apt_get_update=1 fi } - - # TODO remove this section once device-mapper lands + + # aufs is preferred over devicemapper; try to ensure the driver is available. if ! grep -q aufs /proc/filesystems && ! $sh_c 'modprobe aufs'; then kern_extras="linux-image-extra-$(uname -r)" - + apt_get_update ( set -x; $sh_c 'sleep 3; apt-get install -y -q '"$kern_extras" ) || true - + if ! grep -q aufs /proc/filesystems && ! $sh_c 'modprobe aufs'; then echo >&2 'Warning: tried to install '"$kern_extras"' (for AUFS)' echo >&2 ' but we still have no AUFS. Docker may not work. Proceeding anyways!' ( set -x; sleep 10 ) fi fi - + if [ ! -e /usr/lib/apt/methods/https ]; then apt_get_update ( set -x; $sh_c 'sleep 3; apt-get install -y -q apt-transport-https' ) @@ -111,9 +138,9 @@ case "$lsb_dist" in ( set -x if [ "https://get.docker.io/" = "$url" ]; then - $sh_c "apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9" + $sh_c "apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9" elif [ "https://test.docker.io/" = "$url" ]; then - $sh_c "apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 740B314AE3941731B942C66ADF4FD13717AAD7D6" + $sh_c "apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 740B314AE3941731B942C66ADF4FD13717AAD7D6" else $sh_c "$curl ${url}gpg | apt-key add -" fi @@ -138,7 +165,7 @@ case "$lsb_dist" in echo exit 0 ;; - + Gentoo) if [ "$url" = "https://test.docker.io/" ]; then echo >&2 @@ -153,7 +180,7 @@ case "$lsb_dist" in echo >&2 exit 1 fi - + ( set -x $sh_c 'sleep 3; emerge app-emulation/docker' diff --git a/hack/make.sh b/hack/make.sh index 63edca4d4c..e81271370d 100755 --- a/hack/make.sh +++ b/hack/make.sh @@ -43,6 +43,7 @@ DEFAULT_BUNDLES=( binary test test-integration + test-integration-cli dynbinary dyntest dyntest-integration @@ -89,7 +90,7 @@ LDFLAGS=' ' LDFLAGS_STATIC='-linkmode external' EXTLDFLAGS_STATIC='-static' -BUILDFLAGS=( -a -tags "netgo $DOCKER_BUILDTAGS" ) +BUILDFLAGS=( -a -tags "netgo static_build $DOCKER_BUILDTAGS" ) # A few more flags that are specific just to building a completely-static binary (see hack/make/binary) # PLEASE do not use these anywhere else. @@ -125,7 +126,7 @@ go_test_dir() { testcover=( -cover -coverprofile "$coverprofile" $coverpkg ) fi ( - set -x + echo '+ go test' $TESTFLAGS "github.com/dotcloud/docker${dir#.}" cd "$dir" go test ${testcover[@]} -ldflags "$LDFLAGS" "${BUILDFLAGS[@]}" $TESTFLAGS ) @@ -136,11 +137,41 @@ go_test_dir() { # output, one per line. find_dirs() { find -not \( \ - \( -wholename './vendor' -o -wholename './integration' -o -wholename './contrib' -o -wholename './pkg/mflag/example' \) \ + \( \ + -wholename './vendor' \ + -o -wholename './integration' \ + -o -wholename './integration-cli' \ + -o -wholename './contrib' \ + -o -wholename './pkg/mflag/example' \ + -o -wholename './.git' \ + -o -wholename './bundles' \ + -o -wholename './docs' \ + \) \ -prune \ \) -name "$1" -print0 | xargs -0n1 dirname | sort -u } +hash_files() { + while [ $# -gt 0 ]; do + f="$1" + shift + dir="$(dirname "$f")" + base="$(basename "$f")" + for hashAlgo in md5 sha256; do + if command -v "${hashAlgo}sum" &> /dev/null; then + ( + # subshell and cd so that we get output files like: + # $HASH docker-$VERSION + # instead of: + # $HASH /go/src/github.com/.../$VERSION/binary/docker-$VERSION + cd "$dir" + "${hashAlgo}sum" "$base" > "$base.$hashAlgo" + ) + fi + done + done +} + bundle() { bundlescript=$1 bundle=$(basename $bundlescript) diff --git a/hack/make/binary b/hack/make/binary index 7272b1ede0..041e4d1ee8 100644..100755 --- a/hack/make/binary +++ b/hack/make/binary @@ -3,7 +3,7 @@ DEST=$1 go build \ - -o $DEST/docker-$VERSION \ + -o "$DEST/docker-$VERSION" \ "${BUILDFLAGS[@]}" \ -ldflags " $LDFLAGS @@ -11,3 +11,6 @@ go build \ " \ ./docker echo "Created binary: $DEST/docker-$VERSION" +ln -sf "docker-$VERSION" "$DEST/docker" + +hash_files "$DEST/docker-$VERSION" diff --git a/hack/make/dynbinary b/hack/make/dynbinary index d4f583fb62..75cffe3dcc 100644 --- a/hack/make/dynbinary +++ b/hack/make/dynbinary @@ -5,7 +5,7 @@ DEST=$1 if [ -z "$DOCKER_CLIENTONLY" ]; then # dockerinit still needs to be a static binary, even if docker is dynamic go build \ - -o $DEST/dockerinit-$VERSION \ + -o "$DEST/dockerinit-$VERSION" \ "${BUILDFLAGS[@]}" \ -ldflags " $LDFLAGS @@ -14,7 +14,9 @@ if [ -z "$DOCKER_CLIENTONLY" ]; then " \ ./dockerinit echo "Created binary: $DEST/dockerinit-$VERSION" - ln -sf dockerinit-$VERSION $DEST/dockerinit + ln -sf "dockerinit-$VERSION" "$DEST/dockerinit" + + hash_files "$DEST/dockerinit-$VERSION" sha1sum= if command -v sha1sum &> /dev/null; then diff --git a/hack/make/test-integration-cli b/hack/make/test-integration-cli new file mode 100644 index 0000000000..b0506d261a --- /dev/null +++ b/hack/make/test-integration-cli @@ -0,0 +1,40 @@ +#!/bin/bash + +DEST=$1 + +set -e + +DOCKER_GRAPHDRIVER=${DOCKER_GRAPHDRIVER:-vfs} +DOCKER_EXECDRIVER=${DOCKER_EXECDRIVER:-native} + +bundle_test_integration_cli() { + go_test_dir ./integration-cli +} + +# subshell so that we can export PATH without breaking other things +( + export PATH="$DEST/../binary:$DEST/../dynbinary:$PATH" + + if ! command -v docker &> /dev/null; then + echo >&2 'error: binary or dynbinary must be run before test-integration-cli' + false + fi + + ( set -x; exec \ + docker --daemon --debug \ + --storage-driver "$DOCKER_GRAPHDRIVER" \ + --exec-driver "$DOCKER_EXECDRIVER" \ + --pidfile "$DEST/docker.pid" \ + &> "$DEST/docker.log" + ) & + + # pull the busybox image before running the tests + sleep 2 + ( set -x; docker pull busybox ) + + bundle_test_integration_cli + + DOCKERD_PID=$(set -x; cat $DEST/docker.pid) + ( set -x; kill $DOCKERD_PID ) + wait $DOCKERD_PID || true +) 2>&1 | tee $DEST/test.log diff --git a/hack/make/tgz b/hack/make/tgz index 5d03306322..120339976b 100644 --- a/hack/make/tgz +++ b/hack/make/tgz @@ -23,6 +23,8 @@ for d in "$CROSS/"*/*; do tar --numeric-owner --owner 0 -C "$DEST/build" -czf "$TGZ" usr + hash_files "$TGZ" + rm -rf "$DEST/build" echo "Created tgz: $TGZ" diff --git a/hack/make/ubuntu b/hack/make/ubuntu index ebc12f27ec..403a6c7652 100644 --- a/hack/make/ubuntu +++ b/hack/make/ubuntu @@ -38,6 +38,14 @@ bundle_ubuntu() { mkdir -p $DIR/lib/systemd/system cp contrib/init/systemd/docker.service $DIR/lib/systemd/system/ + # Include contributed completions + mkdir -p $DIR/etc/bash_completion.d + cp contrib/completion/bash/docker $DIR/etc/bash_completion.d/ + mkdir -p $DIR/usr/share/zsh/vendor-completions + cp contrib/completion/zsh/_docker $DIR/usr/share/zsh/vendor-completions/ + mkdir -p $DIR/etc/fish/completions + cp contrib/completion/fish/docker.fish $DIR/etc/fish/completions/ + # Copy the binary # This will fail if the binary bundle hasn't been built mkdir -p $DIR/usr/bin diff --git a/hack/release.sh b/hack/release.sh index 50913dd395..d77d454e27 100755 --- a/hack/release.sh +++ b/hack/release.sh @@ -31,7 +31,7 @@ docker run -e AWS_S3_BUCKET=get-staging.docker.io \ -e AWS_ACCESS_KEY=AKI1234... \ -e AWS_SECRET_KEY=sEs4mE... \ -e GPG_PASSPHRASE=m0resEs4mE... \ - -i -t -privileged \ + -i -t --privileged \ docker ./hack/release.sh EOF exit 1 @@ -53,35 +53,22 @@ RELEASE_BUNDLES=( ) if [ "$1" != '--release-regardless-of-test-failure' ]; then - RELEASE_BUNDLES=( test "${RELEASE_BUNDLES[@]}" ) -fi - -if ! ./hack/make.sh "${RELEASE_BUNDLES[@]}"; then - echo >&2 - echo >&2 'The build or tests appear to have failed.' - echo >&2 - echo >&2 'You, as the release maintainer, now have a couple options:' - echo >&2 '- delay release and fix issues' - echo >&2 '- delay release and fix issues' - echo >&2 '- did we mention how important this is? issues need fixing :)' - echo >&2 - echo >&2 'As a final LAST RESORT, you (because only you, the release maintainer,' - echo >&2 ' really knows all the hairy problems at hand with the current release' - echo >&2 ' issues) may bypass this checking by running this script again with the' - echo >&2 ' single argument of "--release-regardless-of-test-failure", which will skip' - echo >&2 ' running the test suite, and will only build the binaries and packages. Please' - echo >&2 ' avoid using this if at all possible.' - echo >&2 - echo >&2 'Regardless, we cannot stress enough the scarcity with which this bypass' - echo >&2 ' should be used. If there are release issues, we should always err on the' - echo >&2 ' side of caution.' - echo >&2 - exit 1 + RELEASE_BUNDLES=( + test test-integration + "${RELEASE_BUNDLES[@]}" + test-integration-cli + ) fi VERSION=$(cat VERSION) BUCKET=$AWS_S3_BUCKET +# These are the 2 keys we've used to sign the deb's +# release (get.docker.io) +# GPG_KEY="36A1D7869245C8950F966E92D8576A8BA88D21E9" +# test (test.docker.io) +# GPG_KEY="740B314AE3941731B942C66ADF4FD13717AAD7D6" + setup_s3() { # Try creating the bucket. Ignore errors (it might already exist). s3cmd mb s3://$BUCKET 2>/dev/null || true @@ -114,76 +101,138 @@ s3_url() { esac } +build_all() { + if ! ./hack/make.sh "${RELEASE_BUNDLES[@]}"; then + echo >&2 + echo >&2 'The build or tests appear to have failed.' + echo >&2 + echo >&2 'You, as the release maintainer, now have a couple options:' + echo >&2 '- delay release and fix issues' + echo >&2 '- delay release and fix issues' + echo >&2 '- did we mention how important this is? issues need fixing :)' + echo >&2 + echo >&2 'As a final LAST RESORT, you (because only you, the release maintainer,' + echo >&2 ' really knows all the hairy problems at hand with the current release' + echo >&2 ' issues) may bypass this checking by running this script again with the' + echo >&2 ' single argument of "--release-regardless-of-test-failure", which will skip' + echo >&2 ' running the test suite, and will only build the binaries and packages. Please' + echo >&2 ' avoid using this if at all possible.' + echo >&2 + echo >&2 'Regardless, we cannot stress enough the scarcity with which this bypass' + echo >&2 ' should be used. If there are release issues, we should always err on the' + echo >&2 ' side of caution.' + echo >&2 + exit 1 + fi +} + +upload_release_build() { + src="$1" + dst="$2" + latest="$3" + + echo + echo "Uploading $src" + echo " to $dst" + echo + s3cmd --follow-symlinks --preserve --acl-public put "$src" "$dst" + if [ "$latest" ]; then + echo + echo "Copying to $latest" + echo + s3cmd --acl-public cp "$dst" "$latest" + fi + + # get hash files too (see hash_files() in hack/make.sh) + for hashAlgo in md5 sha256; do + if [ -e "$src.$hashAlgo" ]; then + echo + echo "Uploading $src.$hashAlgo" + echo " to $dst.$hashAlgo" + echo + s3cmd --follow-symlinks --preserve --acl-public --mime-type='text/plain' put "$src.$hashAlgo" "$dst.$hashAlgo" + if [ "$latest" ]; then + echo + echo "Copying to $latest.$hashAlgo" + echo + s3cmd --acl-public cp "$dst.$hashAlgo" "$latest.$hashAlgo" + fi + fi + done +} + release_build() { GOOS=$1 GOARCH=$2 - BINARY=bundles/$VERSION/cross/$GOOS/$GOARCH/docker-$VERSION - TGZ=bundles/$VERSION/tgz/$GOOS/$GOARCH/docker-$VERSION.tgz + binDir=bundles/$VERSION/cross/$GOOS/$GOARCH + tgzDir=bundles/$VERSION/tgz/$GOOS/$GOARCH + binary=docker-$VERSION + tgz=docker-$VERSION.tgz + + latestBase= + if [ -z "$NOLATEST" ]; then + latestBase=docker-latest + fi # we need to map our GOOS and GOARCH to uname values # see https://en.wikipedia.org/wiki/Uname # ie, GOOS=linux -> "uname -s"=Linux - S3OS=$GOOS - case "$S3OS" in + s3Os=$GOOS + case "$s3Os" in darwin) - S3OS=Darwin + s3Os=Darwin ;; freebsd) - S3OS=FreeBSD + s3Os=FreeBSD ;; linux) - S3OS=Linux + s3Os=Linux ;; *) - echo >&2 "error: can't convert $S3OS to an appropriate value for 'uname -s'" + echo >&2 "error: can't convert $s3Os to an appropriate value for 'uname -s'" exit 1 ;; esac - S3ARCH=$GOARCH - case "$S3ARCH" in + s3Arch=$GOARCH + case "$s3Arch" in amd64) - S3ARCH=x86_64 + s3Arch=x86_64 ;; 386) - S3ARCH=i386 + s3Arch=i386 ;; arm) - S3ARCH=armel + s3Arch=armel # someday, we might potentially support mutliple GOARM values, in which case we might get armhf here too ;; *) - echo >&2 "error: can't convert $S3ARCH to an appropriate value for 'uname -m'" + echo >&2 "error: can't convert $s3Arch to an appropriate value for 'uname -m'" exit 1 ;; esac - S3DIR=s3://$BUCKET/builds/$S3OS/$S3ARCH + s3Dir=s3://$BUCKET/builds/$s3Os/$s3Arch + latest= + latestTgz= + if [ "$latestBase" ]; then + latest="$s3Dir/$latestBase" + latestTgz="$s3Dir/$latestBase.tgz" + fi - if [ ! -x "$BINARY" ]; then - echo >&2 "error: can't find $BINARY - was it compiled properly?" + if [ ! -x "$binDir/$binary" ]; then + echo >&2 "error: can't find $binDir/$binary - was it compiled properly?" exit 1 fi - if [ ! -f "$TGZ" ]; then - echo >&2 "error: can't find $TGZ - was it packaged properly?" + if [ ! -f "$tgzDir/$tgz" ]; then + echo >&2 "error: can't find $tgzDir/$tgz - was it packaged properly?" exit 1 fi - echo "Uploading $BINARY to $S3OS/$S3ARCH/docker-$VERSION" - s3cmd --follow-symlinks --preserve --acl-public put $BINARY $S3DIR/docker-$VERSION - - echo "Uploading $TGZ to $S3OS/$S3ARCH/docker-$VERSION.tgz" - s3cmd --follow-symlinks --preserve --acl-public put $TGZ $S3DIR/docker-$VERSION.tgz - - if [ -z "$NOLATEST" ]; then - echo "Copying $S3OS/$S3ARCH/docker-$VERSION to $S3OS/$S3ARCH/docker-latest" - s3cmd --acl-public cp $S3DIR/docker-$VERSION $S3DIR/docker-latest - - echo "Copying $S3OS/$S3ARCH/docker-$VERSION.tgz to $S3OS/$S3ARCH/docker-latest.tgz" - s3cmd --acl-public cp $S3DIR/docker-$VERSION.tgz $S3DIR/docker-latest.tgz - fi + upload_release_build "$binDir/$binary" "$s3Dir/$binary" "$latest" + upload_release_build "$tgzDir/$tgz" "$s3Dir/$tgz" "$latestTgz" } # Upload the 'ubuntu' bundle to S3: @@ -194,21 +243,6 @@ release_ubuntu() { echo >&2 './hack/make.sh must be run before release_ubuntu' exit 1 } - # Make sure that we have our keys - mkdir -p /.gnupg/ - s3cmd sync s3://$BUCKET/ubuntu/.gnupg/ /.gnupg/ || true - gpg --list-keys releasedocker >/dev/null || { - gpg --gen-key --batch <<EOF -Key-Type: RSA -Key-Length: 2048 -Passphrase: $GPG_PASSPHRASE -Name-Real: Docker Release Tool -Name-Email: docker@dotcloud.com -Name-Comment: releasedocker -Expire-Date: 0 -%commit -EOF - } # Sign our packages dpkg-sig -g "--passphrase $GPG_PASSPHRASE" -k releasedocker \ @@ -243,10 +277,15 @@ EOF # Upload repo s3cmd --acl-public sync $APTDIR/ s3://$BUCKET/ubuntu/ cat <<EOF | write_to_s3 s3://$BUCKET/ubuntu/index +# Check that HTTPS transport is available to APT +if [ ! -e /usr/lib/apt/methods/https ]; then + apt-get update + apt-get install -y apt-transport-https +fi # Add the repository to your APT sources echo deb $(s3_url)/ubuntu docker main > /etc/apt/sources.list.d/docker.list # Then import the repository key -apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 +apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 # Install docker apt-get update ; apt-get install -y lxc-docker @@ -305,8 +344,28 @@ release_test() { fi } +setup_gpg() { + # Make sure that we have our keys + mkdir -p /.gnupg/ + s3cmd sync s3://$BUCKET/ubuntu/.gnupg/ /.gnupg/ || true + gpg --list-keys releasedocker >/dev/null || { + gpg --gen-key --batch <<EOF +Key-Type: RSA +Key-Length: 4096 +Passphrase: $GPG_PASSPHRASE +Name-Real: Docker Release Tool +Name-Email: docker@dotcloud.com +Name-Comment: releasedocker +Expire-Date: 0 +%commit +EOF + } +} + main() { + build_all setup_s3 + setup_gpg release_binaries release_ubuntu release_index diff --git a/hack/vendor.sh b/hack/vendor.sh index 184cb750a5..4200d90867 100755 --- a/hack/vendor.sh +++ b/hack/vendor.sh @@ -39,7 +39,7 @@ clone() { echo done } -clone git github.com/kr/pty 3b1f6487b +clone git github.com/kr/pty 98c7b80083 clone git github.com/gorilla/context 708054d61e5 @@ -58,3 +58,6 @@ mv src/code.google.com/p/go/src/pkg/archive/tar tmp-tar rm -rf src/code.google.com/p/go mkdir -p src/code.google.com/p/go/src/pkg/archive mv tmp-tar src/code.google.com/p/go/src/pkg/archive/tar + +clone git github.com/godbus/dbus cb98efbb933d8389ab549a060e880ea3c375d213 +clone git github.com/coreos/go-systemd 4c14ed39b8a643ac44b4f95b5a53c00e94261475 |