diff options
author | Rasmus Johansson <razze@iki.fi> | 2019-05-28 14:04:39 +0300 |
---|---|---|
committer | Oleksandr Byelkin <sanja@mariadb.com> | 2020-06-16 11:01:23 +0200 |
commit | 39e5e89052b4029b868a578d07f92e180db0d0d9 (patch) | |
tree | a163f2d101f6f460ab1b218f0720e896cbe786c1 | |
parent | 44053ceef75ad34459e55c4d683737d328dd84f3 (diff) | |
download | mariadb-git-10.5-enterprise-rebase.tar.gz |
azure pipelines configuration10.5-enterprise-rebase
Fix Windows build and test
Add galera-es test repository
update azure-pipelines.yml for new galera-es package names
Install needed packages for galera-es testing
add galera-es tests for centos6, centos7
Fix syntax error with galera-es repo setup
Fix galera-es repo url
Update azure-pipelines.yml
Added a comment to explain why the section is only in the 10.4 pipeline script
Move MTR tests out of bintar job to new 'MTR' job
add download and untar steps
try fix for name of artifact to download
more updates to download task for MTR job
Fix paths
Remove bintar build dependency from DEB and RPM jobs
unpack to opt, and add full paths to cd commands
Also turn off verbose when untarring, we don't need that
fix ownership of /opt dir
revert back to using System.ArtifactsDirectory instead of /opt
chown of /opt to the current user failed because the $USER env variable
is not set for some reason, not sure if this is a docker issue, or
something with azure, or something else, don't want to mess with it at
this time
Add debian-8
fix deb publishing
the .changes file was missing, needs to be included with the .deb files
Also added the scanpackages step that we do with our buildbot and galera
builds
add -deb or -rpm suffix to package build artifacts
This is to keep bintar and package artifacts separate, which makes for
easier processing during releases.
include missing deb files
Remove debian-8 for now
It's not building packages correctly
increase job timeout for Windows, from default 1 hour to 3 hours
fix for Debian 8
use the correct number of CPUs to not overload the builder
MENT-160 RPM packages are not installable on CentOS due to to missing MariaDB-compat
MENT-164 Some RPM packages present in community server are missing from enterprise
and MENT-163 RPM installation complains about policy files, mariadb.service is not installed
install missing -devel packages to get cracklib_password_check, oqgraph,
and systemd support
show scripts executing in azure
run ASAN as in buildbot
Desactivating ASan
Package ZIPs
also, echo commands in Windows build scripts
Windows - do not publish exe/dll/pdb, the relevant stuff is in ZIP
make sure artifacts are saved on Windows, even if some MTR tests fail.
use static gnutls for bintars
Adding required prerequisites for successful execution of mtr with Galera tests
run mtr also for ubuntu 16.04 bintars
Test plugins suite on Windows.
MENT-187 Windows build needs to be signed
Checking the need to install lsof for CentOS 7
MENT-261 Store the build docker images in a Azure Container Repository
azure-pipelines.yml install galera library
Galera library is now installed in its own steps so that it can be used both for normal mtr and galera mtr
MENT-176 Set up the Stage Gate to the Azure CI
MENT-265 Azure CI fails on CopyFiles after RPM build
MENT-25 Add test to only have stable plugins in Enterprise
MENT-252 install tests in azure
MENT-198 Build & debug ASAN builds for all supported OS & versions
MENT-329 Build & debug SLES 15
MENT-191 Build & debug SLES 12
Initial Stage Gate implementation for Azure pipeline
[+] STAGE 1 source tarball reworked
[+] STAGE 1 binary release tarballs reworked
[+] STAGE 1 binary debug tarballs reworked
[+] STAGE 1 release MTR reworked
[+] STAGE 1 debug MTR reworked
[+] STAGE 1 Release RPMS reworked
[+] STAGE 1 Windows ZIP and MSI reworked
[+] STAGE 1 Windows release MTR reworked
[+] STAGE 1 Galera on release tarballs reworked
[+] STAGE 2 Azure <=> KVM servers interaction
[+] STAGE 2 RPM installation test
[+] STAGE 2 DEB installation test
[+] STAGE 2 PAM acceptance test
[+] STAGE 2 Simple SELECT/INSERT test
[+] STAGE 2 Galera tests
[+] STAGE 2 MTR tests from buildbot
[+] STAGE 2 Upgrade test, 10.2 => 10.3 => 10.4 current
MENT-355
disable expensive checks on Azure for personal branches
MENT-355
disable expensive checks on Azure for personal branches
MENT-367 move Debug and ASAN builds to separate pipelines
10.4 pipelines
MENT-370 Replace Azure CentOS 6 & 7 builders with RHEL builders
replaced centos with rhel builders
MENT-371 yum install fails on CentOS 7
transformed image name to rpm name variable by deleting dash from image
name
MENT-383 move stage2 to the separate pipeline
created additional YAML file for separate pipeline to run on 03AM EEST
MENT-245 Azure Stage gate pipelines scripts
added Galera-Enterprise repository to SLES
MENT-393 Azure job must fail if VM can't be prepared for tests
do not supress eny error from yum/apt/zypper during uninstall
MENT-398 galera mtr tests - false positives
MENT-402 default Azure timeout breaks pkg build
Azure default timeout is 60m. this commit increases build time (and time
in queue) from 60m to 180m (3x)
MENT-296 Galera MTR failed on Azure on 10.4-enterprise
--test-case-timeout=120 for all Galera tests
MENT-419 Galera: Create pipeline that is run after every commit
added back galera tests to release pipeline
MENT-279 MariaDB Enterprise source code tarballs don't have "enterprise" in the file name
added properly set CPACK_SOURCE_PACKAGE_FILE_NAME, effective only with
-DBUILD_CONFIG=enterprise
(azure part)
MENT-61 Installation / Upgrade tests
in fact installation tests are located in the release pipeline and may
be easily extended to cover all supported systems. so this commit is
intended to bring only upgrade tests into Stage Gates.
the logic is:
* build packages
* prepare VM for installation
* install Community Server
* check installation validity
* install ES over CS
* check installation validity
* install built ES over released
MENT-441 set timeout on copyfiles@1
the goal is to fail quickly if file can't be copied or just hangs
MENT-420 package dependency problem with postfix & perl-DBD-MySQL on CentOS7
it's for -compat package. changed package name to download
MariaDB-shared instead of -compat to have libmysqlclient.so.18
MENT-466 Release pipeline - Quick Ubuntu - 2 failures to analyse
added bash loop, because lockfile may be acquired by OS during run
MENT-445 Release pipeline - task timeout for SLES
added 2m timeout for download task to fail quickly if Azure hangs
MENT-459 ASAN Pipeline - Several builds packages are not built properly
added release and debug configurations, reduced platform list, fixed mtr
call
MENT-199 Build & debug MacOS
enabled MacOS compilation
MENT-241 Save more detailed failures details & files
added conditional step to archive mysql vardir after failed mtr
MENT-496 ASAN pipeline can't archive logs - missing parameters
added missing params for azure/tests/run-mtr-test.yml
MENT-499 fix galera installation script in pipelines
set separate repos for galera-3 and galera-4 in pipelines depending on
ES version
MENT-485 galera-25.3.27-1.sles12.sle12.x86_64.rpm' not found on medium 'https://downloads.mariadb.com
added missing step to rewrite the repo and install galera
MENT-507 short timeout lead to job failures on azure
set timeout to 10m
MENT-505 galera-enterprise-4 is not found for buster
removed apt-add-repository call
MENT-526 Remove Travis & Appveyor configuration files in the MariaDBEnterprise repository branches
removed yaml configs
MENT-465 Upgrade Test Failure - Rolling upgrade needs to be fixed
commented out Centos platforms until MDEV-9584 is fixed
MENT-524 Azure pipeline for checking server installation from the official repo during release process
added separate yaml file to run new pipeline
MENT-558 move Galera MTR tests to the separate pipeline or disable for long MTR run
added one more stage to run MTR tests 2*3 in parallel
fixed typo in engines test
Change the way cygwin is downloaded
MENT-672 debian & ubuntu lack dependencies for new 10.5 dh-exec libpcre2-dev
replaced debian jessie with ubuntu focal
MENT-678 wrong cmake logic related to OpenSSL
on Mac and on other systems with custom OpenSSL installation -DWITH_SSL
must be used instead of specifying OPENSSL_ROOT_DIR and
OPENSSL_LIBRARIES
TODO-2280 ASAN failure on Azure, missing sanitizer/asan_interface.h
changed gcc version to 10 for asan builds
disabled pkg build and installation test on Azure
TODO-2367 no MacOS nightly builds for 10.5
copied MacOS pipeline from 10.4
Fix XPand compile failures in Azure Pipelines on Windows
- Use relative path to submodule
- Checkout submodules also when building source tarball
- Update XPand to include revision which disables compiling XPand on W
(azure part)
Make MacOS build check out submodules
30 files changed, 2527 insertions, 76 deletions
diff --git a/.travis.compiler.sh b/.travis.compiler.sh deleted file mode 100755 index 98d8197d371..00000000000 --- a/.travis.compiler.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/bin/sh -set -v -x - -# Exclude modules from build not directly affecting the current -# test suites found in $MYSQL_TEST_SUITES, to conserve job time -# as well as disk usage - -function exclude_modules() { -# excludes for all -CMAKE_OPT="${CMAKE_OPT} -DPLUGIN_TOKUDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_OQGRAPH=NO -DPLUGIN_PERFSCHEMA=NO -DPLUGIN_SPHINX=NO" -# exclude storage engines not being tested in current job -if [[ ! "${MYSQL_TEST_SUITES}" =~ "archive" ]]; then - CMAKE_OPT="${CMAKE_OPT} -DPLUGIN_ARCHIVE=NO" -fi -if [[ ! "${MYSQL_TEST_SUITES}" =~ "rocksdb" ]]; then - CMAKE_OPT="${CMAKE_OPT} -DPLUGIN_ROCKSDB=NO" -fi -} - -if [[ "${TRAVIS_OS_NAME}" == 'linux' ]]; then - TEST_CASE_TIMEOUT=2 - exclude_modules; - if which ccache ; then - CMAKE_OPT="${CMAKE_OPT} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache" - fi - if [[ "${CXX}" == 'clang++' ]]; then - if [[ "${CC_VERSION}" == '6' ]]; then - export CXX=${CXX}-${CC_VERSION}.0 - else - export CXX=${CXX}-${CC_VERSION} - fi - export CC=${CXX/++/} - # excess warnings about unused include path - export CFLAGS='-Wno-unused-command-line-argument' - export CXXFLAGS='-Wno-unused-command-line-argument' - elif [[ "${CXX}" == 'g++' ]]; then - export CXX=g++-${CC_VERSION} - export CC=gcc-${CC_VERSION} - fi - if [[ ${CC_VERSION} == 7 ]]; then - export WSREP_PROVIDER=/usr/lib/galera/libgalera_smm.so - MYSQL_TEST_SUITES="${MYSQL_TEST_SUITES},wsrep" - fi -fi - -if [[ "${TRAVIS_OS_NAME}" == 'osx' ]]; then - TEST_CASE_TIMEOUT=20 - exclude_modules; - CMAKE_OPT="${CMAKE_OPT} -DOPENSSL_ROOT_DIR=/usr/local/opt/openssl" - if which ccache ; then - CMAKE_OPT="${CMAKE_OPT} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache" - fi -fi - -set +v +x diff --git a/appveyor.yml b/appveyor.yml deleted file mode 100644 index 4a6a49fd8b5..00000000000 --- a/appveyor.yml +++ /dev/null @@ -1,21 +0,0 @@ -version: build-{build}~branch-{branch} - -before_build: - - md %APPVEYOR_BUILD_FOLDER%\win_build - - cd %APPVEYOR_BUILD_FOLDER%\win_build - - cmake .. -G "Visual Studio 15 2017 Win64" -DWITH_UNIT_TESTS=0 -DWITH_MARIABACKUP=0 -DMYSQL_MAINTAINER_MODE=ERR -DPLUGIN_ROCKSDB=NO -DPLUGIN_CONNECT=NO -DBISON_EXECUTABLE=C:\cygwin64\bin\bison - -build: - project: win_build\MySQL.sln - parallel: true - verbosity: minimal - -configuration: RelWithDebInfo -platform: x64 - -test_script: - - set PATH=%PATH%;C:\Program Files (x86)\Windows Kits\10\Debuggers\x64 - - cd %APPVEYOR_BUILD_FOLDER%\win_build\mysql-test - - perl mysql-test-run.pl --force --max-test-fail=10 --parallel=4 --testcase-timeout=10 --skip-test-list=unstable-tests --suite=main - -image: Visual Studio 2017 diff --git a/azure-pipelines-asan.yml b/azure-pipelines-asan.yml new file mode 100644 index 00000000000..fe988efa0bb --- /dev/null +++ b/azure-pipelines-asan.yml @@ -0,0 +1,113 @@ +# C/C++ with GCC +# Build your C/C++ project with GCC using make. +# Add steps that publish test results, save build artifacts, deploy, and more: +# https://docs.microsoft.com/azure/devops/pipelines/apps/c-cpp/gcc + +resources: + containers: + - container: ubuntu-1804 + image: mariadbe.azurecr.io/build-es:ubuntu-1804 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1904 + image: mariadbe.azurecr.io/build-es:ubuntu-1904 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + +variables: + TEST_RESULTS_FILE: 'test-results.log' + MTR_FEEDBACK_PLUGIN: 1 + DEBIAN_FRONTEND: noninteractive + MYSQL_VARDIR: '/var/tmp/mtr' + BUILDDIR: "$(Build.SourcesDirectory)/BUILD" + ASAN_OPTIONS: abort_on_error=1 +# +stages: + +- stage: BuildPackages_Stage0 + + jobs: + + - template: azure/stage0/build-source-tarball.yml + + - template: azure/stage0/build-binary-tarballs-asan.yml + parameters: + BuildType: RelWithDebInfo + Artifact: ASAN + CMakeArgs: '-DWITH_ASAN=ON' + + - template: azure/stage0/build-binary-tarballs-asan.yml + parameters: + BuildType: Debug + Artifact: ASAN + CMakeArgs: '-DWITH_ASAN=ON' + +################################# +########## STAGE 0 END ########## +################################# + + +################################# +######### STAGE 1 BEGIN ######### +################################# + +- stage: TestPackages_ASAN + dependsOn: BuildPackages_Stage0 + condition: succeededOrFailed() + jobs: + + - job: TestTarball_Debug + timeoutInMinutes: 240 + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + ubuntu-1904: + containerImage: ubuntu-1904 + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: azure/fetch-unpack-tarball.yml + parameters: + Artifact: ASAN-Debug + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--force --verbose-restart --parallel=auto --vardir=$(MYSQL_VARDIR)' + TestName: 'Quick MTR Test - ASAN Debug' + tarname: 'asan-mtr-quick-test.tar.gz' + Artifact: ASAN-Debug + + - job: TestTarball_RelWithDebInfo + timeoutInMinutes: 240 + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + ubuntu-1904: + containerImage: ubuntu-1904 + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: azure/fetch-unpack-tarball.yml + parameters: + Artifact: ASAN-RelWithDebInfo + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--force --verbose-restart --parallel=auto --vardir=$(MYSQL_VARDIR)' + TestName: 'Quick MTR Test - ASAN RelWithDebInfo' + tarname: 'asan-mtr-quick-test.tar.gz' + Artifact: ASAN-RelWithDebInfo +# +################################# +########## STAGE 1 END ########## +################################# +# diff --git a/azure-pipelines-debug.yml b/azure-pipelines-debug.yml new file mode 100644 index 00000000000..2fbb662c6dd --- /dev/null +++ b/azure-pipelines-debug.yml @@ -0,0 +1,133 @@ +# C/C++ with GCC +# Build your C/C++ project with GCC using make. +# Add steps that publish test results, save build artifacts, deploy, and more: +# https://docs.microsoft.com/azure/devops/pipelines/apps/c-cpp/gcc + +resources: + containers: + + - container: rhel-6 + image: mariadbe.azurecr.io/build-es:rhel-6 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: rhel-7 + image: mariadbe.azurecr.io/build-es:rhel-7 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: rhel-8 + image: mariadbe.azurecr.io/build-es:rhel-8 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: sles-12 + image: mariadbe.azurecr.io/build-es:sles-12 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: sles-15 + image: mariadbe.azurecr.io/build-es:sles-15 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1604 + image: mariadbe.azurecr.io/build-es:ubuntu-1604 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1804 + image: mariadbe.azurecr.io/build-es:ubuntu-1804 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-8 + image: mariadbe.azurecr.io/build-es:debian-8 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-9 + image: mariadbe.azurecr.io/build-es:debian-9 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-10 + image: mariadbe.azurecr.io/build-es:debian-10 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + +variables: + TEST_RESULTS_FILE: 'test-results.log' + MTR_FEEDBACK_PLUGIN: 1 + DEBIAN_FRONTEND: noninteractive + MYSQL_VARDIR: '/var/tmp/mtr' + BUILDDIR: "$(Build.SourcesDirectory)/BUILD" + VERSION: '10.4' +# +stages: + +- stage: BuildPackages_Stage0 + + jobs: + + - template: azure/stage0/build-source-tarball.yml + + - template: azure/stage0/build-binary-tarballs.yml + parameters: + BuildType: Debug + Artifact: Debug + + - template: azure/stage0/build-windows-packages.yml + parameters: + Artifact: Debug + BuildType: Debug + + - template: azure/stage0/build-macos-packages.yml + parameters: + BuildType: Debug + Artifact: Debug + +################################# +########## STAGE 0 END ########## +################################# + + +################################# +######### STAGE 1 BEGIN ######### +################################# + +- stage: TestPackages_Debug + dependsOn: BuildPackages_Stage0 + condition: succeededOrFailed() + jobs: + + - job: TestTarball_Quick + timeoutInMinutes: 240 + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + rhel-7: + containerImage: rhel-7 + ubuntu-1804: + containerImage: ubuntu-1804 + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: azure/fetch-unpack-tarball.yml + parameters: + Artifact: Debug + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--mem --force --verbose-restart --skip-test-list=unstable-tests --parallel=auto' + TestName: 'Quick Test - Debug' + +# +################################# +########## STAGE 1 END ########## +################################# +# diff --git a/azure-pipelines-installtest.yml b/azure-pipelines-installtest.yml new file mode 100644 index 00000000000..6772ca02089 --- /dev/null +++ b/azure-pipelines-installtest.yml @@ -0,0 +1,127 @@ +# C/C++ with GCC +# Build your C/C++ project with GCC using make. +# Add steps that publish test results, save build artifacts, deploy, and more: +# https://docs.microsoft.com/azure/devops/pipelines/apps/c-cpp/gcc + +variables: + MYSQL_PASSWORD: 'tESt123%_password' + ESTOKEN: '49bc85f8-6e90-443d-901a-b7f01a4ab22c' + ES_REPO_URL: 'https://dlm.mariadb.com/enterprise-release-helpers/mariadb_es_repo_setup' + VERSION: '10.4' + +jobs: + +- job: Install_ES_10_2 + timeoutInMinutes: 30 + workspace: + clean: all + strategy: + maxParallel: 2 + matrix: + rhel-6: + poolName: 'rhel-6' + rhel-7: + poolName: 'rhel-7' + # rhel-8: + # poolName: 'rhel-8' + debian-8: + poolName: 'debian-8' + debian-9: + poolName: 'debian-9' + debian-10: + poolName: 'debian-10' + ubuntu-1604: + poolName: 'ubuntu-1604' + ubuntu-1804: + poolName: 'ubuntu-1804' + pool: + name: $(poolName) + + steps: + - checkout: none + - template: azure/prepare-vm-for-installation.yml + + - bash: | + set -ex + wget $(ES_REPO_URL) -O install_mariadb.sh + chmod +x ./install_mariadb.sh + sudo ./install_mariadb.sh --token=$(ESTOKEN) --apply --verbose --mariadb-server-version=$(VERSION) + displayName: "Install ES-$(VERSION) repo file and key" + + - bash: | + set -ex + sudo yum -y install MariaDB-* + displayName: "Install ES-$(VERSION) on RHEL/Centos" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7', 'rhel-8')) + + - bash: | + set -ex + sudo apt-get update + export DEBIAN_FRONTEND=noninteractive + sudo apt-get -y install mariadb-server mariadb-backup \ + mariadb-plugin-mroonga mariadb-plugin-oqgraph mariadb-plugin-rocksdb mariadb-plugin-spider \ + mariadb-test mariadb-test-data + # we do not install cassandra and tokudb (doesn't exist in ES) + displayName: "Install ES-$(VERSION) on Debian/Ubuntu" + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'debian-10', 'ubuntu-1604', 'ubuntu-1804')) + + - bash: | + set -ex + if [[ -f /etc/init.d/mysql ]]; then + restart_cmd="service mysql restart" + else + restart_cmd="systemctl restart mysql" + fi + sudo ${restart_cmd} + displayName: "Restart ES-$(VERSION) NOT RHEL-7" + condition: ne(variables['poolName'], 'rhel-7') + + - bash: | + set -x + restart_cmd="systemctl restart mariadb" + sudo ${restart_cmd} + displayName: "Restart ES-$(VERSION) RHEL-7" + condition: eq(variables['poolName'], 'rhel-7') + + - bash: | + set -ex + sudo mysql -e 'SELECT 1' + displayName: "Check ES-$(VERSION) installation on RHEL/Centos" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7', 'rhel-8')) + + - bash: | + set -ex + sudo mysql -p$(MYSQL_PASSWORD) -e 'SELECT 1' + displayName: "Check ES-$(VERSION) installation on Debian/Ubuntu" + condition: in(variables['poolName'], 'debian-8', 'debian-9', 'debian-10', 'ubuntu-1604', 'ubuntu-1804') + + - bash: | + set -x + rpm -qa | grep -iE "^maria|^mysql|^percona|^galera" + displayName: "List installed MariaDB packages on RHEL/Centos" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7', 'rhel-8')) + + - bash: | + set -x + dpkg -l | grep -iE "maria|mysql|percona|galera" | awk '{print $1"\t"$2"\t"$3"\t"$4}' + displayName: "List installed MariaDB packages on Debian/Ubuntu" + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'debian-10', 'ubuntu-1604', 'ubuntu-1804')) + + - template: azure/prepare-vm-for-installation.yml + + + + + + + + + + + + + + + + + diff --git a/azure-pipelines-mac.yml b/azure-pipelines-mac.yml new file mode 100644 index 00000000000..bc8afc8dbdf --- /dev/null +++ b/azure-pipelines-mac.yml @@ -0,0 +1,120 @@ +# C/C++ with GCC +# Build your C/C++ project with GCC using make. +# Add steps that publish test results, save build artifacts, deploy, and more: +# https://docs.microsoft.com/azure/devops/pipelines/apps/c-cpp/gcc + +resources: + containers: + - container: ubuntu-1804 + image: mariadbe.azurecr.io/build-es:ubuntu-1804 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + +variables: + configuration: RelWithDebInfo + platform: x64 + TEST_RESULTS_FILE: 'test-results.log' + MTR_FEEDBACK_PLUGIN: 1 + DEBIAN_FRONTEND: noninteractive + MYSQL_VARDIR: '/var/tmp/mtr' + DEB_LOCAL_LIST: '/etc/apt/sources.list.d/localInstall.list' + BUILDDIR: "$(Build.SourcesDirectory)/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE" + VERSION: '10.4' +# +stages: + +- stage: BuildPackages_Stage0 + + jobs: + + - template: azure/stage0/build-source-tarball.yml + - template: azure/stage0/build-macos-packages.yml + parameters: + Artifact: Release + BuildType: RelWithDebInfo + +################################# +########## STAGE 0 END ########## +################################# + + +################################# +######### STAGE 1 BEGIN ######### +################################# + +- stage: TestPackages_Stage1 + dependsOn: BuildPackages_Stage0 + condition: succeededOrFailed() + jobs: + + - job: TestTarball_Quick + timeoutInMinutes: 600 + workspace: + clean: all + pool: + vmImage: 'macOS-latest' + + steps: + + - checkout: none + + - task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: 'macOS-latest-Release' + downloadPath: '$(System.ArtifactsDirectory)' + + - bash: | + df -h + ls -la + pwd + mkdir test + tar xzf '$(System.ArtifactsDirectory)'/macOS-latest-Release/mariadb-*.tar.gz \ + --strip-components=1 -C $(Build.SourcesDirectory) + ls -la + rm -frv '$(System.ArtifactsDirectory)'/macOS-latest-Release/mariadb-*.tar.gz + displayName: "Unpack RelWithDebInfo Tarball" + + - template: azure/tests/mac-mtr-test.yml + parameters: + MtrArgs: '--verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --parallel=auto --vardir=$(MYSQL_VARDIR)' + TestName: 'MTR Normal Test - Release' + tarname: 'mtr-normal-test.tar.gz' + Artifact: Release + + - template: azure/tests/mac-mtr-test.yml + parameters: + MtrArgs: '--verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --parallel=auto --suite=spider,spider/bg,engines/funcs,engines/iuds --testcase-timeout=120 --mysqld=--open-files-limit=0 --mysqld=--log-warnings=1 --vardir=$(MYSQL_VARDIR)' + TestName: 'MTR Engines test' + tarname: 'mtr-engines-test.tar.gz' + Artifact: Release + + - template: azure/tests/mac-mtr-test.yml + parameters: + MtrArgs: '--verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --parallel=auto --suite=funcs_1,funcs_2,stress,jp --testcase-timeout=120 --mysqld=--open-files-limit=0 --mysqld=--log-warnings=1 --vardir=$(MYSQL_VARDIR)' + TestName: 'MTR Extra test' + tarname: 'mtr-extra-test.tar.gz' + Artifact: Release + + - template: azure/tests/mac-mtr-test.yml + parameters: + MtrArgs: '--verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --parallel=auto --ps-protocol --vardir=$(MYSQL_VARDIR)' + TestName: 'MTR PS Protocol test' + tarname: 'mtr-ps-protocol-test.tar.gz' + Artifact: Release + + - template: azure/tests/mac-mtr-test.yml + parameters: + MtrArgs: '--force --retry=3 --max-save-core=1 --vardir=${MYSQL_VARDIR} --max-save-datadir=1 --max-test-fail=20 --big-test --parallel=auto' + TestName: 'MTR -big test' + tarname: 'mtr-big-test.tar.gz' + Artifact: Release + + +# +################################# +########## STAGE 1 END ########## +################################# +# diff --git a/azure-pipelines-mtr.yml b/azure-pipelines-mtr.yml new file mode 100644 index 00000000000..edb6f033e90 --- /dev/null +++ b/azure-pipelines-mtr.yml @@ -0,0 +1,212 @@ +# C/C++ with GCC +# Build your C/C++ project with GCC using make. +# Add steps that publish test results, save build artifacts, deploy, and more: +# https://docs.microsoft.com/azure/devops/pipelines/apps/c-cpp/gcc + +resources: + containers: + + - container: rhel-6 + image: mariadbe.azurecr.io/build-es:rhel-6 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: rhel-7 + image: mariadbe.azurecr.io/build-es:rhel-7 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: rhel-8 + image: mariadbe.azurecr.io/build-es:rhel-8 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: sles-12 + image: mariadbe.azurecr.io/build-es:sles-12 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: sles-15 + image: mariadbe.azurecr.io/build-es:sles-15 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1604 + image: mariadbe.azurecr.io/build-es:ubuntu-1604 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1804 + image: mariadbe.azurecr.io/build-es:ubuntu-1804 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-8 + image: mariadbe.azurecr.io/build-es:debian-8 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-9 + image: mariadbe.azurecr.io/build-es:debian-9 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-10 + image: mariadbe.azurecr.io/build-es:debian-10 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + +variables: + TEST_RESULTS_FILE: 'test-results.log' + MTR_FEEDBACK_PLUGIN: 1 + DEBIAN_FRONTEND: noninteractive + MYSQL_VARDIR: '/var/tmp/mtr' + BUILDDIR: "$(Build.SourcesDirectory)/BUILD" + VERSION: '10.3' + +stages: +- stage: BuildPackages_Stage0 + jobs: + - template: azure/stage0/build-source-tarball.yml + - template: azure/stage0/build-binary-tarballs.yml + parameters: + BuildType: RelWithDebInfo + Artifact: Release + +################################# +######### STAGE 2 BEGIN ######### +################################# +# +# Run install test for RPM/DEB packages + +# +# Checks if binary packages are OK on all supported platforms +# Checks if functionality is OK +# Checks if replication is OK +# +- stage: TestPackages_Stage2 + dependsOn: BuildPackages_Stage0 + condition: succeededOrFailed() + + jobs: + + - job: TestTarball_2 + timeoutInMinutes: 600 + + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + rhel-6: + containerImage: rhel-6 + rhel-7: + containerImage: rhel-7 + rhel-8: + containerImage: rhel-8 + sles-12: + containerImage: sles-12 + sles-15: + containerImage: sles-15 + debian-8: + containerImage: debian-8 + debian-9: + containerImage: debian-9 + debian-10: + containerImage: debian-10 + ubuntu-1604: + containerImage: ubuntu-1604 + ubuntu-1804: + containerImage: ubuntu-1804 + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: azure/install-galera.yml + - template: azure/fetch-unpack-tarball.yml + + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--force --retry=3 --max-save-core=1 --vardir=${MYSQL_VARDIR} --max-save-datadir=1 --max-test-fail=20 --big-test --parallel=auto' + TestName: 'MTR -big test' + tarname: 'mtr-big-test.tar.gz' + Artifact: Release + + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--force --parallel=auto --suite=galera,wsrep --max-test-fail=0 --testcase-timeout=120 --vardir=$(MYSQL_VARDIR) --big-test' + TestName: 'MTR Galera -big test' + tarname: 'mtr-galera-big-test.tar.gz' + Artifact: Release + + +- stage: TestPackages_Stage3 + dependsOn: BuildPackages_Stage0 + condition: succeededOrFailed() + + jobs: + + - job: TestTarball_3 + timeoutInMinutes: 600 + + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + rhel-6: + containerImage: rhel-6 + rhel-7: + containerImage: rhel-7 + rhel-8: + containerImage: rhel-8 + sles-12: + containerImage: sles-12 + sles-15: + containerImage: sles-15 + debian-8: + containerImage: debian-8 + debian-9: + containerImage: debian-9 + ubuntu-1604: + containerImage: ubuntu-1604 + ubuntu-1804: + containerImage: ubuntu-1804 + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: azure/fetch-unpack-tarball.yml + + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --parallel=auto --vardir=$(MYSQL_VARDIR)' + TestName: 'MTR Normal test' + tarname: 'mtr-normal-test.tar.gz' + Artifact: Release + + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --parallel=auto --ps-protocol --vardir=$(MYSQL_VARDIR)' + TestName: 'MTR PS Protocol test' + tarname: 'mtr-ps-protocol-test.tar.gz' + Artifact: Release + + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --parallel=auto --suite=funcs_1,funcs_2,stress,jp --testcase-timeout=120 --mysqld=--open-files-limit=0 --mysqld=--log-warnings=1 --vardir=$(MYSQL_VARDIR)' + TestName: 'MTR Extra test' + tarname: 'mtr-extra-test.tar.gz' + Artifact: Release + + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --parallel=auto --suite=spider,spider/bg,engines/funcs,engines/iuds --testcase-timeout=120 --mysqld=--open-files-limit=0 --mysqld=--log-warnings=1 --vardir=$(MYSQL_VARDIR)' + TestName: 'MTR Engines test' + tarname: 'mtr-engines-test.tar.gz' + Artifact: Release + diff --git a/azure-pipelines-upgrade.yml b/azure-pipelines-upgrade.yml new file mode 100644 index 00000000000..db4bc5ca9ed --- /dev/null +++ b/azure-pipelines-upgrade.yml @@ -0,0 +1,164 @@ +# C/C++ with GCC +# Build your C/C++ project with GCC using make. +# Add steps that publish test results, save build artifacts, deploy, and more: +# https://docs.microsoft.com/azure/devops/pipelines/apps/c-cpp/gcc + +resources: + containers: + + - container: rhel-6 + image: mariadbe.azurecr.io/build-es:rhel-6 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: rhel-7 + image: mariadbe.azurecr.io/build-es:rhel-7 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: rhel-8 + image: mariadbe.azurecr.io/build-es:rhel-8 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: sles-12 + image: mariadbe.azurecr.io/build-es:sles-12 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: sles-15 + image: mariadbe.azurecr.io/build-es:sles-15 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1604 + image: mariadbe.azurecr.io/build-es:ubuntu-1604 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1804 + image: mariadbe.azurecr.io/build-es:ubuntu-1804 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-8 + image: mariadbe.azurecr.io/build-es:debian-8 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-9 + image: mariadbe.azurecr.io/build-es:debian-9 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-10 + image: mariadbe.azurecr.io/build-es:debian-10 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + +variables: + BUILDDIR: "$(Build.SourcesDirectory)/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE" + DEB_LOCAL_LIST: '/etc/apt/sources.list.d/localInstall.list' + MYSQL_PASSWORD: 'tESt123%_password' + ESTOKEN: '49bc85f8-6e90-443d-901a-b7f01a4ab22c' + CS_REPO_URL: 'https://downloads.mariadb.com/MariaDB/mariadb_repo_setup' + ES_REPO_URL: 'https://dlm.mariadb.com/enterprise-release-helpers/mariadb_es_repo_setup' + +stages: + +- stage: BuildPackages + jobs: + - template: azure/stage0/build-source-tarball.yml + - template: azure/stage0/build-rpm-packages.yml + - template: azure/stage0/build-deb-packages.yml + +- stage: Upgrade_CS_10_2_to_ES_10_4 + jobs: + - template: azure/stage3-upgrade.yml + parameters: + VERSION: '10.2' + FROM: 'CS' + FROMVERSION: '10_2' + TO: 'ES' + TOVERSION: '10_4' + REPO: $(CS_REPO_URL) + REPO_PARAMS: '--mariadb-server-version=mariadb-10.2' + +- stage: Upgrade_CS_10_3_to_ES_10_4 + condition: succeededOrFailed() + jobs: + - template: azure/stage3-upgrade.yml + parameters: + VERSION: '10.3' + FROM: 'CS' + FROMVERSION: '10_3' + TO: 'ES' + TOVERSION: '10_4' + REPO: $(CS_REPO_URL) + REPO_PARAMS: '--mariadb-server-version=mariadb-10.3' + +- stage: Upgrade_CS_10_4_to_ES_10_4 + condition: succeededOrFailed() + jobs: + - template: azure/stage3-upgrade.yml + parameters: + VERSION: '10.4' + FROM: 'CS' + FROMVERSION: '10_4' + TO: 'ES' + TOVERSION: '10_4' + REPO: $(CS_REPO_URL) + REPO_PARAMS: '--mariadb-server-version=mariadb-10.4' + + +- stage: Upgrade_ES_10_2_to_ES_10_4 + condition: succeededOrFailed() + jobs: + - template: azure/stage3-upgrade.yml + parameters: + VERSION: '10.2' + FROM: 'ES' + FROMVERSION: '10_2' + TO: 'ES' + TOVERSION: '10_4' + REPO: $(ES_REPO_URL) + REPO_PARAMS: '--token=$(ESTOKEN) --apply --verbose --mariadb-server-version=10.2' + +- stage: Upgrade_ES_10_3_to_ES_10_4 + condition: succeededOrFailed() + jobs: + - template: azure/stage3-upgrade.yml + parameters: + VERSION: '10.3' + FROM: 'ES' + FROMVERSION: '10_3' + TO: 'ES' + TOVERSION: '10_4' + REPO: $(ES_REPO_URL) + REPO_PARAMS: '--token=$(ESTOKEN) --apply --verbose --mariadb-server-version=10.3' + +- stage: Upgrade_ES_10_4_to_ES_10_4 + condition: succeededOrFailed() + jobs: + - template: azure/stage3-upgrade.yml + parameters: + VERSION: '10.4' + FROM: 'ES' + FROMVERSION: '10_4' + TO: 'ES' + TOVERSION: '10_4' + REPO: $(ES_REPO_URL) + REPO_PARAMS: '--token=$(ESTOKEN) --apply --verbose --mariadb-server-version=10.4' + +- stage: RollingUpgrade_ES_10_2_to_ES_10_4 + condition: succeededOrFailed() + jobs: + - template: azure/stage3-rolling-upgrade.yml + parameters: + VERSIONS: [ '10.2', '10.3', '10.4' ] + + + + + + diff --git a/azure-pipelines.yml b/azure-pipelines.yml new file mode 100644 index 00000000000..4a5318c78b9 --- /dev/null +++ b/azure-pipelines.yml @@ -0,0 +1,180 @@ +# C/C++ with GCC +# Build your C/C++ project with GCC using make. +# Add steps that publish test results, save build artifacts, deploy, and more: +# https://docs.microsoft.com/azure/devops/pipelines/apps/c-cpp/gcc + +resources: + containers: + + - container: rhel-6 + image: mariadbe.azurecr.io/build-es:rhel-6 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: rhel-7 + image: mariadbe.azurecr.io/build-es:rhel-7 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: rhel-8 + image: mariadbe.azurecr.io/build-es:rhel-8 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: sles-12 + image: mariadbe.azurecr.io/build-es:sles-12 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: sles-15 + image: mariadbe.azurecr.io/build-es:sles-15 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1604 + image: mariadbe.azurecr.io/build-es:ubuntu-1604 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-1804 + image: mariadbe.azurecr.io/build-es:ubuntu-1804 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: ubuntu-2004 + image: mariadbe.azurecr.io/build-es:ubuntu-2004 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-8 + image: mariadbe.azurecr.io/build-es:debian-8 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-9 + image: mariadbe.azurecr.io/build-es:debian-9 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + + - container: debian-10 + image: mariadbe.azurecr.io/build-es:debian-10 + endpoint: acr-mariadbe-connection + options: --shm-size 12G + +variables: + configuration: RelWithDebInfo + platform: x64 + TEST_RESULTS_FILE: 'test-results.log' + MTR_FEEDBACK_PLUGIN: 1 + DEBIAN_FRONTEND: noninteractive + MYSQL_VARDIR: '/var/tmp/mtr' + DEB_LOCAL_LIST: '/etc/apt/sources.list.d/localInstall.list' + BUILDDIR: "$(Build.SourcesDirectory)/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE" +# +stages: + +- stage: BuildPackages_Stage0 + + jobs: + + - template: azure/stage0/build-source-tarball.yml + - template: azure/stage0/build-binary-tarballs.yml + parameters: + BuildType: RelWithDebInfo + Artifact: Release + + - template: azure/stage0/build-windows-packages.yml + parameters: + Artifact: Release + BuildType: RelWithDebInfo + + - template: azure/stage0/build-macos-packages.yml + parameters: + Artifact: Release + BuildType: RelWithDebInfo + +################################# +########## STAGE 0 END ########## +################################# + + +################################# +######### STAGE 1 BEGIN ######### +################################# + +- stage: TestPackages_Stage1 + dependsOn: BuildPackages_Stage0 + condition: succeededOrFailed() + jobs: + + - job: TestTarball_Quick + timeoutInMinutes: 240 + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + rhel-7: + containerImage: rhel-7 + rhel-8: + containerImage: rhel-8 + ubuntu-1804: + containerImage: ubuntu-1804 + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: azure/install-galera.yml + - template: azure/fetch-unpack-tarball.yml + parameters: + Artifact: Release + + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: '--force --verbose-restart --skip-test-list=unstable-tests --parallel=auto --vardir=$(MYSQL_VARDIR)' + TestName: 'Quick Test - Release' + tarname: 'mtr-quick-test.tar.gz' + Artifact: Release + + - template: azure/tests/run-mtr-test.yml + parameters: + MtrArgs: "--force --parallel=auto --suite=galera,galera_sr,wsrep --max-test-fail=0 --testcase-timeout=120 --vardir=$(MYSQL_VARDIR) --big-test" + TestName: 'Galera --big-test - Release' + tarname: 'galera-big-test.tar.gz' + Artifact: Release + + - job: TestMSI_Quick + timeoutInMinutes: 10 + pool: + vmImage: 'windows-2019' + + steps: + + - checkout: none + - task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: Windows-Release + downloadPath: '$(System.ArtifactsDirectory)' + + - powershell: | + Set-PSDebug -Trace 1 + $MSI=(Get-Childitem -Path $(System.ArtifactsDirectory)\Windows-Release -Filter *.msi).fullname + Start-Process msiexec.exe -Wait -ArgumentList "/I $MSI PASSWORD=tESt123%_password SERVICENAME=MariaDB-Enterprise /qn /l*v .\msi-install-log.txt" + + $ProgramFilesPath = [Environment]::GetEnvironmentVariable("ProgramW6432") + Get-Childitem -Path $ProgramFilesPath -Recurse -Filter mysql.exe + + Get-Content -Path .\msi-install-log.txt + displayName: "Install Windows MSI package" + + - template: azure/tests/win-select-insert.yml +# +################################# +########## STAGE 1 END ########## +################################# +# diff --git a/azure/fetch-unpack-tarball.yml b/azure/fetch-unpack-tarball.yml new file mode 100644 index 00000000000..21744c963a4 --- /dev/null +++ b/azure/fetch-unpack-tarball.yml @@ -0,0 +1,24 @@ + +parameters: + Artifact: Release + +steps: + +- task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: '$(containerImage)-${{ parameters.Artifact }}' + downloadPath: '$(System.ArtifactsDirectory)' + +- bash: | + df -h + ls -la + pwd + mkdir test + tar xzf '$(System.ArtifactsDirectory)'/$(containerImage)-${{ parameters.Artifact }}/mariadb-*.tar.gz \ + --strip-components=1 -C $(Build.SourcesDirectory) + ls -la + rm -frv '$(System.ArtifactsDirectory)'/$(containerImage)-${{ parameters.Artifact }}/mariadb-*.tar.gz + displayName: "Unpack ${{ parameters.Artifact }} Tarball" diff --git a/azure/install-galera.yml b/azure/install-galera.yml new file mode 100644 index 00000000000..fdc9b3cc7a3 --- /dev/null +++ b/azure/install-galera.yml @@ -0,0 +1,39 @@ +steps: + + - bash: | + set -x + + if [[ -e /usr/bin/apt-get ]]; then + sudo apt update + sudo apt install -y dirmngr lsb-release + DEBIAN_VERSION=$(lsb_release -sc) + sudo bash -c "echo deb http://downloads.mariadb.com/galera-test/repo4/deb ${DEBIAN_VERSION} main > /etc/apt/sources.list.d/galera.list" + sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xCE1A3DD5E3C94F49 + sudo apt update + DEBIAN_FRONTEND=noninteractive sudo apt install -y galera-enterprise-4 galera-arbitrator-4 rsync netcat socat + fi + + if [[ -e /usr/bin/yum ]]; then + sudo sh -c "echo '[galera]' > /etc/yum.repos.d/galera.repo" + sudo sh -c "echo 'name=galera' >> /etc/yum.repos.d/galera.repo" + sudo sh -c "echo 'baseurl=http://downloads.mariadb.com/galera-test/repo4/rpm/rhel/\$releasever/\$basearch/' >> /etc/yum.repos.d/galera.repo" + sudo sh -c "echo 'gpgkey=https://downloads.mariadb.com/MariaDB/RPM-GPG-KEY-MariaDB-Ent' >> /etc/yum.repos.d/galera.repo" + sudo sh -c "echo 'gpgcheck=1' >> /etc/yum.repos.d/galera.repo" + sudo cat /etc/yum.repos.d/galera.repo + sudo yum -y clean all + sudo yum install -y galera-enterprise-4 rsync socat lsof + fi + + if [[ -e /usr/bin/zypper ]]; then + sudo zypper -n in wget || true + source /etc/os-release + RELEASE=${VERSION_ID%.*} + wget https://downloads.mariadb.com/MariaDB/RPM-GPG-KEY-MariaDB-Ent -O /tmp/rpm.key + sudo rpm --import /tmp/rpm.key && rm -f /tmp/rpm.key + sudo zypper rr Galera-Enterprise || true + sudo zypper ar -f -g http://downloads.mariadb.com/galera-test/repo4/rpm/sles/${RELEASE}/x86_64/ Galera-Enterprise + sudo zypper refresh -sf + sudo zypper -n in galera-enterprise-4 rsync socat lsof + fi + displayName: "Install Galera-Enterprise-4" + condition: succeededOrFailed() diff --git a/azure/prepare-vm-for-installation.yml b/azure/prepare-vm-for-installation.yml new file mode 100644 index 00000000000..aaa1ff7ea4e --- /dev/null +++ b/azure/prepare-vm-for-installation.yml @@ -0,0 +1,81 @@ +parameters: + CLEAN: yes + +steps: + - bash: | + set -x + + if [[ -e /etc/redhat-release ]]; then + sudo yum -y install patch + sudo yum -y upgrade + sudo yum -y clean all + sudo rm -fv /etc/yum.repos.d/mariadb.repo* + ls -la /etc/yum.repos.d/ + fi + + if [[ -e /etc/debian_version ]]; then + export DEBIAN_FRONTEND=noninteractive + sudo bash -c "rm -fv ${DEB_LOCAL_LIST} /etc/apt/sources.list.d/mariadb.list*" + sudo apt-get update + sudo apt-get -y -f -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade + sudo apt-get -y -f install debconf-utils dpkg-dev zlib1g-dev curl patch + # + echo mariadb-server-10.2 mariadb-server-10.2/postrm_remove_databases boolean false | sudo debconf-set-selections + echo mariadb-server-10.2 mysql-server/root_password password $(MYSQL_PASSWORD) | sudo debconf-set-selections + echo mariadb-server-10.2 mysql-server/root_password_again password $(MYSQL_PASSWORD) | sudo debconf-set-selections + # + echo mariadb-server-10.3 mariadb-server-10.3/postrm_remove_databases boolean false | sudo debconf-set-selections + echo mariadb-server-10.3 mysql-server/root_password password $(MYSQL_PASSWORD) | sudo debconf-set-selections + echo mariadb-server-10.3 mysql-server/root_password_again password $(MYSQL_PASSWORD) | sudo debconf-set-selections + # + echo mariadb-server-10.4 mariadb-server-10.4/postrm_remove_databases boolean false | sudo debconf-set-selections + echo mariadb-server-10.4 mysql-server/root_password password $(MYSQL_PASSWORD) | sudo debconf-set-selections + echo mariadb-server-10.4 mysql-server/root_password_again password $(MYSQL_PASSWORD) | sudo debconf-set-selections + # + echo mariadb-server-10.5 mariadb-server-10.5/postrm_remove_databases boolean false | sudo debconf-set-selections + echo mariadb-server-10.5 mysql-server/root_password password $(MYSQL_PASSWORD) | sudo debconf-set-selections + echo mariadb-server-10.5 mysql-server/root_password_again password $(MYSQL_PASSWORD) | sudo debconf-set-selections + # + fi + displayName: "Prepare and clean repo data" + + - bash: | + set -ex + uname -a + + SHOULDRUN=${{ parameters.CLEAN }} + [[ ${SHOULDRUN} == no ]] && exit 0 + + INSTALLED="" + + if [[ -e /etc/redhat-release ]]; then + cat /etc/*release + INSTALLED=$(rpm -qa | grep -iE "^maria|^mysql|^percona|^galera" ||:) + [[ -n "${INSTALLED}" ]] && sudo yum -y erase ${INSTALLED} + fi + + if [[ -e /etc/SuSE-release ]]; then + cat /etc/*release + INSTALLED=$(rpm -qa | grep -iE "^maria|^mysql|^percona|^galera" ||:) + [[ -n "${INSTALLED}" ]] && sudo zypper -n remove ${INSTALLED} + fi + + if [[ -e /etc/debian_version ]]; then + cat /etc/*_version + INSTALLED=$(dpkg -l | awk '{print $2}' | grep -iE "^maria|^mysql|^percona|^galera" ||:) + if [[ -n "${INSTALLED}" ]]; then + export DEBIAN_FRONTEND=noninteractive + until sudo apt-get -y -f purge ${INSTALLED}; do + sleep 10 + done + sudo apt-get -y -f autoremove + fi + fi + + [[ -n "$(pidof mysqld)" ]] && sudo pkill -9 mysqld + sudo rm -frv /var/lib/mysql /usr/share/mysql* /etc/my.cnf* /etc/mysql* + sudo userdel bombalurina ||: + sudo rm -frv /dev/shm/var* + sudo rm -frv ${MYSQL_VARDIR} + sudo rm -fv ${TEST_RESULTS_FILE} + displayName: "Uninstall existing MariaDB packages" diff --git a/azure/run-mtr-suites.yml b/azure/run-mtr-suites.yml new file mode 100644 index 00000000000..269b7793aab --- /dev/null +++ b/azure/run-mtr-suites.yml @@ -0,0 +1,77 @@ + +steps: + + - bash: | + set -x + MYSQLTEST=$(find /usr/share -name 'mysql-test' ||:) + pushd "${MYSQLTEST}" + + if test -f suite/plugins/pam/pam_mariadb_mtr.so; then + for p in /lib*/security /lib*/*/security ; do + test -f $p/pam_unix.so && sudo cp -v suite/plugins/pam/pam_mariadb_mtr.so $p/ + done + sudo cp -v suite/plugins/pam/mariadb_mtr /etc/pam.d/ + if ! perl mysql-test-run.pl --verbose-restart --force --mem --suite=plugins --do-test=pam --vardir=${MYSQL_VARDIR}/pam; then + RES=1 + ERRMSG="- MTR PAM tests failed" + fi + else + ERRMSG="- No pam_mariadb_mtr.so library found!" + fi + popd + [[ -n "${ERRMSG}" ]] && echo "${ERRMSG}" >> ${TEST_RESULTS_FILE} + displayName: "PAM MTR Test" + failOnStderr: false + + - bash: | + MYSQLTEST=$(find /usr/share -name 'mysql-test' ||:) + cd "${MYSQLTEST}" + perl mysql-test-run.pl --verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --mem --parallel=4 --vardir=${MYSQL_VARDIR}/normal + displayName: "MTR Normal Test" + failOnStderr: false + + - bash: | + MYSQLTEST=$(find /usr/share -name 'mysql-test' ||:) + cd "${MYSQLTEST}" + perl mysql-test-run.pl --verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --mem --parallel=4 --ps-protocol --vardir=${MYSQL_VARDIR}/psproto + displayName: "MTR PS Protocol test" + failOnStderr: false + + - bash: | + MYSQLTEST=$(find /usr/share -name 'mysql-test' ||:) + cd "${MYSQLTEST}" + perl mysql-test-run.pl --verbose-restart --force --max-save-core=0 --max-save-datadir=1 --skip-test='rocksdb\..*' --big --big --parallel=2 --testcase-timeout=120 --vardir=${MYSQL_VARDIR}/big_nops + displayName: "MTR --big --big Test, no ps-protocol" + failOnStderr: false + + - bash: | + MYSQLTEST=$(find /usr/share -name 'mysql-test' ||:) + cd "${MYSQLTEST}" + perl mysql-test-run.pl --verbose-restart --big --big --ps-protocol --force --retry=3 --max-save-core=0 --max-save-datadir=1 --mem --parallel=4 --ps-protocol --vardir=${MYSQL_VARDIR}/big_ps + displayName: "MTR --big --big test, +ps-protocol" + failOnStderr: false + + - bash: | + MYSQLTEST=$(find /usr/share -name 'mysql-test' ||:) + cd "${MYSQLTEST}" + perl mysql-test-run.pl --verbose-restart --big --big --force --retry=3 --max-save-core=0 --max-save-datadir=1 --mem --parallel=4 --testcase-timeout=120 \ + --suites=galera,wsrep --vardir=${MYSQL_VARDIR}/galera4_big + displayName: "MTR Galera --big --big test" + failOnStderr: false + + - bash: | + MYSQLTEST=$(find /usr/share -name 'mysql-test' ||:) + cd "${MYSQLTEST}" + perl mysql-test-run.pl --verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --mem \ + --parallel=2 --suite=funcs_1,funcs_2,stress,jp --testcase-timeout=120 --mysqld=--open-files-limit=0 --mysqld=--log-warnings=1 --vardir=${MYSQL_VARDIR}/xtra + displayName: "MTR Extra test" + failOnStderr: false + + + - bash: | + MYSQLTEST=$(find /usr/share -name 'mysql-test' ||:). + cd "${MYSQLTEST}" + perl mysql-test-run.pl --verbose-restart --force --retry=3 --max-save-core=0 --max-save-datadir=1 --mem \ + --parallel=2 --suite=spider,spider/bg,engines/funcs,engines/iuds --testcase-timeout=120 --mysqld=--open-files-limit=0 --mysqld=--log-warnings=1 --vardir=${MYSQL_VARDIR}/engines + displayName: "MTR Engines test" + failOnStderr: false diff --git a/azure/stage0/build-binary-tarballs-asan.yml b/azure/stage0/build-binary-tarballs-asan.yml new file mode 100644 index 00000000000..2cce23b82c1 --- /dev/null +++ b/azure/stage0/build-binary-tarballs-asan.yml @@ -0,0 +1,58 @@ +parameters: + BuildType: '' + CMakeArgs: '' + Artifact: '' + +jobs: + +- job: BuildTarballs_${{ parameters.Artifact }}_${{ parameters.BuildType }} + timeoutInMinutes: 180 + dependsOn: BuildSourceTarball + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + ubuntu-1904: + containerImage: ubuntu-2004 + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: extract-source-tarball.yml + + - bash: | + set -x + df -h + ls -la + pwd + cd ${BUILDDIR} + mkdir build + + - bash: | + set -x + cd ${BUILDDIR}/build + CC=gcc-10 CXX=g++-10 cmake .. -DBUILD_CONFIG=enterprise \ + -DCMAKE_BUILD_TYPE=${{ parameters.BuildType }} \ + -DMYSQL_MAINTAINER_MODE=OFF ${{ parameters.CMakeArgs }} + displayName: 'CMake configure ${{ parameters.BuildType }} ${{ parameters.CMakeArgs }}' + + - bash: | + set -x + cd ${BUILDDIR}/build + make -j`grep -c processor /proc/cpuinfo` package VERBOSE=1 + displayName: 'Build ${{ parameters.BuildType }} ${{ parameters.CMakeArgs }} Tarball' + + - task: CopyFiles@2 + timeoutInMinutes: 2 + inputs: + targetFolder: '$(Build.ArtifactStagingDirectory)' + sourceFolder: "$(BUILDDIR)/build" + contents: '?(*.tar.gz)' + + - task: PublishBuildArtifacts@1 + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: '$(containerImage)-${{ parameters.Artifact }}-${{ parameters.BuildType }}' diff --git a/azure/stage0/build-binary-tarballs.yml b/azure/stage0/build-binary-tarballs.yml new file mode 100644 index 00000000000..7f7af10a53f --- /dev/null +++ b/azure/stage0/build-binary-tarballs.yml @@ -0,0 +1,89 @@ +parameters: + BuildType: '' + CMakeArgs: '' + Artifact: '' + +jobs: + +- job: BuildTarballs_${{ parameters.Artifact }} + timeoutInMinutes: 180 + dependsOn: BuildSourceTarball + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + ubuntu-1604: + containerImage: ubuntu-1604 + ubuntu-1804: + containerImage: ubuntu-1804 + ubuntu-2004: + containerImage: ubuntu-2004 + debian-9: + containerImage: debian-9 + debian-10: + containerImage: debian-10 + rhel-6: + containerImage: rhel-6 + rhel-7: + containerImage: rhel-7 + rhel-8: + containerImage: rhel-8 + sles-12: + containerImage: sles-12 + sles-15: + containerImage: sles-15 + + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: extract-source-tarball.yml + + - bash: | + set -x + df -h + ls -la + pwd + cd ${BUILDDIR} + mkdir build + + - bash: | + cd ${BUILDDIR}/build + cmake .. \ + -DCMAKE_C_COMPILER=/opt/rh/devtoolset-3/root/usr/bin/gcc \ + -DCMAKE_CXX_COMPILER=/opt/rh/devtoolset-3/root/usr/bin/g++ \ + -DBUILD_CONFIG=enterprise -DCMAKE_BUILD_TYPE=${{ parameters.BuildType }} \ + -DMYSQL_MAINTAINER_MODE=OFF ${{ parameters.CMakeArgs }} + displayName: 'CMake configure ${{ parameters.BuildType }} ${{ parameters.CMakeArgs }} rhel-6' + condition: eq(variables['containerImage'], 'rhel-6') + + - bash: | + set -x + cd ${BUILDDIR}/build + cmake .. \ + -DBUILD_CONFIG=enterprise \ + -DCMAKE_BUILD_TYPE=${{ parameters.BuildType }} \ + -DMYSQL_MAINTAINER_MODE=OFF ${{ parameters.CMakeArgs }} + displayName: 'CMake configure ${{ parameters.BuildType }} ${{ parameters.CMakeArgs }}' + condition: ne(variables['containerImage'], 'rhel-6') + + - bash: | + set -x + cd ${BUILDDIR}/build + make -j`grep -c processor /proc/cpuinfo` package VERBOSE=1 + displayName: 'Build ${{ parameters.BuildType }} ${{ parameters.CMakeArgs }} Tarball' + + - task: CopyFiles@2 + timeoutInMinutes: 2 + inputs: + targetFolder: '$(Build.ArtifactStagingDirectory)' + sourceFolder: "$(BUILDDIR)/build" + contents: '?(*.tar.gz)' + + - task: PublishBuildArtifacts@1 + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: '$(containerImage)-${{ parameters.Artifact }}' diff --git a/azure/stage0/build-deb-packages.yml b/azure/stage0/build-deb-packages.yml new file mode 100644 index 00000000000..40393471a78 --- /dev/null +++ b/azure/stage0/build-deb-packages.yml @@ -0,0 +1,66 @@ + +jobs: + +- job: BuildDEBs + timeoutInMinutes: 180 + dependsOn: BuildSourceTarball + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + ubuntu-1604: + containerImage: ubuntu-1604 + ubuntu-1804: + containerImage: ubuntu-1804 + ubuntu-2004: + containerImage: ubuntu-2004 + debian-9: + containerImage: debian-9 + debian-10: + containerImage: debian-10 + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: extract-source-tarball.yml + - script: | + set -x + cd ${BUILDDIR} + sed s/"dch -b"/"dch -b --force-distribution"/g -i debian/autobake-deb.sh + condition: eq(variables['containerImage'], 'debian-8') + displayName: 'Fix for Debian 8' + + - script: | + set -x + cd ${BUILDDIR} + debian/autobake-deb.sh + condition: eq( variables['Agent.OS'], 'Linux' ) + displayName: 'Build packages' + + - script: | + set -x + cd ${BUILDDIR} + pwd + mkdir -vp $BUILD_ARTIFACTSTAGINGDIRECTORY/binary + mkdir -vp $BUILD_ARTIFACTSTAGINGDIRECTORY/source + ls -lah .. + echo $BUILD_SOURCESDIRECTORY + echo $BUILD_ARTIFACTSTAGINGDIRECTORY + cp -v ../*.deb $BUILD_ARTIFACTSTAGINGDIRECTORY/binary/ + cp -v ../*.changes $BUILD_ARTIFACTSTAGINGDIRECTORY/binary/ + cp -v ../*.ddeb $BUILD_ARTIFACTSTAGINGDIRECTORY/binary/ + cp -v ../*.buildinfo $BUILD_ARTIFACTSTAGINGDIRECTORY/binary/ + cp -v ../*.xz $BUILD_ARTIFACTSTAGINGDIRECTORY/binary/ + cp -v ../*.dsc $BUILD_ARTIFACTSTAGINGDIRECTORY/binary/ + cd $BUILD_ARTIFACTSTAGINGDIRECTORY + dpkg-scanpackages binary /dev/null | gzip -9c > binary/Packages.gz + dpkg-scansources source /dev/null | gzip -9c > source/Sources.gz + displayName: 'Create repo structure' + + - task: PublishBuildArtifacts@1 + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: $(containerImage)-DEBS diff --git a/azure/stage0/build-macos-packages.yml b/azure/stage0/build-macos-packages.yml new file mode 100644 index 00000000000..cebdfb9dc3c --- /dev/null +++ b/azure/stage0/build-macos-packages.yml @@ -0,0 +1,47 @@ +parameters: + BuildType: '' + CMakeArgs: '' + Artifact: '' + +jobs: + +- job: BuildMacPackages_${{ parameters.Artifact }} + dependsOn: BuildSourceTarball + workspace: + clean: all + pool: + vmImage: 'macOS-latest' + + steps: + + - template: extract-source-tarball.yml + + - checkout: self + submodules: true + + - script: | + brew install jemalloc mhash curl gnutls + displayName: "brew install dependencies" + + - script: | + cmake . -DCMAKE_BUILD_TYPE=${{ parameters.BuildType }} \ + -DWITH_SSL=/usr/local/opt/openssl@1.1 -DBUILD_CONFIG=enterprise + displayName: 'cmake configure' + failOnStderr: false + + - script: make -j5 package + displayName: 'make' + failOnStderr: false + +# #- script: cd support-files/macOSpkg && ./mkpkg ../../mariadb*tar.gz && mv -vi *.pkg ../../ + + - task: CopyFiles@2 + inputs: + targetFolder: '$(Build.ArtifactStagingDirectory)' + sourceFolder: '$(Build.SourcesDirectory)' + contents: '?(*.tar.gz|*.pkg)' + + - task: PublishBuildArtifacts@1 + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: 'macOS-latest-${{ parameters.Artifact }}' diff --git a/azure/stage0/build-rpm-packages.yml b/azure/stage0/build-rpm-packages.yml new file mode 100644 index 00000000000..17bf770b835 --- /dev/null +++ b/azure/stage0/build-rpm-packages.yml @@ -0,0 +1,111 @@ + +jobs: + +- job: BuildRPMs + timeoutInMinutes: 180 + dependsOn: BuildSourceTarball + workspace: + clean: all + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + rhel-6: + containerImage: rhel-6 + rhel-7: + containerImage: rhel-7 + rhel-8: + containerImage: rhel-8 + sles-12: + containerImage: sles-12 + sles-15: + containerImage: sles-15 + + container: $[variables['containerImage']] + + steps: + + - checkout: none + - template: extract-source-tarball.yml + + + - bash: | + set -x + cd "${BUILDDIR}" + curl -o ../MariaDB-shared-5.3.x.rpm \ + http://yum.mariadb.org/5.3/centos5-amd64/rpms/MariaDB-shared-5.3.12-122.el5.x86_64.rpm + curl -o ../MariaDB-shared-10.1.x.rpm \ + http://yum.mariadb.org/10.1/centos6-amd64/rpms/MariaDB-10.1.41-centos6-x86_64-shared.rpm + displayName: 'Fetch Compat RPMs for for RHEL-6' + condition: eq(variables['containerImage'], 'rhel-6') + + - bash: | + set -x + cd "${BUILDDIR}" + curl -o ../MariaDB-shared-5.3.x.rpm \ + http://yum.mariadb.org/5.3/centos5-amd64/rpms/MariaDB-shared-5.3.12-122.el5.x86_64.rpm + curl -o ../MariaDB-shared-10.1.x.rpm \ + http://yum.mariadb.org/10.1/centos7-amd64/rpms/MariaDB-shared-10.1.41-1.el7.centos.x86_64.rpm + displayName: 'Fetch Compat RPMs for RHEL-7' + condition: eq(variables['containerImage'], 'rhel-7') + + - bash: | + set -x + cd "${BUILDDIR}" + curl -o ../MariaDB-shared-5.3.x.rpm \ + http://yum.mariadb.org/5.3/centos5-amd64/rpms/MariaDB-shared-5.3.12-122.el5.x86_64.rpm + curl -o ../MariaDB-shared-10.1.x.rpm \ + http://yum.mariadb.org/10.1/sles12-amd64/rpms/MariaDB-shared-10.1.41-1.x86_64.rpm + displayName: 'Fetch Compat RPMs for SLES 12 & 15' + condition: in(variables['containerImage'], 'sles-12', 'sles-15') + + - bash: | + set -x + cd "${BUILDDIR}" + PLATFORM=$(containerImage) + cmake . -DCMAKE_C_COMPILER=/opt/rh/devtoolset-3/root/usr/bin/gcc \ + -DCMAKE_CXX_COMPILER=/opt/rh/devtoolset-3/root/usr/bin/g++ \ + -DRPM=${PLATFORM/-/} -DBUILD_CONFIG=enterprise + displayName: 'CMake configure RPM RHEL 6' + condition: eq(variables['containerImage'], 'rhel-6') + + - bash: | + set -x + cd "${BUILDDIR}" + PLATFORM=$(containerImage) + cmake . -DRPM=${PLATFORM/-/} -DBUILD_CONFIG=enterprise + displayName: 'CMake configure RPM' + condition: ne(variables['containerImage'], 'rhel-6') + + - bash: | + set -x + cd "${BUILDDIR}" + make -j`grep -c processor /proc/cpuinfo` package VERBOSE=1 + displayName: 'Build RPM package' + + - bash: | + set -x + cd "${BUILDDIR}" + pwd + find . -type f -name "*.rpm" + ls -la + echo '$(Build.SourcesDirectory)' + echo '$(Build.ArtifactStagingDirectory)' + condition: succeeded() + + - bash: | + set -x + find -L . -type l -print -delete + displayName: 'Workaround for CopyFiles' + + - task: CopyFiles@2 + inputs: + targetFolder: '$(Build.ArtifactStagingDirectory)' + sourceFolder: '$(BUILDDIR)' + contents: '?(*.rpm)' + condition: succeeded() + + - task: PublishBuildArtifacts@1 + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: $(containerImage)-RPMS diff --git a/azure/stage0/build-source-tarball.yml b/azure/stage0/build-source-tarball.yml new file mode 100644 index 00000000000..56d8c1720b4 --- /dev/null +++ b/azure/stage0/build-source-tarball.yml @@ -0,0 +1,31 @@ + +jobs: + +- job: BuildSourceTarball + pool: + vmImage: 'ubuntu-16.04' + strategy: + matrix: + ubuntu-1804: + containerImage: ubuntu-1804 + container: $[variables['containerImage']] + + steps: + + - checkout: self + submodules: true + + - bash: | + set -x + whoami + pwd + mkdir dist && cd dist + cmake .. -DBUILD_CONFIG=enterprise + make dist + cp *.tar.gz $BUILD_ARTIFACTSTAGINGDIRECTORY + displayName: Build Source Tarball + + - task: PublishBuildArtifacts@1 + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: source diff --git a/azure/stage0/build-windows-packages.yml b/azure/stage0/build-windows-packages.yml new file mode 100644 index 00000000000..77ac5859975 --- /dev/null +++ b/azure/stage0/build-windows-packages.yml @@ -0,0 +1,84 @@ +parameters: + BuildType: '' + Artifact: '' + +jobs: + +- job: BuildWindowsPackages_${{ parameters.BuildType }} + timeoutInMinutes: 180 + dependsOn: BuildSourceTarball + workspace: + clean: all + pool: + vmImage: 'windows-2019' + + steps: + + - checkout: none + - task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: source + downloadPath: '$(System.ArtifactsDirectory)' + + - powershell: | + Invoke-WebRequest -Uri https://cygwin.com/setup-x86_64.exe -OutFile setup-x86_64.exe + .\setup-x86_64.exe --no-admin --packages "bison,tar" -q -R C:\cygwin64 -s http://cygwin.mirror.constant.com -a x86_64 -v + displayName: 'Install cygwin packages' + failOnStderr: false + ignoreLASTEXITCODE: true + + - task: ExtractFiles@1 + inputs: + archiveFilePatterns: '$(System.ArtifactsDirectory)/source/mariadb-*.tar.gz' + cleanDestinationFolder: false + displayName: 'Extract sources' + + + - powershell: | + Set-PSDebug -Trace 1 + dir + move mariadb-*/* $(Build.SourcesDirectory) + md win_build + cd win_build + cmake .. -Wno-deprecated -Wno-dev -DBUILD_CONFIG=enterprise ` + -DPLUGIN_AWS_KEY_MANAGEMENT=NO -DBISON_EXECUTABLE=C:\cygwin64\bin\bison ` + -DWITH_EMBEDDED_SERVER=0 -DSIGNCODE=OFF -DWITH_UNIT_TESTS=0 -DMYSQL_MAINTAINER_MODE=ERR + displayName: 'CMake configure' + failOnStderr: false + + - script: | + @echo on + pwd + cd win_build + cmake --build . --config ${{ parameters.BuildType }} --verbose --parallel 5 + displayName: 'Compile Windows binaries - ${{ parameters.BuildType }}' + + - script: | + @echo on + pwd + cd win_build + cmake --build . --config ${{ parameters.BuildType }} --target win_package + cmake --build . --config ${{ parameters.BuildType }} --target MSI + displayName: 'Build Windows packages' + + - task: CopyFiles@2 + inputs: + sourceFolder: '$(Build.SourcesDirectory)\win_build' + contents: '?(*.msi|*.zip)' + targetFolder: '$(Build.ArtifactStagingDirectory)' + + - task: PublishBuildArtifacts@1 + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: Windows-${{ parameters.Artifact }} + + - script: | + @echo on + cd win_build\mysql-test + C:\Strawberry\perl\bin\perl.exe mysql-test-run.pl ^ + --suite=main,innodb,mariabackup,rocksdb,plugins --force --retry=3 ^ + --max-save-core=1 --max-save-datadir=1 --max-test-fail=20 --parallel=5 + displayName: 'Windows MTR ${{ parameters.BuildType }} tests' diff --git a/azure/stage0/extract-source-tarball.yml b/azure/stage0/extract-source-tarball.yml new file mode 100644 index 00000000000..639cdc531e2 --- /dev/null +++ b/azure/stage0/extract-source-tarball.yml @@ -0,0 +1,20 @@ + +steps: + + - task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: source + downloadPath: '$(System.ArtifactsDirectory)' + + - bash: | + df -h + ls -la + pwd + mkdir -p "${BUILDDIR}" + tar xf '$(System.ArtifactsDirectory)'/source/mariadb-enterprise-*.tar.gz \ + --strip-components=1 -C "${BUILDDIR}" + ls -la + rm -frv '$(System.ArtifactsDirectory)'/source diff --git a/azure/stage3-rolling-upgrade.yml b/azure/stage3-rolling-upgrade.yml new file mode 100644 index 00000000000..8a72632ee4e --- /dev/null +++ b/azure/stage3-rolling-upgrade.yml @@ -0,0 +1,168 @@ +parameters: + VERSIONS: [] + REPO: 'https://dlm.mariadb.com/enterprise-release-helpers/mariadb_es_repo_setup' + REPO_PARAMS: '--token=$(ESTOKEN) --apply --verbose --mariadb-server-version' + +jobs: + +- job: Upgrade_ES_10_2_to_ES_10_4 + timeoutInMinutes: 280 + condition: succeededOrFailed() + workspace: + clean: all + strategy: + maxParallel: 1 + matrix: + # rhel-6: + # poolName: 'rhel-6' + # rhel-7: + # poolName: 'rhel-7' + debian-8: + poolName: 'debian-8' + debian-9: + poolName: 'debian-9' + ubuntu-1604: + poolName: 'ubuntu-1604' + ubuntu-1804: + poolName: 'ubuntu-1804' + pool: + name: $(poolName) + + steps: + - checkout: none + - template: prepare-vm-for-installation.yml + - ${{ each version in parameters.VERSIONS }}: + + - bash: | + sudo yum -y clean all + sudo rm -fv /etc/yum.repos.d/mariadb.repo* + displayName: "Clean repo data on Centos/RHEL" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7')) + + - bash: | + sudo bash -c "rm -fv ${DEB_LOCAL_LIST} /etc/apt/sources.list.d/mariadb.list*" + sudo apt-get update + displayName: "Clean repo data on Debian/Ubuntu" + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804')) + + - bash: | + set -ex + wget ${{ parameters.REPO }} -O install_mariadb.sh + chmod +x ./install_mariadb.sh + sudo ./install_mariadb.sh ${{ parameters.REPO_PARAMS }}=${{ version }} + rm -fv ./install_mariadb.sh + displayName: "Install ES-${{ version }} repo file and key" + + - bash: | + set -ex + sudo yum -y install MariaDB-* + # we must remove cassandra and tokudb (doesn't exist in ES) + sudo yum -y erase MariaDB-cassandra-engine MariaDB-tokudb-engine ||: + displayName: "Install ES-${{ version }} on RHEL/Centos" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7')) + + - bash: | + set -ex + sudo apt-get update + export DEBIAN_FRONTEND=noninteractive + mVersion=${{ version }} + if [[ ${mVersion} == '10.2' ]]; then + BACKUP_PKG=mariadb-backup-${mVersion} + else + BACKUP_PKG=mariadb-backup + fi + sudo apt-get -y install mariadb-server ${BACKUP_PKG} \ + mariadb-plugin-mroonga mariadb-plugin-oqgraph mariadb-plugin-rocksdb mariadb-plugin-spider \ + mariadb-test mariadb-test-data + # we do not install cassandra and tokudb (doesn't exist in ES) + displayName: "Install ES-${{ version }} on Debian/Ubuntu" + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804')) + + - bash: | + set -ex + if [[ -f /etc/init.d/mysql ]]; then + restart_cmd="service mysql restart" + else + restart_cmd="systemctl restart mysql" + fi + sudo ${restart_cmd} + displayName: "Restart ES-${{ version }}" + + - bash: | + set -ex + sudo mysql -e 'SELECT 1' + displayName: "Check ES-${{ version }} installation on RHEL/Centos" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7')) + + - bash: | + set -ex + sudo mysql -p$(MYSQL_PASSWORD) -e 'SELECT 1' + displayName: "Check ES-${{ version }} installation on Debian/Ubuntu" + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804')) + + - task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: '$(poolName)-RPMS' + downloadPath: '$(System.ArtifactsDirectory)/current' + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7')) + + - task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: '$(poolName)-DEBS' + downloadPath: '$(System.ArtifactsDirectory)/current' + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804')) + + - bash: | + set -x + ls -la '$(System.ArtifactsDirectory)' + sudo yum -y --nogpgcheck install '$(System.ArtifactsDirectory)'/current/'$(poolName)'-RPMS/*.rpm + displayName: "Install built RPM packages on RHEL/Centos" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7')) + + - bash: | + set -xe + ls -la '$(System.ArtifactsDirectory)' + sudo sh -c "echo 'deb [trusted=yes] file://$(System.ArtifactsDirectory)/current/$(poolName)-DEBS binary/' > ${DEB_LOCAL_LIST}" + sudo apt update + export DEBIAN_FRONTEND=noninteractive + sudo apt-get --allow-unauthenticated -y -f install \ + mariadb-server mariadb-test mariadb-backup mariadb-plugin-connect \ + mariadb-plugin-cracklib-password-check mariadb-plugin-gssapi-client \ + mariadb-plugin-gssapi-server mariadb-plugin-mroonga mariadb-plugin-oqgraph \ + mariadb-plugin-rocksdb mariadb-plugin-spider + dpkg -l | grep -iE "galera|maria" + displayName: "Install built DEB packages on Debian/Ubuntu" + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804')) + + - bash: | + set -ex + if [[ -f /etc/init.d/mysql ]]; then + restart_cmd="service mysql restart" + else + restart_cmd="systemctl restart mysql" + fi + sudo ${restart_cmd} + displayName: "Restart built ES Server" + + - bash: | + set -ex + sudo mysql -e 'SELECT 1' + displayName: "Check ES installation on RHEL" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7')) + + - bash: | + set -ex + sudo mysql -p$(MYSQL_PASSWORD) -e 'SELECT 1' + displayName: "Check ES installation on Debian/Ubuntu" + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804')) + + + + + diff --git a/azure/stage3-upgrade.yml b/azure/stage3-upgrade.yml new file mode 100644 index 00000000000..42fc24c1982 --- /dev/null +++ b/azure/stage3-upgrade.yml @@ -0,0 +1,164 @@ +parameters: + VERSION: '' + FROM: '' + FROMVERSION: '' + TO: '' + TOVERSION: '' + REPO: '' + REPO_PARAMS: '' + +jobs: + +- job: Upgrade_${{ parameters.FROM }}_${{ parameters.FROMVERSION }}_to_${{ parameters.TO }}_${{ parameters.TOVERSION }} + timeoutInMinutes: 280 + condition: succeededOrFailed() + workspace: + clean: all + strategy: + maxParallel: 1 + matrix: + # rhel-6: + # poolName: 'rhel-6' + # rhel-7: + # poolName: 'rhel-7' + debian-8: + poolName: 'debian-8' + debian-9: + poolName: 'debian-9' + ubuntu-1604: + poolName: 'ubuntu-1604' + ubuntu-1804: + poolName: 'ubuntu-1804' + pool: + name: $(poolName) + + steps: + - checkout: none + - template: prepare-vm-for-installation.yml + - bash: | + set -ex + wget ${{ parameters.REPO }} -O install_mariadb.sh + chmod +x ./install_mariadb.sh + sudo ./install_mariadb.sh ${{ parameters.REPO_PARAMS }} + rm -fv ./install_mariadb.sh + displayName: "Install ${{ parameters.FROM }}-${{ parameters.FROMVERSION }} repo file and key" + + - bash: | + set -ex + sudo yum -y install MariaDB-* + # we must remove cassandra and tokudb (doesn't exist in ES) + sudo yum -y erase MariaDB-cassandra-engine MariaDB-tokudb-engine ||: + displayName: "Install ${{ parameters.FROM }}-${{ parameters.FROMVERSION }} on RHEL/Centos" + condition: in(variables['poolName'], 'rhel-6', 'rhel-7') + + - bash: | + set -ex + sudo apt-get update + export DEBIAN_FRONTEND=noninteractive + mVersion=${{ parameters.VERSION }} + if [[ ${mVersion} == '10.2' ]]; then + BACKUP_PKG=mariadb-backup-${mVersion} + else + BACKUP_PKG=mariadb-backup + fi + sudo apt-get -y install mariadb-server ${BACKUP_PKG} \ + mariadb-plugin-mroonga mariadb-plugin-oqgraph mariadb-plugin-rocksdb mariadb-plugin-spider \ + mariadb-test mariadb-test-data + # we do not install cassandra and tokudb (doesn't exist in ES) + displayName: "Install ${{ parameters.FROM }}-${{ parameters.FROMVERSION }} on Debian/Ubuntu" + condition: in(variables['poolName'], 'debian-8', 'debian-9', 'debian-10', 'ubuntu-1604', 'ubuntu-1804') + + - bash: | + set -ex + if [[ -f /etc/init.d/mysql ]]; then + restart_cmd="service mysql restart" + else + restart_cmd="systemctl restart mysql" + fi + sudo ${restart_cmd} + displayName: "Restart ${{ parameters.FROM }}-${{ parameters.FROMVERSION }}" + + - bash: | + set -ex + sudo mysql -e 'SELECT 1' + displayName: "Check ${{ parameters.FROM }}-${{ parameters.FROMVERSION }} installation on RHEL/Centos" + condition: in(variables['poolName'], 'rhel-6', 'rhel-7') + + - bash: | + set -ex + sudo mysql -p$(MYSQL_PASSWORD) -e 'SELECT 1' + displayName: "Check ${{ parameters.FROM }}-${{ parameters.FROMVERSION }} installation on Debian/Ubuntu" + condition: in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804') + + - task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: '$(poolName)-RPMS' + downloadPath: '$(System.ArtifactsDirectory)/current' + condition: in(variables['poolName'], 'rhel-6', 'rhel-7') + + - task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: '$(poolName)-DEBS' + downloadPath: '$(System.ArtifactsDirectory)/current' + condition: in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804') + + - bash: | + set -x + ls -la '$(System.ArtifactsDirectory)' + sudo yum -y --nogpgcheck install '$(System.ArtifactsDirectory)'/current/'$(poolName)'-RPMS/*.rpm + displayName: "Install built RPM packages on RHEL/Centos" + condition: in(variables['poolName'], 'rhel-6', 'rhel-7') + + - bash: | + set -xe + ls -la '$(System.ArtifactsDirectory)' + sudo sh -c "echo 'deb [trusted=yes] file://$(System.ArtifactsDirectory)/current/$(poolName)-DEBS binary/' > ${DEB_LOCAL_LIST}" + sudo apt update + export DEBIAN_FRONTEND=noninteractive + mVersion=${{ parameters.VERSION }} + if [[ ${mVersion} == '10.2' ]]; then + BACKUP_PKG=mariadb-backup-${mVersion} + else + BACKUP_PKG=mariadb-backup + fi + sudo apt-get --allow-unauthenticated -y -f install \ + mariadb-server mariadb-test ${BACKUP_PKG} mariadb-plugin-connect \ + mariadb-plugin-cracklib-password-check mariadb-plugin-gssapi-client \ + mariadb-plugin-gssapi-server mariadb-plugin-mroonga mariadb-plugin-oqgraph \ + mariadb-plugin-rocksdb mariadb-plugin-spider + dpkg -l | grep -iE "galera|maria" + displayName: "Install built DEB packages on Debian/Ubuntu" + condition: in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804') + + - bash: | + set -ex + if [[ -f /etc/init.d/mysql ]]; then + restart_cmd="service mysql restart" + else + restart_cmd="systemctl restart mysql" + fi + sudo ${restart_cmd} + displayName: "Restart built ES Server" + + - bash: | + set -ex + sudo mysql -e 'SELECT 1' + displayName: "Check ES installation on RHEL" + condition: and(succeeded(), in(variables['poolName'], 'rhel-6', 'rhel-7')) + + - bash: | + set -ex + sudo mysql -p$(MYSQL_PASSWORD) -e 'SELECT 1' + displayName: "Check ES installation on Debian/Ubuntu" + condition: and(succeeded(), in(variables['poolName'], 'debian-8', 'debian-9', 'ubuntu-1604', 'ubuntu-1804')) + + + + + diff --git a/azure/test-linux-tarball.yml b/azure/test-linux-tarball.yml new file mode 100644 index 00000000000..c81f9f3d89c --- /dev/null +++ b/azure/test-linux-tarball.yml @@ -0,0 +1,38 @@ + +parameters: + BuildType: '' + MTRArgs: '--mem --max-test-fail=10 --verbose-restart --max-save-datadir=1 --retry=3 --max-save-core=1' + +steps: + +- task: DownloadBuildArtifacts@0 + inputs: + buildType: 'current' + specificBuildWithTriggering: true + downloadType: 'single' + artifactName: '$(containerImage)-${{ parameters.BuildType }}' + downloadPath: '$(System.ArtifactsDirectory)' + +- bash: | + df -h + ls -la + pwd + mkdir test + tar xzf '$(System.ArtifactsDirectory)'/$(containerImage)-${{ parameters.BuildType }}/mariadb-enterprise-*.tar.gz \ + --strip-components=1 -C $(Build.SourcesDirectory) + ls -la + rm -frv '$(System.ArtifactsDirectory)'/$(containerImage)-${{ parameters.BuildType }}/mariadb-enterprise-*.tar.gz + +- bash: | + set -x + WSREP_PROVIDER=$(sudo find /usr -type f -name 'libgalera_enterprise_smm.so') + export WSREP_PROVIDER + echo "WSREP_PROVIDER=${WSREP_PROVIDER}" + + [[ -e ${WSREP_PROVIDER} ]] && ls -la ${WSREP_PROVIDER} + + cd '$(Build.SourcesDirectory)/mysql-test' + pwd + ls -lah + exec perl mysql-test-run.pl --force --parallel=auto ${{ parameters.MTRArgs }} + displayName: 'Run MTR on $(containerImage)-${{ parameters.BuildType }}' diff --git a/azure/tests/mac-mtr-test.yml b/azure/tests/mac-mtr-test.yml new file mode 100644 index 00000000000..ec798b40088 --- /dev/null +++ b/azure/tests/mac-mtr-test.yml @@ -0,0 +1,52 @@ +parameters: + MtrArgs: '' + TestName: 'MTR Test' + tarname: '' + Artifact: Release + +steps: + + - bash: | + set -x + cd $(Build.SourcesDirectory) + [[ -d bin ]] && export PATH="$(Build.SourcesDirectory)/bin:${PATH}" + + WSREP_PROVIDER=$(sudo find /usr -type f -name 'libgalera_smm.so') + if [[ -n ${WSREP_PROVIDER} ]]; then + export WSREP_PROVIDER + sudo mysqld --help --verbose 2>&1 | grep wsrep + fi + + if [[ -d mysql-test ]]; then + MYSQLTEST=$(Build.SourcesDirectory)/mysql-test + else + MYSQLTEST=$(find /usr/share -type d -name 'mysql-test' ||:) + fi + + if [[ -z "${MYSQLTEST}" ]]; then + echo "- mysql-test directory doesn't exist after PKG installation!" + exit 1 + else + sudo chown -R ${USER} "${MYSQLTEST}" + fi + + cd "${MYSQLTEST}" + exec perl mysql-test-run.pl ${{ parameters.MtrArgs }} + + displayName: "MTR - ${{ parameters.TestName }}" + condition: succeededOrFailed() + + - bash: | + set -x + ls -la '$(Build.ArtifactStagingDirectory)' + rm -frv '$(Build.ArtifactStagingDirectory)'/* + mkdir -p '$(Build.ArtifactStagingDirectory)'/MTR-Results + tar czvf '$(Build.ArtifactStagingDirectory)'/MTR-Results/${{ parameters.tarname }} $(MYSQL_VARDIR)/* + displayName: 'Archive mysql vardir' + condition: failed() + + - task: PublishBuildArtifacts@1 + condition: failed() + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: 'macOS-latest-${{ parameters.Artifact }}' diff --git a/azure/tests/run-acceptance-tests.yml b/azure/tests/run-acceptance-tests.yml new file mode 100644 index 00000000000..32ed76f19e6 --- /dev/null +++ b/azure/tests/run-acceptance-tests.yml @@ -0,0 +1,65 @@ +steps: + + - bash: | + set -x + sudo mysql -e "CREATE USER IF NOT EXISTS 'testuser'@'localhost' IDENTIFIED BY '$(MYSQL_PASSWORD)';" + UNSTABLE=$(mysql -utestuser -p'$(MYSQL_PASSWORD)' -E -e "SELECT PLUGIN_NAME, PLUGIN_MATURITY, PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_MATURITY != 'Stable' AND PLUGIN_STATUS != 'DISABLED';") + if [[ -n "${UNSTABLE}" ]]; then + echo "- Plugins maturity test failed" + echo "${UNSTABLE}" + fi + displayName: "Plugins maturity test" + + - bash: | + set -x + RES=0 + if [ -e /etc/init.d/mysql ] ; then + restart_cmd="/etc/init.d/mysql restart" + else + restart_cmd="systemctl restart mariadb" + fi + + plugindir=`sudo mysql -s -s -e 'select @@plugin_dir'` + mdev19807_workaround="--plugin-dir=$plugindir" + + id -u bombalurina || sudo useradd bombalurina + sudo passwd bombalurina <<EOF + an1rul@bm0b + an1rul@bm0b + EOF + + sudo tee /etc/pam.d/mariadb <<EOF + auth required pam_unix.so audit + account required pam_unix.so audit + EOF + + # PAM v2 + sudo mysql -e "INSTALL SONAME 'auth_pam'; CREATE USER 'bombalurina'@'localhost' IDENTIFIED VIA pam USING 'mariadb'" + if ! mysql $mdev19807_workaround -ubombalurina -pan1rul@bm0b -e "SHOW GRANTS" ; then + echo "Authentication with PAM v2 (pam_unix) failed" + fi + sudo mysql -e "UNINSTALL SONAME 'auth_pam'" + if mysql $mdev19807_workaround -ubombalurina -pan1rul@bm0b -e "SHOW GRANTS" > /dev/null 2>&1 ; then + echo "User authenticated via PAM v2 (pam_unix) could still connect after uninstalling plugin" + fi + + # PAM v1 + sudo mysql -e "INSTALL SONAME 'auth_pam_v1'" + set +e + sudo groupadd shadow + sudo usermod -a -G shadow mysql + sudo chown root:shadow /etc/shadow + sudo chmod g+r /etc/shadow + set -e + + sudo $restart_cmd + + if ! mysql $mdev19807_workaround -ubombalurina -pan1rul@bm0b -e "SHOW GRANTS" ; then + echo "- Authentication with PAM v1 (pam_unix) failed" + fi + + sudo mysql -e "UNINSTALL SONAME 'auth_pam_v1'" + if mysql $mdev19807_workaround -ubombalurina -pan1rul@bm0b -e "SHOW GRANTS" > /dev/null 2>&1 ; then + echo "- User authenticated via PAM v1 (pam_unix) could still connect after uninstalling plugin" + fi + displayName: "PAM Acceptance test" diff --git a/azure/tests/run-mtr-test.yml b/azure/tests/run-mtr-test.yml new file mode 100644 index 00000000000..451793cbb11 --- /dev/null +++ b/azure/tests/run-mtr-test.yml @@ -0,0 +1,52 @@ +parameters: + MtrArgs: '' + TestName: 'MTR Test' + tarname: '' + Artifact: Release + +steps: + + - bash: | + set -x + cd $(Build.SourcesDirectory) + [[ -d bin ]] && export PATH="$(Build.SourcesDirectory)/bin:${PATH}" + + WSREP_PROVIDER=$(sudo find /usr -type f -name 'libgalera_enterprise_smm.so') + if [[ -n ${WSREP_PROVIDER} ]]; then + export WSREP_PROVIDER + sudo mysqld --help --verbose 2>&1 | grep wsrep + fi + + if [[ -d mysql-test ]]; then + MYSQLTEST=$(Build.SourcesDirectory)/mysql-test + else + MYSQLTEST=$(find /usr/share -type d -name 'mysql-test' ||:) + fi + + if [[ -z "${MYSQLTEST}" ]]; then + echo "- mysql-test directory doesn't exist after PKG installation!" + exit 1 + else + sudo chown -R ${USER} "${MYSQLTEST}" + fi + + cd "${MYSQLTEST}" + exec perl mysql-test-run.pl ${{ parameters.MtrArgs }} + + displayName: "MTR - ${{ parameters.TestName }}" + condition: succeededOrFailed() + + - bash: | + set -x + ls -la '$(Build.ArtifactStagingDirectory)' + rm -frv '$(Build.ArtifactStagingDirectory)'/* + mkdir -p '$(Build.ArtifactStagingDirectory)'/MTR-Results + tar czvf '$(Build.ArtifactStagingDirectory)'/MTR-Results/${{ parameters.tarname }} $(MYSQL_VARDIR)/* + displayName: 'Archive mysql vardir' + condition: failed() + + - task: PublishBuildArtifacts@1 + condition: failed() + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: '$(containerImage)-${{ parameters.Artifact }}' diff --git a/azure/tests/select-insert-test.yml b/azure/tests/select-insert-test.yml new file mode 100644 index 00000000000..6358dc961e2 --- /dev/null +++ b/azure/tests/select-insert-test.yml @@ -0,0 +1,61 @@ +steps: + + - bash: | + set -x + INSTALLED="" + + if [[ -e /etc/redhat-release ]]; then + INSTALLED=$(rpm -qa | grep -iE "^maria|^mysql|^percona|^galera" ||:) + fi + + if [[ -e /etc/debian_version ]]; then + INSTALLED=$(dpkg -l | grep -iE "^maria|^mysql|^percona|^galera" ||:) + fi + + [[ -n "${INSTALLED}" ]] && echo "${INSTALLED}" + + if [ -e /etc/init.d/mysql ] ; then + restart_cmd="/etc/init.d/mysql restart" + status_cmd="/etc/init.d/mysql status" + else + systemctl --version + sudo systemctl show mariadb | grep Capa || true + restart_cmd="systemctl restart mariadb" + status_cmd="systemctl status mariadb" + fi + + sudo ${restart_cmd} + sudo ${status_cmd} + + MYSQL_PID=$(pidof mysqld) + if [[ -z "${MYSQL_PID}" ]]; then + echo "- Probably MySQL failed to start!" + exit 1 + fi + displayName: "Restart MySQL after installation" + failOnStderr: false + + - bash: | + set -x + + sudo mysql -e 'drop database if exists test;' + sudo mysql -e 'create database test;' + sudo mysql -e 'create table test.t(a int primary key) engine=innodb;' + sudo mysql -e 'insert into test.t values (1);' + sudo mysql -e 'select * from test.t;' + sudo mysql -e 'drop table test.t;' + + sudo mysql -e "set password= PASSWORD('S1mpl-pw')" + password_option="-pS1mpl-pw" + + sudo mysql -uroot $password_option -e "CREATE DATABASE db" + sudo mysql -uroot $password_option -e "CREATE TABLE db.t_innodb(a1 SERIAL, c1 CHAR(8)) ENGINE=InnoDB; INSERT INTO db.t_innodb VALUES (1,'foo'),(2,'bar')" + sudo mysql -uroot $password_option -e "CREATE TABLE db.t_myisam(a2 SERIAL, c2 CHAR(8)) ENGINE=MyISAM; INSERT INTO db.t_myisam VALUES (1,'foo'),(2,'bar')" + sudo mysql -uroot $password_option -e "CREATE TABLE db.t_aria(a3 SERIAL, c3 CHAR(8)) ENGINE=Aria; INSERT INTO db.t_aria VALUES (1,'foo'),(2,'bar')" + sudo mysql -uroot $password_option -e "CREATE TABLE db.t_memory(a4 SERIAL, c4 CHAR(8)) ENGINE=MEMORY; INSERT INTO db.t_memory VALUES (1,'foo'),(2,'bar')" + sudo mysql -uroot $password_option -e "CREATE ALGORITHM=MERGE VIEW db.v_merge AS SELECT * FROM db.t_innodb, db.t_myisam, db.t_aria" + sudo mysql -uroot $password_option -e "CREATE ALGORITHM=TEMPTABLE VIEW db.v_temptable AS SELECT * FROM db.t_innodb, db.t_myisam, db.t_aria" + sudo mysql -uroot $password_option -e "CREATE PROCEDURE db.p() SELECT * FROM db.v_merge" + sudo mysql -uroot $password_option -e "CREATE FUNCTION db.f() RETURNS INT DETERMINISTIC RETURN 1" + + displayName: "Simple SELECT/INSERT tests" diff --git a/azure/tests/win-select-insert.yml b/azure/tests/win-select-insert.yml new file mode 100644 index 00000000000..b9d8b0f8dde --- /dev/null +++ b/azure/tests/win-select-insert.yml @@ -0,0 +1,47 @@ +steps: + +- powershell: | + Set-PSDebug -Trace 1 + $password_option="-pS1mpl-pw" + $install_password="-ptESt123%_password" + $ProgramFilesPath = [Environment]::GetEnvironmentVariable("ProgramW6432") + $mysql = Get-Childitem -Path $ProgramFilesPath -Recurse -Filter mysql.exe + + & $mysql -uroot $install_password -e 'drop database if exists test;' + & $mysql -uroot $install_password -e 'create database test;' + & $mysql -uroot $install_password -e 'create table test.t(a int primary key) engine=innodb;' + & $mysql -uroot $install_password -e 'insert into test.t values (1);' + & $mysql -uroot $install_password -e 'select * from test.t;' + & $mysql -uroot $install_password -e 'drop table test.t;' + + & $mysql -uroot $install_password -e "set password=PASSWORD('S1mpl-pw')" + + & $mysql -uroot $password_option -e "CREATE DATABASE db" + & $mysql -uroot $password_option -e "CREATE TABLE db.t_innodb(a1 SERIAL, c1 CHAR(8)) ENGINE=InnoDB; INSERT INTO db.t_innodb VALUES (1,'foo'),(2,'bar')" + & $mysql -uroot $password_option -e "CREATE TABLE db.t_myisam(a2 SERIAL, c2 CHAR(8)) ENGINE=MyISAM; INSERT INTO db.t_myisam VALUES (1,'foo'),(2,'bar')" + & $mysql -uroot $password_option -e "CREATE TABLE db.t_aria(a3 SERIAL, c3 CHAR(8)) ENGINE=Aria; INSERT INTO db.t_aria VALUES (1,'foo'),(2,'bar')" + & $mysql -uroot $password_option -e "CREATE TABLE db.t_memory(a4 SERIAL, c4 CHAR(8)) ENGINE=MEMORY; INSERT INTO db.t_memory VALUES (1,'foo'),(2,'bar')" + & $mysql -uroot $password_option -e "CREATE ALGORITHM=MERGE VIEW db.v_merge AS SELECT * FROM db.t_innodb, db.t_myisam, db.t_aria" + & $mysql -uroot $password_option -e "CREATE ALGORITHM=TEMPTABLE VIEW db.v_temptable AS SELECT * FROM db.t_innodb, db.t_myisam, db.t_aria" + & $mysql -uroot $password_option -e "CREATE PROCEDURE db.p() SELECT * FROM db.v_merge" + & $mysql -uroot $password_option -e "CREATE FUNCTION db.f() RETURNS INT DETERMINISTIC RETURN 1" + displayName: "Windows SELECT/INSERT tests" + failOnStderr: false + errorActionPreference: continue + + +- powershell: | + Set-PSDebug -Trace 1 + $password_option="-pS1mpl-pw" + $ProgramFilesPath = [Environment]::GetEnvironmentVariable("ProgramW6432") + $mysql = Get-Childitem -Path $ProgramFilesPath -Recurse -Filter mysql.exe + + $UNSTABLE = & $mysql -uroot $password_option -e "SELECT PLUGIN_NAME, PLUGIN_MATURITY, PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_MATURITY != 'Stable' AND PLUGIN_STATUS != 'DISABLED';" + if ($UNSTABLE) { + Write-Host "- Plugins maturity test failed" + [Environment]::Exit(1) + } + displayName: "Windows plugins maturity test" + failOnStderr: false + errorActionPreference: continue + diff --git a/azure/untitled.yaml b/azure/untitled.yaml new file mode 100644 index 00000000000..5c5c4302480 --- /dev/null +++ b/azure/untitled.yaml @@ -0,0 +1,104 @@ + + + + + + + # Clean install test + + + + - task: DownloadPipelineArtifact@2 + displayName: 'Download ES 10.2' + inputs: + source: 'specific' + project: '550599d3-6165-4abd-8c86-e3f7e53a1847' + artifact: '$(shortName)-rpm' + pipeline: 3 + runVersion: 'latestFromBranch' + runBranch: 'refs/heads/10.2-enterprise' + downloadPath: '$(Build.ArtifactStagingDirectory)/10.2-enterprise' + + - task: DownloadPipelineArtifact@2 + displayName: 'Download ES 10.3' + inputs: + source: 'specific' + project: '550599d3-6165-4abd-8c86-e3f7e53a1847' + artifact: '$(shortName)-rpm' + pipeline: 3 + runVersion: 'latestFromBranch' + runBranch: 'refs/heads/10.3-enterprise' + downloadPath: '$(Build.ArtifactStagingDirectory)/10.3-enterprise' + + - template: azure/prepare-vm-for-installation.yml + + - bash: | + set -x + ls -la '$(System.ArtifactsDirectory)' + sudo yum -y --nogpgcheck install '$(System.ArtifactsDirectory)/10.2-enterprise/$(shortName)-RPMS'/*.rpm + displayName: 'Install 10.2 version' + + - template: azure/select-insert-test.yml + - template: azure/run-acceptance-tests.yml + + - bash: | + set -x + if [ -e /etc/init.d/mysql ] ; then + stop_cmd="sudo /etc/init.d/mysql stop" + else + stop_cmd="sudo systemctl stop mariadb" + fi + + # Stop server before upgrade + [[ -n $(pidof mysqld) ]] && eval $stop_cmd + RES=$? + if [[ ${RES:-} -gt 0 ]]; then + echo "mysqld es-10.2 failed to stop!" + exit 1 + fi + - bash: | + set -x + ls -la '$(System.ArtifactsDirectory)' + sudo yum -y --nogpgcheck install '$(System.ArtifactsDirectory)/10.3-enterprise/$(shortName)-RPMS'/*.rpm + displayName: 'Install 10.3 version' + + - template: azure/select-insert-test.yml + - template: azure/run-acceptance-tests.yml + + - bash: | + set -x + if [ -e /etc/init.d/mysql ] ; then + stop_cmd="sudo /etc/init.d/mysql stop" + else + stop_cmd="sudo systemctl stop mariadb" + fi + + # Stop server before upgrade + [[ -n $(pidof mysqld) ]] && eval $stop_cmd + RES=$? + if [[ ${RES:-} -gt 0 ]]; then + echo "mysqld es-10.3 failed to stop!" + exit 1 + fi + - bash: | + set -x + ls -la '$(System.ArtifactsDirectory)' + sudo yum -y --nogpgcheck install '$(System.ArtifactsDirectory)/current/$(shortName)-RPMS'/*.rpm + displayName: 'Install current 10.4 version' + + - template: azure/select-insert-test.yml + - template: azure/run-acceptance-tests.yml + + - bash: | + rm -fr $(Build.ArtifactStagingDirectory)/10.*-enterprise + displayName: "Clean downloaded packages" + + # Upload log with failed tests to the same directory + - task: PublishBuildArtifacts@1 + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: $(shortName)-RPMS + + + # done? upgrade from ES 10.2 to ES 10.3 to ES 10.4 should be tested, + # then also from community server to enterprise server |