azure pipelines configuration10.5-enterprise-rebase

Fix Windows build and test

Add galera-es test repository

update azure-pipelines.yml for new galera-es package names

Install needed packages for galera-es testing

add galera-es tests for centos6, centos7

Fix syntax error with galera-es repo setup

Fix galera-es repo url

Update azure-pipelines.yml

Added a comment to explain why the section is only in the 10.4 pipeline script

Move MTR tests out of bintar job to new 'MTR' job

add download and untar steps

try fix for name of artifact to download

more updates to download task for MTR job

Fix paths

Remove bintar build dependency from DEB and RPM jobs

unpack to opt, and add full paths to cd commands

Also turn off verbose when untarring, we don't need that

fix ownership of /opt dir

revert back to using System.ArtifactsDirectory instead of /opt

chown of /opt to the current user failed because the $USER env variable
is not set for some reason, not sure if this is a docker issue, or
something with azure, or something else, don't want to mess with it at
this time

Add debian-8

fix deb publishing

the .changes file was missing, needs to be included with the .deb files

Also added the scanpackages step that we do with our buildbot and galera
builds

add -deb or -rpm suffix to package build artifacts

This is to keep bintar and package artifacts separate, which makes for
easier processing during releases.

include missing deb files

Remove debian-8 for now

It's not building packages correctly

increase job timeout for Windows, from default 1 hour to 3 hours

fix for Debian 8

use the correct number of CPUs to not overload the builder

MENT-160 RPM packages are not installable on CentOS due to to missing MariaDB-compat

MENT-164 Some RPM packages present in community server are missing from enterprise

and MENT-163 RPM installation complains about policy files, mariadb.service is not installed

install missing -devel packages to get cracklib_password_check, oqgraph,
and systemd support

show scripts executing in azure

run ASAN as in buildbot

Desactivating ASan

Package ZIPs

also, echo commands in Windows build scripts

Windows - do not publish exe/dll/pdb, the relevant stuff is in ZIP

make sure artifacts are saved on Windows, even if some MTR tests fail.

use static gnutls for bintars

Adding required prerequisites for successful execution of mtr with Galera tests

run mtr also for ubuntu 16.04 bintars

Test plugins suite on Windows.

MENT-187 Windows build needs to be signed

Checking the need to install lsof for CentOS 7

MENT-261 Store the build docker images in a Azure Container Repository

azure-pipelines.yml install galera library

Galera library is now installed in its own steps so that it can be used both for normal mtr and galera mtr

MENT-176 Set up the Stage Gate to the Azure CI
MENT-265 Azure CI fails on CopyFiles after RPM build
MENT-25 Add test to only have stable plugins in Enterprise
MENT-252 install tests in azure
MENT-198 Build & debug ASAN builds for all supported OS & versions
MENT-329 Build & debug SLES 15
MENT-191 Build & debug SLES 12

Initial Stage Gate implementation for Azure pipeline

[+] STAGE 1 source tarball              reworked
[+] STAGE 1 binary release tarballs     reworked
[+] STAGE 1 binary debug tarballs       reworked
[+] STAGE 1 release MTR                 reworked
[+] STAGE 1 debug MTR                   reworked
[+] STAGE 1 Release RPMS                reworked
[+] STAGE 1 Windows ZIP and MSI         reworked
[+] STAGE 1 Windows release MTR         reworked
[+] STAGE 1 Galera on release tarballs  reworked

[+] STAGE 2 Azure <=> KVM servers interaction
[+] STAGE 2 RPM installation test
[+] STAGE 2 DEB installation test
[+] STAGE 2 PAM acceptance test
[+] STAGE 2 Simple SELECT/INSERT test
[+] STAGE 2 Galera tests
[+] STAGE 2 MTR tests from buildbot
[+] STAGE 2 Upgrade test, 10.2 => 10.3 => 10.4 current

MENT-355
disable expensive checks on Azure for personal branches

MENT-355

disable expensive checks on Azure for personal branches

MENT-367 move Debug and ASAN builds to separate pipelines

10.4 pipelines

MENT-370 Replace Azure CentOS 6 & 7 builders with RHEL builders

replaced centos with rhel builders

MENT-371 yum install fails on CentOS 7

transformed image name to rpm name variable by deleting dash from image
name

MENT-383 move stage2 to the separate pipeline

created additional YAML file for separate pipeline to run on 03AM EEST

MENT-245 Azure Stage gate pipelines scripts

added Galera-Enterprise repository to SLES

MENT-393 Azure job must fail if VM can't be prepared for tests

do not supress eny error from yum/apt/zypper during uninstall

MENT-398 galera mtr tests - false positives

MENT-402 default Azure timeout breaks pkg build

Azure default timeout is 60m. this commit increases build time (and time
in queue) from 60m to 180m (3x)

MENT-296 Galera MTR failed on Azure on 10.4-enterprise

--test-case-timeout=120 for all Galera tests

MENT-419 Galera: Create pipeline that is run after every commit

added back galera tests to release pipeline

MENT-279 MariaDB Enterprise source code tarballs don't have "enterprise" in the file name

added properly set CPACK_SOURCE_PACKAGE_FILE_NAME, effective only with
    -DBUILD_CONFIG=enterprise

(azure part)

MENT-61 Installation / Upgrade tests

in fact installation tests are located in the release pipeline and may
be easily extended to cover all supported systems. so this commit is
intended to bring only upgrade tests into Stage Gates.

the logic is:
* build packages
* prepare VM for installation
* install Community Server
* check installation validity
* install ES over CS
* check installation validity
* install built ES over released

MENT-441 set timeout on copyfiles@1

the goal is to fail quickly if file can't be copied or just hangs

MENT-420 package dependency problem with postfix & perl-DBD-MySQL on CentOS7

it's for -compat package. changed package name to download
MariaDB-shared instead of -compat to have libmysqlclient.so.18

MENT-466 Release pipeline - Quick Ubuntu - 2 failures to analyse

added bash loop, because lockfile may be acquired by OS during run

MENT-445 Release pipeline - task timeout for SLES

added 2m timeout for download task to fail quickly if Azure hangs

MENT-459 ASAN Pipeline - Several builds packages are not built properly

added release and debug configurations, reduced platform list, fixed mtr
call

MENT-199 Build & debug MacOS

enabled MacOS compilation

MENT-241 Save more detailed failures details & files

added conditional step to archive mysql vardir after failed mtr

MENT-496 ASAN pipeline can't archive logs - missing parameters

added missing params for azure/tests/run-mtr-test.yml

MENT-499 fix galera installation script in pipelines

set separate repos for galera-3 and galera-4 in pipelines depending on
ES version

MENT-485 galera-25.3.27-1.sles12.sle12.x86_64.rpm' not found on medium 'https://downloads.mariadb.com

added missing step to rewrite the repo and install galera

MENT-507 short timeout lead to job failures on azure

set timeout to 10m

MENT-505 galera-enterprise-4 is not found for buster

removed apt-add-repository call

MENT-526 Remove Travis & Appveyor configuration files in the MariaDBEnterprise repository branches

removed yaml configs

MENT-465 Upgrade Test Failure - Rolling upgrade needs to be fixed

commented out Centos platforms until MDEV-9584 is fixed

MENT-524 Azure pipeline for checking server installation from the official repo during release process

added separate yaml file to run new pipeline

MENT-558 move Galera MTR tests to the separate pipeline or disable for long MTR run

added one more stage to run MTR tests 2*3 in parallel

fixed typo in engines test

Change the way cygwin is downloaded

MENT-672 debian & ubuntu lack dependencies  for new 10.5 dh-exec libpcre2-dev

replaced debian jessie with ubuntu focal

MENT-678 wrong cmake logic related to OpenSSL

on Mac and on other systems with custom OpenSSL installation -DWITH_SSL
must be used instead of specifying OPENSSL_ROOT_DIR and
OPENSSL_LIBRARIES

TODO-2280 ASAN failure on Azure, missing sanitizer/asan_interface.h

changed gcc version to 10 for asan builds

disabled pkg build and installation test on Azure

TODO-2367 no MacOS nightly builds for 10.5

copied MacOS pipeline from 10.4

Fix XPand compile failures in Azure Pipelines on Windows

- Use relative path to submodule
- Checkout submodules also when building source tarball
- Update XPand to include revision which disables compiling XPand on W

(azure part)

Make MacOS build check out submodules

1 files changed, 133 insertions, 0 deletions

diff --git a/azure-pipelines-debug.yml b/azure-pipelines-debug.yml
new file mode 100644
index 00000000000..2fbb662c6dd
--- /dev/null
+++ b/azure-pipelines-debug.yml
@@ -0,0 +1,133 @@
+# C/C++ with GCC
+# Build your C/C++ project with GCC using make.
+# Add steps that publish test results, save build artifacts, deploy, and more:
+# https://docs.microsoft.com/azure/devops/pipelines/apps/c-cpp/gcc
+
+resources:
+    containers:
+
+    - container: rhel-6
+      image: mariadbe.azurecr.io/build-es:rhel-6
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: rhel-7
+      image: mariadbe.azurecr.io/build-es:rhel-7
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: rhel-8
+      image: mariadbe.azurecr.io/build-es:rhel-8
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: sles-12
+      image: mariadbe.azurecr.io/build-es:sles-12
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: sles-15
+      image: mariadbe.azurecr.io/build-es:sles-15
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: ubuntu-1604
+      image: mariadbe.azurecr.io/build-es:ubuntu-1604
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: ubuntu-1804
+      image: mariadbe.azurecr.io/build-es:ubuntu-1804
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: debian-8
+      image: mariadbe.azurecr.io/build-es:debian-8
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: debian-9
+      image: mariadbe.azurecr.io/build-es:debian-9
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+    - container: debian-10
+      image: mariadbe.azurecr.io/build-es:debian-10
+      endpoint: acr-mariadbe-connection
+      options: --shm-size 12G
+
+variables:
+  TEST_RESULTS_FILE: 'test-results.log'
+  MTR_FEEDBACK_PLUGIN: 1
+  DEBIAN_FRONTEND: noninteractive
+  MYSQL_VARDIR: '/var/tmp/mtr'
+  BUILDDIR: "$(Build.SourcesDirectory)/BUILD"
+  VERSION: '10.4'
+#
+stages:
+
+- stage: BuildPackages_Stage0
+
+  jobs:
+
+  - template: azure/stage0/build-source-tarball.yml
+
+  - template: azure/stage0/build-binary-tarballs.yml
+    parameters:
+      BuildType: Debug
+      Artifact: Debug
+
+  - template: azure/stage0/build-windows-packages.yml
+    parameters:
+      Artifact: Debug
+      BuildType: Debug
+
+  - template: azure/stage0/build-macos-packages.yml
+    parameters:
+      BuildType: Debug
+      Artifact: Debug
+
+#################################
+########## STAGE 0 END ##########
+#################################
+
+
+#################################
+######### STAGE 1 BEGIN #########
+#################################
+
+- stage: TestPackages_Debug
+  dependsOn: BuildPackages_Stage0
+  condition: succeededOrFailed()
+  jobs:
+
+  - job: TestTarball_Quick
+    timeoutInMinutes: 240
+    workspace:
+      clean: all
+    pool:
+      vmImage: 'ubuntu-16.04'
+    strategy:
+      matrix:
+        rhel-7:
+          containerImage: rhel-7
+        ubuntu-1804:
+          containerImage: ubuntu-1804
+    container: $[variables['containerImage']]
+
+    steps:
+
+    - checkout: none
+    - template: azure/fetch-unpack-tarball.yml
+      parameters:
+        Artifact: Debug
+    - template: azure/tests/run-mtr-test.yml
+      parameters:
+        MtrArgs: '--mem --force --verbose-restart --skip-test-list=unstable-tests --parallel=auto'
+        TestName: 'Quick Test - Debug'
+
+#
+#################################
+########## STAGE 1 END ##########
+#################################
+#