diff options
Diffstat (limited to 'doc/development')
41 files changed, 1824 insertions, 245 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index 2ff38d68a47..624665a42d1 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -59,6 +59,7 @@ description: 'Learn how to contribute to GitLab.' - [DeclarativePolicy framework](policies.md) - [How Git object deduplication works in GitLab](git_object_deduplication.md) - [Geo development](geo.md) +- [Routing](routing.md) ## Performance guides diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 9a012f4299b..a0e4020da09 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -59,7 +59,7 @@ graph TB PgBouncerExporter[PgBouncer Exporter] --> PgBouncer Prometheus -- TCP 9187 --> PostgreSQLExporter Prometheus -- TCP 9100 --> NodeExporter[Node Exporter] - Prometheus -- TCP 9168 --> GitLabMonito[GitLab Monitor] + Prometheus -- TCP 9168 --> GitLabMonitor[GitLab Monitor] Prometheus -- TCP 9127 --> PgBouncerExporter GitLabMonitor --> PostgreSQL GitLabMonitor --> GitLabShell @@ -106,43 +106,43 @@ Component statuses are linked to configuration documentation for each component. ### Component list -| Component | Description | [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html) | [GitLab chart](https://docs.gitlab.com/charts/) | [Minikube Minimal](https://docs.gitlab.com/charts/development/minikube/#deploying-gitlab-with-minimal-settings) | [GitLab.com](https://gitlab.com) | CE/EE | -| --------- | ----------- |:--------------------:|:------------------:|:-----:|:--------:|:--------:| -| [NGINX](#nginx) | Routes requests to appropriate components, terminates SSL | [✅][nginx-omnibus] | [✅][nginx-charts] | [⚙][nginx-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | -| [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | [✅][unicorn-omnibus] | [✅][unicorn-charts] | [✅][unicorn-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#unicorn) | CE & EE | -| [Sidekiq](#sidekiq) | Background jobs processor | [✅][sidekiq-omnibus] | [✅][sidekiq-charts] | [✅](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#sidekiq) | CE & EE | -| [Gitaly](#gitaly) | Git RPC service for handling all git calls made by GitLab | [✅][gitaly-omnibus] | [✅][gitaly-charts] | [✅][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | -| [GitLab Workhorse](#gitlab-workhorse) | Smart reverse proxy, handles large HTTP requests | [✅][workhorse-omnibus] | [✅][workhorse-charts] | [✅][workhorse-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | -| [GitLab Shell](#gitlab-shell) | Handles `git` over SSH sessions | [✅][shell-omnibus] | [✅][shell-charts] | [✅][shell-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | -| [GitLab Pages](#gitlab-pages) | Hosts static websites | [⚙][pages-omnibus] | [❌][pages-charts] | [❌][pages-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-pages) | CE & EE | -| [Registry](#registry) | Container registry, allows pushing and pulling of images | [⚙][registry-omnibus] | [✅][registry-charts] | [✅][registry-charts] | [✅](https://docs.gitlab.com/ee/user/project/container_registry.html#build-and-push-images) | CE & EE | -| [Redis](#redis) | Caching service | [✅][redis-omnibus] | [✅][redis-omnibus] | [✅][redis-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | -| [PostgreSQL](#postgresql) | Database | [✅][postgres-omnibus] | [✅][postgres-charts] | [✅][postgres-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#postgresql) | CE & EE | -| [PgBouncer](#pgbouncer) | Database connection pooling, failover | [⚙][pgbouncer-omnibus] | [❌][pgbouncer-charts] | [❌][pgbouncer-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#database-architecture) | EE Only | -| [Consul](#consul) | Database node discovery, failover | [⚙][consul-omnibus] | [❌][consul-charts] | [❌][consul-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#consul) | EE Only | -| [GitLab self-monitoring: Prometheus](#prometheus) | Time-series database, metrics collection, and query service | [✅][prometheus-omnibus] | [✅][prometheus-charts] | [⚙][prometheus-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#prometheus) | CE & EE | -| [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | [✅][alertmanager-omnibus] | [✅][alertmanager-charts] | [⚙][alertmanager-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | -| [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | [⚙][grafana-omnibus] | [⤓][grafana-charts] | [⤓][grafana-charts] | [✅](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s) | CE & EE | -| [GitLab self-monitoring: Sentry](#sentry) | Track errors generated by the GitLab instance | [⤓][sentry-omnibus] | [❌][sentry-charts] | [❌][sentry-charts] | [✅](https://about.gitlab.com/handbook/support/workflows/services/gitlab_com/500_errors.html#searching-sentry) | CE & EE | -| [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | [❌][jaeger-omnibus] | [❌][jaeger-charts] | [❌][jaeger-charts] | [❌](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104) | CE & EE | -| [Redis Exporter](#redis-exporter) | Prometheus endpoint with Redis metrics | [✅][redis-exporter-omnibus] | [✅][redis-exporter-charts] | [✅][redis-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | -| [Postgres Exporter](#postgres-exporter) | Prometheus endpoint with PostgreSQL metrics | [✅][postgres-exporter-omnibus] | [✅][postgres-exporter-charts] | [✅][postgres-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | -| [PgBouncer Exporter](#pgbouncer-exporter) | Prometheus endpoint with PgBouncer metrics | [⚙][pgbouncer-exporter-omnibus] | [❌][pgbouncer-exporter-charts] | [❌][pgbouncer-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | -| [GitLab Monitor](#gitlab-monitor) | Generates a variety of GitLab metrics | [✅][gitlab-monitor-omnibus] | [❌][gitab-monitor-charts] | [❌][gitab-monitor-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | -| [Node Exporter](#node-exporter) | Prometheus endpoint with system metrics | [✅][node-exporter-omnibus] | [❌][node-exporter-charts] | [❌][node-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | -| [Mattermost](#mattermost) | Open-source Slack alternative | [⚙][mattermost-omnibus] | [⤓][mattermost-charts] | [⤓][mattermost-charts] | [⤓](https://docs.gitlab.com/ee/user/project/integrations/mattermost_slash_commands.html#manual-configuration), [⤓](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html) | CE & EE | -| [Minio](#minio) | Object storage service | [⤓][minio-omnibus] | [✅][minio-charts] | [✅][minio-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#storage-architecture) | CE & EE | -| [Runner](#gitlab-runner) | Executes GitLab CI jobs | [⤓][runner-omnibus] | [✅][runner-charts] | [⚙][runner-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners) | CE & EE | -| [Database Migrations](#database-migrations) | Database migrations | [✅][database-migrations-omnibus] | [✅]() | [✅][database-migrations-charts] | [✅][database-migrations-charts] | CE & EE | -| [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | [✅][certificate-management-omnibus] | [✅][certificate-management-charts] | [⚙][certificate-management-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | CE & EE | -| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [⚙][geo-omnibus] | [❌][geo-charts] | [❌][geo-charts] | ✅ | EE Only | -| [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [⤓][ldap-omnibus] | [⤓][ldap-charts] | [⤓][ldap-charts] | [❌](https://about.gitlab.com/pricing/#gitlab-com) | CE & EE | -| [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [⤓][outbound-email-omnibus] | [⤓][outbound-email-charts] | [⤓][outbound-email-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | CE & EE | -| [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [⤓][inbound-email-omnibus] | [⤓][inbound-email-charts] | [⤓][inbound-email-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | CE & EE | -| [ElasticSearch](#elasticsearch) | Improved search within GitLab | [⤓][elasticsearch-omnibus] | [⤓][elasticsearch-charts] | [⤓][elasticsearch-charts] | [❌](https://gitlab.com/groups/gitlab-org/-/epics/153) | EE Only | -| [Sentry integration](#sentry) | Error tracking for deployed apps | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | CE & EE | -| [Jaeger integration](#jaeger) | Distributed tracing for deployed apps | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | EE Only | -| [Kubernetes cluster apps](#kubernetes-cluster-apps) | Deploy [Helm](https://docs.helm.sh/), [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/), [Cert-Manager](https://docs.cert-manager.io/en/latest/), [Prometheus](https://prometheus.io/docs/introduction/overview/), a [Runner](https://docs.gitlab.com/runner/), [JupyterHub](http://jupyter.org/), [Knative](https://cloud.google.com/knative) to a cluster | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | CE & EE | +| Component | Description | [Omnibus GitLab](https://docs.gitlab.com/omnibus/) | [GitLab chart](https://docs.gitlab.com/charts/) | [Minikube Minimal](https://docs.gitlab.com/charts/development/minikube/#deploying-gitlab-with-minimal-settings) | [GitLab.com](https://gitlab.com) | [Source](../install/installation.md) | [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) | CE/EE | +| --------- | ----------- |:--------------------:|:------------------:|:-----:|:--------:|:--------:|:-------:|:-------:| +| [NGINX](#nginx) | Routes requests to appropriate components, terminates SSL | [✅][nginx-omnibus] | [✅][nginx-charts] | [⚙][nginx-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⤓][nginx-source] | ❌ | CE & EE | +| [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | [✅][unicorn-omnibus] | [✅][unicorn-charts] | [✅][unicorn-charts] | [✅](../user/gitlab_com/index.md#unicorn) | [⚙][unicorn-source] | [✅][gitlab-yml] | CE & EE | +| [Sidekiq](#sidekiq) | Background jobs processor | [✅][sidekiq-omnibus] | [✅][sidekiq-charts] | [✅](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [✅](../user/gitlab_com/index.md#sidekiq) | [✅][gitlab-yml] | [✅][gitlab-yml] | CE & EE | +| [Gitaly](#gitaly) | Git RPC service for handling all git calls made by GitLab | [✅][gitaly-omnibus] | [✅][gitaly-charts] | [✅][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][gitaly-source] | ✅ | CE & EE | +| [GitLab Workhorse](#gitlab-workhorse) | Smart reverse proxy, handles large HTTP requests | [✅][workhorse-omnibus] | [✅][workhorse-charts] | [✅][workhorse-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][workhorse-source] | ✅ | CE & EE | +| [GitLab Shell](#gitlab-shell) | Handles `git` over SSH sessions | [✅][shell-omnibus] | [✅][shell-charts] | [✅][shell-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][shell-source] | [✅][gitlab-yml] | CE & EE | +| [GitLab Pages](#gitlab-pages) | Hosts static websites | [⚙][pages-omnibus] | [❌][pages-charts] | [❌][pages-charts] | [✅](../user/gitlab_com/index.md#gitlab-pages) | [⚙][pages-source] | [⚙][pages-gdk] | CE & EE | +| [Registry](#registry) | Container registry, allows pushing and pulling of images | [⚙][registry-omnibus] | [✅][registry-charts] | [✅][registry-charts] | [✅](../user/project/container_registry.md#build-and-push-images) | [⤓][registry-source] | [⚙][registry-gdk] | CE & EE | +| [Redis](#redis) | Caching service | [✅][redis-omnibus] | [✅][redis-omnibus] | [✅][redis-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⤓][redis-source] | ✅ | CE & EE | +| [PostgreSQL](#postgresql) | Database | [✅][postgres-omnibus] | [✅][postgres-charts] | [✅][postgres-charts] | [✅](../user/gitlab_com/index.md#postgresql) | [⤓][postgres-source] | ✅ | CE & EE | +| [PgBouncer](#pgbouncer) | Database connection pooling, failover | [⚙][pgbouncer-omnibus] | [❌][pgbouncer-charts] | [❌][pgbouncer-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#database-architecture) | ❌ | ❌ | EE Only | +| [Consul](#consul) | Database node discovery, failover | [⚙][consul-omnibus] | [❌][consul-charts] | [❌][consul-charts] | [✅](../user/gitlab_com/index.md#consul) | ❌ | ❌ | EE Only | +| [GitLab self-monitoring: Prometheus](#prometheus) | Time-series database, metrics collection, and query service | [✅][prometheus-omnibus] | [✅][prometheus-charts] | [⚙][prometheus-charts] | [✅](../user/gitlab_com/index.md#prometheus) | ❌ | ❌ | CE & EE | +| [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | [✅][alertmanager-omnibus] | [✅][alertmanager-charts] | [⚙][alertmanager-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | [⚙][grafana-omnibus] | [⤓][grafana-charts] | [⤓][grafana-charts] | [✅](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s) | ❌ | ❌ | CE & EE | +| [GitLab self-monitoring: Sentry](#sentry) | Track errors generated by the GitLab instance | [⤓][sentry-omnibus] | [❌][sentry-charts] | [❌][sentry-charts] | [✅](https://about.gitlab.com/handbook/support/workflows/services/gitlab_com/500_errors.html#searching-sentry) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | +| [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | [❌][jaeger-omnibus] | [❌][jaeger-charts] | [❌][jaeger-charts] | [❌](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104) | [⤓][jaeger-source] | [⚙][jaeger-gdk] | CE & EE | +| [Redis Exporter](#redis-exporter) | Prometheus endpoint with Redis metrics | [✅][redis-exporter-omnibus] | [✅][redis-exporter-charts] | [✅][redis-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [Postgres Exporter](#postgres-exporter) | Prometheus endpoint with PostgreSQL metrics | [✅][postgres-exporter-omnibus] | [✅][postgres-exporter-charts] | [✅][postgres-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [PgBouncer Exporter](#pgbouncer-exporter) | Prometheus endpoint with PgBouncer metrics | [⚙][pgbouncer-exporter-omnibus] | [❌][pgbouncer-exporter-charts] | [❌][pgbouncer-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [GitLab Monitor](#gitlab-monitor) | Generates a variety of GitLab metrics | [✅][gitlab-monitor-omnibus] | [❌][gitab-monitor-charts] | [❌][gitab-monitor-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [Node Exporter](#node-exporter) | Prometheus endpoint with system metrics | [✅][node-exporter-omnibus] | [❌][node-exporter-charts] | [❌][node-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [Mattermost](#mattermost) | Open-source Slack alternative | [⚙][mattermost-omnibus] | [⤓][mattermost-charts] | [⤓][mattermost-charts] | [⤓](../user/project/integrations/mattermost_slash_commands.md#manual-configuration), [⤓](../user/project/integrations/mattermost.html) | ❌ | ❌ | CE & EE | +| [MinIO](#minio) | Object storage service | [⤓][minio-omnibus] | [✅][minio-charts] | [✅][minio-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#storage-architecture) | ❌ | [⚙][minio-gdk] | CE & EE | +| [Runner](#gitlab-runner) | Executes GitLab CI jobs | [⤓][runner-omnibus] | [✅][runner-charts] | [⚙][runner-charts] | [✅](../user/gitlab_com/index.md#shared-runners) | [⚙][runner-source] | [⚙][runner-gdk] | CE & EE | +| [Database Migrations](#database-migrations) | Database migrations | [✅][database-migrations-omnibus] | [✅][database-migrations-charts] | [✅][database-migrations-charts] | ✅ | [⚙][database-migrations-source] | ✅ | CE & EE | +| [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | [✅][certificate-management-omnibus] | [✅][certificate-management-charts] | [⚙][certificate-management-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | [⚙][certificate-management-source] | [⚙][certificate-management-gdk] | CE & EE | +| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [⚙][geo-omnibus] | [❌][geo-charts] | [❌][geo-charts] | ✅ | ❌ | [⚙][geo-gdk] | EE Only | +| [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [⤓][ldap-omnibus] | [⤓][ldap-charts] | [⤓][ldap-charts] | [❌](https://about.gitlab.com/pricing/#gitlab-com) | [⤓][gitlab-yml] | [⤓][ldap-gdk] | CE & EE | +| [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [⤓][outbound-email-omnibus] | [⤓][outbound-email-charts] | [⤓][outbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | +| [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [⤓][inbound-email-omnibus] | [⤓][inbound-email-charts] | [⤓][inbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | +| [ElasticSearch](#elasticsearch) | Improved search within GitLab | [⤓][elasticsearch-omnibus] | [⤓][elasticsearch-charts] | [⤓][elasticsearch-charts] | [❌](https://gitlab.com/groups/gitlab-org/-/epics/153) | [⤓][elasticsearch-source] | [⤓][elasticsearch-gdk] | EE Only | +| [Sentry integration](#sentry) | Error tracking for deployed apps | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | CE & EE | +| [Jaeger integration](#jaeger) | Distributed tracing for deployed apps | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | EE Only | +| [GitLab Managed Apps](#gitlab-managed-apps) | Deploy [Helm](https://docs.helm.sh/), [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/), [Cert-Manager](https://docs.cert-manager.io/en/latest/), [Prometheus](https://prometheus.io/docs/introduction/overview/), a [Runner](https://docs.gitlab.com/runner/), [JupyterHub](http://jupyter.org/), [Knative](https://cloud.google.com/knative) to a cluster | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | CE & EE | ### Component details @@ -164,13 +164,14 @@ GitLab can be considered to have two layers from a process perspective: - [Project page](https://github.com/prometheus/alertmanager/blob/master/README.md) - Configuration: [Omnibus][alertmanager-omnibus], [Charts][alertmanager-charts] - Layer: Monitoring +- Process: `alertmanager` [Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. #### Certificate management - Project page: [Omnibus](https://github.com/certbot/certbot/blob/master/README.rst), [Charts](https://github.com/jetstack/cert-manager/blob/master/README.md) -- Configuration: [Omnibus][certificate-management-omnibus], [Charts][certificate-management-charts] +- Configuration: [Omnibus][certificate-management-omnibus], [Charts][certificate-management-charts], [Source][certificate-management-source], [GDK][certificate-management-gdk] - Layer: Core Service (Processor) #### Consul @@ -183,13 +184,13 @@ Consul is a tool for service discovery and configuration. Consul is distributed, #### Database migrations -- Configuration: [Omnibus][registry-omnibus], [Charts][registry-charts] +- Configuration: [Omnibus][registry-omnibus], [Charts][registry-charts], [Source][database-migrations-source] - Layer: Core Service (Data) #### Elasticsearch - [Project page](https://github.com/elastic/elasticsearch/blob/master/README.textile) -- Configuration: [Omnibus][elasticsearch-omnibus], [Charts][elasticsearch-charts] +- Configuration: [Omnibus][elasticsearch-omnibus], [Charts][elasticsearch-charts], [Source][elasticsearch-source], [GDK][elasticsearch-gdk] - Layer: Core Service (Data) Elasticsearch is a distributed RESTful search engine built for the cloud. @@ -197,14 +198,15 @@ Elasticsearch is a distributed RESTful search engine built for the cloud. #### Gitaly - [Project page](https://gitlab.com/gitlab-org/gitaly/blob/master/README.md) -- Configuration: [Omnibus][gitaly-omnibus], [Charts][gitaly-charts] +- Configuration: [Omnibus][gitaly-omnibus], [Charts][gitaly-charts], [Source][gitaly-source] - Layer: Core Service (Data) +- Process: `gitaly` Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). #### Gitlab Geo -- Configuration: [Omnibus][geo-omnibus], [Charts][geo-charts] +- Configuration: [Omnibus][geo-omnibus], [Charts][geo-charts], [GDK][geo-gdk] - Layer: Core Service (Processor) #### Gitlab Monitor @@ -212,12 +214,13 @@ Gitaly is a service designed by GitLab to remove our need for NFS for Git storag - [Project page](https://gitlab.com/gitlab-org/gitlab-monitor) - Configuration: [Omnibus][gitlab-monitor-omnibus], [Charts][gitlab-monitor-charts] - Layer: Monitoring +- Process: `gitlab-monitor` GitLab Monitor is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor). #### Gitlab Pages -- Configuration: [Omnibus][pages-omnibus], [Charts][pages-charts] +- Configuration: [Omnibus][pages-omnibus], [Charts][pages-charts], [Source][pages-source], [GDK][pages-gdk] - Layer: Core Service (Processor) GitLab Pages is a feature that allows you to publish static websites directly from a repository in GitLab. @@ -227,7 +230,7 @@ You can use it either for personal or business websites, such as portfolios, doc #### Gitlab Runner - [Project page](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/README.md) -- Configuration: [Omnibus][runner-omnibus], [Charts][runner-charts] +- Configuration: [Omnibus][runner-omnibus], [Charts][runner-charts], [Source][runner-source], [GDK][runner-gdk] - Layer: Core Service (Processor) GitLab Runner runs tests and sends the results to GitLab. @@ -237,7 +240,7 @@ GitLab CI is the open-source continuous integration service included with GitLab #### Gitlab Shell - [Project page](https://gitlab.com/gitlab-org/gitlab-shell/blob/master/README.md) -- Configuration: [Omnibus][shell-omnibus], [Charts][shell-charts] +- Configuration: [Omnibus][shell-omnibus], [Charts][shell-charts], [Source][shell-source], [GDK][gitlab-yml] - Layer: Core Service (Processor) [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) is a program designed at GitLab to handle ssh-based `git` sessions, and modifies the list of authorized keys. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh. @@ -245,8 +248,9 @@ GitLab CI is the open-source continuous integration service included with GitLab #### Gitlab Workhorse - [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md) -- Configuration: [Omnibus][gitlab-workhorse-omnibus], [Charts][gitlab-workhorse-charts] +- Configuration: [Omnibus][gitlab-workhorse-omnibus], [Charts][gitlab-workhorse-charts], [Source][workhorse-source] - Layer: Core Service (Processor) +- Process: `gitlab-workhorse` [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. @@ -261,7 +265,7 @@ Grafana is an open source, feature rich metrics dashboard and graph editor for G #### Jaeger - [Project page](https://github.com/jaegertracing/jaeger/blob/master/README.md) -- Configuration: [Omnibus][jaeger-omnibus], [Charts][jaeger-charts] +- Configuration: [Omnibus][jaeger-omnibus], [Charts][jaeger-charts], [Source][jaeger-source], [GDK][jaeger-gdk] - Layer: Monitoring Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system. It can be used for monitoring microservices-based distributed systems. @@ -271,6 +275,7 @@ Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system. It c - [Project page](https://github.com/logrotate/logrotate/blob/master/README.md) - Configuration: [Omnibus](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) - Layer: Core Service +- Process: `logrotate` GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. @@ -285,7 +290,7 @@ Mattermost is an open source, private cloud, Slack-alternative from https://matt #### MinIO - [Project page](https://github.com/minio/minio/blob/master/README.md) -- Configuration: [Omnibus][minio-omnibus], [Charts][minio-charts] +- Configuration: [Omnibus][minio-omnibus], [Charts][minio-charts], [GDK][minio-gdk] - Layer: Core Service (Data) MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB. @@ -293,8 +298,9 @@ MinIO is an object storage server released under Apache License v2.0. It is comp #### NGINX - Project page: [Omnibus](https://github.com/nginx/nginx), [Charts](https://github.com/kubernetes/ingress-nginx/blob/master/README.md) -- Configuration: [Omnibus][nginx-omnibus], [Charts][nginx-charts] +- Configuration: [Omnibus][nginx-omnibus], [Charts][nginx-charts], [Source][nginx-source] - Layer: Core Service (Processor) +- Process: `nginx` Nginx as an ingress port for all HTTP requests and routes them to the approriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver. @@ -303,6 +309,7 @@ Nginx as an ingress port for all HTTP requests and routes them to the approriate - [Project page](https://github.com/prometheus/node_exporter/blob/master/README.md) - Configuration: [Omnibus][node-exporter-omnibus], [Charts][node-exporter-charts] - Layer: Monitoring +- Process: `node-exporter` [Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine (think CPU/Disk/Load). It's just a packaged version of the common open source offering from the Prometheus project. @@ -325,8 +332,9 @@ Prometheus exporter for PgBouncer. Exports metrics at 9127/metrics. #### Postgresql - [Project page](https://github.com/postgres/postgres/blob/master/README) -- Configuration: [Omnibus][postgres-omnibus], [Charts][postgres-charts] +- Configuration: [Omnibus][postgres-omnibus], [Charts][postgres-charts], [Source][postgres-source] - Layer: Core Service (Data) +- Process: `postgresql` GitLab packages the popular Database to provide storage for Application meta data and user information. @@ -335,6 +343,7 @@ GitLab packages the popular Database to provide storage for Application meta dat - [Project page](https://github.com/wrouesnel/postgres_exporter/blob/master/README.md) - Configuration: [Omnibus][postgres-exporter-omnibus], [Charts][postgres-exporter-charts] - Layer: Monitoring +- Process: `postgres-exporter` [Postgres-exporter](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about Postgres to Prometheus for use in Grafana Dashboards. @@ -343,14 +352,16 @@ GitLab packages the popular Database to provide storage for Application meta dat - [Project page](https://github.com/prometheus/prometheus/blob/master/README.md) - Configuration: [Omnibus][prometheus-omnibus], [Charts][prometheus-charts] - Layer: Monitoring +- Process: `prometheus` Prometheus is a time-series tool that helps GitLab administrators expose metrics about the individual processes used to provide GitLab the service. #### Redis - [Project page](https://github.com/antirez/redis/blob/unstable/README.md) -- Configuration: [Omnibus][redis-omnibus], [Charts][redis-charts] +- Configuration: [Omnibus][redis-omnibus], [Charts][redis-charts], [Source][redis-source] - Layer: Core Service (Data) +- Process: `redis` Redis is packaged to provide a place to store: @@ -363,13 +374,14 @@ Redis is packaged to provide a place to store: - [Project page](https://github.com/oliver006/redis_exporter/blob/master/README.md) - Configuration: [Omnibus][redis-exporter-omnibus], [Charts][redis-exporter-charts] - Layer: Monitoring +- Process: `redis-exporter` [Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Grafana. #### Registry - [Project page](https://github.com/docker/distribution/blob/master/README.md) -- Configuration: [Omnibus][registry-omnibus], [Charts][registry-charts] +- Configuration: [Omnibus][registry-omnibus], [Charts][registry-charts], [Source][registry-source], [GDK][registry-gdk] - Layer: Core Service (Processor) The registry is what users use to store their own Docker images. The bundled @@ -385,7 +397,7 @@ An external registry can also be configured to use GitLab as an auth endpoint. #### Sentry - [Project page](https://github.com/getsentry/sentry/blob/master/README.rst) -- Configuration: [Omnibus][sentry-omnibus], [Charts][sentry-charts] +- Configuration: [Omnibus][sentry-omnibus], [Charts][sentry-charts], [Source][gitlab-yml], [GDK][gitlab-yml] - Layer: Monitoring Sentry fundamentally is a service that helps you monitor and fix crashes in realtime. The server is in Python, but it contains a full API for sending events from any language, in any application. @@ -393,40 +405,42 @@ Sentry fundamentally is a service that helps you monitor and fix crashes in real #### Sidekiq - [Project page](https://github.com/mperham/sidekiq/blob/master/README.md) -- Configuration: [Omnibus][sidekiq-omnibus], [Charts][sidekiq-charts] +- Configuration: [Omnibus][sidekiq-omnibus], [Charts][sidekiq-charts], [Source][gitlab-yml], [GDK][gitlab-yml] - Layer: Core Service (Processor) +- Process: `sidekiq` Sidekiq is a Ruby background job processor that pulls jobs from the redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background. #### Unicorn - [Project page](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/README.md) -- Configuration: [Omnibus][unicorn-omnibus], [Charts][unicorn-charts] +- Configuration: [Omnibus][unicorn-omnibus], [Charts][unicorn-charts], [Source][unicorn-source], [GDK][gitlab-yml] - Layer: Core Service (Processor) +- Process: `unicorn` [Unicorn](https://bogomips.org/unicorn/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version. #### LDAP Authentication -- Configuration: [Omnibus][ldap-omnibus], [Charts][ldap-charts] +- Configuration: [Omnibus][ldap-omnibus], [Charts][ldap-charts], [Source][gitlab-yml], [GDK][ldap-gdk] - Layer: Core Service (Processor) #### Outbound Email -- Configuration: [Omnibus][outbound-email-omnibus], [Charts][outbound-email-charts] +- Configuration: [Omnibus][outbound-email-omnibus], [Charts][outbound-email-charts], [Source][gitlab-yml], [GDK][gitlab-yml] - Layer: Core Service (Processor) #### Inbound Email -- Configuration: [Omnibus][inbound-email-omnibus], [Charts][inbound-email-charts] +- Configuration: [Omnibus][inbound-email-omnibus], [Charts][inbound-email-charts], [Source][gitlab-yml], [GDK][gitlab-yml] - Layer: Core Service (Processor) -#### Kubernetes Cluster Apps +#### GitLab Managed Apps -- Configuration: [Omnibus][managed-k8s-apps], [Charts][managed-k8s-apps] +- Configuration: [Omnibus][managed-k8s-apps], [Charts][managed-k8s-apps], [Source][managed-k8s-apps], [GDK][managed-k8s-apps] - Layer: Core Service (Processor) -GitLab provides [GitLab Managed Apps](https://docs.gitlab.com/ee/user/project/clusters/#installing-applications), a one-click install for various applications which can be added directly to your configured cluster. These applications are needed for Review Apps and deployments when using Auto DevOps. You can install them after you create a cluster. +GitLab provides [GitLab Managed Apps](../user/project/clusters/index.md#installing-applications), a one-click install for various applications which can be added directly to your configured cluster. These applications are needed for Review Apps and deployments when using Auto DevOps. You can install them after you create a cluster. ## GitLab by Request Type @@ -595,68 +609,92 @@ We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/ha [alertmanager-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template [alertmanager-charts]: https://github.com/helm/charts/tree/master/stable/prometheus -[nginx-omnibus]: https://docs.gitlab.com/omnibus/settings/nginx.html -[nginx-charts]: https://docs.gitlab.com/charts/charts/nginx/index.html +[nginx-omnibus]: https://docs.gitlab.com/omnibus/settings/ +[nginx-charts]: https://docs.gitlab.com/charts/charts/nginx/ +[nginx-source]: ../install/installation.md#9-nginx [unicorn-omnibus]: https://docs.gitlab.com/omnibus/settings/unicorn.html -[unicorn-charts]: https://docs.gitlab.com/charts/charts/gitlab/unicorn/index.html +[unicorn-charts]: https://docs.gitlab.com/charts/charts/gitlab/unicorn/ +[unicorn-source]: ../install/installation.md#configure-it +[gitlab-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example [sidekiq-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template -[sidekiq-charts]: https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html -[gitaly-omnibus]: https://docs.gitlab.com/ee/administration/gitaly/ -[gitaly-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitaly/index.html +[sidekiq-charts]: https://docs.gitlab.com/charts/charts/gitlab/sidekiq/ +[gitaly-omnibus]: ../administration/gitaly/index.md +[gitaly-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitaly/ +[gitaly-source]: ../install/installation.md#install-gitaly [workhorse-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template -[workhorse-charts]: https://docs.gitlab.com/charts/charts/gitlab/unicorn/index.html +[workhorse-charts]: https://docs.gitlab.com/charts/charts/gitlab/unicorn/ +[workhorse-source]: ../install/installation.md#install-gitlab-workhorse [shell-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template -[shell-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/index.html -[pages-omnibus]: https://docs.gitlab.com/ee/administration/pages/ +[shell-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/ +[shell-source]: ../install/installation.md#install-gitlab-shell +[pages-omnibus]: ../administration/pages/index.md [pages-charts]: https://gitlab.com/charts/gitlab/issues/37 -[registry-omnibus]: https://docs.gitlab.com/ee/administration/container_registry.html#container-registry-domain-configuration -[registry-charts]: https://docs.gitlab.com/charts/charts/registry/index.html +[pages-source]: ../install/installation.md#install-gitlab-pages +[pages-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/pages.md +[registry-omnibus]: ../administration/container_registry.md#container-registry-domain-configuration +[registry-charts]: https://docs.gitlab.com/charts/charts/registry/ +[registry-source]: ../administration/container_registry.md#enable-the-container-registry +[registry-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/registry.md [redis-omnibus]: https://docs.gitlab.com/omnibus/settings/redis.html -[redis-charts]: https://docs.gitlab.com/charts/charts/redis/index.html +[redis-charts]: https://docs.gitlab.com/charts/charts/redis/ +[redis-source]: ../install/installation.md#7-redis [postgres-omnibus]: https://docs.gitlab.com/omnibus/settings/database.html [postgres-charts]: https://github.com/helm/charts/tree/master/stable/postgresql -[pgbouncer-omnibus]: https://docs.gitlab.com/ee/administration/high_availability/pgbouncer.html +[postgres-source]: ../install/installation.md#6-database +[pgbouncer-omnibus]: ../administration/high_availability/pgbouncer.md [pgbouncer-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql -[consul-omnibus]: https://docs.gitlab.com/ee/administration/high_availability/consul.html +[consul-omnibus]: ../administration/high_availability/consul.md [consul-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql -[prometheus-omnibus]: https://docs.gitlab.com/ee/administration/monitoring/prometheus/ +[prometheus-omnibus]: ../administration/monitoring/prometheus/index.md [prometheus-charts]: https://github.com/helm/charts/tree/master/stable/prometheus -[grafana-omnibus]: https://docs.gitlab.com/ee/administration/monitoring/performance/grafana_configuration.html +[grafana-omnibus]: ../administration/monitoring/performance/grafana_configuration.md [grafana-charts]: https://github.com/helm/charts/tree/master/stable/grafana [sentry-omnibus]: https://docs.gitlab.com/omnibus/settings/configuration.html#error-reporting-and-logging-with-sentry [sentry-charts]: https://gitlab.com/charts/gitlab/issues/1319 [jaeger-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104 [jaeger-charts]: https://gitlab.com/charts/gitlab/issues/1320 -[redis-exporter-omnibus]: https://docs.gitlab.com/ee/administration/monitoring/prometheus/redis_exporter.html -[redis-exporter-charts]: https://docs.gitlab.com/charts/charts/redis/index.html -[postgres-exporter-omnibus]: https://docs.gitlab.com/ee/administration/monitoring/prometheus/postgres_exporter.html +[jaeger-source]: ../development/distributed_tracing.md#enabling-distributed-tracing +[jaeger-gdk]: ../development/distributed_tracing.html#using-jaeger-in-the-gitlab-development-kit +[redis-exporter-omnibus]: ../administration/monitoring/prometheus/redis_exporter.md +[redis-exporter-charts]: https://docs.gitlab.com/charts/charts/redis/ +[postgres-exporter-omnibus]: ../administration/monitoring/prometheus/postgres_exporter.md [postgres-exporter-charts]: https://github.com/helm/charts/tree/master/stable/postgresql -[pgbouncer-exporter-omnibus]: https://docs.gitlab.com/ee/administration/monitoring/prometheus/pgbouncer_exporter.html +[pgbouncer-exporter-omnibus]: ../administration/monitoring/prometheus/pgbouncer_exporter.md [pgbouncer-exporter-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql -[gitlab-monitor-omnibus]: https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_monitor_exporter.html +[gitlab-monitor-omnibus]: ../administration/monitoring/prometheus/gitlab_monitor_exporter.md [gitab-monitor-charts]: https://gitlab.com/charts/gitlab/issues/319 -[node-exporter-omnibus]: https://docs.gitlab.com/ee/administration/monitoring/prometheus/node_exporter.html +[node-exporter-omnibus]: ../administration/monitoring/prometheus/node_exporter.md [node-exporter-charts]: https://gitlab.com/charts/gitlab/issues/1332 [mattermost-omnibus]: https://docs.gitlab.com/omnibus/gitlab-mattermost/ [mattermost-charts]: https://docs.mattermost.com/install/install-mmte-helm-gitlab-helm.html [minio-omnibus]: https://min.io/download -[minio-charts]: https://docs.gitlab.com/charts/charts/minio/index.html +[minio-charts]: https://docs.gitlab.com/charts/charts/minio/ +[minio-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/object_storage.md [runner-omnibus]: https://docs.gitlab.com/runner/ [runner-charts]: https://docs.gitlab.com/runner/install/kubernetes.html +[runner-source]: https://docs.gitlab.com/runner/ +[runner-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/runner.md [database-migrations-omnibus]: https://docs.gitlab.com/omnibus/settings/database.html#disabling-automatic-database-migration -[database-migrations-charts]: https://docs.gitlab.com/charts/charts/gitlab/migrations/index.html +[database-migrations-charts]: https://docs.gitlab.com/charts/charts/gitlab/migrations/ +[database-migrations-source]: ../update/upgrading_from_source.md#13-install-libs-migrations-etc [certificate-management-omnibus]: https://docs.gitlab.com/omnibus/settings/ssl.html [certificate-management-charts]: https://docs.gitlab.com/charts/installation/tls.html -[geo-omnibus]: https://docs.gitlab.com/ee/administration/geo/replication/index.html#setup-instructions +[certificate-management-source]: ../install/installation.md#using-https +[certificate-management-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/https.md +[geo-omnibus]: ../administration/geo/replication/index.md#setup-instructions [geo-charts]: https://gitlab.com/charts/gitlab/issues/8 -[ldap-omnibus]: https://docs.gitlab.com/ee/administration/auth/ldap.html +[geo-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/geo.md +[ldap-omnibus]: ../administration/auth/ldap.md [ldap-charts]: https://docs.gitlab.com/charts/charts/globals.html#ldap +[ldap-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/ldap.md [outbound-email-omnibus]: https://docs.gitlab.com/omnibus/settings/smtp.html [outbound-email-charts]: https://docs.gitlab.com/charts/installation/command-line-options.html#outgoing-email-configuration -[inbound-email-omnibus]: https://docs.gitlab.com/ee/administration/incoming_email.html +[inbound-email-omnibus]: ../administration/incoming_email.md [inbound-email-charts]: https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration -[elasticsearch-omnibus]: https://docs.gitlab.com/ee/integration/elasticsearch.html -[elasticsearch-charts]: https://docs.gitlab.com/ee/integration/elasticsearch.html -[sentry-integration]: https://docs.gitlab.com/ee/user/project/operations/error_tracking.html -[jaeger-integration]: https://docs.gitlab.com/ee/user/project/operations/tracing.html -[managed-k8s-apps]: https://docs.gitlab.com/ee/user/project/clusters/#installing-applications +[elasticsearch-omnibus]: ../integration/elasticsearch.md +[elasticsearch-charts]: ../integration/elasticsearch.md +[elasticsearch-source]: ../integration/elasticsearch.md +[elasticsearch-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/elasticsearch.md +[sentry-integration]: ../user/project/operations/error_tracking.md +[jaeger-integration]: ../user/project/operations/tracing.md +[managed-k8s-apps]: ../user/project/clusters/index.md#installing-applications diff --git a/doc/development/code_review.md b/doc/development/code_review.md index c4e5995714d..29e2aa1a581 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -27,7 +27,7 @@ Depending on the areas your merge request touches, it must be **approved** by on or more [maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#maintainer): For approvals, we use the approval functionality found in the merge request -widget. Reviewers can add their approval by [approving additionally](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html#adding-or-removing-an-approval). +widget. Reviewers can add their approval by [approving additionally](../user/project/merge_requests/merge_request_approvals.md#adding-or-removing-an-approval). Getting your merge request **merged** also requires a maintainer. If it requires more than one approval, the last maintainer to review and approve it will also merge it. @@ -43,7 +43,7 @@ It picks reviewers and maintainers from the list at the [engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page, with these behaviours: -1. It will not pick people whose [GitLab status](../user/profile/#current-status) +1. It will not pick people whose [GitLab status](../user/profile/index.md#current-status) contains the string 'OOO'. 2. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) are three times as likely to be picked as other reviewers. @@ -152,7 +152,7 @@ required approvers. Maintainers must check before merging if the merge request is introducing new vulnerabilities, by inspecting the list in the Merge Request [Security -Widget](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate). +Widget](../user/project/merge_requests/index.md#security-reports-ultimate). When in doubt, a [Security Engineer][team] can be involved. The list of detected vulnerabilities must be either empty or containing: @@ -286,7 +286,7 @@ experience, refactors the existing code). Then: author has already set this option or if the merge request clearly contains a messy commit history that is intended to be squashed. -[squash-and-merge]: https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html#squash-and-merge +[squash-and-merge]: ../user/project/merge_requests/squash_and_merge.md#squash-and-merge ### The right balance @@ -319,7 +319,7 @@ reviewee. GitLab is used in a lot of places. Many users use our [Omnibus packages](https://about.gitlab.com/installation/), but some use the [Docker images](https://docs.gitlab.com/omnibus/docker/), some are -[installed from source](https://docs.gitlab.com/ce/install/installation.html), +[installed from source](../install/installation.md), and there are other installation methods available. GitLab.com itself is a large Enterprise Edition instance. This has some implications: diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 8b1d014e101..59cf5014da4 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -87,7 +87,7 @@ Sometimes style guides will be followed but the code will lack structural integr GitLab will do its best to review community contributions as quickly as possible. Specially appointed developers review community contributions daily. You may take a look at the [team page](https://about.gitlab.com/team/) for the merge request coach who specializes in the type of code you have written and mention them in the merge request. For example, if you have written some JavaScript in your code then you should mention the frontend merge request coach. If your code has multiple disciplines you may mention multiple merge request coaches. -GitLab receives a lot of community contributions, so if your code has not been reviewed within 4 days of its initial submission feel free to re-mention the appropriate merge request coach. +GitLab receives a lot of community contributions, so if your code has not been reviewed within two days (excluding weekend and public holidays) of its initial submission feel free to re-mention the appropriate merge request coach. When submitting code to GitLab, you may feel that your contribution requires the aid of an external library. If your code includes an external library please provide a link to the library, as well as reasons for including it. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 0e1ab8663ed..e3a1dc711fd 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -7,7 +7,7 @@ scheduling into milestones. Labelling is a task for everyone. Most issues will have labels for at least one of the following: - Type: ~feature, ~bug, ~customer, etc. -- Subject: ~wiki, ~"container registry", ~ldap, ~api, ~frontend, etc. +- Subject: ~wiki, ~"Container Registry", ~ldap, ~api, ~frontend, etc. - Team: ~Plan, ~Manage, ~Quality, etc. - Stage: ~"devops:plan", ~"devops:create", etc. - Release Scoping: ~Deliverable, ~Stretch, ~"Next Patch Release" @@ -44,7 +44,7 @@ Subject labels are labels that define what area or feature of GitLab this issue hits. They are not always necessary, but very convenient. Examples of subject labels are ~wiki, ~ldap, ~api, -~issues, ~"merge requests", ~labels, and ~"container registry". +~issues, ~"merge requests", ~labels, and ~"Container Registry". If you are an expert in a particular area, it makes it easier to find issues to work on. You can also subscribe to those labels to receive an email each time an @@ -92,20 +92,21 @@ Stage labels specify which [DevOps stage][devops-stages] the issue belongs to. The current stage labels are: -- ~"devops:manage" -- ~"devops:plan" -- ~"devops:create" -- ~"devops:verify" -- ~"devops:package" -- ~"devops:release" -- ~"devops:configure" -- ~"devops:monitor" -- ~"devops:secure" -- ~"devops:defend" -- ~"devops:enablement" - -These labels should be mutually exclusive. If an issue belongs to multiple -stages, the most relevant should be used. +- ~"devops::manage" +- ~"devops::plan" +- ~"devops::create" +- ~"devops::verify" +- ~"devops::package" +- ~"devops::release" +- ~"devops::configure" +- ~"devops::monitor" +- ~"devops::secure" +- ~"devops::defend" +- ~"devops::growth" +- ~"devops::enablement" + +These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) +and thus are mutually exclusive. They differ from the [Team labels](#team-labels) because teams may work on issues outside their stage. @@ -121,6 +122,25 @@ The Stage labels are used to generate the [direction pages][direction-pages] aut [devops-stages]: https://about.gitlab.com/direction/#devops-stages [direction-pages]: https://about.gitlab.com/direction/ +## Group labels + +Group labels specify which [groups][structure-groups] the issue belongs to. + +Examples include: + +- ~"group::control" +- ~"group::editor" + +These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) +and thus are mutually exclusive. + +Groups are nested beneath a particular stage, so only one stage label and one group label +can be applied to a single issue. You can find the groups listed in the +[Product Categories pages][product-categories]. + +[structure-groups]: https://about.gitlab.com/company/team/structure/#groups +[product-categories]: https://about.gitlab.com/handbook/product/categories/ + ## Release Scoping labels Release Scoping labels help us clearly communicate expectations of the work for the diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 038e3de10d7..bfce7488a8d 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -34,7 +34,7 @@ GITLAB_TRACING=opentracing://<driver>?<param_name>=<param_value>&<param_name_2>= In this example, we have the following hypothetical values: - `driver`: the driver. [GitLab supports - `jaeger`](https://docs.gitlab.com/ee/user/project/operations/tracing.html). In future, other + `jaeger`](../user/project/operations/tracing.md). In future, other tracing implementations may also be supported. - `param_name`, `param_value`: these are driver specific configuration values. Configuration parameters for Jaeger are documented [further on in this diff --git a/doc/development/documentation/feature-change-workflow.md b/doc/development/documentation/feature-change-workflow.md index 1f68b6a6a70..ca29353ecbe 100644 --- a/doc/development/documentation/feature-change-workflow.md +++ b/doc/development/documentation/feature-change-workflow.md @@ -86,7 +86,7 @@ Everyone is encouraged to draft the requirements in the issue, but a product man do the following: - When the issue is assigned a release milestone, review and update the Documentation details. -- By the kickoff, finalizie the Documentation details. +- By the kickoff, finalize the Documentation details. ### Developer and maintainer roles @@ -117,7 +117,7 @@ Follow the process below unless otherwise agreed with the product manager and te #### Reviews and merging -All reviewers can help ensure accuracy, clarity, completeness, and adherence to the plans in the issue, as well as the [Documentation Guidelines](https://docs.gitlab.com/ee/development/documentation/) and [Style Guide](https://docs.gitlab.com/ee/development/documentation/styleguide.html). +All reviewers can help ensure accuracy, clarity, completeness, and adherence to the plans in the issue, as well as the [Documentation Guidelines](index.md) and [Style Guide](styleguide.md). - **Prior to merging**, documentation changes committed by the developer must be reviewed by: @@ -136,7 +136,7 @@ All reviewers can help ensure accuracy, clarity, completeness, and adherence to 1. **The maintainer** who is assigned to merge the MR, to verify clarity, completeness, and quality, to the best of their ability. - Upon merging, if a technical writer review has not been performed and there is not yet a linked issue for a follow-up review, the maintainer should [create an issue using the Doc Review template](https://gitlab.com/gitlab-org/gitlab-ce/issues/new?issuable_template=Doc%20Review), link it from the MR, and - mention the original MR author in the new issue. Alternatively, the mainitainer can ask the MR author to create and link this issue before the MR is merged. + mention the original MR author in the new issue. Alternatively, the maintainer can ask the MR author to create and link this issue before the MR is merged. - After merging, documentation changes are reviewed by: @@ -157,14 +157,14 @@ All reviewers can help ensure accuracy, clarity, completeness, and adherence to #### Collaboration By default, the developer will work on documentation changes independently, but -the developer, PM, or technicial writer can propose a broader collaboration for +the developer, PM, or technical writer can propose a broader collaboration for any given issue. Additionally, technical writers are available for questions at any time. #### Review -- Techncial writers provide non-blocking reviews of all documentation changes, +- Technical writers provide non-blocking reviews of all documentation changes, before or after the change is merged. However, if the docs are ready in the MR while there's time before the freeze, the technical writer's review can commence early, on request. - The technical writer will confirm that the doc is clear, grammatically correct, @@ -173,7 +173,7 @@ Additionally, technical writers are available for questions at any time. the developer and code reviewer should have already made a good-faith effort to ensure: - Clarity. - Adherence to the plans and goals in the issue. - - Location (make sure the docs are in the correct directorkes and has the correct name). + - Location (make sure the docs are in the correct directories and has the correct name). - Syntax, typos, and broken links. - Improvements to the content. - Accordance with the [Documentation Style Guide](styleguide.md), and [Structure and Template](structure.md) doc. diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 4bf8401c0e8..6dfd3b2a690 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -466,7 +466,7 @@ If you want to know the in-depth details, here's what's really happening: The following GitLab features are used among others: - [Manual actions](../../ci/yaml/README.md#whenmanual) -- [Multi project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html) +- [Multi project pipelines](../../ci/multi_project_pipeline_graphs.md) - [Review Apps](../../ci/review_apps/index.md) - [Artifacts](../../ci/yaml/README.md#artifacts) - [Specific Runner](../../ci/runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 95b5fcd99a1..fe676efa94d 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -77,8 +77,8 @@ for use (e.g. variations on the main use case), but if that's not applicable, th Examples of use cases on feature pages: - CE and EE: [Issues](../../user/project/issues/index.md#use-cases) - CE and EE: [Merge Requests](../../user/project/merge_requests/index.md) -- EE-only: [Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html) -- EE-only: [Jenkins integration](https://docs.gitlab.com/ee/integration/jenkins.html) +- EE-only: [Geo](../../administration/geo/replication/index.md) +- EE-only: [Jenkins integration](../../integration/jenkins.md) ## Requirements @@ -121,8 +121,8 @@ but commented out to help encourage others to add to it in the future. --> Notes: -- (1): Apply the [tier badges](https://docs.gitlab.com/ee/development/documentation/styleguide.html#product-badges) accordingly -- (2): Apply the correct format for the [GitLab version introducing the feature](https://docs.gitlab.com/ee/development/documentation/styleguide.html#gitlab-versions-and-tiers) +- (1): Apply the [tier badges](styleguide.md#product-badges) accordingly +- (2): Apply the correct format for the [GitLab version introducing the feature](styleguide.md#gitlab-versions-and-tiers) ``` ## Help and feedback section diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index bc472bb5b0a..d0db1a61935 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -459,15 +459,6 @@ Resolving an EE template path that is relative to the CE view path will not work = render_if_exists 'projects/button' # Will render `ee/app/views/projects/_button` ``` -You should not explicitly set render options like `partial` or provide a `locals` hash. -The first argument should be a path string and the second can be a hash replacing `locals`. - -```ruby -render partial: 'projects/button', locals: { project: project } -# becomes -render_if_exists 'projects/button', project: project -``` - #### Using `render_ce` For `render` and `render_if_exists`, they search for the EE partial first, @@ -557,40 +548,56 @@ due to `prepend`, but Grape is complex internally and we couldn't easily do that, so we'll follow regular object-oriented practices that we define the interface first here. -For example, suppose we have a few more optional params for EE, given this CE -API code: +For example, suppose we have a few more optional params for EE. We can move the +params out of the `Grape::API` class to a helper module, so we can `prepend` it +before it would be used in the class. ```ruby module API - class MergeRequests < Grape::API - # EE::API::MergeRequests would override the following helpers - helpers do - params :optional_params_ee do + class Projects < Grape::API + helpers Helpers::ProjectsHelpers + end +end +``` + +Given this CE API `params`: + +```ruby +module API + module Helpers + module ProjectsHelpers + extend ActiveSupport::Concern + extend Grape::API::Helpers + + params :optional_project_params_ce do + # CE specific params go here... end - end - params :optional_params do - # CE specific params go here... + params :optional_project_params_ee do + end - use :optional_params_ee + params :optional_project_params do + use :optional_project_params_ce + use :optional_project_params_ee + end end end end -API::MergeRequests.prepend(EE::API::MergeRequests) +API::Helpers::ProjectsHelpers.prepend(EE::API::Helpers::ProjectsHelpers) ``` -And then we could override it in EE module: +We could override it in EE module: ```ruby module EE module API - module MergeRequests - extend ActiveSupport::Concern + module Helpers + module ProjectsHelpers + extend ActiveSupport::Concern - prepended do - helpers do - params :optional_params_ee do + prepended do + params :optional_project_params_ee do # EE specific params go here... end end @@ -600,9 +607,6 @@ module EE end ``` -This way, the only difference between CE and EE for that API file would be -`prepend EE::API::MergeRequests`. - #### EE helpers To make it easy for an EE module to override the CE helpers, we need to define @@ -933,7 +937,7 @@ export default { ``` #### For JS code that is EE only, like props, computed properties, methods, etc, we will keep the current approach - - Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](https://docs.gitlab.com/ee/development/ee_features.html#javascript-code-in-assetsjavascripts) alias we already have for webpack. + - Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) alias we already have for webpack. - This means all the EE specific props, computed properties, methods, etc that are EE only should be in a mixin in the `ee/` folder and we need to create a CE counterpart of the mixin ##### Example: @@ -960,7 +964,7 @@ import mixin from 'ee_else_ce/path/mixin'; ### Non Vue Files For regular JS files, the approach is similar. -1. We will keep using the [`ee_else_ce`](https://docs.gitlab.com/ee/development/ee_features.html#javascript-code-in-assetsjavascripts) helper, this means that EE only code should be inside the `ee/` folder. +1. We will keep using the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) helper, this means that EE only code should be inside the `ee/` folder. 1. An EE file should be created with the EE only code, and it should extend the CE counterpart. 1. For code inside functions that can't be extended, the code should be moved into a new file and we should use `ee_else_ce` helper: diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 0c9e7908713..8b0f4f02d19 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -2,7 +2,7 @@ This area is to maintain a compendium of useful information when working with elasticsearch. -Information on how to enable ElasticSearch and perform the initial indexing is kept in https://docs.gitlab.com/ee/integration/elasticsearch.html#enabling-elasticsearch +Information on how to enable ElasticSearch and perform the initial indexing is kept in ../integration/elasticsearch.md#enabling-elasticsearch ## Initial installation on OS X @@ -16,7 +16,7 @@ and use `docker stop elastic56` and `docker start elastic56` to stop/start it. ### Installing on the host -We currently only support Elasticsearch [5.6 to 6.x](https://docs.gitlab.com/ee/integration/elasticsearch.html#requirements) +We currently only support Elasticsearch [5.6 to 6.x](../integration/elasticsearch.md#version-requirements) Version 5.6 is available on homebrew and is the recommended version to use in order to test compatibility. @@ -43,7 +43,7 @@ this adds `gitlab-elasticsearch-indexer` to `$GOPATH/bin`, please make sure that - `gitlab:elastic:test:index_size`: Tells you how much space the current index is using, as well as how many documents are in the index. - `gitlab:elastic:test:index_size_change`: Outputs index size, reindexes, and outputs index size again. Useful when testing improvements to indexing size. -Additionally, if you need large repos or multiple forks for testing, please consider [following these instructions](https://docs.gitlab.com/ee/development/rake_tasks.html#extra-project-seed-options) +Additionally, if you need large repos or multiple forks for testing, please consider [following these instructions](rake_tasks.md#extra-project-seed-options) ## How does it work? diff --git a/doc/development/fe_guide/style_guide_js.md b/doc/development/fe_guide/style_guide_js.md index 060cd8baf7f..b50159c2b75 100644 --- a/doc/development/fe_guide/style_guide_js.md +++ b/doc/development/fe_guide/style_guide_js.md @@ -95,6 +95,7 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ #### Modules, Imports, and Exports 1. Use ES module syntax to import modules + ```javascript // bad const SomeClass = require('some_class'); @@ -168,6 +169,7 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ Do not use them anymore and feel free to remove them when refactoring legacy code. 1. Avoid adding to the global namespace. + ```javascript // bad window.MyClass = class { /* ... */ }; @@ -176,7 +178,8 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ export default class MyClass { /* ... */ } ``` -1. Side effects are forbidden in any script which contains exports +1. Side effects are forbidden in any script which contains export + ```javascript // bad export default class MyClass { /* ... */ } @@ -449,6 +452,7 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. #### Props 1. Props should be declared as an object + ```javascript // bad props: ['foo'] @@ -544,14 +548,24 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. <component @click="eventHandler"/> ``` -1. Shorthand `:` is preferable over `v-bind` +2. Shorthand `:` is preferable over `v-bind` ```javascript // bad <component v-bind:class="btn"/> // good - <component :class="btsn"/> + <component :class="btn"/> + ``` + +3. Shorthand `#` is preferable over `v-slot` + + ```javascript + // bad + <template v-slot:header></template> + + // good + <template #header></template> ``` #### Closing tags diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md index 36880dd746d..b25dce65ffe 100644 --- a/doc/development/fe_guide/style_guide_scss.md +++ b/doc/development/fe_guide/style_guide_scss.md @@ -9,20 +9,44 @@ easy to maintain, and performant for the end-user. As part of the effort for [cleaning up our CSS and moving our components into GitLab-UI](https://gitlab.com/groups/gitlab-org/-/epics/950) led by the [GitLab UI WG](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20623) we prefer the use of utility classes over adding new CSS. However, complex CSS can be addressed by adding component classes. -We have a few internal utility classes in [`common.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/framework/common.scss) -and we use [Bootstrap's Utility Classes](https://getbootstrap.com/docs/4.3/utilities/) +#### Where are utility classes defined? + +- [Bootstrap's Utility Classes](https://getbootstrap.com/docs/4.3/utilities/) +- [`common.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/framework/common.scss) (old) +- [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/utilities.scss) (new) + +#### Where should I put new utility classes? New utility classes should be added to [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/utilities.scss). Existing classes include: -**Background color**: `.bg-variant-shade` e.g. `.bg-warning-400` -**Text color**: `.text-variant-shade` e.g. `.text-success-500` +| Name | Pattern | Example | +|------|---------|---------| +| Background color | `.bg-{variant}-{shade}` | `.bg-warning-400` | +| Text color | `.text-{variant}-{shade}` | `.text-success-500` | +| Font size | `.text-{size}` | `.text-2` | + +- `{variant}` is one of 'primary', 'secondary', 'success', 'warning', 'error' +- `{shade}` is on of the shades listed on [colors](https://design.gitlab.com/foundations/colors/) +- `{size}` is a number from 1-6 from our [Type scale](https://design.gitlab.com/foundations/typography) + +#### When should I create component classes? + +We recommend a "utility-first" approach. + +1. Start with utility classes. +2. If composing utility classes into a component class removes code duplication and encapsulates a clear responsibility, do it. + +This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`). + +Examples of component classes that were created using "utility-first" include: -- variant is one of 'primary', 'secondary', 'success', 'warning', 'error' -- shade is on of the shades listed on [colors](https://design.gitlab.com/foundations/colors/) +- [`.circle-icon-container`](https://gitlab.com/gitlab-org/gitlab-ce/blob/579fa8b8ec7eb38d40c96521f517c9dab8c3b97a/app/assets/stylesheets/framework/icons.scss#L85) +- [`.d-flex-center`](https://gitlab.com/gitlab-org/gitlab-ce/blob/900083d89cd6af391d26ab7922b3f64fa2839bef/app/assets/stylesheets/framework/common.scss#L425) -**Font size**: `.text-size` e.g. `.text-2` +Inspiration: -- **size** is number from 1-6 from our [Type scale](https://design.gitlab.com/foundations/typography) +- https://tailwindcss.com/docs/utility-first +- https://tailwindcss.com/docs/extracting-components ### Naming diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 82b09cc0224..c871015aaf6 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -28,7 +28,7 @@ the time, you should execute `/chatops run feature set my_feature_flag 50`. This document only covers feature flags used in the development of GitLab itself. Feature flags in deployed user applications can be found at -[Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html) +[Feature Flags](../user/project/operations/feature_flags.md) ## Developing with feature flags @@ -111,7 +111,7 @@ Whenever a feature flag is present, make sure to test _both_ states of the feature flag. See the -[testing guide](testing_guide/best_practices.html#feature-flags-in-tests) +[testing guide](testing_guide/best_practices.md#feature-flags-in-tests) for information and examples on how to stub feature flags in tests. ## Enabling a feature flag (in development) diff --git a/doc/development/geo.md b/doc/development/geo.md index c8e6a86eb52..6e59fab34c7 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -10,6 +10,7 @@ the diagram below and are described in more detail within this document. ## Replication layer Geo handles replication for different components: + - [Database](#database-replication): includes the entire application, except cache and jobs. - [Git repositories](#repository-replication): includes both projects and wikis. - [Uploaded blobs](#uploads-replication): includes anything from images attached on issues @@ -90,7 +91,7 @@ projects that need updating. Those projects can be: timestamp that is more recent than the `last_repository_successful_sync_at` timestamp in the `Geo::ProjectRegistry` model. - Manual: The admin can manually flag a repository to resync in the - [Geo admin panel](https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html). + [Geo admin panel](../user/admin_area/geo_nodes.md). When we fail to fetch a repository on the secondary `RETRIES_BEFORE_REDOWNLOAD` times, Geo does a so-called _redownload_. It will do a clean clone @@ -209,20 +210,38 @@ bundle exec rake geo:db:migrate ### Foreign Data Wrapper -The use of [FDW](#fdw) was introduced in GitLab 10.1. +> Introduced in GitLab 10.1. + +Foreign Data Wrapper ([FDW](#fdw)) is used by the [Geo Log Cursor](#geo-log-cursor) and improves +the performance of many synchronization operations. -This is useful for the [Geo Log Cursor](#geo-log-cursor) and improves -the performance of some synchronization operations. +FDW is a PostgreSQL extension ([`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html)) that is enabled within +the Geo Tracking Database (on a **secondary** node), which allows it +to connect to the readonly database replica and perform queries and filter +data from both instances. While FDW is available in older versions of PostgreSQL, we needed to raise the minimum required version to 9.6 as this includes many performance improvements to the FDW implementation. +This persistent connection is configured as an FDW server +named `gitlab_secondary`. This configuration exists within the database's user +context only. To access the `gitlab_secondary`, GitLab needs to use the +same database user that had previously been configured. + +The Geo Tracking Database accesses the readonly database replica via FDW as a regular user, +limited by its own restrictions. The credentials are configured as a +`USER MAPPING` associated with the `SERVER` mapped previously +(`gitlab_secondary`). + +FDW configuration and credentials definition are managed automatically by the +Omnibus GitLab `gitlab-ctl reconfigure` command. + #### Refeshing the Foreign Tables -Whenever the database schema changes on the **primary** node, the -**secondary** node will need to refresh its foreign tables by running -the following: +Whenever a new Geo node is configured or the database schema changes on the +**primary** node, you must refresh the foreign tables on the **secondary** node +by running the following: ```sh bundle exec rake geo:db:refresh_foreign_tables @@ -243,6 +262,53 @@ STATEMENT: SELECT a.attname, format_type(a.atttypid, a.atttypmod) ORDER BY a.attnum ``` +#### Accessing data from a Foreign Table + +At the SQL level, all you have to do is `SELECT` data from `gitlab_secondary.*`. + +Here's an example of how to access all projects from the Geo Tracking Database's FDW: + +```sql +SELECT * FROM gitlab_secondary.projects; +``` + +As a more real-world example, this is how you filter for unarchived projects +on the Tracking Database: + +```sql +SELECT project_registry.* + FROM project_registry + JOIN gitlab_secondary.projects + ON (project_registry.project_id = gitlab_secondary.projects.id + AND gitlab_secondary.projects.archived IS FALSE) +``` + +At the ActiveRecord level, we have additional Models that represent the +foreign tables. They must be mapped in a slightly different way, and they are read-only. + +Check the existing FDW models in `ee/app/models/geo/fdw` for reference. + +From a developer's perspective, it's no different than creating a model that +represents a Database View. + +With the examples above, you can access the projects with: + +```ruby +Geo::Fdw::Project.all +``` + +and to access the `ProjectRegistry` filtering by unarchived projects: + +```ruby +# We have to use Arel here: +project_registry_table = Geo::ProjectRegistry.arel_table +fdw_project_table = Geo::Fdw::Project.arel_table + +project_registry_table.join(fdw_project_table) + .on(project_registry_table[:project_id].eq(fdw_project_table[:id])) + .where((fdw_project_table[:archived]).eq(true)) # if you append `.to_sql` you can check generated query +``` + ## Finders Geo uses [Finders](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/app/finders), @@ -299,7 +365,7 @@ basically hashes all Git refs together and stores that hash in the The **secondary** node does the same to calculate the hash of its clone, and compares the hash with the value the **primary** node calculated. If there is a mismatch, Geo will mark this as a mismatch -and the administrator can see this in the [Geo admin panel](https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html). +and the administrator can see this in the [Geo admin panel](../user/admin_area/geo_nodes.md). ## Glossary diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index cf78b792ffd..4dad8815fcb 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -40,7 +40,7 @@ of possible security breaches in our code: - SQL injections Remember to run -[SAST](https://docs.gitlab.com/ee/user/application_security/sast/index) +[SAST](../../user/application_security/sast/index.md) **[ULTIMATE]** on your project (or at least the [gosec analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), and to follow our [Security @@ -94,9 +94,9 @@ become available, you will be able to share job templates like this Dependencies should be kept to the minimum. The introduction of a new dependency should be argued in the merge request, as per our [Approval Guidelines](../code_review.md#approval-guidelines). Both [License -Management](https://docs.gitlab.com/ee/user/project/merge_requests/license_management.html) +Management](../../user/project/merge_requests/license_management.md) **[ULTIMATE]** and [Dependency -Scanning](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index) +Scanning](../../user/application_security/dependency_scanning/index.md) **[ULTIMATE]** should be activated on all projects to ensure new dependencies security status and license compatibility. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 38d56322de8..38425674567 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -195,6 +195,7 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Sometimes you need to add some context to the text that you want to translate (if the word occurs in a sentence and/or the word is ambiguous). +Namespaces should be PascalCase. - In Ruby/HAML: diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index 85284d8c714..eadf1cd74c5 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -32,7 +32,7 @@ clicking `Pause sync` on the [Crowdin integration settings page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). When all failures are resolved, the translations need to be double -checked once more as discussed in [confidential issue](https://docs.gitlab.com/ee/user/project/issues/confidential_issues.html) `https://gitlab.com/gitlab-org/gitlab-ce/issues/37850`. +checked once more as discussed in [confidential issue](../../user/project/issues/confidential_issues.md) `https://gitlab.com/gitlab-org/gitlab-ce/issues/37850`. ## Merging translations diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index eb492c9818b..fb5cfb6c157 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -27,7 +27,7 @@ are very appreciative of the work done by translators and proofreaders! - Czech - Proofreaders needed. - Danish - - Proofreaders needed. + - Saederup92 - [GitLab](https://gitlab.com/Saederup92), [Crowdin](https://crowdin.com/profile/Saederup92) - Dutch - Emily Hendle - [GitLab](https://gitlab.com/pundachan), [Crowdin](https://crowdin.com/profile/pandachan) - Esperanto diff --git a/doc/development/import_export.md b/doc/development/import_export.md index f7f48b03651..fd067b80c16 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -27,7 +27,7 @@ Read through the current performance problems using the Import/Export below. ### OOM errors -Out of memory (OOM) errors are normally caused by the [Sidekiq Memory Killer](https://docs.gitlab.com/ee/administration/operations/sidekiq_memory_killer.html): +Out of memory (OOM) errors are normally caused by the [Sidekiq Memory Killer](../administration/operations/sidekiq_memory_killer.md): ```bash SIDEKIQ_MEMORY_KILLER_MAX_RSS = 2GB in GitLab.com diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 1657d73e0c9..6f3dd59b2c3 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -7,8 +7,8 @@ on-premise or GitLab.com plans and features. ## Restricting features scoped by namespaces or projects GitLab.com plans are persisted on user groups and namespaces, therefore, if you're adding a -feature such as [Related issues](https://docs.gitlab.com/ee/user/project/issues/related_issues.html) or -[Service desk](https://docs.gitlab.com/ee/user/project/service_desk.html), +feature such as [Related issues](../user/project/issues/related_issues.md) or +[Service desk](../user/project/service_desk.md), it should be restricted on namespace scope. 1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in @@ -22,8 +22,8 @@ project.feature_available?(:feature_symbol) ## Restricting global features (instance) -However, for features such as [Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html) and -[Load balancing](https://docs.gitlab.com/ee/administration/database_load_balancing.html), which cannot be restricted +However, for features such as [Geo](../administration/geo/replication/index.md) and +[Load balancing](../administration/database_load_balancing.md), which cannot be restricted to only a subset of projects or namespaces, the check will be made directly in the instance license. diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 0c326eeb851..9b26f691b55 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -186,7 +186,11 @@ end When adding a foreign-key constraint to either an existing or new column remember to also add a index on the column. -This is _required_ for all foreign-keys. +This is **required** for all foreign-keys, e.g., to support efficient cascading +deleting: when a lot of rows in a table get deleted, the referenced records need +to be deleted too. The database has to look for corresponding records in the +referenced table. Without an index, this will result in a sequential scan on the +table which can take a long time. Here's an example where we add a new column with a foreign key constraint. Note it includes `index: true` to create an index for it. diff --git a/doc/development/new_fe_guide/development/components.md b/doc/development/new_fe_guide/development/components.md index 8ae58d30c35..963ce53423b 100644 --- a/doc/development/new_fe_guide/development/components.md +++ b/doc/development/new_fe_guide/development/components.md @@ -17,5 +17,5 @@ D3 is very popular across many projects outside of GitLab: Within GitLab, D3 has been used for the following notable features -- [Prometheus graphs](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html) +- [Prometheus graphs](../../../user/project/integrations/prometheus.md) - Contribution calendars diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md index 889a5aab2b7..4564f678ec0 100644 --- a/doc/development/new_fe_guide/tips.md +++ b/doc/development/new_fe_guide/tips.md @@ -10,16 +10,16 @@ yarn clean ## Creating feature flags in development -The process for creating a feature flag is the same as [enabling a feature flag in development](https://docs.gitlab.com/ee/development/feature_flags.html#enabling-a-feature-flag-in-development). +The process for creating a feature flag is the same as [enabling a feature flag in development](../feature_flags.md#enabling-a-feature-flag-in-development). Your feature flag can now be: -- [made available to the frontend](https://docs.gitlab.com/ee/development/feature_flags.html#frontend) via the `gon` -- queried in [tests](https://docs.gitlab.com/ee/development/feature_flags.html#specs) +- [made available to the frontend](../feature_flags.md#frontend) via the `gon` +- queried in [tests](../feature_flags.md#specs) - queried in HAML templates and ruby files via the `Feature.enabled?(:my_shiny_new_feature_flag)` method ### More on feature flags -- [Deleting a feature flag](https://docs.gitlab.com/ee/api/features.html#delete-a-feature) -- [Manage feature flags](https://docs.gitlab.com/ee/development/feature_flags.html) -- [Feature flags API](https://docs.gitlab.com/ee/api/features.html) +- [Deleting a feature flag](../../api/features.md#delete-a-feature) +- [Manage feature flags](../feature_flags.md) +- [Feature flags API](../../api/features.md) diff --git a/doc/development/packages.md b/doc/development/packages.md index a3b891d7834..ab0c5f9904d 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -1,8 +1,8 @@ # Packages **[PREMIUM]** -This document will guide you through adding another [package management system](https://docs.gitlab.com/ee/administration/packages.html) support to GitLab. +This document will guide you through adding another [package management system](../administration/packages.md) support to GitLab. -See already supported package types in [Packages documentation](https://docs.gitlab.com/ee/administration/packages.html) +See already supported package types in [Packages documentation](../administration/packages.md) Since GitLab packages' UI is pretty generic, it is possible to add new package system support by solely backend changes. This guide is superficial and does @@ -46,7 +46,7 @@ Group-level and instance-level endpoints are good to have but are optional. NOTE: **Note:** To avoid name conflict for instance-level endpoints we use -[the package naming convention](https://docs.gitlab.com/ee/user/project/packages/npm_registry.html#package-naming-convention) +[the package naming convention](../user/project/packages/npm_registry.md#package-naming-convention) ## Configuration diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 27fc3231218..28a12572961 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -31,7 +31,7 @@ project. #### Seeding issues for Insights charts **[ULTIMATE]** You can seed issues specifically for working with the -[Insights charts](https://docs.gitlab.com/ee/user/group/insights/index.html) with the +[Insights charts](../user/group/insights/index.md) with the `gitlab:seed:insights:issues` task: ```shell @@ -108,11 +108,13 @@ To make sure that indices still fit. You could find great details in: In order to run the test you can use the following commands: -- `rake spec` to run the rspec suite -- `rake karma` to run the karma test suite -- `rake gitlab:test` to run all the tests +- `bin/rake spec` to run the rspec suite +- `bin/rake spec:unit` to run the only the unit tests +- `bin/rake spec:integration` to run the only the integration tests +- `bin/rake spec:system` to run the only the system tests +- `bin/rake karma` to run the karma test suite -Note: `rake spec` takes significant time to pass. +Note: `bin/rake spec` takes significant time to pass. Instead of running full test suite locally you can save a lot of time by running a single test or directory related to your changes. After you submit merge request CI will run full test suite for you. Green CI status in the merge request means @@ -121,6 +123,9 @@ full test suite is passed. Note: You can't run `rspec .` since this will try to run all the `_spec.rb` files it can find, also the ones in `/tmp` +Note: You can pass RSpec command line options to the `spec:unit`, +`spec:integration`, and `spec:system` tasks, e.g. `bin/rake "spec:unit[--tag ~geo --dry-run]"`. + To run a single test file you can use: - `bin/rspec spec/controllers/commit_controller_spec.rb` for a rspec test diff --git a/doc/development/rolling_out_changes_using_feature_flags.md b/doc/development/rolling_out_changes_using_feature_flags.md index 8e4e07c4319..84028b1b342 100644 --- a/doc/development/rolling_out_changes_using_feature_flags.md +++ b/doc/development/rolling_out_changes_using_feature_flags.md @@ -102,7 +102,7 @@ GitLab's feature library (using [Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature Flags](feature_flags.md) guide) supports rolling out changes to a percentage of users. This in turn can be controlled using [GitLab -chatops](https://docs.gitlab.com/ee/ci/chatops/). +chatops](../ci/chatops/README.md). For an up to date list of feature flag commands please see [the source code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). @@ -200,10 +200,9 @@ isn't gated by a License or Plan. ### Undefined feature flags default to "on" -An important side-effect of the [implicit feature -flags][#implicit-feature-flags] mentioned above is that unless the feature is -explicitly disabled or limited to a percentage of users, the feature flag check -will default to `true`. +An important side-effect of the [implicit feature flags](#implicit-feature-flags) +mentioned above is that unless the feature is explicitly disabled or limited to a +percentage of users, the feature flag check will default to `true`. As an example, if you were to ship the backend half of a feature behind a flag, you'd want to explicitly disable that flag until the frontend half is also ready diff --git a/doc/development/routing.md b/doc/development/routing.md new file mode 100644 index 00000000000..e9c0ad8d4e8 --- /dev/null +++ b/doc/development/routing.md @@ -0,0 +1,63 @@ +# Routing + +The GitLab backend is written primarily with Rails so it uses [Rails +routing](https://guides.rubyonrails.org/routing.html). Beside Rails best +practices, there are few rules unique to the GitLab application. To +support subgroups, GitLab project and group routes use the wildcard +character to match project and group routes. For example, we might have +a path such as: + + /gitlab-com/customer-success/north-america/west/customerA + +However, paths can be ambiguous. Consider the following example: + + /gitlab-com/edit + +It's ambiguous whether there is a subgroup named `edit` or whether +this is a special endpoint to edit the `gitlab-com` group. + +To eliminate the ambiguity and to make the backend easier to maintain, +we introduced the `/-/` scope. The purpose of it is to separate group or +project paths from the rest of the routes. Also it helps to reduce the +number of [reserved names](../user/reserved_names.md). + +## Global routes + +We have a number of global routes. For example: + + /-/health + /-/metrics + +## Group routes + +Every group route must be under the `/-/` scope. + +Examples: + + gitlab-org/-/edit + gitlab-org/-/activity + gitlab-org/-/security/dashboard + gitlab-org/serverless/-/activity + +To achieve that, use the `scope '-'` method. + +## Project routes + +Every project route must be under the `/-/` scope, except cases where a Git +client or other software requires something different. + +Examples: + + gitlab-org/gitlab-ce/-/activity + gitlab-org/gitlab-ce/-/jobs/123 + gitlab-org/gitlab-ce/-/settings/repository + gitlab-org/serverless/runtimes/-/settings/repository + +Currently, only some project routes are placed under the `/-/` scope. However, +you can help us migrate more of them! To migrate project routes: + +1. Modify existing routes by adding `-` scope. +1. Add redirects for legacy routes by using `Gitlab::Routing.redirect_legacy_paths`. +1. Create a technical debt issue to remove deprecated routes in later releases. + +To get started, see an [example merge request](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28435). diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 63ec9755462..82439c94c5a 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -2,19 +2,29 @@ ## Test Design -Testing at GitLab is a first class citizen, not an afterthought. It's important we consider the design of our tests -as we do the design of our features. +Testing at GitLab is a first class citizen, not an afterthought. It's important we consider the design of our tests +as we do the design of our features. -When implementing a feature, we think about developing the right capabilities the right way, which helps us -narrow our scope to a manageable level. When implementing tests for a feature, we must think about developing -the right tests, but then cover _all_ the important ways the test may fail, which can quickly widen our scope to +When implementing a feature, we think about developing the right capabilities the right way, which helps us +narrow our scope to a manageable level. When implementing tests for a feature, we must think about developing +the right tests, but then cover _all_ the important ways the test may fail, which can quickly widen our scope to a level that is difficult to manage. -Test heuristics can help solve this problem. They concisely address many of the common ways bugs -manifest themselves within our code. When designing our tests, take time to review known test heuristics to inform -our test design. We can find some helpful heuristics documented in the Handbook in the +Test heuristics can help solve this problem. They concisely address many of the common ways bugs +manifest themselves within our code. When designing our tests, take time to review known test heuristics to inform +our test design. We can find some helpful heuristics documented in the Handbook in the [Test Design](https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/test-design/) section. +## Run tests against MySQL + +By default, tests are only run againts PostgreSQL, but you can run them on +demand against MySQL by following one of the following conventions: + +| Convention | Valid example | +|:----------------------|:-----------------------------| +| Include `mysql` in your branch name | `enhance-mysql-support` | +| Include `[run mysql]` in your commit message | `Fix MySQL support<br><br>[run mysql]` | + ## Test speed GitLab has a massive test suite that, without [parallelization], can take hours @@ -184,11 +194,11 @@ instead of 30+ seconds in case of a regular `spec_helper`. ### `let` variables GitLab's RSpec suite has made extensive use of `let`(along with it strict, non-lazy -version `let!`) variables to reduce duplication. However, this sometimes [comes at the cost of clarity][lets-not], +version `let!`) variables to reduce duplication. However, this sometimes [comes at the cost of clarity][lets-not], so we need to set some guidelines for their use going forward: - `let!` variables are preferable to instance variables. `let` variables - are preferable to `let!` variables. Local variables are preferable to + are preferable to `let!` variables. Local variables are preferable to `let` variables. - Use `let` to reduce duplication throughout an entire spec file. - Don't use `let` to define variables used by a single test; define them as @@ -199,8 +209,8 @@ so we need to set some guidelines for their use going forward: - Try to avoid overriding the definition of one `let` variable with another. - Don't define a `let` variable that's only used by the definition of another. Use a helper method instead. -- `let!` variables should be used only in case if strict evaluation with defined - order is required, otherwise `let` will suffice. Remember that `let` is lazy and won't +- `let!` variables should be used only in case if strict evaluation with defined + order is required, otherwise `let` will suffice. Remember that `let` is lazy and won't be evaluated until it is referenced. [lets-not]: https://robots.thoughtbot.com/lets-not diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md new file mode 100644 index 00000000000..89500ef9a90 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -0,0 +1,38 @@ +# Best practices when writing end-to-end tests + +The majority of the end-to-end tests require some state to be built in the application for the tests to happen. + +A good example is a user being logged in as a pre-condition for testing the feature. + +But if the login feature is already covered with end-to-end tests through the GUI, there is no reason to perform such an expensive task to test the functionality of creating a project, or importing a repo, even if these features depend on a user being logged in. Let's see an example to make things clear. + +Let's say that, on average, the process to perform a successful login through the GUI takes 2 seconds. + +Now, realize that almost all tests need the user to be logged in, and that we need every test to run in isolation, meaning that tests cannot interfere with each other. This would mean that for every test the user needs to log in, and "waste 2 seconds". + +Now, multiply the number of tests per 2 seconds, and as your test suite grows, the time to run it grows with it, and this is not sustainable. + +An alternative to perform a login in a cheaper way would be having an endpoint (available only for testing) where we could pass the user's credentials as encrypted values as query strings, and then we would be redirected to the logged in home page if the credentials are valid. Let's say that, on average, this process takes only 200 miliseconds. + +You see the point right? + +Performing a login through the GUI for every test would cost a lot in terms of tests' execution. + +And there is another reason. + +Let's say that you don't follow the above suggestion, and depend on the GUI for the creation of every application state in order to test a specific feature. In this case we could be talking about the **Issues** feature, that depends on a project to exist, and the user to be logged in. + +What would happen if there was a bug in the project creation page, where the 'Create' button is disabled, not allowing for the creation of a project through the GUI, but the API logic is still working? + +In this case, instead of having only the project creation test failing, we would have many tests that depend on a project to be failing too. + +But, if we were following the best practices, only one test would be failing, and tests for other features that depend on a project to exist would continue to pass, since they could be creating the project behind the scenes interacting directly with the public APIs, ensuring a more reliable metric of test failure rate. + +Finally, interacting with the application only by its GUI generates a higher rate of test flakiness, and we want to avoid that at max. + +**The takeaways here are:** + +- Building state through the GUI is time consuming and it's not sustainable as the test suite grows. +- When depending only on the GUI to create the application's state and tests fail due to front-end issues, we can't rely on the test failures rate, and we generate a higher rate of test flakiness. + +Now that we are aware of all of it, [let's go create some tests](quick_start_guide.md). diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end/index.md index fc7aaedca29..afd81ff00b2 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -65,7 +65,7 @@ Below you can read more about how to use it and how does it work. Currently, we are using _multi-project pipeline_-like approach to run QA pipelines. - + <details> <summary>Show mermaid source</summary> @@ -136,6 +136,12 @@ Once you decided where to put [test environment orchestration scenarios] and the [GitLab QA orchestrator README][gitlab-qa-readme], and [the already existing instance-level scenarios][instance-level scenarios]. +Continued reading: + +- [Quick Start Guide](quick_start_guide.md) +- [Style Guide](style_guide.md) +- [Best Practices](best_practices.md) + ## Where can I ask for help? You can ask question in the `#quality` channel on Slack (GitLab internal) or @@ -149,7 +155,7 @@ you can find an issue you would like to work on in [gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md [quality-nightly-pipelines]: https://gitlab.com/gitlab-org/quality/nightly/pipelines [quality-staging-pipelines]: https://gitlab.com/gitlab-org/quality/staging/pipelines -[review-apps]: ./review_apps.md +[review-apps]: ../review_apps.md [gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md [gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario [gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name[]=QA&label_name[]=test diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md new file mode 100644 index 00000000000..d0de33892c4 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -0,0 +1,134 @@ +# Page objects in GitLab QA + +In GitLab QA we are using a known pattern, called _Page Objects_. + +This means that we have built an abstraction for all GitLab pages that we use +to drive GitLab QA scenarios. Whenever we do something on a page, like filling +in a form, or clicking a button, we do that only through a page object +associated with this area of GitLab. + +For example, when GitLab QA test harness signs in into GitLab, it needs to fill +in a user login and user password. In order to do that, we have a class, called +`Page::Main::Login` and `sign_in_using_credentials` methods, that is the only +piece of the code, that has knowledge about `user_login` and `user_password` +fields. + +## Why do we need that? + +We need page objects, because we need to reduce duplication and avoid problems +whenever someone changes some selectors in GitLab's source code. + +Imagine that we have a hundred specs in GitLab QA, and we need to sign into +GitLab each time, before we make assertions. Without a page object one would +need to rely on volatile helpers or invoke Capybara methods directly. Imagine +invoking `fill_in :user_login` in every `*_spec.rb` file / test example. + +When someone later changes `t.text_field :login` in the view associated with +this page to `t.text_field :username` it will generate a different field +identifier, what would effectively break all tests. + +Because we are using `Page::Main::Login.act { sign_in_using_credentials }` +everywhere, when we want to sign into GitLab, the page object is the single +source of truth, and we will need to update `fill_in :user_login` +to `fill_in :user_username` only in a one place. + +## What problems did we have in the past? + +We do not run QA tests for every commit, because of performance reasons, and +the time it would take to build packages and test everything. + +That is why when someone changes `t.text_field :login` to +`t.text_field :username` in the _new session_ view we won't know about this +change until our GitLab QA nightly pipeline fails, or until someone triggers +`package-and-qa` action in their merge request. + +Obviously such a change would break all tests. We call this problem a _fragile +tests problem_. + +In order to make GitLab QA more reliable and robust, we had to solve this +problem by introducing coupling between GitLab CE / EE views and GitLab QA. + +## How did we solve fragile tests problem? + +Currently, when you add a new `Page::Base` derived class, you will also need to +define all selectors that your page objects depends on. + +Whenever you push your code to CE / EE repository, `qa:selectors` sanity test +job is going to be run as a part of a CI pipeline. + +This test is going to validate all page objects that we have implemented in +`qa/page` directory. When it fails, you will be notified about missing +or invalid views / selectors definition. + +## How to properly implement a page object? + +We have built a DSL to define coupling between a page object and GitLab views +it is actually implemented by. See an example below. + +```ruby +module Page + module Main + class Login < Page::Base + view 'app/views/devise/passwords/edit.html.haml' do + element :password_field + element :password_confirmation + element :change_password_button + end + + view 'app/views/devise/sessions/_new_base.html.haml' do + element :login_field + element :password_field + element :sign_in_button + end + + # ... + end +end +``` + +The `view` DSL method declares the filename of the view where an +`element` is implemented. + +The `element` DSL method in turn declares an element for which a corresponding +`qa-element-name-dasherized` CSS class need to be added to the view file. + +You can also define a value (String or Regexp) to match to the actual view +code but **this is deprecated** in favor of the above method for two reasons: + +- Consistency: there is only one way to define an element +- Separation of concerns: QA uses dedicated CSS classes instead of reusing code + or classes used by other components (e.g. `js-*` classes etc.) + +```ruby +view 'app/views/my/view.html.haml' do + # Implicitly require `.qa-logout-button` CSS class to be present in the view + element :logout_button + + ## This is deprecated and forbidden by the `QA/ElementWithPattern` RuboCop cop. + # Require `f.submit "Sign in"` to be present in `my/view.html.haml + element :my_button, 'f.submit "Sign in"' # rubocop:disable QA/ElementWithPattern + + ## This is deprecated and forbidden by the `QA/ElementWithPattern` RuboCop cop. + # Match every line in `my/view.html.haml` against + # `/link_to .* "My Profile"/` regexp. + element :profile_link, /link_to .* "My Profile"/ # rubocop:disable QA/ElementWithPattern +end +``` + +## Running the test locally + +During development, you can run the `qa:selectors` test by running + +```shell +bin/qa Test::Sanity::Selectors +``` + +from within the `qa` directory. + +## Where to ask for help? + +If you need more information, ask for help on `#quality` channel on Slack +(internal, GitLab Team only). + +If you are not a Team Member, and you still need help to contribute, please +open an issue in GitLab CE issue tracker with the `~QA` label. diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md new file mode 100644 index 00000000000..afe76acf9c9 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/quick_start_guide.md @@ -0,0 +1,585 @@ +# Writing end-to-end tests step-by-step + +In this tutorial, you will find different examples, and the steps involved, in the creation of end-to-end (_e2e_) tests for GitLab CE and GitLab EE, using GitLab QA. + +> When referring to end-to-end tests in this document, this means testing a specific feature end-to-end, such as a user logging in, the creation of a project, the management of labels, breaking down epics into sub-epics and issues, etc. + +## Important information before we start writing tests + +It's important to understand that end-to-end tests of isolated features, such as the ones described in the above note, doesn't mean that everything needs to happen through the GUI. + +If you don't exactly understand what we mean by **not everything needs to happen through the GUI,** please make sure you've read the [best practices](best_practices.md) before moving on. + +## This document covers the following items: + +- [0.](#0-are-end-to-end-tests-needed) Identifying if end-to-end tests are really needed +- [1.](#1-identifying-the-devops-stage) Identifying the [DevOps stage](https://about.gitlab.com/stages-devops-lifecycle/) of the feature that you are going to cover with end-to-end tests +- [2.](#2-test-skeleton) Creating the skeleton of the test file (`*_spec.rb`) +- [3.](#3-test-cases-mvc) The [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of the test cases' logic +- [4.](#4-extracting-duplicated-code) Extracting duplicated code into methods +- [5.](#5-tests-pre-conditions-using-resources-and-page-objects) Tests' pre-conditions (`before :context` and `before`) using resources and [Page Objects] +- [6.](#6-optimization) Optimizing the test suite +- [7.](#7-resources) Using and implementing resources +- [8.](#8-page-objects) Moving element definitions and methods to [Page Objects] + +### 0. Are end-to-end tests needed? + +At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend you check the code coverage of a specific feature before writing end-to-end tests, for both [CE](https://gitlab-org.gitlab.io/gitlab-ce/coverage-ruby/#_AllFiles) and [EE](https://gitlab-org.gitlab.io/gitlab-ee/coverage-ruby/#_AllFiles) projects. + +Sometimes you may notice that there is already good coverage in other test levels, and we can stay confident that if we break a feature, we will still have quick feedback about it, even without having end-to-end tests. + +If after this analysis you still think that end-to-end tests are needed, keep reading. + +### 1. Identifying the DevOps stage + +The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features/browser_ui), and so, if you are creating tests for issue creation, for instance, you would locate the spec files under the `qa/qa/specs/features/browser_ui/2_plan/` directory since issue creation is part of the Plan stage. + + In another case of a test for listing merged merge requests (MRs), the test should go under the `qa/qa/specs/features/browser_ui/3_create/` directory since merge requests are a feature from the Create stage. + +> There may be sub-directories inside the stages directories, for different features. For example: `.../browser_ui/2_plan/ee_epics/` and `.../browser_ui/2_plan/issues/`. + +Now, let's say we want to create tests for the [scoped labels](https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#scoped-labels) feature, available on GitLab EE Premium (this feature is part of the Plan stage.) + +> Because these tests are for a feature available only on GitLab EE, we need to create them in the [EE repository](https://gitlab.com/gitlab-org/gitlab-ee). + +Since [there is no specific directory for this feature](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/qa/qa/specs/features/browser_ui/2_plan), we should create a sub-directory for it. + +Under `.../browser_ui/2_plan/`, let's create a sub-directory called `ee_scoped_labels/`. + +> Notice that since this feature is only available for GitLab EE we prefix the sub-directory with `ee_`. + +### 2. Test skeleton + +Inside the newly created sub-directory, let's create a file describing the test suite (e.g. `editing_scoped_labels_spec.rb`.) + +#### The `context` and `describe` blocks + +Specs have an outer `context` that indicates the DevOps stage. The next level is the `describe` block, that briefly states the subject of the test suite. See the following example: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + end + end +end +``` + +#### The `it` blocks + +Every test suite is composed of at least one `it` block, and a good way to start writing end-to-end tests is by writing test cases descriptions as `it` blocks. These might help you to think of different test scenarios. Take a look at the following example: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + it 'replaces an existing label if it has the same key' do + end + + it 'keeps both scoped labels when adding a label with a different key' do + end + end + end +end +``` + +### 3. Test cases MVC + +For the [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of our test cases, let's say that we already have the application in the state needed for the tests, and then let's focus on the logic of the test cases only. + +To evolve the test cases drafted on step 2, let's imagine that the user is already logged into a GitLab EE instance, they already have at least a Premium license in use, there is already a project created, there is already an issue opened in the project, the issue already has a scoped label (e.g. `animal::fox`), there are other scoped labels (for the same scope and for a different scope (e.g. `animal::dolphin` and `plant::orchid`), and finally, the user is already on the issue's page. Let's also suppose that for every test case the application is in a clean state, meaning that one test case won't affect another. + +> Note: there are different approaches to creating an application state for end-to-end tests. Some of them are very time consuming and subject to failures, such as when using the GUI for all the pre-conditions of the tests. On the other hand, other approaches are more efficient, such as using the public APIs. The latter is more efficient since it doesn't depend on the GUI. We won't focus on this part yet, but it's good to keep it in mind. + +Let's now focus on the first test case. + +```ruby +it 'replaces an existing label if it has the same key' do + # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects (which we cover on section 8). + page.find('.block.labels .edit-link').click + page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['animal::dolphin', :enter] + page.find('#content-body').click + page.refresh + + labels_block = page.find('.qa-labels-block') + + expect(labels_block).to have_content('animal::dolphin') + expect(labels_block).not_to have_content('animal::fox') + expect(page).to have_content('added animal::dolphin label and removed animal::fox') +end +``` + +> Notice that the test itself is simple. The most challenging part is the creation of the application state, which will be covered later. + +> The exemplified test case's MVC is not enough for the change to be merged, but it helps to build up the test logic. The reason is that we do not want to use locators directly in the tests, and tests **must** use [Page Objects] before they can be merged. This way we better separate the responsibilities, where the Page Objects encapsulate elements and methods that allow us to interact with pages, while the spec files describe the test cases in more business-related language. + +Below are the steps that the test covers: + +1. The test finds the 'Edit' link for the labels and clicks on it. +2. Then it fills in the 'Assign labels' input field with the value 'animal::dolphin' and press enters. +3. Then it clicks in the content body to apply the label and refreshes the page. +4. Finally, the expectations check that the previous scoped label was removed and that the new one was added. + +Let's now see how the second test case would look. + +```ruby +it 'keeps both scoped labels when adding a label with a different key' do + # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects (which we cover on section 8). + page.find('.block.labels .edit-link').click + page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['plant::orchid', :enter] + page.find('#content-body').click + page.refresh + + labels_block = page.find('.qa-labels-block') + + expect(labels_block).to have_content('animal::fox') + expect(labels_block).to have_content('plant::orchid') + expect(page).to have_content('added animal::fox') + expect(page).to have_content('added plant::orchid') +end +``` + +> Note that elements are always located using CSS selectors, and a good practice is to add test-specific selectors (this is called adding testability to the application and we will talk more about it later.) For example, the `labels_block` element uses the selector `.qa-labels-block`, which was added specifically for testing purposes. + +Below are the steps that the test covers: + +1. The test finds the 'Edit' link for the labels and clicks on it. +2. Then it fills in the 'Assign labels' input field with the value 'plant::orchid' and press enters. +3. Then it clicks in the content body to apply the label and refreshes the page. +4. Finally, the expectations check that both scoped labels are present. + +> Similar to the previous test, this one is also very straightforward, but there is some code duplication. Let's address it. + +### 4. Extracting duplicated code + +If we refactor the tests created on step 3 we could come up with something like this: + +```ruby +before do + ... + + @initial_label = 'animal::fox' + @new_label_same_scope = 'animal::dolphin' + @new_label_different_scope = 'plant::orchid' + + ... +end + +it 'replaces an existing label if it has the same key' do + select_label_and_refresh @new_label_same_scope + + labels_block = page.find('.qa-labels-block') + + expect(labels_block).to have_content(@new_label_same_scope) + expect(labels_block).not_to have_content(@initial_label) + expect(page).to have_content("added #{@new_label_same_scope}") + expect(page).to have_content("and removed #{@initial_label}") +end + +it 'keeps both scoped label when adding a label with a different key' do + select_label_and_refresh @new_label_different_scope + + labels_block = page.find('.qa-labels-block') + + expect(labels_blocks).to have_content(@new_label_different_scope) + expect(labels_blocks).to have_content(@initial_label) + expect(page).to have_content("added #{@new_label_different_scope}") + expect(page).to have_content("added #{@initial_label}") +end + +def select_label_and_refresh(label) + page.find('.block.labels .edit-link').click + page.find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] + page.find('#content-body').click + page.refresh +end +``` + +First, we remove the duplication of strings by defining the global variables `@initial_label`, `@new_label_same_scope` and `@new_label_different_scope` in the `before` block, and by using them in the expectations. + +Then, by creating a reusable `select_label_and_refresh` method, we remove the code duplication of this action, and later we can move this method to a Page Object class that will be created for easier maintenance purposes. + +> Notice that the reusable method is created at the bottom of the file. The reason for that is that reading the code should be similar to reading a newspaper, where high-level information is at the top, like the title and summary of the news, while low level, or more specific information, is at the bottom (this helps readability). + +### 5. Tests' pre-conditions using resources and Page Objects + +In this section, we will address the previously mentioned subject of creating the application state for the tests, using the `before :context` and `before` blocks, together with resources and Page Objects. + +#### `before :context` + +A pre-condition for the entire test suite is defined in the `before :context` block. + +> For our test suite, due to the need of the tests being completely independent of each other, we won't use the `before :context` block. The `before :context` block would make the tests dependent on each other because the first test changes the label of the issue, and the second one depends on the `'animal::fox'` label being set. + +> **Tip:** In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. + +#### `before` + +As the pre-conditions for our test suite, the things that needs to happen before each test starts are: + +- The user logging in; +- A premium license already being set; +- A project being created with an issue and labels already set; +- The issue page being opened with only one scoped label applied to the it. + +> When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-remote-grid-environment-variables), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). + +#### Implementation + +In the following code we will focus only on the test suite's pre-conditions: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + before do + Runtime::Browser.visit(:gitlab, Page::Main::Login) + Page::Main::Login.perform(&:sign_in_using_credentials) + + @initial_label = 'animal::fox' + @new_label_same_scope = 'animal::dolphin' + @new_label_different_scope = 'plant::orchid' + + issue = Resource::Issue.fabricate_via_api! do |issue| + issue.title = 'Issue to test the scoped labels' + issue.labels = @initial_label + end + + [@new_label_same_scope, @new_label_different_scope].each do |label| + Resource::Label.fabricate_via_api! do |l| + l.project = issue.project.id + l.title = label + end + end + + issue.visit! + end + + it 'replaces an existing label if it has the same key' do + ... + end + + it 'keeps both scoped labels when adding a label with a different key' do + ... + end + + def select_label_and_refresh(label) + ... + end + end + end +end +``` + +In the `before` block we create all the application state needed for the tests to run. We do that by using the `Runtime::Browser.visit` method to go to the login page, by performing a `sign_in_using_credentials` from the `Login` Page Object, by fabricating resources via APIs (`issue`, and `Resource::Label`), and by using the `issue.visit!` to visit the issue page. + +> A project is created in the background by creating the `issue` resource. + +> When creating the [Resources], notice that when calling the `fabricate_via_api` method, we pass some attribute:values, like `title`, and `labels` for the `issue` resource; and `project` and `title` for the `label` resource. + +> What's important to understand here is that by creating the application state mostly using the public APIs we save a lot of time in the test suite setup stage. + +> Soon we will cover the use of the already existing resources' methods and the creation of your own `fabricate_via_api` methods for resources where this is still not available, but first, let's optimize our implementation. + +### 6. Optimization + +As already mentioned in the [best practices](best_practices.md) document, end-to-end tests are very costly in terms of execution time, and it's our responsibility as software engineers to ensure that we optimize them as much as possible. + +> Note that end-to-end tests are slow to run and so they can have several actions and assertions in a single test, which helps us get feedback from the tests sooner. In comparison, unit tests are much faster to run and can exercise every little piece of the application in isolation, and so they usually have only one assertion per test. + +Some improvements that we could make in our test suite to optimize its time to run are: + +1. Having a single test case (an `it` block) that exercises both scenarios to avoid "wasting" time in the tests' pre-conditions, instead of having two different test cases. +2. Making the selection of labels more performant by allowing for the selection of more than one label in the same reusable method. + +Let's look at a suggestion that addresses the above points, one by one: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + before do + ... + end + + it 'correctly applies scoped labels depending on if they are from the same or a different scope' do + select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] + + labels_block = page.all('.qa-labels-block') + + expect(labels_block).to have_content(@new_label_same_scope) + expect(labels_block).to have_content(@new_label_different_scope) + expect(labels_block).not_to have_content(@initial_label) + expect(page).to have_content("added #{@initial_label}") + expect(page).to have_content("added #{@new_label_same_scope} #{@new_label_different_scope} labels and removed #{@initial_label}") + end + + def select_labels_and_refresh(labels) + find('.block.labels .edit-link').click + labels.each do |label| + find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] + end + find('#content-body').click + refresh + end + end + end + end +``` + +To address point 1, we changed the test implementation from two `it` blocks into a single one that exercises both scenarios. Now the new test description is: `'correctly applies the scoped labels depending if they are from the same or a different scope'`. It's a long description, but it describes well what the test does. + +> Notice that the implementation of the new and unique `it` block had to change a little bit. Below we describe in details what it does. + +1. It selects two scoped labels simultaneously, one from the same scope of the one already applied in the issue during the setup phase (in the `before` block), and another one from a different scope. +2. It asserts that the correct labels are visible in the `labels_block`, and that the labels were correctly added and removed; +3. Finally, the `select_label_and_refresh` method is changed to `select_labels_and_refresh`, which accepts an array of labels instead of a single label, and it iterates on them for faster label selection (this is what is used in step 1 explained above.) + +### 7. Resources + +**Note:** When writing this document, some code that is now merged to master was not implemented yet, but we left them here for the readers to understand the whole process of end-to-end test creation. + +You can think of [Resources] as anything that can be created on GitLab CE or EE, either through the GUI, the API, or the CLI. + +With that in mind, resources can be a project, an epic, an issue, a label, a commit, etc. + +As you saw in the tests' pre-conditions and the optimization sections, we're already creating some of these resources, and we are doing that by calling the `fabricate_via_api!` method. + +> We could be using the `fabricate!` method instead, which would use the `fabricate_via_api!` method if it exists, and fallback to GUI fabrication otherwise, but we recommend being explicit to make it clear what the test does. Also, we always recommend fabricating resources via API since this makes tests faster and more reliable. + +For our test suite example, the resources that we need to create don't have the necessary code for the `fabricate_via_api!` method to correctly work (e.g., the issue and label resources), so we will have to create them. + +#### Implementation + +In the following we describe the changes needed in each of the resource files mentioned above. + +**Issue resource** + +Now, let's make it possible to create an issue resource through the API. + +First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its labels attribute. + +Add the following `attribute :labels` right below the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). + +> This line is needed to allow for labels to be automatically added to an issue when fabricating it via API. + +Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. + +```ruby +def api_get_path + "/projects/#{project.id}/issues/#{id}" +end + +def api_post_path + "/projects/#{project.id}/issues" +end + +def api_post_body + { + title: title, + labels: [labels] + } +end +``` + +By defining the `api_get_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single issue. + +> This `GET` path can be found in the [public API documentation](https://docs.gitlab.com/ee/api/issues.html#single-issue). + +By defining the `api_post_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new issue in a specific project. + +> This `POST` path can be found in the [public API documentation](https://docs.gitlab.com/ee/api/issues.html#new-issue). + +By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. + +> Notice that we pass both `title` and `labels` attributes in the `api_post_body`, where `labels` receives an array of labels, and [`title` is required](https://docs.gitlab.com/ee/api/issues.html#new-issue). + +**Label resource** + +Finally, let's make it possible to create label resources through the API. + +Add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/label.rb#L36) method. + +```ruby +def resource_web_url(resource) + super +rescue ResourceURLMissingError + # this particular resource does not expose a web_url property +end + +def api_get_path + raise NotImplementedError, "The Labels API doesn't expose a single-resource endpoint so this method cannot be properly implemented." +end + +def api_post_path + "/projects/#{project}/labels" +end + +def api_post_body + { + name: @title, + color: @color + } +end +``` + +By defining the `resource_web_url(resource)` method, we override the one from the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb#L44) module. We do that to avoid failing the test due to this particular resource not exposing a `web_url` property. + +By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the publich API, we raise a `NotImplementedError` instead. + +By defining the `api_post_path` method, we allow for the [`ApiFabricator `](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. + +By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. + +> Notice that we pass both `name` and `color` attributes in the `api_post_body` since [those are required](https://docs.gitlab.com/ee/api/labels.html#create-a-new-label). + +### 8. Page Objects + +Page Objects are used in end-to-end tests for maintenance reasons, where a page's elements and methods are defined to be reused in any test. + +> Page Objects are auto-loaded in the `qa/qa.rb` file and available in all the test files (`*_spec.rb`). + +Take a look at the [Page Objects] documentation. + +Now, let's go back to our example. + +As you may have noticed, we are defining elements with CSS selectors and the `select_labels_and_refresh` method directly in the test file, and this is an anti-pattern since we need to better separate the responsibilities. + +To address this issue, we will move the implementation to Page Objects, and the test suite will only focus on the business rules that we are testing. + +#### Updates in the test file + +As in a test-driven development approach, let's start changing the test file even before the Page Object implementation is in place. + +Replace the code of the `it` block in the test file by the following: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + before do + ... + end + + it 'correctly applies scoped labels depending on if they are from the same or a different scope' do + Page::Project::Issue::Show.perform do |issue_page| + issue_page.select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] + + expect(page).to have_content("added #{@initial_label}") + expect(page).to have_content("added #{@new_label_same_scope} #{@new_label_different_scope} labels and removed #{@initial_label}") + expect(issue_page.text_of_labels_block).to have_content(@new_label_same_scope) + expect(issue_page.text_of_labels_block).to have_content(@new_label_different_scope) + expect(issue_page.text_of_labels_block).not_to have_content(@initial_label) + end + end + end + end +end +``` + +Notice that `select_labels_and_refresh` is now a method from the issue Page Object (which is not yet implemented), and that we verify the labels' text by using `text_of_labels_block`, instead of via the `labels_block` element. The `text_of_labels_block` method will also be implemented in the issue Page Object. + +Let's now update the Issue Page Object. + +#### Updates in the Issue Page Object + +> Page Objects are located in the `qa/qa/page/` directory, and its sub-directories. + +The file we will have to change is the [Issue Page Object](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/page/project/issue/show.rb). + +First, add the following code right below the definition of an already implemented view: + +```ruby +view 'app/views/shared/issuable/_sidebar.html.haml' do + element :labels_block + element :edit_link_labels + element :dropdown_menu_labels +end + +view 'app/helpers/dropdowns_helper.rb' do + element :dropdown_input_field +end +``` + +Similarly to what we did before, let's first change the Page Object even without the elements being defined in the view (`_sidebar.html.haml`) and the `dropdowns_helper.rb` files, and later we will update them by adding the appropriate CSS selectors. + +Now, let's implement the methods `select_labels_and_refresh` and `text_of_labels_block`. + +Somewhere between the definition of the views and the private methods, add the following snippet of code: + +```ruby +def select_labels_and_refresh(labels) + click_element(:edit_link_labels) + labels.each do |label| + within_element(:dropdown_menu_labels, text: label) do + send_keys_to_element(:dropdown_input_field, [label, :enter]) + end + end + click_body + labels.each do |label| + has_element?(:labels_block, text: label) + end + refresh +end + +def text_of_labels_block + find_element(:labels_block) +end +``` + +##### Details of `select_labels_and_refresh` + +Notice that we have not only moved the `select_labels_and_refresh` method, but we have also changed its implementation to: +1. Click the `:edit_link_labels` element previously defined, instead of using `find('.block.labels .edit-link').click` +2. Use `within_element(:dropdown_menu_labels, text: label)`, and inside of it, we call `send_keys_to_element(:dropdown_input_field, [label, :enter])`, which is a method that we will implement in the `QA::Page::Base` class to replace `find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter]` +3. Use `click_body` after iterating on each label, instead of using `find('#content-body').click` +4. Iterate on every label again, and then we use `has_element?(:labels_block, text: label)` after clicking the page body (which applies the labels), and before refreshing the page, to avoid test flakiness due to refreshing too fast. + +##### Details of `text_of_labels_block` + +The `text_of_labels_block` method is a simple method that returns the `:labels_block` element (`find_element(:labels_block)`). + +#### Updates in the view (*.html.haml) and `dropdowns_helper.rb` files + +Now let's change the view and the `dropdowns_helper` files to add the selectors that relate to the Page Object. + +In the [app/views/shared/issuable/_sidebar.html.haml](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/views/shared/issuable/_sidebar.html.haml) file, on [line 105 ](https://gitlab.com/gitlab-org/gitlab-ee/blob/84043fa72ca7f83ae9cde48ad670e6d5d16501a3/app/views/shared/issuable/_sidebar.html.haml#L105), add an extra class `qa-edit-link-labels`. + +The code should look like this: `= link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link qa-edit-link-labels float-right'`. + +In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab-ee/blob/84043fa72ca7f83ae9cde48ad670e6d5d16501a3/app/views/shared/issuable/_sidebar.html.haml#L121), add an extra class `.qa-dropdown-menu-labels`. + +The code should look like this: `.dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.qa-dropdown-menu-labels.dropdown-menu-selectable`. + +In the [`dropdowns_helper.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/helpers/dropdowns_helper.rb) file, on [line 94](https://gitlab.com/gitlab-org/gitlab-ee/blob/99e51a374f2c20bee0989cac802e4b5621f72714/app/helpers/dropdowns_helper.rb#L94), add an extra class `qa-dropdown-input-field`. + +The code should look like this: `filter_output = search_field_tag search_id, nil, class: "dropdown-input-field qa-dropdown-input-field", placeholder: placeholder, autocomplete: 'off'`. + +> Classes starting with `qa-` are used for testing purposes only, and by defining such classes in the elements we add **testability** in the application. + +> When defining a class like `qa-labels-block`, it is transformed into `:labels_block` for usage in the Page Objects. So, `qa-edit-link-labels` is tranformed into `:edit_link_labels`, `qa-dropdown-menu-labels` is transformed into `:dropdown_menu_labels`, and `qa-dropdown-input-field` is transformed into `:dropdown_input_field`. Also, we use a [sanity test](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective `qa-` selectors in the specified views. + +> We did not define the `qa-labels-block` class in the `app/views/shared/issuable/_sidebar.html.haml` file because it was already there to be used. + +#### Updates in the `QA::Page::Base` class + +The last thing that we have to do is to update `QA::Page::Base` class to add the `send_keys_to_element` method on it. + +Add the following snippet of code somewhere where class methods are defined: + +```ruby +def send_keys_to_element(name, keys) + find_element(name).send_keys(keys) +end +``` + +This method receives an element (`name`) and the `keys` that it will send to that element, and the keys are an array that can receive strings, or "special" keys, like `:enter`. + +As you might remember, in the Issue Page Object we call this method like this: `send_keys_to_element(:dropdown_input_field, [label, :enter])`. + +___ + +With that, you should be able to start writing end-to-end tests yourself. *Congratulations!* + +[Page Objects]: page_objects.md +[Resources]: resources.md diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md new file mode 100644 index 00000000000..1e32db4f633 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -0,0 +1,392 @@ +# Resource class in GitLab QA + +Resources are primarily created using Browser UI steps, but can also +be created via the API or the CLI. + +## How to properly implement a resource class? + +All resource classes should inherit from `Resource::Base`. + +There is only one mandatory method to implement to define a resource class. +This is the `#fabricate!` method, which is used to build the resource via the +browser UI. Note that you should only use [Page objects](page_objects.md) to +interact with a Web page in this method. + +Here is an imaginary example: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + def fabricate! + Page::Dashboard::Index.perform do |dashboard_index| + dashboard_index.go_to_new_shirt + end + + Page::Shirt::New.perform do |shirt_new| + shirt_new.set_name(name) + shirt_new.create_shirt! + end + end + end + end +end +``` + +### Define API implementation + +A resource class may also implement the three following methods to be able to +create the resource via the public GitLab API: + +- `#api_get_path`: The `GET` path to fetch an existing resource. +- `#api_post_path`: The `POST` path to create a new resource. +- `#api_post_body`: The `POST` body (as a Ruby hash) to create a new resource. + +Let's take the `Shirt` resource class, and add these three API methods: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + def fabricate! + # ... same as before + end + + def api_get_path + "/shirt/#{name}" + end + + def api_post_path + "/shirts" + end + + def api_post_body + { + name: name + } + end + end + end +end +``` + +The `Project` resource is a good real example of Browser +UI and API implementations. + +#### Resource attributes + +A resource may need another resource to exist first. For instance, a project +needs a group to be created in. + +To define a resource attribute, you can use the `attribute` method with a +block using the other resource class to fabricate the resource. + +That will allow access to the other resource from your resource object's +methods. You would usually use it in `#fabricate!`, `#api_get_path`, +`#api_post_path`, `#api_post_body`. + +Let's take the `Shirt` resource class, and add a `project` attribute to it: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + attribute :project do + Project.fabricate! do |resource| + resource.name = 'project-to-create-a-shirt' + end + end + + def fabricate! + project.visit! + + Page::Project::Show.perform do |project_show| + project_show.go_to_new_shirt + end + + Page::Shirt::New.perform do |shirt_new| + shirt_new.set_name(name) + shirt_new.create_shirt! + end + end + + def api_get_path + "/project/#{project.path}/shirt/#{name}" + end + + def api_post_path + "/project/#{project.path}/shirts" + end + + def api_post_body + { + name: name + } + end + end + end +end +``` + +**Note that all the attributes are lazily constructed. This means if you want +a specific attribute to be fabricated first, you'll need to call the +attribute method first even if you're not using it.** + +#### Product data attributes + +Once created, you may want to populate a resource with attributes that can be +found in the Web page, or in the API response. +For instance, once you create a project, you may want to store its repository +SSH URL as an attribute. + +Again we could use the `attribute` method with a block, using a page object +to retrieve the data on the page. + +Let's take the `Shirt` resource class, and define a `:brand` attribute: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + attribute :project do + Project.fabricate! do |resource| + resource.name = 'project-to-create-a-shirt' + end + end + + # Attribute populated from the Browser UI (using the block) + attribute :brand do + Page::Shirt::Show.perform do |shirt_show| + shirt_show.fetch_brand_from_page + end + end + + # ... same as before + end + end +end +``` + +**Note again that all the attributes are lazily constructed. This means if +you call `shirt.brand` after moving to the other page, it'll not properly +retrieve the data because we're no longer on the expected page.** + +Consider this: + +```ruby +shirt = + QA::Resource::Shirt.fabricate! do |resource| + resource.name = "GitLab QA" + end + +shirt.project.visit! + +shirt.brand # => FAIL! +``` + +The above example will fail because now we're on the project page, trying to +construct the brand data from the shirt page, however we moved to the project +page already. There are two ways to solve this, one is that we could try to +retrieve the brand before visiting the project again: + +```ruby +shirt = + QA::Resource::Shirt.fabricate! do |resource| + resource.name = "GitLab QA" + end + +shirt.brand # => OK! + +shirt.project.visit! + +shirt.brand # => OK! +``` + +The attribute will be stored in the instance therefore all the following calls +will be fine, using the data previously constructed. If we think that this +might be too brittle, we could eagerly construct the data right before +ending fabrication: + +```ruby +module QA + module Resource + class Shirt < Base + # ... same as before + + def fabricate! + project.visit! + + Page::Project::Show.perform do |project_show| + project_show.go_to_new_shirt + end + + Page::Shirt::New.perform do |shirt_new| + shirt_new.set_name(name) + shirt_new.create_shirt! + end + + populate(:brand) # Eagerly construct the data + end + end + end +end +``` + +The `populate` method will iterate through its arguments and call each +attribute respectively. Here `populate(:brand)` has the same effect as +just `brand`. Using the populate method makes the intention clearer. + +With this, it will make sure we construct the data right after we create the +shirt. The drawback is that this will always construct the data when the +resource is fabricated even if we don't need to use the data. + +Alternatively, we could just make sure we're on the right page before +constructing the brand data: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + attribute :project do + Project.fabricate! do |resource| + resource.name = 'project-to-create-a-shirt' + end + end + + # Attribute populated from the Browser UI (using the block) + attribute :brand do + back_url = current_url + visit! + + Page::Shirt::Show.perform do |shirt_show| + shirt_show.fetch_brand_from_page + end + + visit(back_url) + end + + # ... same as before + end + end +end +``` + +This will make sure it's on the shirt page before constructing brand, and +move back to the previous page to avoid breaking the state. + +#### Define an attribute based on an API response + +Sometimes, you want to define a resource attribute based on the API response +from its `GET` or `POST` request. For instance, if the creation of a shirt via +the API returns + +```ruby +{ + brand: 'a-brand-new-brand', + style: 't-shirt', + materials: [[:cotton, 80], [:polyamide, 20]] +} +``` + +you may want to store `style` as-is in the resource, and fetch the first value +of the first `materials` item in a `main_fabric` attribute. + +Let's take the `Shirt` resource class, and define a `:style` and a +`:main_fabric` attributes: + +```ruby +module QA + module Resource + class Shirt < Base + # ... same as before + + # @style from the instance if present, + # or fetched from the API response if present, + # or a QA::Resource::Base::NoValueError is raised otherwise + attribute :style + + # If @main_fabric is not present, + # and if the API does not contain this field, this block will be + # used to construct the value based on the API response, and + # store the result in @main_fabric + attribute :main_fabric do + api_response.&dig(:materials, 0, 0) + end + + # ... same as before + end + end +end +``` + +**Notes on attributes precedence:** + +- resource instance variables have the highest precedence +- attributes from the API response take precedence over attributes from the + block (usually from Browser UI) +- attributes without a value will raise a `QA::Resource::Base::NoValueError` error + +## Creating resources in your tests + +To create a resource in your tests, you can call the `.fabricate!` method on +the resource class. +Note that if the resource class supports API fabrication, this will use this +fabrication by default. + +Here is an example that will use the API fabrication method under the hood +since it's supported by the `Shirt` resource class: + +```ruby +my_shirt = Resource::Shirt.fabricate! do |shirt| + shirt.name = 'my-shirt' +end + +expect(page).to have_text(my_shirt.name) # => "my-shirt" from the resource's instance variable +expect(page).to have_text(my_shirt.brand) # => "a-brand-new-brand" from the API response +expect(page).to have_text(my_shirt.style) # => "t-shirt" from the API response +expect(page).to have_text(my_shirt.main_fabric) # => "cotton" from the API response via the block +``` + +If you explicitly want to use the Browser UI fabrication method, you can call +the `.fabricate_via_browser_ui!` method instead: + +```ruby +my_shirt = Resource::Shirt.fabricate_via_browser_ui! do |shirt| + shirt.name = 'my-shirt' +end + +expect(page).to have_text(my_shirt.name) # => "my-shirt" from the resource's instance variable +expect(page).to have_text(my_shirt.brand) # => the brand name fetched from the `Page::Shirt::Show` page via the block +expect(page).to have_text(my_shirt.style) # => QA::Resource::Base::NoValueError will be raised because no API response nor a block is provided +expect(page).to have_text(my_shirt.main_fabric) # => QA::Resource::Base::NoValueError will be raised because no API response and the block didn't provide a value (because it's also based on the API response) +``` + +You can also explicitly use the API fabrication method, by calling the +`.fabricate_via_api!` method: + +```ruby +my_shirt = Resource::Shirt.fabricate_via_api! do |shirt| + shirt.name = 'my-shirt' +end +``` + +In this case, the result will be similar to calling +`Resource::Shirt.fabricate!`. + +## Where to ask for help? + +If you need more information, ask for help on `#quality` channel on Slack +(internal, GitLab Team only). + +If you are not a Team Member, and you still need help to contribute, please +open an issue in GitLab CE issue tracker with the `~QA` label. diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md new file mode 100644 index 00000000000..0272e1810f2 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -0,0 +1,99 @@ +# Style guide for writing end-to-end tests + +This document describes the conventions used at GitLab for writing End-to-end (E2E) tests using the GitLab QA project. + +## `click_` versus `go_to_` + +### When to use `click_`? + +When clicking in a single link to navigate, use `click_`. + +E.g.: + +```ruby +def click_ci_cd_pipelines + within_sidebar do + click_element :link_pipelines + end +end +``` + +From a testing perspective, if we want to check that clicking a link, or a button (a single interaction) is working as intended, we would want the test to read as: + +- Click a certain element +- Verify the action took place + +### When to use `go_to_`? + +When interacting with multiple elements to go to a page, use `go_to_`. + +E.g.: + +```ruby +def go_to_operations_environments + hover_operations do + within_submenu do + click_element(:operations_environments_link) + end + end +end +``` + +`go_to_` fits the definition of interacting with multiple elements very well given it's more of a meta-navigation action that includes multiple interactions. + +Notice that in the above example, before clicking the `:operations_environments_link`, another element is hovered over. + +> We can create these methods as helpers to abstract multi-step navigation. + +### Element naming convention + +When adding new elements to a page, it's important that we have a uniform element naming convention. + +We follow a simple formula roughly based on hungarian notation. + +*Formula*: `element :<descriptor>_<type>` + +- `descriptor`: The natural-language description of what the element is. On the login page, this could be `username`, or `password`. +- `type`: A physical control on the page that can be seen by a user. + - `_button` + - `_link` + - `_tab` + - `_dropdown` + - `_field` + - `_checkbox` + - `_radio` + - `_content` + +*Note: This list is a work in progress. This list will eventually be the end-all enumeration of all available types. + I.e., any element that does not end with something in this list is bad form.* + +#### Examples + +**Good** + +```ruby +view '...' do + element :edit_button + element :notes_tab + element :squash_checkbox + element :username_field + element :issue_title_content +end +``` + +**Bad** + +```ruby +view '...' do + # `_confirmation` should be `_field`. what sort of confirmation? a checkbox confirmation? no real way to disambiguate. + # an appropriate replacement would be `element :password_confirmation_field` + element :password_confirmation + + # `clone_options` is too vague. If it's a dropdown menu, it should be `clone_dropdown`. + # If it's a checkbox, it should be `clone_checkbox` + element :clone_options + + # how is this url being displayed? is it a textbox? a simple span? + element :ssh_clone_url +end +``` diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 58d6f08954d..4c9d1684c00 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -67,9 +67,6 @@ Remember that the performance of each test depends on the environment. GitLab uses the [Karma][karma] test runner with [Jasmine] as its test framework for our JavaScript unit and integration tests. -We generate HTML and JSON fixtures from backend views and controllers -using RSpec (see `spec/javascripts/fixtures/*.rb` for examples). -Fixtures are served during testing by the [jasmine-jquery][jasmine-jquery] plugin. JavaScript tests live in `spec/javascripts/`, matching the folder structure of `app/assets/javascripts/`: `app/assets/javascripts/behaviors/autosize.js` @@ -421,7 +418,7 @@ See this [section][vue-test]. For running the frontend tests, you need the following commands: -- `rake karma:fixtures` (re-)generates fixtures. +- `rake karma:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). - `yarn test` executes the tests. As long as the fixtures don't change, `yarn test` is sufficient (and saves you some time). @@ -469,6 +466,48 @@ yarn karma -f 'spec/javascripts/ide/**/file_spec.js' Information on setting up and running RSpec integration tests with [Capybara] can be found in the [Testing Best Practices](best_practices.md). +## Frontend test fixtures + +Code that is added to HAML templates (in `app/views/`) or makes Ajax requests to the backend has tests that require HTML or JSON from the backend. +Fixtures for these tests are located at: + +- `spec/javascripts/fixtures/`, for running tests in CE. +- `ee/spec/javascripts/fixtures/`, for running tests in EE. + +Fixture files in: + +- The Karma test suite are served by [jasmine-jquery](https://github.com/velesin/jasmine-jquery). +- Jest use `spec/frontend/helpers/fixtures.js`. + +The following are examples of tests that work for both Karma and Jest: + +```javascript +it('makes a request', () => { + const responseBody = getJSONFixture('some/fixture.json'); // loads spec/javascripts/fixtures/some/fixture.json + axiosMock.onGet(endpoint).reply(200, responseBody); + + myButton.click(); + + // ... +}); + +it('uses some HTML element', () => { + loadFixtures('some/page.html'); // loads spec/javascripts/fixtures/some/page.html and adds it to the DOM + + const element = document.getElementById('#my-id'); + + // ... +}); +``` + +HTML and JSON fixtures are generated from backend views and controllers using RSpec (see `spec/javascripts/fixtures/*.rb`). + +For each fixture, the content of the `response` variable is stored in the output file. +This variable gets automagically set if the test is marked as `type: :request` or `type: :controller`. +Fixtures are regenerated using the `bin/rake karma:fixtures` command but you can also generate them individually, +for example `bin/rspec spec/javascripts/fixtures/merge_requests.rb`. +When creating a new fixture, it often makes sense to take a look at the corresponding tests for the endpoint in `(ee/)spec/controllers/` or `(ee/)spec/requests/`. + ## Gotchas ### RSpec errors due to JavaScript @@ -501,7 +540,6 @@ end ``` [jasmine-focus]: https://jasmine.github.io/2.5/focused_specs.html -[jasmine-jquery]: https://github.com/velesin/jasmine-jquery [karma]: http://karma-runner.github.io/ [vue-test]: https://docs.gitlab.com/ce/development/fe_guide/vue.html#testing-vue-components [rspec]: https://github.com/rspec/rspec-rails#feature-specs diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index ecad9ba48a3..93ee2a6371a 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -71,7 +71,7 @@ Everything you should know about how to test Rake tasks. --- -## [End-to-end tests](end_to_end_tests.md) +## [End-to-end tests](end_to_end/index.md) Everything you should know about how to run end-to-end tests using [GitLab QA][gitlab-qa] testing framework. diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index f34793c11f4..b5fde92a23d 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -113,7 +113,7 @@ You can also manually start the `review-qa-all`: it runs the full QA suite. On every [pipeline][gitlab-pipeline] in the `qa` stage, the `review-performance` job is automatically started: this job does basic browser performance testing using a -[Sitespeed.io Container](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). +[Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md). ## How to: @@ -203,7 +203,7 @@ find a way to limit it to only us.** [automated_cleanup.rb]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/review_apps/automated_cleanup.rb [Auto-DevOps.gitlab-ci.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml [gitlab-ci-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml -[gitlab-k8s-integration]: https://docs.gitlab.com/ee/user/project/clusters/index.html +[gitlab-k8s-integration]: ../../user/project/clusters/index.md [password-bug]: https://gitlab.com/gitlab-org/gitlab-ce/issues/53621 --- diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md index 30d861d7d68..c9d3238fbe9 100644 --- a/doc/development/testing_guide/smoke.md +++ b/doc/development/testing_guide/smoke.md @@ -17,7 +17,7 @@ Currently, our suite consists of this basic functionality coverage: Smoke tests have the `:smoke` RSpec metadata. -See [End-to-end Testing](./end_to_end_tests.md) for more details about +See [End-to-end Testing](end_to_end/index.md) for more details about end-to-end tests. --- diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 1fa6e38ea5a..e1ce4d3b7d1 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -4,12 +4,14 @@ _This diagram demonstrates the relative priority of each test type we use. `e2e` stands for end-to-end._ -As of 2019-04-16, we have the following distribution of tests per level: +As of 2019-05-01, we have the following distribution of tests per level: -- 67 black-box tests at the system level (aka end-to-end or QA tests) in CE, 98 in EE. This represents 0.3% of all the CE tests (0.3% in EE). -- 5,457 white-box tests at the system level (aka system or feature tests) in CE, 6,585 in EE. This represents 24.6% of all the CE tests (20.3% in EE). -- 8,298 integration tests in CE, 10,633 in EE: 0.3% of all the CE tests (0.3% in EE). This represents 37.2% of all the CE tests (32.8% in EE). -- 8,403 unit tests in CE, 15,090 in EE: 0.3% of all the CE tests (0.3% in EE). This represents 37.8% of all the CE tests (46.6% in EE). +| Test level | Community Edition | Enterprise Edition | Community + Enterprise Edition | +| --------- | ---------- | -------------- | ----- | +| Black-box tests at the system level (aka end-to-end or QA tests) | 68 (0.14%) | 31 (0.2%) | 99 (0.17%) | +| White-box tests at the system level (aka system or feature tests) | 5,471 (11.9%) | 969 (7.4%) | 6440 (10.9%) | +| Integration tests | 8,333 (18.2%) | 2,244 (17.2%) | 10,577 (17.9%) | +| Unit tests | 32,031 (69.7%) | 9,778 (75.1%) | 41,809 (71%) | ## Unit tests @@ -176,7 +178,7 @@ Every new feature should come with a [test plan]. | ---------- | -------------- | ----- | | `qa/qa/specs/features/` | [Capybara] + [RSpec] + Custom QA framework | Tests should be placed under their corresponding [Product category] | -> See [end-to-end tests](end_to_end_tests.md) for more information. +> See [end-to-end tests](end_to_end/index.md) for more information. Note that `qa/spec` contains unit tests of the QA framework itself, not to be confused with the application's [unit tests](#unit-tests) or diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index 01a0044f096..2ef8b3148e4 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -80,8 +80,9 @@ Planning time: 2.861 ms Execution time: 3428.596 ms ``` -For more information, refer to the official [EXPLAIN -documentation](https://www.postgresql.org/docs/current/static/sql-explain.html). +For more information, refer to the official +[`EXPLAIN` documentation](https://www.postgresql.org/docs/current/sql-explain.html) +and [using `EXPLAIN` guide](https://www.postgresql.org/docs/current/using-explain.html). ## Nodes @@ -653,6 +654,35 @@ and related tools such as: - <https://explain.depesz.com/> - <http://tatiyants.com/postgres-query-plan-visualization/> +## Producing query plans + +There are a few ways to get the output of a query plan. Of course you +can directly run the `EXPLAIN` query in the `psql` console, or you can +follow one of the other options below. + +### Rails console + +Using the [`activerecord-explain-analyze`](https://github.com/6/activerecord-explain-analyze) +you can directly generate the query plan from the Rails console: + +```ruby +pry(main)> require 'activerecord-explain-analyze' +=> true +pry(main)> Project.where('build_timeout > ?', 3600).explain(analyze: true) + Project Load (1.9ms) SELECT "projects".* FROM "projects" WHERE (build_timeout > 3600) + ↳ (pry):12 +=> EXPLAIN for: SELECT "projects".* FROM "projects" WHERE (build_timeout > 3600) +Seq Scan on public.projects (cost=0.00..2.17 rows=1 width=742) (actual time=0.040..0.041 rows=0 loops=1) + Output: id, name, path, description, created_at, updated_at, creator_id, namespace_id, ... + Filter: (projects.build_timeout > 3600) + Rows Removed by Filter: 14 + Buffers: shared hit=2 +Planning time: 0.411 ms +Execution time: 0.113 ms +``` + +### Chatops + GitLab employees can also use our chatops solution, available in Slack using the `/chatops` slash command. You can use chatops to get a query plan by running the following: @@ -674,3 +704,9 @@ For more information about the available options, run: ``` /chatops run explain --help ``` + +## Further reading + +A more extensive guide on understanding query plans can be found in +the [presentation](https://www.dalibo.org/_media/understanding_explain.pdf) +from [Dalibo.org](https://www.dalibo.org/en/). |