diff options
Diffstat (limited to 'doc/development')
22 files changed, 273 insertions, 40 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index 3096d9f25f0..1448a4c0414 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -44,6 +44,7 @@ - [Building a package for testing purposes](build_test_package.md) - [Manage feature flags](feature_flags.md) - [View sent emails or preview mailers](emails.md) +- [Working with Gitaly](gitaly.md) ## Databases diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index f83a60e49e8..5452b0e7a2f 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -215,14 +215,29 @@ same time will ensure that both existing and new data is migrated. In the next release we can remove the `after_commit` hooks and related code. We will also need to add a post-deployment migration that consumes any remaining -jobs. Such a migration would look like this: +jobs and manually run on any un-migrated rows. Such a migration would look like +this: ```ruby class ConsumeRemainingExtractServicesUrlJobs < ActiveRecord::Migration disable_ddl_transaction! + class Service < ActiveRecord::Base + include ::EachBatch + + self.table_name = 'services' + end + def up + # This must be included Gitlab::BackgroundMigration.steal('ExtractServicesUrl') + + # This should be included, but can be skipped - see below + Service.where(url: nil).each_batch(of: 50) do |batch| + range = batch.pluck('MIN(id)', 'MAX(id)').first + + Gitlab::BackgroundMigration::ExtractServicesUrl.new.perform(*range) + end end def down @@ -230,6 +245,15 @@ class ConsumeRemainingExtractServicesUrlJobs < ActiveRecord::Migration end ``` +The final step runs for any un-migrated rows after all of the jobs have been +processed. This is in case a Sidekiq process running the background migrations +received SIGKILL, leading to the jobs being lost. (See +[more reliable Sidekiq queue][reliable-sidekiq] for more information.) + +If the application does not depend on the data being 100% migrated (for +instance, the data is advisory, and not mission-critical), then this final step +can be skipped. + This migration will then process any jobs for the ExtractServicesUrl migration and continue once all jobs have been processed. Once done you can safely remove the `services.properties` column. @@ -254,6 +278,9 @@ for more details. 1. Make sure that background migration jobs are idempotent. 1. Make sure that tests you write are not false positives. +1. Make sure that if the data being migrated is critical and cannot be lost, the + clean-up migration also checks the final state of the data before completing. [migrations-readme]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/migrations/README.md [issue-rspec-hooks]: https://gitlab.com/gitlab-org/gitlab-ce/issues/35351 +[reliable-sidekiq]: https://gitlab.com/gitlab-org/gitlab-ce/issues/36791 diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index d84801f91d4..031b12a8e91 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -29,34 +29,6 @@ For our currently-supported browsers, see our [requirements][requirements]. ## Development Process -When you are assigned an issue please follow the next steps: - -### Divide a big feature into small Merge Requests -1. Big Merge Request are painful to review. In order to make this process easier we -must break a big feature into smaller ones and create a Merge Request for each step. -1. First step is to create a branch from `master`, let's call it `new-feature`. This branch -will be the recipient of all the smaller Merge Requests. Only this one will be merged to master. -1. Don't do any work on this one, let's keep it synced with master. -1. Create a new branch from `new-feature`, let's call it `new-feature-step-1`. We advise you -to clearly identify which step the branch represents. -1. Do the first part of the modifications in this branch. The target branch of this Merge Request -should be `new-feature`. -1. Once `new-feature-step-1` gets merged into `new-feature` we can continue our work. Create a new -branch from `new-feature`, let's call it `new-feature-step-2` and repeat the process done before. - -```shell -master -└─ new-feature - ├─ new-feature-step-1 - ├─ new-feature-step-2 - └─ new-feature-step-3 -``` - -**Tips** -- Make sure `new-feature` branch is always synced with `master`: merge master frequently. -- Do the same for the feature branch you have opened. This can be accomplished by merging `master` into `new-feature` and `new-feature` into `new-feature-step-*` -- Avoid rewriting history. - ### Share your work early 1. Before writing code guarantee your vision of the architecture is aligned with GitLab's architecture. diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md new file mode 100644 index 00000000000..e41d258bec6 --- /dev/null +++ b/doc/development/gitaly.md @@ -0,0 +1,84 @@ +# GitLab Developers Guide to Working with Gitaly + +[Gitaly](https://gitlab.com/gitlab-org/gitaly) is a high-level Git RPC service used by GitLab CE/EE, +Workhorse and GitLab-Shell. All Rugged operations in GitLab CE/EE are currently being phased out to +be replaced by Gitaly API calls. + +Visit the [Gitaly Migration Board](https://gitlab.com/gitlab-org/gitaly/boards/331341) for current +status of the migration. + +## Feature Flags + +Gitaly makes heavy use of [feature flags](feature_flags.md). + +Each Rugged-to-Gitaly migration goes through a [series of phases](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md): + +* **Opt-In**: by default the Rugged implementation is used. + * Production instances can choose to enable the Gitaly endpoint by enabling the feature flag. + * For testing purposes, you may wish to enable all feature flags by default. This can be done by exporting the following + environment variable: `GITALY_FEATURE_DEFAULT_ON=1`. + * On developer instances (ie, when `Rails.env.development?` is true), the Gitaly endpoint + is enabled by default, but can be _disabled_ using feature flags. +* **Opt-Out**: by default, the Gitaly endpoint is used, but the feature can be explicitly disabled using the feature flag. +* **Mandatory**: The migration is complete and cannot be disabled. The old codepath is removed. + +### Enabling and Disabling Feature + +In the Rails console, type: + +```ruby +Feature.enable(:gitaly_feature_name) +Feature.disable(:gitaly_feature_name) +``` + +Where `gitaly_feature_name` is the name of the Gitaly feature. This can be determined by finding the appropriate +`gitaly_migrate` code block, for example: + +```ruby +gitaly_migrate(:tag_names) do +... +end +``` + +Since Gitaly features are always prefixed with `gitaly_`, the name of the feature flag in this case would be `gitaly_tag_names`. + +## Gitaly-Related Test Failures + +If your test-suite is failing with Gitaly issues, as a first step, try running: + +```shell +rm -rf tmp/tests/gitaly +``` + +## `TooManyInvocationsError` errors + +During development and testing, you may experience `Gitlab::GitalyClient::TooManyInvocationsError` failures. +The `GitalyClient` will attempt to block against potential n+1 issues by raising this error +when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution. + +As a temporary measure, export `GITALY_DISABLE_REQUEST_LIMITS=1` to suppress the error. This will disable the n+1 detection +in your development environment. + +Please raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly +~performance ~"technical debt". Please ensure that the issue contains the full stack trace and error message of the +`TooManyInvocationsError`. Also include any known failing tests if possible. + +Isolate the source of the n+1 problem. This will normally be a loop that results in Gitaly being called for each +element in an array. If you are unable to isolate the problem, please contact a member +of the [Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members) for assistance. + +Once the source has been found, wrap it in an `allow_n_plus_1_calls` block, as follows: + +```ruby +# n+1: link to n+1 issue +Gitlab::GitalyClient.allow_n_plus_1_calls do + # original code + commits.each { |commit| ... } +end +``` + +Once the code is wrapped in this block, this code-path will be excluded from n+1 detection. + +--- + +[Return to Development documentation](README.md) diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 9a5811d8474..a75cdf22f40 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -65,6 +65,7 @@ Libraries with the following licenses are unacceptable for use: - [GNU AGPLv3][AGPLv3]: AGPL-licensed libraries cannot be linked to from non-GPL projects. - [Open Software License (OSL)][OSL]: is a copyleft license. In addition, the FSF [recommend against its use][OSL-GNU]. - [Facebook BSD + PATENTS][Facebook]: is a 3-clause BSD license with a patent grant that has been deemed [Category X][x-list] by the Apache foundation. +- [WTFPL][WTFPL]: is a public domain dedication [rejected by the OSI (3.2)][WTFPL-OSI]. Also has a strong language which is not in accordance with our diversity policy. ## Requesting Approval for Licenses @@ -108,3 +109,5 @@ Gems which are included only in the "development" or "test" groups by Bundler ar [x-list]: https://www.apache.org/legal/resolved.html#category-x [Acceptable-Licenses]: #acceptable-licenses [Unacceptable-Licenses]: #unacceptable-licenses +[WTFPL]: https://wtfpl.net +[WTFPL-OSI]: https://opensource.org/minutes20090304 diff --git a/doc/development/testing.md b/doc/development/testing.md index 83269303005..d856b003353 100644 --- a/doc/development/testing.md +++ b/doc/development/testing.md @@ -150,6 +150,16 @@ always in-sync with the codebase. [GitLab QA]: https://gitlab.com/gitlab-org/gitlab-qa [part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa +## Test for what should not be there + +This is particularly important for permission calls and might be called a +negative assertion: make sure only the bare minimum is returned and nothing else. + +See an issue about [leaking tokens] as an example of a vulnerability that is +captured by such a test. + +[leaking tokens]: https://gitlab.com/gitlab-org/gitlab-ce/issues/37948 + ## How to test at the correct level? As many things in life, deciding what to test at each level of testing is a @@ -493,24 +503,24 @@ Here are some things to keep in mind regarding test performance: Our current CI parallelization setup is as follows: -1. The `knapsack` job in the prepare stage that is supposed to ensure we have a - `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file: +1. The `retrieve-tests-metadata` job in the `prepare` stage ensures that we have + a `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file: - The `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file is fetched from S3, if it's not here we initialize the file with `{}`. -1. Each `rspec x y` job are run with `knapsack rspec` and should have an evenly - distributed share of tests: +1. Each `rspec-pg x y`/`rspec-mysql x y` job is run with `knapsack rspec` and + should have an evenly distributed share of tests: - It works because the jobs have access to the `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` since the "artifacts from all previous stages are passed by default". [^1] - - the jobs set their own report path to + - The jobs set their own report path to `KNAPSACK_REPORT_PATH=knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json`. - - if knapsack is doing its job, test files that are run should be listed under + - If knapsack is doing its job, test files that are run should be listed under `Report specs`, not under `Leftover specs`. -1. The `update-knapsack` job takes all the +1. The `update-tests-metadata` job takes all the `knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json` - files from the `rspec x y` jobs and merge them all together into a single - `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file that is then - uploaded to S3. + files from the `rspec-pg x y`/`rspec-mysql x y`jobs and merge them all together + into a single `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file that + is then uploaded to S3. After that, the next pipeline will use the up-to-date `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file. The same strategy diff --git a/doc/development/ux_guide/illustrations.md b/doc/development/ux_guide/illustrations.md new file mode 100644 index 00000000000..7e16c300921 --- /dev/null +++ b/doc/development/ux_guide/illustrations.md @@ -0,0 +1,86 @@ +# Illustrations + +The illustrations should always align with topics and goals in specific context. + +## Principles + +#### Be simple. +- For clarity, we use simple and specific elements to create our illustrations. + +#### Be optimistic. +- We are an open-minded, optimistic, and friendly team. We should reflect those values in our illustrations to connect with our brand experience. + +#### Be gentle. +- Our illustrations assist users in understanding context and guide users in the right direction. Illustrations are supportive, so they should be obvious but not aggressive. + + +## Style + +#### Shapes +- All illustrations are geometric rather than organic. +- The illustrations are made by circles, rectangles, squares, and triangles. + +<img src="img/illustrations-geometric.png" width=224px alt="Example for geometric" /> + +#### Stroke +- Standard border thickness: **4px** +- Depending on the situation, border thickness can be changed to **3px**. For example, when the illustration size is small, an illustration with 4px border thickness would look tight. In this case, the border thickness can be changed to 3px. +- We use **rounded caps** and **rounded corner**. + +| Do | Don't | +| -------- | -------- | +| <img src="img/illustrations-caps-do.png" width= 133px alt="Do: caps and corner" /> | <img src="img/illustrations-caps-don't.png" width= 133px alt="Don't: caps and corner"/> | + +#### Radius +- Standard corner radius: **10px** +- Depending on the situation, corner radius can be changed to **5px**. For example, when the illustration size is small, an illustration with 10px corner radius would be over-rounded. In this case, the corner radius can be changed to 5px. + +<img src="img/illustrations-border-radius.png" width= 464px alt="Example for border radius"/> + +#### Size +Depends on the situation, the illustration size can be the 3 types below: + +**Large** +* Use case: Empty states, error pages(e.g. 404, 403) +* For vertical layout, the illustration should not larger than **430*300 px**. +* For horizontal layout, the illustration should not larger than **430*380 px**. + +| Vertical layout | Horizontal layout | +| --------------- | ----------------- | +| <img src="img/illustration-size-large-vertical.png" /> | <img src="img/illustration-size-large-horizontal.png" /> + +**Medium** +* Use case: Banner +* The illustration should not larger than **240*160 px** +* The illustration should keep simple and clear. We recommend not including too many elements in the medium size illustration. + +<img src="img/illustration-size-medium.png" width=983px /> + +**Small** +* Use case: Graphics for explanatory text, graphics for status. +* The illustration should not larger than **160*90 px**. +* The illustration should keep simple and clear. We recommend not including too many elements in the small size illustration. + +<img src="img/illustration-size-small.png" width=983px /> + +**Illustration on mobile** +- Keep the proportions in original ratio. + +#### Colors palette + +For consistency, we recommend choosing colors from our color palette. + +| Orange | Purple | Grey | +| -------- | -------- | -------- | +| <img src="img/illustrations-color-orange.png" width= 160px alt="Orange" /> | <img src="img/illustrations-color-purple.png" width= 160px alt="Purple" /> | <img src="img/illustrations-color-grey.png" width= 160px alt="Grey" /> | +| #FC6D26 | #6B4FBB | #EEEEEE | + +#### Don't +- Don't include the typography in the illustration. +- Don't include tanuki in the illustration. If necessary, we recommend having tanuki monochromatic. + +--- + +| Orange | Purple | +| -------- | -------- | +| <img src="img/illustrations-palette-oragne.png" width= 160px alt="Palette - Orange" /> | <img src="img/illustrations-palette-purple.png" width= 160px alt="Palette - Purple" /> | diff --git a/doc/development/ux_guide/img/illustration-size-large-horizontal.png b/doc/development/ux_guide/img/illustration-size-large-horizontal.png Binary files differnew file mode 100755 index 00000000000..8aa835adccc --- /dev/null +++ b/doc/development/ux_guide/img/illustration-size-large-horizontal.png diff --git a/doc/development/ux_guide/img/illustration-size-large-vertical.png b/doc/development/ux_guide/img/illustration-size-large-vertical.png Binary files differnew file mode 100644 index 00000000000..813b6a065e5 --- /dev/null +++ b/doc/development/ux_guide/img/illustration-size-large-vertical.png diff --git a/doc/development/ux_guide/img/illustration-size-medium.png b/doc/development/ux_guide/img/illustration-size-medium.png Binary files differnew file mode 100755 index 00000000000..55cfe1dcb91 --- /dev/null +++ b/doc/development/ux_guide/img/illustration-size-medium.png diff --git a/doc/development/ux_guide/img/illustration-size-small.png b/doc/development/ux_guide/img/illustration-size-small.png Binary files differnew file mode 100644 index 00000000000..0124f58f48e --- /dev/null +++ b/doc/development/ux_guide/img/illustration-size-small.png diff --git a/doc/development/ux_guide/img/illustrations-border-radius.png b/doc/development/ux_guide/img/illustrations-border-radius.png Binary files differnew file mode 100755 index 00000000000..4e2fef5c7f5 --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-border-radius.png diff --git a/doc/development/ux_guide/img/illustrations-caps-do.png b/doc/development/ux_guide/img/illustrations-caps-do.png Binary files differnew file mode 100755 index 00000000000..7a2c74382f6 --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-caps-do.png diff --git a/doc/development/ux_guide/img/illustrations-caps-don't.png b/doc/development/ux_guide/img/illustrations-caps-don't.png Binary files differnew file mode 100755 index 00000000000..848f72dbe30 --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-caps-don't.png diff --git a/doc/development/ux_guide/img/illustrations-color-grey.png b/doc/development/ux_guide/img/illustrations-color-grey.png Binary files differnew file mode 100755 index 00000000000..63855026c2b --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-color-grey.png diff --git a/doc/development/ux_guide/img/illustrations-color-orange.png b/doc/development/ux_guide/img/illustrations-color-orange.png Binary files differnew file mode 100755 index 00000000000..96765c8c28c --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-color-orange.png diff --git a/doc/development/ux_guide/img/illustrations-color-purple.png b/doc/development/ux_guide/img/illustrations-color-purple.png Binary files differnew file mode 100755 index 00000000000..745d2c853ba --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-color-purple.png diff --git a/doc/development/ux_guide/img/illustrations-geometric.png b/doc/development/ux_guide/img/illustrations-geometric.png Binary files differnew file mode 100755 index 00000000000..33f05547bac --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-geometric.png diff --git a/doc/development/ux_guide/img/illustrations-palette-oragne.png b/doc/development/ux_guide/img/illustrations-palette-oragne.png Binary files differnew file mode 100755 index 00000000000..15f35912646 --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-palette-oragne.png diff --git a/doc/development/ux_guide/img/illustrations-palette-purple.png b/doc/development/ux_guide/img/illustrations-palette-purple.png Binary files differnew file mode 100755 index 00000000000..e0f5839705e --- /dev/null +++ b/doc/development/ux_guide/img/illustrations-palette-purple.png diff --git a/doc/development/ux_guide/index.md b/doc/development/ux_guide/index.md index 8a849f239dc..42bcf234e12 100644 --- a/doc/development/ux_guide/index.md +++ b/doc/development/ux_guide/index.md @@ -21,6 +21,11 @@ Guidance on the timing, curving and motion for GitLab. --- +### [Illustrations](illustrations.md) +Guidelines for principals and styles related to illustrations for GitLab. + +--- + ### [Copy](copy.md) Conventions on text and messaging within labels, buttons, and other components. diff --git a/doc/development/writing_documentation.md b/doc/development/writing_documentation.md index b1eb020a592..68ba3dd2da3 100644 --- a/doc/development/writing_documentation.md +++ b/doc/development/writing_documentation.md @@ -1,6 +1,6 @@ # Writing documentation - - **General Documentation**: written by the developers responsible by creating features. Should be submitted in the same merge request containing code. Feature proposals (by GitLab contributors) should also be accompanied by its respective documentation. They can be later improved by PMs and Technical Writers. + - **General Documentation**: written by the [developers responsible by creating features](#contributing-to-docs). Should be submitted in the same merge request containing code. Feature proposals (by GitLab contributors) should also be accompanied by its respective documentation. They can be later improved by PMs and Technical Writers. - **Technical Articles**: written by any [GitLab Team](https://about.gitlab.com/team/) member, GitLab contributors, or [Community Writers](https://about.gitlab.com/handbook/product/technical-writing/community-writers/). - **Indexes per topic**: initially prepared by the Technical Writing Team, and kept up-to-date by developers and PMs in the same merge request containing code. They gather all resources for that topic in a single page (user and admin documentation, articles, and third-party docs). @@ -69,6 +69,51 @@ Use the [writing method](https://about.gitlab.com/handbook/product/technical-wri All the docs follow the same [styleguide](doc_styleguide.md). +### Contributing to docs + +Whenever a feature is changed, updated, introduced, or deprecated, the merge +request introducing these changes must be accompanied by the documentation +(either updating existing ones or creating new ones). This is also valid when +changes are introduced to the UI. + +The one resposible for writing the first piece of documentation is the developer who +wrote the code. It's the job of the Product Manager to ensure all features are +shipped with its docs, whether is a small or big change. At the pace GitLab evolves, +this is the only way to keep the docs up-to-date. If you have any questions about it, +please ask a Technical Writer. Otherwise, when your content is ready, assign one of +them to review it for you. + +We use the [monthly release blog post](https://about.gitlab.com/handbook/marketing/blog/release-posts/#monthly-releases) as a changelog checklist to ensure everything +is documented. + +### Feature overview and use cases + +Every major feature (regardless if present in GitLab Community or Enterprise editions) +should present, at the beginning of the document, two main sections: **overview** and +**use cases**. Every GitLab EE-only feature should also contain these sections. + +**Overview**: at the name suggests, the goal here is to provide an overview of the feature. +Describe what is it, what it does, why it is important/cool/nice-to-have, +what problem it solves, and what you can do with this feature that you couldn't +do before. + +**Use cases**: provide at least two, ideally three, use cases for every major feature. +You should answer this question: what can you do with this feature/change? Use cases +are examples of how this feauture or change can be used in real life. + +Examples: +- CE and EE: [Issues](../user/project/issues/index.md#use-cases) +- CE and EE: [Merge Requests](../user/project/merge_requests/index.md#overview) +- EE-only: [Geo](https://docs.gitlab.com/ee/gitlab-geo/README.html#overview) +- EE-only: [Jenkins integration](https://docs.gitlab.com/ee/integration/jenkins.md#overview) + +Note that if you don't have anything to add between the doc title (`<h1>`) and +the header `## Overview`, you can omit the header, but keep the content of the +overview there. + +> **Overview** and **use cases** are required to **every** Enterprise Edition feature, +and for every **major** feature present in Community Edition. + ### Markdown Currently GitLab docs use Redcarpet as [markdown](../user/markdown.md) engine, but there's an [open discussion](https://gitlab.com/gitlab-com/gitlab-docs/issues/50) for implementing Kramdown in the near future. |
