diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-05-05 09:08:00 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-05-05 09:08:00 +0000 |
commit | 17ef30f3df6d3939e41e69efc7cfa3deaa08605d (patch) | |
tree | 6852730e03de7e85e7a42952ec85960ab9832fa5 /doc/development/experiment_guide | |
parent | cd9bbd8a3e8af73864ca3c7704211309fae8ce0e (diff) | |
download | gitlab-ce-17ef30f3df6d3939e41e69efc7cfa3deaa08605d.tar.gz |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development/experiment_guide')
6 files changed, 663 insertions, 630 deletions
diff --git a/doc/development/experiment_guide/experiment_code_reviews.md b/doc/development/experiment_guide/experiment_code_reviews.md new file mode 100644 index 00000000000..fdde89caa34 --- /dev/null +++ b/doc/development/experiment_guide/experiment_code_reviews.md @@ -0,0 +1,25 @@ +--- +stage: Growth +group: Adoption +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Experiment code reviews + +Experiments' code quality can fail our standards for several reasons. These +reasons can include not being added to the codebase for a long time, or because +of fast iteration to retrieve data. However, having the experiment run (or not +run) shouldn't impact GitLab availability. To avoid or identify issues, +experiments are initially deployed to a small number of users. Regardless, +experiments still need tests. + +Experiments must have corresponding [frontend or feature tests](../testing_guide/index.md) to ensure they +exist in the application. These tests should help prevent the experiment code from +being removed before the [experiment cleanup process](https://about.gitlab.com/handbook/engineering/development/growth/experimentation/#experiment-cleanup-issue) starts. + +If, as a reviewer or maintainer, you find code that would usually fail review +but is acceptable for now, mention your concerns with a note that there's no +need to change the code. The author can then add a comment to this piece of code +and link to the issue that resolves the experiment. The author or reviewer can add a link to this concern in the +experiment rollout issue under the `Experiment Successful Cleanup Concerns` section of the description. +If the experiment is successful and becomes part of the product, any items that appear under this section will be addressed. diff --git a/doc/development/experiment_guide/experiment_rollout.md b/doc/development/experiment_guide/experiment_rollout.md new file mode 100644 index 00000000000..bc700b13600 --- /dev/null +++ b/doc/development/experiment_guide/experiment_rollout.md @@ -0,0 +1,77 @@ +--- +stage: Growth +group: Adoption +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Experiment rollouts and feature flags + +## Experiment rollout issue + +Each experiment should have an [experiment rollout](https://gitlab.com/groups/gitlab-org/-/boards/1352542) issue to track the experiment from rollout through to cleanup and removal. +The rollout issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. + +When an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). +After the deadline, the issue needs to be resolved and either: + +- It was successful and the experiment becomes the new default. +- It was not successful and all code related to the experiment is removed. + +In either case, an outcome of the experiment should be posted to the issue with the reasoning for the decision. + +## Turn off all experiments + +When there is a case on GitLab.com (SaaS) that necessitates turning off all experiments, we have this control. + +You can toggle experiments on SaaS on and off using the `gitlab_experiment` [feature flag](../feature_flags). + +This can be done via chatops: + +- [disable](../feature_flags/controls.md#disabling-feature-flags): `/chatops run feature set gitlab_experiment false` +- [enable](../feature_flags/controls.md#process): `/chatops run feature delete gitlab_experiment` +- This allows the `default_enabled` [value of true in the yml](https://gitlab.com/gitlab-org/gitlab/-/blob/016430f6751b0c34abb24f74608c80a1a8268f20/config/feature_flags/ops/gitlab_experiment.yml#L8) to be honored. + +## Notes on feature flags + +NOTE: +We use the terms "enabled" and "disabled" here, even though it's against our +[documentation style guide recommendations](../documentation/styleguide/word_list.md#enable) +because these are the terms that the feature flag documentation uses. + +You may already be familiar with the concept of feature flags in GitLab, but using +feature flags in experiments is a bit different. While in general terms, a feature flag +is viewed as being either `on` or `off`, this isn't accurate for experiments. + +Generally, `off` means that when we ask if a feature flag is enabled, it will always +return `false`, and `on` means that it will always return `true`. An interim state, +considered `conditional`, also exists. We take advantage of this trinary state of +feature flags. To understand this `conditional` aspect: consider that either of these +settings puts a feature flag into this state: + +- Setting a `percentage_of_actors` of any percent greater than 0%. +- Enabling it for a single user or group. + +Conditional means that it returns `true` in some situations, but not all situations. + +When a feature flag is disabled (meaning the state is `off`), the experiment is +considered _inactive_. You can visualize this in the [decision tree diagram](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment#how-it-works) +as reaching the first `Running?` node, and traversing the negative path. + +When a feature flag is rolled out to a `percentage_of_actors` or similar (meaning the +state is `conditional`) the experiment is considered to be _running_ +where sometimes the control is assigned, and sometimes the candidate is assigned. +We don't refer to this as being enabled, because that's a confusing and overloaded +term here. In the experiment terms, our experiment is _running_, and the feature flag is +`conditional`. + +When a feature flag is enabled (meaning the state is `on`), the candidate will always be +assigned. + +We should try to be consistent with our terms, and so for experiments, we have an +_inactive_ experiment until we set the feature flag to `conditional`. After which, +our experiment is then considered _running_. If you choose to "enable" your feature flag, +you should consider the experiment to be _resolved_, because everyone is assigned +the candidate unless they've opted out of experimentation. + +As of GitLab 13.10, work is being done to improve this process and how we communicate +about it. diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index 78e1f84d701..5ddbe9b3de9 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -1,586 +1,11 @@ --- -stage: Growth -group: Adoption -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2022-08-05' --- -# Implementing an A/B/n experiment +This document was moved to [another location](index.md). -## Introduction - -Experiments in GitLab are tightly coupled with the concepts provided by -[Feature flags in development of GitLab](../feature_flags/index.md). You're strongly encouraged -to read and understand the [Feature flags in development of GitLab](../feature_flags/index.md) -portion of the documentation before considering running experiments. Experiments add additional -concepts which may seem confusing or advanced without understanding the underpinnings of how GitLab -uses feature flags in development. One concept: experiments can be run with multiple variants, -which are sometimes referred to as A/B/n tests. - -We use the [`gitlab-experiment` gem](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment), -sometimes referred to as GLEX, to run our experiments. The gem exists in a separate repository -so it can be shared across any GitLab property that uses Ruby. You should feel comfortable reading -the documentation on that project if you want to dig into more advanced topics or open issues. Be -aware that the documentation there reflects what's in the main branch and may not be the same as -the version being used within GitLab. - -## Glossary of terms - -To ensure a shared language, you should understand these fundamental terms we use -when communicating about experiments: - -- `experiment`: Any deviation of code paths we want to run at some times, but not others. -- `context`: A consistent experience we provide in an experiment. -- `control`: The default, or "original" code path. -- `candidate`: Defines an experiment with only one code path. -- `variant(s)`: Defines an experiment with multiple code paths. -- `behaviors`: Used to reference all possible code paths of an experiment, including the control. - -## Implementing an experiment - -[Examples](https://gitlab.com/gitlab-org/growth/growth/-/wikis/GLEX-Framework-code-examples) - -Start by generating a feature flag using the `bin/feature-flag` command as you -normally would for a development feature flag, making sure to use `experiment` for -the type. For the sake of documentation let's name our feature flag (and experiment) -"pill_color". - -```shell -bin/feature-flag pill_color -t experiment -``` - -After you generate the desired feature flag, you can immediately implement an -experiment in code. An experiment implementation can be as simple as: - -```ruby -experiment(:pill_color, actor: current_user) do |e| - e.control { 'control' } - e.variant(:red) { 'red' } - e.variant(:blue) { 'blue' } -end -``` - -When this code executes, the experiment is run, a variant is assigned, and (if within a -controller or view) a `window.gl.experiments.pill_color` object will be available in the -client layer, with details like: - -- The assigned variant. -- The context key for client tracking events. - -In addition, when an experiment runs, an event is tracked for -the experiment `:assignment`. We cover more about events, tracking, and -the client layer later. - -In local development, you can make the experiment active by using the feature flag -interface. You can also target specific cases by providing the relevant experiment -to the call to enable the feature flag: - -```ruby -# Enable for everyone -Feature.enable(:pill_color) - -# Get the `experiment` method -- already available in controllers, views, and mailers. -include Gitlab::Experiment::Dsl -# Enable for only the first user -Feature.enable(:pill_color, experiment(:pill_color, actor: User.first)) -``` - -To roll out your experiment feature flag on an environment, run -the following command using ChatOps (which is covered in more depth in the -[Feature flags in development of GitLab](../feature_flags/index.md) documentation). -This command creates a scenario where half of everyone who encounters -the experiment would be assigned the _control_, 25% would be assigned the _red_ -variant, and 25% would be assigned the _blue_ variant: - -```slack -/chatops run feature set pill_color 50 --actors -``` - -For an even distribution in this example, change the command to set it to 66% instead -of 50. - -NOTE: -To immediately stop running an experiment, use the -`/chatops run feature set pill_color false` command. - -WARNING: -We strongly recommend using the `--actors` flag when using the ChatOps commands, -as anything else may give odd behaviors due to how the caching of variant assignment is -handled. - -We can also implement this experiment in a HAML file with HTML wrappings: - -```haml -#cta-interface - - experiment(:pill_color, actor: current_user) do |e| - - e.control do - .pill-button control - - e.variant(:red) do - .pill-button.red red - - e.variant(:blue) do - .pill-button.blue blue -``` - -### The importance of context - -In our previous example experiment, our context (this is an important term) is a hash -that's set to `{ actor: current_user }`. Context must be unique based on how you -want to run your experiment, and should be understood at a lower level. - -It's expected, and recommended, that you use some of these -contexts to simplify reporting: - -- `{ actor: current_user }`: Assigns a variant and is "sticky" to each user - (or "client" if `current_user` is nil) who enters the experiment. -- `{ project: project }`: Assigns a variant and is "sticky" to the project currently - being viewed. If running your experiment is more useful when viewing a project, - rather than when a specific user is viewing any project, consider this approach. -- `{ group: group }`: Similar to the project example, but applies to a wider - scope of projects and users. -- `{ actor: current_user, project: project }`: Assigns a variant and is "sticky" - to the user who is viewing the given project. This creates a different variant - assignment possibility for every project that `current_user` views. Understand this - can create a large cache size if an experiment like this in a highly trafficked part - of the application. -- `{ wday: Time.current.wday }`: Assigns a variant based on the current day of the - week. In this example, it would consistently assign one variant on Friday, and a - potentially different variant on Saturday. - -Context is critical to how you define and report on your experiment. It's usually -the most important aspect of how you choose to implement your experiment, so consider -it carefully, and discuss it with the wider team if needed. Also, take into account -that the context you choose affects our cache size. - -After the above examples, we can state the general case: *given a specific -and consistent context, we can provide a consistent experience and track events for -that experience.* To dive a bit deeper into the implementation details: a context key -is generated from the context that's provided. Use this context key to: - -- Determine the assigned variant. -- Identify events tracked against that context key. - -We can think about this as the experience that we've rendered, which is both dictated -and tracked by the context key. The context key is used to track the interaction and -results of the experience we've rendered to that context key. These concepts are -somewhat abstract and hard to understand initially, but this approach enables us to -communicate about experiments as something that's wider than just user behavior. - -NOTE: -Using `actor:` utilizes cookies if the `current_user` is nil. If you don't need -cookies though - meaning that the exposed functionality would only be visible to -signed in users - `{ user: current_user }` would be just as effective. - -WARNING: -The caching of variant assignment is done by using this context, and so consider -your impact on the cache size when defining your experiment. If you use -`{ time: Time.current }` you would be inflating the cache size every time the -experiment is run. Not only that, your experiment would not be "sticky" and events -wouldn't be resolvable. - -### Advanced experimentation - -There are two ways to implement an experiment: - -1. The simple experiment style described previously. -1. A more advanced style where an experiment class is provided. - -The advanced style is handled by naming convention, and works similar to what you -would expect in Rails. - -To generate a custom experiment class that can override the defaults in -`ApplicationExperiment` use the Rails generator: - -```shell -rails generate gitlab:experiment pill_color control red blue -``` - -This generates an experiment class in `app/experiments/pill_color_experiment.rb` -with the _behaviors_ we've provided to the generator. Here's an example -of how that class would look after migrating our previous example into it: - -```ruby -class PillColorExperiment < ApplicationExperiment - control { 'control' } - variant(:red) { 'red' } - variant(:blue) { 'blue' } -end -``` - -We can now simplify where we run our experiment to the following call, instead of -providing the block we were initially providing, by explicitly calling `run`: - -```ruby -experiment(:pill_color, actor: current_user).run -``` - -The _behaviors_ we defined in our experiment class represent the default -implementation. You can still use the block syntax to override these _behaviors_ -however, so the following would also be valid: - -```ruby -experiment(:pill_color, actor: current_user) do |e| - e.control { '<strong>control</strong>' } -end -``` - -NOTE: -When passing a block to the `experiment` method, it is implicitly invoked as -if `run` has been called. - -#### Segmentation rules - -You can use runtime segmentation rules to, for instance, segment contexts into a specific -variant. The `segment` method is a callback (like `before_action`) and so allows providing -a block or method name. - -In this example, any user named `'Richard'` would always be assigned the _red_ -variant, and any account older than 2 weeks old would be assigned the _blue_ variant: - -```ruby -class PillColorExperiment < ApplicationExperiment - # ...registered behaviors - - segment(variant: :red) { context.actor.first_name == 'Richard' } - segment :old_account?, variant: :blue - - private - - def old_account? - context.actor.created_at < 2.weeks.ago - end -end -``` - -When an experiment runs, the segmentation rules are executed in the order they're -defined. The first segmentation rule to produce a truthy result assigns the variant. - -In our example, any user named `'Richard'`, regardless of account age, will always -be assigned the _red_ variant. If you want the opposite logic, flip the order. - -NOTE: -Keep in mind when defining segmentation rules: after a truthy result, the remaining -segmentation rules are skipped to achieve optimal performance. - -#### Exclusion rules - -Exclusion rules are similar to segmentation rules, but are intended to determine -if a context should even be considered as something we should include in the experiment -and track events toward. Exclusion means we don't care about the events in relation -to the given context. - -These examples exclude all users named `'Richard'`, *and* any account -older than 2 weeks old. Not only are they given the control behavior - which could -be nothing - but no events are tracked in these cases as well. - -```ruby -class PillColorExperiment < ApplicationExperiment - # ...registered behaviors - - exclude :old_account?, ->{ context.actor.first_name == 'Richard' } - - private - - def old_account? - context.actor.created_at < 2.weeks.ago - end -end -``` - -You may also need to check exclusion in custom tracking logic by calling `should_track?`: - -```ruby -class PillColorExperiment < ApplicationExperiment - # ...registered behaviors - - def expensive_tracking_logic - return unless should_track? - - track(:my_event, value: expensive_method_call) - end -end -``` - -### Tracking events - -One of the most important aspects of experiments is gathering data and reporting on -it. You can use the `track` method to track events across an experimental implementation. -You can track events consistently to an experiment if you provide the same context between -calls to your experiment. If you do not yet understand context, you should read -about contexts now. - -We can assume we run the experiment in one or a few places, but -track events potentially in many places. The tracking call remains the same, with -the arguments you would normally use when -[tracking events using snowplow](../snowplow/index.md). The easiest example -of tracking an event in Ruby would be: - -```ruby -experiment(:pill_color, actor: current_user).track(:clicked) -``` - -When you run an experiment with any of the examples so far, an `:assignment` event -is tracked automatically by default. All events that are tracked from an -experiment have a special -[experiment context](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-3) -added to the event. This can be used - typically by the data team - to create a connection -between the events on a given experiment. - -If our current user hasn't encountered the experiment yet (meaning where the experiment -is run), and we track an event for them, they are assigned a variant and see -that variant if they ever encountered the experiment later, when an `:assignment` -event would be tracked at that time for them. - -NOTE: -GitLab tries to be sensitive and respectful of our customers regarding tracking, -so our experimentation library allows us to implement an experiment without ever tracking identifying -IDs. It's not always possible, though, based on experiment reporting requirements. -You may be asked from time to time to track a specific record ID in experiments. -The approach is largely up to the PM and engineer creating the implementation. -No recommendations are provided here at this time. - -## Testing with RSpec - -In the course of working with experiments, you'll probably want to utilize the RSpec -tooling that's built in. This happens automatically for files in `spec/experiments`, but -for other files and specs you want to include it in, you can specify the `:experiment` type: - -```ruby -it "tests experiments nicely", :experiment do -end -``` - -### Stub helpers - -You can stub experiments using `stub_experiments`. Pass it a hash using experiment -names as the keys, and the variants you want each to resolve to, as the values: - -```ruby -# Ensures the experiments named `:example` & `:example2` are both "enabled" and -# that each will resolve to the given variant (`:my_variant` and `:control` -# respectively). -stub_experiments(example: :my_variant, example2: :control) - -experiment(:example) do |e| - e.enabled? # => true - e.assigned.name # => 'my_variant' -end - -experiment(:example2) do |e| - e.enabled? # => true - e.assigned.name # => 'control' -end -``` - -### Exclusion, segmentation, and behavior matchers - -You can also test things like the registered behaviors, the exclusions, and -segmentations using the matchers. - -```ruby -class ExampleExperiment < ApplicationExperiment - control { } - candidate { '_candidate_' } - - exclude { context.actor.first_name == 'Richard' } - segment(variant: :candidate) { context.actor.username == 'jejacks0n' } -end - -excluded = double(username: 'rdiggitty', first_name: 'Richard') -segmented = double(username: 'jejacks0n', first_name: 'Jeremy') - -# register_behavior matcher -expect(experiment(:example)).to register_behavior(:control) -expect(experiment(:example)).to register_behavior(:candidate).with('_candidate_') - -# exclude matcher -expect(experiment(:example)).to exclude(actor: excluded) -expect(experiment(:example)).not_to exclude(actor: segmented) - -# segment matcher -expect(experiment(:example)).to segment(actor: segmented).into(:candidate) -expect(experiment(:example)).not_to segment(actor: excluded) -``` - -### Tracking matcher - -Tracking events is a major aspect of experimentation. We try -to provide a flexible way to ensure your tracking calls are covered. - -You can do this on the instance level or at an "any instance" level: - -```ruby -subject = experiment(:example) - -expect(subject).to track(:my_event) - -subject.track(:my_event) -``` - -You can use the `on_next_instance` chain method to specify that it will happen -on the next instance of the experiment. This helps you if you're calling -`experiment(:example).track` downstream: - -```ruby -expect(experiment(:example)).to track(:my_event).on_next_instance - -experiment(:example).track(:my_event) -``` - -A full example of the methods you can chain onto the `track` matcher: - -```ruby -expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_') - .on_next_instance - .with_context(foo: :bar) - .for(:variant_name) - -experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_') -``` - -## Experiments in the client layer - -Any experiment that's been run in the request lifecycle surfaces in `window.gl.experiments`, -and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-3) -so it can be used when resolving experimentation in the client layer. - -Given that we've defined a class for our experiment, and have defined the variants for it, we can publish that experiment in a couple ways. - -The first way is simply by running the experiment. Assuming the experiment has been run, it will surface in the client layer without having to do anything special. - -The second way doesn't run the experiment and is intended to be used if the experiment only needs to surface in the client layer. To accomplish this we can simply `.publish` the experiment. This won't run any logic, but does surface the experiment details in the client layer so they can be utilized there. - -An example might be to publish an experiment in a `before_action` in a controller. Assuming we've defined the `PillColorExperiment` class, like we have above, we can surface it to the client by publishing it instead of running it: - -```ruby -before_action -> { experiment(:pill_color).publish }, only: [:show] -``` - -You can then see this surface in the JavaScript console: - -```javascript -window.gl.experiments // => { pill_color: { excluded: false, experiment: "pill_color", key: "ca63ac02", variant: "candidate" } } -``` - -### Using experiments in Vue - -With the `gitlab-experiment` component, you can define slots that match the name of the -variants pushed to `window.gl.experiments`. - -We can make use of the named slots in the Vue component, that match the behaviors defined in : - -```vue -<script> -import GitlabExperiment from '~/experimentation/components/gitlab_experiment.vue'; - -export default { - components: { GitlabExperiment } -} -</script> - -<template> - <gitlab-experiment name="pill_color"> - <template #control> - <button class="bg-default">Click default button</button> - </template> - - <template #red> - <button class="bg-red">Click red button</button> - </template> - - <template #blue> - <button class="bg-blue">Click blue button</button> - </template> - </gitlab-experiment> -</template> -``` - -NOTE: -When there is no experiment data in the `window.gl.experiments` object for the given experiment name, the `control` slot will be used, if it exists. - -## Test with Jest - -### Stub Helpers - -You can stub experiments using the `stubExperiments` helper defined in `spec/frontend/__helpers__/experimentation_helper.js`. - -```javascript -import { stubExperiments } from 'helpers/experimentation_helper'; -import { getExperimentData } from '~/experimentation/utils'; - -describe('when my_experiment is enabled', () => { - beforeEach(() => { - stubExperiments({ my_experiment: 'candidate' }); - }); - - it('sets the correct data', () => { - expect(getExperimentData('my_experiment')).toEqual({ experiment: 'my_experiment', variant: 'candidate' }); - }); -}); -``` - -NOTE: -This method of stubbing in Jest specs will not automatically un-stub itself at the end of the test. We merge our stubbed experiment in with all the other global data in `window.gl`. If you need to remove the stubbed experiment(s) after your test or ensure a clean global object before your test, you'll need to manage the global object directly yourself: - -```javascript -describe('tests that care about global state', () => { - const originalObjects = []; - - beforeEach(() => { - // For backwards compatibility for now, we're using both window.gon & window.gl - originalObjects.push(window.gon, window.gl); - }); - - afterEach(() => { - [window.gon, window.gl] = originalObjects; - }); - - it('stubs experiment in fresh global state', () => { - stubExperiment({ my_experiment: 'candidate' }); - // ... - }); -}) -``` - -## Notes on feature flags - -NOTE: -We use the terms "enabled" and "disabled" here, even though it's against our -[documentation style guide recommendations](../documentation/styleguide/word_list.md#enable) -because these are the terms that the feature flag documentation uses. - -You may already be familiar with the concept of feature flags in GitLab, but using -feature flags in experiments is a bit different. While in general terms, a feature flag -is viewed as being either `on` or `off`, this isn't accurate for experiments. - -Generally, `off` means that when we ask if a feature flag is enabled, it will always -return `false`, and `on` means that it will always return `true`. An interim state, -considered `conditional`, also exists. We take advantage of this trinary state of -feature flags. To understand this `conditional` aspect: consider that either of these -settings puts a feature flag into this state: - -- Setting a `percentage_of_actors` of any percent greater than 0%. -- Enabling it for a single user or group. - -Conditional means that it returns `true` in some situations, but not all situations. - -When a feature flag is disabled (meaning the state is `off`), the experiment is -considered _inactive_. You can visualize this in the [decision tree diagram](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment#how-it-works) -as reaching the first `Running?` node, and traversing the negative path. - -When a feature flag is rolled out to a `percentage_of_actors` or similar (meaning the -state is `conditional`) the experiment is considered to be _running_ -where sometimes the control is assigned, and sometimes the candidate is assigned. -We don't refer to this as being enabled, because that's a confusing and overloaded -term here. In the experiment terms, our experiment is _running_, and the feature flag is -`conditional`. - -When a feature flag is enabled (meaning the state is `on`), the candidate will always be -assigned. - -We should try to be consistent with our terms, and so for experiments, we have an -_inactive_ experiment until we set the feature flag to `conditional`. After which, -our experiment is then considered _running_. If you choose to "enable" your feature flag, -you should consider the experiment to be _resolved_, because everyone is assigned -the candidate unless they've opted out of experimentation. - -As of GitLab 13.10, work is being done to improve this process and how we communicate -about it. +<!-- This redirect file can be deleted after 2022-08-05. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/experiment_guide/implementing_experiments.md b/doc/development/experiment_guide/implementing_experiments.md new file mode 100644 index 00000000000..3c33d015108 --- /dev/null +++ b/doc/development/experiment_guide/implementing_experiments.md @@ -0,0 +1,369 @@ +--- +stage: Growth +group: Adoption +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Implementing an A/B/n experiment + +## Implementing an experiment + +[Examples](https://gitlab.com/gitlab-org/growth/growth/-/wikis/GLEX-Framework-code-examples) + +Start by generating a feature flag using the `bin/feature-flag` command as you +normally would for a development feature flag, making sure to use `experiment` for +the type. For the sake of documentation let's name our feature flag (and experiment) +"pill_color". + +```shell +bin/feature-flag pill_color -t experiment +``` + +After you generate the desired feature flag, you can immediately implement an +experiment in code. An experiment implementation can be as simple as: + +```ruby +experiment(:pill_color, actor: current_user) do |e| + e.control { 'control' } + e.variant(:red) { 'red' } + e.variant(:blue) { 'blue' } +end +``` + +When this code executes, the experiment is run, a variant is assigned, and (if within a +controller or view) a `window.gl.experiments.pill_color` object will be available in the +client layer, with details like: + +- The assigned variant. +- The context key for client tracking events. + +In addition, when an experiment runs, an event is tracked for +the experiment `:assignment`. We cover more about events, tracking, and +the client layer later. + +In local development, you can make the experiment active by using the feature flag +interface. You can also target specific cases by providing the relevant experiment +to the call to enable the feature flag: + +```ruby +# Enable for everyone +Feature.enable(:pill_color) + +# Get the `experiment` method -- already available in controllers, views, and mailers. +include Gitlab::Experiment::Dsl +# Enable for only the first user +Feature.enable(:pill_color, experiment(:pill_color, actor: User.first)) +``` + +To roll out your experiment feature flag on an environment, run +the following command using ChatOps (which is covered in more depth in the +[Feature flags in development of GitLab](../feature_flags/index.md) documentation). +This command creates a scenario where half of everyone who encounters +the experiment would be assigned the _control_, 25% would be assigned the _red_ +variant, and 25% would be assigned the _blue_ variant: + +```slack +/chatops run feature set pill_color 50 --actors +``` + +For an even distribution in this example, change the command to set it to 66% instead +of 50. + +NOTE: +To immediately stop running an experiment, use the +`/chatops run feature set pill_color false` command. + +WARNING: +We strongly recommend using the `--actors` flag when using the ChatOps commands, +as anything else may give odd behaviors due to how the caching of variant assignment is +handled. + +We can also implement this experiment in a HAML file with HTML wrappings: + +```haml +#cta-interface + - experiment(:pill_color, actor: current_user) do |e| + - e.control do + .pill-button control + - e.variant(:red) do + .pill-button.red red + - e.variant(:blue) do + .pill-button.blue blue +``` + +### The importance of context + +In our previous example experiment, our context (this is an important term) is a hash +that's set to `{ actor: current_user }`. Context must be unique based on how you +want to run your experiment, and should be understood at a lower level. + +It's expected, and recommended, that you use some of these +contexts to simplify reporting: + +- `{ actor: current_user }`: Assigns a variant and is "sticky" to each user + (or "client" if `current_user` is nil) who enters the experiment. +- `{ project: project }`: Assigns a variant and is "sticky" to the project currently + being viewed. If running your experiment is more useful when viewing a project, + rather than when a specific user is viewing any project, consider this approach. +- `{ group: group }`: Similar to the project example, but applies to a wider + scope of projects and users. +- `{ actor: current_user, project: project }`: Assigns a variant and is "sticky" + to the user who is viewing the given project. This creates a different variant + assignment possibility for every project that `current_user` views. Understand this + can create a large cache size if an experiment like this in a highly trafficked part + of the application. +- `{ wday: Time.current.wday }`: Assigns a variant based on the current day of the + week. In this example, it would consistently assign one variant on Friday, and a + potentially different variant on Saturday. + +Context is critical to how you define and report on your experiment. It's usually +the most important aspect of how you choose to implement your experiment, so consider +it carefully, and discuss it with the wider team if needed. Also, take into account +that the context you choose affects our cache size. + +After the above examples, we can state the general case: *given a specific +and consistent context, we can provide a consistent experience and track events for +that experience.* To dive a bit deeper into the implementation details: a context key +is generated from the context that's provided. Use this context key to: + +- Determine the assigned variant. +- Identify events tracked against that context key. + +We can think about this as the experience that we've rendered, which is both dictated +and tracked by the context key. The context key is used to track the interaction and +results of the experience we've rendered to that context key. These concepts are +somewhat abstract and hard to understand initially, but this approach enables us to +communicate about experiments as something that's wider than just user behavior. + +NOTE: +Using `actor:` utilizes cookies if the `current_user` is nil. If you don't need +cookies though - meaning that the exposed functionality would only be visible to +signed in users - `{ user: current_user }` would be just as effective. + +WARNING: +The caching of variant assignment is done by using this context, and so consider +your impact on the cache size when defining your experiment. If you use +`{ time: Time.current }` you would be inflating the cache size every time the +experiment is run. Not only that, your experiment would not be "sticky" and events +wouldn't be resolvable. + +### Advanced experimentation + +There are two ways to implement an experiment: + +1. The simple experiment style described previously. +1. A more advanced style where an experiment class is provided. + +The advanced style is handled by naming convention, and works similar to what you +would expect in Rails. + +To generate a custom experiment class that can override the defaults in +`ApplicationExperiment` use the Rails generator: + +```shell +rails generate gitlab:experiment pill_color control red blue +``` + +This generates an experiment class in `app/experiments/pill_color_experiment.rb` +with the _behaviors_ we've provided to the generator. Here's an example +of how that class would look after migrating our previous example into it: + +```ruby +class PillColorExperiment < ApplicationExperiment + control { 'control' } + variant(:red) { 'red' } + variant(:blue) { 'blue' } +end +``` + +We can now simplify where we run our experiment to the following call, instead of +providing the block we were initially providing, by explicitly calling `run`: + +```ruby +experiment(:pill_color, actor: current_user).run +``` + +The _behaviors_ we defined in our experiment class represent the default +implementation. You can still use the block syntax to override these _behaviors_ +however, so the following would also be valid: + +```ruby +experiment(:pill_color, actor: current_user) do |e| + e.control { '<strong>control</strong>' } +end +``` + +NOTE: +When passing a block to the `experiment` method, it is implicitly invoked as +if `run` has been called. + +#### Segmentation rules + +You can use runtime segmentation rules to, for instance, segment contexts into a specific +variant. The `segment` method is a callback (like `before_action`) and so allows providing +a block or method name. + +In this example, any user named `'Richard'` would always be assigned the _red_ +variant, and any account older than 2 weeks old would be assigned the _blue_ variant: + +```ruby +class PillColorExperiment < ApplicationExperiment + # ...registered behaviors + + segment(variant: :red) { context.actor.first_name == 'Richard' } + segment :old_account?, variant: :blue + + private + + def old_account? + context.actor.created_at < 2.weeks.ago + end +end +``` + +When an experiment runs, the segmentation rules are executed in the order they're +defined. The first segmentation rule to produce a truthy result assigns the variant. + +In our example, any user named `'Richard'`, regardless of account age, will always +be assigned the _red_ variant. If you want the opposite logic, flip the order. + +NOTE: +Keep in mind when defining segmentation rules: after a truthy result, the remaining +segmentation rules are skipped to achieve optimal performance. + +#### Exclusion rules + +Exclusion rules are similar to segmentation rules, but are intended to determine +if a context should even be considered as something we should include in the experiment +and track events toward. Exclusion means we don't care about the events in relation +to the given context. + +These examples exclude all users named `'Richard'`, *and* any account +older than 2 weeks old. Not only are they given the control behavior - which could +be nothing - but no events are tracked in these cases as well. + +```ruby +class PillColorExperiment < ApplicationExperiment + # ...registered behaviors + + exclude :old_account?, ->{ context.actor.first_name == 'Richard' } + + private + + def old_account? + context.actor.created_at < 2.weeks.ago + end +end +``` + +You may also need to check exclusion in custom tracking logic by calling `should_track?`: + +```ruby +class PillColorExperiment < ApplicationExperiment + # ...registered behaviors + + def expensive_tracking_logic + return unless should_track? + + track(:my_event, value: expensive_method_call) + end +end +``` + +### Tracking events + +One of the most important aspects of experiments is gathering data and reporting on +it. You can use the `track` method to track events across an experimental implementation. +You can track events consistently to an experiment if you provide the same context between +calls to your experiment. If you do not yet understand context, you should read +about contexts now. + +We can assume we run the experiment in one or a few places, but +track events potentially in many places. The tracking call remains the same, with +the arguments you would normally use when +[tracking events using snowplow](../snowplow/index.md). The easiest example +of tracking an event in Ruby would be: + +```ruby +experiment(:pill_color, actor: current_user).track(:clicked) +``` + +When you run an experiment with any of the examples so far, an `:assignment` event +is tracked automatically by default. All events that are tracked from an +experiment have a special +[experiment context](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-3) +added to the event. This can be used - typically by the data team - to create a connection +between the events on a given experiment. + +If our current user hasn't encountered the experiment yet (meaning where the experiment +is run), and we track an event for them, they are assigned a variant and see +that variant if they ever encountered the experiment later, when an `:assignment` +event would be tracked at that time for them. + +NOTE: +GitLab tries to be sensitive and respectful of our customers regarding tracking, +so our experimentation library allows us to implement an experiment without ever tracking identifying +IDs. It's not always possible, though, based on experiment reporting requirements. +You may be asked from time to time to track a specific record ID in experiments. +The approach is largely up to the PM and engineer creating the implementation. +No recommendations are provided here at this time. + +## Experiments in the client layer + +Any experiment that's been run in the request lifecycle surfaces in `window.gl.experiments`, +and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-3) +so it can be used when resolving experimentation in the client layer. + +Given that we've defined a class for our experiment, and have defined the variants for it, we can publish that experiment in a couple ways. + +The first way is simply by running the experiment. Assuming the experiment has been run, it will surface in the client layer without having to do anything special. + +The second way doesn't run the experiment and is intended to be used if the experiment only needs to surface in the client layer. To accomplish this we can simply `.publish` the experiment. This won't run any logic, but does surface the experiment details in the client layer so they can be utilized there. + +An example might be to publish an experiment in a `before_action` in a controller. Assuming we've defined the `PillColorExperiment` class, like we have above, we can surface it to the client by publishing it instead of running it: + +```ruby +before_action -> { experiment(:pill_color).publish }, only: [:show] +``` + +You can then see this surface in the JavaScript console: + +```javascript +window.gl.experiments // => { pill_color: { excluded: false, experiment: "pill_color", key: "ca63ac02", variant: "candidate" } } +``` + +### Using experiments in Vue + +With the `gitlab-experiment` component, you can define slots that match the name of the +variants pushed to `window.gl.experiments`. + +We can make use of the named slots in the Vue component, that match the behaviors defined in : + +```vue +<script> +import GitlabExperiment from '~/experimentation/components/gitlab_experiment.vue'; + +export default { + components: { GitlabExperiment } +} +</script> + +<template> + <gitlab-experiment name="pill_color"> + <template #control> + <button class="bg-default">Click default button</button> + </template> + + <template #red> + <button class="bg-red">Click red button</button> + </template> + + <template #blue> + <button class="bg-blue">Click blue button</button> + </template> + </gitlab-experiment> +</template> +``` + +NOTE: +When there is no experiment data in the `window.gl.experiments` object for the given experiment name, the `control` slot will be used, if it exists. diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index f7af1113b6e..b140cce34fc 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -6,47 +6,46 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Experiment Guide -Experiments can be conducted by any GitLab team, most often the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). Experiments are not tied to releases because they primarily target GitLab.com. - -Experiments are run as an A/B/n test, and are behind an [experiment feature flag](../feature_flags/#experiment-type) to turn the test on or off. Based on the data the experiment generates, the team decides if the experiment had a positive impact and should be made the new default, or rolled back. - -## Experiment rollout issue - -Each experiment should have an [experiment rollout](https://gitlab.com/groups/gitlab-org/-/boards/1352542) issue to track the experiment from rollout through to cleanup and removal. -The rollout issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. -When an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). -After the deadline, the issue needs to be resolved and either: - -- It was successful and the experiment becomes the new default. -- It was not successful and all code related to the experiment is removed. - -In either case, an outcome of the experiment should be posted to the issue with the reasoning for the decision. - -## Code reviews - -Experiments' code quality can fail our standards for several reasons. These -reasons can include not being added to the codebase for a long time, or because -of fast iteration to retrieve data. However, having the experiment run (or not -run) shouldn't impact GitLab availability. To avoid or identify issues, -experiments are initially deployed to a small number of users. Regardless, -experiments still need tests. - -Experiments must have corresponding [frontend or feature tests](../testing_guide/index.md) to ensure they -exist in the application. These tests should help prevent the experiment code from -being removed before the [experiment cleanup process](https://about.gitlab.com/handbook/engineering/development/growth/experimentation/#experiment-cleanup-issue) starts. - -If, as a reviewer or maintainer, you find code that would usually fail review -but is acceptable for now, mention your concerns with a note that there's no -need to change the code. The author can then add a comment to this piece of code -and link to the issue that resolves the experiment. The author or reviewer can add a link to this concern in the -experiment rollout issue under the `Experiment Successful Cleanup Concerns` section of the description. -If the experiment is successful and becomes part of the product, any items that appear under this section will be addressed. +Experiments can be conducted by any GitLab team, most often the teams from the +[Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). +Experiments are not tied to releases because they primarily target GitLab.com. + +Experiments are run as an A/B/n test, and are behind an [experiment feature flag](../feature_flags/#experiment-type) +to turn the test on or off. Based on the data the experiment generates, the team decides +if the experiment had a positive impact and should be made the new default, or rolled back. + +Experiments in GitLab are tightly coupled with the concepts provided by +[Feature flags in development of GitLab](../feature_flags/index.md). You're strongly encouraged +to read and understand the [Feature flags in development of GitLab](../feature_flags/index.md) +portion of the documentation before considering running experiments. Experiments add additional +concepts which may seem confusing or advanced without understanding the underpinnings of how GitLab +uses feature flags in development. One concept: experiments can be run with multiple variants, +which are sometimes referred to as A/B/n tests. + +We use the [`gitlab-experiment` gem](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment), +sometimes referred to as GLEX, to run our experiments. The gem exists in a separate repository +so it can be shared across any GitLab property that uses Ruby. You should feel comfortable reading +the documentation on that project if you want to dig into more advanced topics or open issues. Be +aware that the documentation there reflects what's in the main branch and may not be the same as +the version being used within GitLab. + +## Glossary of terms + +To ensure a shared language, you should understand these fundamental terms we use +when communicating about experiments: + +- `experiment`: Any deviation of code paths we want to run at some times, but not others. +- `context`: A consistent experience we provide in an experiment. +- `control`: The default, or "original" code path. +- `candidate`: Defines an experiment with only one code path. +- `variant(s)`: Defines an experiment with multiple code paths. +- `behaviors`: Used to reference all possible code paths of an experiment, including the control. ## Implementing an experiment [`GLEX`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) - or `Gitlab::Experiment`, the `gitlab-experiment` gem - is the preferred option for implementing an experiment in GitLab. -For more information, see [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md). +For more information, see [Implementing an A/B/n experiment using GLEX](implementing_experiments.md). This uses [experiment](../feature_flags/index.md#experiment-type) feature flags. @@ -64,15 +63,3 @@ We recommend the following workflow: 1. **If the experiment is a success**, designers add the new icon or illustration to the Pajamas UI kit as part of the cleanup process. Engineers can then add it to the [SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) and modify the implementation based on the [Frontend Development Guidelines](../fe_guide/icons.md#usage-in-hamlrails-2). - -## Turn off all experiments - -When there is a case on GitLab.com (SaaS) that necessitates turning off all experiments, we have this control. - -You can toggle experiments on SaaS on and off using the `gitlab_experiment` [feature flag](../feature_flags). - -This can be done via chatops: - -- [disable](../feature_flags/controls.md#disabling-feature-flags): `/chatops run feature set gitlab_experiment false` -- [enable](../feature_flags/controls.md#process): `/chatops run feature delete gitlab_experiment` - - This allows the `default_enabled` [value of true in the yml](https://gitlab.com/gitlab-org/gitlab/-/blob/016430f6751b0c34abb24f74608c80a1a8268f20/config/feature_flags/ops/gitlab_experiment.yml#L8) to be honored. diff --git a/doc/development/experiment_guide/testing_experiments.md b/doc/development/experiment_guide/testing_experiments.md new file mode 100644 index 00000000000..c51c3c7cb11 --- /dev/null +++ b/doc/development/experiment_guide/testing_experiments.md @@ -0,0 +1,150 @@ +--- +stage: Growth +group: Activation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Testing experiments + +## Testing experiments with RSpec + +In the course of working with experiments, you'll probably want to utilize the RSpec +tooling that's built in. This happens automatically for files in `spec/experiments`, but +for other files and specs you want to include it in, you can specify the `:experiment` type: + +```ruby +it "tests experiments nicely", :experiment do +end +``` + +### Stub helpers + +You can stub experiments using `stub_experiments`. Pass it a hash using experiment +names as the keys, and the variants you want each to resolve to, as the values: + +```ruby +# Ensures the experiments named `:example` & `:example2` are both "enabled" and +# that each will resolve to the given variant (`:my_variant` and `:control` +# respectively). +stub_experiments(example: :my_variant, example2: :control) + +experiment(:example) do |e| + e.enabled? # => true + e.assigned.name # => 'my_variant' +end + +experiment(:example2) do |e| + e.enabled? # => true + e.assigned.name # => 'control' +end +``` + +### Exclusion, segmentation, and behavior matchers + +You can also test things like the registered behaviors, the exclusions, and +segmentations using the matchers. + +```ruby +class ExampleExperiment < ApplicationExperiment + control { } + candidate { '_candidate_' } + + exclude { context.actor.first_name == 'Richard' } + segment(variant: :candidate) { context.actor.username == 'jejacks0n' } +end + +excluded = double(username: 'rdiggitty', first_name: 'Richard') +segmented = double(username: 'jejacks0n', first_name: 'Jeremy') + +# register_behavior matcher +expect(experiment(:example)).to register_behavior(:control) +expect(experiment(:example)).to register_behavior(:candidate).with('_candidate_') + +# exclude matcher +expect(experiment(:example)).to exclude(actor: excluded) +expect(experiment(:example)).not_to exclude(actor: segmented) + +# segment matcher +expect(experiment(:example)).to segment(actor: segmented).into(:candidate) +expect(experiment(:example)).not_to segment(actor: excluded) +``` + +### Tracking matcher + +Tracking events is a major aspect of experimentation. We try +to provide a flexible way to ensure your tracking calls are covered. + +You can do this on the instance level or at an "any instance" level: + +```ruby +subject = experiment(:example) + +expect(subject).to track(:my_event) + +subject.track(:my_event) +``` + +You can use the `on_next_instance` chain method to specify that it will happen +on the next instance of the experiment. This helps you if you're calling +`experiment(:example).track` downstream: + +```ruby +expect(experiment(:example)).to track(:my_event).on_next_instance + +experiment(:example).track(:my_event) +``` + +A full example of the methods you can chain onto the `track` matcher: + +```ruby +expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_') + .on_next_instance + .with_context(foo: :bar) + .for(:variant_name) + +experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_') +``` + +## Test with Jest + +### Stub Helpers + +You can stub experiments using the `stubExperiments` helper defined in `spec/frontend/__helpers__/experimentation_helper.js`. + +```javascript +import { stubExperiments } from 'helpers/experimentation_helper'; +import { getExperimentData } from '~/experimentation/utils'; + +describe('when my_experiment is enabled', () => { + beforeEach(() => { + stubExperiments({ my_experiment: 'candidate' }); + }); + + it('sets the correct data', () => { + expect(getExperimentData('my_experiment')).toEqual({ experiment: 'my_experiment', variant: 'candidate' }); + }); +}); +``` + +NOTE: +This method of stubbing in Jest specs will not automatically un-stub itself at the end of the test. We merge our stubbed experiment in with all the other global data in `window.gl`. If you need to remove the stubbed experiment(s) after your test or ensure a clean global object before your test, you'll need to manage the global object directly yourself: + +```javascript +describe('tests that care about global state', () => { + const originalObjects = []; + + beforeEach(() => { + // For backwards compatibility for now, we're using both window.gon & window.gl + originalObjects.push(window.gon, window.gl); + }); + + afterEach(() => { + [window.gon, window.gl] = originalObjects; + }); + + it('stubs experiment in fresh global state', () => { + stubExperiment({ my_experiment: 'candidate' }); + // ... + }); +}) +``` |