GitLab Experiment
=================

<img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">

Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) docs if you're curious about how we use this gem internally.

This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.

When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.

- `experiment` is any deviation of code paths we want to run sometimes and not others.
- `context` is used to identify a consistent experience we'll provide in an experiment.
- `control` is the default, or "original" code path.
- `candidate` defines that there's one experimental code path.
- `variant(s)` is used when more than one experimental code path exists.

Candidate and variant are the same concept, but simplify how we speak about experimental paths -- this concept is also widely referred to as the "experiment group".<br clear="all">

[[_TOC_]]

## Installation

Add the gem to your Gemfile and then `bundle install`.

```ruby
gem 'gitlab-experiment'
```

If you're using Rails, you can install the initializer which provides basic configuration, documentation, and the base experiment class that all your experiments can inherit from.

```shell
$ rails generate gitlab:experiment:install
```

## Implementing an experiment

For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.

In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.

The behavior will be the same, but the interface will be different and may involve more or fewer steps.

Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.

We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.

When you implement an experiment you'll need to provide a name, and a context. The name can show up in tracking calls, and potentially other aspects. The context determines the variant assigned, and should be consistent between calls. We'll discuss migrating context in later examples.

A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching. This is how an experiment remains "sticky" to a given context.

Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.

```ruby
class SubscriptionsController < ApplicationController
  def show
    experiment(:notification_toggle, actor: user) do |e|
      e.use { render_toggle } # control
      e.try { render_button } # candidate
    end
  end
end
```

You can define the experiment using simple control/candidate paths, or provide named variants.

Handling [multivariate](https://en.wikipedia.org/wiki/Multivariate_statistics) experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.

```ruby
experiment(:notification_toggle, actor: user) do |e|
  e.use { render_toggle } # control
  e.try(:variant_one) { render_button(confirmation: true) }
  e.try(:variant_two) { render_button(confirmation: false) }
end
```

You can specify what the experiment should be "sticky" to by providing a `:sticky_to` option. By default this will be the entire context provided, but this can be overridden manually if needed.

```ruby
experiment(:notification_toggle, actor: user, project: project, sticky_to: project) #...
```

Understanding how an experiment can change behavior is important in evaluating its performance.

To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.

```ruby
experiment(:notification_toggle, actor: user).track(:clicked_button)
```

### Custom experiments

You can craft more advanced behaviors by defining custom experiments at a higher level. To do this you can define a class that inherits from `ApplicationExperiment` (or `Gitlab::Experiment`).

Let's say you want to do more advanced segmentation, or provide default behavior for the variants on the experiment we've already outlined above -- that way if the variants aren't defined in the block at the time the experiment is run, these methods will be used. 

You can generate a custom experiment by running:

```shell
$ rails generate gitlab:experiment NotificationToggle control candidate
``` 

This will generate a file in `app/experiments/notification_toggle_experiment.rb`, as well as a test file for you to further expand on.

Here are some examples of what you can introduce once you have a custom experiment defined.

```ruby
class NotificationToggleExperiment < ApplicationExperiment
  # Exclude any users that aren't me.
  exclude :users_named_richard

  # Segment any account older than 2 weeks into the candidate, without
  # asking the variant resolver to decide which variant to provide.
  segment :old_account?, variant: :candidate

  # Define the default control behavior, which can be overridden at
  # experiment time.
  def control_behavior
    # render_toggle
  end

  # Define the default candidate behavior, which can be overridden
  # at experiment time.
  def candidate_behavior
    # render_button
  end

  private

  def users_named_richard
    context.actor.first_name == 'Richard'
  end

  def old_account?
    context.actor.created_at < 2.weeks.ago
  end
end

# The class will be looked up based on the experiment name provided. 
exp = experiment(:notification_toggle, actor: user)
exp # => instance of NotificationToggleExperiment

# Run the experiment -- returning the result.
exp.run

# Track an event on the experiment we've defined.
exp.track(:clicked_button)
```

You can now also do things very similar to the simple examples and override the default variant behaviors defined in the custom experiment -- keeping in mind that this should be carefully considered within the scope of your experiment.

```ruby
experiment(:notification_toggle, actor: user) do |e|
  e.use { render_special_toggle } # override default control behavior
end
```

<details>
  <summary>You can also specify the variant to use for segmentation...</summary>

Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration.

Any time a specific variant is provided (including `:control`) it will be cached for that context, if caching is enabled. 

```ruby
experiment(:notification_toggle, :no_interface, actor: user) do |e|
  e.use { render_toggle } # control
  e.try { render_button } # candidate
  e.try(:no_interface) { no_interface! } # no_interface variant
end
```

Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it.

```ruby
experiment(:notification_toggle, actor: user) do |e|
  # Variant selection must be done before calling run or track.
  e.variant(:no_interface) # set the variant
  # ...
end
```

Or it can be specified in the call to run if you call it from within the block.

```ruby
experiment(:notification_toggle, actor: user) do |e|
  # ...
  # Variant selection can be specified when calling run.
  e.run(:no_interface)
end
```

</details>

### Segmentation rules

This library comes with the capability to segment contexts into a specific variant, before asking the variant resolver which variant to provide.

Segmentation can be achieved by using a custom experiment class and specifying the segmentation rules at a class level.

```ruby
class ExampleExperiment < ApplicationExperiment
  segment(variant: :variant_one) { context.actor.first_name == 'Richard' }
  segment :old_account?, variant: :variant_two
  
  private

  def old_account?
    context.actor.created_at < 2.weeks.ago
  end
end
```

In the previous examples, any user named `'Richard'` would always receive the experience defined in "variant_one". As well, any account older than 2 weeks old would get the alternate experience defined in "variant_two".

When an experiment is run, the segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped.

This means that any user named `'Richard'`, regardless of account age, will always be provided the experience as defined in "variant_one". If you wanted the opposite logic, you can flip the order.

### Exclusion rules

Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.

```ruby
class ExampleExperiment < ApplicationExperiment
  exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
  
  private

  def old_account? 
    context.actor.created_at < 2.weeks.ago
  end
end
```

The previous examples will exclude all users named `'Richard'` as well as any account older than 2 weeks old. Not only will they be given the control behavior, but no events will be tracked in these cases as well. 

You may need to check exclusion in custom tracking logic by calling `should_track?`:

```ruby
def expensive_tracking_logic
  return unless should_track?
  
  track(:my_event, value: expensive_method_call)
end
```

Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can implement.

### Return value

By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.

```ruby
experiment(:notification_toggle) do |e|
  e.use { 'A' }
  e.try { 'B' }
  e.run
end # => 'A'
```

### Including the DSL

By default, `Gitlab::Experiment` injects itself into the controller and view layers. This exposes the `experiment` method application wide in those layers.

Some experiments may extend outside of those layers, so you may want to include it elsewhere. For instance in a mailer, service object, background job, or similar.

Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.

```ruby
class WelcomeMailer < ApplicationMailer
  include Gitlab::Experiment::Dsl # include the `experiment` method

  def welcome
    @user = params[:user]

    ex = experiment(:project_suggestions, actor: @user) do |e|
      e.use { 'welcome' }
      e.try { 'welcome_with_project_suggestions' }
    end

    mail(to: @user.email, subject: 'Welcome!', template: ex.run)
  end
end
```

### Context migrations

There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.

Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.

In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.

```ruby
# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
experiment(:example, actor: project, version: 2, migrated_with: { version: 1 })
```

You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.

If you wanted to introduce a `version` to your context, provide the full previous context.

```ruby
# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
experiment(:example, actor: project, version: 1, migrated_from: { actor: project })
```

This can impact an experience if you:

1. haven't implemented the concept of migrations in your variant resolver
1. haven't enabled a reasonable caching mechanism

### When there isn't an actor (cookie fallback)

When there isn't an identifying key in the context (this is `actor` by default), we fall back to cookies to provide a consistent experience for the client viewing them.

Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.

This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.

To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.

You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.

```ruby
experiment(:example, actor: user, request: request)
```

The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.

```ruby
# actor is not present, so no cookie is set
experiment(:example, project: project)

# actor is present and is nil, so the cookie is set and used
experiment(:example, actor: nil, project: project)

# actor is present and set to a value, so no cookie is set
experiment(:example, actor: user, project: project)
```

For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.

## How it works

The way the gem works is best described using the following decision tree illustration. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and the context provided.
 
```mermaid
graph TD
    GP[General Pool/Population] --> Enabled?
    Enabled? -->|Yes| Cached?[Cached? / Pre-segmented?]
    Enabled? -->|No| Excluded[Control / No Tracking]
    Cached? -->|No| Excluded?
    Cached? -->|Yes| Cached[Cached Value]
    Excluded? -->|Yes / Cached| Excluded
    Excluded? -->|No| Segmented?
    Segmented? -->|Yes / Cached| VariantA
    Segmented? -->|No| Included?[Experiment Group?]
    Included? -->|Yes| Rollout
    Included? -->|No| Control
    Rollout -->|Cached| VariantA
    Rollout -->|Cached| VariantB
    Rollout -->|Cached| VariantC

classDef included fill:#380d75,color:#ffffff,stroke:none
classDef excluded fill:#fca121,stroke:none
classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
classDef default fill:#fff,stroke:#6e49cb

class VariantA,VariantB,VariantC included
class Control,Excluded excluded
class Cached cached
```

## Configuration

This gem needs to be configured before being used in a meaningful way.

The default configuration will always render the control, so it's important to configure your own logic for resolving variants.

Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.

```ruby
Gitlab::Experiment.configure do |config|
  # The block here is evaluated within the scope of the experiment instance,
  # which is why we are able to access things like name and context.
  config.variant_resolver = lambda do |requested_variant|
    # Return the requested variant if a specific one has been provided in code.
    return requested_variant unless requested_variant.nil?

    # Ask Unleash to determine the variant, given the context we've built,
    # using the control as the fallback.
    fallback = Unleash::Variant.new(name: 'control', enabled: true)
    UNLEASH.get_variant(name, context.value, fallback)
  end
end
```

More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt).

### Client layer / JavaScript

This library doesn't attempt to provide any logic for the client layer.

Instead it allows you to do this yourself in configuration. Using [Gon](https://github.com/gazay/gon) to publish your experiment information to the client layer is pretty simple.

```ruby
Gitlab::Experiment.configure do |config|
  config.publishing_behavior = lambda do |_result|
    # Push the experiment knowledge into the front end. The signature contains
    # the context key, and the variant that has been determined.
    Gon.push({ experiment: { name => signature } }, true)
  end
end
```

In the client you can now access `window.gon.experiment.notificationToggle`.

### Caching

Caching can be enabled in configuration, and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.

It's important to understand that using caching can drastically change or override your rollout strategy logic.

```ruby
Gitlab::Experiment.configure do |config|
  config.cache = Rails.cache
end
```

### Middleware

There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you won't be able to implement tracking. For these cases, gitlab-experiment comes with middleware that will redirect to a given URL while also tracking that the URL was visited.

In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`. If this path is empty the middleware won't be mounted at all.

Once mounted, the redirect URLs can be generated using the Rails route helpers. If not using Rails, mount the middleware and generate these URLs yourself.

```ruby
Gitlab::Experiment.configure do |config|
  config.mount_at = '/experiment'
end

ex = experiment(:example, foo: :bar)

# using rails path/url helpers 
experiment_redirect_path(ex, 'https//docs.gitlab.com/') # => /experiment/example:[context_key]?https//docs.gitlab.com/

# manually
"#{Gitlab::Experiment.configure.mount_at}/#{ex.to_param}?https//docs.gitlab.com/"
```

URLS that match the base path will be handled by the middleware and will redirect to the provided redirect path.

## Testing (rspec support)

This gem comes with some rspec helpers and custom matchers. These are in flux at the time of writing.

First, require the rspec support file:

```ruby
require 'gitlab/experiment/rspec'
```

This mixes in some of the basics, but the matchers and other aspects need to be included. This happens automatically for files in `spec/experiments`, but for other files and specs you want to include it in, you can specify the `:experiment` type:

```ruby
it "tests", :experiment do
end
```

### Stub helpers

You can stub experiments using `stub_experiments`. Pass it a hash using experiment names as the keys and the variants you want each to resolve to as the values:

```ruby
# Ensures the experiments named `:example` & `:example2` are both
# "enabled" and that each will resolve to the given variant
# (`:my_variant` & `:control` respectively).
stub_experiments(example: :my_variant, example2: :control)

experiment(:example) do |e|
  e.enabled? # => true
  e.variant.name # => 'my_variant'
end

experiment(:example2) do |e|
  e.enabled? # => true
  e.variant.name # => 'control'
end
```

### Exclusion and segmentation matchers

You can also easily test the exclusion and segmentation matchers.

```ruby
class ExampleExperiment < ApplicationExperiment
  exclude { context.actor.first_name == 'Richard' }
  segment(variant: :candidate) { context.actor.username == 'jejacks0n' }
end

excluded = double(username: 'rdiggitty', first_name: 'Richard')
segmented = double(username: 'jejacks0n', first_name: 'Jeremy')

# exclude matcher
expect(experiment(:example)).to exclude(actor: excluded)
expect(experiment(:example)).not_to exclude(actor: segmented)

# segment matcher
expect(experiment(:example)).to segment(actor: segmented).into(:candidate)
expect(experiment(:example)).not_to segment(actor: excluded)
```

### Tracking matcher

Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.

You can do this on the instance level or at an "any instance" level. At an instance level this is pretty straight forward:

```ruby
subject = experiment(:example)

expect(subject).to track(:my_event)

subject.track(:my_event)
```

You can use the `on_any_instance` chain method to specify that it could happen on any instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream:

```ruby
expect(experiment(:example)).to track(:my_event).on_any_instance

experiment(:example).track(:my_event)
```

And here's a full example of the methods that can be chained onto the `track` matcher:

```ruby
expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_')
  .on_any_instance
  .with_context(foo: :bar)
  .for(:variant_name)

experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_')
```

## Tracking, anonymity and GDPR

We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).

We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.

This library attempts to be non-user-centric, in that a context can contain things like a user or a project.

If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.

Each of these approaches could be desirable given the objectives of your experiment.

## Development

After checking out the repo, run `bundle install` to install dependencies.
Then, run `bundle exec rake` to run the tests. You can also run `bundle exec pry` for an
interactive prompt that will allow you to experiment.

## Contributing

Bug reports and merge requests are welcome on GitLab at
https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be a
safe, welcoming space for collaboration, and contributors are expected to adhere
to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.

## Release process

Please refer to the [Release Process](docs/release_process.md).

## License

The gem is available as open source under the terms of the
[MIT License](http://opensource.org/licenses/MIT).

## Code of conduct

Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers,
chat rooms and mailing lists is expected to follow the
[code of conduct](CODE_OF_CONDUCT.md).

***Make code not war***
