diff options
Diffstat (limited to 'doc/development')
-rw-r--r-- | doc/development/README.md | 2 | ||||
-rw-r--r-- | doc/development/architecture.md | 8 | ||||
-rw-r--r-- | doc/development/feature_flags.md | 7 | ||||
-rw-r--r-- | doc/development/i18n_guide.md | 3 | ||||
-rw-r--r-- | doc/development/serializing_data.md | 84 |
5 files changed, 99 insertions, 5 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index 934c6849ff9..af4131c4a8f 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -42,6 +42,7 @@ - [Sidekiq debugging](sidekiq_debugging.md) - [Object state models](object_state_models.md) - [Building a package for testing purposes](build_test_package.md) +- [Manage feature flags](feature_flags.md) ## Databases @@ -49,6 +50,7 @@ - [Adding database indexes](adding_database_indexes.md) - [Post Deployment Migrations](post_deployment_migrations.md) - [Foreign Keys & Associations](foreign_keys.md) +- [Serializing Data](serializing_data.md) ## i18n diff --git a/doc/development/architecture.md b/doc/development/architecture.md index b36fd52603b..acd5e3c2093 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -12,7 +12,7 @@ Both EE and CE require some add-on components called gitlab-shell and Gitaly. Th You can imagine GitLab as a physical office. -**The repositories** are the goods GitLab handling. +**The repositories** are the goods GitLab handles. They can be stored in a warehouse. This can be either a hard disk, or something more complex, such as a NFS filesystem; @@ -54,7 +54,7 @@ To serve repositories over SSH there's an add-on application called gitlab-shell ### Components - +<img src="https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/pub?w=987&h=797"> _[edit diagram (for GitLab team members only)](https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/edit)_ @@ -66,7 +66,9 @@ When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to reso The add-on component gitlab-shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. gitlab-shell accesses the bare repositories through Gitaly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. gitlab-shell queries the GitLab API to determine authorization and access. -Gitaly executes git operations from gitlab-shell and Workhorse, and provides an API to the GitLab web app to get attributes from git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files) +Gitaly executes git operations from gitlab-shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). + +You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/infrastructure/production-architecture/). ### Installation Folder Summary diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md new file mode 100644 index 00000000000..5c6316b9ac6 --- /dev/null +++ b/doc/development/feature_flags.md @@ -0,0 +1,7 @@ +# Manage feature flags + +Starting from GitLab 9.3 we support feature flags via +[Flipper](https://github.com/jnunemaker/flipper/). You should use the `Feature` +class (defined in `lib/feature.rb`) in your code to get, set and list feature +flags. During runtime you can set the values for the gates via the +[admin API](../api/features.md). diff --git a/doc/development/i18n_guide.md b/doc/development/i18n_guide.md index 735345bd126..bfb0779fbfa 100644 --- a/doc/development/i18n_guide.md +++ b/doc/development/i18n_guide.md @@ -233,8 +233,7 @@ Let's suppose you want to add translations for a new language, let's say French. containing the translations: ```sh - bundle exec rake gettext:pack - bundle exec rake gettext:po_to_json + bundle exec rake gettext:compile ``` 1. In order to see the translated content we need to change our preferred language diff --git a/doc/development/serializing_data.md b/doc/development/serializing_data.md new file mode 100644 index 00000000000..2b56f48bc44 --- /dev/null +++ b/doc/development/serializing_data.md @@ -0,0 +1,84 @@ +# Serializing Data + +**Summary:** don't store serialized data in the database, use separate columns +and/or tables instead. + +Rails makes it possible to store serialized data in JSON, YAML or other formats. +Such a field can be defined as follows: + +```ruby +class Issue < ActiveRecord::Model + serialize :custom_fields +end +``` + +While it may be tempting to store serialized data in the database there are many +problems with this. This document will outline these problems and provide an +alternative. + +## Serialized Data Is Less Powerful + +When using a relational database you have the ability to query individual +fields, change the schema, index data and so forth. When you use serialized data +all of that becomes either very difficult or downright impossible. While +PostgreSQL does offer the ability to query JSON fields it is mostly meant for +very specialized use cases, and not for more general use. If you use YAML in +turn there's no way to query the data at all. + +## Waste Of Space + +Storing serialized data such as JSON or YAML will end up wasting a lot of space. +This is because these formats often include additional characters (e.g. double +quotes or newlines) besides the data that you are storing. + +## Difficult To Manage + +There comes a time where you will need to add a new field to the serialized +data, or change an existing one. Using serialized data this becomes difficult +and very time consuming as the only way of doing so is to re-write all the +stored values. To do so you would have to: + +1. Retrieve the data +1. Parse it into a Ruby structure +1. Mutate it +1. Serialize it back to a String +1. Store it in the database + +On the other hand, if one were to use regular columns adding a column would be +as easy as this: + +```sql +ALTER TABLE table_name ADD COLUMN column_name type; +``` + +Such a query would take very little to no time and would immediately apply to +all rows, without having to re-write large JSON or YAML structures. + +Finally, there comes a time when the JSON or YAML structure is no longer +sufficient and you need to migrate away from it. When storing only a few rows +this may not be a problem, but when storing millions of rows such a migration +can easily take hours or even days to complete. + +## Relational Databases Are Not Document Stores + +When storing data as JSON or YAML you're essentially using your database as if +it were a document store (e.g. MongoDB), except you're not using any of the +powerful features provided by a typical RDBMS _nor_ are you using any of the +features provided by a typical document store (e.g. the ability to index fields +of documents with variable fields). In other words, it's a waste. + +## Consistent Fields + +One argument sometimes made in favour of serialized data is having to store +widely varying fields and values. Sometimes this is truly the case, and then +perhaps it might make sense to use serialized data. However, in 99% of the cases +the fields and types stored tend to be the same for every row. Even if there is +a slight difference you can still use separate columns and just not set the ones +you don't need. + +## The Solution + +The solution is very simple: just use separate columns and/or separate tables. +This will allow you to use all the features provided by your database, it will +make it easier to manage and migrate the data, you'll conserve space, you can +index the data efficiently and so forth. |