From 03a40a7ee1262d5b68e5f7aba26077dc3bef6b33 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Wed, 16 Dec 2015 11:51:37 +0800 Subject: Avoid allocations in Ability class. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It won't change anything after they are first invoke, so add method cache to avoid allocations and avoid GC. Benchmarks: ``` Calculating ------------------------------------- project_guest_rules without method cache 79.352k i/100ms project_guest_rules with method cache 93.634k i/100ms ------------------------------------------------- project_guest_rules without method cache 2.865M (±32.5%) i/s - 11.982M project_guest_rules with method cache 4.419M (± 7.4%) i/s - 22.004M Comparison: project_guest_rules with method cache: 4418908.0 i/s project_guest_rules without method cache: 2864514.0 i/s - 1.54x slower Calculating ------------------------------------- project_report_rules without method cache 53.126k i/100ms project_report_rules with method cache 97.473k i/100ms ------------------------------------------------- project_report_rules without method cache 1.093M (±36.5%) i/s - 4.675M project_report_rules with method cache 4.420M (± 7.2%) i/s - 22.029M Comparison: project_report_rules with method cache: 4420054.3 i/s project_report_rules without method cache: 1092509.6 i/s - 4.05x slower ``` https://gist.github.com/huacnlee/b04788ae6df42fe769e4 --- app/models/ability.rb | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/app/models/ability.rb b/app/models/ability.rb index cd5ae0fb0fd..1b3ee757040 100644 --- a/app/models/ability.rb +++ b/app/models/ability.rb @@ -132,14 +132,14 @@ class Ability end def public_project_rules - project_guest_rules + [ + @public_project_rules ||= project_guest_rules + [ :download_code, :fork_project ] end def project_guest_rules - [ + @project_guest_rules ||= [ :read_project, :read_wiki, :read_issue, @@ -157,7 +157,7 @@ class Ability end def project_report_rules - project_guest_rules + [ + @project_report_rules ||= project_guest_rules + [ :create_commit_status, :read_commit_statuses, :download_code, @@ -170,7 +170,7 @@ class Ability end def project_dev_rules - project_report_rules + [ + @project_dev_rules ||= project_report_rules + [ :admin_merge_request, :create_merge_request, :create_wiki, @@ -181,7 +181,7 @@ class Ability end def project_archived_rules - [ + @project_archived_rules ||= [ :create_merge_request, :push_code, :push_code_to_protected_branches, @@ -191,7 +191,7 @@ class Ability end def project_master_rules - project_dev_rules + [ + @project_master_rules ||= project_dev_rules + [ :push_code_to_protected_branches, :update_project_snippet, :update_merge_request, @@ -206,7 +206,7 @@ class Ability end def project_admin_rules - project_master_rules + [ + @project_admin_rules ||= project_master_rules + [ :change_namespace, :change_visibility_level, :rename_project, @@ -332,7 +332,7 @@ class Ability end if snippet.public? || snippet.internal? - rules << :read_personal_snippet + rules << :read_personal_snippet end rules -- cgit v1.2.1 From 141e946c3da97c7af02aaca5324c6e4ce7362a04 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Wed, 9 Dec 2015 16:45:51 +0100 Subject: Storing of application metrics in InfluxDB This adds the ability to write application metrics (e.g. SQL timings) to InfluxDB. These metrics can in turn be visualized using Grafana, or really anything else that can read from InfluxDB. These metrics can be used to track application performance over time, between different Ruby versions, different GitLab versions, etc. == Transaction Metrics Currently the following is tracked on a per transaction basis (a transaction is a Rails request or a single Sidekiq job): * Timings per query along with the raw (obfuscated) SQL and information about what file the query originated from. * Timings per view along with the path of the view and information about what file triggered the rendering process. * The duration of a request itself along with the controller/worker class and method name. * The duration of any instrumented method calls (more below). == Sampled Metrics Certain metrics can't be directly associated with a transaction. For example, a process' total memory usage is unrelated to any running transactions. While a transaction can result in the memory usage going up there's no accurate way to determine what transaction is to blame, this becomes especially problematic in multi-threaded environments. To solve this problem there's a separate thread that takes samples at a fixed interval. This thread (using the class Gitlab::Metrics::Sampler) currently tracks the following: * The process' total memory usage. * The number of file descriptors opened by the process. * The amount of Ruby objects (using ObjectSpace.count_objects). * GC statistics such as timings, heap slots, etc. The default/current interval is 15 seconds, any smaller interval might put too much pressure on InfluxDB (especially when running dozens of processes). == Method Instrumentation While currently not yet used methods can be instrumented to track how long they take to run. Unlike the likes of New Relic this doesn't require modifying the source code (e.g. including modules), it all happens from the outside. For example, to track `User.by_login` we'd add the following code somewhere in an initializer: Gitlab::Metrics::Instrumentation. instrument_method(User, :by_login) to instead instrument an instance method: Gitlab::Metrics::Instrumentation. instrument_instance_method(User, :save) Instrumentation for either all public model methods or a few crucial ones will be added in the near future, I simply haven't gotten to doing so just yet. == Configuration By default metrics are disabled. This means users don't have to bother setting anything up if they don't want to. Metrics can be enabled by editing one's gitlab.yml configuration file (see config/gitlab.yml.example for example settings). == Writing Data To InfluxDB Because InfluxDB is still a fairly young product I expect the worse. Data loss, unexpected reboots, the database not responding, you name it. Because of this data is _not_ written to InfluxDB directly, instead it's queued and processed by Sidekiq. This ensures that users won't notice anything when InfluxDB is giving trouble. The metrics worker can be started in a standalone manner as following: bundle exec sidekiq -q metrics The corresponding class is called MetricsWorker. --- Gemfile | 6 ++ Gemfile.lock | 30 ++++--- Procfile | 2 +- app/workers/metrics_worker.rb | 29 +++++++ config/gitlab.yml.example | 17 ++++ config/initializers/metrics.rb | 25 ++++++ lib/gitlab/metrics.rb | 52 ++++++++++++ lib/gitlab/metrics/delta.rb | 32 ++++++++ lib/gitlab/metrics/instrumentation.rb | 47 +++++++++++ lib/gitlab/metrics/metric.rb | 34 ++++++++ lib/gitlab/metrics/obfuscated_sql.rb | 39 +++++++++ lib/gitlab/metrics/rack_middleware.rb | 49 ++++++++++++ lib/gitlab/metrics/sampler.rb | 77 ++++++++++++++++++ lib/gitlab/metrics/sidekiq_middleware.rb | 30 +++++++ lib/gitlab/metrics/subscribers/action_view.rb | 48 +++++++++++ lib/gitlab/metrics/subscribers/active_record.rb | 43 ++++++++++ lib/gitlab/metrics/subscribers/method_call.rb | 42 ++++++++++ lib/gitlab/metrics/system.rb | 35 ++++++++ lib/gitlab/metrics/transaction.rb | 66 ++++++++++++++++ spec/lib/gitlab/metrics/delta_spec.rb | 16 ++++ spec/lib/gitlab/metrics/instrumentation_spec.rb | 91 +++++++++++++++++++++ spec/lib/gitlab/metrics/metric_spec.rb | 57 ++++++++++++++ spec/lib/gitlab/metrics/obfuscated_sql_spec.rb | 79 +++++++++++++++++++ spec/lib/gitlab/metrics/rack_middleware_spec.rb | 63 +++++++++++++++ spec/lib/gitlab/metrics/sampler_spec.rb | 92 ++++++++++++++++++++++ spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb | 34 ++++++++ .../gitlab/metrics/subscribers/action_view_spec.rb | 32 ++++++++ .../metrics/subscribers/active_record_spec.rb | 31 ++++++++ .../gitlab/metrics/subscribers/method_call_spec.rb | 41 ++++++++++ spec/lib/gitlab/metrics/system_spec.rb | 29 +++++++ spec/lib/gitlab/metrics/transaction_spec.rb | 77 ++++++++++++++++++ spec/lib/gitlab/metrics_spec.rb | 36 +++++++++ 32 files changed, 1368 insertions(+), 13 deletions(-) create mode 100644 app/workers/metrics_worker.rb create mode 100644 config/initializers/metrics.rb create mode 100644 lib/gitlab/metrics.rb create mode 100644 lib/gitlab/metrics/delta.rb create mode 100644 lib/gitlab/metrics/instrumentation.rb create mode 100644 lib/gitlab/metrics/metric.rb create mode 100644 lib/gitlab/metrics/obfuscated_sql.rb create mode 100644 lib/gitlab/metrics/rack_middleware.rb create mode 100644 lib/gitlab/metrics/sampler.rb create mode 100644 lib/gitlab/metrics/sidekiq_middleware.rb create mode 100644 lib/gitlab/metrics/subscribers/action_view.rb create mode 100644 lib/gitlab/metrics/subscribers/active_record.rb create mode 100644 lib/gitlab/metrics/subscribers/method_call.rb create mode 100644 lib/gitlab/metrics/system.rb create mode 100644 lib/gitlab/metrics/transaction.rb create mode 100644 spec/lib/gitlab/metrics/delta_spec.rb create mode 100644 spec/lib/gitlab/metrics/instrumentation_spec.rb create mode 100644 spec/lib/gitlab/metrics/metric_spec.rb create mode 100644 spec/lib/gitlab/metrics/obfuscated_sql_spec.rb create mode 100644 spec/lib/gitlab/metrics/rack_middleware_spec.rb create mode 100644 spec/lib/gitlab/metrics/sampler_spec.rb create mode 100644 spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb create mode 100644 spec/lib/gitlab/metrics/subscribers/action_view_spec.rb create mode 100644 spec/lib/gitlab/metrics/subscribers/active_record_spec.rb create mode 100644 spec/lib/gitlab/metrics/subscribers/method_call_spec.rb create mode 100644 spec/lib/gitlab/metrics/system_spec.rb create mode 100644 spec/lib/gitlab/metrics/transaction_spec.rb create mode 100644 spec/lib/gitlab/metrics_spec.rb diff --git a/Gemfile b/Gemfile index b23e274081b..90db2c43006 100644 --- a/Gemfile +++ b/Gemfile @@ -208,6 +208,12 @@ gem 'select2-rails', '~> 3.5.9' gem 'virtus', '~> 1.0.1' gem 'net-ssh', '~> 3.0.1' +# Metrics +group :metrics do + gem 'influxdb', '~> 0.2', require: false + gem 'connection_pool', '~> 2.0', require: false +end + group :development do gem "foreman" gem 'brakeman', '3.0.1', require: false diff --git a/Gemfile.lock b/Gemfile.lock index 4dfff211134..7d592ba93a7 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -65,7 +65,7 @@ GEM attr_encrypted (1.3.4) encryptor (>= 1.3.0) attr_required (1.0.0) - autoprefixer-rails (6.1.1) + autoprefixer-rails (6.1.2) execjs json awesome_print (1.2.0) @@ -102,7 +102,7 @@ GEM bundler-audit (0.4.0) bundler (~> 1.2) thor (~> 0.18) - byebug (8.2.0) + byebug (8.2.1) cal-heatmap-rails (0.0.1) capybara (2.4.4) mime-types (>= 1.16) @@ -117,6 +117,7 @@ GEM activemodel (>= 3.2.0) activesupport (>= 3.2.0) json (>= 1.7) + cause (0.1) charlock_holmes (0.7.3) chunky_png (1.3.5) cliver (0.3.2) @@ -140,10 +141,10 @@ GEM term-ansicolor (~> 1.3) thor (~> 0.19.1) tins (~> 1.6.0) - crack (0.4.2) + crack (0.4.3) safe_yaml (~> 1.0.0) creole (0.5.0) - d3_rails (3.5.6) + d3_rails (3.5.11) railties (>= 3.1.0) daemons (1.2.3) database_cleaner (1.4.1) @@ -230,7 +231,7 @@ GEM ipaddress (~> 0.5) nokogiri (~> 1.5, >= 1.5.11) opennebula - fog-brightbox (0.9.0) + fog-brightbox (0.10.1) fog-core (~> 1.22) fog-json inflecto (~> 0.0.2) @@ -249,7 +250,7 @@ GEM fog-core (>= 1.21.0) fog-json fog-xml (>= 0.0.1) - fog-sakuracloud (1.4.0) + fog-sakuracloud (1.5.0) fog-core fog-json fog-softlayer (1.0.2) @@ -277,11 +278,11 @@ GEM ruby-progressbar (~> 1.4) gemnasium-gitlab-service (0.2.6) rugged (~> 0.21) - gemojione (2.1.0) + gemojione (2.1.1) json get_process_mem (0.2.0) gherkin-ruby (0.3.2) - github-linguist (4.7.2) + github-linguist (4.7.3) charlock_holmes (~> 0.7.3) escape_utils (~> 1.1.0) mime-types (>= 1.19) @@ -298,7 +299,7 @@ GEM posix-spawn (~> 0.3) gitlab_emoji (0.2.0) gemojione (~> 2.1) - gitlab_git (7.2.21) + gitlab_git (7.2.22) activesupport (~> 4.0) charlock_holmes (~> 0.7.3) github-linguist (~> 4.7.0) @@ -370,6 +371,9 @@ GEM i18n (0.7.0) ice_nine (0.11.1) inflecto (0.0.2) + influxdb (0.2.3) + cause + json ipaddress (0.8.0) jquery-atwho-rails (1.3.2) jquery-rails (3.1.4) @@ -416,11 +420,11 @@ GEM net-ldap (0.12.1) net-ssh (3.0.1) netrc (0.11.0) - newrelic-grape (2.0.0) + newrelic-grape (2.1.0) grape newrelic_rpm newrelic_rpm (3.9.4.245) - nokogiri (1.6.7) + nokogiri (1.6.7.1) mini_portile2 (~> 2.0.0.rc2) nprogress-rails (0.1.6.7) oauth (0.4.7) @@ -783,7 +787,7 @@ GEM coercible (~> 1.0) descendants_tracker (~> 0.0, >= 0.0.3) equalizer (~> 0.0, >= 0.0.9) - warden (1.2.3) + warden (1.2.4) rack (>= 1.0) web-console (2.2.1) activemodel (>= 4.0) @@ -836,6 +840,7 @@ DEPENDENCIES charlock_holmes (~> 0.7.3) coffee-rails (~> 4.1.0) colorize (~> 0.7.0) + connection_pool (~> 2.0) coveralls (~> 0.8.2) creole (~> 0.5.0) d3_rails (~> 3.5.5) @@ -873,6 +878,7 @@ DEPENDENCIES hipchat (~> 1.5.0) html-pipeline (~> 1.11.0) httparty (~> 0.13.3) + influxdb (~> 0.2) jquery-atwho-rails (~> 1.3.2) jquery-rails (~> 3.1.3) jquery-scrollto-rails (~> 1.4.3) diff --git a/Procfile b/Procfile index 9cfdee7040f..bbafdd33a2d 100644 --- a/Procfile +++ b/Procfile @@ -3,5 +3,5 @@ # lib/support/init.d, which call scripts in bin/ . # web: bundle exec unicorn_rails -p ${PORT:="3000"} -E ${RAILS_ENV:="development"} -c ${UNICORN_CONFIG:="config/unicorn.rb"} -worker: bundle exec sidekiq -q post_receive -q mailers -q archive_repo -q system_hook -q project_web_hook -q gitlab_shell -q incoming_email -q runner -q common -q default +worker: bundle exec sidekiq -q post_receive -q mailers -q archive_repo -q system_hook -q project_web_hook -q gitlab_shell -q incoming_email -q runner -q common -q default -q metrics # mail_room: bundle exec mail_room -q -c config/mail_room.yml diff --git a/app/workers/metrics_worker.rb b/app/workers/metrics_worker.rb new file mode 100644 index 00000000000..8fffe371572 --- /dev/null +++ b/app/workers/metrics_worker.rb @@ -0,0 +1,29 @@ +class MetricsWorker + include Sidekiq::Worker + + sidekiq_options queue: :metrics + + def perform(metrics) + prepared = prepare_metrics(metrics) + + Gitlab::Metrics.pool.with do |connection| + connection.write_points(prepared) + end + end + + def prepare_metrics(metrics) + metrics.map do |hash| + new_hash = hash.symbolize_keys + + new_hash[:tags].each do |key, value| + new_hash[:tags][key] = escape_value(value) + end + + new_hash + end + end + + def escape_value(value) + value.gsub('=', '\\=') + end +end diff --git a/config/gitlab.yml.example b/config/gitlab.yml.example index db378118f85..38b0920890f 100644 --- a/config/gitlab.yml.example +++ b/config/gitlab.yml.example @@ -421,9 +421,22 @@ production: &base # # Ban an IP for one hour (3600s) after too many auth attempts # bantime: 3600 + metrics: + enabled: false + # The name of the InfluxDB database to store metrics in. + database: gitlab + # Credentials to use for logging in to InfluxDB. + # username: + # password: + # The amount of InfluxDB connections to open. + # pool_size: 16 + # The timeout of a connection in seconds. + # timeout: 10 development: <<: *base + metrics: + enabled: false test: <<: *base @@ -466,6 +479,10 @@ test: user_filter: '' group_base: 'ou=groups,dc=example,dc=com' admin_group: '' + metrics: + enabled: false staging: <<: *base + metrics: + enabled: false diff --git a/config/initializers/metrics.rb b/config/initializers/metrics.rb new file mode 100644 index 00000000000..bde04232725 --- /dev/null +++ b/config/initializers/metrics.rb @@ -0,0 +1,25 @@ +if Gitlab::Metrics.enabled? + require 'influxdb' + require 'socket' + require 'connection_pool' + + # These are manually require'd so the classes are registered properly with + # ActiveSupport. + require 'gitlab/metrics/subscribers/action_view' + require 'gitlab/metrics/subscribers/active_record' + require 'gitlab/metrics/subscribers/method_call' + + Gitlab::Application.configure do |config| + config.middleware.use(Gitlab::Metrics::RackMiddleware) + end + + Sidekiq.configure_server do |config| + config.server_middleware do |chain| + chain.add Gitlab::Metrics::SidekiqMiddleware + end + end + + GC::Profiler.enable + + Gitlab::Metrics::Sampler.new.start +end diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb new file mode 100644 index 00000000000..08393995165 --- /dev/null +++ b/lib/gitlab/metrics.rb @@ -0,0 +1,52 @@ +module Gitlab + module Metrics + def self.pool_size + Settings.metrics['pool_size'] || 16 + end + + def self.timeout + Settings.metrics['timeout'] || 10 + end + + def self.enabled? + !!Settings.metrics['enabled'] + end + + def self.pool + @pool + end + + def self.hostname + @hostname + end + + def self.last_relative_application_frame + root = Rails.root.to_s + metrics = Rails.root.join('lib', 'gitlab', 'metrics').to_s + + frame = caller_locations.find do |l| + l.path.start_with?(root) && !l.path.start_with?(metrics) + end + + if frame + return frame.path.gsub(/^#{Rails.root.to_s}\/?/, ''), frame.lineno + else + return nil, nil + end + end + + @hostname = Socket.gethostname + + # When enabled this should be set before being used as the usual pattern + # "@foo ||= bar" is _not_ thread-safe. + if enabled? + @pool = ConnectionPool.new(size: pool_size, timeout: timeout) do + db = Settings.metrics['database'] + user = Settings.metrics['username'] + pw = Settings.metrics['password'] + + InfluxDB::Client.new(db, username: user, password: pw) + end + end + end +end diff --git a/lib/gitlab/metrics/delta.rb b/lib/gitlab/metrics/delta.rb new file mode 100644 index 00000000000..bcf28eed84d --- /dev/null +++ b/lib/gitlab/metrics/delta.rb @@ -0,0 +1,32 @@ +module Gitlab + module Metrics + # Class for calculating the difference between two numeric values. + # + # Every call to `compared_with` updates the internal value. This makes it + # possible to use a single Delta instance to calculate the delta over time + # of an ever increasing number. + # + # Example usage: + # + # delta = Delta.new(0) + # + # delta.compared_with(10) # => 10 + # delta.compared_with(15) # => 5 + # delta.compared_with(20) # => 5 + class Delta + def initialize(value = 0) + @value = value + end + + # new_value - The value to compare with as a Numeric. + # + # Returns a new Numeric (depending on the type of `new_value`). + def compared_with(new_value) + delta = new_value - @value + @value = new_value + + delta + end + end + end +end diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb new file mode 100644 index 00000000000..1c2f84fb09a --- /dev/null +++ b/lib/gitlab/metrics/instrumentation.rb @@ -0,0 +1,47 @@ +module Gitlab + module Metrics + # Module for instrumenting methods. + # + # This module allows instrumenting of methods without having to actually + # alter the target code (e.g. by including modules). + # + # Example usage: + # + # Gitlab::Metrics::Instrumentation.instrument_method(User, :by_login) + module Instrumentation + # Instruments a class method. + # + # mod - The module to instrument as a Module/Class. + # name - The name of the method to instrument. + def self.instrument_method(mod, name) + instrument(:class, mod, name) + end + + # Instruments an instance method. + # + # mod - The module to instrument as a Module/Class. + # name - The name of the method to instrument. + def self.instrument_instance_method(mod, name) + instrument(:instance, mod, name) + end + + def self.instrument(type, mod, name) + return unless Metrics.enabled? + + alias_name = "_original_#{name}" + target = type == :instance ? mod : mod.singleton_class + + target.class_eval do + alias_method(alias_name, name) + + define_method(name) do |*args, &block| + ActiveSupport::Notifications. + instrument("#{type}_method.method_call", module: mod, name: name) do + __send__(alias_name, *args, &block) + end + end + end + end + end + end +end diff --git a/lib/gitlab/metrics/metric.rb b/lib/gitlab/metrics/metric.rb new file mode 100644 index 00000000000..f592f4e571f --- /dev/null +++ b/lib/gitlab/metrics/metric.rb @@ -0,0 +1,34 @@ +module Gitlab + module Metrics + # Class for storing details of a single metric (label, value, etc). + class Metric + attr_reader :series, :values, :tags, :created_at + + # series - The name of the series (as a String) to store the metric in. + # values - A Hash containing the values to store. + # tags - A Hash containing extra tags to add to the metrics. + def initialize(series, values, tags = {}) + @values = values + @series = series + @tags = tags + @created_at = Time.now.utc + end + + # Returns a Hash in a format that can be directly written to InfluxDB. + def to_hash + { + series: @series, + tags: @tags.merge( + hostname: Metrics.hostname, + ruby_engine: RUBY_ENGINE, + ruby_version: RUBY_VERSION, + gitlab_version: Gitlab::VERSION, + process_type: Sidekiq.server? ? 'sidekiq' : 'rails' + ), + values: @values, + timestamp: @created_at.to_i + } + end + end + end +end diff --git a/lib/gitlab/metrics/obfuscated_sql.rb b/lib/gitlab/metrics/obfuscated_sql.rb new file mode 100644 index 00000000000..45f2e2bc62a --- /dev/null +++ b/lib/gitlab/metrics/obfuscated_sql.rb @@ -0,0 +1,39 @@ +module Gitlab + module Metrics + # Class for producing SQL queries with sensitive data stripped out. + class ObfuscatedSQL + REPLACEMENT = / + \d+(\.\d+)? # integers, floats + | '.+?' # single quoted strings + | \/.+?(? 'GET', 'REQUEST_URI' => '/foo' } } + + describe '#call' do + before do + expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:finish) + end + + it 'tracks a transaction' do + expect(app).to receive(:call).with(env).and_return('yay') + + expect(middleware.call(env)).to eq('yay') + end + + it 'tags a transaction with the name and action of a controller' do + klass = double(:klass, name: 'TestController') + controller = double(:controller, class: klass, action_name: 'show') + + env['action_controller.instance'] = controller + + allow(app).to receive(:call).with(env) + + expect(middleware).to receive(:tag_controller). + with(an_instance_of(Gitlab::Metrics::Transaction), env) + + middleware.call(env) + end + end + + describe '#transaction_from_env' do + let(:transaction) { middleware.transaction_from_env(env) } + + it 'returns a Transaction' do + expect(transaction).to be_an_instance_of(Gitlab::Metrics::Transaction) + end + + it 'tags the transaction with the request method and URI' do + expect(transaction.tags[:request_method]).to eq('GET') + expect(transaction.tags[:request_uri]).to eq('/foo') + end + end + + describe '#tag_controller' do + let(:transaction) { middleware.transaction_from_env(env) } + + it 'tags a transaction with the name and action of a controller' do + klass = double(:klass, name: 'TestController') + controller = double(:controller, class: klass, action_name: 'show') + + env['action_controller.instance'] = controller + + middleware.tag_controller(transaction, env) + + expect(transaction.tags[:action]).to eq('TestController#show') + end + end +end diff --git a/spec/lib/gitlab/metrics/sampler_spec.rb b/spec/lib/gitlab/metrics/sampler_spec.rb new file mode 100644 index 00000000000..b486d38870f --- /dev/null +++ b/spec/lib/gitlab/metrics/sampler_spec.rb @@ -0,0 +1,92 @@ +require 'spec_helper' + +describe Gitlab::Metrics::Sampler do + let(:sampler) { described_class.new(5) } + + describe '#start' do + it 'gathers a sample at a given interval' do + expect(sampler).to receive(:sleep).with(5) + expect(sampler).to receive(:sample) + expect(sampler).to receive(:loop).and_yield + + sampler.start.join + end + end + + describe '#sample' do + it 'samples various statistics' do + expect(sampler).to receive(:sample_memory_usage) + expect(sampler).to receive(:sample_file_descriptors) + expect(sampler).to receive(:sample_objects) + expect(sampler).to receive(:sample_gc) + expect(sampler).to receive(:flush) + + sampler.sample + end + + it 'clears any GC profiles' do + expect(sampler).to receive(:flush) + expect(GC::Profiler).to receive(:clear) + + sampler.sample + end + end + + describe '#flush' do + it 'schedules the metrics using Sidekiq' do + expect(MetricsWorker).to receive(:perform_async). + with([an_instance_of(Hash)]) + + sampler.sample_memory_usage + sampler.flush + end + end + + describe '#sample_memory_usage' do + it 'adds a metric containing the memory usage' do + expect(Gitlab::Metrics::System).to receive(:memory_usage). + and_return(9000) + + expect(Gitlab::Metrics::Metric).to receive(:new). + with('memory_usage', value: 9000). + and_call_original + + sampler.sample_memory_usage + end + end + + describe '#sample_file_descriptors' do + it 'adds a metric containing the amount of open file descriptors' do + expect(Gitlab::Metrics::System).to receive(:file_descriptor_count). + and_return(4) + + expect(Gitlab::Metrics::Metric).to receive(:new). + with('file_descriptors', value: 4). + and_call_original + + sampler.sample_file_descriptors + end + end + + describe '#sample_objects' do + it 'adds a metric containing the amount of allocated objects' do + expect(Gitlab::Metrics::Metric).to receive(:new). + with('object_counts', an_instance_of(Hash)). + and_call_original + + sampler.sample_objects + end + end + + describe '#sample_gc' do + it 'adds a metric containing garbage collection statistics' do + expect(GC::Profiler).to receive(:total_time).and_return(0.24) + + expect(Gitlab::Metrics::Metric).to receive(:new). + with('gc_statistics', an_instance_of(Hash)). + and_call_original + + sampler.sample_gc + end + end +end diff --git a/spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb b/spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb new file mode 100644 index 00000000000..05214efc565 --- /dev/null +++ b/spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb @@ -0,0 +1,34 @@ +require 'spec_helper' + +describe Gitlab::Metrics::SidekiqMiddleware do + let(:middleware) { described_class.new } + + describe '#call' do + it 'tracks the transaction' do + worker = Class.new.new + + expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:finish) + + middleware.call(worker, 'test', :test) { nil } + end + + it 'does not track jobs of the MetricsWorker' do + worker = MetricsWorker.new + + expect(Gitlab::Metrics::Transaction).to_not receive(:new) + + middleware.call(worker, 'test', :test) { nil } + end + end + + describe '#tag_worker' do + it 'adds the worker class and action to the transaction' do + trans = Gitlab::Metrics::Transaction.new + worker = double(:worker, class: double(:class, name: 'TestWorker')) + + expect(trans).to receive(:add_tag).with(:action, 'TestWorker#perform') + + middleware.tag_worker(trans, worker) + end + end +end diff --git a/spec/lib/gitlab/metrics/subscribers/action_view_spec.rb b/spec/lib/gitlab/metrics/subscribers/action_view_spec.rb new file mode 100644 index 00000000000..77f3e69d523 --- /dev/null +++ b/spec/lib/gitlab/metrics/subscribers/action_view_spec.rb @@ -0,0 +1,32 @@ +require 'spec_helper' + +describe Gitlab::Metrics::Subscribers::ActionView do + let(:transaction) { Gitlab::Metrics::Transaction.new } + + let(:subscriber) { described_class.new } + + let(:event) do + root = Rails.root.to_s + + double(:event, duration: 2.1, + payload: { identifier: "#{root}/app/views/x.html.haml" }) + end + + before do + allow(subscriber).to receive(:current_transaction).and_return(transaction) + + allow(Gitlab::Metrics).to receive(:last_relative_application_frame). + and_return(['app/views/x.html.haml', 4]) + end + + describe '#render_template' do + it 'tracks rendering of a template' do + values = { duration: 2.1, file: 'app/views/x.html.haml', line: 4 } + + expect(transaction).to receive(:add_metric). + with(described_class::SERIES, values, path: 'app/views/x.html.haml') + + subscriber.render_template(event) + end + end +end diff --git a/spec/lib/gitlab/metrics/subscribers/active_record_spec.rb b/spec/lib/gitlab/metrics/subscribers/active_record_spec.rb new file mode 100644 index 00000000000..58e8e84df9b --- /dev/null +++ b/spec/lib/gitlab/metrics/subscribers/active_record_spec.rb @@ -0,0 +1,31 @@ +require 'spec_helper' + +describe Gitlab::Metrics::Subscribers::ActiveRecord do + let(:transaction) { Gitlab::Metrics::Transaction.new } + + let(:subscriber) { described_class.new } + + let(:event) do + double(:event, duration: 0.2, + payload: { sql: 'SELECT * FROM users WHERE id = 10' }) + end + + before do + allow(subscriber).to receive(:current_transaction).and_return(transaction) + + allow(Gitlab::Metrics).to receive(:last_relative_application_frame). + and_return(['app/models/foo.rb', 4]) + end + + describe '#sql' do + it 'tracks the execution of a SQL query' do + values = { duration: 0.2, file: 'app/models/foo.rb', line: 4 } + sql = 'SELECT * FROM users WHERE id = ?' + + expect(transaction).to receive(:add_metric). + with(described_class::SERIES, values, sql: sql) + + subscriber.sql(event) + end + end +end diff --git a/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb b/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb new file mode 100644 index 00000000000..65de95d6d1e --- /dev/null +++ b/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb @@ -0,0 +1,41 @@ +require 'spec_helper' + +describe Gitlab::Metrics::Subscribers::MethodCall do + let(:transaction) { Gitlab::Metrics::Transaction.new } + + let(:subscriber) { described_class.new } + + let(:event) do + double(:event, duration: 0.2, + payload: { module: double(:mod, name: 'Foo'), name: :foo }) + end + + before do + allow(subscriber).to receive(:current_transaction).and_return(transaction) + + allow(Gitlab::Metrics).to receive(:last_relative_application_frame). + and_return(['app/models/foo.rb', 4]) + end + + describe '#instance_method' do + it 'tracks the execution of an instance method' do + values = { duration: 0.2, file: 'app/models/foo.rb', line: 4 } + + expect(transaction).to receive(:add_metric). + with(described_class::SERIES, values, method: 'Foo#foo') + + subscriber.instance_method(event) + end + end + + describe '#class_method' do + it 'tracks the execution of a class method' do + values = { duration: 0.2, file: 'app/models/foo.rb', line: 4 } + + expect(transaction).to receive(:add_metric). + with(described_class::SERIES, values, method: 'Foo.foo') + + subscriber.class_method(event) + end + end +end diff --git a/spec/lib/gitlab/metrics/system_spec.rb b/spec/lib/gitlab/metrics/system_spec.rb new file mode 100644 index 00000000000..f8c1d956ca1 --- /dev/null +++ b/spec/lib/gitlab/metrics/system_spec.rb @@ -0,0 +1,29 @@ +require 'spec_helper' + +describe Gitlab::Metrics::System do + if File.exist?('/proc') + describe '.memory_usage' do + it "returns the process' memory usage in bytes" do + expect(described_class.memory_usage).to be > 0 + end + end + + describe '.file_descriptor_count' do + it 'returns the amount of open file descriptors' do + expect(described_class.file_descriptor_count).to be > 0 + end + end + else + describe '.memory_usage' do + it 'returns 0.0' do + expect(described_class.memory_usage).to eq(0.0) + end + end + + describe '.file_descriptor_count' do + it 'returns 0' do + expect(described_class.file_descriptor_count).to eq(0) + end + end + end +end diff --git a/spec/lib/gitlab/metrics/transaction_spec.rb b/spec/lib/gitlab/metrics/transaction_spec.rb new file mode 100644 index 00000000000..5f17ff8ee75 --- /dev/null +++ b/spec/lib/gitlab/metrics/transaction_spec.rb @@ -0,0 +1,77 @@ +require 'spec_helper' + +describe Gitlab::Metrics::Transaction do + let(:transaction) { described_class.new } + + describe '#duration' do + it 'returns the duration of a transaction in seconds' do + transaction.run { sleep(0.5) } + + expect(transaction.duration).to be >= 0.5 + end + end + + describe '#run' do + it 'yields the supplied block' do + expect { |b| transaction.run(&b) }.to yield_control + end + + it 'stores the transaction in the current thread' do + transaction.run do + expect(Thread.current[described_class::THREAD_KEY]).to eq(transaction) + end + end + + it 'removes the transaction from the current thread upon completion' do + transaction.run { } + + expect(Thread.current[described_class::THREAD_KEY]).to be_nil + end + end + + describe '#add_metric' do + it 'adds a metric tagged with the transaction UUID' do + expect(Gitlab::Metrics::Metric).to receive(:new). + with('foo', { number: 10 }, { transaction_id: transaction.uuid }) + + transaction.add_metric('foo', number: 10) + end + end + + describe '#add_tag' do + it 'adds a tag' do + transaction.add_tag(:foo, 'bar') + + expect(transaction.tags).to eq({ foo: 'bar' }) + end + end + + describe '#finish' do + it 'tracks the transaction details and submits them to Sidekiq' do + expect(transaction).to receive(:track_self) + expect(transaction).to receive(:submit) + + transaction.finish + end + end + + describe '#track_self' do + it 'adds a metric for the transaction itself' do + expect(transaction).to receive(:add_metric). + with(described_class::SERIES, { duration: transaction.duration }, {}) + + transaction.track_self + end + end + + describe '#submit' do + it 'submits the metrics to Sidekiq' do + transaction.track_self + + expect(MetricsWorker).to receive(:perform_async). + with([an_instance_of(Hash)]) + + transaction.submit + end + end +end diff --git a/spec/lib/gitlab/metrics_spec.rb b/spec/lib/gitlab/metrics_spec.rb new file mode 100644 index 00000000000..ebc69f8a75f --- /dev/null +++ b/spec/lib/gitlab/metrics_spec.rb @@ -0,0 +1,36 @@ +require 'spec_helper' + +describe Gitlab::Metrics do + describe '.pool_size' do + it 'returns a Fixnum' do + expect(described_class.pool_size).to be_an_instance_of(Fixnum) + end + end + + describe '.timeout' do + it 'returns a Fixnum' do + expect(described_class.timeout).to be_an_instance_of(Fixnum) + end + end + + describe '.enabled?' do + it 'returns a boolean' do + expect([true, false].include?(described_class.enabled?)).to eq(true) + end + end + + describe '.hostname' do + it 'returns a String containing the hostname' do + expect(described_class.hostname).to eq(Socket.gethostname) + end + end + + describe '.last_relative_application_frame' do + it 'returns an Array containing a file path and line number' do + file, line = described_class.last_relative_application_frame + + expect(line).to eq(30) + expect(file).to eq('spec/lib/gitlab/metrics_spec.rb') + end + end +end -- cgit v1.2.1 From 60a6a240ea21cbfa564b9373b1c3bb57316aae46 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 13:25:16 +0100 Subject: Improved last_relative_application_frame timings The previous setup wasn't exactly fast, resulting in instrumented method calls taking about 600 times longer than non instrumented calls (including any ActiveSupport code involved). With this commit this slowdown has been reduced to around 185 times. --- lib/gitlab/metrics.rb | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb index 08393995165..007429fa194 100644 --- a/lib/gitlab/metrics.rb +++ b/lib/gitlab/metrics.rb @@ -1,5 +1,9 @@ module Gitlab module Metrics + RAILS_ROOT = Rails.root.to_s + METRICS_ROOT = Rails.root.join('lib', 'gitlab', 'metrics').to_s + PATH_REGEX = /^#{RAILS_ROOT}\/?/ + def self.pool_size Settings.metrics['pool_size'] || 16 end @@ -20,16 +24,15 @@ module Gitlab @hostname end + # Returns a relative path and line number based on the last application call + # frame. def self.last_relative_application_frame - root = Rails.root.to_s - metrics = Rails.root.join('lib', 'gitlab', 'metrics').to_s - frame = caller_locations.find do |l| - l.path.start_with?(root) && !l.path.start_with?(metrics) + l.path.start_with?(RAILS_ROOT) && !l.path.start_with?(METRICS_ROOT) end if frame - return frame.path.gsub(/^#{Rails.root.to_s}\/?/, ''), frame.lineno + return frame.path.sub(PATH_REGEX, ''), frame.lineno else return nil, nil end -- cgit v1.2.1 From b66a16c8384b64eabeb04f3f32017581e4711eb8 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 13:38:45 +0100 Subject: Use string evaluation for method instrumentation This is faster than using define_method since we don't have to keep block bindings around. --- lib/gitlab/metrics/instrumentation.rb | 16 +++++++++------- lib/gitlab/metrics/subscribers/method_call.rb | 4 ++-- spec/lib/gitlab/metrics/instrumentation_spec.rb | 6 ++++-- spec/lib/gitlab/metrics/subscribers/method_call_spec.rb | 3 +-- 4 files changed, 16 insertions(+), 13 deletions(-) diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index 1c2f84fb09a..982e35adfc9 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -31,16 +31,18 @@ module Gitlab alias_name = "_original_#{name}" target = type == :instance ? mod : mod.singleton_class - target.class_eval do - alias_method(alias_name, name) + target.class_eval <<-EOF, __FILE__, __LINE__ + 1 + alias_method :#{alias_name}, :#{name} - define_method(name) do |*args, &block| - ActiveSupport::Notifications. - instrument("#{type}_method.method_call", module: mod, name: name) do - __send__(alias_name, *args, &block) + def #{name}(*args, &block) + ActiveSupport::Notifications + .instrument("#{type}_method.method_call", + module: #{mod.name.inspect}, + name: #{name.inspect}) do + #{alias_name}(*args, &block) end end - end + EOF end end end diff --git a/lib/gitlab/metrics/subscribers/method_call.rb b/lib/gitlab/metrics/subscribers/method_call.rb index 1606134b7e5..0094ed0dc6a 100644 --- a/lib/gitlab/metrics/subscribers/method_call.rb +++ b/lib/gitlab/metrics/subscribers/method_call.rb @@ -10,7 +10,7 @@ module Gitlab def instance_method(event) return unless current_transaction - label = "#{event.payload[:module].name}##{event.payload[:name]}" + label = "#{event.payload[:module]}##{event.payload[:name]}" add_metric(label, event.duration) end @@ -18,7 +18,7 @@ module Gitlab def class_method(event) return unless current_transaction - label = "#{event.payload[:module].name}.#{event.payload[:name]}" + label = "#{event.payload[:module]}.#{event.payload[:name]}" add_metric(label, event.duration) end diff --git a/spec/lib/gitlab/metrics/instrumentation_spec.rb b/spec/lib/gitlab/metrics/instrumentation_spec.rb index eb31c9e52cd..3e7e9869e30 100644 --- a/spec/lib/gitlab/metrics/instrumentation_spec.rb +++ b/spec/lib/gitlab/metrics/instrumentation_spec.rb @@ -11,6 +11,8 @@ describe Gitlab::Metrics::Instrumentation do text end end + + allow(@dummy).to receive(:name).and_return('Dummy') end describe '.instrument_method' do @@ -31,7 +33,7 @@ describe Gitlab::Metrics::Instrumentation do it 'fires an ActiveSupport notification upon calling the method' do expect(ActiveSupport::Notifications).to receive(:instrument). - with('class_method.method_call', module: @dummy, name: :foo) + with('class_method.method_call', module: 'Dummy', name: :foo) @dummy.foo end @@ -69,7 +71,7 @@ describe Gitlab::Metrics::Instrumentation do it 'fires an ActiveSupport notification upon calling the method' do expect(ActiveSupport::Notifications).to receive(:instrument). - with('instance_method.method_call', module: @dummy, name: :bar) + with('instance_method.method_call', module: 'Dummy', name: :bar) @dummy.new.bar end diff --git a/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb b/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb index 65de95d6d1e..5c76eb3ba50 100644 --- a/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb +++ b/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb @@ -6,8 +6,7 @@ describe Gitlab::Metrics::Subscribers::MethodCall do let(:subscriber) { described_class.new } let(:event) do - double(:event, duration: 0.2, - payload: { module: double(:mod, name: 'Foo'), name: :foo }) + double(:event, duration: 0.2, payload: { module: 'Foo', name: :foo }) end before do -- cgit v1.2.1 From 1b077d2d81bd25fe37492ea56c8bd884f944ce52 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 15:16:56 +0100 Subject: Use custom code for instrumenting method calls The use of ActiveSupport would slow down instrumented method calls by about 180x due to: 1. ActiveSupport itself not being the fastest thing on the planet 2. caller_locations() having quite some overhead The use of caller_locations() has been removed because it's not _that_ useful since we already know the full namespace of receivers and the names of the called methods. The use of ActiveSupport has been replaced with some custom code that's generated using eval() (which can be quite a bit faster than using define_method). This new setup results in instrumented methods only being about 35-40x slower (compared to non instrumented methods). --- config/initializers/metrics.rb | 1 - lib/gitlab/metrics/instrumentation.rb | 37 +++++++++++++++---- lib/gitlab/metrics/subscribers/method_call.rb | 42 ---------------------- spec/lib/gitlab/metrics/instrumentation_spec.rb | 22 ++++++++---- .../gitlab/metrics/subscribers/method_call_spec.rb | 40 --------------------- 5 files changed, 47 insertions(+), 95 deletions(-) delete mode 100644 lib/gitlab/metrics/subscribers/method_call.rb delete mode 100644 spec/lib/gitlab/metrics/subscribers/method_call_spec.rb diff --git a/config/initializers/metrics.rb b/config/initializers/metrics.rb index bde04232725..556e968137e 100644 --- a/config/initializers/metrics.rb +++ b/config/initializers/metrics.rb @@ -7,7 +7,6 @@ if Gitlab::Metrics.enabled? # ActiveSupport. require 'gitlab/metrics/subscribers/action_view' require 'gitlab/metrics/subscribers/active_record' - require 'gitlab/metrics/subscribers/method_call' Gitlab::Application.configure do |config| config.middleware.use(Gitlab::Metrics::RackMiddleware) diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index 982e35adfc9..5b56f2e8513 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -9,6 +9,8 @@ module Gitlab # # Gitlab::Metrics::Instrumentation.instrument_method(User, :by_login) module Instrumentation + SERIES = 'method_calls' + # Instruments a class method. # # mod - The module to instrument as a Module/Class. @@ -31,19 +33,42 @@ module Gitlab alias_name = "_original_#{name}" target = type == :instance ? mod : mod.singleton_class + if type == :instance + target = mod + label = "#{mod.name}##{name}" + else + target = mod.singleton_class + label = "#{mod.name}.#{name}" + end + target.class_eval <<-EOF, __FILE__, __LINE__ + 1 alias_method :#{alias_name}, :#{name} def #{name}(*args, &block) - ActiveSupport::Notifications - .instrument("#{type}_method.method_call", - module: #{mod.name.inspect}, - name: #{name.inspect}) do - #{alias_name}(*args, &block) - end + trans = Gitlab::Metrics::Instrumentation.transaction + + if trans + start = Time.now + retval = #{alias_name}(*args, &block) + duration = (Time.now - start) * 1000.0 + + trans.add_metric(Gitlab::Metrics::Instrumentation::SERIES, + { duration: duration }, + method: #{label.inspect}) + + retval + else + #{alias_name}(*args, &block) + end end EOF end + + # Small layer of indirection to make it easier to stub out the current + # transaction. + def self.transaction + Transaction.current + end end end end diff --git a/lib/gitlab/metrics/subscribers/method_call.rb b/lib/gitlab/metrics/subscribers/method_call.rb deleted file mode 100644 index 0094ed0dc6a..00000000000 --- a/lib/gitlab/metrics/subscribers/method_call.rb +++ /dev/null @@ -1,42 +0,0 @@ -module Gitlab - module Metrics - module Subscribers - # Class for tracking method call timings. - class MethodCall < ActiveSupport::Subscriber - attach_to :method_call - - SERIES = 'method_calls' - - def instance_method(event) - return unless current_transaction - - label = "#{event.payload[:module]}##{event.payload[:name]}" - - add_metric(label, event.duration) - end - - def class_method(event) - return unless current_transaction - - label = "#{event.payload[:module]}.#{event.payload[:name]}" - - add_metric(label, event.duration) - end - - private - - def add_metric(label, duration) - file, line = Metrics.last_relative_application_frame - - values = { duration: duration, file: file, line: line } - - current_transaction.add_metric(SERIES, values, method: label) - end - - def current_transaction - Transaction.current - end - end - end - end -end diff --git a/spec/lib/gitlab/metrics/instrumentation_spec.rb b/spec/lib/gitlab/metrics/instrumentation_spec.rb index 3e7e9869e30..9308fb157d8 100644 --- a/spec/lib/gitlab/metrics/instrumentation_spec.rb +++ b/spec/lib/gitlab/metrics/instrumentation_spec.rb @@ -1,6 +1,8 @@ require 'spec_helper' describe Gitlab::Metrics::Instrumentation do + let(:transaction) { Gitlab::Metrics::Transaction.new } + before do @dummy = Class.new do def self.foo(text = 'foo') @@ -31,9 +33,13 @@ describe Gitlab::Metrics::Instrumentation do expect(@dummy.foo).to eq('foo') end - it 'fires an ActiveSupport notification upon calling the method' do - expect(ActiveSupport::Notifications).to receive(:instrument). - with('class_method.method_call', module: 'Dummy', name: :foo) + it 'tracks the call duration upon calling the method' do + allow(Gitlab::Metrics::Instrumentation).to receive(:transaction). + and_return(transaction) + + expect(transaction).to receive(:add_metric). + with(described_class::SERIES, an_instance_of(Hash), + method: 'Dummy.foo') @dummy.foo end @@ -69,9 +75,13 @@ describe Gitlab::Metrics::Instrumentation do expect(@dummy.new.bar).to eq('bar') end - it 'fires an ActiveSupport notification upon calling the method' do - expect(ActiveSupport::Notifications).to receive(:instrument). - with('instance_method.method_call', module: 'Dummy', name: :bar) + it 'tracks the call duration upon calling the method' do + allow(Gitlab::Metrics::Instrumentation).to receive(:transaction). + and_return(transaction) + + expect(transaction).to receive(:add_metric). + with(described_class::SERIES, an_instance_of(Hash), + method: 'Dummy#bar') @dummy.new.bar end diff --git a/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb b/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb deleted file mode 100644 index 5c76eb3ba50..00000000000 --- a/spec/lib/gitlab/metrics/subscribers/method_call_spec.rb +++ /dev/null @@ -1,40 +0,0 @@ -require 'spec_helper' - -describe Gitlab::Metrics::Subscribers::MethodCall do - let(:transaction) { Gitlab::Metrics::Transaction.new } - - let(:subscriber) { described_class.new } - - let(:event) do - double(:event, duration: 0.2, payload: { module: 'Foo', name: :foo }) - end - - before do - allow(subscriber).to receive(:current_transaction).and_return(transaction) - - allow(Gitlab::Metrics).to receive(:last_relative_application_frame). - and_return(['app/models/foo.rb', 4]) - end - - describe '#instance_method' do - it 'tracks the execution of an instance method' do - values = { duration: 0.2, file: 'app/models/foo.rb', line: 4 } - - expect(transaction).to receive(:add_metric). - with(described_class::SERIES, values, method: 'Foo#foo') - - subscriber.instance_method(event) - end - end - - describe '#class_method' do - it 'tracks the execution of a class method' do - values = { duration: 0.2, file: 'app/models/foo.rb', line: 4 } - - expect(transaction).to receive(:add_metric). - with(described_class::SERIES, values, method: 'Foo.foo') - - subscriber.class_method(event) - end - end -end -- cgit v1.2.1 From ad69ba57d6e7d52b2a44a20393c072538c299653 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 16:15:02 +0100 Subject: Proper method instrumentation for special symbols This ensures that methods such as "==" can be instrumented without producing syntax errors. --- lib/gitlab/metrics/instrumentation.rb | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index 5b56f2e8513..2ed75495650 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -30,7 +30,8 @@ module Gitlab def self.instrument(type, mod, name) return unless Metrics.enabled? - alias_name = "_original_#{name}" + name = name.to_sym + alias_name = :"_original_#{name}" target = type == :instance ? mod : mod.singleton_class if type == :instance @@ -42,14 +43,14 @@ module Gitlab end target.class_eval <<-EOF, __FILE__, __LINE__ + 1 - alias_method :#{alias_name}, :#{name} + alias_method #{alias_name.inspect}, #{name.inspect} def #{name}(*args, &block) trans = Gitlab::Metrics::Instrumentation.transaction if trans start = Time.now - retval = #{alias_name}(*args, &block) + retval = duration = (Time.now - start) * 1000.0 trans.add_metric(Gitlab::Metrics::Instrumentation::SERIES, @@ -58,7 +59,7 @@ module Gitlab retval else - #{alias_name}(*args, &block) + __send__(#{alias_name.inspect}, *args, &block) end end EOF -- cgit v1.2.1 From 5dbcb635a17aff6543150a66b597c75b819801e2 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 16:15:46 +0100 Subject: Methods for instrumenting multiple methods The methods Instrumentation.instrument_methods and Instrumentation.instrument_instance_methods can be used to instrument all methods of a module at once. --- lib/gitlab/metrics/instrumentation.rb | 23 +++++++++++++++++ spec/lib/gitlab/metrics/instrumentation_spec.rb | 34 +++++++++++++++++++++---- 2 files changed, 52 insertions(+), 5 deletions(-) diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index 2ed75495650..12d5ede4be3 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -27,6 +27,29 @@ module Gitlab instrument(:instance, mod, name) end + # Instruments all public methods of a module. + # + # mod - The module to instrument. + def self.instrument_methods(mod) + mod.public_methods(false).each do |name| + instrument_method(mod, name) + end + end + + # Instruments all public instance methods of a module. + # + # mod - The module to instrument. + def self.instrument_instance_methods(mod) + mod.public_instance_methods(false).each do |name| + instrument_instance_method(mod, name) + end + end + + # Instruments a method. + # + # type - The type (:class or :instance) of method to instrument. + # mod - The module containing the method. + # name - The name of the method to instrument. def self.instrument(type, mod, name) return unless Metrics.enabled? diff --git a/spec/lib/gitlab/metrics/instrumentation_spec.rb b/spec/lib/gitlab/metrics/instrumentation_spec.rb index 9308fb157d8..5427bacdc94 100644 --- a/spec/lib/gitlab/metrics/instrumentation_spec.rb +++ b/spec/lib/gitlab/metrics/instrumentation_spec.rb @@ -34,7 +34,7 @@ describe Gitlab::Metrics::Instrumentation do end it 'tracks the call duration upon calling the method' do - allow(Gitlab::Metrics::Instrumentation).to receive(:transaction). + allow(described_class).to receive(:transaction). and_return(transaction) expect(transaction).to receive(:add_metric). @@ -51,7 +51,7 @@ describe Gitlab::Metrics::Instrumentation do end it 'does not instrument the method' do - Gitlab::Metrics::Instrumentation.instrument_method(@dummy, :foo) + described_class.instrument_method(@dummy, :foo) expect(@dummy).to_not respond_to(:_original_foo) end @@ -63,7 +63,7 @@ describe Gitlab::Metrics::Instrumentation do before do allow(Gitlab::Metrics).to receive(:enabled?).and_return(true) - Gitlab::Metrics::Instrumentation. + described_class. instrument_instance_method(@dummy, :bar) end @@ -76,7 +76,7 @@ describe Gitlab::Metrics::Instrumentation do end it 'tracks the call duration upon calling the method' do - allow(Gitlab::Metrics::Instrumentation).to receive(:transaction). + allow(described_class).to receive(:transaction). and_return(transaction) expect(transaction).to receive(:add_metric). @@ -93,11 +93,35 @@ describe Gitlab::Metrics::Instrumentation do end it 'does not instrument the method' do - Gitlab::Metrics::Instrumentation. + described_class. instrument_instance_method(@dummy, :bar) expect(@dummy.method_defined?(:_original_bar)).to eq(false) end end end + + describe '.instrument_methods' do + before do + allow(Gitlab::Metrics).to receive(:enabled?).and_return(true) + end + + it 'instruments all public class methods' do + described_class.instrument_methods(@dummy) + + expect(@dummy).to respond_to(:_original_foo) + end + end + + describe '.instrument_instance_methods' do + before do + allow(Gitlab::Metrics).to receive(:enabled?).and_return(true) + end + + it 'instruments all public instance methods' do + described_class.instrument_instance_methods(@dummy) + + expect(@dummy.method_defined?(:_original_bar)).to eq(true) + end + end end -- cgit v1.2.1 From f43f3b89a633b5ceee4e71acba0c83ed5cb28963 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 16:16:48 +0100 Subject: Added Instrumentation.configure This makes it easier to instrument multiple modules without having to type the full namespace over and over again. --- lib/gitlab/metrics/instrumentation.rb | 4 ++++ spec/lib/gitlab/metrics/instrumentation_spec.rb | 10 +++++++++- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index 12d5ede4be3..8822a907967 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -11,6 +11,10 @@ module Gitlab module Instrumentation SERIES = 'method_calls' + def self.configure + yield self + end + # Instruments a class method. # # mod - The module to instrument as a Module/Class. diff --git a/spec/lib/gitlab/metrics/instrumentation_spec.rb b/spec/lib/gitlab/metrics/instrumentation_spec.rb index 5427bacdc94..fdb0820b875 100644 --- a/spec/lib/gitlab/metrics/instrumentation_spec.rb +++ b/spec/lib/gitlab/metrics/instrumentation_spec.rb @@ -17,12 +17,20 @@ describe Gitlab::Metrics::Instrumentation do allow(@dummy).to receive(:name).and_return('Dummy') end + describe '.configure' do + it 'yields self' do + described_class.configure do |c| + expect(c).to eq(described_class) + end + end + end + describe '.instrument_method' do describe 'with metrics enabled' do before do allow(Gitlab::Metrics).to receive(:enabled?).and_return(true) - Gitlab::Metrics::Instrumentation.instrument_method(@dummy, :foo) + described_class.instrument_method(@dummy, :foo) end it 'renames the original method' do -- cgit v1.2.1 From 641761f1d6af5a94c0007e8d7463ee86fc047229 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 16:25:28 +0100 Subject: Only instrument methods defined directly When using instrument_methods/instrument_instance_methods we only want to instrument methods defined directly in a class, not those included via mixins (e.g. whatever RSpec throws in during development). In case an externally included method _has_ to be instrumented we can still use the regular instrument_method/instrument_instance_method methods. --- lib/gitlab/metrics/instrumentation.rb | 10 +++++++--- spec/lib/gitlab/metrics/instrumentation_spec.rb | 26 +++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 3 deletions(-) diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index 8822a907967..cf22aa93cdd 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -36,7 +36,9 @@ module Gitlab # mod - The module to instrument. def self.instrument_methods(mod) mod.public_methods(false).each do |name| - instrument_method(mod, name) + method = mod.method(name) + + instrument_method(mod, name) if method.owner == mod.singleton_class end end @@ -45,7 +47,9 @@ module Gitlab # mod - The module to instrument. def self.instrument_instance_methods(mod) mod.public_instance_methods(false).each do |name| - instrument_instance_method(mod, name) + method = mod.instance_method(name) + + instrument_instance_method(mod, name) if method.owner == mod end end @@ -77,7 +81,7 @@ module Gitlab if trans start = Time.now - retval = + retval = __send__(#{alias_name.inspect}, *args, &block) duration = (Time.now - start) * 1000.0 trans.add_metric(Gitlab::Metrics::Instrumentation::SERIES, diff --git a/spec/lib/gitlab/metrics/instrumentation_spec.rb b/spec/lib/gitlab/metrics/instrumentation_spec.rb index fdb0820b875..80dc160ebd2 100644 --- a/spec/lib/gitlab/metrics/instrumentation_spec.rb +++ b/spec/lib/gitlab/metrics/instrumentation_spec.rb @@ -119,6 +119,19 @@ describe Gitlab::Metrics::Instrumentation do expect(@dummy).to respond_to(:_original_foo) end + + it 'only instruments methods directly defined in the module' do + mod = Module.new do + def kittens + end + end + + @dummy.extend(mod) + + described_class.instrument_methods(@dummy) + + expect(@dummy).to_not respond_to(:_original_kittens) + end end describe '.instrument_instance_methods' do @@ -131,5 +144,18 @@ describe Gitlab::Metrics::Instrumentation do expect(@dummy.method_defined?(:_original_bar)).to eq(true) end + + it 'only instruments methods directly defined in the module' do + mod = Module.new do + def kittens + end + end + + @dummy.include(mod) + + described_class.instrument_instance_methods(@dummy) + + expect(@dummy.method_defined?(:_original_kittens)).to eq(false) + end end end -- cgit v1.2.1 From 6dc25ad58c7bf2218cdda6d60061111a5365affb Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 17:09:20 +0100 Subject: Instrument Gitlab::Shel and Gitlab::Git --- config/initializers/metrics.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/config/initializers/metrics.rb b/config/initializers/metrics.rb index 556e968137e..0ac4299dcba 100644 --- a/config/initializers/metrics.rb +++ b/config/initializers/metrics.rb @@ -18,6 +18,18 @@ if Gitlab::Metrics.enabled? end end + Gitlab::Metrics::Instrumentation.configure do |config| + config.instrument_instance_methods(Gitlab::Shell) + + config.instrument_methods(Gitlab::Git) + + Gitlab::Git.constants.each do |name| + const = Gitlab::Git.const_get(name) + + config.instrument_methods(const) if const.is_a?(Module) + end + end + GC::Profiler.enable Gitlab::Metrics::Sampler.new.start -- cgit v1.2.1 From 09a311568abee739fae0c2577a9cf6aa01516977 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 10 Dec 2015 17:48:14 +0100 Subject: Track object count types as tags --- lib/gitlab/metrics/sampler.rb | 4 +++- spec/lib/gitlab/metrics/sampler_spec.rb | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/gitlab/metrics/sampler.rb b/lib/gitlab/metrics/sampler.rb index 141953dc985..03afa6324dd 100644 --- a/lib/gitlab/metrics/sampler.rb +++ b/lib/gitlab/metrics/sampler.rb @@ -53,7 +53,9 @@ module Gitlab end def sample_objects - @metrics << Metric.new('object_counts', ObjectSpace.count_objects) + ObjectSpace.count_objects.each do |type, count| + @metrics << Metric.new('object_counts', { count: count }, type: type) + end end def sample_gc diff --git a/spec/lib/gitlab/metrics/sampler_spec.rb b/spec/lib/gitlab/metrics/sampler_spec.rb index b486d38870f..319f287178d 100644 --- a/spec/lib/gitlab/metrics/sampler_spec.rb +++ b/spec/lib/gitlab/metrics/sampler_spec.rb @@ -71,7 +71,8 @@ describe Gitlab::Metrics::Sampler do describe '#sample_objects' do it 'adds a metric containing the amount of allocated objects' do expect(Gitlab::Metrics::Metric).to receive(:new). - with('object_counts', an_instance_of(Hash)). + with('object_counts', an_instance_of(Hash), an_instance_of(Hash)). + at_least(:once). and_call_original sampler.sample_objects -- cgit v1.2.1 From f932b781a79d9b829001c39bc214372b7efd8610 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 14 Dec 2015 12:31:56 +0100 Subject: Replace double quotes when obfuscating SQL InfluxDB escapes double quotes upon output which makes it a pain to deal with. This ensures that if we're using PostgreSQL we don't store any queries containing double quotes in InfluxDB, solving the escaping problem. --- lib/gitlab/metrics/obfuscated_sql.rb | 10 +++++++++- spec/lib/gitlab/metrics/obfuscated_sql_spec.rb | 8 ++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/lib/gitlab/metrics/obfuscated_sql.rb b/lib/gitlab/metrics/obfuscated_sql.rb index 45f2e2bc62a..7b15670aa6b 100644 --- a/lib/gitlab/metrics/obfuscated_sql.rb +++ b/lib/gitlab/metrics/obfuscated_sql.rb @@ -30,9 +30,17 @@ module Gitlab regex = Regexp.union(regex, MYSQL_REPLACEMENTS) end - @sql.gsub(regex, '?').gsub(CONSECUTIVE) do |match| + sql = @sql.gsub(regex, '?').gsub(CONSECUTIVE) do |match| "#{match.count(',') + 1} values" end + + # InfluxDB escapes double quotes upon output, so lets get rid of them + # whenever we can. + if Gitlab::Database.postgresql? + sql = sql.gsub('"', '') + end + + sql end end end diff --git a/spec/lib/gitlab/metrics/obfuscated_sql_spec.rb b/spec/lib/gitlab/metrics/obfuscated_sql_spec.rb index 6e9b62016d6..0f01ee588c9 100644 --- a/spec/lib/gitlab/metrics/obfuscated_sql_spec.rb +++ b/spec/lib/gitlab/metrics/obfuscated_sql_spec.rb @@ -75,5 +75,13 @@ describe Gitlab::Metrics::ObfuscatedSQL do expect(sql.to_s).to eq('SELECT x FROM y WHERE z IN (2 values)') end end + + if Gitlab::Database.postgresql? + it 'replaces double quotes' do + sql = described_class.new('SELECT "x" FROM "y"') + + expect(sql.to_s).to eq('SELECT x FROM y') + end + end end end -- cgit v1.2.1 From d0352e6604915a1a94fbe84252d1606f5adac57c Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 14 Dec 2015 12:32:30 +0100 Subject: Added specs for MetricsWorker --- spec/workers/metrics_worker_spec.rb | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 spec/workers/metrics_worker_spec.rb diff --git a/spec/workers/metrics_worker_spec.rb b/spec/workers/metrics_worker_spec.rb new file mode 100644 index 00000000000..0d12516c1a3 --- /dev/null +++ b/spec/workers/metrics_worker_spec.rb @@ -0,0 +1,40 @@ +require 'spec_helper' + +describe MetricsWorker do + let(:worker) { described_class.new } + + describe '#perform' do + it 'prepares and writes the metrics to InfluxDB' do + connection = double(:connection) + pool = double(:pool) + + expect(pool).to receive(:with).and_yield(connection) + expect(connection).to receive(:write_points).with(an_instance_of(Array)) + expect(Gitlab::Metrics).to receive(:pool).and_return(pool) + + worker.perform([{ 'series' => 'kittens', 'tags' => {} }]) + end + end + + describe '#prepare_metrics' do + it 'returns a Hash with the keys as Symbols' do + metrics = worker.prepare_metrics([{ 'values' => {}, 'tags' => {} }]) + + expect(metrics).to eq([{ values: {}, tags: {} }]) + end + + it 'escapes tag values' do + metrics = worker.prepare_metrics([ + { 'values' => {}, 'tags' => { 'foo' => 'bar=' } } + ]) + + expect(metrics).to eq([{ values: {}, tags: { 'foo' => 'bar\\=' } }]) + end + end + + describe '#escape_value' do + it 'escapes an equals sign' do + expect(worker.escape_value('foo=')).to eq('foo\\=') + end + end +end -- cgit v1.2.1 From 9f95ff0d90802467a04816f1d38e30770a026820 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 14 Dec 2015 16:51:38 +0100 Subject: Track location information as tags This allows the information to be displayed when using certain functions (e.g. top()) as well as making it easier to aggregate on a per file basis. --- lib/gitlab/metrics/subscribers/action_view.rb | 17 +++++++++++------ lib/gitlab/metrics/subscribers/active_record.rb | 17 +++++++++++------ spec/lib/gitlab/metrics/subscribers/action_view_spec.rb | 9 +++++++-- .../gitlab/metrics/subscribers/active_record_spec.rb | 5 +++-- 4 files changed, 32 insertions(+), 16 deletions(-) diff --git a/lib/gitlab/metrics/subscribers/action_view.rb b/lib/gitlab/metrics/subscribers/action_view.rb index 2e88e4bea6a..7e0dcf99d92 100644 --- a/lib/gitlab/metrics/subscribers/action_view.rb +++ b/lib/gitlab/metrics/subscribers/action_view.rb @@ -16,10 +16,10 @@ module Gitlab private def track(event) - path = relative_path(event.payload[:identifier]) values = values_for(event) + tags = tags_for(event) - current_transaction.add_metric(SERIES, values, path: path) + current_transaction.add_metric(SERIES, values, tags) end def relative_path(path) @@ -27,16 +27,21 @@ module Gitlab end def values_for(event) - values = { duration: event.duration } + { duration: event.duration } + end + + def tags_for(event) + path = relative_path(event.payload[:identifier]) + tags = { view: path } file, line = Metrics.last_relative_application_frame if file and line - values[:file] = file - values[:line] = line + tags[:file] = file + tags[:line] = line end - values + tags end def current_transaction diff --git a/lib/gitlab/metrics/subscribers/active_record.rb b/lib/gitlab/metrics/subscribers/active_record.rb index 3cc9b1addf6..d947c128ce2 100644 --- a/lib/gitlab/metrics/subscribers/active_record.rb +++ b/lib/gitlab/metrics/subscribers/active_record.rb @@ -13,25 +13,30 @@ module Gitlab def sql(event) return unless current_transaction - sql = ObfuscatedSQL.new(event.payload[:sql]).to_s values = values_for(event) + tags = tags_for(event) - current_transaction.add_metric(SERIES, values, sql: sql) + current_transaction.add_metric(SERIES, values, tags) end private def values_for(event) - values = { duration: event.duration } + { duration: event.duration } + end + + def tags_for(event) + sql = ObfuscatedSQL.new(event.payload[:sql]).to_s + tags = { sql: sql } file, line = Metrics.last_relative_application_frame if file and line - values[:file] = file - values[:line] = line + tags[:file] = file + tags[:line] = line end - values + tags end def current_transaction diff --git a/spec/lib/gitlab/metrics/subscribers/action_view_spec.rb b/spec/lib/gitlab/metrics/subscribers/action_view_spec.rb index 77f3e69d523..c6cd584663f 100644 --- a/spec/lib/gitlab/metrics/subscribers/action_view_spec.rb +++ b/spec/lib/gitlab/metrics/subscribers/action_view_spec.rb @@ -21,10 +21,15 @@ describe Gitlab::Metrics::Subscribers::ActionView do describe '#render_template' do it 'tracks rendering of a template' do - values = { duration: 2.1, file: 'app/views/x.html.haml', line: 4 } + values = { duration: 2.1 } + tags = { + view: 'app/views/x.html.haml', + file: 'app/views/x.html.haml', + line: 4 + } expect(transaction).to receive(:add_metric). - with(described_class::SERIES, values, path: 'app/views/x.html.haml') + with(described_class::SERIES, values, tags) subscriber.render_template(event) end diff --git a/spec/lib/gitlab/metrics/subscribers/active_record_spec.rb b/spec/lib/gitlab/metrics/subscribers/active_record_spec.rb index 58e8e84df9b..05b6cc14716 100644 --- a/spec/lib/gitlab/metrics/subscribers/active_record_spec.rb +++ b/spec/lib/gitlab/metrics/subscribers/active_record_spec.rb @@ -19,11 +19,12 @@ describe Gitlab::Metrics::Subscribers::ActiveRecord do describe '#sql' do it 'tracks the execution of a SQL query' do - values = { duration: 0.2, file: 'app/models/foo.rb', line: 4 } sql = 'SELECT * FROM users WHERE id = ?' + values = { duration: 0.2 } + tags = { sql: sql, file: 'app/models/foo.rb', line: 4 } expect(transaction).to receive(:add_metric). - with(described_class::SERIES, values, sql: sql) + with(described_class::SERIES, values, tags) subscriber.sql(event) end -- cgit v1.2.1 From 5142c61707cc4169a3f8d9e378aacb8f88760db5 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 14 Dec 2015 16:52:05 +0100 Subject: Cast values to strings before escaping them This ensures that e.g. line numbers used in tags are first casted to strings. --- app/workers/metrics_worker.rb | 2 +- spec/workers/metrics_worker_spec.rb | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/app/workers/metrics_worker.rb b/app/workers/metrics_worker.rb index 8fffe371572..90a65579382 100644 --- a/app/workers/metrics_worker.rb +++ b/app/workers/metrics_worker.rb @@ -24,6 +24,6 @@ class MetricsWorker end def escape_value(value) - value.gsub('=', '\\=') + value.to_s.gsub('=', '\\=') end end diff --git a/spec/workers/metrics_worker_spec.rb b/spec/workers/metrics_worker_spec.rb index 0d12516c1a3..f5650494c7c 100644 --- a/spec/workers/metrics_worker_spec.rb +++ b/spec/workers/metrics_worker_spec.rb @@ -36,5 +36,9 @@ describe MetricsWorker do it 'escapes an equals sign' do expect(worker.escape_value('foo=')).to eq('foo\\=') end + + it 'casts values to Strings' do + expect(worker.escape_value(10)).to eq('10') + end end end -- cgit v1.2.1 From d67e2045a02d105bfbc7abf1805fe477eb9155ca Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 14 Dec 2015 17:37:20 +0100 Subject: Drop empty tag values from metrics InfluxDB throws an error when trying to store a list of tags where one or more have an empty value. --- app/workers/metrics_worker.rb | 6 +++++- spec/workers/metrics_worker_spec.rb | 8 ++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/app/workers/metrics_worker.rb b/app/workers/metrics_worker.rb index 90a65579382..b15dc819c5c 100644 --- a/app/workers/metrics_worker.rb +++ b/app/workers/metrics_worker.rb @@ -16,7 +16,11 @@ class MetricsWorker new_hash = hash.symbolize_keys new_hash[:tags].each do |key, value| - new_hash[:tags][key] = escape_value(value) + if value.blank? + new_hash[:tags].delete(key) + else + new_hash[:tags][key] = escape_value(value) + end end new_hash diff --git a/spec/workers/metrics_worker_spec.rb b/spec/workers/metrics_worker_spec.rb index f5650494c7c..2acd0f8ba30 100644 --- a/spec/workers/metrics_worker_spec.rb +++ b/spec/workers/metrics_worker_spec.rb @@ -30,6 +30,14 @@ describe MetricsWorker do expect(metrics).to eq([{ values: {}, tags: { 'foo' => 'bar\\=' } }]) end + + it 'drops empty tags' do + metrics = worker.prepare_metrics([ + { 'values' => {}, 'tags' => { 'cats' => '', 'dogs' => nil }} + ]) + + expect(metrics).to eq([{ values: {}, tags: {} }]) + end end describe '#escape_value' do -- cgit v1.2.1 From 13dbd663acbbe91ddac77b650a90377cd12c54b8 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Tue, 15 Dec 2015 16:31:24 +0100 Subject: Allow filtering of what methods to instrument This makes it possible to determine if a method should be instrumented or not using a block. --- lib/gitlab/metrics/instrumentation.rb | 19 +++++++++++++++++-- spec/lib/gitlab/metrics/instrumentation_spec.rb | 16 ++++++++++++++++ 2 files changed, 33 insertions(+), 2 deletions(-) diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index cf22aa93cdd..91e09694cd8 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -33,23 +33,38 @@ module Gitlab # Instruments all public methods of a module. # + # This method optionally takes a block that can be used to determine if a + # method should be instrumented or not. The block is passed the receiving + # module and an UnboundMethod. If the block returns a non truthy value the + # method is not instrumented. + # # mod - The module to instrument. def self.instrument_methods(mod) mod.public_methods(false).each do |name| method = mod.method(name) - instrument_method(mod, name) if method.owner == mod.singleton_class + if method.owner == mod.singleton_class + if !block_given? || block_given? && yield(mod, method) + instrument_method(mod, name) + end + end end end # Instruments all public instance methods of a module. # + # See `instrument_methods` for more information. + # # mod - The module to instrument. def self.instrument_instance_methods(mod) mod.public_instance_methods(false).each do |name| method = mod.instance_method(name) - instrument_instance_method(mod, name) if method.owner == mod + if method.owner == mod + if !block_given? || block_given? && yield(mod, method) + instrument_instance_method(mod, name) + end + end end end diff --git a/spec/lib/gitlab/metrics/instrumentation_spec.rb b/spec/lib/gitlab/metrics/instrumentation_spec.rb index 80dc160ebd2..5fe7a369cba 100644 --- a/spec/lib/gitlab/metrics/instrumentation_spec.rb +++ b/spec/lib/gitlab/metrics/instrumentation_spec.rb @@ -132,6 +132,14 @@ describe Gitlab::Metrics::Instrumentation do expect(@dummy).to_not respond_to(:_original_kittens) end + + it 'can take a block to determine if a method should be instrumented' do + described_class.instrument_methods(@dummy) do + false + end + + expect(@dummy).to_not respond_to(:_original_foo) + end end describe '.instrument_instance_methods' do @@ -157,5 +165,13 @@ describe Gitlab::Metrics::Instrumentation do expect(@dummy.method_defined?(:_original_kittens)).to eq(false) end + + it 'can take a block to determine if a method should be instrumented' do + described_class.instrument_instance_methods(@dummy) do + false + end + + expect(@dummy.method_defined?(:_original_bar)).to eq(false) + end end end -- cgit v1.2.1 From a41287d8989d7d49b405fd8f658d6c6e4edfd307 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Tue, 15 Dec 2015 16:38:25 +0100 Subject: Only track method calls above a certain threshold This ensures we don't end up wasting resources by tracking method calls that only take a few microseconds. By default the threshold is 10 milliseconds but this can be changed using the gitlab.yml configuration file. --- config/gitlab.yml.example | 3 +++ lib/gitlab/metrics.rb | 4 ++++ lib/gitlab/metrics/instrumentation.rb | 8 +++++--- spec/lib/gitlab/metrics/instrumentation_spec.rb | 24 ++++++++++++++++++++++++ 4 files changed, 36 insertions(+), 3 deletions(-) diff --git a/config/gitlab.yml.example b/config/gitlab.yml.example index 38b0920890f..24f3f6f02dc 100644 --- a/config/gitlab.yml.example +++ b/config/gitlab.yml.example @@ -432,6 +432,9 @@ production: &base # pool_size: 16 # The timeout of a connection in seconds. # timeout: 10 + # The minimum amount of milliseconds a method call has to take before it's + # tracked. Defaults to 10. + # method_call_threshold: 10 development: <<: *base diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb index 007429fa194..4b92c3244fa 100644 --- a/lib/gitlab/metrics.rb +++ b/lib/gitlab/metrics.rb @@ -16,6 +16,10 @@ module Gitlab !!Settings.metrics['enabled'] end + def self.method_call_threshold + Settings.metrics['method_call_threshold'] || 10 + end + def self.pool @pool end diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index 91e09694cd8..ca2dffbc46a 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -99,9 +99,11 @@ module Gitlab retval = __send__(#{alias_name.inspect}, *args, &block) duration = (Time.now - start) * 1000.0 - trans.add_metric(Gitlab::Metrics::Instrumentation::SERIES, - { duration: duration }, - method: #{label.inspect}) + if duration >= Gitlab::Metrics.method_call_threshold + trans.add_metric(Gitlab::Metrics::Instrumentation::SERIES, + { duration: duration }, + method: #{label.inspect}) + end retval else diff --git a/spec/lib/gitlab/metrics/instrumentation_spec.rb b/spec/lib/gitlab/metrics/instrumentation_spec.rb index 5fe7a369cba..71d7209db0f 100644 --- a/spec/lib/gitlab/metrics/instrumentation_spec.rb +++ b/spec/lib/gitlab/metrics/instrumentation_spec.rb @@ -42,6 +42,9 @@ describe Gitlab::Metrics::Instrumentation do end it 'tracks the call duration upon calling the method' do + allow(Gitlab::Metrics).to receive(:method_call_threshold). + and_return(0) + allow(described_class).to receive(:transaction). and_return(transaction) @@ -51,6 +54,15 @@ describe Gitlab::Metrics::Instrumentation do @dummy.foo end + + it 'does not track method calls below a given duration threshold' do + allow(Gitlab::Metrics).to receive(:method_call_threshold). + and_return(100) + + expect(transaction).to_not receive(:add_metric) + + @dummy.foo + end end describe 'with metrics disabled' do @@ -84,6 +96,9 @@ describe Gitlab::Metrics::Instrumentation do end it 'tracks the call duration upon calling the method' do + allow(Gitlab::Metrics).to receive(:method_call_threshold). + and_return(0) + allow(described_class).to receive(:transaction). and_return(transaction) @@ -93,6 +108,15 @@ describe Gitlab::Metrics::Instrumentation do @dummy.new.bar end + + it 'does not track method calls below a given duration threshold' do + allow(Gitlab::Metrics).to receive(:method_call_threshold). + and_return(100) + + expect(transaction).to_not receive(:add_metric) + + @dummy.new.bar + end end describe 'with metrics disabled' do -- cgit v1.2.1 From a93a32a290c8e134763188ebd2b62935f5698e6c Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Tue, 15 Dec 2015 17:22:46 +0100 Subject: Support for instrumenting class hierarchies This will be used to (for example) instrument all ActiveRecord models. --- lib/gitlab/metrics/instrumentation.rb | 23 +++++++++++++++++ spec/lib/gitlab/metrics/instrumentation_spec.rb | 33 +++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/lib/gitlab/metrics/instrumentation.rb b/lib/gitlab/metrics/instrumentation.rb index ca2dffbc46a..06fc2f25948 100644 --- a/lib/gitlab/metrics/instrumentation.rb +++ b/lib/gitlab/metrics/instrumentation.rb @@ -31,6 +31,29 @@ module Gitlab instrument(:instance, mod, name) end + # Recursively instruments all subclasses of the given root module. + # + # This can be used to for example instrument all ActiveRecord models (as + # these all inherit from ActiveRecord::Base). + # + # This method can optionally take a block to pass to `instrument_methods` + # and `instrument_instance_methods`. + # + # root - The root module for which to instrument subclasses. The root + # module itself is not instrumented. + def self.instrument_class_hierarchy(root, &block) + visit = root.subclasses + + until visit.empty? + klass = visit.pop + + instrument_methods(klass, &block) + instrument_instance_methods(klass, &block) + + klass.subclasses.each { |c| visit << c } + end + end + # Instruments all public methods of a module. # # This method optionally takes a block that can be used to determine if a diff --git a/spec/lib/gitlab/metrics/instrumentation_spec.rb b/spec/lib/gitlab/metrics/instrumentation_spec.rb index 71d7209db0f..a7eab9d11cc 100644 --- a/spec/lib/gitlab/metrics/instrumentation_spec.rb +++ b/spec/lib/gitlab/metrics/instrumentation_spec.rb @@ -133,6 +133,39 @@ describe Gitlab::Metrics::Instrumentation do end end + describe '.instrument_class_hierarchy' do + before do + allow(Gitlab::Metrics).to receive(:enabled?).and_return(true) + + @child1 = Class.new(@dummy) do + def self.child1_foo; end + def child1_bar; end + end + + @child2 = Class.new(@child1) do + def self.child2_foo; end + def child2_bar; end + end + end + + it 'recursively instruments a class hierarchy' do + described_class.instrument_class_hierarchy(@dummy) + + expect(@child1).to respond_to(:_original_child1_foo) + expect(@child2).to respond_to(:_original_child2_foo) + + expect(@child1.method_defined?(:_original_child1_bar)).to eq(true) + expect(@child2.method_defined?(:_original_child2_bar)).to eq(true) + end + + it 'does not instrument the root module' do + described_class.instrument_class_hierarchy(@dummy) + + expect(@dummy).to_not respond_to(:_original_foo) + expect(@dummy.method_defined?(:_original_bar)).to eq(false) + end + end + describe '.instrument_methods' do before do allow(Gitlab::Metrics).to receive(:enabled?).and_return(true) -- cgit v1.2.1 From bcee44ad33d8a84822a8df068d47812594c445a3 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Tue, 15 Dec 2015 17:23:23 +0100 Subject: Instrument all ActiveRecord model methods This works by searching the raw source code for any references to commonly used ActiveRecord methods. While not bulletproof it saves us from having to list hundreds of methods by hand. It also ensures that (most) newly added methods are instrumented automatically. This _only_ instruments models defined in app/models, should a model reside somewhere else (e.g. somewhere in lib/) it _won't_ be instrumented. --- Gemfile | 1 + Gemfile.lock | 1 + config/initializers/metrics.rb | 21 +++++++++++++++++++++ 3 files changed, 23 insertions(+) diff --git a/Gemfile b/Gemfile index 90db2c43006..e9e5c7df075 100644 --- a/Gemfile +++ b/Gemfile @@ -210,6 +210,7 @@ gem 'net-ssh', '~> 3.0.1' # Metrics group :metrics do + gem 'method_source', '~> 0.8', require: false gem 'influxdb', '~> 0.2', require: false gem 'connection_pool', '~> 2.0', require: false end diff --git a/Gemfile.lock b/Gemfile.lock index 7d592ba93a7..3f301111224 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -887,6 +887,7 @@ DEPENDENCIES kaminari (~> 0.16.3) letter_opener (~> 1.1.2) mail_room (~> 0.6.1) + method_source (~> 0.8) minitest (~> 5.7.0) mousetrap-rails (~> 1.4.6) mysql2 (~> 0.3.16) diff --git a/config/initializers/metrics.rb b/config/initializers/metrics.rb index 0ac4299dcba..a47d2bf59a6 100644 --- a/config/initializers/metrics.rb +++ b/config/initializers/metrics.rb @@ -2,6 +2,7 @@ if Gitlab::Metrics.enabled? require 'influxdb' require 'socket' require 'connection_pool' + require 'method_source' # These are manually require'd so the classes are registered properly with # ActiveSupport. @@ -18,6 +19,26 @@ if Gitlab::Metrics.enabled? end end + # This instruments all methods residing in app/models that (appear to) use any + # of the ActiveRecord methods. This has to take place _after_ initializing as + # for some unknown reason calling eager_load! earlier breaks Devise. + Gitlab::Application.config.after_initialize do + Rails.application.eager_load! + + models = Rails.root.join('app', 'models').to_s + + regex = Regexp.union( + ActiveRecord::Querying.public_instance_methods(false).map(&:to_s) + ) + + Gitlab::Metrics::Instrumentation. + instrument_class_hierarchy(ActiveRecord::Base) do |_, method| + loc = method.source_location + + loc && loc[0].start_with?(models) && method.source =~ regex + end + end + Gitlab::Metrics::Instrumentation.configure do |config| config.instrument_instance_methods(Gitlab::Shell) -- cgit v1.2.1 From f181f05e8abd7b1066c11578193f6d7170764bf5 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 17 Dec 2015 17:17:18 +0100 Subject: Track object counts using the "allocations" Gem This allows us to track the counts of actual classes instead of "T_XXX" nodes. This is only enabled on CRuby as it uses CRuby specific APIs. --- Gemfile | 1 + Gemfile.lock | 2 ++ lib/gitlab/metrics.rb | 4 ++++ lib/gitlab/metrics/sampler.rb | 25 ++++++++++++++++++++++--- spec/lib/gitlab/metrics/sampler_spec.rb | 4 ++++ 5 files changed, 33 insertions(+), 3 deletions(-) diff --git a/Gemfile b/Gemfile index e9e5c7df075..94660ab5217 100644 --- a/Gemfile +++ b/Gemfile @@ -210,6 +210,7 @@ gem 'net-ssh', '~> 3.0.1' # Metrics group :metrics do + gem 'allocations', '~> 1.0', require: false, platform: :mri gem 'method_source', '~> 0.8', require: false gem 'influxdb', '~> 0.2', require: false gem 'connection_pool', '~> 2.0', require: false diff --git a/Gemfile.lock b/Gemfile.lock index 3f301111224..13e8168ee8a 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -49,6 +49,7 @@ GEM addressable (2.3.8) after_commit_queue (1.3.0) activerecord (>= 3.0) + allocations (1.0.1) annotate (2.6.10) activerecord (>= 3.2, <= 4.3) rake (~> 10.4) @@ -818,6 +819,7 @@ DEPENDENCIES acts-as-taggable-on (~> 3.4) addressable (~> 2.3.8) after_commit_queue + allocations (~> 1.0) annotate (~> 2.6.0) asana (~> 0.4.0) asciidoctor (~> 1.5.2) diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb index 4b92c3244fa..ce89be636d3 100644 --- a/lib/gitlab/metrics.rb +++ b/lib/gitlab/metrics.rb @@ -16,6 +16,10 @@ module Gitlab !!Settings.metrics['enabled'] end + def self.mri? + RUBY_ENGINE == 'ruby' + end + def self.method_call_threshold Settings.metrics['method_call_threshold'] || 10 end diff --git a/lib/gitlab/metrics/sampler.rb b/lib/gitlab/metrics/sampler.rb index 03afa6324dd..828ee1f8c62 100644 --- a/lib/gitlab/metrics/sampler.rb +++ b/lib/gitlab/metrics/sampler.rb @@ -13,6 +13,12 @@ module Gitlab @last_minor_gc = Delta.new(GC.stat[:minor_gc_count]) @last_major_gc = Delta.new(GC.stat[:major_gc_count]) + + if Gitlab::Metrics.mri? + require 'allocations' + + Allocations.start + end end def start @@ -52,9 +58,22 @@ module Gitlab new('file_descriptors', value: System.file_descriptor_count) end - def sample_objects - ObjectSpace.count_objects.each do |type, count| - @metrics << Metric.new('object_counts', { count: count }, type: type) + if Metrics.mri? + def sample_objects + sample = Allocations.to_hash + counts = sample.each_with_object({}) do |(klass, count), hash| + hash[klass.name] = count + end + + # Symbols aren't allocated so we'll need to add those manually. + counts['Symbol'] = Symbol.all_symbols.length + + counts.each do |name, count| + @metrics << Metric.new('object_counts', { count: count }, type: name) + end + end + else + def sample_objects end end diff --git a/spec/lib/gitlab/metrics/sampler_spec.rb b/spec/lib/gitlab/metrics/sampler_spec.rb index 319f287178d..69376c0b79b 100644 --- a/spec/lib/gitlab/metrics/sampler_spec.rb +++ b/spec/lib/gitlab/metrics/sampler_spec.rb @@ -3,6 +3,10 @@ require 'spec_helper' describe Gitlab::Metrics::Sampler do let(:sampler) { described_class.new(5) } + after do + Allocations.stop if Gitlab::Metrics.mri? + end + describe '#start' do it 'gathers a sample at a given interval' do expect(sampler).to receive(:sleep).with(5) -- cgit v1.2.1 From a3de46654b2fe0f02995913a771e6423bb584d64 Mon Sep 17 00:00:00 2001 From: Jose Torres Date: Sat, 19 Dec 2015 15:16:50 -0600 Subject: Adding how we manage CRIME vulnerability to security docs [ci skip] --- doc/security/README.md | 1 + doc/security/crime_vulnerability.md | 59 +++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 doc/security/crime_vulnerability.md diff --git a/doc/security/README.md b/doc/security/README.md index fba6013d9c1..7df7cef6aa5 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -6,3 +6,4 @@ - [Information exclusivity](information_exclusivity.md) - [Reset your root password](reset_root_password.md) - [User File Uploads](user_file_uploads.md) +- [How we manage the CRIME vulnerability](crime_vulnerability.md) diff --git a/doc/security/crime_vulnerability.md b/doc/security/crime_vulnerability.md new file mode 100644 index 00000000000..d716bff85a5 --- /dev/null +++ b/doc/security/crime_vulnerability.md @@ -0,0 +1,59 @@ +# How we manage the TLS protocol CRIME vulnerability + +> CRIME ("Compression Ratio Info-leak Made Easy") is a security exploit against +secret web cookies over connections using the HTTPS and SPDY protocols that also +use data compression.[1][2] When used to recover the content of secret +authentication cookies, it allows an attacker to perform session hijacking on an +authenticated web session, allowing the launching of further attacks. +([CRIME](https://en.wikipedia.org/w/index.php?title=CRIME&oldid=692423806)) + +### Description + +The TLS Protocol CRIME Vulnerability affects compression over HTTPS therefore +it warns against using SSL Compression, take gzip for example, or SPDY which +optionally uses compression as well. + +GitLab support both gzip and SPDY and manages the CRIME vulnerability by +deactivating gzip when https is enabled and not activating the compression +feature on SDPY. + +Take a look at our configuration file for NGINX if you'd like to explore how the +conditions are setup for gzip deactivation on this link: +[GitLab NGINX File](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb). + +For SPDY you can also watch how its implmented on NGINX at [GitLab NGINX File](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb) +but take into consideration the NGINX documentation on its default state here: +[Module ngx_http_spdy_module](http://nginx.org/en/docs/http/ngx_http_spdy_module.html). + + +### Nessus + +The Nessus scanner reports a possible CRIME vunerability for GitLab similar to the +following format: + + Description + + This remote service has one of two configurations that are known to be required for the CRIME attack: + SSL/TLS compression is enabled. + TLS advertises the SPDY protocol earlier than version 4. + + ... + + Output + + The following configuration indicates that the remote service may be vulnerable to the CRIME attack: + SPDY support earlier than version 4 is advertised. + +*[This](http://www.tenable.com/plugins/index.php?view=single&id=62565) is a complete description from Nessus.* + +From the report above its important to note that Nessus is only checkng if TLS +advertises the SPDY protocol earlier than version 4, it does not perform an +attack nor does it check if compression is enabled. With just this approach it +cannot tell that SPDY's compression is disabled and not subject to the CRIME +vulnerbility. + + +### Reference +* Nginx. "Module ngx_http_spdy_module", Fri. 18 Dec. +* Tenable Network Security, Inc. "Transport Layer Security (TLS) Protocol CRIME Vulnerability", Web. 15 Dec. +* Wikipedia contributors. "CRIME." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 25 Nov. 2015. Web. 15 Dec. 2015. \ No newline at end of file -- cgit v1.2.1 From 33ef13c641a63c4c82c5867a6f01a90c57f3aa04 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Mon, 21 Dec 2015 15:53:08 +0200 Subject: Init documentation on Triggers [ci skip] --- doc/ci/README.md | 39 ++++++++++++++++++++------------------- doc/ci/triggers/README.md | 11 +++++++++++ 2 files changed, 31 insertions(+), 19 deletions(-) create mode 100644 doc/ci/triggers/README.md diff --git a/doc/ci/README.md b/doc/ci/README.md index 5d9d7a81db3..aef6e69b7a6 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -2,34 +2,35 @@ ### User documentation -+ [Quick Start](quick_start/README.md) -+ [Configuring project (.gitlab-ci.yml)](yaml/README.md) -+ [Configuring runner](runners/README.md) -+ [Configuring deployment](deployment/README.md) -+ [Using Docker Images](docker/using_docker_images.md) -+ [Using Docker Build](docker/using_docker_build.md) -+ [Using Variables](variables/README.md) -+ [Using SSH keys](ssh_keys/README.md) +* [Quick Start](quick_start/README.md) +* [Configuring project (.gitlab-ci.yml)](yaml/README.md) +* [Configuring runner](runners/README.md) +* [Configuring deployment](deployment/README.md) +* [Using Docker Images](docker/using_docker_images.md) +* [Using Docker Build](docker/using_docker_build.md) +* [Using Variables](variables/README.md) +* [Using SSH keys](ssh_keys/README.md) +* [Triggering builds through the API](triggers/README.md) ### Languages -+ [Testing PHP](languages/php.md) +* [Testing PHP](languages/php.md) ### Services -+ [Using MySQL](services/mysql.md) -+ [Using PostgreSQL](services/postgres.md) -+ [Using Redis](services/redis.md) -+ [Using Other Services](docker/using_docker_images.md#how-to-use-other-images-as-services) +* [Using MySQL](services/mysql.md) +* [Using PostgreSQL](services/postgres.md) +* [Using Redis](services/redis.md) +* [Using Other Services](docker/using_docker_images.md#how-to-use-other-images-as-services) ### Examples -+ [Test and deploy Ruby applications to Heroku](examples/test-and-deploy-ruby-application-to-heroku.md) -+ [Test and deploy Python applications to Heroku](examples/test-and-deploy-python-application-to-heroku.md) -+ [Test Clojure applications](examples/test-clojure-application.md) -+ Help your favorite programming language and GitLab by sending a merge request with a guide for that language. +* [Test and deploy Ruby applications to Heroku](examples/test-and-deploy-ruby-application-to-heroku.md) +* [Test and deploy Python applications to Heroku](examples/test-and-deploy-python-application-to-heroku.md) +* [Test Clojure applications](examples/test-clojure-application.md) +* Help your favorite programming language and GitLab by sending a merge request with a guide for that language. ### Administrator documentation -+ [User permissions](permissions/README.md) -+ [API](api/README.md) +* [User permissions](permissions/README.md) +* [API](api/README.md) diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md new file mode 100644 index 00000000000..5990d74fda1 --- /dev/null +++ b/doc/ci/triggers/README.md @@ -0,0 +1,11 @@ +# Triggering Builds through the API + +_**Note:** This feature was [introduced][ci-229] in GitLab CE 7.14_ + +Triggers can be used to force a rebuild of a specific branch or tag with an API +call. + +## Add a trigger + + +[ci-229]: https://gitlab.com/gitlab-org/gitlab-ci/merge_requests/229 -- cgit v1.2.1 From 70dfa3a721700cf0151a7d097933d75684e69fc9 Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Mon, 21 Dec 2015 13:06:09 -0500 Subject: open and close issue via ajax request. With tests --- app/assets/javascripts/issue.js.coffee | 30 +++++++++++++++++++++ app/helpers/issues_helper.rb | 4 +++ app/views/projects/issues/show.html.haml | 13 +++------ spec/javascripts/fixtures/issues_show.html.haml | 5 +++- spec/javascripts/issue_spec.js.coffee | 36 +++++++++++++++++++++++++ 5 files changed, 78 insertions(+), 10 deletions(-) diff --git a/app/assets/javascripts/issue.js.coffee b/app/assets/javascripts/issue.js.coffee index eff80bf63bb..8d028268b81 100644 --- a/app/assets/javascripts/issue.js.coffee +++ b/app/assets/javascripts/issue.js.coffee @@ -8,11 +8,41 @@ class @Issue if $("a.btn-close").length @initTaskList() + @initIssueBtnEventListeners() initTaskList: -> $('.detail-page-description .js-task-list-container').taskList('enable') $(document).on 'tasklist:changed', '.detail-page-description .js-task-list-container', @updateTaskList + initIssueBtnEventListeners: -> + $("a.btn-close, a.btn-reopen").on "click", (e) -> + e.preventDefault() + e.stopImmediatePropagation() + $this = $(this) + isClose = $this.hasClass('btn-close') + $this.prop("disabled", true) + url = $this.data('url') + $.ajax + type: 'PUT' + url: url, + error: (jqXHR, textStatus, errorThrown) -> + issueStatus = if isClose then 'close' else 'open' + console.log("Cannot #{issueStatus} this issue, at this time.") + success: (data, textStatus, jqXHR) -> + if data.saved + $this.addClass('hidden') + if isClose + $('a.btn-reopen').removeClass('hidden') + $('div.issue-box-closed').removeClass('hidden') + $('div.issue-box-open').addClass('hidden') + else + $('a.btn-close').removeClass('hidden') + $('div.issue-box-closed').addClass('hidden') + $('div.issue-box-open').removeClass('hidden') + else + console.log("Did not work") + $this.prop('disabled', false) + disableTaskList: -> $('.detail-page-description .js-task-list-container').taskList('disable') $(document).off 'tasklist:changed', '.detail-page-description .js-task-list-container' diff --git a/app/helpers/issues_helper.rb b/app/helpers/issues_helper.rb index d2186427dba..1f0f6aeeac2 100644 --- a/app/helpers/issues_helper.rb +++ b/app/helpers/issues_helper.rb @@ -69,6 +69,10 @@ module IssuesHelper end end + def issue_button_visibility(issue, closed) + return 'hidden' if issue.closed? == closed + end + def issue_to_atom(xml, issue) xml.entry do xml.id namespace_project_issue_url(issue.project.namespace, diff --git a/app/views/projects/issues/show.html.haml b/app/views/projects/issues/show.html.haml index 2fe6f88b2a9..9444a1c1d84 100644 --- a/app/views/projects/issues/show.html.haml +++ b/app/views/projects/issues/show.html.haml @@ -3,11 +3,8 @@ .issue .detail-page-header - .status-box{ class: status_box_class(@issue) } - - if @issue.closed? - Closed - - else - Open + .status-box{ class: "status-box-closed #{issue_button_visibility(@issue, false)}"} Closed + .status-box{ class: "status-box-open #{issue_button_visibility(@issue, true)}"} Open %span.identifier Issue ##{@issue.iid} %span.creator @@ -27,10 +24,8 @@ = icon('plus') New Issue - if can?(current_user, :update_issue, @issue) - - if @issue.closed? - = link_to 'Reopen', issue_path(@issue, issue: {state_event: :reopen}, status_only: true), method: :put, class: 'btn btn-grouped btn-reopen' - - else - = link_to 'Close', issue_path(@issue, issue: {state_event: :close}, status_only: true), method: :put, class: 'btn btn-grouped btn-close', title: 'Close Issue' + = link_to 'Reopen', '#', data: {no_turbolink: true, url: issue_path(@issue, issue: {state_event: :reopen}, status_only: true, format: 'json')}, class: "btn btn-grouped btn-reopen #{issue_button_visibility(@issue, false)}", title: 'Reopen Issue' + = link_to 'Close', '#', data: {no_turbolink: true, url: issue_path(@issue, issue: {state_event: :close}, status_only: true, format: 'json')}, class: "btn btn-grouped btn-close #{issue_button_visibility(@issue, true)}", title: 'Close Issue' = link_to edit_namespace_project_issue_path(@project.namespace, @project, @issue), class: 'btn btn-grouped issuable-edit' do = icon('pencil-square-o') diff --git a/spec/javascripts/fixtures/issues_show.html.haml b/spec/javascripts/fixtures/issues_show.html.haml index 8447dfdda32..6c985a32966 100644 --- a/spec/javascripts/fixtures/issues_show.html.haml +++ b/spec/javascripts/fixtures/issues_show.html.haml @@ -1,4 +1,7 @@ -%a.btn-close +.issue-box.issue-box-open Open +.issue-box.issue-box-closed.hidden Closed +%a.btn-close{"data-url" => "http://gitlab/issues/6/close"} Close +%a.btn-reopen.hidden{"data-url" => "http://gitlab/issues/6/reopen"} Reopen .detail-page-description .description.js-task-list-container diff --git a/spec/javascripts/issue_spec.js.coffee b/spec/javascripts/issue_spec.js.coffee index 268e4c68c31..60df3a8878b 100644 --- a/spec/javascripts/issue_spec.js.coffee +++ b/spec/javascripts/issue_spec.js.coffee @@ -20,3 +20,39 @@ describe 'Issue', -> expect(req.data.issue.description).not.toBe(null) $('.js-task-list-field').trigger('tasklist:changed') +describe 'reopen/close issue', -> + fixture.preload('issues_show.html') + beforeEach -> + fixture.load('issues_show.html') + @issue = new Issue() + it 'closes an issue', -> + $.ajax = (obj) -> + expect(obj.type).toBe('PUT') + expect(obj.url).toBe('http://gitlab/issues/6/close') + obj.success saved: true + $btnClose = $('a.btn-close') + $btnReopen = $('a.btn-reopen') + expect($btnReopen.hasClass('hidden')).toBe(true) + expect($btnClose.text()).toBe('Close') + expect(typeof $btnClose.prop('disabled')).toBe('undefined') + $btnClose.trigger('click') + expect($btnClose.hasClass('hidden')).toBe(true) + expect($btnReopen.hasClass('hidden')).toBe(false) + expect($btnClose.prop('disabled')).toBe(false) + expect($('div.issue-box-open').hasClass('hidden')).toBe(true) + expect($('div.issue-box-closed').hasClass('hidden')).toBe(false) + it 'reopens an issue', -> + $.ajax = (obj) -> + expect(obj.type).toBe('PUT') + expect(obj.url).toBe('http://gitlab/issues/6/reopen') + obj.success saved: true + $btnClose = $('a.btn-close') + $btnReopen = $('a.btn-reopen') + expect(typeof $btnReopen.prop('disabled')).toBe('undefined') + expect($btnReopen.text()).toBe('Reopen') + $btnReopen.trigger('click') + expect($btnReopen.hasClass('hidden')).toBe(true) + expect($btnClose.hasClass('hidden')).toBe(false) + expect($btnReopen.prop('disabled')).toBe(false) + expect($('div.issue-box-open').hasClass('hidden')).toBe(false) + expect($('div.issue-box-closed').hasClass('hidden')).toBe(true) \ No newline at end of file -- cgit v1.2.1 From d5e9436033d75da74c40ced450e060c8a5c307f9 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Mon, 21 Dec 2015 20:41:51 +0200 Subject: Document triggers in yaml/README.md [ci skip] --- doc/ci/yaml/README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index fd0d49de4e4..6862116cc5b 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -66,6 +66,7 @@ There are a few reserved `keywords` that **cannot** be used as job names: | before_script | no | Define commands that run before each job's script | | variables | no | Define build variables | | cache | no | Define list of files that should be cached between subsequent runs | +| trigger | no | Force a rebuild of a specific branch or tag with an API call | ### image and services @@ -152,6 +153,32 @@ cache: - binaries/ ``` +### trigger + +Triggers can be used to force a rebuild of a specific branch or tag with an API +call. You can add a trigger by visiting the project's **Settings > Triggers**. + +Every new trigger you create, gets assigned a different token which you can +then use inside your `.gitlab-ci.yml`: + +```yaml +trigger: + stage: deploy + script: + - "curl -X POST -F token=TOKEN -F ref=master https://gitlab.com/api/v3/projects/9/trigger/builds" +``` + +In the example above, a rebuild on master branch will be triggered after all +previous stages build successfully (denoted by `stage: deploy`). + +You can find the endpoint containing the ID of the project on the **Triggers** +page. + +_**Warning:** Be careful how you set up your triggers, because you could end up +in an infinite loop._ + +Read more in the dedicated [triggers documentation](../triggers/README.md). + ## Jobs `.gitlab-ci.yml` allows you to specify an unlimited number of jobs. Each job -- cgit v1.2.1 From 531d06d170fd5cfcb6d4f6c919034c49266e42e9 Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Mon, 21 Dec 2015 16:16:04 -0500 Subject: removes `console.log`s --- app/assets/javascripts/issue.js.coffee | 2 -- 1 file changed, 2 deletions(-) diff --git a/app/assets/javascripts/issue.js.coffee b/app/assets/javascripts/issue.js.coffee index 8d028268b81..1d57c24587d 100644 --- a/app/assets/javascripts/issue.js.coffee +++ b/app/assets/javascripts/issue.js.coffee @@ -27,7 +27,6 @@ class @Issue url: url, error: (jqXHR, textStatus, errorThrown) -> issueStatus = if isClose then 'close' else 'open' - console.log("Cannot #{issueStatus} this issue, at this time.") success: (data, textStatus, jqXHR) -> if data.saved $this.addClass('hidden') @@ -40,7 +39,6 @@ class @Issue $('div.issue-box-closed').addClass('hidden') $('div.issue-box-open').removeClass('hidden') else - console.log("Did not work") $this.prop('disabled', false) disableTaskList: -> -- cgit v1.2.1 From fec7c99e2f2221f22680c140b7d9b1d959d8aeb0 Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Mon, 21 Dec 2015 16:27:52 -0500 Subject: updates tests style for four-phase-testing as per: https://robots.thoughtbot.com/four-phase-test --- spec/javascripts/issue_spec.js.coffee | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/spec/javascripts/issue_spec.js.coffee b/spec/javascripts/issue_spec.js.coffee index 60df3a8878b..f8e60565b9a 100644 --- a/spec/javascripts/issue_spec.js.coffee +++ b/spec/javascripts/issue_spec.js.coffee @@ -30,29 +30,45 @@ describe 'reopen/close issue', -> expect(obj.type).toBe('PUT') expect(obj.url).toBe('http://gitlab/issues/6/close') obj.success saved: true + + # setup $btnClose = $('a.btn-close') $btnReopen = $('a.btn-reopen') expect($btnReopen.hasClass('hidden')).toBe(true) expect($btnClose.text()).toBe('Close') expect(typeof $btnClose.prop('disabled')).toBe('undefined') + + # excerize $btnClose.trigger('click') + + # verify expect($btnClose.hasClass('hidden')).toBe(true) expect($btnReopen.hasClass('hidden')).toBe(false) expect($btnClose.prop('disabled')).toBe(false) expect($('div.issue-box-open').hasClass('hidden')).toBe(true) expect($('div.issue-box-closed').hasClass('hidden')).toBe(false) + + # teardown it 'reopens an issue', -> $.ajax = (obj) -> expect(obj.type).toBe('PUT') expect(obj.url).toBe('http://gitlab/issues/6/reopen') obj.success saved: true + + # setup $btnClose = $('a.btn-close') $btnReopen = $('a.btn-reopen') expect(typeof $btnReopen.prop('disabled')).toBe('undefined') expect($btnReopen.text()).toBe('Reopen') + + # excerize $btnReopen.trigger('click') + + # verify expect($btnReopen.hasClass('hidden')).toBe(true) expect($btnClose.hasClass('hidden')).toBe(false) expect($btnReopen.prop('disabled')).toBe(false) expect($('div.issue-box-open').hasClass('hidden')).toBe(false) - expect($('div.issue-box-closed').hasClass('hidden')).toBe(true) \ No newline at end of file + expect($('div.issue-box-closed').hasClass('hidden')).toBe(true) + + # teardown \ No newline at end of file -- cgit v1.2.1 From 3e7e7ae0aceaa8a805093841b07b9b5d9f911cb1 Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Mon, 21 Dec 2015 16:35:03 -0500 Subject: clarifies tests with methods like `toBeVisible()` etc. --- spec/javascripts/issue_spec.js.coffee | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/spec/javascripts/issue_spec.js.coffee b/spec/javascripts/issue_spec.js.coffee index f8e60565b9a..f50640d1821 100644 --- a/spec/javascripts/issue_spec.js.coffee +++ b/spec/javascripts/issue_spec.js.coffee @@ -34,7 +34,7 @@ describe 'reopen/close issue', -> # setup $btnClose = $('a.btn-close') $btnReopen = $('a.btn-reopen') - expect($btnReopen.hasClass('hidden')).toBe(true) + expect($btnReopen.toBeHidden()) expect($btnClose.text()).toBe('Close') expect(typeof $btnClose.prop('disabled')).toBe('undefined') @@ -42,11 +42,10 @@ describe 'reopen/close issue', -> $btnClose.trigger('click') # verify - expect($btnClose.hasClass('hidden')).toBe(true) - expect($btnReopen.hasClass('hidden')).toBe(false) - expect($btnClose.prop('disabled')).toBe(false) - expect($('div.issue-box-open').hasClass('hidden')).toBe(true) - expect($('div.issue-box-closed').hasClass('hidden')).toBe(false) + expect($btnClose.toBeHidden()) + expect($btnReopen.toBeVisible()) + expect($('div.issue-box-open').toBeVisible()) + expect($('div.issue-box-closed').toBeHidden()) # teardown it 'reopens an issue', -> @@ -58,17 +57,15 @@ describe 'reopen/close issue', -> # setup $btnClose = $('a.btn-close') $btnReopen = $('a.btn-reopen') - expect(typeof $btnReopen.prop('disabled')).toBe('undefined') expect($btnReopen.text()).toBe('Reopen') # excerize $btnReopen.trigger('click') # verify - expect($btnReopen.hasClass('hidden')).toBe(true) - expect($btnClose.hasClass('hidden')).toBe(false) - expect($btnReopen.prop('disabled')).toBe(false) - expect($('div.issue-box-open').hasClass('hidden')).toBe(false) - expect($('div.issue-box-closed').hasClass('hidden')).toBe(true) + expect($btnReopen.toBeHidden()) + expect($btnClose.toBeVisible()) + expect($('div.issue-box-open').toBeVisible()) + expect($('div.issue-box-closed').toBeHidden()) # teardown \ No newline at end of file -- cgit v1.2.1 From 1f5b8e88f3fedc382555e89b5a6ba9be57d551c7 Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Mon, 21 Dec 2015 16:45:52 -0500 Subject: changes `data-url` to `href` for javascript url grabbing --- app/assets/javascripts/issue.js.coffee | 3 ++- app/views/projects/issues/show.html.haml | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/app/assets/javascripts/issue.js.coffee b/app/assets/javascripts/issue.js.coffee index 1d57c24587d..ba5205fb471 100644 --- a/app/assets/javascripts/issue.js.coffee +++ b/app/assets/javascripts/issue.js.coffee @@ -16,12 +16,13 @@ class @Issue initIssueBtnEventListeners: -> $("a.btn-close, a.btn-reopen").on "click", (e) -> + console.log('closing') e.preventDefault() e.stopImmediatePropagation() $this = $(this) isClose = $this.hasClass('btn-close') $this.prop("disabled", true) - url = $this.data('url') + url = $this.attr('href') $.ajax type: 'PUT' url: url, diff --git a/app/views/projects/issues/show.html.haml b/app/views/projects/issues/show.html.haml index 9444a1c1d84..4ca122ba55f 100644 --- a/app/views/projects/issues/show.html.haml +++ b/app/views/projects/issues/show.html.haml @@ -24,8 +24,8 @@ = icon('plus') New Issue - if can?(current_user, :update_issue, @issue) - = link_to 'Reopen', '#', data: {no_turbolink: true, url: issue_path(@issue, issue: {state_event: :reopen}, status_only: true, format: 'json')}, class: "btn btn-grouped btn-reopen #{issue_button_visibility(@issue, false)}", title: 'Reopen Issue' - = link_to 'Close', '#', data: {no_turbolink: true, url: issue_path(@issue, issue: {state_event: :close}, status_only: true, format: 'json')}, class: "btn btn-grouped btn-close #{issue_button_visibility(@issue, true)}", title: 'Close Issue' + = link_to 'Reopen', issue_path(@issue, issue: {state_event: :reopen}, status_only: true, format: 'json'), data: {no_turbolink: true}, class: "btn btn-grouped btn-reopen #{issue_button_visibility(@issue, false)}", title: 'Reopen Issue' + = link_to 'Close', issue_path(@issue, issue: {state_event: :close}, status_only: true, format: 'json'), data: {no_turbolink: true}, class: "btn btn-grouped btn-close #{issue_button_visibility(@issue, true)}", title: 'Close Issue' = link_to edit_namespace_project_issue_path(@project.namespace, @project, @issue), class: 'btn btn-grouped issuable-edit' do = icon('pencil-square-o') -- cgit v1.2.1 From 801b801bf08a0bc6df8ba13775f3050d091ce84d Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Mon, 21 Dec 2015 16:46:36 -0500 Subject: removes console logs --- app/assets/javascripts/issue.js.coffee | 1 - 1 file changed, 1 deletion(-) diff --git a/app/assets/javascripts/issue.js.coffee b/app/assets/javascripts/issue.js.coffee index ba5205fb471..1423daa281b 100644 --- a/app/assets/javascripts/issue.js.coffee +++ b/app/assets/javascripts/issue.js.coffee @@ -16,7 +16,6 @@ class @Issue initIssueBtnEventListeners: -> $("a.btn-close, a.btn-reopen").on "click", (e) -> - console.log('closing') e.preventDefault() e.stopImmediatePropagation() $this = $(this) -- cgit v1.2.1 From 453479143dd4784d7284b2e6f98694ff8fc18ce8 Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Mon, 21 Dec 2015 17:03:28 -0500 Subject: adds alerts for when http request errors out in some way. --- app/assets/javascripts/issue.js.coffee | 2 ++ spec/javascripts/issue_spec.js.coffee | 11 +---------- 2 files changed, 3 insertions(+), 10 deletions(-) diff --git a/app/assets/javascripts/issue.js.coffee b/app/assets/javascripts/issue.js.coffee index 1423daa281b..2e0ef99a587 100644 --- a/app/assets/javascripts/issue.js.coffee +++ b/app/assets/javascripts/issue.js.coffee @@ -27,6 +27,7 @@ class @Issue url: url, error: (jqXHR, textStatus, errorThrown) -> issueStatus = if isClose then 'close' else 'open' + new Flash("Issues update failed", 'alert') success: (data, textStatus, jqXHR) -> if data.saved $this.addClass('hidden') @@ -39,6 +40,7 @@ class @Issue $('div.issue-box-closed').addClass('hidden') $('div.issue-box-open').removeClass('hidden') else + new Flash("Issues update failed", 'alert') $this.prop('disabled', false) disableTaskList: -> diff --git a/spec/javascripts/issue_spec.js.coffee b/spec/javascripts/issue_spec.js.coffee index f50640d1821..ef78cfbc653 100644 --- a/spec/javascripts/issue_spec.js.coffee +++ b/spec/javascripts/issue_spec.js.coffee @@ -31,41 +31,32 @@ describe 'reopen/close issue', -> expect(obj.url).toBe('http://gitlab/issues/6/close') obj.success saved: true - # setup $btnClose = $('a.btn-close') $btnReopen = $('a.btn-reopen') expect($btnReopen.toBeHidden()) expect($btnClose.text()).toBe('Close') expect(typeof $btnClose.prop('disabled')).toBe('undefined') - # excerize $btnClose.trigger('click') - # verify expect($btnClose.toBeHidden()) expect($btnReopen.toBeVisible()) expect($('div.issue-box-open').toBeVisible()) expect($('div.issue-box-closed').toBeHidden()) - # teardown it 'reopens an issue', -> $.ajax = (obj) -> expect(obj.type).toBe('PUT') expect(obj.url).toBe('http://gitlab/issues/6/reopen') obj.success saved: true - # setup $btnClose = $('a.btn-close') $btnReopen = $('a.btn-reopen') expect($btnReopen.text()).toBe('Reopen') - # excerize $btnReopen.trigger('click') - # verify expect($btnReopen.toBeHidden()) expect($btnClose.toBeVisible()) expect($('div.issue-box-open').toBeVisible()) - expect($('div.issue-box-closed').toBeHidden()) - - # teardown \ No newline at end of file + expect($('div.issue-box-closed').toBeHidden()) \ No newline at end of file -- cgit v1.2.1 From 3ea3d9ef284fcfaaa1b631e98926df7dae4129c9 Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Mon, 21 Dec 2015 17:10:27 -0500 Subject: changes `issue-box` to `status-box` since html was changed as well --- app/assets/javascripts/issue.js.coffee | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/app/assets/javascripts/issue.js.coffee b/app/assets/javascripts/issue.js.coffee index 2e0ef99a587..1a9e03e4ee3 100644 --- a/app/assets/javascripts/issue.js.coffee +++ b/app/assets/javascripts/issue.js.coffee @@ -6,7 +6,7 @@ class @Issue # Prevent duplicate event bindings @disableTaskList() - if $("a.btn-close").length + if $('a.btn-close').length @initTaskList() @initIssueBtnEventListeners() @@ -15,32 +15,32 @@ class @Issue $(document).on 'tasklist:changed', '.detail-page-description .js-task-list-container', @updateTaskList initIssueBtnEventListeners: -> - $("a.btn-close, a.btn-reopen").on "click", (e) -> + $('a.btn-close, a.btn-reopen').on 'click', (e) -> e.preventDefault() e.stopImmediatePropagation() $this = $(this) isClose = $this.hasClass('btn-close') - $this.prop("disabled", true) + $this.prop('disabled', true) url = $this.attr('href') $.ajax type: 'PUT' url: url, error: (jqXHR, textStatus, errorThrown) -> issueStatus = if isClose then 'close' else 'open' - new Flash("Issues update failed", 'alert') + new Flash('Issues update failed', 'alert') success: (data, textStatus, jqXHR) -> if data.saved $this.addClass('hidden') if isClose $('a.btn-reopen').removeClass('hidden') - $('div.issue-box-closed').removeClass('hidden') - $('div.issue-box-open').addClass('hidden') + $('div.status-box-closed').removeClass('hidden') + $('div.status-box-open').addClass('hidden') else $('a.btn-close').removeClass('hidden') - $('div.issue-box-closed').addClass('hidden') - $('div.issue-box-open').removeClass('hidden') + $('div.status-box-closed').addClass('hidden') + $('div.status-box-open').removeClass('hidden') else - new Flash("Issues update failed", 'alert') + new Flash('Issues update failed', 'alert') $this.prop('disabled', false) disableTaskList: -> -- cgit v1.2.1 From 9a166c1ed85dc4ba96cbd43098d71c83bda2a93a Mon Sep 17 00:00:00 2001 From: Raymii Date: Tue, 22 Dec 2015 05:29:19 +0000 Subject: `update-init-script` was listed two times. removed one without explanation. --- doc/update/8.2-to-8.3.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/update/8.2-to-8.3.md b/doc/update/8.2-to-8.3.md index c4661dc16af..3748941b781 100644 --- a/doc/update/8.2-to-8.3.md +++ b/doc/update/8.2-to-8.3.md @@ -99,8 +99,6 @@ sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production # Clean up assets and cache sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production -# Update init.d script -sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` ### 7. Update configuration files -- cgit v1.2.1 From 947a09583b25140ee386229f11c82a0984be2451 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Wed, 23 Dec 2015 13:55:15 -0500 Subject: Add Open Graph meta tags --- app/views/layouts/_head.html.haml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/app/views/layouts/_head.html.haml b/app/views/layouts/_head.html.haml index 74174a72f5a..be88ad9e2a6 100644 --- a/app/views/layouts/_head.html.haml +++ b/app/views/layouts/_head.html.haml @@ -1,10 +1,14 @@ - page_title "GitLab" -%head +%head{prefix: "og: http://ogp.me/ns#"} %meta{charset: "utf-8"} %meta{'http-equiv' => 'X-UA-Compatible', content: 'IE=edge'} %meta{content: "GitLab Community Edition", name: "description"} %meta{name: 'referrer', content: 'origin-when-cross-origin'} + -# Open Graph - http://ogp.me/ + %meta{property: 'og:title', content: page_title} + %meta{property: 'og:url', content: url_for(only_path: false)} + %title= page_title = favicon_link_tag 'favicon.ico' -- cgit v1.2.1 From 3f452f539b909d1a379a4f3c5d48307246a14fff Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Wed, 23 Dec 2015 16:45:37 -0500 Subject: Add gitlab_logo.png, remove brand_logo.png --- app/assets/images/brand_logo.png | Bin 27059 -> 0 bytes app/assets/images/gitlab_logo.png | Bin 0 -> 5189 bytes 2 files changed, 0 insertions(+), 0 deletions(-) delete mode 100644 app/assets/images/brand_logo.png create mode 100644 app/assets/images/gitlab_logo.png diff --git a/app/assets/images/brand_logo.png b/app/assets/images/brand_logo.png deleted file mode 100644 index 9c564bb6141..00000000000 Binary files a/app/assets/images/brand_logo.png and /dev/null differ diff --git a/app/assets/images/gitlab_logo.png b/app/assets/images/gitlab_logo.png new file mode 100644 index 00000000000..0c157546b9c Binary files /dev/null and b/app/assets/images/gitlab_logo.png differ -- cgit v1.2.1 From b26eb782f5536d8c383aebe3d65571fb16a20bcd Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Wed, 23 Dec 2015 16:56:27 -0500 Subject: Add page descriptions and images A limited number of pages have defined their own descriptions, but otherwise we default to the Project's description (if `@project` is set), or the old `brand_title` fallback. The image will either be the uploaded project icon (never a generated one), the user's uploaded icon or Gravatar, or, finally, the GitLab logo. --- app/helpers/page_layout_helper.rb | 50 ++++++++++++++ app/views/layouts/_head.html.haml | 12 ++-- app/views/projects/issues/show.html.haml | 4 +- app/views/projects/merge_requests/_show.html.haml | 4 +- app/views/projects/milestones/show.html.haml | 4 +- app/views/users/show.html.haml | 5 +- spec/helpers/page_layout_helper_spec.rb | 79 +++++++++++++++++++++++ 7 files changed, 149 insertions(+), 9 deletions(-) create mode 100644 spec/helpers/page_layout_helper_spec.rb diff --git a/app/helpers/page_layout_helper.rb b/app/helpers/page_layout_helper.rb index 9bf750124b2..6b16528dde6 100644 --- a/app/helpers/page_layout_helper.rb +++ b/app/helpers/page_layout_helper.rb @@ -8,6 +8,56 @@ module PageLayoutHelper @page_title.join(" \u00b7 ") end + # Define or get a description for the current page + # + # description - String (default: nil) + # + # If this helper is called multiple times with an argument, only the last + # description will be returned when called without an argument. Descriptions + # have newlines replaced with spaces and all HTML tags are sanitized. + # + # Examples: + # + # page_description # => "GitLab Community Edition" + # page_description("Foo") + # page_description # => "Foo" + # + # page_description("Bar\nBaz") + # page_description # => "Bar Baz" + # + # Returns an HTML-safe String. + def page_description(description = nil) + @page_description ||= page_description_default + + if description.present? + @page_description = description + else + sanitize(@page_description.squish, tags: []) + end + end + + # Default value for page_description when one hasn't been defined manually by + # a view + def page_description_default + if @project + @project.description + else + brand_title + end + end + + def page_image + default = image_url('gitlab_logo.png') + + if @project + @project.avatar_url || default + elsif @user + avatar_icon(@user) + else + default + end + end + def header_title(title = nil, title_url = nil) if title @header_title = title diff --git a/app/views/layouts/_head.html.haml b/app/views/layouts/_head.html.haml index be88ad9e2a6..ad1dacac1c8 100644 --- a/app/views/layouts/_head.html.haml +++ b/app/views/layouts/_head.html.haml @@ -1,13 +1,17 @@ - page_title "GitLab" + %head{prefix: "og: http://ogp.me/ns#"} %meta{charset: "utf-8"} %meta{'http-equiv' => 'X-UA-Compatible', content: 'IE=edge'} - %meta{content: "GitLab Community Edition", name: "description"} - %meta{name: 'referrer', content: 'origin-when-cross-origin'} + + %meta{name: "description", content: page_description} + %meta{name: 'referrer', content: 'origin-when-cross-origin'} -# Open Graph - http://ogp.me/ - %meta{property: 'og:title', content: page_title} - %meta{property: 'og:url', content: url_for(only_path: false)} + %meta{property: 'og:title', content: page_title} + %meta{property: 'og:description', content: page_description} + %meta{property: 'og:image', content: page_image} + %meta{property: 'og:url', content: url_for(only_path: false)} %title= page_title diff --git a/app/views/projects/issues/show.html.haml b/app/views/projects/issues/show.html.haml index b6efa05a1ae..f2a261ab426 100644 --- a/app/views/projects/issues/show.html.haml +++ b/app/views/projects/issues/show.html.haml @@ -1,4 +1,6 @@ -- page_title "#{@issue.title} (##{@issue.iid})", "Issues" +- page_title "#{@issue.title} (##{@issue.iid})", "Issues" +- page_description @issue.description + = render "header_title" .issue diff --git a/app/views/projects/merge_requests/_show.html.haml b/app/views/projects/merge_requests/_show.html.haml index e9ffbd06be2..75f44557964 100644 --- a/app/views/projects/merge_requests/_show.html.haml +++ b/app/views/projects/merge_requests/_show.html.haml @@ -1,4 +1,6 @@ -- page_title "#{@merge_request.title} (##{@merge_request.iid})", "Merge Requests" +- page_title "#{@merge_request.title} (##{@merge_request.iid})", "Merge Requests" +- page_description @merge_request.description + = render "header_title" - if params[:view] == 'parallel' diff --git a/app/views/projects/milestones/show.html.haml b/app/views/projects/milestones/show.html.haml index 7e73ae274e9..1670ea8741a 100644 --- a/app/views/projects/milestones/show.html.haml +++ b/app/views/projects/milestones/show.html.haml @@ -1,4 +1,6 @@ -- page_title @milestone.title, "Milestones" +- page_title @milestone.title, "Milestones" +- page_description @milestone.description + = render "header_title" .detail-page-header diff --git a/app/views/users/show.html.haml b/app/views/users/show.html.haml index b7a7eb4e6f7..0bca8177e14 100644 --- a/app/views/users/show.html.haml +++ b/app/views/users/show.html.haml @@ -1,5 +1,6 @@ -- page_title @user.name -- header_title @user.name, user_path(@user) +- page_title @user.name +- page_description @user.bio +- header_title @user.name, user_path(@user) = content_for :meta_tags do = auto_discovery_link_tag(:atom, user_url(@user, format: :atom), title: "#{@user.name} activity") diff --git a/spec/helpers/page_layout_helper_spec.rb b/spec/helpers/page_layout_helper_spec.rb new file mode 100644 index 00000000000..a60b8f508f2 --- /dev/null +++ b/spec/helpers/page_layout_helper_spec.rb @@ -0,0 +1,79 @@ +require 'rails_helper' + +describe PageLayoutHelper do + describe 'page_description' do + it 'defaults to value returned by page_description_default helper' do + allow(helper).to receive(:page_description_default).and_return('Foo') + + expect(helper.page_description).to eq 'Foo' + end + + it 'returns the last-pushed description' do + helper.page_description('Foo') + helper.page_description('Bar') + helper.page_description('Baz') + + expect(helper.page_description).to eq 'Baz' + end + + it 'squishes multiple newlines' do + helper.page_description("Foo\nBar\nBaz") + + expect(helper.page_description).to eq 'Foo Bar Baz' + end + + it 'sanitizes all HTML' do + helper.page_description("Bold

Header

") + + expect(helper.page_description).to eq 'Bold Header' + end + end + + describe 'page_description_default' do + it 'uses Project description when available' do + project = double(description: 'Project Description') + helper.instance_variable_set(:@project, project) + + expect(helper.page_description_default).to eq 'Project Description' + end + + it 'falls back to brand_title' do + allow(helper).to receive(:brand_title).and_return('Brand Title') + + expect(helper.page_description_default).to eq 'Brand Title' + end + end + + describe 'page_image' do + it 'defaults to the GitLab logo' do + expect(helper.page_image).to end_with 'assets/gitlab_logo.png' + end + + context 'with @project' do + it 'uses Project avatar if available' do + project = double(avatar_url: 'http://example.com/uploads/avatar.png') + helper.instance_variable_set(:@project, project) + + expect(helper.page_image).to eq project.avatar_url + end + + it 'falls back to the default' do + project = double(avatar_url: nil) + helper.instance_variable_set(:@project, project) + + expect(helper.page_image).to end_with 'assets/gitlab_logo.png' + end + end + + context 'with @user' do + it 'delegates to avatar_icon helper' do + user = double('User') + helper.instance_variable_set(:@user, user) + + expect(helper).to receive(:avatar_icon).with(user) + + helper.page_image + end + end + end +end -- cgit v1.2.1 From 7e43fa67096f60856d1cf862f245367bc710a44f Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Wed, 23 Dec 2015 16:56:36 -0500 Subject: fixes tests to work with jasmine/jquery --- spec/javascripts/fixtures/issues_show.html.haml | 11 +++++++---- spec/javascripts/issue_spec.js.coffee | 18 +++++++++--------- 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/spec/javascripts/fixtures/issues_show.html.haml b/spec/javascripts/fixtures/issues_show.html.haml index 6c985a32966..f67ea5a13c7 100644 --- a/spec/javascripts/fixtures/issues_show.html.haml +++ b/spec/javascripts/fixtures/issues_show.html.haml @@ -1,7 +1,10 @@ -.issue-box.issue-box-open Open -.issue-box.issue-box-closed.hidden Closed -%a.btn-close{"data-url" => "http://gitlab/issues/6/close"} Close -%a.btn-reopen.hidden{"data-url" => "http://gitlab/issues/6/reopen"} Reopen +:css + .hidden { display: none !important; } + +.status-box.status-box-open Open +.status-box.status-box-closed.hidden Closed +%a.btn-close{"href" => "http://gitlab/issues/6/close"} Close +%a.btn-reopen.hidden{"href" => "http://gitlab/issues/6/reopen"} Reopen .detail-page-description .description.js-task-list-container diff --git a/spec/javascripts/issue_spec.js.coffee b/spec/javascripts/issue_spec.js.coffee index ef78cfbc653..142eb05cc67 100644 --- a/spec/javascripts/issue_spec.js.coffee +++ b/spec/javascripts/issue_spec.js.coffee @@ -33,16 +33,16 @@ describe 'reopen/close issue', -> $btnClose = $('a.btn-close') $btnReopen = $('a.btn-reopen') - expect($btnReopen.toBeHidden()) + expect($btnReopen).toBeHidden() expect($btnClose.text()).toBe('Close') expect(typeof $btnClose.prop('disabled')).toBe('undefined') $btnClose.trigger('click') - expect($btnClose.toBeHidden()) - expect($btnReopen.toBeVisible()) - expect($('div.issue-box-open').toBeVisible()) - expect($('div.issue-box-closed').toBeHidden()) + expect($btnReopen).toBeVisible() + expect($btnClose).toBeHidden() + expect($('div.status-box-closed')).toBeVisible() + expect($('div.status-box-open')).toBeHidden() it 'reopens an issue', -> $.ajax = (obj) -> @@ -56,7 +56,7 @@ describe 'reopen/close issue', -> $btnReopen.trigger('click') - expect($btnReopen.toBeHidden()) - expect($btnClose.toBeVisible()) - expect($('div.issue-box-open').toBeVisible()) - expect($('div.issue-box-closed').toBeHidden()) \ No newline at end of file + expect($btnReopen).toBeHidden() + expect($btnClose).toBeVisible() + expect($('div.status-box-open')).toBeVisible() + expect($('div.status-box-closed')).toBeHidden() \ No newline at end of file -- cgit v1.2.1 From 5a3b9c97e34ee69312ef9bcf575894a106c5a271 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Wed, 23 Dec 2015 17:14:18 -0500 Subject: Account for `@project.description` being nil --- app/helpers/page_layout_helper.rb | 2 +- spec/helpers/page_layout_helper_spec.rb | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/app/helpers/page_layout_helper.rb b/app/helpers/page_layout_helper.rb index 6b16528dde6..c4eb09bea2b 100644 --- a/app/helpers/page_layout_helper.rb +++ b/app/helpers/page_layout_helper.rb @@ -40,7 +40,7 @@ module PageLayoutHelper # a view def page_description_default if @project - @project.description + @project.description || brand_title else brand_title end diff --git a/spec/helpers/page_layout_helper_spec.rb b/spec/helpers/page_layout_helper_spec.rb index a60b8f508f2..5d95beac908 100644 --- a/spec/helpers/page_layout_helper_spec.rb +++ b/spec/helpers/page_layout_helper_spec.rb @@ -37,6 +37,14 @@ describe PageLayoutHelper do expect(helper.page_description_default).to eq 'Project Description' end + it 'uses brand_title when Project description is nil' do + project = double(description: nil) + helper.instance_variable_set(:@project, project) + + expect(helper).to receive(:brand_title).and_return('Brand Title') + expect(helper.page_description_default).to eq 'Brand Title' + end + it 'falls back to brand_title' do allow(helper).to receive(:brand_title).and_return('Brand Title') -- cgit v1.2.1 From e11ee5ff01bf031cd4fda377a7444f2ba4d50f40 Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Wed, 23 Dec 2015 17:41:05 -0500 Subject: adds test for issue close/reopen failure --- app/assets/javascripts/issue.js.coffee | 5 ++- spec/javascripts/fixtures/issues_show.html.haml | 13 ++++++- spec/javascripts/issue_spec.js.coffee | 51 ++++++++++++++++++++++++- 3 files changed, 63 insertions(+), 6 deletions(-) diff --git a/app/assets/javascripts/issue.js.coffee b/app/assets/javascripts/issue.js.coffee index 1a9e03e4ee3..3d9d514a9c7 100644 --- a/app/assets/javascripts/issue.js.coffee +++ b/app/assets/javascripts/issue.js.coffee @@ -15,6 +15,7 @@ class @Issue $(document).on 'tasklist:changed', '.detail-page-description .js-task-list-container', @updateTaskList initIssueBtnEventListeners: -> + issueFailMessage = 'Unable to update this issue at this time.' $('a.btn-close, a.btn-reopen').on 'click', (e) -> e.preventDefault() e.stopImmediatePropagation() @@ -27,7 +28,7 @@ class @Issue url: url, error: (jqXHR, textStatus, errorThrown) -> issueStatus = if isClose then 'close' else 'open' - new Flash('Issues update failed', 'alert') + new Flash(issueFailMessage, 'alert') success: (data, textStatus, jqXHR) -> if data.saved $this.addClass('hidden') @@ -40,7 +41,7 @@ class @Issue $('div.status-box-closed').addClass('hidden') $('div.status-box-open').removeClass('hidden') else - new Flash('Issues update failed', 'alert') + new Flash(issueFailMessage, 'alert') $this.prop('disabled', false) disableTaskList: -> diff --git a/spec/javascripts/fixtures/issues_show.html.haml b/spec/javascripts/fixtures/issues_show.html.haml index f67ea5a13c7..dd01b6378c5 100644 --- a/spec/javascripts/fixtures/issues_show.html.haml +++ b/spec/javascripts/fixtures/issues_show.html.haml @@ -1,10 +1,19 @@ :css .hidden { display: none !important; } +.flash-container + - if alert + .flash-alert + = alert + + - elsif notice + .flash-notice + = notice + .status-box.status-box-open Open .status-box.status-box-closed.hidden Closed -%a.btn-close{"href" => "http://gitlab/issues/6/close"} Close -%a.btn-reopen.hidden{"href" => "http://gitlab/issues/6/reopen"} Reopen +%a.btn-close{"href" => "http://gitlab.com/issues/6/close"} Close +%a.btn-reopen.hidden{"href" => "http://gitlab.com/issues/6/reopen"} Reopen .detail-page-description .description.js-task-list-container diff --git a/spec/javascripts/issue_spec.js.coffee b/spec/javascripts/issue_spec.js.coffee index 142eb05cc67..f0a57d99c56 100644 --- a/spec/javascripts/issue_spec.js.coffee +++ b/spec/javascripts/issue_spec.js.coffee @@ -1,3 +1,4 @@ +#= require flash #= require issue describe 'Issue', -> @@ -28,7 +29,7 @@ describe 'reopen/close issue', -> it 'closes an issue', -> $.ajax = (obj) -> expect(obj.type).toBe('PUT') - expect(obj.url).toBe('http://gitlab/issues/6/close') + expect(obj.url).toBe('http://gitlab.com/issues/6/close') obj.success saved: true $btnClose = $('a.btn-close') @@ -44,10 +45,56 @@ describe 'reopen/close issue', -> expect($('div.status-box-closed')).toBeVisible() expect($('div.status-box-open')).toBeHidden() + it 'fails to closes an issue with success:false', -> + + $.ajax = (obj) -> + expect(obj.type).toBe('PUT') + expect(obj.url).toBe('http://goesnowhere.nothing/whereami') + obj.success saved: false + + $btnClose = $('a.btn-close') + $btnReopen = $('a.btn-reopen') + $btnClose.attr('href','http://goesnowhere.nothing/whereami') + expect($btnReopen).toBeHidden() + expect($btnClose.text()).toBe('Close') + expect(typeof $btnClose.prop('disabled')).toBe('undefined') + + $btnClose.trigger('click') + + expect($btnReopen).toBeHidden() + expect($btnClose).toBeVisible() + expect($('div.status-box-closed')).toBeHidden() + expect($('div.status-box-open')).toBeVisible() + expect($('div.flash-alert')).toBeVisible() + expect($('div.flash-alert').text()).toBe('Unable to update this issue at this time.') + + it 'fails to closes an issue with HTTP error', -> + + $.ajax = (obj) -> + expect(obj.type).toBe('PUT') + expect(obj.url).toBe('http://goesnowhere.nothing/whereami') + obj.error() + + $btnClose = $('a.btn-close') + $btnReopen = $('a.btn-reopen') + $btnClose.attr('href','http://goesnowhere.nothing/whereami') + expect($btnReopen).toBeHidden() + expect($btnClose.text()).toBe('Close') + expect(typeof $btnClose.prop('disabled')).toBe('undefined') + + $btnClose.trigger('click') + + expect($btnReopen).toBeHidden() + expect($btnClose).toBeVisible() + expect($('div.status-box-closed')).toBeHidden() + expect($('div.status-box-open')).toBeVisible() + expect($('div.flash-alert')).toBeVisible() + expect($('div.flash-alert').text()).toBe('Unable to update this issue at this time.') + it 'reopens an issue', -> $.ajax = (obj) -> expect(obj.type).toBe('PUT') - expect(obj.url).toBe('http://gitlab/issues/6/reopen') + expect(obj.url).toBe('http://gitlab.com/issues/6/reopen') obj.success saved: true $btnClose = $('a.btn-close') -- cgit v1.2.1 From 00e967a07f476f7df7aaddc652258668af392f5d Mon Sep 17 00:00:00 2001 From: Jacob Schatz Date: Wed, 23 Dec 2015 20:57:16 -0500 Subject: removes unused `alert` from issue spec. Requires flash in the *implementation* instead of the spec. --- app/assets/javascripts/issue.js.coffee | 1 + spec/javascripts/fixtures/issues_show.html.haml | 9 ++------- spec/javascripts/issue_spec.js.coffee | 1 - 3 files changed, 3 insertions(+), 8 deletions(-) diff --git a/app/assets/javascripts/issue.js.coffee b/app/assets/javascripts/issue.js.coffee index 3d9d514a9c7..c256ec8f41b 100644 --- a/app/assets/javascripts/issue.js.coffee +++ b/app/assets/javascripts/issue.js.coffee @@ -1,3 +1,4 @@ +#= require flash #= require jquery.waitforimages #= require task_list diff --git a/spec/javascripts/fixtures/issues_show.html.haml b/spec/javascripts/fixtures/issues_show.html.haml index dd01b6378c5..470cabeafbb 100644 --- a/spec/javascripts/fixtures/issues_show.html.haml +++ b/spec/javascripts/fixtures/issues_show.html.haml @@ -2,13 +2,8 @@ .hidden { display: none !important; } .flash-container - - if alert - .flash-alert - = alert - - - elsif notice - .flash-notice - = notice + .flash-alert + .flash-notice .status-box.status-box-open Open .status-box.status-box-closed.hidden Closed diff --git a/spec/javascripts/issue_spec.js.coffee b/spec/javascripts/issue_spec.js.coffee index f0a57d99c56..7e67c778861 100644 --- a/spec/javascripts/issue_spec.js.coffee +++ b/spec/javascripts/issue_spec.js.coffee @@ -1,4 +1,3 @@ -#= require flash #= require issue describe 'Issue', -> -- cgit v1.2.1 From 7309b848e2e67e3206e986facc0104a11af18d88 Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Thu, 24 Dec 2015 12:20:00 +0000 Subject: Move changelog items to their correct place. --- CHANGELOG | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 0f9ae1e3b52..55e242cff50 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -4,12 +4,13 @@ v 8.4.0 (unreleased) - Fix Error 500 when doing a search in dashboard before visiting any project (Stan Hu) - Implement new UI for group page - Add project permissions to all project API endpoints (Stan Hu) + - Add CAS support (tduehr) + - Add link to merge request on build detail page. + - Expose Git's version in the admin area v 8.3.0 - - Add CAS support (tduehr) - Bump rack-attack to 4.3.1 for security fix (Stan Hu) - API support for starred projects for authorized user (Zeger-Jan van de Weg) - - Add link to merge request on build detail page. - Add open_issues_count to project API (Stan Hu) - Expand character set of usernames created by Omniauth (Corey Hinshaw) - Add button to automatically merge a merge request when the build succeeds (Zeger-Jan van de Weg) @@ -69,7 +70,6 @@ v 8.3.0 - Do not show build status unless builds are enabled and `.gitlab-ci.yml` is present - Persist runners registration token in database - Fix online editor should not remove newlines at the end of the file - - Expose Git's version in the admin area v 8.2.3 - Fix application settings cache not expiring after changes (Stan Hu) -- cgit v1.2.1 From 672cbbff959c66ba10740ed17671d93060410d93 Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Thu, 24 Dec 2015 15:33:51 +0100 Subject: Only allow group/project members to mention `@all` --- CHANGELOG | 1 + app/controllers/projects_controller.rb | 2 +- app/models/concerns/mentionable.rb | 2 +- lib/banzai/filter/redactor_filter.rb | 6 +++--- lib/banzai/filter/reference_filter.rb | 6 +++++- lib/banzai/filter/reference_gatherer_filter.rb | 8 +++++++- lib/banzai/filter/user_reference_filter.rb | 14 +++++++++++++- lib/gitlab/reference_extractor.rb | 19 ++++++++++++------- spec/lib/banzai/filter/user_reference_filter_spec.rb | 19 ++++++++++++++++--- 9 files changed, 59 insertions(+), 18 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index a80b776affa..e7897a0fe77 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -5,6 +5,7 @@ v 8.4.0 (unreleased) - Implement new UI for group page - Implement search inside emoji picker - Add project permissions to all project API endpoints (Stan Hu) + - Only allow group/project members to mention `@all` v 8.3.1 (unreleased) - Fix Error 500 when global milestones have slashes (Stan Hu) diff --git a/app/controllers/projects_controller.rb b/app/controllers/projects_controller.rb index bf5e25ff895..682dbf2766a 100644 --- a/app/controllers/projects_controller.rb +++ b/app/controllers/projects_controller.rb @@ -178,7 +178,7 @@ class ProjectsController < ApplicationController def markdown_preview text = params[:text] - ext = Gitlab::ReferenceExtractor.new(@project, current_user) + ext = Gitlab::ReferenceExtractor.new(@project, current_user, current_user) ext.analyze(text) render json: { diff --git a/app/models/concerns/mentionable.rb b/app/models/concerns/mentionable.rb index 1fdcda97520..6316ee208b5 100644 --- a/app/models/concerns/mentionable.rb +++ b/app/models/concerns/mentionable.rb @@ -44,7 +44,7 @@ module Mentionable end def all_references(current_user = self.author, text = nil) - ext = Gitlab::ReferenceExtractor.new(self.project, current_user) + ext = Gitlab::ReferenceExtractor.new(self.project, current_user, self.author) if text ext.analyze(text) diff --git a/lib/banzai/filter/redactor_filter.rb b/lib/banzai/filter/redactor_filter.rb index 89e7a79789a..f01a32b5ae5 100644 --- a/lib/banzai/filter/redactor_filter.rb +++ b/lib/banzai/filter/redactor_filter.rb @@ -11,7 +11,7 @@ module Banzai class RedactorFilter < HTML::Pipeline::Filter def call doc.css('a.gfm').each do |node| - unless user_can_reference?(node) + unless user_can_see_reference?(node) # The reference should be replaced by the original text, # which is not always the same as the rendered text. text = node.attr('data-original') || node.text @@ -24,12 +24,12 @@ module Banzai private - def user_can_reference?(node) + def user_can_see_reference?(node) if node.has_attribute?('data-reference-filter') reference_type = node.attr('data-reference-filter') reference_filter = Banzai::Filter.const_get(reference_type) - reference_filter.user_can_reference?(current_user, node, context) + reference_filter.user_can_see_reference?(current_user, node, context) else true end diff --git a/lib/banzai/filter/reference_filter.rb b/lib/banzai/filter/reference_filter.rb index 33457a3f361..5275ac8bf95 100644 --- a/lib/banzai/filter/reference_filter.rb +++ b/lib/banzai/filter/reference_filter.rb @@ -12,7 +12,7 @@ module Banzai # :project (required) - Current project, ignored if reference is cross-project. # :only_path - Generate path-only links. class ReferenceFilter < HTML::Pipeline::Filter - def self.user_can_reference?(user, node, context) + def self.user_can_see_reference?(user, node, context) if node.has_attribute?('data-project') project_id = node.attr('data-project').to_i return true if project_id == context[:project].try(:id) @@ -24,6 +24,10 @@ module Banzai end end + def self.user_can_reference?(user, node, context) + true + end + def self.referenced_by(node) raise NotImplementedError, "#{self} does not implement #{__method__}" end diff --git a/lib/banzai/filter/reference_gatherer_filter.rb b/lib/banzai/filter/reference_gatherer_filter.rb index 855f238ac1e..12412ff7ea9 100644 --- a/lib/banzai/filter/reference_gatherer_filter.rb +++ b/lib/banzai/filter/reference_gatherer_filter.rb @@ -35,7 +35,9 @@ module Banzai return if context[:reference_filter] && reference_filter != context[:reference_filter] - return unless reference_filter.user_can_reference?(current_user, node, context) + return if author && !reference_filter.user_can_reference?(author, node, context) + + return unless reference_filter.user_can_see_reference?(current_user, node, context) references = reference_filter.referenced_by(node) return unless references @@ -57,6 +59,10 @@ module Banzai def current_user context[:current_user] end + + def author + context[:author] + end end end end diff --git a/lib/banzai/filter/user_reference_filter.rb b/lib/banzai/filter/user_reference_filter.rb index 67c24faf991..7ec771266ed 100644 --- a/lib/banzai/filter/user_reference_filter.rb +++ b/lib/banzai/filter/user_reference_filter.rb @@ -39,7 +39,7 @@ module Banzai end end - def self.user_can_reference?(user, node, context) + def self.user_can_see_reference?(user, node, context) if node.has_attribute?('data-group') group = Group.find(node.attr('data-group')) rescue nil Ability.abilities.allowed?(user, :read_group, group) @@ -48,6 +48,18 @@ module Banzai end end + def self.user_can_reference?(user, node, context) + # Only team members can reference `@all` + if node.has_attribute?('data-project') + project = Project.find(node.attr('data-project')) rescue nil + return false unless project + + user && project.team.member?(user) + else + super + end + end + def call replace_text_nodes_matching(User.reference_pattern) do |content| user_link_filter(content) diff --git a/lib/gitlab/reference_extractor.rb b/lib/gitlab/reference_extractor.rb index 0a70d21b1ce..be795649e59 100644 --- a/lib/gitlab/reference_extractor.rb +++ b/lib/gitlab/reference_extractor.rb @@ -3,11 +3,12 @@ require 'banzai' module Gitlab # Extract possible GFM references from an arbitrary String for further processing. class ReferenceExtractor < Banzai::ReferenceExtractor - attr_accessor :project, :current_user + attr_accessor :project, :current_user, :author - def initialize(project, current_user = nil) + def initialize(project, current_user = nil, author = nil) @project = project @current_user = current_user + @author = author @references = {} @@ -20,18 +21,22 @@ module Gitlab %i(user label merge_request snippet commit commit_range).each do |type| define_method("#{type}s") do - @references[type] ||= references(type, project: project, current_user: current_user) + @references[type] ||= references(type, reference_context) end end def issues - options = { project: project, current_user: current_user } - if project && project.jira_tracker? - @references[:external_issue] ||= references(:external_issue, options) + @references[:external_issue] ||= references(:external_issue, reference_context) else - @references[:issue] ||= references(:issue, options) + @references[:issue] ||= references(:issue, reference_context) end end + + private + + def reference_context + { project: project, current_user: current_user, author: author } + end end end diff --git a/spec/lib/banzai/filter/user_reference_filter_spec.rb b/spec/lib/banzai/filter/user_reference_filter_spec.rb index 3534bf97784..8bdebae1841 100644 --- a/spec/lib/banzai/filter/user_reference_filter_spec.rb +++ b/spec/lib/banzai/filter/user_reference_filter_spec.rb @@ -37,9 +37,22 @@ describe Banzai::Filter::UserReferenceFilter, lib: true do .to eq urls.namespace_project_url(project.namespace, project) end - it 'adds to the results hash' do - result = reference_pipeline_result("Hey #{reference}") - expect(result[:references][:user]).to eq [project.creator] + context "when the author is a member of the project" do + + it 'adds to the results hash' do + result = reference_pipeline_result("Hey #{reference}", author: project.creator) + expect(result[:references][:user]).to eq [project.creator] + end + end + + context "when the author is not a member of the project" do + + let(:other_user) { create(:user) } + + it "doesn't add to the results hash" do + result = reference_pipeline_result("Hey #{reference}", author: other_user) + expect(result[:references][:user]).to eq [] + end end end -- cgit v1.2.1 From 5a8c65b508614dd8896ff8af7cad6e2b33fb7244 Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Sat, 12 Dec 2015 22:02:05 -0800 Subject: Add API support for looking up a user by username Needed to support Huboard --- CHANGELOG | 1 + doc/api/users.md | 12 +++++++++++- lib/api/users.rb | 14 ++++++++++---- spec/requests/api/users_spec.rb | 7 +++++++ 4 files changed, 29 insertions(+), 5 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index a80b776affa..29a175bea80 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -4,6 +4,7 @@ v 8.4.0 (unreleased) - Fix Error 500 when doing a search in dashboard before visiting any project (Stan Hu) - Implement new UI for group page - Implement search inside emoji picker + - Add API support for looking up a user by username (Stan Hu) - Add project permissions to all project API endpoints (Stan Hu) v 8.3.1 (unreleased) diff --git a/doc/api/users.md b/doc/api/users.md index 7ba2db248ff..66d2fd52526 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -90,7 +90,17 @@ GET /users You can search for users by email or username with: `/users?search=John` -Also see `def search query` in `app/models/user.rb`. +In addition, you can lookup users by username: + +``` +GET /users?username=:username +``` + +For example: + +``` +GET /users?username=jack_smith +``` ## Single user diff --git a/lib/api/users.rb b/lib/api/users.rb index a98d668e02d..3400f0713ef 100644 --- a/lib/api/users.rb +++ b/lib/api/users.rb @@ -8,11 +8,17 @@ module API # # Example Request: # GET /users + # GET /users?search=Admin + # GET /users?username=root get do - @users = User.all - @users = @users.active if params[:active].present? - @users = @users.search(params[:search]) if params[:search].present? - @users = paginate @users + if params[:username].present? + @users = User.where(username: params[:username]) + else + @users = User.all + @users = @users.active if params[:active].present? + @users = @users.search(params[:search]) if params[:search].present? + @users = paginate @users + end if current_user.is_admin? present @users, with: Entities::UserFull diff --git a/spec/requests/api/users_spec.rb b/spec/requests/api/users_spec.rb index 2f609c63330..4f278551d07 100644 --- a/spec/requests/api/users_spec.rb +++ b/spec/requests/api/users_spec.rb @@ -27,6 +27,13 @@ describe API::API, api: true do user['username'] == username end['username']).to eq(username) end + + it "should return one user" do + get api("/users?username=#{omniauth_user.username}", user) + expect(response.status).to eq(200) + expect(json_response).to be_an Array + expect(json_response.first['username']).to eq(omniauth_user.username) + end end context "when admin" do -- cgit v1.2.1 From e3befaed82f9aa52c79a1d4c437fe4fc63f8d07a Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 24 Dec 2015 14:05:30 -0500 Subject: Update CHANGELOG [ci skip] --- CHANGELOG | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index 96d71331321..de8ac5161a0 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,7 +1,6 @@ Please view this file on the master branch, on stable branches it's out of date. v 8.4.0 (unreleased) - - Fix Error 500 when doing a search in dashboard before visiting any project (Stan Hu) - Implement new UI for group page - Implement search inside emoji picker - Add project permissions to all project API endpoints (Stan Hu) @@ -12,6 +11,9 @@ v 8.4.0 (unreleased) v 8.3.1 - Fix Error 500 when global milestones have slashes (Stan Hu) + - Fix Error 500 when doing a search in dashboard before visiting any project (Stan Hu) + - Fix LDAP identity and user retrieval when special characters are used + - Move Sidekiq-cron configuration to gitlab.yml v 8.3.0 - Bump rack-attack to 4.3.1 for security fix (Stan Hu) -- cgit v1.2.1 From 396c5c97ddbbe0744d9c27d855c22e2a57546ea1 Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Thu, 24 Dec 2015 20:34:13 +0100 Subject: Fix specs --- spec/models/concerns/mentionable_spec.rb | 4 ++++ spec/services/notification_service_spec.rb | 1 + 2 files changed, 5 insertions(+) diff --git a/spec/models/concerns/mentionable_spec.rb b/spec/models/concerns/mentionable_spec.rb index 6653621a83e..20f0c561e44 100644 --- a/spec/models/concerns/mentionable_spec.rb +++ b/spec/models/concerns/mentionable_spec.rb @@ -3,6 +3,10 @@ require 'spec_helper' describe Mentionable do include Mentionable + def author + nil + end + describe :references do let(:project) { create(:project) } diff --git a/spec/services/notification_service_spec.rb b/spec/services/notification_service_spec.rb index d7a898e85ff..213adace14b 100644 --- a/spec/services/notification_service_spec.rb +++ b/spec/services/notification_service_spec.rb @@ -115,6 +115,7 @@ describe NotificationService, services: true do before do build_team(note.project) + note.project.team << [note.author, :master] ActionMailer::Base.deliveries.clear end -- cgit v1.2.1 From 8309ef45a959715ed6d0727dabe6cc072cea8781 Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Thu, 24 Dec 2015 11:08:46 -0800 Subject: Enable "Add key" button when user fills in a proper key Closes #4295 --- CHANGELOG | 3 +++ app/views/profiles/keys/new.html.haml | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index 96d71331321..e4aefdea259 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -10,6 +10,9 @@ v 8.4.0 (unreleased) - Add CAS support (tduehr) - Add link to merge request on build detail page. +v 8.3.2 (unreleased) + - Enable "Add key" button when user fills in a proper key + v 8.3.1 - Fix Error 500 when global milestones have slashes (Stan Hu) diff --git a/app/views/profiles/keys/new.html.haml b/app/views/profiles/keys/new.html.haml index 11166dc6d99..13a18269d11 100644 --- a/app/views/profiles/keys/new.html.haml +++ b/app/views/profiles/keys/new.html.haml @@ -12,6 +12,6 @@ comment = val.match(/^\S+ \S+ (.+)\n?$/); if( comment && comment.length > 1 && title.val() == '' ){ - $('#key_title').val( comment[1] ); + $('#key_title').val( comment[1] ).change(); } }); -- cgit v1.2.1 From 6fce8b6f5cfee1fe17ef7cdb5508b66da93f1f14 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 24 Dec 2015 21:48:24 +0200 Subject: Add triggers doc in top level readme [ci skip] --- doc/README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/doc/README.md b/doc/README.md index 8bac00f2f23..f40a257b42f 100644 --- a/doc/README.md +++ b/doc/README.md @@ -28,17 +28,18 @@ - [Using SSH keys](ci/ssh_keys/README.md) - [User permissions](ci/permissions/README.md) - [API](ci/api/README.md) +- [Triggering builds through the API](ci/triggers/README.md) ### CI Languages -+ [Testing PHP](ci/languages/php.md) +- [Testing PHP](ci/languages/php.md) ### CI Services -+ [Using MySQL](ci/services/mysql.md) -+ [Using PostgreSQL](ci/services/postgres.md) -+ [Using Redis](ci/services/redis.md) -+ [Using Other Services](ci/docker/using_docker_images.md#how-to-use-other-images-as-services) +- [Using MySQL](ci/services/mysql.md) +- [Using PostgreSQL](ci/services/postgres.md) +- [Using Redis](ci/services/redis.md) +- [Using Other Services](ci/docker/using_docker_images.md#how-to-use-other-images-as-services) ### CI Examples -- cgit v1.2.1 From aafd7ce83bf1f6c1074b93ad489caa03e5b20786 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 24 Dec 2015 21:50:03 +0200 Subject: Remove triggers from yaml doc [ci skip] --- doc/ci/yaml/README.md | 27 --------------------------- 1 file changed, 27 deletions(-) diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 6862116cc5b..fd0d49de4e4 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -66,7 +66,6 @@ There are a few reserved `keywords` that **cannot** be used as job names: | before_script | no | Define commands that run before each job's script | | variables | no | Define build variables | | cache | no | Define list of files that should be cached between subsequent runs | -| trigger | no | Force a rebuild of a specific branch or tag with an API call | ### image and services @@ -153,32 +152,6 @@ cache: - binaries/ ``` -### trigger - -Triggers can be used to force a rebuild of a specific branch or tag with an API -call. You can add a trigger by visiting the project's **Settings > Triggers**. - -Every new trigger you create, gets assigned a different token which you can -then use inside your `.gitlab-ci.yml`: - -```yaml -trigger: - stage: deploy - script: - - "curl -X POST -F token=TOKEN -F ref=master https://gitlab.com/api/v3/projects/9/trigger/builds" -``` - -In the example above, a rebuild on master branch will be triggered after all -previous stages build successfully (denoted by `stage: deploy`). - -You can find the endpoint containing the ID of the project on the **Triggers** -page. - -_**Warning:** Be careful how you set up your triggers, because you could end up -in an infinite loop._ - -Read more in the dedicated [triggers documentation](../triggers/README.md). - ## Jobs `.gitlab-ci.yml` allows you to specify an unlimited number of jobs. Each job -- cgit v1.2.1 From 37993d39577058d5c76ef9c35e40d1c8f9aa7982 Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Thu, 24 Dec 2015 21:36:33 +0100 Subject: Escape all the things. --- lib/banzai/filter/abstract_reference_filter.rb | 23 ++++++++++++---------- .../filter/external_issue_reference_filter.rb | 6 +++--- lib/banzai/filter/label_reference_filter.rb | 2 +- lib/banzai/filter/reference_filter.rb | 4 ++-- lib/banzai/filter/user_reference_filter.rb | 2 +- 5 files changed, 20 insertions(+), 17 deletions(-) diff --git a/lib/banzai/filter/abstract_reference_filter.rb b/lib/banzai/filter/abstract_reference_filter.rb index bdaa4721b4b..63ad8910c0f 100644 --- a/lib/banzai/filter/abstract_reference_filter.rb +++ b/lib/banzai/filter/abstract_reference_filter.rb @@ -98,7 +98,7 @@ module Banzai project = project_from_ref(project_ref) if project && object = find_object(project, id) - title = escape_once(object_link_title(object)) + title = object_link_title(object) klass = reference_class(object_sym) data = data_attribute( @@ -110,17 +110,11 @@ module Banzai url = matches[:url] if matches.names.include?("url") url ||= url_for_object(object, project) - text = link_text - unless text - text = object.reference_link_text(context[:project]) - - extras = object_link_text_extras(object, matches) - text += " (#{extras.join(", ")})" if extras.any? - end + text = link_text || object_link_text(object, matches) %(#{text}) + title="#{escape_once(title)}" + class="#{klass}">#{escape_once(text)}) else match end @@ -140,6 +134,15 @@ module Banzai def object_link_title(object) "#{object_class.name.titleize}: #{object.title}" end + + def object_link_text(object, matches) + text = object.reference_link_text(context[:project]) + + extras = object_link_text_extras(object, matches) + text += " (#{extras.join(", ")})" if extras.any? + + text + end end end end diff --git a/lib/banzai/filter/external_issue_reference_filter.rb b/lib/banzai/filter/external_issue_reference_filter.rb index f5942740cd6..6136e73c096 100644 --- a/lib/banzai/filter/external_issue_reference_filter.rb +++ b/lib/banzai/filter/external_issue_reference_filter.rb @@ -63,15 +63,15 @@ module Banzai url = url_for_issue(id, project, only_path: context[:only_path]) - title = escape_once("Issue in #{project.external_issue_tracker.title}") + title = "Issue in #{project.external_issue_tracker.title}" klass = reference_class(:issue) data = data_attribute(project: project.id, external_issue: id) text = link_text || match %(#{text}) + title="#{escape_once(title)}" + class="#{klass}">#{escape_once(text)}) end end diff --git a/lib/banzai/filter/label_reference_filter.rb b/lib/banzai/filter/label_reference_filter.rb index 07bac2dd7fd..a3a7a23c1e6 100644 --- a/lib/banzai/filter/label_reference_filter.rb +++ b/lib/banzai/filter/label_reference_filter.rb @@ -60,7 +60,7 @@ module Banzai text = link_text || render_colored_label(label) %(#{text}) + class="#{klass}">#{escape_once(text)}) else match end diff --git a/lib/banzai/filter/reference_filter.rb b/lib/banzai/filter/reference_filter.rb index 33457a3f361..a22a7a7afd3 100644 --- a/lib/banzai/filter/reference_filter.rb +++ b/lib/banzai/filter/reference_filter.rb @@ -44,11 +44,11 @@ module Banzai # Returns a String def data_attribute(attributes = {}) attributes[:reference_filter] = self.class.name.demodulize - attributes.map { |key, value| %Q(data-#{key.to_s.dasherize}="#{value}") }.join(" ") + attributes.map { |key, value| %Q(data-#{key.to_s.dasherize}="#{escape_once(value)}") }.join(" ") end def escape_once(html) - ERB::Util.html_escape_once(html) + html.html_safe? ? html : ERB::Util.html_escape_once(html) end def ignore_parents diff --git a/lib/banzai/filter/user_reference_filter.rb b/lib/banzai/filter/user_reference_filter.rb index 67c24faf991..7f302d51dd7 100644 --- a/lib/banzai/filter/user_reference_filter.rb +++ b/lib/banzai/filter/user_reference_filter.rb @@ -122,7 +122,7 @@ module Banzai end def link_tag(url, data, text) - %(#{text}) + %(#{escape_once(text)}) end end end -- cgit v1.2.1 From 33964469b38e2b36b200b20fe3061371a5f5ab18 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Fri, 18 Dec 2015 18:29:13 -0200 Subject: WIP require two factor authentication --- app/controllers/application_controller.rb | 12 +++++ .../profiles/two_factor_auths_controller.rb | 2 + app/models/application_setting.rb | 59 ++++++++++++---------- ...151218154042_add_tfa_to_application_settings.rb | 8 +++ db/schema.rb | 2 + spec/models/application_setting_spec.rb | 1 + 6 files changed, 58 insertions(+), 26 deletions(-) create mode 100644 db/migrate/20151218154042_add_tfa_to_application_settings.rb diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb index 01e2e7b2f98..e15d83631b3 100644 --- a/app/controllers/application_controller.rb +++ b/app/controllers/application_controller.rb @@ -13,6 +13,7 @@ class ApplicationController < ActionController::Base before_action :validate_user_service_ticket! before_action :reject_blocked! before_action :check_password_expiration + before_action :check_tfa_requirement before_action :ldap_security_check before_action :default_headers before_action :add_gon_variables @@ -223,6 +224,13 @@ class ApplicationController < ActionController::Base end end + def check_tfa_requirement + if two_factor_authentication_required? && current_user && !current_user.two_factor_enabled + redirect_to new_profile_two_factor_auth_path, + alert: 'You must configure Two-Factor Authentication in your account' + end + end + def ldap_security_check if current_user && current_user.requires_ldap_check? unless Gitlab::LDAP::Access.allowed?(current_user) @@ -357,6 +365,10 @@ class ApplicationController < ActionController::Base current_application_settings.import_sources.include?('git') end + def two_factor_authentication_required? + current_application_settings.require_two_factor_authentication + end + def redirect_to_home_page_url? # If user is not signed-in and tries to access root_path - redirect him to landing page # Don't redirect to the default URL to prevent endless redirections diff --git a/app/controllers/profiles/two_factor_auths_controller.rb b/app/controllers/profiles/two_factor_auths_controller.rb index e6b99be37fb..05c84fb720e 100644 --- a/app/controllers/profiles/two_factor_auths_controller.rb +++ b/app/controllers/profiles/two_factor_auths_controller.rb @@ -1,4 +1,6 @@ class Profiles::TwoFactorAuthsController < Profiles::ApplicationController + skip_before_action :check_tfa_requirement + def new unless current_user.otp_secret current_user.otp_secret = User.generate_otp_secret(32) diff --git a/app/models/application_setting.rb b/app/models/application_setting.rb index 724429e7558..7c107da116c 100644 --- a/app/models/application_setting.rb +++ b/app/models/application_setting.rb @@ -2,32 +2,34 @@ # # Table name: application_settings # -# id :integer not null, primary key -# default_projects_limit :integer -# signup_enabled :boolean -# signin_enabled :boolean -# gravatar_enabled :boolean -# sign_in_text :text -# created_at :datetime -# updated_at :datetime -# home_page_url :string(255) -# default_branch_protection :integer default(2) -# twitter_sharing_enabled :boolean default(TRUE) -# restricted_visibility_levels :text -# version_check_enabled :boolean default(TRUE) -# max_attachment_size :integer default(10), not null -# default_project_visibility :integer -# default_snippet_visibility :integer -# restricted_signup_domains :text -# user_oauth_applications :boolean default(TRUE) -# after_sign_out_path :string(255) -# session_expire_delay :integer default(10080), not null -# import_sources :text -# help_page_text :text -# admin_notification_email :string(255) -# shared_runners_enabled :boolean default(TRUE), not null -# max_artifacts_size :integer default(100), not null -# runners_registration_token :string(255) +# id :integer not null, primary key +# default_projects_limit :integer +# signup_enabled :boolean +# signin_enabled :boolean +# gravatar_enabled :boolean +# sign_in_text :text +# created_at :datetime +# updated_at :datetime +# home_page_url :string(255) +# default_branch_protection :integer default(2) +# twitter_sharing_enabled :boolean default(TRUE) +# restricted_visibility_levels :text +# version_check_enabled :boolean default(TRUE) +# max_attachment_size :integer default(10), not null +# default_project_visibility :integer +# default_snippet_visibility :integer +# restricted_signup_domains :text +# user_oauth_applications :boolean default(TRUE) +# after_sign_out_path :string(255) +# session_expire_delay :integer default(10080), not null +# import_sources :text +# help_page_text :text +# admin_notification_email :string(255) +# shared_runners_enabled :boolean default(TRUE), not null +# max_artifacts_size :integer default(100), not null +# runners_registration_token :string(255) +# require_two_factor_authentication :boolean default(TRUE) +# two_factor_grace_period :integer default(48) # class ApplicationSetting < ActiveRecord::Base @@ -58,6 +60,9 @@ class ApplicationSetting < ActiveRecord::Base allow_blank: true, email: true + validates :two_factor_grace_period, + numericality: { greater_than_or_equal_to: 0 } + validates_each :restricted_visibility_levels do |record, attr, value| unless value.nil? value.each do |level| @@ -112,6 +117,8 @@ class ApplicationSetting < ActiveRecord::Base import_sources: ['github','bitbucket','gitlab','gitorious','google_code','fogbugz','git'], shared_runners_enabled: Settings.gitlab_ci['shared_runners_enabled'], max_artifacts_size: Settings.artifacts['max_size'], + require_two_factor_authentication: false, + two_factor_grace_period: 48 ) end diff --git a/db/migrate/20151218154042_add_tfa_to_application_settings.rb b/db/migrate/20151218154042_add_tfa_to_application_settings.rb new file mode 100644 index 00000000000..dd95db775c5 --- /dev/null +++ b/db/migrate/20151218154042_add_tfa_to_application_settings.rb @@ -0,0 +1,8 @@ +class AddTfaToApplicationSettings < ActiveRecord::Migration + def change + change_table :application_settings do |t| + t.boolean :require_two_factor_authentication, default: false + t.integer :two_factor_grace_period, default: 48 + end + end +end diff --git a/db/schema.rb b/db/schema.rb index 0d53105b057..631979a7fd8 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -50,6 +50,8 @@ ActiveRecord::Schema.define(version: 20151224123230) do t.boolean "shared_runners_enabled", default: true, null: false t.integer "max_artifacts_size", default: 100, null: false t.string "runners_registration_token" + t.boolean "require_two_factor_authentication", default: false + t.integer "two_factor_grace_period", default: 48 end create_table "audit_events", force: :cascade do |t| diff --git a/spec/models/application_setting_spec.rb b/spec/models/application_setting_spec.rb index 5f64453a35f..35d8220ae54 100644 --- a/spec/models/application_setting_spec.rb +++ b/spec/models/application_setting_spec.rb @@ -27,6 +27,7 @@ # admin_notification_email :string(255) # shared_runners_enabled :boolean default(TRUE), not null # max_artifacts_size :integer default(100), not null +# runners_registration_token :string(255) # require 'spec_helper' -- cgit v1.2.1 From 31fb2b7702345fbf597c2cb17466567776433a56 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 24 Dec 2015 00:02:52 -0200 Subject: Grace period support for TFA --- app/controllers/application_controller.rb | 20 ++++++++++++++++++-- .../profiles/two_factor_auths_controller.rb | 14 +++++++++++++- app/helpers/auth_helper.rb | 12 ++++++++++++ app/views/profiles/two_factor_auths/new.html.haml | 1 + config/routes.rb | 1 + .../20151221234414_add_tfa_additional_fields.rb | 7 +++++++ db/schema.rb | 1 + 7 files changed, 53 insertions(+), 3 deletions(-) create mode 100644 db/migrate/20151221234414_add_tfa_additional_fields.rb diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb index e15d83631b3..978a269ca52 100644 --- a/app/controllers/application_controller.rb +++ b/app/controllers/application_controller.rb @@ -225,9 +225,13 @@ class ApplicationController < ActionController::Base end def check_tfa_requirement - if two_factor_authentication_required? && current_user && !current_user.two_factor_enabled + if two_factor_authentication_required? && current_user && !current_user.two_factor_enabled && !skip_two_factor? + grace_period_started = current_user.otp_grace_period_started_at + grace_period_deadline = grace_period_started + two_factor_grace_period.hours + + deadline_text = "until #{l(grace_period_deadline)}" unless two_factor_grace_period_expired?(grace_period_started) redirect_to new_profile_two_factor_auth_path, - alert: 'You must configure Two-Factor Authentication in your account' + alert: "You must configure Two-Factor Authentication in your account #{deadline_text}" end end @@ -369,6 +373,18 @@ class ApplicationController < ActionController::Base current_application_settings.require_two_factor_authentication end + def two_factor_grace_period + current_application_settings.two_factor_grace_period + end + + def two_factor_grace_period_expired?(date) + date && (date + two_factor_grace_period.hours) < Time.current + end + + def skip_two_factor? + session[:skip_tfa] && session[:skip_tfa] > Time.current + end + def redirect_to_home_page_url? # If user is not signed-in and tries to access root_path - redirect him to landing page # Don't redirect to the default URL to prevent endless redirections diff --git a/app/controllers/profiles/two_factor_auths_controller.rb b/app/controllers/profiles/two_factor_auths_controller.rb index 05c84fb720e..49629e9894a 100644 --- a/app/controllers/profiles/two_factor_auths_controller.rb +++ b/app/controllers/profiles/two_factor_auths_controller.rb @@ -4,8 +4,11 @@ class Profiles::TwoFactorAuthsController < Profiles::ApplicationController def new unless current_user.otp_secret current_user.otp_secret = User.generate_otp_secret(32) - current_user.save! end + unless current_user.otp_grace_period_started_at && two_factor_grace_period + current_user.otp_grace_period_started_at = Time.current + end + current_user.save! if current_user.changed? @qr_code = build_qr_code end @@ -36,6 +39,15 @@ class Profiles::TwoFactorAuthsController < Profiles::ApplicationController redirect_to profile_account_path end + def skip + if two_factor_grace_period_expired?(current_user.otp_grace_period_started_at) + redirect_to new_profile_two_factor_auth_path, alert: 'Cannot skip two factor authentication setup' + else + session[:skip_tfa] = current_user.otp_grace_period_started_at + two_factor_grace_period.hours + redirect_to root_path + end + end + private def build_qr_code diff --git a/app/helpers/auth_helper.rb b/app/helpers/auth_helper.rb index 2c81ea1623c..0cfc0565e84 100644 --- a/app/helpers/auth_helper.rb +++ b/app/helpers/auth_helper.rb @@ -50,5 +50,17 @@ module AuthHelper current_user.identities.exists?(provider: provider.to_s) end + def two_factor_skippable? + current_application_settings.require_two_factor_authentication && + !current_user.two_factor_enabled && + current_application_settings.two_factor_grace_period && + !two_factor_grace_period_expired? + end + + def two_factor_grace_period_expired? + current_user.otp_grace_period_started_at && + (current_user.otp_grace_period_started_at + current_application_settings.two_factor_grace_period.hours) < Time.current + end + extend self end diff --git a/app/views/profiles/two_factor_auths/new.html.haml b/app/views/profiles/two_factor_auths/new.html.haml index 92dc58c10d7..1a5b6efce35 100644 --- a/app/views/profiles/two_factor_auths/new.html.haml +++ b/app/views/profiles/two_factor_auths/new.html.haml @@ -38,3 +38,4 @@ = text_field_tag :pin_code, nil, class: "form-control", required: true, autofocus: true .form-actions = submit_tag 'Submit', class: 'btn btn-success' + = link_to 'Configure it later', skip_profile_two_factor_auth_path, :method => :patch, class: 'btn btn-cancel' if two_factor_skippable? diff --git a/config/routes.rb b/config/routes.rb index b9242327de1..3e7d9f78710 100644 --- a/config/routes.rb +++ b/config/routes.rb @@ -297,6 +297,7 @@ Rails.application.routes.draw do resource :two_factor_auth, only: [:new, :create, :destroy] do member do post :codes + patch :skip end end end diff --git a/db/migrate/20151221234414_add_tfa_additional_fields.rb b/db/migrate/20151221234414_add_tfa_additional_fields.rb new file mode 100644 index 00000000000..c16df47932f --- /dev/null +++ b/db/migrate/20151221234414_add_tfa_additional_fields.rb @@ -0,0 +1,7 @@ +class AddTfaAdditionalFields < ActiveRecord::Migration + def change + change_table :users do |t| + t.datetime :otp_grace_period_started_at, null: true + end + end +end diff --git a/db/schema.rb b/db/schema.rb index 631979a7fd8..49fa258660d 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -840,6 +840,7 @@ ActiveRecord::Schema.define(version: 20151224123230) do t.integer "layout", default: 0 t.boolean "hide_project_limit", default: false t.string "unlock_token" + t.datetime "otp_grace_period_started_at" end add_index "users", ["admin"], name: "index_users_on_admin", using: :btree -- cgit v1.2.1 From b61a5bc20cbfcd8a2c914f19e8786a989bf69198 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 24 Dec 2015 02:04:41 -0200 Subject: specs for forced two-factor authentication and grace period simplified code and fixed stuffs --- app/controllers/application_controller.rb | 10 ++--- .../profiles/two_factor_auths_controller.rb | 9 +++- spec/features/login_spec.rb | 52 ++++++++++++++++++++++ 3 files changed, 63 insertions(+), 8 deletions(-) diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb index 978a269ca52..a945b38e35f 100644 --- a/app/controllers/application_controller.rb +++ b/app/controllers/application_controller.rb @@ -226,12 +226,7 @@ class ApplicationController < ActionController::Base def check_tfa_requirement if two_factor_authentication_required? && current_user && !current_user.two_factor_enabled && !skip_two_factor? - grace_period_started = current_user.otp_grace_period_started_at - grace_period_deadline = grace_period_started + two_factor_grace_period.hours - - deadline_text = "until #{l(grace_period_deadline)}" unless two_factor_grace_period_expired?(grace_period_started) - redirect_to new_profile_two_factor_auth_path, - alert: "You must configure Two-Factor Authentication in your account #{deadline_text}" + redirect_to new_profile_two_factor_auth_path end end @@ -377,7 +372,8 @@ class ApplicationController < ActionController::Base current_application_settings.two_factor_grace_period end - def two_factor_grace_period_expired?(date) + def two_factor_grace_period_expired? + date = current_user.otp_grace_period_started_at date && (date + two_factor_grace_period.hours) < Time.current end diff --git a/app/controllers/profiles/two_factor_auths_controller.rb b/app/controllers/profiles/two_factor_auths_controller.rb index 49629e9894a..4f125eb7e05 100644 --- a/app/controllers/profiles/two_factor_auths_controller.rb +++ b/app/controllers/profiles/two_factor_auths_controller.rb @@ -10,6 +10,13 @@ class Profiles::TwoFactorAuthsController < Profiles::ApplicationController end current_user.save! if current_user.changed? + if two_factor_grace_period_expired? + flash.now[:alert] = 'You must configure Two-Factor Authentication in your account.' + else + grace_period_deadline = current_user.otp_grace_period_started_at + two_factor_grace_period.hours + flash.now[:alert] = "You must configure Two-Factor Authentication in your account until #{l(grace_period_deadline)}." + end + @qr_code = build_qr_code end @@ -40,7 +47,7 @@ class Profiles::TwoFactorAuthsController < Profiles::ApplicationController end def skip - if two_factor_grace_period_expired?(current_user.otp_grace_period_started_at) + if two_factor_grace_period_expired? redirect_to new_profile_two_factor_auth_path, alert: 'Cannot skip two factor authentication setup' else session[:skip_tfa] = current_user.otp_grace_period_started_at + two_factor_grace_period.hours diff --git a/spec/features/login_spec.rb b/spec/features/login_spec.rb index 922c76285d1..2451e56fe7c 100644 --- a/spec/features/login_spec.rb +++ b/spec/features/login_spec.rb @@ -98,4 +98,56 @@ feature 'Login', feature: true do expect(page).to have_content('Invalid login or password.') end end + + describe 'with required two-factor authentication enabled' do + let(:user) { create(:user) } + before(:each) { stub_application_setting(require_two_factor_authentication: true) } + + context 'with grace period defined' do + before(:each) do + stub_application_setting(two_factor_grace_period: 48) + login_with(user) + end + + context 'within the grace period' do + it 'redirects to two-factor configuration page' do + expect(current_path).to eq new_profile_two_factor_auth_path + expect(page).to have_content('You must configure Two-Factor Authentication in your account until') + end + + it 'two-factor configuration is skippable' do + expect(current_path).to eq new_profile_two_factor_auth_path + + click_link 'Configure it later' + expect(current_path).to eq root_path + end + end + + context 'after the grace period' do + let(:user) { create(:user, otp_grace_period_started_at: 9999.hours.ago) } + + it 'redirects to two-factor configuration page' do + expect(current_path).to eq new_profile_two_factor_auth_path + expect(page).to have_content('You must configure Two-Factor Authentication in your account.') + end + + it 'two-factor configuration is not skippable' do + expect(current_path).to eq new_profile_two_factor_auth_path + expect(page).not_to have_link('Configure it later') + end + end + end + + context 'without grace pariod defined' do + before(:each) do + stub_application_setting(two_factor_grace_period: 0) + login_with(user) + end + + it 'redirects to two-factor configuration page' do + expect(current_path).to eq new_profile_two_factor_auth_path + expect(page).to have_content('You must configure Two-Factor Authentication in your account.') + end + end + end end -- cgit v1.2.1 From cde06999c939c6856a62cfdf764857d712d7a863 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 24 Dec 2015 04:18:34 -0200 Subject: Add to application_settings forced TFA options --- app/controllers/admin/application_settings_controller.rb | 2 ++ app/views/admin/application_settings/_form.html.haml | 12 ++++++++++++ 2 files changed, 14 insertions(+) diff --git a/app/controllers/admin/application_settings_controller.rb b/app/controllers/admin/application_settings_controller.rb index 9dd16f8c735..2f4a855c118 100644 --- a/app/controllers/admin/application_settings_controller.rb +++ b/app/controllers/admin/application_settings_controller.rb @@ -49,6 +49,8 @@ class Admin::ApplicationSettingsController < Admin::ApplicationController :default_branch_protection, :signup_enabled, :signin_enabled, + :require_two_factor_authentication, + :two_factor_grace_period, :gravatar_enabled, :twitter_sharing_enabled, :sign_in_text, diff --git a/app/views/admin/application_settings/_form.html.haml b/app/views/admin/application_settings/_form.html.haml index 6c355366948..58f5c621f4a 100644 --- a/app/views/admin/application_settings/_form.html.haml +++ b/app/views/admin/application_settings/_form.html.haml @@ -104,6 +104,18 @@ = f.label :signin_enabled do = f.check_box :signin_enabled Sign-in enabled + .form-group + = f.label :two_factor_authentication, 'Two-Factor authentication', class: 'control-label col-sm-2' + .col-sm-10 + .checkbox + = f.label :require_two_factor_authentication do + = f.check_box :require_two_factor_authentication + Require all users to setup Two-Factor authentication + .form-group + = f.label :two_factor_authentication, 'Two-Factor grace period (hours)', class: 'control-label col-sm-2' + .col-sm-10 + = f.number_field :two_factor_grace_period, min: 0, class: 'form-control', placeholder: '0' + .help-block Amount of time (in hours) that users are allowed to skip forced configuration of two-factor authentication .form-group = f.label :restricted_signup_domains, 'Restricted domains for sign-ups', class: 'control-label col-sm-2' .col-sm-10 -- cgit v1.2.1 From 6e3fb5024addf86d48c5723168f664d09d81a969 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 24 Dec 2015 05:20:12 -0200 Subject: Updated CHANGELOG --- CHANGELOG | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG b/CHANGELOG index 2e12889cb70..4591dc3013d 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -17,6 +17,7 @@ v 8.3.1 - Fix Error 500 when doing a search in dashboard before visiting any project (Stan Hu) - Fix LDAP identity and user retrieval when special characters are used - Move Sidekiq-cron configuration to gitlab.yml + - Enable forcing Two-Factor authentication sitewide, with optional grace period v 8.3.0 - Bump rack-attack to 4.3.1 for security fix (Stan Hu) -- cgit v1.2.1 From 1249289f89feba725109ce769e685b07cf746e4b Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 24 Dec 2015 18:58:46 -0200 Subject: Fixed codestyle and added 2FA documentation --- app/controllers/application_controller.rb | 4 +-- .../profiles/two_factor_auths_controller.rb | 4 ++- doc/security/README.md | 1 + doc/security/two_factor_authentication.md | 38 ++++++++++++++++++++++ 4 files changed, 44 insertions(+), 3 deletions(-) create mode 100644 doc/security/two_factor_authentication.md diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb index a945b38e35f..d9a37a4d45f 100644 --- a/app/controllers/application_controller.rb +++ b/app/controllers/application_controller.rb @@ -13,7 +13,7 @@ class ApplicationController < ActionController::Base before_action :validate_user_service_ticket! before_action :reject_blocked! before_action :check_password_expiration - before_action :check_tfa_requirement + before_action :check_2fa_requirement before_action :ldap_security_check before_action :default_headers before_action :add_gon_variables @@ -224,7 +224,7 @@ class ApplicationController < ActionController::Base end end - def check_tfa_requirement + def check_2fa_requirement if two_factor_authentication_required? && current_user && !current_user.two_factor_enabled && !skip_two_factor? redirect_to new_profile_two_factor_auth_path end diff --git a/app/controllers/profiles/two_factor_auths_controller.rb b/app/controllers/profiles/two_factor_auths_controller.rb index 4f125eb7e05..6e91d9b4ad9 100644 --- a/app/controllers/profiles/two_factor_auths_controller.rb +++ b/app/controllers/profiles/two_factor_auths_controller.rb @@ -1,13 +1,15 @@ class Profiles::TwoFactorAuthsController < Profiles::ApplicationController - skip_before_action :check_tfa_requirement + skip_before_action :check_2fa_requirement def new unless current_user.otp_secret current_user.otp_secret = User.generate_otp_secret(32) end + unless current_user.otp_grace_period_started_at && two_factor_grace_period current_user.otp_grace_period_started_at = Time.current end + current_user.save! if current_user.changed? if two_factor_grace_period_expired? diff --git a/doc/security/README.md b/doc/security/README.md index fba6013d9c1..384df570394 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -6,3 +6,4 @@ - [Information exclusivity](information_exclusivity.md) - [Reset your root password](reset_root_password.md) - [User File Uploads](user_file_uploads.md) +- [Enforce Two-Factor authentication](two_factor_authentication.md) diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md new file mode 100644 index 00000000000..4e25a1fdc3f --- /dev/null +++ b/doc/security/two_factor_authentication.md @@ -0,0 +1,38 @@ +# Enforce Two-factor Authentication (2FA) + +Two-factor Authentication (2FA) provides an additional level of security to your +users' GitLab account. Once enabled, in addition to supplying their username and +password to login, they'll be prompted for a code generated by an application on +their phone. + +You can read more about it here: +[Two-factor Authentication (2FA)](doc/profile/two_factor_authentication.md) + +## Enabling 2FA + +Users on GitLab, can enable it without any admin's intervention. If you want to +enforce everyone to setup 2FA, you can choose from two different ways: + + 1. Enforce on next login + 2. Suggest on next login, but allow a grace period before enforcing. + +In the Admin area under **Settings** (`/admin/application_settings`), look for +the "Sign-in Restrictions" area, where you can configure both. + +If you want 2FA enforcement to take effect on next login, change the grace +period to `0` + +## Disabling 2FA for everyone + +There may be some special situations where you want to disable 2FA for everyone +even when forced 2FA is disabled. There is a rake task for that: + +``` +# use this command if you've installed GitLab with the Omnibus package +sudo gitlab-rake gitlab:two_factor:disable_for_all_users + +# if you've installed GitLab from source +sudo -u git -H bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production +``` + +**IMPORTANT: this is a permanent and irreversible action. Users will have to reactivate 2FA from scratch if they want to use it again.** -- cgit v1.2.1 From c6d25083626c9a97a0ae442298b59c2ff9351f3a Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 24 Dec 2015 16:26:52 -0500 Subject: Truncate page_description to 30 words --- app/helpers/page_layout_helper.rb | 4 ++-- spec/helpers/page_layout_helper_spec.rb | 13 +++++++++++++ 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/app/helpers/page_layout_helper.rb b/app/helpers/page_layout_helper.rb index c4eb09bea2b..4f1276f93ec 100644 --- a/app/helpers/page_layout_helper.rb +++ b/app/helpers/page_layout_helper.rb @@ -30,9 +30,9 @@ module PageLayoutHelper @page_description ||= page_description_default if description.present? - @page_description = description + @page_description = description.squish else - sanitize(@page_description.squish, tags: []) + sanitize(@page_description, tags: []).truncate_words(30) end end diff --git a/spec/helpers/page_layout_helper_spec.rb b/spec/helpers/page_layout_helper_spec.rb index 5d95beac908..530e9bab343 100644 --- a/spec/helpers/page_layout_helper_spec.rb +++ b/spec/helpers/page_layout_helper_spec.rb @@ -22,6 +22,19 @@ describe PageLayoutHelper do expect(helper.page_description).to eq 'Foo Bar Baz' end + it 'truncates' do + helper.page_description <<-LOREM.strip_heredoc + Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo + ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis + dis parturient montes, nascetur ridiculus mus. Donec quam felis, + ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa + quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, + arcu. + LOREM + + expect(helper.page_description).to end_with 'quam felis,...' + end + it 'sanitizes all HTML' do helper.page_description("Bold

Header

") -- cgit v1.2.1 From 99dc1fce5ed84fb78bd993423db9c470021ea3a2 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 24 Dec 2015 16:53:35 -0500 Subject: Use `request.fullpath` for `og:url` tag --- app/views/layouts/_head.html.haml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/views/layouts/_head.html.haml b/app/views/layouts/_head.html.haml index ad1dacac1c8..d1cb07eaa66 100644 --- a/app/views/layouts/_head.html.haml +++ b/app/views/layouts/_head.html.haml @@ -11,7 +11,7 @@ %meta{property: 'og:title', content: page_title} %meta{property: 'og:description', content: page_description} %meta{property: 'og:image', content: page_image} - %meta{property: 'og:url', content: url_for(only_path: false)} + %meta{property: 'og:url', content: request.base_url + request.fullpath} %title= page_title -- cgit v1.2.1 From ab3d855c0e1869fd1986c3bcdf7519f6b1cf1fa8 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 24 Dec 2015 17:03:54 -0500 Subject: Add support for `twitter:label` meta tags --- app/helpers/page_layout_helper.rb | 24 +++++++++++++++++++ app/models/concerns/issuable.rb | 8 +++++++ app/views/layouts/_head.html.haml | 1 + app/views/projects/issues/show.html.haml | 5 ++-- app/views/projects/merge_requests/_show.html.haml | 5 ++-- spec/helpers/page_layout_helper_spec.rb | 29 +++++++++++++++++++++++ spec/models/concerns/issuable_spec.rb | 18 ++++++++++++++ 7 files changed, 86 insertions(+), 4 deletions(-) diff --git a/app/helpers/page_layout_helper.rb b/app/helpers/page_layout_helper.rb index 4f1276f93ec..791cb9e50bd 100644 --- a/app/helpers/page_layout_helper.rb +++ b/app/helpers/page_layout_helper.rb @@ -58,6 +58,30 @@ module PageLayoutHelper end end + # Define or get attributes to be used as Twitter card metadata + # + # map - Hash of label => data pairs. Keys become labels, values become data + # + # Raises ArgumentError if given more than two attributes + def page_card_attributes(map = {}) + raise ArgumentError, 'cannot provide more than two attributes' if map.length > 2 + + @page_card_attributes ||= {} + @page_card_attributes = map.reject { |_,v| v.blank? } if map.present? + @page_card_attributes + end + + def page_card_meta_tags + tags = '' + + page_card_attributes.each_with_index do |pair, i| + tags << tag(:meta, property: "twitter:label#{i + 1}", content: pair[0]) + tags << tag(:meta, property: "twitter:data#{i + 1}", content: pair[1]) + end + + tags.html_safe + end + def header_title(title = nil, title_url = nil) if title @header_title = title diff --git a/app/models/concerns/issuable.rb b/app/models/concerns/issuable.rb index f56fd3e02d4..919833f6df5 100644 --- a/app/models/concerns/issuable.rb +++ b/app/models/concerns/issuable.rb @@ -161,6 +161,14 @@ module Issuable self.class.to_s.underscore end + # Returns a Hash of attributes to be used for Twitter card metadata + def card_attributes + { + 'Author' => author.try(:name), + 'Assignee' => assignee.try(:name) + } + end + def notes_with_associations notes.includes(:author, :project) end diff --git a/app/views/layouts/_head.html.haml b/app/views/layouts/_head.html.haml index d1cb07eaa66..434605e59ae 100644 --- a/app/views/layouts/_head.html.haml +++ b/app/views/layouts/_head.html.haml @@ -12,6 +12,7 @@ %meta{property: 'og:description', content: page_description} %meta{property: 'og:image', content: page_image} %meta{property: 'og:url', content: request.base_url + request.fullpath} + = page_card_meta_tags %title= page_title diff --git a/app/views/projects/issues/show.html.haml b/app/views/projects/issues/show.html.haml index f2a261ab426..f548383008d 100644 --- a/app/views/projects/issues/show.html.haml +++ b/app/views/projects/issues/show.html.haml @@ -1,5 +1,6 @@ -- page_title "#{@issue.title} (##{@issue.iid})", "Issues" -- page_description @issue.description +- page_title "#{@issue.title} (##{@issue.iid})", "Issues" +- page_description @issue.description +- page_card_attributes @issue.card_attributes = render "header_title" diff --git a/app/views/projects/merge_requests/_show.html.haml b/app/views/projects/merge_requests/_show.html.haml index 75f44557964..ba7c2c01e93 100644 --- a/app/views/projects/merge_requests/_show.html.haml +++ b/app/views/projects/merge_requests/_show.html.haml @@ -1,5 +1,6 @@ -- page_title "#{@merge_request.title} (##{@merge_request.iid})", "Merge Requests" -- page_description @merge_request.description +- page_title "#{@merge_request.title} (##{@merge_request.iid})", "Merge Requests" +- page_description @merge_request.description +- page_card_attributes @merge_request.card_attributes = render "header_title" diff --git a/spec/helpers/page_layout_helper_spec.rb b/spec/helpers/page_layout_helper_spec.rb index 530e9bab343..fd7107779f6 100644 --- a/spec/helpers/page_layout_helper_spec.rb +++ b/spec/helpers/page_layout_helper_spec.rb @@ -97,4 +97,33 @@ describe PageLayoutHelper do end end end + + describe 'page_card_attributes' do + it 'raises ArgumentError when given more than two attributes' do + map = { foo: 'foo', bar: 'bar', baz: 'baz' } + + expect { helper.page_card_attributes(map) }. + to raise_error(ArgumentError, /more than two attributes/) + end + + it 'rejects blank values' do + map = { foo: 'foo', bar: '' } + helper.page_card_attributes(map) + + expect(helper.page_card_attributes).to eq({ foo: 'foo' }) + end + end + + describe 'page_card_meta_tags' do + it 'returns the twitter:label and twitter:data tags' do + allow(helper).to receive(:page_card_attributes).and_return(foo: 'bar') + + tags = helper.page_card_meta_tags + + aggregate_failures do + expect(tags).to include %q() + expect(tags).to include %q() + end + end + end end diff --git a/spec/models/concerns/issuable_spec.rb b/spec/models/concerns/issuable_spec.rb index 0f13c4410cd..901eb936688 100644 --- a/spec/models/concerns/issuable_spec.rb +++ b/spec/models/concerns/issuable_spec.rb @@ -81,4 +81,22 @@ describe Issue, "Issuable" do expect(hook_data[:object_attributes]).to eq(issue.hook_attrs) end end + + describe '#card_attributes' do + it 'includes the author name' do + allow(issue).to receive(:author).and_return(double(name: 'Robert')) + allow(issue).to receive(:assignee).and_return(nil) + + expect(issue.card_attributes). + to eq({'Author' => 'Robert', 'Assignee' => nil}) + end + + it 'includes the assignee name' do + allow(issue).to receive(:author).and_return(double(name: 'Robert')) + allow(issue).to receive(:assignee).and_return(double(name: 'Douwe')) + + expect(issue.card_attributes). + to eq({'Author' => 'Robert', 'Assignee' => 'Douwe'}) + end + end end -- cgit v1.2.1 From e74affcfa84acaddc236d6dfed7be1a61470dc0e Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Fri, 25 Dec 2015 01:29:27 +0200 Subject: Add images and examples --- doc/ci/triggers/README.md | 97 ++++++++++++++++++++++++++- doc/ci/triggers/img/builds_page.png | Bin 0 -> 39713 bytes doc/ci/triggers/img/trigger_single_build.png | Bin 0 -> 2895 bytes doc/ci/triggers/img/trigger_variables.png | Bin 0 -> 5418 bytes doc/ci/triggers/img/triggers_page.png | Bin 0 -> 15889 bytes 5 files changed, 95 insertions(+), 2 deletions(-) create mode 100644 doc/ci/triggers/img/builds_page.png create mode 100644 doc/ci/triggers/img/trigger_single_build.png create mode 100644 doc/ci/triggers/img/trigger_variables.png create mode 100644 doc/ci/triggers/img/triggers_page.png diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 5990d74fda1..63661ee4858 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -2,10 +2,103 @@ _**Note:** This feature was [introduced][ci-229] in GitLab CE 7.14_ -Triggers can be used to force a rebuild of a specific branch or tag with an API -call. +Triggers can be used to force a rebuild of a specific branch, tag or commit, +with an API call. ## Add a trigger +You can add a new trigger by going to your project's **Settings > Triggers**. +The **Add trigger** button will create a new token which you can then use to +trigger a rebuild of this particular project. + +Once at least one trigger is created, on the **Triggers** page you will find +some descriptive information on how you can + +Every new trigger you create, gets assigned a different token which you can +then use inside your scripts or `.gitlab-ci.yml`. You also have a nice +overview of the time the triggers were last used. + +![Triggers page overview](img/triggers_page.png) + +## Revoke a trigger + +You can revoke a trigger any time by going at your project's +**Settings > Triggers** and hitting the **Revoke** button. The action is +irreversible. + +## Trigger a build + +To trigger a build you need to send a `POST` request to GitLab's API endpoint: + +``` +POST /projects/:id/trigger/builds +``` + +The required parameters are the trigger's `token` and the Git `ref` on which +the trigger will be performed. Valid refs are the branch, the tag or the commit +SHA. The `:id` of a project can be found by [querying the API](../api/projects.md) +or by visiting the **Triggers** page which provides self-explanatory examples. + +When a rebuild is triggered, the information is exposed in GitLab's UI under +the **Builds** page and the builds are marked as `triggered`. + +![Marked rebuilds as triggered on builds page](img/builds_page.png) + +--- + +You can see which trigger caused the rebuild by visiting the single build page. +The token of the trigger is exposed in the UI as you can see from the image +below. + +![Marked rebuilds as triggered on a single build page](img/trigger_single_build.png) + +--- + +See the [Examples](#examples) section for more details on how to actually +trigger a rebuild. + +## Pass build variables to a trigger + +You can pass any number of arbitrary variables in the trigger API call and they +will be available in GitLab CI so that they can be used in your `.gitlab-ci.yml` +file. The parameter is of the form: + +``` +variables[key]=value +``` + +This information is also exposed in the UI. + +![Build variables in UI](img/trigger_variables.png) + +--- + +See the [Examples](#examples) section below for more details. + +## Examples + +Using cURL you can trigger a rebuild with minimal effort, for example: + +```bash +curl -X POST \ + -F token=TOKEN \ + -F ref=master \ + https://gitlab.com/api/v3/projects/9/trigger/builds +``` + +In this case, the project with ID `9` will get rebuilt on `master` branch. + +You can also use triggers in your `.gitlab-ci.yml`. Let's say that you have +two projects, A and B, and you want to trigger a rebuild on the `master` +branch of project B whenever a tag on project A is created. + +```yaml +build_docs: + stage: deploy + script: + - "curl -X POST -F token=TOKEN -F ref=master https://gitlab.com/api/v3/projects/9/trigger/builds" + only: + - tags +``` [ci-229]: https://gitlab.com/gitlab-org/gitlab-ci/merge_requests/229 diff --git a/doc/ci/triggers/img/builds_page.png b/doc/ci/triggers/img/builds_page.png new file mode 100644 index 00000000000..e78794fbee7 Binary files /dev/null and b/doc/ci/triggers/img/builds_page.png differ diff --git a/doc/ci/triggers/img/trigger_single_build.png b/doc/ci/triggers/img/trigger_single_build.png new file mode 100644 index 00000000000..c25f27409d6 Binary files /dev/null and b/doc/ci/triggers/img/trigger_single_build.png differ diff --git a/doc/ci/triggers/img/trigger_variables.png b/doc/ci/triggers/img/trigger_variables.png new file mode 100644 index 00000000000..2207e8b34cb Binary files /dev/null and b/doc/ci/triggers/img/trigger_variables.png differ diff --git a/doc/ci/triggers/img/triggers_page.png b/doc/ci/triggers/img/triggers_page.png new file mode 100644 index 00000000000..268368dc3c5 Binary files /dev/null and b/doc/ci/triggers/img/triggers_page.png differ -- cgit v1.2.1 From ff8cd116a04d187f63222b3b29ebb6818dfd65e5 Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Fri, 25 Dec 2015 01:07:00 -0800 Subject: Disable --follow in `git log` to avoid loading duplicate commit data in infinite scroll `git` doesn't work properly when `--follow` and `--skip` are specified together. We could even be **omitting commits in the Web log** as a result. Here are the gory details. Let's say you ran: ``` git log -n=5 --skip=2 README ``` This is the working case since it omits `--follow`. This is what happens: 1. `git` starts at `HEAD` and traverses down the tree until it finds the top-most commit relevant to README. 2. Once this is found, this commit is returned via `get_revision_1()`. 3. If the `skip_count` is positive, decrement and repeat step 2. Otherwise go onto step 4. 4. `show_log()` gets called with that commit. 5. Repeat step 1 until we have all five entries. That's exactly what we want. What happens when you use `--follow`? You have to understand how step 1 is performed: * When you specify a pathspec on the command-line (e.g. README), a flag `prune` [gets set here](https://github.com/git/git/blob/master/revision.c#L2351). * If the `prune` flag is active, `get_commit_action()` determines whether the commit should be [scanned for matching paths](https://github.com/git/git/blob/master/revision.c#L2989). * In the case of `--follow`, however, `prune` is [disabled here](https://github.com/git/git/blob/master/revision.c#L2350). * As a result, a commit is never scanned for matching paths and therefore never pruned. `HEAD` will always get returned as the first commit, even if it's not relevant to the README. * Making matters worse, the `--skip` in the example above would actually skip every other after `HEAD` N times. If README were changed in these skipped commits, we would actually miss information! Since git uses a matching algorithm to determine whether a file was renamed, I believe `git` needs to generate a diff of each commit to do this and traverse each commit one-by-one to do this. I think that's the rationale for disabling the `prune` functionality since you can't just do a simple string comparison. Closes #4181, #4229, #3574, #2410 --- CHANGELOG | 1 + app/models/repository.rb | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index 2e12889cb70..f110b16e1f2 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -10,6 +10,7 @@ v 8.4.0 (unreleased) - Add link to merge request on build detail page. v 8.3.2 (unreleased) + - Disable --follow in `git log` to avoid loading duplicate commit data in infinite scroll (Stan Hu) - Enable "Add key" button when user fills in a proper key v 8.3.1 diff --git a/app/models/repository.rb b/app/models/repository.rb index 9f688e3b45b..a9bf4eb4033 100644 --- a/app/models/repository.rb +++ b/app/models/repository.rb @@ -76,7 +76,9 @@ class Repository path: path, limit: limit, offset: offset, - follow: path.present? + # --follow doesn't play well with --skip. See: + # https://gitlab.com/gitlab-org/gitlab-ce/issues/3574#note_3040520 + follow: false } commits = Gitlab::Git::Commit.where(options) -- cgit v1.2.1 From 662e9cffe1dc278bad06c8b44c527cac1d0e913d Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Fri, 25 Dec 2015 13:37:46 +0200 Subject: Add examples for triggers [ci skip] --- doc/ci/triggers/README.md | 81 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 76 insertions(+), 5 deletions(-) diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 63661ee4858..2c1de5859f8 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -83,22 +83,93 @@ Using cURL you can trigger a rebuild with minimal effort, for example: curl -X POST \ -F token=TOKEN \ -F ref=master \ - https://gitlab.com/api/v3/projects/9/trigger/builds + https://gitlab.example.com/api/v3/projects/9/trigger/builds ``` In this case, the project with ID `9` will get rebuilt on `master` branch. -You can also use triggers in your `.gitlab-ci.yml`. Let's say that you have -two projects, A and B, and you want to trigger a rebuild on the `master` -branch of project B whenever a tag on project A is created. + +### Triggering a build within `.gitlab-ci.yml` + +You can also benefit by using triggers in your `.gitlab-ci.yml`. Let's say that +you have two projects, A and B, and you want to trigger a rebuild on the `master` +branch of project B whenever a tag on project A is created. This is the job you +need to add in project's A `.gitlab-ci.yml`: ```yaml build_docs: stage: deploy script: - - "curl -X POST -F token=TOKEN -F ref=master https://gitlab.com/api/v3/projects/9/trigger/builds" + - "curl -X POST -F token=TOKEN -F ref=master https://gitlab.example.com/api/v3/projects/9/trigger/builds" only: - tags ``` +Now, whenever a new tag is pushed on project A, the build will run and the +`build_docs` job will be executed, triggering a rebuild of project B. The +`stage: deploy` ensures that this job will run only after all jobs with +`stage: test` complete successfully. + +_**Note:** If your project is public, passing the token in plain text is +probably not the wiser idea, so you might want to use a +[secure variable](../variables/README.md#user-defined-variables-secure-variables) +for that purpose._ + +### Making use of trigger variables + +Using trigger variables can be proven useful for a variety of reasons. + +* Identifiable jobs. Since the variable is exposed in the UI you can know + why the rebuild was triggered if you pass a variable that explains the + purpose. +* Conditional job processing. You can have conditional jobs that run whenever + a certain variable is present. + +Consider the following `.gitlab-ci.yml` where we set three +[stages](../yaml/README.md#stages) and the `upload_package` job is run only +when all jobs from the test and build stages pass. When the `UPLOAD_TO_S3` +variable is non-zero, `make upload` is run. + +```yaml +stages: +- test +- build +- package + +run_tests: + script: + - make test + +build_package: + stage: build + script: + - make build + +upload_package: + stage: package + script: + - if [ -n "${UPLOAD_TO_S3}" ]; then make upload; fi +``` + +You can then trigger a rebuild while you pass the `UPLOAD_TO_S3` variable +and the script of the `upload_package` job will run: + +```bash +curl -X POST \ + -F token=TOKEN \ + -F ref=master \ + -F "variables[UPLOAD_TO_S3]=true" \ + https://gitlab.example.com/api/v3/projects/9/trigger/builds +``` + +### Using cron to trigger nightly builds + +Whether you craft a script or just run cURL directly, you can trigger builds +in conjunction with cron. The example below triggers a build on the `master` +branch of project with ID `9` every night at `00:30`: + +```bash +30 0 * * * curl -X POST -F token=TOKEN -F ref=master https://gitlab.example.com/api/v3/projects/9/trigger/builds +``` + [ci-229]: https://gitlab.com/gitlab-org/gitlab-ci/merge_requests/229 -- cgit v1.2.1 From a1b63e12526c069a8754b5e64deb9eb15668832a Mon Sep 17 00:00:00 2001 From: Valery Sizov Date: Thu, 24 Dec 2015 19:46:35 +0200 Subject: revert back vote buttons to issue and MR pages --- CHANGELOG | 1 + app/assets/javascripts/awards_handler.coffee | 15 ++++++++------- app/models/note.rb | 9 ++++++++- spec/models/note_spec.rb | 11 ++++++++--- 4 files changed, 25 insertions(+), 11 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 78d717a709f..a20c3978a11 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -9,6 +9,7 @@ v 8.4.0 (unreleased) - Add "Frequently used" category to emoji picker - Add CAS support (tduehr) - Add link to merge request on build detail page. + - Revert back upvote and downvote button to the issue and MR pages v 8.3.2 (unreleased) - Enable "Add key" button when user fills in a proper key diff --git a/app/assets/javascripts/awards_handler.coffee b/app/assets/javascripts/awards_handler.coffee index 04bf5cc7bb5..eb1c3669032 100644 --- a/app/assets/javascripts/awards_handler.coffee +++ b/app/assets/javascripts/awards_handler.coffee @@ -43,15 +43,19 @@ class @AwardsHandler decrementCounter: (emoji) -> counter = @findEmojiIcon(emoji).siblings(".counter") + emojiIcon = counter.parent() if parseInt(counter.text()) > 1 counter.text(parseInt(counter.text()) - 1) - counter.parent().removeClass("active") + emojiIcon.removeClass("active") @removeMeFromAuthorList(emoji) + else if emoji =="thumbsup" || emoji == "thumbsdown" + emojiIcon.tooltip("destroy") + counter.text(0) + emojiIcon.removeClass("active") else - award = counter.parent() - award.tooltip("destroy") - award.remove() + emojiIcon.tooltip("destroy") + emojiIcon.remove() removeMeFromAuthorList: (emoji) -> award_block = @findEmojiIcon(emoji).parent() @@ -127,9 +131,6 @@ class @AwardsHandler getFrequentlyUsedEmojis: -> frequently_used_emojis = ($.cookie('frequently_used_emojis') || "").split(",") - - frequently_used_emojis = ["thumbsup", "thumbsdown"].concat(frequently_used_emojis) - _.compact(_.uniq(frequently_used_emojis)) renderFrequentlyUsedBlock: -> diff --git a/app/models/note.rb b/app/models/note.rb index 8c5b5836f9a..1222d99cf1f 100644 --- a/app/models/note.rb +++ b/app/models/note.rb @@ -107,9 +107,16 @@ class Note < ActiveRecord::Base end def grouped_awards + notes = {} + awards.select(:note).distinct.map do |note| - [ note.note, where(note: note.note) ] + notes[note.note] = where(note: note.note) end + + notes["thumbsup"] ||= Note.none + notes["thumbsdown"] ||= Note.none + + notes end end diff --git a/spec/models/note_spec.rb b/spec/models/note_spec.rb index b7006fa5e68..593d8f76215 100644 --- a/spec/models/note_spec.rb +++ b/spec/models/note_spec.rb @@ -137,9 +137,14 @@ describe Note, models: true do create :note, note: "smile", is_award: true end - it "returns grouped array of notes" do - expect(Note.grouped_awards.first.first).to eq("smile") - expect(Note.grouped_awards.first.last).to match_array(Note.all) + it "returns grouped hash of notes" do + expect(Note.grouped_awards.keys.size).to eq(3) + expect(Note.grouped_awards["smile"]).to match_array(Note.all) + end + + it "returns thumbsup and thumbsdown always" do + expect(Note.grouped_awards["thumbsup"]).to match_array(Note.none) + expect(Note.grouped_awards["thumbsdown"]).to match_array(Note.none) end end -- cgit v1.2.1 From 195dc3a7461468b060b9a9c6860aca37cebc9d8d Mon Sep 17 00:00:00 2001 From: Valery Sizov Date: Fri, 25 Dec 2015 12:08:53 +0200 Subject: add sorting of awards --- app/helpers/issues_helper.rb | 12 ++++++++++++ app/views/votes/_votes_block.html.haml | 2 +- features/steps/project/issues/award_emoji.rb | 2 +- spec/helpers/issues_helper_spec.rb | 7 +++++++ 4 files changed, 21 insertions(+), 2 deletions(-) diff --git a/app/helpers/issues_helper.rb b/app/helpers/issues_helper.rb index 4fe84322199..c1053554fbd 100644 --- a/app/helpers/issues_helper.rb +++ b/app/helpers/issues_helper.rb @@ -120,6 +120,18 @@ module IssuesHelper end end + def awards_sort(awards) + awards.sort_by do |award, notes| + if award == "thumbsup" + 0 + elsif award == "thumbsdown" + 1 + else + 2 + end + end.to_h + end + # Required for Banzai::Filter::IssueReferenceFilter module_function :url_for_issue end diff --git a/app/views/votes/_votes_block.html.haml b/app/views/votes/_votes_block.html.haml index e16187bb42f..ce0a0113403 100644 --- a/app/views/votes/_votes_block.html.haml +++ b/app/views/votes/_votes_block.html.haml @@ -1,5 +1,5 @@ .awards.votes-block - - votable.notes.awards.grouped_awards.each do |emoji, notes| + - awards_sort(votable.notes.awards.grouped_awards).each do |emoji, notes| .award{class: (note_active_class(notes, current_user)), title: emoji_author_list(notes, current_user)} = emoji_icon(emoji) .counter diff --git a/features/steps/project/issues/award_emoji.rb b/features/steps/project/issues/award_emoji.rb index a7e15398819..12cf8860ec6 100644 --- a/features/steps/project/issues/award_emoji.rb +++ b/features/steps/project/issues/award_emoji.rb @@ -37,7 +37,7 @@ class Spinach::Features::AwardEmoji < Spinach::FeatureSteps step 'I have award added' do page.within '.awards' do expect(page).to have_selector '.award' - expect(page.find('.award .counter')).to have_content '1' + expect(page.find('.award.active .counter')).to have_content '1' end end diff --git a/spec/helpers/issues_helper_spec.rb b/spec/helpers/issues_helper_spec.rb index 04e795025d2..68df460da2d 100644 --- a/spec/helpers/issues_helper_spec.rb +++ b/spec/helpers/issues_helper_spec.rb @@ -141,4 +141,11 @@ describe IssuesHelper do expect(note_active_class(Note.all, @note.author)).to eq("active") end end + + describe "#awards_sort" do + it "sorts a hash so thumbsup and thumbsdown are always on top" do + data = {"thumbsdown" => "some value", "lifter" => "some value", "thumbsup" => "some value"} + expect(awards_sort(data).keys).to eq(["thumbsup", "thumbsdown", "lifter"]) + end + end end -- cgit v1.2.1 From 5605e1591401796c19c91967976c98ad1ae32015 Mon Sep 17 00:00:00 2001 From: Valery Sizov Date: Fri, 25 Dec 2015 14:10:55 +0200 Subject: fix spinach --- features/steps/project/issues/award_emoji.rb | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/features/steps/project/issues/award_emoji.rb b/features/steps/project/issues/award_emoji.rb index 12cf8860ec6..1404f34cfe0 100644 --- a/features/steps/project/issues/award_emoji.rb +++ b/features/steps/project/issues/award_emoji.rb @@ -15,15 +15,17 @@ class Spinach::Features::AwardEmoji < Spinach::FeatureSteps end step 'I click to emoji in the picker' do - page.within '.emoji-menu' do + page.within '.emoji-menu-content' do page.first('.emoji-icon').click end end step 'I can remove it by clicking to icon' do page.within '.awards' do - page.first('.award').click - expect(page).to_not have_selector '.award' + expect do + page.find('.award.active').click + sleep 0.1 + end.to change{ page.all(".award").size }.from(3).to(2) end end -- cgit v1.2.1 From 9a0e16f4548bca25f6efc6cd7a4dd0af42b60042 Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Fri, 25 Dec 2015 13:20:20 +0100 Subject: Fix spec --- spec/services/notification_service_spec.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/spec/services/notification_service_spec.rb b/spec/services/notification_service_spec.rb index 213adace14b..c103752198d 100644 --- a/spec/services/notification_service_spec.rb +++ b/spec/services/notification_service_spec.rb @@ -127,6 +127,8 @@ describe NotificationService, services: true do note.project.team.members.each do |member| # User with disabled notification should not be notified next if member.id == @u_disabled.id + # Author should not be notified + next if member.id == note.author.id should_email(member) end -- cgit v1.2.1 From 6438c288a89ad1e17893eded187eba2785035aa3 Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Fri, 25 Dec 2015 13:22:32 +0100 Subject: Satisfy Rubocop --- spec/models/concerns/issuable_spec.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/models/concerns/issuable_spec.rb b/spec/models/concerns/issuable_spec.rb index 901eb936688..f9d3c56750f 100644 --- a/spec/models/concerns/issuable_spec.rb +++ b/spec/models/concerns/issuable_spec.rb @@ -88,7 +88,7 @@ describe Issue, "Issuable" do allow(issue).to receive(:assignee).and_return(nil) expect(issue.card_attributes). - to eq({'Author' => 'Robert', 'Assignee' => nil}) + to eq({ 'Author' => 'Robert', 'Assignee' => nil }) end it 'includes the assignee name' do @@ -96,7 +96,7 @@ describe Issue, "Issuable" do allow(issue).to receive(:assignee).and_return(double(name: 'Douwe')) expect(issue.card_attributes). - to eq({'Author' => 'Robert', 'Assignee' => 'Douwe'}) + to eq({ 'Author' => 'Robert', 'Assignee' => 'Douwe' }) end end end -- cgit v1.2.1 From 4a2514ff0e1cf88786dcaf168b50af540568e79d Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Fri, 25 Dec 2015 13:27:43 +0100 Subject: Add og:site_name --- app/views/layouts/_head.html.haml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/app/views/layouts/_head.html.haml b/app/views/layouts/_head.html.haml index 434605e59ae..2002f66af0e 100644 --- a/app/views/layouts/_head.html.haml +++ b/app/views/layouts/_head.html.haml @@ -1,5 +1,3 @@ -- page_title "GitLab" - %head{prefix: "og: http://ogp.me/ns#"} %meta{charset: "utf-8"} %meta{'http-equiv' => 'X-UA-Compatible', content: 'IE=edge'} @@ -8,12 +6,14 @@ %meta{name: 'referrer', content: 'origin-when-cross-origin'} -# Open Graph - http://ogp.me/ + %meta{property: 'og:site_name', content: "GitLab"} %meta{property: 'og:title', content: page_title} %meta{property: 'og:description', content: page_description} %meta{property: 'og:image', content: page_image} %meta{property: 'og:url', content: request.base_url + request.fullpath} = page_card_meta_tags + - page_title "GitLab" %title= page_title = favicon_link_tag 'favicon.ico' -- cgit v1.2.1 From 0980dda6cdab6dc35cfa9a8b7b7b67fcd73ea699 Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Fri, 25 Dec 2015 13:35:57 +0100 Subject: Add more twitter metatags. --- app/views/layouts/_head.html.haml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/app/views/layouts/_head.html.haml b/app/views/layouts/_head.html.haml index 2002f66af0e..0be00da0b9e 100644 --- a/app/views/layouts/_head.html.haml +++ b/app/views/layouts/_head.html.haml @@ -6,11 +6,17 @@ %meta{name: 'referrer', content: 'origin-when-cross-origin'} -# Open Graph - http://ogp.me/ + %meta{property: 'og:type', content: "object"} %meta{property: 'og:site_name', content: "GitLab"} %meta{property: 'og:title', content: page_title} %meta{property: 'og:description', content: page_description} %meta{property: 'og:image', content: page_image} %meta{property: 'og:url', content: request.base_url + request.fullpath} + + %meta{property: 'twitter:card', content: "summary"} + %meta{property: 'twitter:title', content: page_title} + %meta{property: 'twitter:description', content: page_description} + %meta{property: 'twitter:image', content: page_image} = page_card_meta_tags - page_title "GitLab" -- cgit v1.2.1 From 6d385e3365ffa68a3e4ddb7bf332522cceaccaff Mon Sep 17 00:00:00 2001 From: Douwe Maan Date: Fri, 25 Dec 2015 13:38:54 +0100 Subject: Add link to twitter docs --- app/views/layouts/_head.html.haml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/app/views/layouts/_head.html.haml b/app/views/layouts/_head.html.haml index 0be00da0b9e..2e0bd2007a3 100644 --- a/app/views/layouts/_head.html.haml +++ b/app/views/layouts/_head.html.haml @@ -1,9 +1,9 @@ %head{prefix: "og: http://ogp.me/ns#"} %meta{charset: "utf-8"} %meta{'http-equiv' => 'X-UA-Compatible', content: 'IE=edge'} + %meta{name: 'referrer', content: 'origin-when-cross-origin'} %meta{name: "description", content: page_description} - %meta{name: 'referrer', content: 'origin-when-cross-origin'} -# Open Graph - http://ogp.me/ %meta{property: 'og:type', content: "object"} @@ -13,6 +13,7 @@ %meta{property: 'og:image', content: page_image} %meta{property: 'og:url', content: request.base_url + request.fullpath} + -# Twitter Card - https://dev.twitter.com/cards/types/summary %meta{property: 'twitter:card', content: "summary"} %meta{property: 'twitter:title', content: page_title} %meta{property: 'twitter:description', content: page_description} -- cgit v1.2.1 From e081edc1c474dec558f54983f0d0dc8c5841eaf6 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Fri, 25 Dec 2015 15:23:06 +0200 Subject: Clean up CRIME security doc [ci skip] --- doc/security/crime_vulnerability.md | 78 +++++++++++++++++++------------------ 1 file changed, 41 insertions(+), 37 deletions(-) diff --git a/doc/security/crime_vulnerability.md b/doc/security/crime_vulnerability.md index d716bff85a5..94ba5d1375d 100644 --- a/doc/security/crime_vulnerability.md +++ b/doc/security/crime_vulnerability.md @@ -1,59 +1,63 @@ # How we manage the TLS protocol CRIME vulnerability -> CRIME ("Compression Ratio Info-leak Made Easy") is a security exploit against -secret web cookies over connections using the HTTPS and SPDY protocols that also -use data compression.[1][2] When used to recover the content of secret -authentication cookies, it allows an attacker to perform session hijacking on an +> CRIME ("Compression Ratio Info-leak Made Easy") is a security exploit against +secret web cookies over connections using the HTTPS and SPDY protocols that also +use data compression. When used to recover the content of secret +authentication cookies, it allows an attacker to perform session hijacking on an authenticated web session, allowing the launching of further attacks. ([CRIME](https://en.wikipedia.org/w/index.php?title=CRIME&oldid=692423806)) ### Description -The TLS Protocol CRIME Vulnerability affects compression over HTTPS therefore -it warns against using SSL Compression, take gzip for example, or SPDY which -optionally uses compression as well. +The TLS Protocol CRIME Vulnerability affects compression over HTTPS, therefore +it warns against using SSL Compression (for example gzip) or SPDY which +optionally uses compression as well. -GitLab support both gzip and SPDY and manages the CRIME vulnerability by -deactivating gzip when https is enabled and not activating the compression -feature on SDPY. +GitLab supports both gzip and [SPDY][ngx-spdy] and mitigates the CRIME +vulnerability by deactivating gzip when HTTPS is enabled. You can see the +sources of the files in question: -Take a look at our configuration file for NGINX if you'd like to explore how the -conditions are setup for gzip deactivation on this link: -[GitLab NGINX File](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb). - -For SPDY you can also watch how its implmented on NGINX at [GitLab NGINX File](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb) -but take into consideration the NGINX documentation on its default state here: -[Module ngx_http_spdy_module](http://nginx.org/en/docs/http/ngx_http_spdy_module.html). +* [Source installation NGINX file][source-nginx] +* [Omnibus installation NGINX file][omnibus-nginx] +Although SPDY is enabled in Omnibus installations, CRIME relies on compression +(the 'C') and the default compression level in NGINX's SPDY module is 0 +(no compression). ### Nessus -The Nessus scanner reports a possible CRIME vunerability for GitLab similar to the -following format: - - Description +The Nessus scanner, [reports a possible CRIME vulnerability][nessus] in GitLab +similar to the following format: - This remote service has one of two configurations that are known to be required for the CRIME attack: - SSL/TLS compression is enabled. - TLS advertises the SPDY protocol earlier than version 4. +``` +Description - ... +This remote service has one of two configurations that are known to be required for the CRIME attack: +SSL/TLS compression is enabled. +TLS advertises the SPDY protocol earlier than version 4. - Output +... - The following configuration indicates that the remote service may be vulnerable to the CRIME attack: - SPDY support earlier than version 4 is advertised. +Output -*[This](http://www.tenable.com/plugins/index.php?view=single&id=62565) is a complete description from Nessus.* +The following configuration indicates that the remote service may be vulnerable to the CRIME attack: +SPDY support earlier than version 4 is advertised. +``` -From the report above its important to note that Nessus is only checkng if TLS -advertises the SPDY protocol earlier than version 4, it does not perform an -attack nor does it check if compression is enabled. With just this approach it +From the report above it is important to note that Nessus is only checking if +TLS advertises the SPDY protocol earlier than version 4, it does not perform an +attack nor does it check if compression is enabled. With just this approach, it cannot tell that SPDY's compression is disabled and not subject to the CRIME -vulnerbility. +vulnerability. + +### References +* Nginx ["Module ngx_http_spdy_module"][ngx-spdy] +* Tenable Network Security, Inc. ["Transport Layer Security (TLS) Protocol CRIME Vulnerability"][nessus] +* Wikipedia contributors, ["CRIME"][wiki-crime] Wikipedia, The Free Encyclopedia -### Reference -* Nginx. "Module ngx_http_spdy_module", Fri. 18 Dec. -* Tenable Network Security, Inc. "Transport Layer Security (TLS) Protocol CRIME Vulnerability", Web. 15 Dec. -* Wikipedia contributors. "CRIME." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 25 Nov. 2015. Web. 15 Dec. 2015. \ No newline at end of file +[source-nginx]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/gitlab-ssl +[omnibus-nginx]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb +[ngx-spdy]: http://nginx.org/en/docs/http/ngx_http_spdy_module.html +[nessus]: https://www.tenable.com/plugins/index.php?view=single&id=62565 +[wiki-crime]: https://en.wikipedia.org/wiki/CRIME -- cgit v1.2.1 From c79ffa01b49128c609099b7048649082b5e327fb Mon Sep 17 00:00:00 2001 From: Valery Sizov Date: Fri, 25 Dec 2015 15:46:01 +0200 Subject: satisfy rubocop --- spec/helpers/issues_helper_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/helpers/issues_helper_spec.rb b/spec/helpers/issues_helper_spec.rb index 68df460da2d..ffd8ebae029 100644 --- a/spec/helpers/issues_helper_spec.rb +++ b/spec/helpers/issues_helper_spec.rb @@ -144,7 +144,7 @@ describe IssuesHelper do describe "#awards_sort" do it "sorts a hash so thumbsup and thumbsdown are always on top" do - data = {"thumbsdown" => "some value", "lifter" => "some value", "thumbsup" => "some value"} + data = { "thumbsdown" => "some value", "lifter" => "some value", "thumbsup" => "some value" } expect(awards_sort(data).keys).to eq(["thumbsup", "thumbsdown", "lifter"]) end end -- cgit v1.2.1 From 17ee4e6cc633199d2134b8c75dce2418898c8aec Mon Sep 17 00:00:00 2001 From: Igor Matsko Date: Thu, 17 Dec 2015 15:15:29 +0300 Subject: Fix user autocomplete uri --- app/assets/javascripts/users_select.js.coffee | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/assets/javascripts/users_select.js.coffee b/app/assets/javascripts/users_select.js.coffee index 12abf806bfa..9467011799f 100644 --- a/app/assets/javascripts/users_select.js.coffee +++ b/app/assets/javascripts/users_select.js.coffee @@ -117,5 +117,5 @@ class @UsersSelect callback(users) buildUrl: (url) -> - url = gon.relative_url_root + url if gon.relative_url_root? + url = gon.relative_url_root.replace(/\/$/, '') + url if gon.relative_url_root? return url -- cgit v1.2.1 From f0d46b4de6acce7d09deda5449972ebb238591cb Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Sat, 26 Dec 2015 13:50:47 +0200 Subject: Remove incomplete text [ci skip] --- doc/ci/triggers/README.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 2c1de5859f8..d41aed2eb2e 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -11,9 +11,6 @@ You can add a new trigger by going to your project's **Settings > Triggers**. The **Add trigger** button will create a new token which you can then use to trigger a rebuild of this particular project. -Once at least one trigger is created, on the **Triggers** page you will find -some descriptive information on how you can - Every new trigger you create, gets assigned a different token which you can then use inside your scripts or `.gitlab-ci.yml`. You also have a nice overview of the time the triggers were last used. -- cgit v1.2.1 From 8eed3ab35953a4d142fbb8e14fdaba3cdb9d3a29 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Sat, 26 Dec 2015 13:56:33 +0200 Subject: Fix typo on triggers docs [ci skip] --- doc/ci/triggers/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index d41aed2eb2e..9f7c1bfe6a0 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -108,7 +108,7 @@ Now, whenever a new tag is pushed on project A, the build will run and the `stage: test` complete successfully. _**Note:** If your project is public, passing the token in plain text is -probably not the wiser idea, so you might want to use a +probably not the wisest idea, so you might want to use a [secure variable](../variables/README.md#user-defined-variables-secure-variables) for that purpose._ -- cgit v1.2.1 From 4ebd447a056ad3af624814ac3094ce132d38b1e9 Mon Sep 17 00:00:00 2001 From: Sytse Sijbrandij Date: Sun, 27 Dec 2015 17:36:08 +0100 Subject: Use environment variables to configure GitLab. --- config/initializers/1_settings.rb | 2 +- doc/README.md | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/config/initializers/1_settings.rb b/config/initializers/1_settings.rb index 816cb0c02a9..c151ea01d55 100644 --- a/config/initializers/1_settings.rb +++ b/config/initializers/1_settings.rb @@ -144,7 +144,7 @@ Settings.gitlab['default_projects_limit'] ||= 10 Settings.gitlab['default_branch_protection'] ||= 2 Settings.gitlab['default_can_create_group'] = true if Settings.gitlab['default_can_create_group'].nil? Settings.gitlab['default_theme'] = Gitlab::Themes::APPLICATION_DEFAULT if Settings.gitlab['default_theme'].nil? -Settings.gitlab['host'] ||= 'localhost' +Settings.gitlab['host'] ||= ENV['GITLAB_HOST'] || 'localhost' Settings.gitlab['ssh_host'] ||= Settings.gitlab.host Settings.gitlab['https'] = false if Settings.gitlab['https'].nil? Settings.gitlab['port'] ||= Settings.gitlab.https ? 443 : 80 diff --git a/doc/README.md b/doc/README.md index f40a257b42f..d82ff8b908b 100644 --- a/doc/README.md +++ b/doc/README.md @@ -56,6 +56,7 @@ - [Issue closing](customization/issue_closing.md) Customize how to close an issue from commit messages. - [Libravatar](customization/libravatar.md) Use Libravatar for user avatars. - [Log system](logs/logs.md) Log system. +- [Environmental Variables](administration/environmental_variables.md) to configure GitLab. - [Operations](operations/README.md) Keeping GitLab up and running - [Raketasks](raketasks/README.md) Backups, maintenance, automatic web hook setup and the importing of projects. - [Security](security/README.md) Learn what you can do to further secure your GitLab instance. -- cgit v1.2.1 From fe2f1b44813116ed1e1cafef293460ae8a4cf11a Mon Sep 17 00:00:00 2001 From: Sytse Sijbrandij Date: Sun, 27 Dec 2015 17:36:50 +0100 Subject: Add documentation and example file for environent variables. --- config/database.yml.env | 9 ++++++ doc/administration/enviroment_variables.md | 45 ++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) create mode 100644 config/database.yml.env create mode 100644 doc/administration/enviroment_variables.md diff --git a/config/database.yml.env b/config/database.yml.env new file mode 100644 index 00000000000..4fdc8eee7f5 --- /dev/null +++ b/config/database.yml.env @@ -0,0 +1,9 @@ +<%= ENV['RAILS_ENV'] %>: + adapter: <%= ENV['GITLAB_DATABASE_ADAPTER'] || 'postgresql'' %> + encoding: <%= ENV['GITLAB_DATABASE_ENCODING'] || 'unicode' %> + database: <%= ENV['GITLAB_DATABASE_DATABASE'] || "gitlab_#{ENV['RAILS_ENV']}" %> + pool: <%= ENV['GITLAB_DATABASE_POOL'] || '10' %> + username: <%= ENV['GITLAB_DATABASE_USERNAME'] || 'root' %> + password: <%= ENV['GITLAB_DATABASE_PASSWORD'] || '' %> + host: <%= ENV['GITLAB_DATABASE_HOST'] || 'localhost' %> + port: <%= ENV['GITLAB_DATABASE_PORT'] || '5432' %> diff --git a/doc/administration/enviroment_variables.md b/doc/administration/enviroment_variables.md new file mode 100644 index 00000000000..8c9e2fd03ad --- /dev/null +++ b/doc/administration/enviroment_variables.md @@ -0,0 +1,45 @@ +# Environment Variables + +## Introduction + +Commonly people configure GitLab via the gitlab.rb configuration file in the Omnibus package. + +But if you prefer to use environment variables we allow that too. + +## Supported environment variables + +Variable | Type | Explanation +--- | --- | --- +GITLAB_ROOT_PASSWORD | string | sets the password for the `root` user on installation +GITLAB_HOST | url | hostname of the GitLab server includes http or https +RAILS_ENV | production/development/staging/test | Rails environment +DATABASE_URL | url | For example: postgresql://localhost/blog_development?pool=5 + +## Complete database variables + +As explained in the [Heroku documentation](https://devcenter.heroku.com/articles/rails-database-connection-behavior) the DATABASE_URL doesn't let you set: + +- adapter +- database +- username +- password +- host +- port + +To do set these please `cp config/database.yml.rb config/database.yml` and use the following variables: + +Variable | Default +--- | --- +GITLAB_DATABASE_ADAPTER | postgresql +GITLAB_DATABASE_ENCODING | unicode +GITLAB_DATABASE_DATABASE | gitlab_#{ENV['RAILS_ENV'] +GITLAB_DATABASE_POOL | 10 +GITLAB_DATABASE_USERNAME | root +GITLAB_DATABASE_PASSWORD | +GITLAB_DATABASE_HOST | localhost +GITLAB_DATABASE_PORT | 5432 + +## Other variables + +We welcome merge requests to make more settings configurable via variables. +Please stick to the naming scheme "GITLAB_#{name 1_settings.rb in upper case}". -- cgit v1.2.1 From 9f7d379c2a018c86671bfc157fe1f0cf4e31e25e Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Sun, 27 Dec 2015 09:03:06 -0800 Subject: Add support for Google reCAPTCHA in user registration to prevent spammers --- CHANGELOG | 1 + Gemfile | 3 +++ Gemfile.lock | 3 +++ app/controllers/registrations_controller.rb | 23 +++++++++++++++++++++++ app/controllers/sessions_controller.rb | 13 +++++++------ app/views/devise/shared/_signup_box.html.haml | 3 +++ config/gitlab.yml.example | 6 ++++++ config/initializers/1_settings.rb | 7 +++++++ config/initializers/recaptcha.rb | 6 ++++++ 9 files changed, 59 insertions(+), 6 deletions(-) create mode 100644 config/initializers/recaptcha.rb diff --git a/CHANGELOG b/CHANGELOG index a20c3978a11..e54a6669080 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,6 +1,7 @@ Please view this file on the master branch, on stable branches it's out of date. v 8.4.0 (unreleased) + - Add support for Google reCAPTCHA in user registration to prevent spammers (Stan Hu) - Implement new UI for group page - Implement search inside emoji picker - Add API support for looking up a user by username (Stan Hu) diff --git a/Gemfile b/Gemfile index db54bf2f186..a63c84fe94b 100644 --- a/Gemfile +++ b/Gemfile @@ -35,6 +35,9 @@ gem 'omniauth-twitter', '~> 1.2.0' gem 'omniauth_crowd' gem 'rack-oauth2', '~> 1.2.1' +# reCAPTCHA protection +gem 'recaptcha', require: 'recaptcha/rails' + # Two-factor authentication gem 'devise-two-factor', '~> 2.0.0' gem 'rqrcode-rails3', '~> 0.1.7' diff --git a/Gemfile.lock b/Gemfile.lock index 4f4b10c0fb7..664a02c4bab 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -566,6 +566,8 @@ GEM trollop rdoc (3.12.2) json (~> 1.4) + recaptcha (1.0.2) + json redcarpet (3.3.3) redis (3.2.2) redis-actionpack (4.0.1) @@ -924,6 +926,7 @@ DEPENDENCIES raphael-rails (~> 2.1.2) rblineprof rdoc (~> 3.6) + recaptcha redcarpet (~> 3.3.3) redis-namespace redis-rails (~> 4.0.0) diff --git a/app/controllers/registrations_controller.rb b/app/controllers/registrations_controller.rb index 3b3dc86cb68..283831f8149 100644 --- a/app/controllers/registrations_controller.rb +++ b/app/controllers/registrations_controller.rb @@ -1,10 +1,21 @@ class RegistrationsController < Devise::RegistrationsController before_action :signup_enabled? + include Recaptcha::Verify def new redirect_to(new_user_session_path) end + def create + if !Gitlab.config.recaptcha.enabled || verify_recaptcha + super + else + flash[:alert] = "There was an error with the reCAPTCHA code below. Please re-enter the code." + flash.delete :recaptcha_error + render action: 'new' + end + end + def destroy DeleteUserService.new(current_user).execute(current_user) @@ -38,4 +49,16 @@ class RegistrationsController < Devise::RegistrationsController def sign_up_params params.require(:user).permit(:username, :email, :name, :password, :password_confirmation) end + + def resource_name + :user + end + + def resource + @resource ||= User.new + end + + def devise_mapping + @devise_mapping ||= Devise.mappings[:user] + end end diff --git a/app/controllers/sessions_controller.rb b/app/controllers/sessions_controller.rb index 1b60d3e27d0..da4b35d322b 100644 --- a/app/controllers/sessions_controller.rb +++ b/app/controllers/sessions_controller.rb @@ -1,5 +1,6 @@ class SessionsController < Devise::SessionsController include AuthenticatesWithTwoFactor + include Recaptcha::ClientHelper prepend_before_action :authenticate_with_two_factor, only: [:create] prepend_before_action :store_redirect_path, only: [:new] @@ -40,7 +41,7 @@ class SessionsController < Devise::SessionsController User.find(session[:otp_user_id]) end end - + def store_redirect_path redirect_path = if request.referer.present? && (params['redirect_to_referer'] == 'yes') @@ -87,14 +88,14 @@ class SessionsController < Devise::SessionsController provider = Gitlab.config.omniauth.auto_sign_in_with_provider return unless provider.present? - # Auto sign in with an Omniauth provider only if the standard "you need to sign-in" alert is - # registered or no alert at all. In case of another alert (such as a blocked user), it is safer + # Auto sign in with an Omniauth provider only if the standard "you need to sign-in" alert is + # registered or no alert at all. In case of another alert (such as a blocked user), it is safer # to do nothing to prevent redirection loops with certain Omniauth providers. return unless flash[:alert].blank? || flash[:alert] == I18n.t('devise.failure.unauthenticated') - + # Prevent alert from popping up on the first page shown after authentication. - flash[:alert] = nil - + flash[:alert] = nil + redirect_to user_omniauth_authorize_path(provider.to_sym) end diff --git a/app/views/devise/shared/_signup_box.html.haml b/app/views/devise/shared/_signup_box.html.haml index 9dc6aeffd59..3c079d7e2f8 100644 --- a/app/views/devise/shared/_signup_box.html.haml +++ b/app/views/devise/shared/_signup_box.html.haml @@ -17,6 +17,9 @@ = f.email_field :email, class: "form-control middle", placeholder: "Email", required: true .form-group.append-bottom-20#password-strength = f.password_field :password, class: "form-control bottom", id: "user_password_sign_up", placeholder: "Password", required: true + %div + - if Gitlab.config.recaptcha.enabled + = recaptcha_tags %div = f.submit "Sign up", class: "btn-create btn" diff --git a/config/gitlab.yml.example b/config/gitlab.yml.example index db68b5512b8..79fc7423b31 100644 --- a/config/gitlab.yml.example +++ b/config/gitlab.yml.example @@ -346,6 +346,12 @@ production: &base # cas3: # session_duration: 28800 + # reCAPTCHA settings. See: http://www.google.com/recaptcha + recaptcha: + enabled: false + public_key: 'YOUR_PUBLIC_KEY' + private_key: 'YOUR_PRIVATE_KEY' + # Shared file storage settings shared: # path: /mnt/gitlab # Default: shared diff --git a/config/initializers/1_settings.rb b/config/initializers/1_settings.rb index 816cb0c02a9..0dc4838fec1 100644 --- a/config/initializers/1_settings.rb +++ b/config/initializers/1_settings.rb @@ -131,6 +131,13 @@ Settings.omniauth.cas3['session_duration'] ||= 8.hours Settings.omniauth['session_tickets'] ||= Settingslogic.new({}) Settings.omniauth.session_tickets['cas3'] = 'ticket' +# ReCAPTCHA settings +Settings['recaptcha'] ||= Settingslogic.new({}) +Settings.recaptcha['enabled'] = false if Settings.recaptcha['enabled'].nil? +Settings.recaptcha['public_key'] ||= Settings.recaptcha['public_key'] +Settings.recaptcha['private_key'] ||= Settings.recaptcha['private_key'] + + Settings['shared'] ||= Settingslogic.new({}) Settings.shared['path'] = File.expand_path(Settings.shared['path'] || "shared", Rails.root) diff --git a/config/initializers/recaptcha.rb b/config/initializers/recaptcha.rb new file mode 100644 index 00000000000..7509e327ae1 --- /dev/null +++ b/config/initializers/recaptcha.rb @@ -0,0 +1,6 @@ +if Gitlab.config.recaptcha.enabled + Recaptcha.configure do |config| + config.public_key = Gitlab.config.recaptcha['public_key'] + config.private_key = Gitlab.config.recaptcha['private_key'] + end +end -- cgit v1.2.1 From e7314e4c278dc754882c95a4c037e72fd583f4d2 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Sun, 27 Dec 2015 20:34:46 -0500 Subject: Fix the "Show all" link for the keyboard shortcut modal --- app/assets/javascripts/dispatcher.js.coffee | 2 +- app/assets/javascripts/shortcuts.js.coffee | 10 +++++++--- app/views/help/_shortcuts.html.haml | 8 -------- 3 files changed, 8 insertions(+), 12 deletions(-) diff --git a/app/assets/javascripts/dispatcher.js.coffee b/app/assets/javascripts/dispatcher.js.coffee index 599b4c49540..69e061ce6e9 100644 --- a/app/assets/javascripts/dispatcher.js.coffee +++ b/app/assets/javascripts/dispatcher.js.coffee @@ -49,7 +49,7 @@ class Dispatcher new DropzoneInput($('.release-form')) when 'projects:merge_requests:show' new Diff() - shortcut_handler = new ShortcutsIssuable() + shortcut_handler = new ShortcutsIssuable(true) new ZenMode() when "projects:merge_requests:diffs" new Diff() diff --git a/app/assets/javascripts/shortcuts.js.coffee b/app/assets/javascripts/shortcuts.js.coffee index e9aeb1e9525..4d915bfc8c5 100644 --- a/app/assets/javascripts/shortcuts.js.coffee +++ b/app/assets/javascripts/shortcuts.js.coffee @@ -7,7 +7,7 @@ class @Shortcuts selectiveHelp: (e) => Shortcuts.showHelp(e, @enabledHelp) - + @showHelp: (e, location) -> if $('#modal-shortcuts').length > 0 $('#modal-shortcuts').modal('show') @@ -17,8 +17,7 @@ class @Shortcuts dataType: 'script', success: (e) -> if location and location.length > 0 - for l in location - $(l).show() + $(l).show() for l in location else $('.hidden-shortcut').show() $('.js-more-help-button').remove() @@ -28,3 +27,8 @@ class @Shortcuts @focusSearch: (e) -> $('#search').focus() e.preventDefault() + +$(document).on 'click.more_help', '.js-more-help-button', (e) -> + $(@).remove() + $('.hidden-shortcut').show() + e.preventDefault() diff --git a/app/views/help/_shortcuts.html.haml b/app/views/help/_shortcuts.html.haml index 7e801b5332d..e8e331dd109 100644 --- a/app/views/help/_shortcuts.html.haml +++ b/app/views/help/_shortcuts.html.haml @@ -219,11 +219,3 @@ %td.shortcut .key r %td Reply (quoting selected text) - - -:javascript - $('.js-more-help-button').click(function (e) { - $(this).remove()l - $('.hidden-shortcut').show(); - e.preventDefault(); - }); -- cgit v1.2.1 From 7a20c6da9155fe112ceb3ec9e83cd0255eb15858 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Sun, 27 Dec 2015 21:19:01 -0500 Subject: Bump brakeman to ~> 3.1.0 --- Gemfile | 2 +- Gemfile.lock | 22 ++++++++++++++-------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/Gemfile b/Gemfile index db54bf2f186..cdf8884da91 100644 --- a/Gemfile +++ b/Gemfile @@ -214,7 +214,7 @@ gem 'net-ssh', '~> 3.0.1' group :development do gem "foreman" - gem 'brakeman', '3.0.1', require: false + gem 'brakeman', '~> 3.1.0', require: false gem "annotate", "~> 2.6.0" gem "letter_opener", '~> 1.1.2' diff --git a/Gemfile.lock b/Gemfile.lock index 4f4b10c0fb7..59051b9f36f 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -84,15 +84,17 @@ GEM bootstrap-sass (3.3.5) autoprefixer-rails (>= 5.0.0.1) sass (>= 3.2.19) - brakeman (3.0.1) + brakeman (3.1.4) erubis (~> 2.6) fastercsv (~> 1.5) haml (>= 3.0, < 5.0) - highline (~> 1.6.20) + highline (>= 1.6.20, < 2.0) multi_json (~> 1.2) - ruby2ruby (~> 2.1.1) - ruby_parser (~> 3.5.0) + ruby2ruby (>= 2.1.1, < 2.3.0) + ruby_parser (~> 3.7.0) + safe_yaml (>= 1.0) sass (~> 3.0) + slim (>= 1.3.6, < 4.0) terminal-table (~> 1.4) browser (1.0.1) builder (3.2.2) @@ -347,7 +349,7 @@ GEM html2haml (>= 1.0.1) railties (>= 4.0.1) hashie (3.4.3) - highline (1.6.21) + highline (1.7.8) hike (1.2.3) hipchat (1.5.2) httparty @@ -636,10 +638,10 @@ GEM ruby-saml (1.0.0) nokogiri (>= 1.5.10) uuid (~> 2.3) - ruby2ruby (2.1.4) + ruby2ruby (2.2.0) ruby_parser (~> 3.1) sexp_processor (~> 4.0) - ruby_parser (3.5.0) + ruby_parser (3.7.2) sexp_processor (~> 4.1) rubyntlm (0.5.2) rubypants (0.2.0) @@ -693,6 +695,9 @@ GEM tilt (>= 1.3, < 3) six (0.2.0) slack-notifier (1.2.1) + slim (3.0.6) + temple (~> 0.7.3) + tilt (>= 1.3.3, < 2.1) slop (3.6.0) spinach (0.8.10) colorize @@ -734,6 +739,7 @@ GEM railties (>= 3.2.5, < 5) teaspoon-jasmine (2.2.0) teaspoon (>= 1.0.0) + temple (0.7.6) term-ansicolor (1.3.2) tins (~> 1.0) terminal-table (1.5.2) @@ -830,7 +836,7 @@ DEPENDENCIES better_errors (~> 1.0.1) binding_of_caller (~> 0.7.2) bootstrap-sass (~> 3.0) - brakeman (= 3.0.1) + brakeman (~> 3.1.0) browser (~> 1.0.0) bullet bundler-audit -- cgit v1.2.1 From 1bda2e43a2f5ffdd4afa7ae73798ca4e36c0de9f Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Sun, 27 Dec 2015 21:19:14 -0500 Subject: Prevent an XSS warning from the updated Brakeman --- app/controllers/projects/commits_controller.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/controllers/projects/commits_controller.rb b/app/controllers/projects/commits_controller.rb index 58fb946dbc2..04a88990bf4 100644 --- a/app/controllers/projects/commits_controller.rb +++ b/app/controllers/projects/commits_controller.rb @@ -9,7 +9,7 @@ class Projects::CommitsController < Projects::ApplicationController def show @repo = @project.repository - @limit, @offset = (params[:limit] || 40), (params[:offset] || 0) + @limit, @offset = (params[:limit] || 40).to_i, (params[:offset] || 0).to_i @commits = @repo.commits(@ref, @path, @limit, @offset) @note_counts = project.notes.where(commit_id: @commits.map(&:id)). -- cgit v1.2.1 From 6f0ee5c9089d469a59879fbc0ffd6a2f3d69687e Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Sun, 27 Dec 2015 19:47:10 -0800 Subject: Fix failed spec --- app/controllers/registrations_controller.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/controllers/registrations_controller.rb b/app/controllers/registrations_controller.rb index 283831f8149..ee1006dea49 100644 --- a/app/controllers/registrations_controller.rb +++ b/app/controllers/registrations_controller.rb @@ -55,7 +55,7 @@ class RegistrationsController < Devise::RegistrationsController end def resource - @resource ||= User.new + @resource ||= User.new(sign_up_params) end def devise_mapping -- cgit v1.2.1 From 4c6591c9220676c97ddf2dda36e8e855d3196a74 Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Sun, 27 Dec 2015 19:47:21 -0800 Subject: Make sign-up form retain fields after failed CAPTCHA test --- app/views/devise/shared/_signup_box.html.haml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/app/views/devise/shared/_signup_box.html.haml b/app/views/devise/shared/_signup_box.html.haml index 3c079d7e2f8..49fab016bfa 100644 --- a/app/views/devise/shared/_signup_box.html.haml +++ b/app/views/devise/shared/_signup_box.html.haml @@ -6,17 +6,18 @@ .login-heading %h3 Create an account .login-body + - user = params[:user].present? ? params[:user] : {} = form_for(resource, as: resource_name, url: registration_path(resource_name)) do |f| .devise-errors = devise_error_messages! %div - = f.text_field :name, class: "form-control top", placeholder: "Name", required: true + = f.text_field :name, class: "form-control top", value: user[:name], placeholder: "Name", required: true %div - = f.text_field :username, class: "form-control middle", placeholder: "Username", required: true + = f.text_field :username, class: "form-control middle", value: user[:username], placeholder: "Username", required: true %div - = f.email_field :email, class: "form-control middle", placeholder: "Email", required: true + = f.email_field :email, class: "form-control middle", value: user[:email], placeholder: "Email", required: true .form-group.append-bottom-20#password-strength - = f.password_field :password, class: "form-control bottom", id: "user_password_sign_up", placeholder: "Password", required: true + = f.password_field :password, class: "form-control bottom", value: user[:password], id: "user_password_sign_up", placeholder: "Password", required: true %div - if Gitlab.config.recaptcha.enabled = recaptcha_tags -- cgit v1.2.1 From 9e0f532f3eca474bbb4bdf49ea744afb23178b82 Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Sun, 27 Dec 2015 20:36:33 -0800 Subject: Add documentation for using reCAPTCHA --- config/initializers/1_settings.rb | 2 +- doc/integration/README.md | 1 + doc/integration/recaptcha.md | 56 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 1 deletion(-) create mode 100644 doc/integration/recaptcha.md diff --git a/config/initializers/1_settings.rb b/config/initializers/1_settings.rb index 0dc4838fec1..fb7adc77284 100644 --- a/config/initializers/1_settings.rb +++ b/config/initializers/1_settings.rb @@ -133,7 +133,7 @@ Settings.omniauth.session_tickets['cas3'] = 'ticket' # ReCAPTCHA settings Settings['recaptcha'] ||= Settingslogic.new({}) -Settings.recaptcha['enabled'] = false if Settings.recaptcha['enabled'].nil? +Settings.recaptcha['enabled'] = false if Settings.recaptcha['enabled'].nil? Settings.recaptcha['public_key'] ||= Settings.recaptcha['public_key'] Settings.recaptcha['private_key'] ||= Settings.recaptcha['private_key'] diff --git a/doc/integration/README.md b/doc/integration/README.md index 6263353851f..2a9f76533b7 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -13,6 +13,7 @@ See the documentation below for details on how to configure these services. - [Slack](slack.md) Integrate with the Slack chat service - [OAuth2 provider](oauth_provider.md) OAuth2 application creation - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages +- [reCAPTCHA](recaptcha.md) Configure GitLab to use Google reCAPTCHA for new users GitLab Enterprise Edition contains [advanced JIRA support](http://doc.gitlab.com/ee/integration/jira.html) and [advanced Jenkins support](http://doc.gitlab.com/ee/integration/jenkins.html). diff --git a/doc/integration/recaptcha.md b/doc/integration/recaptcha.md new file mode 100644 index 00000000000..7e6f7e7e30a --- /dev/null +++ b/doc/integration/recaptcha.md @@ -0,0 +1,56 @@ +# reCAPTCHA + +GitLab leverages [Google's reCAPTCHA](https://www.google.com/recaptcha/intro/index.html) +to protect against spam and abuse. GitLab displays the CAPTCHA form on the sign-up page +to confirm that a real user, not a bot, is attempting to create an account. + +## Configuration + +To use reCAPTCHA, first you must create a public and private key. + +1. Go to the URL: https://www.google.com/recaptcha/admin + +1. Fill out the form necessary to obtain reCAPTCHA keys. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. Enable reCAPTCHA and add the settings: + + For omnibus package: + + ```ruby + gitlab_rails['recaptcha_enabled'] = true + gitlab_rails['recaptcha_public_key'] = 'YOUR_PUBLIC_KEY' + gitlab_rails['recaptcha_private_key'] = 'YOUR_PUBLIC_KEY' + ``` + + For installation from source: + + ``` + recaptcha: + enabled: true + public_key: 'YOUR_PUBLIC_KEY' + private_key: 'YOUR_PRIVATE_KEY' + ``` + +1. Change 'YOUR_PUBLIC_KEY' to the public key from step 2. + +1. Change 'YOUR_PRIVATE_KEY' to the private key from step 2. + +1. Save the configuration file. + +1. Restart GitLab. -- cgit v1.2.1 From e4ec34468fe84633ac6a66f85d71405746af9be1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rub=C3=A9n=20D=C3=A1vila?= Date: Sun, 27 Dec 2015 21:37:50 -0500 Subject: Precompile assets before running feature specs. #4662 --- .gitlab-ci.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index a8da3de83f8..c23a7a3bf0e 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -12,6 +12,7 @@ before_script: spec:feature: script: + - RAILS_ENV=test bundle exec rake assets:precompile 2>/dev/null - RAILS_ENV=test SIMPLECOV=true bundle exec rake spec:feature tags: - ruby -- cgit v1.2.1 From 004b85540f2542a0c4bbdf62bd19db71adee7917 Mon Sep 17 00:00:00 2001 From: Valery Sizov Date: Mon, 28 Dec 2015 10:11:01 +0200 Subject: Do not show frequently used emojis when empty --- app/assets/javascripts/awards_handler.coffee | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/app/assets/javascripts/awards_handler.coffee b/app/assets/javascripts/awards_handler.coffee index eb1c3669032..619abb1fb07 100644 --- a/app/assets/javascripts/awards_handler.coffee +++ b/app/assets/javascripts/awards_handler.coffee @@ -134,15 +134,16 @@ class @AwardsHandler _.compact(_.uniq(frequently_used_emojis)) renderFrequentlyUsedBlock: -> - frequently_used_emojis = @getFrequentlyUsedEmojis() + if $.cookie('frequently_used_emojis') + frequently_used_emojis = @getFrequentlyUsedEmojis() - ul = $("
    ") + ul = $("
      ") - for emoji in frequently_used_emojis - do (emoji) -> - $(".emoji-menu-content [data-emoji='#{emoji}']").closest("li").clone().appendTo(ul) + for emoji in frequently_used_emojis + do (emoji) -> + $(".emoji-menu-content [data-emoji='#{emoji}']").closest("li").clone().appendTo(ul) - $("input.emoji-search").after(ul).after($("
      ").text("Frequently used")) + $("input.emoji-search").after(ul).after($("
      ").text("Frequently used")) setupSearch: -> $("input.emoji-search").keyup (ev) => -- cgit v1.2.1 From 61fba0f4cca1348978efd510ebb57ad55d991fdd Mon Sep 17 00:00:00 2001 From: Sytse Sijbrandij Date: Mon, 28 Dec 2015 09:37:59 +0100 Subject: Thanks Robert for the corrections in the environment variables docs. --- config/database.yml.env | 2 +- doc/administration/enviroment_variables.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/config/database.yml.env b/config/database.yml.env index 4fdc8eee7f5..b2ff23cb5ab 100644 --- a/config/database.yml.env +++ b/config/database.yml.env @@ -1,5 +1,5 @@ <%= ENV['RAILS_ENV'] %>: - adapter: <%= ENV['GITLAB_DATABASE_ADAPTER'] || 'postgresql'' %> + adapter: <%= ENV['GITLAB_DATABASE_ADAPTER'] || 'postgresql' %> encoding: <%= ENV['GITLAB_DATABASE_ENCODING'] || 'unicode' %> database: <%= ENV['GITLAB_DATABASE_DATABASE'] || "gitlab_#{ENV['RAILS_ENV']}" %> pool: <%= ENV['GITLAB_DATABASE_POOL'] || '10' %> diff --git a/doc/administration/enviroment_variables.md b/doc/administration/enviroment_variables.md index 8c9e2fd03ad..d7f5cb7c21f 100644 --- a/doc/administration/enviroment_variables.md +++ b/doc/administration/enviroment_variables.md @@ -26,7 +26,7 @@ As explained in the [Heroku documentation](https://devcenter.heroku.com/articles - host - port -To do set these please `cp config/database.yml.rb config/database.yml` and use the following variables: +To do so please `cp config/database.yml.env config/database.yml` and use the following variables: Variable | Default --- | --- -- cgit v1.2.1 From 83d42c1518274dc0af0f49fda3a10e846569cbcc Mon Sep 17 00:00:00 2001 From: Valery Sizov Date: Fri, 25 Dec 2015 18:13:55 +0200 Subject: Revert upvotes and downvotes params to MR API --- app/models/concerns/issuable.rb | 6 ++---- app/models/note.rb | 2 -- doc/api/merge_requests.md | 17 +++++------------ doc/api/notes.md | 3 +-- lib/api/entities.rb | 1 - spec/models/concerns/issuable_spec.rb | 14 ++++++++++++++ 6 files changed, 22 insertions(+), 21 deletions(-) diff --git a/app/models/concerns/issuable.rb b/app/models/concerns/issuable.rb index 919833f6df5..18a00f95b48 100644 --- a/app/models/concerns/issuable.rb +++ b/app/models/concerns/issuable.rb @@ -95,14 +95,12 @@ module Issuable opened? || reopened? end - # Deprecated. Still exists to preserve API compatibility. def downvotes - 0 + notes.awards.where(note: "thumbsdown").count end - # Deprecated. Still exists to preserve API compatibility. def upvotes - 0 + notes.awards.where(note: "thumbsup").count end def subscribed?(user) diff --git a/app/models/note.rb b/app/models/note.rb index 1222d99cf1f..1ca86b2592f 100644 --- a/app/models/note.rb +++ b/app/models/note.rb @@ -346,12 +346,10 @@ class Note < ActiveRecord::Base read_attribute(:system) end - # Deprecated. Still exists to preserve API compatibility. def downvote? false end - # Deprecated. Still exists to preserve API compatibility. def upvote? false end diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 366a1f8abec..8bc0a67067a 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -4,8 +4,7 @@ Get all merge requests for this project. The `state` parameter can be used to get only merge requests with a given state (`opened`, `closed`, or `merged`) or all of them (`all`). -The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. With GitLab 8.2 the return fields `upvotes` and -`downvotes` are deprecated and always return `0`. +The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. ``` GET /projects/:id/merge_requests @@ -58,7 +57,7 @@ Parameters: ## Get single MR -Shows information about a single merge request. With GitLab 8.2 the return fields `upvotes` and `downvotes` are deprecated and always return `0`. +Shows information about a single merge request. ``` GET /projects/:id/merge_request/:merge_request_id @@ -141,8 +140,6 @@ Parameters: ## Get single MR changes Shows information about the merge request including its files and changes. -With GitLab 8.2 the return fields `upvotes` and `downvotes` are deprecated and -always return `0`. ``` GET /projects/:id/merge_request/:merge_request_id/changes @@ -213,9 +210,7 @@ Parameters: ## Create MR -Creates a new merge request. With GitLab 8.2 the return fields `upvotes` and ` -downvotes` are deprecated and always return `0`. - +Creates a new merge request. ``` POST /projects/:id/merge_requests ``` @@ -266,8 +261,7 @@ If an error occurs, an error number and a message explaining the reason is retur ## Update MR -Updates an existing merge request. You can change the target branch, title, or even close the MR. With GitLab 8.2 the return fields `upvotes` and `downvotes` -are deprecated and always return `0`. +Updates an existing merge request. You can change the target branch, title, or even close the MR. ``` PUT /projects/:id/merge_request/:merge_request_id @@ -318,8 +312,7 @@ If an error occurs, an error number and a message explaining the reason is retur ## Accept MR -Merge changes submitted with MR using this API. With GitLab 8.2 the return -fields `upvotes` and `downvotes` are deprecated and always return `0`. +Merge changes submitted with MR using this API. If merge success you get `200 OK`. diff --git a/doc/api/notes.md b/doc/api/notes.md index 4d7ef288df8..d4d63e825ab 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -6,8 +6,7 @@ Notes are comments on snippets, issues or merge requests. ### List project issue notes -Gets a list of all notes for a single issue. With GitLab 8.2 the return fields -`upvote` and `downvote` are deprecated and always return `false`. +Gets a list of all notes for a single issue. ``` GET /projects/:id/issues/:issue_id/notes diff --git a/lib/api/entities.rb b/lib/api/entities.rb index f8511ac5f5c..26e7c956e8f 100644 --- a/lib/api/entities.rb +++ b/lib/api/entities.rb @@ -166,7 +166,6 @@ module API class MergeRequest < ProjectEntity expose :target_branch, :source_branch - # deprecated, always returns 0 expose :upvotes, :downvotes expose :author, :assignee, using: Entities::UserBasic expose :source_project_id, :target_project_id diff --git a/spec/models/concerns/issuable_spec.rb b/spec/models/concerns/issuable_spec.rb index f9d3c56750f..021d62cdf0c 100644 --- a/spec/models/concerns/issuable_spec.rb +++ b/spec/models/concerns/issuable_spec.rb @@ -99,4 +99,18 @@ describe Issue, "Issuable" do to eq({ 'Author' => 'Robert', 'Assignee' => 'Douwe' }) end end + + describe "votes" do + before do + author = create :user + project = create :empty_project + issue.notes.awards.create!(note: "thumbsup", author: author, project: project) + issue.notes.awards.create!(note: "thumbsdown", author: author, project: project) + end + + it "returns correct values" do + expect(issue.upvotes).to eq(1) + expect(issue.downvotes).to eq(1) + end + end end -- cgit v1.2.1 From ddca57d3f25e52e5e4061dd3610f1269efe2400d Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 28 Dec 2015 11:33:40 +0100 Subject: Use String#delete for removing double quotes --- lib/gitlab/metrics/obfuscated_sql.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/gitlab/metrics/obfuscated_sql.rb b/lib/gitlab/metrics/obfuscated_sql.rb index 7b15670aa6b..481aca56efb 100644 --- a/lib/gitlab/metrics/obfuscated_sql.rb +++ b/lib/gitlab/metrics/obfuscated_sql.rb @@ -37,7 +37,7 @@ module Gitlab # InfluxDB escapes double quotes upon output, so lets get rid of them # whenever we can. if Gitlab::Database.postgresql? - sql = sql.gsub('"', '') + sql = sql.delete('"') end sql -- cgit v1.2.1 From db7bbadf95d9f2266a055ee50a67fa895b0f21a9 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 28 Dec 2015 11:34:11 +0100 Subject: Fixed styling of MetricsWorker specs --- spec/workers/metrics_worker_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/workers/metrics_worker_spec.rb b/spec/workers/metrics_worker_spec.rb index 2acd0f8ba30..18260ea0c24 100644 --- a/spec/workers/metrics_worker_spec.rb +++ b/spec/workers/metrics_worker_spec.rb @@ -33,7 +33,7 @@ describe MetricsWorker do it 'drops empty tags' do metrics = worker.prepare_metrics([ - { 'values' => {}, 'tags' => { 'cats' => '', 'dogs' => nil }} + { 'values' => {}, 'tags' => { 'cats' => '', 'dogs' => nil } } ]) expect(metrics).to eq([{ values: {}, tags: {} }]) -- cgit v1.2.1 From 1be5668ae0e663015d384ea7d8b404f9eeb5b478 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 28 Dec 2015 13:14:48 +0100 Subject: Added host option for InfluxDB --- config/gitlab.yml.example | 1 + lib/gitlab/metrics.rb | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/config/gitlab.yml.example b/config/gitlab.yml.example index 24f3f6f02dc..acfc86bf4d1 100644 --- a/config/gitlab.yml.example +++ b/config/gitlab.yml.example @@ -422,6 +422,7 @@ production: &base # Ban an IP for one hour (3600s) after too many auth attempts # bantime: 3600 metrics: + host: localhost enabled: false # The name of the InfluxDB database to store metrics in. database: gitlab diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb index ce89be636d3..d6f60732455 100644 --- a/lib/gitlab/metrics.rb +++ b/lib/gitlab/metrics.rb @@ -52,11 +52,12 @@ module Gitlab # "@foo ||= bar" is _not_ thread-safe. if enabled? @pool = ConnectionPool.new(size: pool_size, timeout: timeout) do + host = Settings.metrics['host'] db = Settings.metrics['database'] user = Settings.metrics['username'] pw = Settings.metrics['password'] - InfluxDB::Client.new(db, username: user, password: pw) + InfluxDB::Client.new(db, host: host, username: user, password: pw) end end end -- cgit v1.2.1 From c68f8533aabe5e6fcc516ac8a67d29aaacb1b40b Mon Sep 17 00:00:00 2001 From: Job van der Voort Date: Mon, 28 Dec 2015 14:02:18 +0100 Subject: add issue weight to contributing --- CONTRIBUTING.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 950824e35ab..b9c2b3d2f8e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -155,6 +155,28 @@ sudo -u git -H bundle exec rake gitlab:env:info) ``` +### Issue weight + +Issue weight allows us to get an idea of the amount of work required to solve +one or multiple issues. This makes it possible to schedule work more accurately. + +You are encouraged to set the weight of any issue. Following the guidelines +below will make it easy to manage this, without unnecessary overhead. + +1. Set weight for any issue at the earliest possible convenience +1. If you don't agree with a set weight, discuss with other developers until +consensus is reached about the weight +1. Issue weights are an abstract measurement of complexity of the issue. Do not +relate issue weight directly to time. This is called [anchoring](https://en.wikipedia.org/wiki/Anchoring) +and something you want to avoid. +1. Something that has a weight of 1 (or no weight) is really small and simple. +Something that is 9 is rewriting a large fundamental part of GitLab, +which might lead to many hard problems to solve. Changing some text in GitLab +is probably 1, adding a new Git Hook maybe 4 or 5, big features 7-9. +1. If something is very large, it should probably be split up in multiple +issues or chunks. You can simply not set the weight of a parent issue and set +weights to children issues. + ## Merge requests We welcome merge requests with fixes and improvements to GitLab code, tests, -- cgit v1.2.1 From 42592201d9ce57817d439c5899b63776872a4175 Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Mon, 28 Dec 2015 15:28:39 +0100 Subject: Hotfix for builds trace data integrity Issue #4246 --- app/models/ci/build.rb | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/app/models/ci/build.rb b/app/models/ci/build.rb index 7b89fe069ea..e251b1dcd97 100644 --- a/app/models/ci/build.rb +++ b/app/models/ci/build.rb @@ -194,8 +194,11 @@ module Ci end def raw_trace - if File.exist?(path_to_trace) + if File.file?(path_to_trace) File.read(path_to_trace) + elsif File.file?(old_path_to_trace) + # Temporary fix for build trace data integrity + File.read(old_path_to_trace) else # backward compatibility read_attribute :trace @@ -231,6 +234,24 @@ module Ci "#{dir_to_trace}/#{id}.log" end + ## + # Deprecated + # + def old_dir_to_trace + File.join( + Settings.gitlab_ci.builds_path, + created_at.utc.strftime("%Y_%m"), + project.ci_id.to_s + ) + end + + ## + # Deprecated + # + def old_path_to_trace + "#{old_dir_to_trace}/#{id}.log" + end + def token project.runners_token end -- cgit v1.2.1 From 297f83425e8917d6fbebdc00a612aa08ec855b5e Mon Sep 17 00:00:00 2001 From: Sytse Sijbrandij Date: Mon, 28 Dec 2015 16:33:37 +0100 Subject: Restart settings are moved too. --- config/gitlab.yml.example | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/gitlab.yml.example b/config/gitlab.yml.example index db68b5512b8..0fa6cd306f2 100644 --- a/config/gitlab.yml.example +++ b/config/gitlab.yml.example @@ -4,8 +4,8 @@ # ########################### NOTE ##################################### # This file should not receive new settings. All configuration options # -# that do not require an application restart are being moved to # -# ApplicationSetting model! # +* are being moved to ApplicationSetting model! # +# If a setting requires an application restart say so in that screen. # # If you change this file in a Merge Request, please also create # # a MR on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests # ######################################################################## -- cgit v1.2.1 From 4465e2eca0162a27142a401e6a46aa78add9177f Mon Sep 17 00:00:00 2001 From: Sytse Sijbrandij Date: Mon, 28 Dec 2015 17:08:15 +0100 Subject: Fix spelling mistake, thanks Connor. --- app/views/groups/edit.html.haml | 2 +- spec/lib/ci/gitlab_ci_yaml_processor_spec.rb | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/app/views/groups/edit.html.haml b/app/views/groups/edit.html.haml index 8daac585960..1dea77c2e96 100644 --- a/app/views/groups/edit.html.haml +++ b/app/views/groups/edit.html.haml @@ -31,7 +31,7 @@ .col-sm-10 .checkbox = f.check_box :public - %span.descr Make this group public (even if there is no any public project inside this group) + %span.descr Make this group public (even if there are no public projects inside this group) .form-actions = f.submit 'Save group', class: "btn btn-save" diff --git a/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb b/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb index c90133fbf03..d15100fc6d8 100644 --- a/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb +++ b/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb @@ -3,7 +3,7 @@ require 'spec_helper' module Ci describe GitlabCiYamlProcessor, lib: true do let(:path) { 'path' } - + describe "#builds_for_ref" do let(:type) { 'test' } @@ -29,7 +29,7 @@ module Ci when: "on_success" }) end - + describe :only do it "does not return builds if only has another branch" do config = YAML.dump({ @@ -517,7 +517,7 @@ module Ci end.to raise_error(GitlabCiYamlProcessor::ValidationError, "Unknown parameter: extra") end - it "returns errors if there is no any jobs defined" do + it "returns errors if there are no jobs defined" do config = YAML.dump({ before_script: ["bundle update"] }) expect do GitlabCiYamlProcessor.new(config, path) -- cgit v1.2.1 From 4d925f2147884812e349031a19f0d7ced9d5fdaf Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 28 Dec 2015 18:00:32 +0100 Subject: Move InfluxDB settings to ApplicationSetting --- .../admin/application_settings_controller.rb | 8 ++++ .../admin/application_settings/_form.html.haml | 53 ++++++++++++++++++++++ config/gitlab.yml.example | 21 --------- config/initializers/metrics.rb | 13 ++++-- db/migrate/20151228150906_influxdb_settings.rb | 18 ++++++++ db/schema.rb | 10 +++- lib/gitlab/metrics.rb | 32 +++++++++---- 7 files changed, 122 insertions(+), 33 deletions(-) create mode 100644 db/migrate/20151228150906_influxdb_settings.rb diff --git a/app/controllers/admin/application_settings_controller.rb b/app/controllers/admin/application_settings_controller.rb index 2f4a855c118..3c332adf1fa 100644 --- a/app/controllers/admin/application_settings_controller.rb +++ b/app/controllers/admin/application_settings_controller.rb @@ -67,6 +67,14 @@ class Admin::ApplicationSettingsController < Admin::ApplicationController :user_oauth_applications, :shared_runners_enabled, :max_artifacts_size, + :metrics_enabled, + :metrics_host, + :metrics_database, + :metrics_username, + :metrics_password, + :metrics_pool_size, + :metrics_timeout, + :metrics_method_call_threshold, restricted_visibility_levels: [], import_sources: [] ) diff --git a/app/views/admin/application_settings/_form.html.haml b/app/views/admin/application_settings/_form.html.haml index 58f5c621f4a..3cada08c2ba 100644 --- a/app/views/admin/application_settings/_form.html.haml +++ b/app/views/admin/application_settings/_form.html.haml @@ -156,5 +156,58 @@ .col-sm-10 = f.number_field :max_artifacts_size, class: 'form-control' + %fieldset + %legend Metrics + %p + These settings require a restart to take effect. + .form-group + .col-sm-offset-2.col-sm-10 + .checkbox + = f.label :metrics_enabled do + = f.check_box :metrics_enabled + Enable InfluxDB Metrics + .form-group + = f.label :metrics_host, 'InfluxDB host', class: 'control-label col-sm-2' + .col-sm-10 + = f.text_field :metrics_host, class: 'form-control', placeholder: 'influxdb.example.com' + .form-group + = f.label :metrics_database, 'InfluxDB database', class: 'control-label col-sm-2' + .col-sm-10 + = f.text_field :metrics_database, class: 'form-control', placeholder: 'gitlab' + .help-block + The name of the InfluxDB database to store data in. Users will have to + create this database manually, GitLab does not do so automatically. + .form-group + = f.label :metrics_username, 'InfluxDB username', class: 'control-label col-sm-2' + .col-sm-10 + = f.text_field :metrics_username, class: 'form-control' + .form-group + = f.label :metrics_password, 'InfluxDB password', class: 'control-label col-sm-2' + .col-sm-10 + = f.text_field :metrics_password, class: 'form-control' + .form-group + = f.label :metrics_pool_size, 'Connection pool size', class: 'control-label col-sm-2' + .col-sm-10 + = f.number_field :metrics_pool_size, class: 'form-control' + .help-block + The amount of InfluxDB connections to open. Connections are opened + lazily. Users using multi-threaded application servers should ensure + enough connections are available (at minimum the amount of application + server threads). + .form-group + = f.label :metrics_timeout, 'Connection timeout', class: 'control-label col-sm-2' + .col-sm-10 + = f.number_field :metrics_timeout, class: 'form-control' + .help-block + The amount of seconds after which an InfluxDB connection will time + out. + .form-group + = f.label :metrics_method_call_threshold, 'Method Call Threshold (ms)', class: 'control-label col-sm-2' + .col-sm-10 + = f.number_field :metrics_method_call_threshold, class: 'form-control' + .help-block + A method call is only tracked when it takes longer to complete than + the given amount of milliseconds. + .form-actions = f.submit 'Save', class: 'btn btn-primary' diff --git a/config/gitlab.yml.example b/config/gitlab.yml.example index da6d4005da6..79fc7423b31 100644 --- a/config/gitlab.yml.example +++ b/config/gitlab.yml.example @@ -449,26 +449,9 @@ production: &base # # Ban an IP for one hour (3600s) after too many auth attempts # bantime: 3600 - metrics: - host: localhost - enabled: false - # The name of the InfluxDB database to store metrics in. - database: gitlab - # Credentials to use for logging in to InfluxDB. - # username: - # password: - # The amount of InfluxDB connections to open. - # pool_size: 16 - # The timeout of a connection in seconds. - # timeout: 10 - # The minimum amount of milliseconds a method call has to take before it's - # tracked. Defaults to 10. - # method_call_threshold: 10 development: <<: *base - metrics: - enabled: false test: <<: *base @@ -511,10 +494,6 @@ test: user_filter: '' group_base: 'ou=groups,dc=example,dc=com' admin_group: '' - metrics: - enabled: false staging: <<: *base - metrics: - enabled: false diff --git a/config/initializers/metrics.rb b/config/initializers/metrics.rb index a47d2bf59a6..2e4908192a1 100644 --- a/config/initializers/metrics.rb +++ b/config/initializers/metrics.rb @@ -32,10 +32,17 @@ if Gitlab::Metrics.enabled? ) Gitlab::Metrics::Instrumentation. - instrument_class_hierarchy(ActiveRecord::Base) do |_, method| - loc = method.source_location + instrument_class_hierarchy(ActiveRecord::Base) do |klass, method| + # Instrumenting the ApplicationSetting class can lead to an infinite + # loop. Since the data is cached any way we don't really need to + # instrument it. + if klass == ApplicationSetting + false + else + loc = method.source_location - loc && loc[0].start_with?(models) && method.source =~ regex + loc && loc[0].start_with?(models) && method.source =~ regex + end end end diff --git a/db/migrate/20151228150906_influxdb_settings.rb b/db/migrate/20151228150906_influxdb_settings.rb new file mode 100644 index 00000000000..3012bd52cfd --- /dev/null +++ b/db/migrate/20151228150906_influxdb_settings.rb @@ -0,0 +1,18 @@ +class InfluxdbSettings < ActiveRecord::Migration + def change + add_column :application_settings, :metrics_enabled, :boolean, default: false + + add_column :application_settings, :metrics_host, :string, + default: 'localhost' + + add_column :application_settings, :metrics_database, :string, + default: 'gitlab' + + add_column :application_settings, :metrics_username, :string + add_column :application_settings, :metrics_password, :string + add_column :application_settings, :metrics_pool_size, :integer, default: 16 + add_column :application_settings, :metrics_timeout, :integer, default: 10 + add_column :application_settings, :metrics_method_call_threshold, + :integer, default: 10 + end +end diff --git a/db/schema.rb b/db/schema.rb index 49fa258660d..dc9ba36d0c7 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -11,7 +11,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema.define(version: 20151224123230) do +ActiveRecord::Schema.define(version: 20151228150906) do # These are extensions that must be enabled in order to support this database enable_extension "plpgsql" @@ -52,6 +52,14 @@ ActiveRecord::Schema.define(version: 20151224123230) do t.string "runners_registration_token" t.boolean "require_two_factor_authentication", default: false t.integer "two_factor_grace_period", default: 48 + t.boolean "metrics_enabled", default: false + t.string "metrics_host", default: "localhost" + t.string "metrics_database", default: "gitlab" + t.string "metrics_username" + t.string "metrics_password" + t.integer "metrics_pool_size", default: 16 + t.integer "metrics_timeout", default: 10 + t.integer "metrics_method_call_threshold", default: 10 end create_table "audit_events", force: :cascade do |t| diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb index d6f60732455..8039e8e9e9d 100644 --- a/lib/gitlab/metrics.rb +++ b/lib/gitlab/metrics.rb @@ -4,16 +4,29 @@ module Gitlab METRICS_ROOT = Rails.root.join('lib', 'gitlab', 'metrics').to_s PATH_REGEX = /^#{RAILS_ROOT}\/?/ + # Returns the current settings, ensuring we _always_ have a default set of + # metrics settings (even during tests, when the migrations are lacking, + # etc). This ensures the application is able to boot up even when the + # migrations have not been executed. + def self.settings + ApplicationSetting.current || { + metrics_pool_size: 16, + metrics_timeout: 10, + metrics_enabled: false, + metrics_method_call_threshold: 10 + } + end + def self.pool_size - Settings.metrics['pool_size'] || 16 + settings[:metrics_pool_size] end def self.timeout - Settings.metrics['timeout'] || 10 + settings[:metrics_timeout] end def self.enabled? - !!Settings.metrics['enabled'] + settings[:metrics_enabled] end def self.mri? @@ -21,7 +34,10 @@ module Gitlab end def self.method_call_threshold - Settings.metrics['method_call_threshold'] || 10 + # This is memoized since this method is called for every instrumented + # method. Loading data from an external cache on every method call slows + # things down too much. + @method_call_threshold ||= settings[:metrics_method_call_threshold] end def self.pool @@ -52,10 +68,10 @@ module Gitlab # "@foo ||= bar" is _not_ thread-safe. if enabled? @pool = ConnectionPool.new(size: pool_size, timeout: timeout) do - host = Settings.metrics['host'] - db = Settings.metrics['database'] - user = Settings.metrics['username'] - pw = Settings.metrics['password'] + host = settings[:metrics_host] + db = settings[:metrics_database] + user = settings[:metrics_username] + pw = settings[:metrics_password] InfluxDB::Client.new(db, host: host, username: user, password: pw) end -- cgit v1.2.1 From 0eab0c6efd6779e4e7c78fae4959b8897c055ecf Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 28 Dec 2015 20:28:40 +0100 Subject: Fixed syntax in gitlab.yml.example --- config/gitlab.yml.example | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/gitlab.yml.example b/config/gitlab.yml.example index 7725fa34031..a26002ec07d 100644 --- a/config/gitlab.yml.example +++ b/config/gitlab.yml.example @@ -4,7 +4,7 @@ # ########################### NOTE ##################################### # This file should not receive new settings. All configuration options # -* are being moved to ApplicationSetting model! # +# * are being moved to ApplicationSetting model! # # If a setting requires an application restart say so in that screen. # # If you change this file in a Merge Request, please also create # # a MR on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests # -- cgit v1.2.1 From a3469d914aaf28a1184247cbe72e5197ce7ca006 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Mon, 28 Dec 2015 18:21:34 -0200 Subject: reCAPTCHA is configurable through Admin Settings, no reload needed. --- .../admin/application_settings_controller.rb | 3 ++ app/controllers/registrations_controller.rb | 2 +- app/controllers/sessions_controller.rb | 5 +++ app/models/application_setting.rb | 28 ++++++++----- .../admin/application_settings/_form.html.haml | 22 ++++++++++ app/views/devise/shared/_signup_box.html.haml | 2 +- config/gitlab.yml.example | 6 --- config/initializers/1_settings.rb | 6 --- config/initializers/recaptcha.rb | 6 --- ...175719_add_recaptcha_to_application_settings.rb | 9 +++++ db/schema.rb | 5 ++- doc/integration/recaptcha.md | 47 ++++------------------ lib/gitlab/recaptcha.rb | 14 +++++++ 13 files changed, 84 insertions(+), 71 deletions(-) delete mode 100644 config/initializers/recaptcha.rb create mode 100644 db/migrate/20151228175719_add_recaptcha_to_application_settings.rb create mode 100644 lib/gitlab/recaptcha.rb diff --git a/app/controllers/admin/application_settings_controller.rb b/app/controllers/admin/application_settings_controller.rb index 3c332adf1fa..005db13fb9b 100644 --- a/app/controllers/admin/application_settings_controller.rb +++ b/app/controllers/admin/application_settings_controller.rb @@ -75,6 +75,9 @@ class Admin::ApplicationSettingsController < Admin::ApplicationController :metrics_pool_size, :metrics_timeout, :metrics_method_call_threshold, + :recaptcha_enabled, + :recaptcha_site_key, + :recaptcha_private_key, restricted_visibility_levels: [], import_sources: [] ) diff --git a/app/controllers/registrations_controller.rb b/app/controllers/registrations_controller.rb index ee1006dea49..485aaf45b01 100644 --- a/app/controllers/registrations_controller.rb +++ b/app/controllers/registrations_controller.rb @@ -7,7 +7,7 @@ class RegistrationsController < Devise::RegistrationsController end def create - if !Gitlab.config.recaptcha.enabled || verify_recaptcha + if Gitlab::Recaptcha.load_configurations! && verify_recaptcha super else flash[:alert] = "There was an error with the reCAPTCHA code below. Please re-enter the code." diff --git a/app/controllers/sessions_controller.rb b/app/controllers/sessions_controller.rb index da4b35d322b..825f85199be 100644 --- a/app/controllers/sessions_controller.rb +++ b/app/controllers/sessions_controller.rb @@ -5,6 +5,7 @@ class SessionsController < Devise::SessionsController prepend_before_action :authenticate_with_two_factor, only: [:create] prepend_before_action :store_redirect_path, only: [:new] before_action :auto_sign_in_with_provider, only: [:new] + before_action :load_recaptcha def new if Gitlab.config.ldap.enabled @@ -108,4 +109,8 @@ class SessionsController < Devise::SessionsController AuditEventService.new(user, user, options). for_authentication.security_event end + + def load_recaptcha + Gitlab::Recaptcha.load_configurations! + end end diff --git a/app/models/application_setting.rb b/app/models/application_setting.rb index 7c107da116c..be69d317d73 100644 --- a/app/models/application_setting.rb +++ b/app/models/application_setting.rb @@ -44,24 +44,32 @@ class ApplicationSetting < ActiveRecord::Base attr_accessor :restricted_signup_domains_raw validates :session_expire_delay, - presence: true, - numericality: { only_integer: true, greater_than_or_equal_to: 0 } + presence: true, + numericality: { only_integer: true, greater_than_or_equal_to: 0 } validates :home_page_url, - allow_blank: true, - url: true, - if: :home_page_url_column_exist + allow_blank: true, + url: true, + if: :home_page_url_column_exist validates :after_sign_out_path, - allow_blank: true, - url: true + allow_blank: true, + url: true validates :admin_notification_email, - allow_blank: true, - email: true + allow_blank: true, + email: true validates :two_factor_grace_period, - numericality: { greater_than_or_equal_to: 0 } + numericality: { greater_than_or_equal_to: 0 } + + validates :recaptcha_site_key, + presence: true, + if: :recaptcha_enabled + + validates :recaptcha_private_key, + presence: true, + if: :recaptcha_enabled validates_each :restricted_visibility_levels do |record, attr, value| unless value.nil? diff --git a/app/views/admin/application_settings/_form.html.haml b/app/views/admin/application_settings/_form.html.haml index 3cada08c2ba..6b240ffc97b 100644 --- a/app/views/admin/application_settings/_form.html.haml +++ b/app/views/admin/application_settings/_form.html.haml @@ -209,5 +209,27 @@ A method call is only tracked when it takes longer to complete than the given amount of milliseconds. + %fieldset + %legend Spam and Anti-bot Protection + .form-group + .col-sm-offset-2.col-sm-10 + .checkbox + = f.label :recaptcha_enabled do + = f.check_box :recaptcha_enabled + Enable reCAPTCHA + %span.help-block#recaptcha_help_block Helps preventing bots from creating accounts + + .form-group + = f.label :recaptcha_site_key, 'reCAPTCHA Site Key', class: 'control-label col-sm-2' + .col-sm-10 + = f.text_field :recaptcha_site_key, class: 'form-control' + .help-block + Generate site and private keys here: + %a{ href: 'http://www.google.com/recaptcha', target: 'blank'} http://www.google.com/recaptcha + .form-group + = f.label :recaptcha_private_key, 'reCAPTCHA Private Key', class: 'control-label col-sm-2' + .col-sm-10 + = f.text_field :recaptcha_private_key, class: 'form-control' + .form-actions = f.submit 'Save', class: 'btn btn-primary' diff --git a/app/views/devise/shared/_signup_box.html.haml b/app/views/devise/shared/_signup_box.html.haml index 49fab016bfa..cb93ff2465e 100644 --- a/app/views/devise/shared/_signup_box.html.haml +++ b/app/views/devise/shared/_signup_box.html.haml @@ -19,7 +19,7 @@ .form-group.append-bottom-20#password-strength = f.password_field :password, class: "form-control bottom", value: user[:password], id: "user_password_sign_up", placeholder: "Password", required: true %div - - if Gitlab.config.recaptcha.enabled + - if current_application_settings.recaptcha_enabled = recaptcha_tags %div = f.submit "Sign up", class: "btn-create btn" diff --git a/config/gitlab.yml.example b/config/gitlab.yml.example index 84f0dfb64c8..2d9f730c183 100644 --- a/config/gitlab.yml.example +++ b/config/gitlab.yml.example @@ -346,12 +346,6 @@ production: &base # cas3: # session_duration: 28800 - # reCAPTCHA settings. See: http://www.google.com/recaptcha - recaptcha: - enabled: false - public_key: 'YOUR_PUBLIC_KEY' - private_key: 'YOUR_PRIVATE_KEY' - # Shared file storage settings shared: # path: /mnt/gitlab # Default: shared diff --git a/config/initializers/1_settings.rb b/config/initializers/1_settings.rb index 045bab739ea..dea59f4fec8 100644 --- a/config/initializers/1_settings.rb +++ b/config/initializers/1_settings.rb @@ -131,12 +131,6 @@ Settings.omniauth.cas3['session_duration'] ||= 8.hours Settings.omniauth['session_tickets'] ||= Settingslogic.new({}) Settings.omniauth.session_tickets['cas3'] = 'ticket' -# ReCAPTCHA settings -Settings['recaptcha'] ||= Settingslogic.new({}) -Settings.recaptcha['enabled'] = false if Settings.recaptcha['enabled'].nil? -Settings.recaptcha['public_key'] ||= Settings.recaptcha['public_key'] -Settings.recaptcha['private_key'] ||= Settings.recaptcha['private_key'] - Settings['shared'] ||= Settingslogic.new({}) Settings.shared['path'] = File.expand_path(Settings.shared['path'] || "shared", Rails.root) diff --git a/config/initializers/recaptcha.rb b/config/initializers/recaptcha.rb deleted file mode 100644 index 7509e327ae1..00000000000 --- a/config/initializers/recaptcha.rb +++ /dev/null @@ -1,6 +0,0 @@ -if Gitlab.config.recaptcha.enabled - Recaptcha.configure do |config| - config.public_key = Gitlab.config.recaptcha['public_key'] - config.private_key = Gitlab.config.recaptcha['private_key'] - end -end diff --git a/db/migrate/20151228175719_add_recaptcha_to_application_settings.rb b/db/migrate/20151228175719_add_recaptcha_to_application_settings.rb new file mode 100644 index 00000000000..259fd0248d2 --- /dev/null +++ b/db/migrate/20151228175719_add_recaptcha_to_application_settings.rb @@ -0,0 +1,9 @@ +class AddRecaptchaToApplicationSettings < ActiveRecord::Migration + def change + change_table :application_settings do |t| + t.boolean :recaptcha_enabled, default: false + t.string :recaptcha_site_key + t.string :recaptcha_private_key + end + end +end diff --git a/db/schema.rb b/db/schema.rb index dc9ba36d0c7..ac6bd905eea 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -11,7 +11,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema.define(version: 20151228150906) do +ActiveRecord::Schema.define(version: 20151228175719) do # These are extensions that must be enabled in order to support this database enable_extension "plpgsql" @@ -60,6 +60,9 @@ ActiveRecord::Schema.define(version: 20151228150906) do t.integer "metrics_pool_size", default: 16 t.integer "metrics_timeout", default: 10 t.integer "metrics_method_call_threshold", default: 10 + t.boolean "recaptcha_enabled", default: false + t.string "recaptcha_site_key" + t.string "recaptcha_private_key" end create_table "audit_events", force: :cascade do |t| diff --git a/doc/integration/recaptcha.md b/doc/integration/recaptcha.md index 7e6f7e7e30a..a301d1a613c 100644 --- a/doc/integration/recaptcha.md +++ b/doc/integration/recaptcha.md @@ -6,51 +6,18 @@ to confirm that a real user, not a bot, is attempting to create an account. ## Configuration -To use reCAPTCHA, first you must create a public and private key. +To use reCAPTCHA, first you must create a site and private key. 1. Go to the URL: https://www.google.com/recaptcha/admin -1. Fill out the form necessary to obtain reCAPTCHA keys. +2. Fill out the form necessary to obtain reCAPTCHA keys. -1. On your GitLab server, open the configuration file. +3. Login to your GitLab server, with administrator credentials. - For omnibus package: +4. Go to Applications Settings on Admin Area (`admin/application_settings`) - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` +5. Fill all recaptcha fields with keys from previous steps - For installations from source: +6. Check the `Enable reCAPTCHA` checkbox - ```sh - cd /home/git/gitlab - - sudo -u git -H editor config/gitlab.yml - ``` - -1. Enable reCAPTCHA and add the settings: - - For omnibus package: - - ```ruby - gitlab_rails['recaptcha_enabled'] = true - gitlab_rails['recaptcha_public_key'] = 'YOUR_PUBLIC_KEY' - gitlab_rails['recaptcha_private_key'] = 'YOUR_PUBLIC_KEY' - ``` - - For installation from source: - - ``` - recaptcha: - enabled: true - public_key: 'YOUR_PUBLIC_KEY' - private_key: 'YOUR_PRIVATE_KEY' - ``` - -1. Change 'YOUR_PUBLIC_KEY' to the public key from step 2. - -1. Change 'YOUR_PRIVATE_KEY' to the private key from step 2. - -1. Save the configuration file. - -1. Restart GitLab. +7. Save the configuration. diff --git a/lib/gitlab/recaptcha.rb b/lib/gitlab/recaptcha.rb new file mode 100644 index 00000000000..70e7f25d518 --- /dev/null +++ b/lib/gitlab/recaptcha.rb @@ -0,0 +1,14 @@ +module Gitlab + module Recaptcha + def self.load_configurations! + if current_application_settings.recaptcha_enabled + ::Recaptcha.configure do |config| + config.public_key = current_application_settings.recaptcha_site_key + config.private_key = current_application_settings.recaptcha_private_key + end + + true + end + end + end +end -- cgit v1.2.1 From c5c6a945f29921b969663aa3e79e09148145e60c Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Mon, 28 Dec 2015 23:06:40 +0200 Subject: Fix broken link in permissions page [ci skip] --- doc/permissions/permissions.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/permissions/permissions.md b/doc/permissions/permissions.md index bcd00cfc6bf..1be78ac1823 100644 --- a/doc/permissions/permissions.md +++ b/doc/permissions/permissions.md @@ -6,11 +6,11 @@ If a user is both in a project group and in the project itself, the highest perm If a user is a GitLab administrator they receive all permissions. -On public projects the Guest role is not enforced. -All users will be able to create issues, leave comments, and pull or download the project code. +On public projects the Guest role is not enforced. +All users will be able to create issues, leave comments, and pull or download the project code. To add or import a user, you can follow the [project users and members -documentation](doc/workflow/add-user/add-user.md). +documentation](../workflow/add-user/add-user.md). ## Project -- cgit v1.2.1 From ed214a11ca1566582a084b9dc4b9e79470c1cc18 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 28 Dec 2015 22:14:33 +0100 Subject: Handle missing settings table for metrics This ensures we can still boot, even when the "application_settings" table doesn't exist. --- lib/gitlab/metrics.rb | 16 ++++++++++------ spec/lib/gitlab/metrics_spec.rb | 2 +- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb index 8039e8e9e9d..9470633b065 100644 --- a/lib/gitlab/metrics.rb +++ b/lib/gitlab/metrics.rb @@ -9,12 +9,16 @@ module Gitlab # etc). This ensures the application is able to boot up even when the # migrations have not been executed. def self.settings - ApplicationSetting.current || { - metrics_pool_size: 16, - metrics_timeout: 10, - metrics_enabled: false, - metrics_method_call_threshold: 10 - } + if ApplicationSetting.table_exists? and curr = ApplicationSetting.current + curr + else + { + metrics_pool_size: 16, + metrics_timeout: 10, + metrics_enabled: false, + metrics_method_call_threshold: 10 + } + end end def self.pool_size diff --git a/spec/lib/gitlab/metrics_spec.rb b/spec/lib/gitlab/metrics_spec.rb index ebc69f8a75f..944642909aa 100644 --- a/spec/lib/gitlab/metrics_spec.rb +++ b/spec/lib/gitlab/metrics_spec.rb @@ -29,7 +29,7 @@ describe Gitlab::Metrics do it 'returns an Array containing a file path and line number' do file, line = described_class.last_relative_application_frame - expect(line).to eq(30) + expect(line).to eq(__LINE__ - 2) expect(file).to eq('spec/lib/gitlab/metrics_spec.rb') end end -- cgit v1.2.1 From 41c74cec09af146ed7fdb4778dd72fe578eb1407 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stefan=20Kahlh=C3=B6fer?= Date: Wed, 2 Dec 2015 07:56:34 +0000 Subject: Downcased user or email search for avatar_icon. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Rubén Dávila --- app/helpers/application_helper.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/helpers/application_helper.rb b/app/helpers/application_helper.rb index 0b00b9a0702..f7f7a1a02d3 100644 --- a/app/helpers/application_helper.rb +++ b/app/helpers/application_helper.rb @@ -72,7 +72,7 @@ module ApplicationHelper if user_or_email.is_a?(User) user = user_or_email else - user = User.find_by(email: user_or_email) + user = User.find_by(email: user_or_email.downcase) end if user -- cgit v1.2.1 From e619d0b615a394a08ca1d0be59f0028c8e390b88 Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Mon, 28 Dec 2015 16:59:59 -0800 Subject: When reCAPTCHA is disabled, allow registrations to go through without a code --- app/controllers/registrations_controller.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/controllers/registrations_controller.rb b/app/controllers/registrations_controller.rb index 485aaf45b01..c48175a4c5a 100644 --- a/app/controllers/registrations_controller.rb +++ b/app/controllers/registrations_controller.rb @@ -7,7 +7,7 @@ class RegistrationsController < Devise::RegistrationsController end def create - if Gitlab::Recaptcha.load_configurations! && verify_recaptcha + if !Gitlab::Recaptcha.load_configurations! || verify_recaptcha super else flash[:alert] = "There was an error with the reCAPTCHA code below. Please re-enter the code." -- cgit v1.2.1 From d3807328d8ed9be2915f67708b093269bf0b1b9f Mon Sep 17 00:00:00 2001 From: Valery Sizov Date: Tue, 29 Dec 2015 10:11:20 +0200 Subject: note votes methids implementation --- app/models/note.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/app/models/note.rb b/app/models/note.rb index 1ca86b2592f..3d5b663c99f 100644 --- a/app/models/note.rb +++ b/app/models/note.rb @@ -347,11 +347,11 @@ class Note < ActiveRecord::Base end def downvote? - false + is_award && note == "thumbsdown" end def upvote? - false + is_award && note == "thumbsup" end def editable? -- cgit v1.2.1 From 504696453bb7d89278bf021393274e475b84ebbd Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Tue, 29 Dec 2015 08:52:06 +0100 Subject: Add hotfix that allows to access build artifacts created before 8.3 This is a temporary hotfix that allows to access build artifacts created before 8.3. See #5257. This needs to be changed after migrating CI build files. Note that `ArtifactUploader` uses `artifacts_path` to create a storage directory before and after parsisting `Ci::Build` instance, before and after moving a file to store (save and fetch a file). --- app/models/ci/build.rb | 37 ++++++++++++++++++++++++++++++++++--- app/uploaders/artifact_uploader.rb | 8 ++------ 2 files changed, 36 insertions(+), 9 deletions(-) diff --git a/app/models/ci/build.rb b/app/models/ci/build.rb index e251b1dcd97..3e67b2771c1 100644 --- a/app/models/ci/build.rb +++ b/app/models/ci/build.rb @@ -196,7 +196,7 @@ module Ci def raw_trace if File.file?(path_to_trace) File.read(path_to_trace) - elsif File.file?(old_path_to_trace) + elsif project.ci_id && File.file?(old_path_to_trace) # Temporary fix for build trace data integrity File.read(old_path_to_trace) else @@ -215,8 +215,8 @@ module Ci end def trace=(trace) - unless Dir.exists? dir_to_trace - FileUtils.mkdir_p dir_to_trace + unless Dir.exists?(dir_to_trace) + FileUtils.mkdir_p(dir_to_trace) end File.write(path_to_trace, trace) @@ -237,6 +237,9 @@ module Ci ## # Deprecated # + # This is a hotfix for CI build data integrity, see #4246 + # Should be removed in 8.4, after CI files migration has been done. + # def old_dir_to_trace File.join( Settings.gitlab_ci.builds_path, @@ -248,10 +251,38 @@ module Ci ## # Deprecated # + # This is a hotfix for CI build data integrity, see #4246 + # Should be removed in 8.4, after CI files migration has been done. + # def old_path_to_trace "#{old_dir_to_trace}/#{id}.log" end + ## + # Deprecated + # + # This contains a hotfix for CI build data integrity, see #4246 + # + # This method is used by `ArtifactUploader` to create a store_dir. + # Warning: Uploader uses it after AND before file has been stored. + # + # This method returns old path to artifacts only if it already exists. + # + def artifacts_path + old = File.join(created_at.utc.strftime('%Y_%m'), + project.ci_id.to_s, + id.to_s) + + old_store = File.join(ArtifactUploader.artifacts_path, old) + return old if project.ci_id && File.directory?(old_store) + + File.join( + created_at.utc.strftime('%Y_%m'), + project.id.to_s, + id.to_s + ) + end + def token project.runners_token end diff --git a/app/uploaders/artifact_uploader.rb b/app/uploaders/artifact_uploader.rb index 1dccc39e7e2..1b0ae6c0056 100644 --- a/app/uploaders/artifact_uploader.rb +++ b/app/uploaders/artifact_uploader.rb @@ -20,16 +20,12 @@ class ArtifactUploader < CarrierWave::Uploader::Base @build, @field = build, field end - def artifacts_path - File.join(build.created_at.utc.strftime('%Y_%m'), build.project.id.to_s, build.id.to_s) - end - def store_dir - File.join(ArtifactUploader.artifacts_path, artifacts_path) + File.join(self.class.artifacts_path, @build.artifacts_path) end def cache_dir - File.join(ArtifactUploader.artifacts_cache_path, artifacts_path) + File.join(self.class.artifacts_cache_path, @build.artifacts_path) end def file_storage? -- cgit v1.2.1 From 03478e6d5b98a723fbb349dac2c8495f75909a08 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Tue, 29 Dec 2015 13:40:08 +0100 Subject: Strip newlines from obfuscated SQL Newlines aren't really needed and they may mess with InfluxDB's line protocol. --- lib/gitlab/metrics/obfuscated_sql.rb | 2 +- spec/lib/gitlab/metrics/obfuscated_sql_spec.rb | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/lib/gitlab/metrics/obfuscated_sql.rb b/lib/gitlab/metrics/obfuscated_sql.rb index 481aca56efb..2e932fb3049 100644 --- a/lib/gitlab/metrics/obfuscated_sql.rb +++ b/lib/gitlab/metrics/obfuscated_sql.rb @@ -40,7 +40,7 @@ module Gitlab sql = sql.delete('"') end - sql + sql.gsub("\n", ' ') end end end diff --git a/spec/lib/gitlab/metrics/obfuscated_sql_spec.rb b/spec/lib/gitlab/metrics/obfuscated_sql_spec.rb index 0f01ee588c9..2b681c9fe34 100644 --- a/spec/lib/gitlab/metrics/obfuscated_sql_spec.rb +++ b/spec/lib/gitlab/metrics/obfuscated_sql_spec.rb @@ -2,6 +2,12 @@ require 'spec_helper' describe Gitlab::Metrics::ObfuscatedSQL do describe '#to_s' do + it 'replaces newlines with a space' do + sql = described_class.new("SELECT x\nFROM y") + + expect(sql.to_s).to eq('SELECT x FROM y') + end + describe 'using single values' do it 'replaces a single integer' do sql = described_class.new('SELECT x FROM y WHERE a = 10') -- cgit v1.2.1 From 620e7bb3d60c3685b494b26e256b793a47621da4 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Tue, 29 Dec 2015 13:40:42 +0100 Subject: Write to InfluxDB directly via UDP This removes the need for Sidekiq and any overhead/problems introduced by TCP. There are a few things to take into account: 1. When writing data to InfluxDB you may still get an error if the server becomes unavailable during the write. Because of this we're catching all exceptions and just ignore them (for now). 2. Writing via UDP apparently requires the timestamp to be in nanoseconds. Without this data either isn't written properly. 3. Due to the restrictions on UDP buffer sizes we're writing metrics one by one, instead of writing all of them at once. --- Procfile | 2 +- .../admin/application_settings_controller.rb | 2 +- .../admin/application_settings/_form.html.haml | 10 +-- app/workers/metrics_worker.rb | 33 --------- .../20151229102248_influxdb_udp_port_setting.rb | 5 ++ ...51229112614_influxdb_remote_database_setting.rb | 5 ++ db/schema.rb | 82 +++++++++++----------- lib/gitlab/metrics.rb | 38 +++++++++- lib/gitlab/metrics/metric.rb | 2 +- lib/gitlab/metrics/obfuscated_sql.rb | 2 +- lib/gitlab/metrics/sampler.rb | 2 +- lib/gitlab/metrics/sidekiq_middleware.rb | 7 -- lib/gitlab/metrics/transaction.rb | 2 +- spec/lib/gitlab/metrics/sampler_spec.rb | 2 +- spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb | 8 --- spec/lib/gitlab/metrics/transaction_spec.rb | 2 +- spec/lib/gitlab/metrics_spec.rb | 48 +++++++++++++ spec/workers/metrics_worker_spec.rb | 52 -------------- 18 files changed, 149 insertions(+), 155 deletions(-) delete mode 100644 app/workers/metrics_worker.rb create mode 100644 db/migrate/20151229102248_influxdb_udp_port_setting.rb create mode 100644 db/migrate/20151229112614_influxdb_remote_database_setting.rb delete mode 100644 spec/workers/metrics_worker_spec.rb diff --git a/Procfile b/Procfile index bbafdd33a2d..9cfdee7040f 100644 --- a/Procfile +++ b/Procfile @@ -3,5 +3,5 @@ # lib/support/init.d, which call scripts in bin/ . # web: bundle exec unicorn_rails -p ${PORT:="3000"} -E ${RAILS_ENV:="development"} -c ${UNICORN_CONFIG:="config/unicorn.rb"} -worker: bundle exec sidekiq -q post_receive -q mailers -q archive_repo -q system_hook -q project_web_hook -q gitlab_shell -q incoming_email -q runner -q common -q default -q metrics +worker: bundle exec sidekiq -q post_receive -q mailers -q archive_repo -q system_hook -q project_web_hook -q gitlab_shell -q incoming_email -q runner -q common -q default # mail_room: bundle exec mail_room -q -c config/mail_room.yml diff --git a/app/controllers/admin/application_settings_controller.rb b/app/controllers/admin/application_settings_controller.rb index 005db13fb9b..10e736fd362 100644 --- a/app/controllers/admin/application_settings_controller.rb +++ b/app/controllers/admin/application_settings_controller.rb @@ -69,7 +69,7 @@ class Admin::ApplicationSettingsController < Admin::ApplicationController :max_artifacts_size, :metrics_enabled, :metrics_host, - :metrics_database, + :metrics_port, :metrics_username, :metrics_password, :metrics_pool_size, diff --git a/app/views/admin/application_settings/_form.html.haml b/app/views/admin/application_settings/_form.html.haml index 6b240ffc97b..214e0209bb7 100644 --- a/app/views/admin/application_settings/_form.html.haml +++ b/app/views/admin/application_settings/_form.html.haml @@ -171,12 +171,14 @@ .col-sm-10 = f.text_field :metrics_host, class: 'form-control', placeholder: 'influxdb.example.com' .form-group - = f.label :metrics_database, 'InfluxDB database', class: 'control-label col-sm-2' + = f.label :metrics_port, 'InfluxDB port', class: 'control-label col-sm-2' .col-sm-10 - = f.text_field :metrics_database, class: 'form-control', placeholder: 'gitlab' + = f.text_field :metrics_port, class: 'form-control', placeholder: '8089' .help-block - The name of the InfluxDB database to store data in. Users will have to - create this database manually, GitLab does not do so automatically. + The UDP port to use for connecting to InfluxDB. InfluxDB requires that + your server configuration specifies a database to store data in when + sending messages to this port, without it metrics data will not be + saved. .form-group = f.label :metrics_username, 'InfluxDB username', class: 'control-label col-sm-2' .col-sm-10 diff --git a/app/workers/metrics_worker.rb b/app/workers/metrics_worker.rb deleted file mode 100644 index b15dc819c5c..00000000000 --- a/app/workers/metrics_worker.rb +++ /dev/null @@ -1,33 +0,0 @@ -class MetricsWorker - include Sidekiq::Worker - - sidekiq_options queue: :metrics - - def perform(metrics) - prepared = prepare_metrics(metrics) - - Gitlab::Metrics.pool.with do |connection| - connection.write_points(prepared) - end - end - - def prepare_metrics(metrics) - metrics.map do |hash| - new_hash = hash.symbolize_keys - - new_hash[:tags].each do |key, value| - if value.blank? - new_hash[:tags].delete(key) - else - new_hash[:tags][key] = escape_value(value) - end - end - - new_hash - end - end - - def escape_value(value) - value.to_s.gsub('=', '\\=') - end -end diff --git a/db/migrate/20151229102248_influxdb_udp_port_setting.rb b/db/migrate/20151229102248_influxdb_udp_port_setting.rb new file mode 100644 index 00000000000..ae0499f936d --- /dev/null +++ b/db/migrate/20151229102248_influxdb_udp_port_setting.rb @@ -0,0 +1,5 @@ +class InfluxdbUdpPortSetting < ActiveRecord::Migration + def change + add_column :application_settings, :metrics_port, :integer, default: 8089 + end +end diff --git a/db/migrate/20151229112614_influxdb_remote_database_setting.rb b/db/migrate/20151229112614_influxdb_remote_database_setting.rb new file mode 100644 index 00000000000..f0e1ee1e7a7 --- /dev/null +++ b/db/migrate/20151229112614_influxdb_remote_database_setting.rb @@ -0,0 +1,5 @@ +class InfluxdbRemoteDatabaseSetting < ActiveRecord::Migration + def change + remove_column :application_settings, :metrics_database + end +end diff --git a/db/schema.rb b/db/schema.rb index ac6bd905eea..df7f72d5ad4 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -11,7 +11,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema.define(version: 20151228175719) do +ActiveRecord::Schema.define(version: 20151229112614) do # These are extensions that must be enabled in order to support this database enable_extension "plpgsql" @@ -33,36 +33,36 @@ ActiveRecord::Schema.define(version: 20151228175719) do t.datetime "created_at" t.datetime "updated_at" t.string "home_page_url" - t.integer "default_branch_protection", default: 2 - t.boolean "twitter_sharing_enabled", default: true + t.integer "default_branch_protection", default: 2 + t.boolean "twitter_sharing_enabled", default: true t.text "restricted_visibility_levels" - t.boolean "version_check_enabled", default: true - t.integer "max_attachment_size", default: 10, null: false + t.boolean "version_check_enabled", default: true + t.integer "max_attachment_size", default: 10, null: false t.integer "default_project_visibility" t.integer "default_snippet_visibility" t.text "restricted_signup_domains" - t.boolean "user_oauth_applications", default: true + t.boolean "user_oauth_applications", default: true t.string "after_sign_out_path" - t.integer "session_expire_delay", default: 10080, null: false + t.integer "session_expire_delay", default: 10080, null: false t.text "import_sources" t.text "help_page_text" t.string "admin_notification_email" - t.boolean "shared_runners_enabled", default: true, null: false - t.integer "max_artifacts_size", default: 100, null: false + t.boolean "shared_runners_enabled", default: true, null: false + t.integer "max_artifacts_size", default: 100, null: false t.string "runners_registration_token" - t.boolean "require_two_factor_authentication", default: false - t.integer "two_factor_grace_period", default: 48 - t.boolean "metrics_enabled", default: false - t.string "metrics_host", default: "localhost" - t.string "metrics_database", default: "gitlab" + t.boolean "require_two_factor_authentication", default: false + t.integer "two_factor_grace_period", default: 48 + t.boolean "metrics_enabled", default: false + t.string "metrics_host", default: "localhost" t.string "metrics_username" t.string "metrics_password" - t.integer "metrics_pool_size", default: 16 - t.integer "metrics_timeout", default: 10 - t.integer "metrics_method_call_threshold", default: 10 + t.integer "metrics_pool_size", default: 16 + t.integer "metrics_timeout", default: 10 + t.integer "metrics_method_call_threshold", default: 10 t.boolean "recaptcha_enabled", default: false t.string "recaptcha_site_key" t.string "recaptcha_private_key" + t.integer "metrics_port", default: 8089 end create_table "audit_events", force: :cascade do |t| @@ -796,12 +796,12 @@ ActiveRecord::Schema.define(version: 20151228175719) do add_index "tags", ["name"], name: "index_tags_on_name", unique: true, using: :btree create_table "users", force: :cascade do |t| - t.string "email", default: "", null: false - t.string "encrypted_password", default: "", null: false + t.string "email", default: "", null: false + t.string "encrypted_password", default: "", null: false t.string "reset_password_token" t.datetime "reset_password_sent_at" t.datetime "remember_created_at" - t.integer "sign_in_count", default: 0 + t.integer "sign_in_count", default: 0 t.datetime "current_sign_in_at" t.datetime "last_sign_in_at" t.string "current_sign_in_ip" @@ -809,22 +809,22 @@ ActiveRecord::Schema.define(version: 20151228175719) do t.datetime "created_at" t.datetime "updated_at" t.string "name" - t.boolean "admin", default: false, null: false - t.integer "projects_limit", default: 10 - t.string "skype", default: "", null: false - t.string "linkedin", default: "", null: false - t.string "twitter", default: "", null: false + t.boolean "admin", default: false, null: false + t.integer "projects_limit", default: 10 + t.string "skype", default: "", null: false + t.string "linkedin", default: "", null: false + t.string "twitter", default: "", null: false t.string "authentication_token" - t.integer "theme_id", default: 1, null: false + t.integer "theme_id", default: 1, null: false t.string "bio" - t.integer "failed_attempts", default: 0 + t.integer "failed_attempts", default: 0 t.datetime "locked_at" t.string "username" - t.boolean "can_create_group", default: true, null: false - t.boolean "can_create_team", default: true, null: false + t.boolean "can_create_group", default: true, null: false + t.boolean "can_create_team", default: true, null: false t.string "state" - t.integer "color_scheme_id", default: 1, null: false - t.integer "notification_level", default: 1, null: false + t.integer "color_scheme_id", default: 1, null: false + t.integer "notification_level", default: 1, null: false t.datetime "password_expires_at" t.integer "created_by_id" t.datetime "last_credential_check_at" @@ -833,23 +833,23 @@ ActiveRecord::Schema.define(version: 20151228175719) do t.datetime "confirmed_at" t.datetime "confirmation_sent_at" t.string "unconfirmed_email" - t.boolean "hide_no_ssh_key", default: false - t.string "website_url", default: "", null: false + t.boolean "hide_no_ssh_key", default: false + t.string "website_url", default: "", null: false t.string "notification_email" - t.boolean "hide_no_password", default: false - t.boolean "password_automatically_set", default: false + t.boolean "hide_no_password", default: false + t.boolean "password_automatically_set", default: false t.string "location" t.string "encrypted_otp_secret" t.string "encrypted_otp_secret_iv" t.string "encrypted_otp_secret_salt" - t.boolean "otp_required_for_login", default: false, null: false + t.boolean "otp_required_for_login", default: false, null: false t.text "otp_backup_codes" - t.string "public_email", default: "", null: false - t.integer "dashboard", default: 0 - t.integer "project_view", default: 0 + t.string "public_email", default: "", null: false + t.integer "dashboard", default: 0 + t.integer "project_view", default: 0 t.integer "consumed_timestep" - t.integer "layout", default: 0 - t.boolean "hide_project_limit", default: false + t.integer "layout", default: 0 + t.boolean "hide_project_limit", default: false t.string "unlock_token" t.datetime "otp_grace_period_started_at" end diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb index 9470633b065..0869d007ca5 100644 --- a/lib/gitlab/metrics.rb +++ b/lib/gitlab/metrics.rb @@ -66,6 +66,39 @@ module Gitlab end end + def self.submit_metrics(metrics) + prepared = prepare_metrics(metrics) + + pool.with do |connection| + prepared.each do |metric| + begin + connection.write_points([metric]) + rescue StandardError + end + end + end + end + + def self.prepare_metrics(metrics) + metrics.map do |hash| + new_hash = hash.symbolize_keys + + new_hash[:tags].each do |key, value| + if value.blank? + new_hash[:tags].delete(key) + else + new_hash[:tags][key] = escape_value(value) + end + end + + new_hash + end + end + + def self.escape_value(value) + value.to_s.gsub('=', '\\=') + end + @hostname = Socket.gethostname # When enabled this should be set before being used as the usual pattern @@ -73,11 +106,12 @@ module Gitlab if enabled? @pool = ConnectionPool.new(size: pool_size, timeout: timeout) do host = settings[:metrics_host] - db = settings[:metrics_database] user = settings[:metrics_username] pw = settings[:metrics_password] + port = settings[:metrics_port] - InfluxDB::Client.new(db, host: host, username: user, password: pw) + InfluxDB::Client. + new(udp: { host: host, port: port }, username: user, password: pw) end end end diff --git a/lib/gitlab/metrics/metric.rb b/lib/gitlab/metrics/metric.rb index f592f4e571f..79241f56874 100644 --- a/lib/gitlab/metrics/metric.rb +++ b/lib/gitlab/metrics/metric.rb @@ -26,7 +26,7 @@ module Gitlab process_type: Sidekiq.server? ? 'sidekiq' : 'rails' ), values: @values, - timestamp: @created_at.to_i + timestamp: @created_at.to_i * 1_000_000_000 } end end diff --git a/lib/gitlab/metrics/obfuscated_sql.rb b/lib/gitlab/metrics/obfuscated_sql.rb index 2e932fb3049..fe97d7a0534 100644 --- a/lib/gitlab/metrics/obfuscated_sql.rb +++ b/lib/gitlab/metrics/obfuscated_sql.rb @@ -40,7 +40,7 @@ module Gitlab sql = sql.delete('"') end - sql.gsub("\n", ' ') + sql.tr("\n", ' ') end end end diff --git a/lib/gitlab/metrics/sampler.rb b/lib/gitlab/metrics/sampler.rb index 828ee1f8c62..998578e1c0a 100644 --- a/lib/gitlab/metrics/sampler.rb +++ b/lib/gitlab/metrics/sampler.rb @@ -46,7 +46,7 @@ module Gitlab end def flush - MetricsWorker.perform_async(@metrics.map(&:to_hash)) + Metrics.submit_metrics(@metrics.map(&:to_hash)) end def sample_memory_usage diff --git a/lib/gitlab/metrics/sidekiq_middleware.rb b/lib/gitlab/metrics/sidekiq_middleware.rb index ec10707d1fb..ad441decfa2 100644 --- a/lib/gitlab/metrics/sidekiq_middleware.rb +++ b/lib/gitlab/metrics/sidekiq_middleware.rb @@ -5,13 +5,6 @@ module Gitlab # This middleware is intended to be used as a server-side middleware. class SidekiqMiddleware def call(worker, message, queue) - # We don't want to track the MetricsWorker itself as otherwise we'll end - # up in an infinite loop. - if worker.class == MetricsWorker - yield - return - end - trans = Transaction.new begin diff --git a/lib/gitlab/metrics/transaction.rb b/lib/gitlab/metrics/transaction.rb index 568f9d6ae0c..a61dbd989e7 100644 --- a/lib/gitlab/metrics/transaction.rb +++ b/lib/gitlab/metrics/transaction.rb @@ -59,7 +59,7 @@ module Gitlab end def submit - MetricsWorker.perform_async(@metrics.map(&:to_hash)) + Metrics.submit_metrics(@metrics.map(&:to_hash)) end end end diff --git a/spec/lib/gitlab/metrics/sampler_spec.rb b/spec/lib/gitlab/metrics/sampler_spec.rb index 69376c0b79b..51a941c48cd 100644 --- a/spec/lib/gitlab/metrics/sampler_spec.rb +++ b/spec/lib/gitlab/metrics/sampler_spec.rb @@ -38,7 +38,7 @@ describe Gitlab::Metrics::Sampler do describe '#flush' do it 'schedules the metrics using Sidekiq' do - expect(MetricsWorker).to receive(:perform_async). + expect(Gitlab::Metrics).to receive(:submit_metrics). with([an_instance_of(Hash)]) sampler.sample_memory_usage diff --git a/spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb b/spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb index 05214efc565..5882e7d81c7 100644 --- a/spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb +++ b/spec/lib/gitlab/metrics/sidekiq_middleware_spec.rb @@ -11,14 +11,6 @@ describe Gitlab::Metrics::SidekiqMiddleware do middleware.call(worker, 'test', :test) { nil } end - - it 'does not track jobs of the MetricsWorker' do - worker = MetricsWorker.new - - expect(Gitlab::Metrics::Transaction).to_not receive(:new) - - middleware.call(worker, 'test', :test) { nil } - end end describe '#tag_worker' do diff --git a/spec/lib/gitlab/metrics/transaction_spec.rb b/spec/lib/gitlab/metrics/transaction_spec.rb index 5f17ff8ee75..6862fc9e2d1 100644 --- a/spec/lib/gitlab/metrics/transaction_spec.rb +++ b/spec/lib/gitlab/metrics/transaction_spec.rb @@ -68,7 +68,7 @@ describe Gitlab::Metrics::Transaction do it 'submits the metrics to Sidekiq' do transaction.track_self - expect(MetricsWorker).to receive(:perform_async). + expect(Gitlab::Metrics).to receive(:submit_metrics). with([an_instance_of(Hash)]) transaction.submit diff --git a/spec/lib/gitlab/metrics_spec.rb b/spec/lib/gitlab/metrics_spec.rb index 944642909aa..6c0682cac4d 100644 --- a/spec/lib/gitlab/metrics_spec.rb +++ b/spec/lib/gitlab/metrics_spec.rb @@ -33,4 +33,52 @@ describe Gitlab::Metrics do expect(file).to eq('spec/lib/gitlab/metrics_spec.rb') end end + + describe '#submit_metrics' do + it 'prepares and writes the metrics to InfluxDB' do + connection = double(:connection) + pool = double(:pool) + + expect(pool).to receive(:with).and_yield(connection) + expect(connection).to receive(:write_points).with(an_instance_of(Array)) + expect(Gitlab::Metrics).to receive(:pool).and_return(pool) + + described_class.submit_metrics([{ 'series' => 'kittens', 'tags' => {} }]) + end + end + + describe '#prepare_metrics' do + it 'returns a Hash with the keys as Symbols' do + metrics = described_class. + prepare_metrics([{ 'values' => {}, 'tags' => {} }]) + + expect(metrics).to eq([{ values: {}, tags: {} }]) + end + + it 'escapes tag values' do + metrics = described_class.prepare_metrics([ + { 'values' => {}, 'tags' => { 'foo' => 'bar=' } } + ]) + + expect(metrics).to eq([{ values: {}, tags: { 'foo' => 'bar\\=' } }]) + end + + it 'drops empty tags' do + metrics = described_class.prepare_metrics([ + { 'values' => {}, 'tags' => { 'cats' => '', 'dogs' => nil } } + ]) + + expect(metrics).to eq([{ values: {}, tags: {} }]) + end + end + + describe '#escape_value' do + it 'escapes an equals sign' do + expect(described_class.escape_value('foo=')).to eq('foo\\=') + end + + it 'casts values to Strings' do + expect(described_class.escape_value(10)).to eq('10') + end + end end diff --git a/spec/workers/metrics_worker_spec.rb b/spec/workers/metrics_worker_spec.rb deleted file mode 100644 index 18260ea0c24..00000000000 --- a/spec/workers/metrics_worker_spec.rb +++ /dev/null @@ -1,52 +0,0 @@ -require 'spec_helper' - -describe MetricsWorker do - let(:worker) { described_class.new } - - describe '#perform' do - it 'prepares and writes the metrics to InfluxDB' do - connection = double(:connection) - pool = double(:pool) - - expect(pool).to receive(:with).and_yield(connection) - expect(connection).to receive(:write_points).with(an_instance_of(Array)) - expect(Gitlab::Metrics).to receive(:pool).and_return(pool) - - worker.perform([{ 'series' => 'kittens', 'tags' => {} }]) - end - end - - describe '#prepare_metrics' do - it 'returns a Hash with the keys as Symbols' do - metrics = worker.prepare_metrics([{ 'values' => {}, 'tags' => {} }]) - - expect(metrics).to eq([{ values: {}, tags: {} }]) - end - - it 'escapes tag values' do - metrics = worker.prepare_metrics([ - { 'values' => {}, 'tags' => { 'foo' => 'bar=' } } - ]) - - expect(metrics).to eq([{ values: {}, tags: { 'foo' => 'bar\\=' } }]) - end - - it 'drops empty tags' do - metrics = worker.prepare_metrics([ - { 'values' => {}, 'tags' => { 'cats' => '', 'dogs' => nil } } - ]) - - expect(metrics).to eq([{ values: {}, tags: {} }]) - end - end - - describe '#escape_value' do - it 'escapes an equals sign' do - expect(worker.escape_value('foo=')).to eq('foo\\=') - end - - it 'casts values to Strings' do - expect(worker.escape_value(10)).to eq('10') - end - end -end -- cgit v1.2.1 From 701e5de9108faa65a118e8822560d39fd8ea9996 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Tue, 29 Dec 2015 15:49:12 +0100 Subject: Use Gitlab::CurrentSettings for InfluxDB This ensures we can still start up even when not connecting to a database. --- lib/gitlab/metrics.rb | 36 +++++++++++------------------------- 1 file changed, 11 insertions(+), 25 deletions(-) diff --git a/lib/gitlab/metrics.rb b/lib/gitlab/metrics.rb index 0869d007ca5..2d266ccfe9e 100644 --- a/lib/gitlab/metrics.rb +++ b/lib/gitlab/metrics.rb @@ -1,36 +1,21 @@ module Gitlab module Metrics + extend Gitlab::CurrentSettings + RAILS_ROOT = Rails.root.to_s METRICS_ROOT = Rails.root.join('lib', 'gitlab', 'metrics').to_s PATH_REGEX = /^#{RAILS_ROOT}\/?/ - # Returns the current settings, ensuring we _always_ have a default set of - # metrics settings (even during tests, when the migrations are lacking, - # etc). This ensures the application is able to boot up even when the - # migrations have not been executed. - def self.settings - if ApplicationSetting.table_exists? and curr = ApplicationSetting.current - curr - else - { - metrics_pool_size: 16, - metrics_timeout: 10, - metrics_enabled: false, - metrics_method_call_threshold: 10 - } - end - end - def self.pool_size - settings[:metrics_pool_size] + current_application_settings[:metrics_pool_size] || 16 end def self.timeout - settings[:metrics_timeout] + current_application_settings[:metrics_timeout] || 10 end def self.enabled? - settings[:metrics_enabled] + current_application_settings[:metrics_enabled] || false end def self.mri? @@ -41,7 +26,8 @@ module Gitlab # This is memoized since this method is called for every instrumented # method. Loading data from an external cache on every method call slows # things down too much. - @method_call_threshold ||= settings[:metrics_method_call_threshold] + @method_call_threshold ||= + (current_application_settings[:metrics_method_call_threshold] || 10) end def self.pool @@ -105,10 +91,10 @@ module Gitlab # "@foo ||= bar" is _not_ thread-safe. if enabled? @pool = ConnectionPool.new(size: pool_size, timeout: timeout) do - host = settings[:metrics_host] - user = settings[:metrics_username] - pw = settings[:metrics_password] - port = settings[:metrics_port] + host = current_application_settings[:metrics_host] + user = current_application_settings[:metrics_username] + pw = current_application_settings[:metrics_password] + port = current_application_settings[:metrics_port] InfluxDB::Client. new(udp: { host: host, port: port }, username: user, password: pw) -- cgit v1.2.1 From a80f2b792cd2bdd6626bfe712750a58b863cc037 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Tue, 29 Dec 2015 16:29:45 -0500 Subject: Update CHANGELOG [ci skip] --- CHANGELOG | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 57f0b9f30d5..5d0726a52ce 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,7 +1,6 @@ Please view this file on the master branch, on stable branches it's out of date. v 8.4.0 (unreleased) - - Add support for Google reCAPTCHA in user registration to prevent spammers (Stan Hu) - Implement new UI for group page - Implement search inside emoji picker - Add API support for looking up a user by username (Stan Hu) @@ -10,12 +9,13 @@ v 8.4.0 (unreleased) - Expose Git's version in the admin area - Add "Frequently used" category to emoji picker - Add CAS support (tduehr) - - Add link to merge request on build detail page. + - Add link to merge request on build detail page - Revert back upvote and downvote button to the issue and MR pages + - Enable "Add key" button when user fills in a proper key (Stan Hu) -v 8.3.2 (unreleased) +v 8.3.2 - Disable --follow in `git log` to avoid loading duplicate commit data in infinite scroll (Stan Hu) - - Enable "Add key" button when user fills in a proper key + - Add support for Google reCAPTCHA in user registration v 8.3.1 - Fix Error 500 when global milestones have slashes (Stan Hu) -- cgit v1.2.1 From 59533d47dd901a1b4a7e4eda382a0a25ce9a1eef Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Mon, 28 Dec 2015 18:10:46 -0800 Subject: Fix project transfer e-mail sending incorrect paths in e-mail notification The introduction of ActiveJob and `deliver_now` in 7f214cee7 caused a race condition where the mailer would be invoked before the project was committed to the database, causing the transfer e-mail notification to show the old path instead of the new one. Closes #4670 --- CHANGELOG | 3 +++ app/models/project.rb | 4 +++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index 5d0726a52ce..425beb17743 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -13,6 +13,9 @@ v 8.4.0 (unreleased) - Revert back upvote and downvote button to the issue and MR pages - Enable "Add key" button when user fills in a proper key (Stan Hu) +v 8.3.3 (unreleased) + - Fix project transfer e-mail sending incorrect paths in e-mail notification (Stan Hu) + v 8.3.2 - Disable --follow in `git log` to avoid loading duplicate commit data in infinite scroll (Stan Hu) - Add support for Google reCAPTCHA in user registration diff --git a/app/models/project.rb b/app/models/project.rb index 75f85310d5f..017471995ec 100644 --- a/app/models/project.rb +++ b/app/models/project.rb @@ -555,7 +555,9 @@ class Project < ActiveRecord::Base end def send_move_instructions(old_path_with_namespace) - NotificationService.new.project_was_moved(self, old_path_with_namespace) + # New project path needs to be committed to the DB or notification will + # retrieve stale information + run_after_commit { NotificationService.new.project_was_moved(self, old_path_with_namespace) } end def owner -- cgit v1.2.1