diff options
Diffstat (limited to 'chromium/net/sdch')
| -rw-r--r-- | chromium/net/sdch/README.md | 119 | ||||
| -rw-r--r-- | chromium/net/sdch/sdch_owner.cc | 784 | ||||
| -rw-r--r-- | chromium/net/sdch/sdch_owner.h | 224 | ||||
| -rw-r--r-- | chromium/net/sdch/sdch_owner_unittest.cc | 922 |
4 files changed, 2049 insertions, 0 deletions
diff --git a/chromium/net/sdch/README.md b/chromium/net/sdch/README.md new file mode 100644 index 00000000000..8200ee50528 --- /dev/null +++ b/chromium/net/sdch/README.md @@ -0,0 +1,119 @@ +# SDCH + +"SDCH" stands for "Shared Dictionary Compression over HTTP". It is a +protocol for compressing URL responses used when the server and +the client share a dictionary that can be referred to for +compression/encoding and decompression/decoding. The details of the +SDCH protocol are specified in +[the spec](https://docs.google.com/a/chromium.org/document/d/1REMkwjXY5yFOkJwtJPjCMwZ4Shx3D9vfdAytV_KQCUo/edit?pli=1) +(soon to be moved to github) but in brief: + +1. If the client supports SDCH decoding, it advertises "sdch" in the + "Accept-Encoding" header. +2. If the server could have encoded a response with a dictionary (but + didn't, because the client didn't have the dictionary), it includes + an advisory "Get-Dictionary: <url>" header in its response. +3. If the client has a dictionary that the server has previously + advertised as being usable for encoding a particular requests, it + advertises that dictionary as being available via an + "Avail-Dictionary: <hash>" header in the request. +4. If the server chooses to encode a response with a dictionary, it + includes "sdch" in a "Content-Encoding" header, in which case the + body will reference the dictionary to be used for decoding (which + must be one the client advertised in the original request). + Encodings may be chained; often responses are SDCH encoded, and then + gzip encoded. + +## SDCH in Chromium: Overview + +The SDCH implementation in Chromium is spread across several classes +in several different directories: + +* SdchManager (in net/base): This class contains all + dictionaries currently known to Chromium. Each URLRequestContext + points to an SdchManager; at the chrome/ level, there is one + SdchManager per profile. URLRequestHttpJob consults the SdchManager + for what dictionaries should be advertised with a URLRequest, and + notifies the SdchManager whenever it sees a "Get-Dictionary" + header. The SdchManager does *not* mediate fetching of + dictionaries; it is conceptually layered underneath URLRequest and + has no knowledge of URLRequests. There are several nested classes of + SdchManager (Dictionary, DictionarySet) used in the SDCH + implementation; see sdch_manager.h for details. +* SdchObserver (in net/base). This is an Abstract Base + Class which other classes may implement if those classes wish to + receive notifications about SDCH events. Such classes should also + register as observers with the SdchManager. +* SdchFilter (int net/filter). This class is derived from net::Filter + that is used for decoding the SDCH response; it cooperates with + SdchManager and the URLRequestJob to decode SDCH encoded responses. +* SdchDictionaryFetcher (int net/url_request): + This class implements the nuts&bolts of fetching an SDCH + dictionary. +* SdchOwner (in net/sdch): This class is an SdchObserver. + It contains policy for the SDCH implementation, including mediation + of fetching dictionaries, prioritization and eviction of + dictionaries in response to new fetches, and constraints on the + amount of memory that is usable by SDCH dictionaries. It initiates + dictionary fetches as appropriate when it receives notification of + a "Get-Dictionary" header from the SdchManager. + +A net/ embedder should instantiate an SdchManager and an SdchOwner, +and guarantee that the SdchManager outlive the SdchOwner. + +Note the layering of the above classes: + +1. The SdchManager and SdchOwner classes have no knowledge of + URLRequests. URLRequest is dependent on those classes, not the + reverse. +2. SdchDictionaryFetcher is dependent on URLRequest, but is still a + utility class exported by the net/ library for use by higher levels. +3. SdchOwner manages the entire system on behalf of the embedder. The + intent is that the embedder can change policies through methods on + SdchOwner, while letting the SdchOwner class take care of policy + implementation. + +## SDCH in Chromium: Debugging + +Data that is useful in debugging SDCH problems: + +* The SDCH UMA prefix is "Sdch3", and histograms that have been found + useful for debugging include + * ProblemCodes_* (though this requires trawling the source for each bucket). + * ResponseCorruptionDetection.{Cached,Uncached}: An attempt to make + sense of the twisted mess in SdchFilter::ReadFilteredData mentioned + above. + * BlacklistReason: Why requests avoid using SDCH when they could use + it. +* about:net-internals has an SDCH tab, showing loaded dictionaries and + other information. Searching in net-internals for "Get-Dictionary", + the URLRequest that actually fetches that dictionary, and then the + hash of that dictionary (often used as the file name) can also be + useful. + +## SDCH in Chromium: Gotchas and corner cases + +There are a couple of known issues in SDCH in Chromium that developers +in this space should be aware of: + +* As noted in the spec above, there have historically been problems + with middleboxes stripping or corrupting SDCH encoded responses. + For this reason, the protocol requires that if a server is not using + SDCH encoding when it has previously advertised the availability of + doing such, it includes an "X-SDCH-Encode: 0" header in the + response. Servers don't always do this (especially multi-servers), + and that can result in failed decodings and requests being dropped + on the floor. The code to handle this is a twisted mess (see + SdchFilter::ReadFilteredData()) and problems have often been seen + from or associated with it. +* If the decoding logic trips over a problem, it will often blacklist + the server in question, temporarily (if it can recover that request) + or permanently (if it can't). This can lead to a mysterious lack of + SDCH encoding when it's expected to be present. +* The network cache currently stores the response precisely as received from + the network. This means that requests that don't advertise SDCH + may get a cached value that is SDCH encoded, and requests that do + advertise SDCH may get a cached value that is not SDCH encoded. + The second case is handled transparently, but the first case may + lead to request failure. + diff --git a/chromium/net/sdch/sdch_owner.cc b/chromium/net/sdch/sdch_owner.cc new file mode 100644 index 00000000000..a3ef30ec853 --- /dev/null +++ b/chromium/net/sdch/sdch_owner.cc @@ -0,0 +1,784 @@ +// Copyright 2014 The Chromium Authors. All rights reserved. +// Use of this source code is governed by a BSD-style license that can be +// found in the LICENSE file. + +#include "net/sdch/sdch_owner.h" + +#include "base/bind.h" +#include "base/debug/alias.h" +#include "base/logging.h" +#include "base/metrics/histogram_macros.h" +#include "base/prefs/persistent_pref_store.h" +#include "base/prefs/value_map_pref_store.h" +#include "base/strings/string_util.h" +#include "base/time/default_clock.h" +#include "base/values.h" +#include "net/base/sdch_manager.h" +#include "net/base/sdch_net_log_params.h" + +namespace net { + +namespace { + +enum PersistenceFailureReason { + // File didn't exist; is being created. + PERSISTENCE_FAILURE_REASON_NO_FILE = 1, + + // Error reading in information, but should be able to write. + PERSISTENCE_FAILURE_REASON_READ_FAILED = 2, + + // Error leading to abort on attempted persistence. + PERSISTENCE_FAILURE_REASON_WRITE_FAILED = 3, + + PERSISTENCE_FAILURE_REASON_MAX = 4 +}; + +// Dictionaries that haven't been touched in 24 hours may be evicted +// to make room for new dictionaries. +const int kFreshnessLifetimeHours = 24; + +// Dictionaries that have never been used only stay fresh for one hour. +const int kNeverUsedFreshnessLifetimeHours = 1; + +void RecordPersistenceFailure(PersistenceFailureReason failure_reason) { + UMA_HISTOGRAM_ENUMERATION("Sdch3.PersistenceFailureReason", failure_reason, + PERSISTENCE_FAILURE_REASON_MAX); +} + +// Schema specifications and access routines. + +// The persistent prefs store is conceptually shared with any other network +// stack systems that want to persist data over browser restarts, and so +// use of it must be namespace restricted. +// Schema: +// pref_store_->GetValue(kPreferenceName) -> Dictionary { +// 'version' -> 1 [int] +// 'dictionaries' -> Dictionary { +// server_hash -> { +// 'url' -> URL [string] +// 'last_used' -> seconds since unix epoch [double] +// 'use_count' -> use count [int] +// 'size' -> size [int] +// } +// } +const char kPreferenceName[] = "SDCH"; +const char kVersionKey[] = "version"; +const char kDictionariesKey[] = "dictionaries"; +const char kDictionaryUrlKey[] = "url"; +const char kDictionaryLastUsedKey[] = "last_used"; +const char kDictionaryUseCountKey[] = "use_count"; +const char kDictionarySizeKey[] = "size"; + +const int kVersion = 1; + +// This function returns store[kPreferenceName/kDictionariesKey]. The caller +// is responsible for making sure any needed calls to +// |store->ReportValueChanged()| occur. +base::DictionaryValue* GetPersistentStoreDictionaryMap( + WriteablePrefStore* store) { + base::Value* result = nullptr; + bool success = store->GetMutableValue(kPreferenceName, &result); + DCHECK(success); + + base::DictionaryValue* preference_dictionary = nullptr; + success = result->GetAsDictionary(&preference_dictionary); + DCHECK(success); + DCHECK(preference_dictionary); + + base::DictionaryValue* dictionary_list_dictionary = nullptr; + success = preference_dictionary->GetDictionary(kDictionariesKey, + &dictionary_list_dictionary); + DCHECK(success); + DCHECK(dictionary_list_dictionary); + + return dictionary_list_dictionary; +} + +// This function initializes a pref store with an empty version of the +// above schema, removing anything previously in the store under +// kPreferenceName. +void InitializePrefStore(WriteablePrefStore* store) { + base::DictionaryValue* empty_store(new base::DictionaryValue); + empty_store->SetInteger(kVersionKey, kVersion); + empty_store->Set(kDictionariesKey, + make_scoped_ptr(new base::DictionaryValue)); + store->SetValue(kPreferenceName, empty_store, + WriteablePrefStore::DEFAULT_PREF_WRITE_FLAGS); +} + +// A class to allow iteration over all dictionaries in the pref store, and +// easy lookup of the information associated with those dictionaries. +// Note that this is an "Iterator" in the same sense (and for the same +// reasons) that base::Dictionary::Iterator is an iterator--it allows +// iterating over all the dictionaries in the preference store, but it +// does not allow use as an STL iterator because the container it +// is iterating over does not export begin()/end() methods. This iterator can +// only be safely used on sanitized pref stores that are known to conform to the +// pref store schema. +class DictionaryPreferenceIterator { + public: + explicit DictionaryPreferenceIterator(WriteablePrefStore* pref_store); + + bool IsAtEnd() const; + void Advance(); + + const std::string& server_hash() const { return server_hash_; } + const GURL url() const { return url_; } + base::Time last_used() const { return last_used_; } + int use_count() const { return use_count_; } + int size() const { return size_; } + + private: + void LoadDictionaryOrDie(); + + std::string server_hash_; + GURL url_; + base::Time last_used_; + int use_count_; + int size_; + + base::DictionaryValue::Iterator dictionary_iterator_; +}; + +DictionaryPreferenceIterator::DictionaryPreferenceIterator( + WriteablePrefStore* pref_store) + : dictionary_iterator_(*GetPersistentStoreDictionaryMap(pref_store)) { + if (!IsAtEnd()) + LoadDictionaryOrDie(); +} + +bool DictionaryPreferenceIterator::IsAtEnd() const { + return dictionary_iterator_.IsAtEnd(); +} + +void DictionaryPreferenceIterator::Advance() { + dictionary_iterator_.Advance(); + if (!IsAtEnd()) + LoadDictionaryOrDie(); +} + +void DictionaryPreferenceIterator::LoadDictionaryOrDie() { + double last_used_seconds_from_epoch; + const base::DictionaryValue* dict = nullptr; + bool success = + dictionary_iterator_.value().GetAsDictionary(&dict); + DCHECK(success); + + server_hash_ = dictionary_iterator_.key(); + + std::string url_spec; + success = dict->GetString(kDictionaryUrlKey, &url_spec); + DCHECK(success); + url_ = GURL(url_spec); + + success = dict->GetDouble(kDictionaryLastUsedKey, + &last_used_seconds_from_epoch); + DCHECK(success); + last_used_ = base::Time::FromDoubleT(last_used_seconds_from_epoch); + + success = dict->GetInteger(kDictionaryUseCountKey, &use_count_); + DCHECK(success); + + success = dict->GetInteger(kDictionarySizeKey, &size_); + DCHECK(success); +} + +// Triggers a ReportValueChanged() on the specified WriteablePrefStore +// when the object goes out of scope. +class ScopedPrefNotifier { + public: + // Caller must guarantee lifetime of |*pref_store| exceeds the + // lifetime of this object. + ScopedPrefNotifier(WriteablePrefStore* pref_store) + : pref_store_(pref_store) {} + ~ScopedPrefNotifier() { + pref_store_->ReportValueChanged( + kPreferenceName, WriteablePrefStore::DEFAULT_PREF_WRITE_FLAGS); + } + + private: + WriteablePrefStore* pref_store_; + + DISALLOW_COPY_AND_ASSIGN(ScopedPrefNotifier); +}; + +} // namespace + +// Adjust SDCH limits downwards for mobile. +#if defined(OS_ANDROID) || defined(OS_IOS) +// static +const size_t SdchOwner::kMaxTotalDictionarySize = 2 * 500 * 1000; +#else +// static +const size_t SdchOwner::kMaxTotalDictionarySize = 20 * 1000 * 1000; +#endif + +// Somewhat arbitrary, but we assume a dictionary smaller than +// 50K isn't going to do anyone any good. Note that this still doesn't +// prevent download and addition unless there is less than this +// amount of space available in storage. +const size_t SdchOwner::kMinSpaceForDictionaryFetch = 50 * 1000; + +void SdchOwner::RecordDictionaryFate(enum DictionaryFate fate) { + UMA_HISTOGRAM_ENUMERATION("Sdch3.DictionaryFate", fate, DICTIONARY_FATE_MAX); +} + +void SdchOwner::RecordDictionaryEvictionOrUnload(const std::string& server_hash, + size_t size, + int use_count, + DictionaryFate fate) { + DCHECK(fate == DICTIONARY_FATE_EVICT_FOR_DICT || + fate == DICTIONARY_FATE_EVICT_FOR_MEMORY || + fate == DICTIONARY_FATE_EVICT_FOR_DESTRUCTION || + fate == DICTIONARY_FATE_UNLOAD_FOR_DESTRUCTION); + + UMA_HISTOGRAM_COUNTS_100("Sdch3.DictionaryUseCount", use_count); + RecordDictionaryFate(fate); + + DCHECK(load_times_.count(server_hash) == 1); + base::Time now = clock_->Now(); + base::TimeDelta dict_lifetime = now - load_times_[server_hash]; + consumed_byte_seconds_.push_back(size * dict_lifetime.InMilliseconds()); + load_times_.erase(server_hash); +} + +SdchOwner::SdchOwner(SdchManager* sdch_manager, URLRequestContext* context) + : manager_(sdch_manager->GetWeakPtr()), + fetcher_(new SdchDictionaryFetcher(context)), + total_dictionary_bytes_(0), + clock_(new base::DefaultClock), + max_total_dictionary_size_(kMaxTotalDictionarySize), + min_space_for_dictionary_fetch_(kMinSpaceForDictionaryFetch), +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + destroyed_(0), +#endif + memory_pressure_listener_( + base::Bind(&SdchOwner::OnMemoryPressure, + // Because |memory_pressure_listener_| is owned by + // SdchOwner, the SdchOwner object will be available + // for the lifetime of |memory_pressure_listener_|. + base::Unretained(this))), + in_memory_pref_store_(new ValueMapPrefStore()), + external_pref_store_(nullptr), + pref_store_(in_memory_pref_store_.get()), + creation_time_(clock_->Now()) { +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + CHECK(clock_.get()); +#endif + manager_->AddObserver(this); + InitializePrefStore(pref_store_); +} + +SdchOwner::~SdchOwner() { +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + CHECK_EQ(0u, destroyed_); + CHECK(clock_.get()); + CHECK(manager_.get()); +#endif + + for (DictionaryPreferenceIterator it(pref_store_); !it.IsAtEnd(); + it.Advance()) { + int new_uses = it.use_count() - use_counts_at_load_[it.server_hash()]; + DictionaryFate fate = IsPersistingDictionaries() ? + DICTIONARY_FATE_UNLOAD_FOR_DESTRUCTION : + DICTIONARY_FATE_EVICT_FOR_DESTRUCTION; + RecordDictionaryEvictionOrUnload(it.server_hash(), it.size(), new_uses, + fate); + } + manager_->RemoveObserver(this); + + // This object only observes the external store during loading, + // i.e. before it's made the default preferences store. + if (external_pref_store_) + external_pref_store_->RemoveObserver(this); + + int64 object_lifetime = + (clock_->Now() - creation_time_).InMilliseconds(); + for (const auto& val : consumed_byte_seconds_) { + if (object_lifetime > 0) { + // Objects that are created and immediately destroyed don't add any memory + // pressure over time (and also cause a crash here). + UMA_HISTOGRAM_MEMORY_KB("Sdch3.TimeWeightedMemoryUse", + val / object_lifetime); + } + } + +#if defined(OS_CHROMEOS) + destroyed_ = 0xdeadbeef; +#endif +} + +void SdchOwner::EnablePersistentStorage(PersistentPrefStore* pref_store) { + DCHECK(!external_pref_store_); + external_pref_store_ = pref_store; + external_pref_store_->AddObserver(this); + + if (external_pref_store_->IsInitializationComplete()) + OnInitializationCompleted(true); +} + +void SdchOwner::SetMaxTotalDictionarySize(size_t max_total_dictionary_size) { + max_total_dictionary_size_ = max_total_dictionary_size; +} + +void SdchOwner::SetMinSpaceForDictionaryFetch( + size_t min_space_for_dictionary_fetch) { + min_space_for_dictionary_fetch_ = min_space_for_dictionary_fetch; +} + +void SdchOwner::OnDictionaryFetched(base::Time last_used, + int use_count, + const std::string& dictionary_text, + const GURL& dictionary_url, + const BoundNetLog& net_log, + bool was_from_cache) { + struct DictionaryItem { + base::Time last_used; + std::string server_hash; + int use_count; + size_t dictionary_size; + + DictionaryItem() : use_count(0), dictionary_size(0) {} + DictionaryItem(const base::Time& last_used, + const std::string& server_hash, + int use_count, + size_t dictionary_size) + : last_used(last_used), + server_hash(server_hash), + use_count(use_count), + dictionary_size(dictionary_size) {} + DictionaryItem(const DictionaryItem& rhs) = default; + DictionaryItem& operator=(const DictionaryItem& rhs) = default; + bool operator<(const DictionaryItem& rhs) const { + return last_used < rhs.last_used; + } + }; + +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + CHECK_EQ(0u, destroyed_); + CHECK(clock_.get()); +#endif + + if (!was_from_cache) + UMA_HISTOGRAM_COUNTS("Sdch3.NetworkBytesSpent", dictionary_text.size()); + + // Figure out if there is space for the incoming dictionary; evict + // stale dictionaries if needed to make space. + + std::vector<DictionaryItem> stale_dictionary_list; + size_t recoverable_bytes = 0; + base::Time now(clock_->Now()); + // Dictionaries whose last used time is before |stale_boundary| are candidates + // for eviction if necessary. + base::Time stale_boundary( + now - base::TimeDelta::FromHours(kFreshnessLifetimeHours)); + // Dictionaries that have never been used and are from before + // |never_used_stale_boundary| are candidates for eviction if necessary. + base::Time never_used_stale_boundary( + now - base::TimeDelta::FromHours(kNeverUsedFreshnessLifetimeHours)); + for (DictionaryPreferenceIterator it(pref_store_); !it.IsAtEnd(); + it.Advance()) { + if (it.last_used() < stale_boundary || + (it.use_count() == 0 && it.last_used() < never_used_stale_boundary)) { + stale_dictionary_list.push_back(DictionaryItem( + it.last_used(), it.server_hash(), it.use_count(), it.size())); + recoverable_bytes += it.size(); + } + } + +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + CHECK_EQ(0u, destroyed_); + CHECK(clock_.get()); +#endif + + if (total_dictionary_bytes_ + dictionary_text.size() - recoverable_bytes > + max_total_dictionary_size_) { + RecordDictionaryFate(DICTIONARY_FATE_FETCH_IGNORED_NO_SPACE); + SdchManager::SdchErrorRecovery(SDCH_DICTIONARY_NO_ROOM); + net_log.AddEvent(NetLog::TYPE_SDCH_DICTIONARY_ERROR, + base::Bind(&NetLogSdchDictionaryFetchProblemCallback, + SDCH_DICTIONARY_NO_ROOM, dictionary_url, true)); + return; + } + + // Add the new dictionary. This is done before removing the stale + // dictionaries so that no state change will occur if dictionary addition + // fails. + std::string server_hash; + SdchProblemCode rv = manager_->AddSdchDictionary( + dictionary_text, dictionary_url, &server_hash); + if (rv != SDCH_OK) { + RecordDictionaryFate(DICTIONARY_FATE_FETCH_MANAGER_REFUSED); + SdchManager::SdchErrorRecovery(rv); + net_log.AddEvent(NetLog::TYPE_SDCH_DICTIONARY_ERROR, + base::Bind(&NetLogSdchDictionaryFetchProblemCallback, rv, + dictionary_url, true)); + return; + } + + base::DictionaryValue* pref_dictionary_map = + GetPersistentStoreDictionaryMap(pref_store_); + ScopedPrefNotifier scoped_pref_notifier(pref_store_); + + // Remove the old dictionaries. + std::sort(stale_dictionary_list.begin(), stale_dictionary_list.end()); + size_t avail_bytes = max_total_dictionary_size_ - total_dictionary_bytes_; + auto stale_it = stale_dictionary_list.begin(); + while (avail_bytes < dictionary_text.size() && + stale_it != stale_dictionary_list.end()) { + manager_->RemoveSdchDictionary(stale_it->server_hash); + + DCHECK(pref_dictionary_map->HasKey(stale_it->server_hash)); + bool success = pref_dictionary_map->RemoveWithoutPathExpansion( + stale_it->server_hash, nullptr); + DCHECK(success); + + avail_bytes += stale_it->dictionary_size; + + int new_uses = stale_it->use_count - + use_counts_at_load_[stale_it->server_hash]; + RecordDictionaryEvictionOrUnload(stale_it->server_hash, + stale_it->dictionary_size, + new_uses, + DICTIONARY_FATE_EVICT_FOR_DICT); + + ++stale_it; + } + DCHECK_GE(avail_bytes, dictionary_text.size()); + + RecordDictionaryFate( + // Distinguish between loads triggered by network responses and + // loads triggered by persistence. + last_used.is_null() ? DICTIONARY_FATE_ADD_RESPONSE_TRIGGERED + : DICTIONARY_FATE_ADD_PERSISTENCE_TRIGGERED); + + // If a dictionary has never been used, its dictionary addition time + // is recorded as its last used time. Never used dictionaries are treated + // specially in the freshness logic. + if (last_used.is_null()) + last_used = clock_->Now(); + + total_dictionary_bytes_ += dictionary_text.size(); + +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + CHECK_EQ(0u, destroyed_); + CHECK(clock_.get()); +#endif + + // Record the addition in the pref store. + scoped_ptr<base::DictionaryValue> dictionary_description( + new base::DictionaryValue()); + dictionary_description->SetString(kDictionaryUrlKey, dictionary_url.spec()); + dictionary_description->SetDouble(kDictionaryLastUsedKey, + last_used.ToDoubleT()); + dictionary_description->SetInteger(kDictionaryUseCountKey, use_count); + dictionary_description->SetInteger(kDictionarySizeKey, + dictionary_text.size()); + pref_dictionary_map->Set(server_hash, dictionary_description.Pass()); + load_times_[server_hash] = clock_->Now(); +} + +void SdchOwner::OnDictionaryAdded(const GURL& dictionary_url, + const std::string& server_hash) { } + +void SdchOwner::OnDictionaryRemoved(const std::string& server_hash) { } + +void SdchOwner::OnDictionaryUsed(const std::string& server_hash) { + base::Time now(clock_->Now()); + base::DictionaryValue* pref_dictionary_map = + GetPersistentStoreDictionaryMap(pref_store_); + ScopedPrefNotifier scoped_pref_notifier(pref_store_); + + base::Value* value = nullptr; + bool success = pref_dictionary_map->Get(server_hash, &value); + if (!success) { + // SdchManager::GetDictionarySet() pins the referenced dictionaries in + // memory past a possible deletion. For this reason, OnDictionaryUsed() + // notifications may occur after SdchOwner thinks that dictionaries + // have been deleted. + SdchManager::SdchErrorRecovery(SDCH_DICTIONARY_USED_AFTER_DELETION); + return; + } + base::DictionaryValue* specific_dictionary_map = nullptr; + success = value->GetAsDictionary(&specific_dictionary_map); + // TODO(rdsmith); Switch back to DCHECK() after http://crbug.com/454198 is + // resolved. + CHECK(success); + + double last_used_seconds_since_epoch = 0.0; + success = specific_dictionary_map->GetDouble(kDictionaryLastUsedKey, + &last_used_seconds_since_epoch); + // TODO(rdsmith); Switch back to DCHECK() after http://crbug.com/454198 is + // resolved. + CHECK(success); + int use_count = 0; + success = + specific_dictionary_map->GetInteger(kDictionaryUseCountKey, &use_count); + // TODO(rdsmith); Switch back to DCHECK() after http://crbug.com/454198 is + // resolved. + CHECK(success); + + if (use_counts_at_load_.count(server_hash) == 0) { + use_counts_at_load_[server_hash] = use_count; + } + + base::TimeDelta time_since_last_used(now - + base::Time::FromDoubleT(last_used_seconds_since_epoch)); + + // TODO(rdsmith): Distinguish between "Never used" and "Actually not + // touched for 48 hours". + UMA_HISTOGRAM_CUSTOM_TIMES( + "Sdch3.UsageInterval", + use_count ? time_since_last_used : base::TimeDelta::FromHours(48), + base::TimeDelta(), base::TimeDelta::FromHours(48), 50); + + specific_dictionary_map->SetDouble(kDictionaryLastUsedKey, now.ToDoubleT()); + specific_dictionary_map->SetInteger(kDictionaryUseCountKey, use_count + 1); +} + +void SdchOwner::OnGetDictionary(const GURL& request_url, + const GURL& dictionary_url) { +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + char url_buf[128]; + if (0u != destroyed_ || !clock_.get()) { + base::strlcpy(url_buf, request_url.spec().c_str(), arraysize(url_buf)); + } + base::debug::Alias(url_buf); + + CHECK_EQ(0u, destroyed_); + CHECK(clock_.get()); +#endif + + base::Time stale_boundary(clock_->Now() - base::TimeDelta::FromDays(1)); + size_t avail_bytes = 0; + for (DictionaryPreferenceIterator it(pref_store_); !it.IsAtEnd(); + it.Advance()) { + if (it.last_used() < stale_boundary) + avail_bytes += it.size(); + } + + // Don't initiate the fetch if we wouldn't be able to store any + // reasonable dictionary. + // TODO(rdsmith): Maybe do a HEAD request to figure out how much + // storage we'd actually need? + if (max_total_dictionary_size_ < (total_dictionary_bytes_ - avail_bytes + + min_space_for_dictionary_fetch_)) { + RecordDictionaryFate(DICTIONARY_FATE_GET_IGNORED); + // TODO(rdsmith): Log a net-internals error. This requires + // SdchManager to forward the URLRequest that detected the + // Get-Dictionary header to its observers, which is tricky + // because SdchManager is layered underneath URLRequest. + return; + } + + fetcher_->Schedule(dictionary_url, + base::Bind(&SdchOwner::OnDictionaryFetched, + // SdchOwner will outlive its member variables. + base::Unretained(this), base::Time(), 0)); +} + +void SdchOwner::OnClearDictionaries() { + total_dictionary_bytes_ = 0; + fetcher_->Cancel(); + + InitializePrefStore(pref_store_); +} + +void SdchOwner::OnPrefValueChanged(const std::string& key) { +} + +void SdchOwner::OnInitializationCompleted(bool succeeded) { + PersistentPrefStore::PrefReadError error = + external_pref_store_->GetReadError(); + // Errors on load are self-correcting; if dictionaries were not + // persisted from the last instance of the browser, they will be + // faulted in by user action over time. However, if a load error + // means that the dictionary information won't be able to be persisted, + // the in memory pref store is left in place. + if (!succeeded) { + // Failure means a write failed, since read failures are recoverable. + DCHECK_NE( + error, + PersistentPrefStore::PREF_READ_ERROR_ASYNCHRONOUS_TASK_INCOMPLETE); + DCHECK_NE(error, + PersistentPrefStore::PREF_READ_ERROR_MAX_ENUM); + + LOG(ERROR) << "Pref store write failed: " << error; + external_pref_store_->RemoveObserver(this); + external_pref_store_ = nullptr; + RecordPersistenceFailure(PERSISTENCE_FAILURE_REASON_WRITE_FAILED); + return; + } + switch (external_pref_store_->GetReadError()) { + case PersistentPrefStore::PREF_READ_ERROR_NONE: + break; + + case PersistentPrefStore::PREF_READ_ERROR_NO_FILE: + // First time reading; the file will be created. + RecordPersistenceFailure(PERSISTENCE_FAILURE_REASON_NO_FILE); + break; + + case PersistentPrefStore::PREF_READ_ERROR_JSON_PARSE: + case PersistentPrefStore::PREF_READ_ERROR_JSON_TYPE: + case PersistentPrefStore::PREF_READ_ERROR_FILE_OTHER: + case PersistentPrefStore::PREF_READ_ERROR_FILE_LOCKED: + case PersistentPrefStore::PREF_READ_ERROR_JSON_REPEAT: + case PersistentPrefStore::PREF_READ_ERROR_LEVELDB_IO: + case PersistentPrefStore::PREF_READ_ERROR_LEVELDB_CORRUPTION_READ_ONLY: + case PersistentPrefStore::PREF_READ_ERROR_LEVELDB_CORRUPTION: + RecordPersistenceFailure(PERSISTENCE_FAILURE_REASON_READ_FAILED); + break; + + case PersistentPrefStore::PREF_READ_ERROR_ACCESS_DENIED: + case PersistentPrefStore::PREF_READ_ERROR_FILE_NOT_SPECIFIED: + case PersistentPrefStore::PREF_READ_ERROR_ASYNCHRONOUS_TASK_INCOMPLETE: + case PersistentPrefStore::PREF_READ_ERROR_MAX_ENUM: + // Shouldn't ever happen. ACCESS_DENIED and FILE_NOT_SPECIFIED should + // imply !succeeded, and TASK_INCOMPLETE should never be delivered. + NOTREACHED(); + break; + } + + + // Load in what was stored before chrome exited previously. + const base::Value* sdch_persistence_value = nullptr; + const base::DictionaryValue* sdch_persistence_dictionary = nullptr; + + // The GetPersistentStore() routine above assumes data formatted + // according to the schema described at the top of this file. Since + // this data comes from disk, to avoid disk corruption resulting in + // persistent chrome errors this code avoids those assupmtions. + if (external_pref_store_->GetValue(kPreferenceName, + &sdch_persistence_value) && + sdch_persistence_value->GetAsDictionary(&sdch_persistence_dictionary)) { + SchedulePersistedDictionaryLoads(*sdch_persistence_dictionary); + } + + // Reset the persistent store and update it with the accumulated + // information from the local store. + InitializePrefStore(external_pref_store_); + + ScopedPrefNotifier scoped_pref_notifier(external_pref_store_); + GetPersistentStoreDictionaryMap(external_pref_store_) + ->Swap(GetPersistentStoreDictionaryMap(in_memory_pref_store_.get())); + + // This object can stop waiting on (i.e. observing) the external preference + // store and switch over to using it as the primary preference store. + pref_store_ = external_pref_store_; + external_pref_store_->RemoveObserver(this); + external_pref_store_ = nullptr; + in_memory_pref_store_ = nullptr; +} + +void SdchOwner::SetClockForTesting(scoped_ptr<base::Clock> clock) { + clock_ = clock.Pass(); + +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + CHECK_EQ(0u, destroyed_); + CHECK(clock_.get()); +#endif +} + +int SdchOwner::GetDictionaryCountForTesting() const { + int count = 0; + for (DictionaryPreferenceIterator it(pref_store_); !it.IsAtEnd(); + it.Advance()) { + count++; + } + return count; +} + +bool SdchOwner::HasDictionaryFromURLForTesting(const GURL& url) const { + for (DictionaryPreferenceIterator it(pref_store_); !it.IsAtEnd(); + it.Advance()) { + if (it.url() == url) + return true; + } + return false; +} + +void SdchOwner::SetFetcherForTesting( + scoped_ptr<SdchDictionaryFetcher> fetcher) { + fetcher_.reset(fetcher.release()); +} + +void SdchOwner::OnMemoryPressure( + base::MemoryPressureListener::MemoryPressureLevel level) { + DCHECK_NE(base::MemoryPressureListener::MEMORY_PRESSURE_LEVEL_NONE, level); + + for (DictionaryPreferenceIterator it(pref_store_); !it.IsAtEnd(); + it.Advance()) { + int new_uses = it.use_count() - use_counts_at_load_[it.server_hash()]; + RecordDictionaryEvictionOrUnload(it.server_hash(), + it.size(), + new_uses, + DICTIONARY_FATE_EVICT_FOR_MEMORY); + } + + // TODO(rdsmith): Make a distinction between moderate and critical + // memory pressure. + manager_->ClearData(); +} + +bool SdchOwner::SchedulePersistedDictionaryLoads( + const base::DictionaryValue& persisted_info) { + // Any schema error will result in dropping the persisted info. + int version = 0; + if (!persisted_info.GetInteger(kVersionKey, &version)) + return false; + + // Any version mismatch will result in dropping the persisted info; + // it will be faulted in at small performance cost as URLs using + // dictionaries for encoding are visited. + if (version != kVersion) + return false; + + const base::DictionaryValue* dictionary_set = nullptr; + if (!persisted_info.GetDictionary(kDictionariesKey, &dictionary_set)) + return false; + + // Any formatting error will result in skipping that particular + // dictionary. + for (base::DictionaryValue::Iterator dict_it(*dictionary_set); + !dict_it.IsAtEnd(); dict_it.Advance()) { + const base::DictionaryValue* dict_info = nullptr; + if (!dict_it.value().GetAsDictionary(&dict_info)) + continue; + + std::string url_string; + if (!dict_info->GetString(kDictionaryUrlKey, &url_string)) + continue; + GURL dict_url(url_string); + + double last_used; + if (!dict_info->GetDouble(kDictionaryLastUsedKey, &last_used)) + continue; + + int use_count; + if (!dict_info->GetInteger(kDictionaryUseCountKey, &use_count)) + continue; + + fetcher_->ScheduleReload( + dict_url, base::Bind(&SdchOwner::OnDictionaryFetched, + // SdchOwner will outlive its member variables. + base::Unretained(this), + base::Time::FromDoubleT(last_used), + use_count)); + } + + return true; +} + +bool SdchOwner::IsPersistingDictionaries() const { + return in_memory_pref_store_.get() != nullptr; +} + +} // namespace net diff --git a/chromium/net/sdch/sdch_owner.h b/chromium/net/sdch/sdch_owner.h new file mode 100644 index 00000000000..ec5d82d1224 --- /dev/null +++ b/chromium/net/sdch/sdch_owner.h @@ -0,0 +1,224 @@ +// Copyright 2014 The Chromium Authors. All rights reserved. +// Use of this source code is governed by a BSD-style license that can be +// found in the LICENSE file. + +#ifndef NET_SDCH_SDCH_OWNER_H_ +#define NET_SDCH_SDCH_OWNER_H_ + +#include <map> +#include <string> + +#include "base/memory/memory_pressure_listener.h" +#include "base/memory/ref_counted.h" +#include "base/memory/weak_ptr.h" +#include "base/prefs/pref_store.h" +#include "net/base/sdch_observer.h" +#include "net/url_request/sdch_dictionary_fetcher.h" + +class GURL; +class PersistentPrefStore; +class ValueMapPrefStore; +class WriteablePrefStore; + +namespace base { +class Clock; +} + +namespace net { +class SdchManager; +class URLRequestContext; + +// This class owns the SDCH objects not owned as part of URLRequestContext, and +// exposes interface for setting SDCH policy. It should be instantiated by +// the net/ embedder. +// TODO(rdsmith): Implement dictionary prioritization. +class NET_EXPORT SdchOwner : public SdchObserver, public PrefStore::Observer { + public: + static const size_t kMaxTotalDictionarySize; + static const size_t kMinSpaceForDictionaryFetch; + + // Consumer must guarantee that |sdch_manager| and |context| outlive + // this object. + SdchOwner(SdchManager* sdch_manager, URLRequestContext* context); + ~SdchOwner() override; + + // Enables use of pref persistence. Note that |pref_store| is owned + // by the caller, but must be guaranteed to outlive SdchOwner. The + // actual mechanisms by which the PersistentPrefStore are persisted + // are the responsibility of the caller. This routine may only be + // called once per SdchOwner instance. + void EnablePersistentStorage(PersistentPrefStore* pref_store); + + // Defaults to kMaxTotalDictionarySize. + void SetMaxTotalDictionarySize(size_t max_total_dictionary_size); + + // Defaults to kMinSpaceForDictionaryFetch. + void SetMinSpaceForDictionaryFetch(size_t min_space_for_dictionary_fetch); + + // SdchObserver implementation. + void OnDictionaryAdded(const GURL& dictionary_url, + const std::string& server_hash) override; + void OnDictionaryRemoved(const std::string& server_hash) override; + void OnDictionaryUsed(const std::string& server_hash) override; + void OnGetDictionary(const GURL& request_url, + const GURL& dictionary_url) override; + void OnClearDictionaries() override; + + // PrefStore::Observer implementation. + void OnPrefValueChanged(const std::string& key) override; + void OnInitializationCompleted(bool succeeded) override; + + // Implementation detail--this is the function callback by the callback passed + // to the fetcher through which the fetcher informs the SdchOwner that it's + // gotten the dictionary. The first two arguments are bound locally. + // Public for testing. + void OnDictionaryFetched(base::Time last_used, + int use_count, + const std::string& dictionary_text, + const GURL& dictionary_url, + const BoundNetLog& net_log, + bool was_from_cache); + + void SetClockForTesting(scoped_ptr<base::Clock> clock); + + // Returns the total number of dictionaries loaded. + int GetDictionaryCountForTesting() const; + + // Returns whether this SdchOwner has dictionary from |url| loaded. + bool HasDictionaryFromURLForTesting(const GURL& url) const; + + void SetFetcherForTesting(scoped_ptr<SdchDictionaryFetcher> fetcher); + + private: + // For each active dictionary, stores local info. + // Indexed by the server hash of the dictionary. + struct DictionaryInfo { + base::Time last_used; + int use_count; + size_t size; + + DictionaryInfo() : use_count(0), size(0) {} + DictionaryInfo(const base::Time& last_used, size_t size) + : last_used(last_used), use_count(0), size(size) {} + DictionaryInfo(const DictionaryInfo& rhs) = default; + DictionaryInfo& operator=(const DictionaryInfo& rhs) = default; + }; + + void OnMemoryPressure( + base::MemoryPressureListener::MemoryPressureLevel level); + + // Schedule loading of all dictionaries described in |persisted_info|. + // Returns false and does not schedule a load if |persisted_info| has an + // unsupported version or no dictionaries key. Skips any dictionaries that are + // malformed in |persisted_info|. + bool SchedulePersistedDictionaryLoads( + const base::DictionaryValue& persisted_info); + + bool IsPersistingDictionaries() const; + + enum DictionaryFate { + // A Get-Dictionary header wasn't acted on. + DICTIONARY_FATE_GET_IGNORED = 1, + + // A fetch was attempted, but failed. + // TODO(rdsmith): Actually record this case. + DICTIONARY_FATE_FETCH_FAILED = 2, + + // A successful fetch was dropped on the floor, no space. + DICTIONARY_FATE_FETCH_IGNORED_NO_SPACE = 3, + + // A successful fetch was refused by the SdchManager. + DICTIONARY_FATE_FETCH_MANAGER_REFUSED = 4, + + // A dictionary was successfully added based on + // a Get-Dictionary header in a response. + DICTIONARY_FATE_ADD_RESPONSE_TRIGGERED = 5, + + // A dictionary was evicted by an incoming dict. + DICTIONARY_FATE_EVICT_FOR_DICT = 6, + + // A dictionary was evicted by memory pressure. + DICTIONARY_FATE_EVICT_FOR_MEMORY = 7, + + // A dictionary was evicted on destruction. + DICTIONARY_FATE_EVICT_FOR_DESTRUCTION = 8, + + // A dictionary was successfully added based on + // persistence from a previous browser revision. + DICTIONARY_FATE_ADD_PERSISTENCE_TRIGGERED = 9, + + // A dictionary was unloaded on destruction, but is still present on disk. + DICTIONARY_FATE_UNLOAD_FOR_DESTRUCTION = 10, + + DICTIONARY_FATE_MAX = 11 + }; + + void RecordDictionaryFate(DictionaryFate fate); + + // Record the lifetime memory use of a specified dictionary, identified by + // server hash. + void RecordDictionaryEvictionOrUnload( + const std::string& server_hash, + size_t size, + int use_count, DictionaryFate fate); + + // For investigation of http://crbug.com/454198; remove when resolved. + base::WeakPtr<SdchManager> manager_; + scoped_ptr<SdchDictionaryFetcher> fetcher_; + + size_t total_dictionary_bytes_; + + scoped_ptr<base::Clock> clock_; + + size_t max_total_dictionary_size_; + size_t min_space_for_dictionary_fetch_; + +#if defined(OS_CHROMEOS) + // For debugging http://crbug.com/454198; remove when resolved. + unsigned int destroyed_; +#endif + + base::MemoryPressureListener memory_pressure_listener_; + + // Dictionary persistence machinery. + // * |in_memory_pref_store_| is created on construction and used in + // the absence of any call to EnablePersistentStorage(). + // * |external_pref_store_| holds the preference store specified + // by EnablePersistentStorage() (if any), while it is being read in. + // A non-null value here signals that the SdchOwner is observing + // the pref store; when read-in completes and observation is no longer + // needed, the pointer is set to null. This is to avoid lots of + // extra irrelevant function calls; the only observer interface this + // class is interested in is OnInitializationCompleted(). + // * |pref_store_| holds an unowned pointer to the currently + // active pref store (one of the preceding two). + scoped_refptr<ValueMapPrefStore> in_memory_pref_store_; + PersistentPrefStore* external_pref_store_; + + WriteablePrefStore* pref_store_; + + // The use counts of dictionaries when they were loaded from the persistent + // store, keyed by server hash. These are stored to avoid generating + // misleading ever-increasing use counts for dictionaries that are persisted, + // since the UMA histogram for use counts is only supposed to be since last + // load. + std::map<std::string, int> use_counts_at_load_; + + // Load times for loaded dictionaries, keyed by server hash. These are used to + // track the durations that dictionaries are in memory. + std::map<std::string, base::Time> load_times_; + + // Byte-seconds consumed by dictionaries that have been unloaded. These are + // stored for later uploading in the SdchOwner destructor. + std::vector<int64> consumed_byte_seconds_; + + // Creation time for this SdchOwner object, used for reporting temporal memory + // pressure. + base::Time creation_time_; + + DISALLOW_COPY_AND_ASSIGN(SdchOwner); +}; + +} // namespace net + +#endif // NET_SDCH_SDCH_OWNER_H_ diff --git a/chromium/net/sdch/sdch_owner_unittest.cc b/chromium/net/sdch/sdch_owner_unittest.cc new file mode 100644 index 00000000000..788a9929b6f --- /dev/null +++ b/chromium/net/sdch/sdch_owner_unittest.cc @@ -0,0 +1,922 @@ +// Copyright 2014 The Chromium Authors. All rights reserved. +// Use of this source code is governed by a BSD-style license that can be +// found in the LICENSE file. + +#include "base/memory/memory_pressure_listener.h" +#include "base/prefs/testing_pref_store.h" +#include "base/run_loop.h" +#include "base/strings/stringprintf.h" +#include "base/test/histogram_tester.h" +#include "base/test/simple_test_clock.h" +#include "base/values.h" +#include "net/base/sdch_manager.h" +#include "net/log/net_log.h" +#include "net/sdch/sdch_owner.h" +#include "net/url_request/url_request.h" +#include "net/url_request/url_request_context.h" +#include "net/url_request/url_request_error_job.h" +#include "net/url_request/url_request_job.h" +#include "net/url_request/url_request_job_factory.h" +#include "net/url_request/url_request_test_util.h" +#include "testing/gtest/include/gtest/gtest.h" + +namespace { + +bool GetDictionaryForURL(TestingPrefStore* store, + const GURL& url, + std::string* hash, + base::DictionaryValue** dict) { + base::Value* sdch_val = nullptr; + base::DictionaryValue* sdch_dict = nullptr; + if (!store->GetMutableValue("SDCH", &sdch_val)) + return false; + if (!sdch_val->GetAsDictionary(&sdch_dict)) + return false; + + base::DictionaryValue* dicts_dict = nullptr; + if (!sdch_dict->GetDictionary("dictionaries", &dicts_dict)) + return false; + + base::DictionaryValue::Iterator it(*dicts_dict); + while (!it.IsAtEnd()) { + const base::DictionaryValue* d = nullptr; + if (!it.value().GetAsDictionary(&d)) + continue; + std::string dict_url; + if (d->GetString("url", &dict_url) && dict_url == url.spec()) { + if (hash) + *hash = it.key(); + if (dict) + dicts_dict->GetDictionary(it.key(), dict); + return true; + } + it.Advance(); + } + + return false; +} + +} // namespace + +namespace net { + +static const char generic_url[] = "http://www.example.com"; +static const char generic_domain[] = "www.example.com"; + +static std::string NewSdchDictionary(size_t dictionary_size) { + std::string dictionary; + dictionary.append("Domain: "); + dictionary.append(generic_domain); + dictionary.append("\n"); + dictionary.append("\n"); + + size_t original_dictionary_size = dictionary.size(); + dictionary.resize(dictionary_size); + for (size_t i = original_dictionary_size; i < dictionary_size; ++i) + dictionary[i] = static_cast<char>((i % 127) + 1); + + return dictionary; +} + +int outstanding_url_request_error_counting_jobs = 0; +base::Closure* empty_url_request_jobs_callback = 0; + +// Variation of URLRequestErrorJob to count number of outstanding +// instances and notify when that goes to zero. +class URLRequestErrorCountingJob : public URLRequestJob { + public: + URLRequestErrorCountingJob(URLRequest* request, + NetworkDelegate* network_delegate, + int error) + : URLRequestJob(request, network_delegate), + error_(error), + weak_factory_(this) { + ++outstanding_url_request_error_counting_jobs; + } + + void Start() override { + base::MessageLoop::current()->PostTask( + FROM_HERE, base::Bind(&URLRequestErrorCountingJob::StartAsync, + weak_factory_.GetWeakPtr())); + } + + private: + ~URLRequestErrorCountingJob() override { + --outstanding_url_request_error_counting_jobs; + if (0 == outstanding_url_request_error_counting_jobs && + empty_url_request_jobs_callback) { + empty_url_request_jobs_callback->Run(); + } + } + + void StartAsync() { + NotifyStartError(URLRequestStatus(URLRequestStatus::FAILED, error_)); + } + + int error_; + + base::WeakPtrFactory<URLRequestErrorCountingJob> weak_factory_; + + DISALLOW_COPY_AND_ASSIGN(URLRequestErrorCountingJob); +}; + +static int error_jobs_created = 0; + +class MockURLRequestJobFactory : public URLRequestJobFactory { + public: + MockURLRequestJobFactory() {} + + ~MockURLRequestJobFactory() override {} + + URLRequestJob* MaybeCreateJobWithProtocolHandler( + const std::string& scheme, + URLRequest* request, + NetworkDelegate* network_delegate) const override { + ++error_jobs_created; + return new URLRequestErrorCountingJob(request, network_delegate, + ERR_INTERNET_DISCONNECTED); + } + + URLRequestJob* MaybeInterceptRedirect(URLRequest* request, + NetworkDelegate* network_delegate, + const GURL& location) const override { + return nullptr; + } + + URLRequestJob* MaybeInterceptResponse( + URLRequest* request, + NetworkDelegate* network_delegate) const override { + return nullptr; + } + + bool IsHandledProtocol(const std::string& scheme) const override { + return scheme == "http"; + }; + + bool IsHandledURL(const GURL& url) const override { + return url.SchemeIs("http"); + } + + bool IsSafeRedirectTarget(const GURL& location) const override { + return false; + } +}; + +class MockSdchDictionaryFetcher : public SdchDictionaryFetcher { + public: + MockSdchDictionaryFetcher() : SdchDictionaryFetcher(&test_context_) {} + ~MockSdchDictionaryFetcher() {} + + struct PendingRequest { + PendingRequest(const GURL& url, + const OnDictionaryFetchedCallback& callback) + : url_(url), callback_(callback) {} + GURL url_; + OnDictionaryFetchedCallback callback_; + }; + + virtual bool Schedule(const GURL& dictionary_url, + const OnDictionaryFetchedCallback& callback) { + if (!HasPendingRequest(dictionary_url)) { + requests_.push_back(PendingRequest(dictionary_url, callback)); + return true; + } + return false; + } + + virtual bool ScheduleReload(const GURL& dictionary_url, + const OnDictionaryFetchedCallback& callback) { + if (!HasPendingRequest(dictionary_url)) { + requests_.push_back(PendingRequest(dictionary_url, callback)); + return true; + } + return false; + } + + virtual void Cancel() { + requests_.clear(); + } + + bool HasPendingRequest(const GURL& dictionary_url) { + for (std::vector<PendingRequest>::iterator it = requests_.begin(); + it != requests_.end(); ++it) { + if (it->url_ == dictionary_url) + return true; + } + return false; + } + + bool CompletePendingRequest(const GURL& dictionary_url, + const std::string& dictionary_text, + const BoundNetLog& net_log, + bool was_from_cache) { + for (std::vector<PendingRequest>::iterator it = requests_.begin(); + it != requests_.end(); ++it) { + if (it->url_ == dictionary_url) { + it->callback_.Run(dictionary_text, dictionary_url, net_log, + was_from_cache); + requests_.erase(it); + return true; + } + } + return false; + } + + private: + TestURLRequestContext test_context_; + std::vector<PendingRequest> requests_; + DISALLOW_COPY_AND_ASSIGN(MockSdchDictionaryFetcher); +}; + +// File testing infrastructure summary: +// * NewSdchDictionary(): Creates a dictionary of a specific size. +// * URLRequestErrorCountingJob: A URLRequestJob that returns an error +// and counts the number of outstanding (started but not finished) +// jobs, and calls a global callback when that number transitions to zero. +// * MockURLRequestJobFactory: Factory to create the above jobs. Tracks +// the number of jobs created. +// * SdchOwnerTest: Interfaces +// * Access manager, owner, and net log +// * Return the number of jobs created in a time interval +// * Return dictionary present in the manager +// * Notify SdchOwner of an incoming dictionary (& wait until jobs clear) +// * Attempt to add a dictionary and test for success. +// Test patterns: +// * Let the owner know about a Get-Dictionary header and test for +// appropriate jobs being created. +// * Let the owner know that a dictionary was successfully fetched +// and test for appropriate outcome. +// * Either of the above, having previously added dictionaries to create +// a particular initial state. +class SdchOwnerTest : public testing::Test { + public: + static const size_t kMaxSizeForTesting = 1000 * 50; + static const size_t kMinFetchSpaceForTesting = 500; + + SdchOwnerTest() + : last_jobs_created_(error_jobs_created), + dictionary_creation_index_(0), + pref_store_(new TestingPrefStore), + sdch_owner_(new SdchOwner(&sdch_manager_, &url_request_context_)) { + // Any jobs created on this context will immediately error, + // which leaves the test in control of signals to SdchOwner. + url_request_context_.set_job_factory(&job_factory_); + + // Reduce sizes to reduce time for string operations. + sdch_owner_->SetMaxTotalDictionarySize(kMaxSizeForTesting); + sdch_owner_->SetMinSpaceForDictionaryFetch(kMinFetchSpaceForTesting); + } + + SdchManager& sdch_manager() { return sdch_manager_; } + SdchOwner& sdch_owner() { return *(sdch_owner_.get()); } + BoundNetLog& bound_net_log() { return net_log_; } + TestingPrefStore& pref_store() { return *(pref_store_.get()); } + + int JobsRecentlyCreated() { + int result = error_jobs_created - last_jobs_created_; + last_jobs_created_ = error_jobs_created; + return result; + } + + bool DictionaryPresentInManager(const std::string& server_hash) { + // Presumes all tests use generic url. + SdchProblemCode tmp; + scoped_ptr<SdchManager::DictionarySet> set( + sdch_manager_.GetDictionarySetByHash(GURL(generic_url), server_hash, + &tmp)); + return !!set.get(); + } + + void WaitForNoJobs() { + if (outstanding_url_request_error_counting_jobs == 0) + return; + + base::RunLoop run_loop; + base::Closure quit_closure(run_loop.QuitClosure()); + empty_url_request_jobs_callback = &quit_closure; + run_loop.Run(); + empty_url_request_jobs_callback = NULL; + } + + void SignalGetDictionaryAndClearJobs(GURL request_url, GURL dictionary_url) { + sdch_owner().OnGetDictionary(request_url, dictionary_url); + WaitForNoJobs(); + } + + // Create a unique (by hash) dictionary of the given size, + // associate it with a unique URL, add it to the manager through + // SdchOwner::OnDictionaryFetched(), and return whether that + // addition was successful or not. + bool CreateAndAddDictionary(size_t size, + std::string* server_hash_p, + base::Time last_used_time) { + GURL dictionary_url( + base::StringPrintf("%s/d%d", generic_url, dictionary_creation_index_)); + std::string dictionary_text(NewSdchDictionary(size - 4)); + dictionary_text += base::StringPrintf("%04d", dictionary_creation_index_); + ++dictionary_creation_index_; + std::string client_hash; + std::string server_hash; + SdchManager::GenerateHash(dictionary_text, &client_hash, &server_hash); + + if (DictionaryPresentInManager(server_hash)) + return false; + sdch_owner().OnDictionaryFetched(last_used_time, 0, dictionary_text, + dictionary_url, net_log_, false); + if (server_hash_p) + *server_hash_p = server_hash; + return DictionaryPresentInManager(server_hash); + } + + void ResetOwner() { + sdch_owner_.reset(new SdchOwner(&sdch_manager_, &url_request_context_)); + } + + private: + int last_jobs_created_; + BoundNetLog net_log_; + int dictionary_creation_index_; + + // The dependencies of these objects (sdch_owner_ -> {sdch_manager_, + // url_request_context_}, url_request_context_->job_factory_) require + // this order for correct destruction semantics. + MockURLRequestJobFactory job_factory_; + URLRequestContext url_request_context_; + SdchManager sdch_manager_; + scoped_refptr<TestingPrefStore> pref_store_; + scoped_ptr<SdchOwner> sdch_owner_; + + DISALLOW_COPY_AND_ASSIGN(SdchOwnerTest); +}; + +// Does OnGetDictionary result in a fetch when there's enough space, and not +// when there's not? +TEST_F(SdchOwnerTest, OnGetDictionary_Fetching) { + GURL request_url(std::string(generic_url) + "/r1"); + + // Fetch generated when empty. + GURL dict_url1(std::string(generic_url) + "/d1"); + EXPECT_EQ(0, JobsRecentlyCreated()); + SignalGetDictionaryAndClearJobs(request_url, dict_url1); + EXPECT_EQ(1, JobsRecentlyCreated()); + + // Fetch generated when half full. + GURL dict_url2(std::string(generic_url) + "/d2"); + std::string dictionary1(NewSdchDictionary(kMaxSizeForTesting / 2)); + sdch_owner().OnDictionaryFetched(base::Time::Now(), 1, dictionary1, + dict_url1, bound_net_log(), false); + EXPECT_EQ(0, JobsRecentlyCreated()); + SignalGetDictionaryAndClearJobs(request_url, dict_url2); + EXPECT_EQ(1, JobsRecentlyCreated()); + + // Fetch not generated when close to completely full. + GURL dict_url3(std::string(generic_url) + "/d3"); + std::string dictionary2(NewSdchDictionary( + (kMaxSizeForTesting / 2 - kMinFetchSpaceForTesting / 2))); + sdch_owner().OnDictionaryFetched(base::Time::Now(), 1, dictionary2, + dict_url2, bound_net_log(), false); + EXPECT_EQ(0, JobsRecentlyCreated()); + SignalGetDictionaryAndClearJobs(request_url, dict_url3); + EXPECT_EQ(0, JobsRecentlyCreated()); +} + +// Make sure attempts to add dictionaries do what they should. +TEST_F(SdchOwnerTest, OnDictionaryFetched_Fetching) { + GURL request_url(std::string(generic_url) + "/r1"); + std::string client_hash; + std::string server_hash; + + // In the past, but still fresh for an unused dictionary. + base::Time dictionary_last_used_time(base::Time::Now() - + base::TimeDelta::FromMinutes(30)); + + // Add successful when empty. + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 2, nullptr, + dictionary_last_used_time)); + EXPECT_EQ(0, JobsRecentlyCreated()); + + // Add successful when half full. + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 2, nullptr, + dictionary_last_used_time)); + EXPECT_EQ(0, JobsRecentlyCreated()); + + // Add unsuccessful when full. + EXPECT_FALSE(CreateAndAddDictionary(kMaxSizeForTesting / 2, nullptr, + dictionary_last_used_time)); + EXPECT_EQ(0, JobsRecentlyCreated()); +} + +// Confirm auto-eviction happens if space is needed. +TEST_F(SdchOwnerTest, ConfirmAutoEviction) { + base::Time start_time = base::Time::Now(); + std::string server_hash_d1; + std::string server_hash_d2; + std::string server_hash_d3; + + base::SimpleTestClock* test_clock = new base::SimpleTestClock(); + sdch_owner().SetClockForTesting(make_scoped_ptr(test_clock)); + test_clock->SetNow(base::Time::Now()); + + // Add two dictionaries, one recent, one more than a day in the past. + base::Time fresh(base::Time::Now() - base::TimeDelta::FromHours(23)); + base::Time stale(base::Time::Now() - base::TimeDelta::FromHours(25)); + + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d1, fresh)); + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d2, stale)); + + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d2)); + + base::HistogramTester tester; + const base::TimeDelta synthetic_delta = base::TimeDelta::FromSeconds(5); + + test_clock->Advance(synthetic_delta); + + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d3, fresh)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d3)); + + base::TimeDelta expected_proc_lifetime = synthetic_delta * 3 + + base::Time::Now() - start_time; + size_t expected_value_base = ((kMaxSizeForTesting / 2) * + synthetic_delta.InMilliseconds()) / + expected_proc_lifetime.InMilliseconds(); + + const char *kHistogram = "Sdch3.TimeWeightedMemoryUse"; + tester.ExpectTotalCount(kHistogram, 0); + + // Dictionary insertions and deletions: + // T = 0: insert d1 and d2 + // T = 5: insert d3, which evicts d2 + // T = 15: destroy SdchOwner, which evicts d1 and d3 + // Therefore, d2's lifetime is synthetic_delta, d1's is synthetic_delta * 3, + // and d3's is synthetic_delta * 2. The expected_value_base variable is the + // base factor for d2's memory consumption, of which d1's and d3's are + // multiples. + test_clock->Advance(synthetic_delta * 2); + ResetOwner(); + + tester.ExpectTotalCount(kHistogram, 3); + tester.ExpectBucketCount(kHistogram, expected_value_base, 1); + tester.ExpectBucketCount(kHistogram, expected_value_base * 2, 1); + tester.ExpectBucketCount(kHistogram, expected_value_base * 3, 1); +} + +// Confirm auto-eviction happens if space is needed, with a more complicated +// situation +TEST_F(SdchOwnerTest, ConfirmAutoEviction_2) { + std::string server_hash_d1; + std::string server_hash_d2; + std::string server_hash_d3; + + // Add dictionaries, one recent, two more than a day in the past that + // between them add up to the space needed. + base::Time fresh(base::Time::Now() - base::TimeDelta::FromHours(23)); + base::Time stale(base::Time::Now() - base::TimeDelta::FromHours(25)); + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d1, fresh)); + + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d2, stale)); + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d3, stale)); + + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d3)); + + std::string server_hash_d4; + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d4, fresh)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d3)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d4)); +} + +// Confirm if only one dictionary needs to be evicted it's the oldest. +TEST_F(SdchOwnerTest, ConfirmAutoEviction_Oldest) { + std::string server_hash_d1; + std::string server_hash_d2; + std::string server_hash_d3; + + // Add dictionaries, one recent, one two days in the past, and one + // four days in the past. + base::Time fresh(base::Time::Now() - base::TimeDelta::FromHours(23)); + base::Time stale_newer(base::Time::Now() - base::TimeDelta::FromHours(47)); + base::Time stale_older(base::Time::Now() - base::TimeDelta::FromHours(71)); + + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d1, fresh)); + + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d2, + stale_newer)); + + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d3, + stale_older)); + + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d3)); + + // The addition of a new dictionary should succeed, evicting only the + // oldest one. + + std::string server_hash_d4; + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d4, fresh)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d3)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d4)); +} + +// Confirm using a dictionary changes eviction behavior properly. +TEST_F(SdchOwnerTest, UseChangesEviction) { + std::string server_hash_d1; + std::string server_hash_d2; + std::string server_hash_d3; + + // Add dictionaries, one recent, one two days in the past, and one + // four days in the past. + base::Time fresh(base::Time::Now() - base::TimeDelta::FromHours(23)); + base::Time stale_newer(base::Time::Now() - base::TimeDelta::FromHours(47)); + base::Time stale_older(base::Time::Now() - base::TimeDelta::FromHours(71)); + + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d1, fresh)); + + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d2, + stale_newer)); + + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d3, + stale_older)); + + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d3)); + + // Use the oldest dictionary. + sdch_owner().OnDictionaryUsed(server_hash_d3); + + // The addition of a new dictionary should succeed, evicting only the + // newer stale one. + std::string server_hash_d4; + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d4, fresh)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d3)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d4)); +} + +// Confirm using a dictionary can prevent the addition of a new dictionary. +TEST_F(SdchOwnerTest, UsePreventsAddition) { + std::string server_hash_d1; + std::string server_hash_d2; + std::string server_hash_d3; + + // Add dictionaries, one recent, one two days in the past, and one + // four days in the past. + base::Time fresh(base::Time::Now() - base::TimeDelta::FromMinutes(30)); + base::Time stale_newer(base::Time::Now() - base::TimeDelta::FromHours(47)); + base::Time stale_older(base::Time::Now() - base::TimeDelta::FromHours(71)); + + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d1, fresh)); + + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d2, + stale_newer)); + + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 4, &server_hash_d3, + stale_older)); + + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d3)); + + // Use the older dictionaries. + sdch_owner().OnDictionaryUsed(server_hash_d2); + sdch_owner().OnDictionaryUsed(server_hash_d3); + + // The addition of a new dictionary should fail, not evicting anything. + std::string server_hash_d4; + EXPECT_FALSE( + CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d4, fresh)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d2)); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d3)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d4)); +} + +// Confirm clear gets all the space back. +TEST_F(SdchOwnerTest, ClearReturnsSpace) { + std::string server_hash_d1; + std::string server_hash_d2; + + // Take up all the space. + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting, &server_hash_d1, + base::Time::Now())); + // Addition should fail. + EXPECT_FALSE(CreateAndAddDictionary(kMaxSizeForTesting, &server_hash_d2, + base::Time::Now())); + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d2)); + sdch_manager().ClearData(); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d2)); + + // Addition should now succeed. + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting, nullptr, base::Time::Now())); +} + +// Confirm memory pressure gets all the space back. +TEST_F(SdchOwnerTest, MemoryPressureReturnsSpace) { + std::string server_hash_d1; + std::string server_hash_d2; + + // Take up all the space. + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting, &server_hash_d1, + base::Time::Now())); + + // Addition should fail. + EXPECT_FALSE(CreateAndAddDictionary(kMaxSizeForTesting, &server_hash_d2, + base::Time::Now())); + + EXPECT_TRUE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d2)); + + base::MemoryPressureListener::NotifyMemoryPressure( + base::MemoryPressureListener::MEMORY_PRESSURE_LEVEL_MODERATE); + // The notification may have (implementation note: does :-}) use a PostTask, + // so we drain the local message queue. This should be safe (i.e. not have + // an inifinite number of messages) in a unit test. + base::RunLoop().RunUntilIdle(); + + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d1)); + EXPECT_FALSE(DictionaryPresentInManager(server_hash_d2)); + + // Addition should now succeed. + EXPECT_TRUE( + CreateAndAddDictionary(kMaxSizeForTesting, nullptr, base::Time::Now())); +} + +// Confirm that use of a pinned dictionary after its removal works properly. +TEST_F(SdchOwnerTest, PinRemoveUse) { + pref_store().SetInitializationCompleted(); + sdch_owner().EnablePersistentStorage(&pref_store()); + + std::string server_hash_d1; + EXPECT_TRUE(CreateAndAddDictionary(kMaxSizeForTesting / 2, &server_hash_d1, + base::Time::Now())); + + scoped_ptr<SdchManager::DictionarySet> return_set( + sdch_manager().GetDictionarySet( + GURL(std::string(generic_url) + "/x.html"))); + ASSERT_TRUE(return_set.get()); + EXPECT_TRUE(return_set->GetDictionaryText(server_hash_d1)); + + const base::Value* result = nullptr; + const base::DictionaryValue* dict_result = nullptr; + ASSERT_TRUE(pref_store().GetValue("SDCH", &result)); + ASSERT_TRUE(result->GetAsDictionary(&dict_result)); + EXPECT_TRUE(dict_result->Get("dictionaries", &result)); + EXPECT_TRUE(dict_result->Get("dictionaries." + server_hash_d1, &result)); + + sdch_manager().ClearData(); + + ASSERT_TRUE(pref_store().GetValue("SDCH", &result)); + ASSERT_TRUE(result->GetAsDictionary(&dict_result)); + EXPECT_TRUE(dict_result->Get("dictionaries", &result)); + EXPECT_FALSE(dict_result->Get("dictionaries." + server_hash_d1, &result)); + + scoped_ptr<SdchManager::DictionarySet> return_set2( + sdch_manager().GetDictionarySet( + GURL(std::string(generic_url) + "/x.html"))); + EXPECT_FALSE(return_set2.get()); + + sdch_manager().OnDictionaryUsed(server_hash_d1); + + ASSERT_TRUE(pref_store().GetValue("SDCH", &result)); + ASSERT_TRUE(result->GetAsDictionary(&dict_result)); + EXPECT_TRUE(dict_result->Get("dictionaries", &result)); + EXPECT_FALSE(dict_result->Get("dictionaries." + server_hash_d1, &result)); +} + +class SdchOwnerPersistenceTest : public ::testing::Test { + public: + SdchOwnerPersistenceTest() : pref_store_(new TestingPrefStore()) { + pref_store_->SetInitializationCompleted(); + } + virtual ~SdchOwnerPersistenceTest() {} + + void ClearOwner() { + owner_.reset(NULL); + } + + void ResetOwner(bool delay) { + // This has to be done first, since SdchOwner may be observing SdchManager, + // and SdchManager can't be destroyed with a live observer. + owner_.reset(NULL); + manager_.reset(new SdchManager()); + fetcher_ = new MockSdchDictionaryFetcher(); + owner_.reset(new SdchOwner(manager_.get(), + &url_request_context_)); + owner_->SetMaxTotalDictionarySize(SdchOwnerTest::kMaxSizeForTesting); + owner_->SetMinSpaceForDictionaryFetch( + SdchOwnerTest::kMinFetchSpaceForTesting); + owner_->SetFetcherForTesting(make_scoped_ptr(fetcher_)); + if (!delay) + owner_->EnablePersistentStorage(pref_store_.get()); + } + + void InsertDictionaryForURL(const GURL& url, const std::string& nonce) { + owner_->OnDictionaryFetched(base::Time::Now(), 1, + CreateDictionary(url, nonce), + url, net_log_, false); + } + + bool CompleteLoadFromURL(const GURL& url, const std::string& nonce, + bool was_from_cache) { + return fetcher_->CompletePendingRequest(url, CreateDictionary(url, nonce), + net_log_, was_from_cache); + } + + std::string CreateDictionary(const GURL& url, const std::string& nonce) { + std::string dict; + dict.append("Domain: "); + dict.append(url.host()); + dict.append("\n\n"); + dict.append(url.spec()); + dict.append(nonce); + return dict; + } + + protected: + BoundNetLog net_log_; + scoped_refptr<TestingPrefStore> pref_store_; + scoped_ptr<SdchManager> manager_; + MockSdchDictionaryFetcher* fetcher_; + scoped_ptr<SdchOwner> owner_; + TestURLRequestContext url_request_context_; +}; + +// Test an empty persistence store. +TEST_F(SdchOwnerPersistenceTest, Empty) { + ResetOwner(false); + EXPECT_EQ(0, owner_->GetDictionaryCountForTesting()); +} + +// Test a persistence store with an empty dictionary. +TEST_F(SdchOwnerPersistenceTest, Persistent_EmptyDict) { + pref_store_->SetValue("SDCH", new base::DictionaryValue(), + WriteablePrefStore::DEFAULT_PREF_WRITE_FLAGS); + ResetOwner(false); + EXPECT_EQ(0, owner_->GetDictionaryCountForTesting()); +} + +// Test a persistence store with a bad version number. +TEST_F(SdchOwnerPersistenceTest, Persistent_BadVersion) { + base::DictionaryValue* sdch_dict = new base::DictionaryValue(); + sdch_dict->SetInteger("version", 2); + pref_store_->SetValue("SDCH", sdch_dict, + WriteablePrefStore::DEFAULT_PREF_WRITE_FLAGS); + + ResetOwner(false); + EXPECT_EQ(0, owner_->GetDictionaryCountForTesting()); +} + +// Test a persistence store with an empty dictionaries map. +TEST_F(SdchOwnerPersistenceTest, Persistent_EmptyDictList) { + base::DictionaryValue* sdch_dict = new base::DictionaryValue(); + scoped_ptr<base::DictionaryValue> dicts(new base::DictionaryValue()); + sdch_dict->SetInteger("version", 1); + sdch_dict->Set("dictionaries", dicts.Pass()); + pref_store_->SetValue("SDCH", sdch_dict, + WriteablePrefStore::DEFAULT_PREF_WRITE_FLAGS); + + ResetOwner(false); + EXPECT_EQ(0, owner_->GetDictionaryCountForTesting()); +} + +TEST_F(SdchOwnerPersistenceTest, OneDict) { + const GURL url("http://www.example.com/dict"); + ResetOwner(false); + EXPECT_EQ(0, owner_->GetDictionaryCountForTesting()); + InsertDictionaryForURL(url, "0"); + EXPECT_EQ(1, owner_->GetDictionaryCountForTesting()); + + ResetOwner(false); + EXPECT_EQ(0, owner_->GetDictionaryCountForTesting()); + EXPECT_TRUE(CompleteLoadFromURL(url, "0", true)); + EXPECT_EQ(1, owner_->GetDictionaryCountForTesting()); +} + +TEST_F(SdchOwnerPersistenceTest, TwoDicts) { + const GURL url0("http://www.example.com/dict0"); + const GURL url1("http://www.example.com/dict1"); + ResetOwner(false); + InsertDictionaryForURL(url0, "0"); + InsertDictionaryForURL(url1, "1"); + + ResetOwner(false); + EXPECT_TRUE(CompleteLoadFromURL(url0, "0", true)); + EXPECT_TRUE(CompleteLoadFromURL(url1, "1", true)); + EXPECT_EQ(2, owner_->GetDictionaryCountForTesting()); + EXPECT_TRUE(owner_->HasDictionaryFromURLForTesting(url0)); + EXPECT_TRUE(owner_->HasDictionaryFromURLForTesting(url1)); +} + +TEST_F(SdchOwnerPersistenceTest, OneGoodDictOneBadDict) { + const GURL url0("http://www.example.com/dict0"); + const GURL url1("http://www.example.com/dict1"); + ResetOwner(false); + InsertDictionaryForURL(url0, "0"); + InsertDictionaryForURL(url1, "1"); + + // Mutate the pref store a bit now. Clear the owner first, to ensure that the + // SdchOwner doesn't observe these changes and object. The manual dictionary + // manipulation is a bit icky. + ClearOwner(); + base::DictionaryValue* dict = nullptr; + ASSERT_TRUE(GetDictionaryForURL(pref_store_.get(), url1, nullptr, &dict)); + dict->Remove("use_count", nullptr); + + ResetOwner(false); + EXPECT_TRUE(CompleteLoadFromURL(url0, "0", true)); + EXPECT_FALSE(CompleteLoadFromURL(url1, "1", true)); + EXPECT_EQ(1, owner_->GetDictionaryCountForTesting()); + EXPECT_TRUE(owner_->HasDictionaryFromURLForTesting(url0)); + EXPECT_FALSE(owner_->HasDictionaryFromURLForTesting(url1)); +} + +TEST_F(SdchOwnerPersistenceTest, UsingDictionaryUpdatesUseCount) { + const GURL url("http://www.example.com/dict"); + ResetOwner(false); + InsertDictionaryForURL(url, "0"); + + std::string hash; + int old_count; + { + ClearOwner(); + base::DictionaryValue* dict = nullptr; + ASSERT_TRUE(GetDictionaryForURL(pref_store_.get(), url, &hash, &dict)); + ASSERT_TRUE(dict->GetInteger("use_count", &old_count)); + } + + ResetOwner(false); + ASSERT_TRUE(CompleteLoadFromURL(url, "0", true)); + owner_->OnDictionaryUsed(hash); + + int new_count; + { + ClearOwner(); + base::DictionaryValue* dict = nullptr; + ASSERT_TRUE(GetDictionaryForURL(pref_store_.get(), url, nullptr, &dict)); + ASSERT_TRUE(dict->GetInteger("use_count", &new_count)); + } + + EXPECT_EQ(old_count + 1, new_count); +} + +TEST_F(SdchOwnerPersistenceTest, LoadingDictionaryMerges) { + const GURL url0("http://www.example.com/dict0"); + const GURL url1("http://www.example.com/dict1"); + + ResetOwner(false); + InsertDictionaryForURL(url1, "1"); + + ResetOwner(true); + InsertDictionaryForURL(url0, "0"); + EXPECT_EQ(1, owner_->GetDictionaryCountForTesting()); + owner_->EnablePersistentStorage(pref_store_.get()); + ASSERT_TRUE(CompleteLoadFromURL(url1, "1", true)); + EXPECT_EQ(2, owner_->GetDictionaryCountForTesting()); +} + +TEST_F(SdchOwnerPersistenceTest, PersistenceMetrics) { + const GURL url0("http://www.example.com/dict0"); + const GURL url1("http://www.example.com/dict1"); + ResetOwner(false); + + InsertDictionaryForURL(url0, "0"); + InsertDictionaryForURL(url1, "1"); + + ResetOwner(false); + + base::HistogramTester tester; + + EXPECT_TRUE(CompleteLoadFromURL(url0, "0", true)); + EXPECT_TRUE(CompleteLoadFromURL(url1, "1", false)); + + tester.ExpectTotalCount("Sdch3.NetworkBytesSpent", 1); + tester.ExpectUniqueSample("Sdch3.NetworkBytesSpent", + CreateDictionary(url1, "1").size(), 1); +} + +} // namespace net |