diff options
author | Jordan Cook <jordan.cook@pioneer.com> | 2021-03-20 15:19:58 -0500 |
---|---|---|
committer | Jordan Cook <jordan.cook@pioneer.com> | 2021-03-24 16:46:27 -0500 |
commit | ab18e06217a8a1902d1859aae6b033f469b2232b (patch) | |
tree | 98136dffa4d16ed3d645e2bdee1ac60421001ff7 | |
parent | 78819897213ec3bff57d4c7094fd585dbb8d48d4 (diff) | |
download | requests-cache-ab18e06217a8a1902d1859aae6b033f469b2232b.tar.gz |
More usage examples, formatting, and editing for Readme + Sphinx docs
* Closes #135, #165
* Add a 'Summary' section at the top of the Readme explaining the scope of requests-cache and why you would want to use it
* Add some more info explaining the difference between using `CachedSession` directly vs. patching with `install_cache()`
* Move basic examples from 'User Guide' section into Readme
* Include Readme in Sphinx docs (using `.. mdinclude::`) and remove duplicate sections
* Include Contributing guide in Sphinx docs
* Convert History doc to markdown and include in Sphinx docs
* Use `automod` options to move main cache documentation from `CacheMixin` back to to `CachedSession`, since that's probably where a user will look first
* Add more detailed usage examples to an 'Advanced Usage' section for `CachedSession` options (`filter_fn`, `ignore_parameters`, etc.)
* Update example scripts and move them to `examples/` folder
-rw-r--r-- | CONTRIBUTING.md | 4 | ||||
-rw-r--r-- | HISTORY.md | 123 | ||||
-rw-r--r-- | HISTORY.rst | 184 | ||||
-rw-r--r-- | README.md | 143 | ||||
-rw-r--r-- | docs/advanced_usage.rst | 349 | ||||
-rw-r--r-- | docs/api.rst | 45 | ||||
-rw-r--r-- | docs/conf.py | 4 | ||||
-rw-r--r-- | docs/contributing.rst | 1 | ||||
-rw-r--r-- | docs/history.rst | 1 | ||||
-rw-r--r-- | docs/index.rst | 29 | ||||
-rw-r--r-- | docs/user_guide.rst | 229 | ||||
-rw-r--r-- | example.py | 30 | ||||
-rw-r--r-- | example_request.py | 32 | ||||
-rw-r--r-- | examples/basic_usage.py | 31 | ||||
-rw-r--r-- | examples/expiration.py | 42 | ||||
-rw-r--r-- | examples/session_patch.py | 36 | ||||
-rw-r--r-- | pyproject.toml | 2 | ||||
-rw-r--r-- | setup.cfg | 6 |
18 files changed, 738 insertions, 553 deletions
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9a217e9..64d0cd2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,7 +15,7 @@ If you have an issue or PR that hasn't received a response in a timely manner, o discuss ideas about the project in general, please reach out on the Code Shelter chat server, under [projects/requests-cache](https://codeshelter.zulipchat.com/#narrow/stream/186993-projects/topic/requests-cache). -## Installation +## Dev Installation To set up for local development: ```bash @@ -24,7 +24,7 @@ $ cd requests-cache $ pip install -Ue ".[dev]" ``` -## Pre-commit hooks +## Pre-commit Hooks [Pre-commit](https://github.com/pre-commit/pre-commit) config is included to run the same checks locally that are run in CI jobs by GitHub Actions. This is optional but recommended. ```bash diff --git a/HISTORY.md b/HISTORY.md new file mode 100644 index 0000000..ce3e6dd --- /dev/null +++ b/HISTORY.md @@ -0,0 +1,123 @@ +# History + +## 0.6.0 (TBD) +[See all included issues and PRs](https://github.com/reclosedev/requests-cache/milestone/1?closed=1) + +* Drop support for python <= 3.5 +* Add support for setting expiration for individual requests +* Add support for setting expiration based on URL glob patterns +* Add support for overriding original expiration (i.e., revalidating) in `CachedSession.remove_expired_responses()` +* Add `CacheMixin` class to be make the features of `CachedSession` usable as a mixin class, + for compatibility with other `requests`-based libraries +* Add `CachedResponse` class to wrapped cached `requests.Response` objects, make additional cache + information available to client code +* Add property `BaseCache.urls` to get all URLs persisted in the cache +* Update usage of deprecated MongoClient `save()` method +* Fix TypeError with `DbPickleDict` initialization +* Fix usage of `CachedSession.cache_disabled` if used within another contextmanager +* Fix non-thread-safe iteration in `BaseCache` +* Fix `get_cache()`, `clear()`, and `remove_expired_responses()` so they will do nothing if + requests-cache is not installed +* Add `HEAD` to default `allowable_methods` + +## 0.5.2 (2019-08-14) +* Fix DeprecationWarning from collections #140 + +## 0.5.1 (2019-08-13) +* Remove Python 2.6 Testing from travis #133 +* Fix DeprecationWarning from collections #131 +* vacuum the sqlite database after clearing a table #134 +* Fix handling of unpickle errors #128 + +## 0.5.0 (2019-04-18) +Project is now added to [Code Shelter](https://www.codeshelter.co) + +* Add gridfs support, thanks to @chengguangnan +* Add dynamodb support, thanks to @ar90n +* Add response filter #104, thanks to @christopher-dG +* Fix bulk_commit #78 +* Fix remove_expired_responses missed in __init__.py #93 +* Fix deprecation warnings #122, thanks to mbarkhau + +## 0.4.13 (2016-12-23) +* Support PyMongo3, thanks to @craigls #72 +* Fix streaming releate issue #68 + +## 0.4.12 (2016-03-19) +* Fix ability to pass backend instance in `install_cache` #61 + + +## 0.4.11 (2016-03-07) +* `ignore_parameters` feature, thanks to @themiurgo and @YetAnotherNerd (#52, #55) +* More informative message for missing backend dependencies, thanks to @Garrett-R (#60) + +## 0.4.10 (2015-04-28) +* Better transactional handling in sqlite #50, thanks to @rgant +* Compatibility with streaming in requests >= 2.6.x + +## 0.4.9 (2015-01-17) +* `expire_after` now also accepts `timedelta`, thanks to @femtotrader +* Added Ability to include headers to cache key (`include_get_headers` option) +* Added string representation for `CachedSession` + +## 0.4.8 (2014-12-13) +* Fix bug in reading cached streaming response + +## 0.4.7 (2014-12-06) +* Fix compatibility with Requests > 2.4.1 (json arg, response history) + +## 0.4.6 (2014-10-13) +* Monkey patch now uses class instead lambda (compatibility with rauth) +* Normalize (sort) parameters passed as builtin dict + +## 0.4.5 (2014-08-22) +* Requests==2.3.0 compatibility, thanks to @gwillem + +## 0.4.4 (2013-10-31) +* Check for backend availability in install_cache(), not at the first request +* Default storage fallbacks to memory if `sqlite` is not available + +## 0.4.3 (2013-09-12) +* Fix `response.from_cache` not set in hooks + +## 0.4.2 (2013-08-25) +* Fix `UnpickleableError` for gzip responses + + +## 0.4.1 (2013-08-19) +* `requests_cache.enabled()` context manager +* Compatibility with Requests 1.2.3 cookies handling + +## 0.4.0 (2013-04-25) +* Redis backend. Thanks to @michaelbeaumont +* Fix for changes in Requests 1.2.0 hooks dispatching + + +## 0.3.0 (2013-02-24) +* Support for `Requests` 1.x.x +* `CachedSession` +* Many backward incompatible changes + +## 0.2.1 (2013-01-13) +* Fix broken PyPi package + +## 0.2.0 (2013-01-12) +* Last backward compatible version for `Requests` 0.14.2 + +## 0.1.3 (2012-05-04) +* Thread safety for default `sqlite` backend +* Take into account the POST parameters when cache is configured + with 'POST' in `allowable_methods` + +## 0.1.2 (2012-05-02) +* Reduce number of `sqlite` database write operations +* `fast_save` option for `sqlite` backend + +## 0.1.1 (2012-04-11) +* Fix: restore responses from response.history +* Internal refactoring (`MemoryCache` -> `BaseCache`, `reduce_response` + and `restore_response` moved to `BaseCache`) +* `connection` option for `MongoCache` + +## 0.1.0 (2012-04-10) +* initial PyPI release diff --git a/HISTORY.rst b/HISTORY.rst deleted file mode 100644 index 6be510b..0000000 --- a/HISTORY.rst +++ /dev/null @@ -1,184 +0,0 @@ -.. :changelog: - -History -------- - -0.6.0 (TBD) -++++++++++++++++++ -`See all included issues and PRs <https://github.com/reclosedev/requests-cache/milestone/1?closed=1>`_ - -* Drop support for python <= 3.5 -* Fix TypeError with ``DbPickleDict`` initialization -* Fix usage of `CachedSession.cache_disabled` if used within another contextmanager -* Update usage of deprecated MongoClient ``save()`` method -* Fix non-thread-safe iteration in ``BaseCache`` -* Add support for setting ``expire_after`` per request -* Refactor CachedSession to be usable as a mixin class, for compatibility with other ``requests.Session`` extensions -* Fix get_cache, clear, and remove_expired_responses to do nothing if requests-cache is not installed -* Add ``HEAD`` to default ``allowable_methods`` - - -0.5.2 (2019-08-14) -++++++++++++++++++ - -* Fix DeprecationWarning from collections #140 - - -0.5.1 (2019-08-13) -++++++++++++++++++ - -* Remove Python 2.6 Testing from travis #133 -* Fix DeprecationWarning from collections #131 -* vacuum the sqlite database after clearing a table #134 -* Fix handling of unpickle errors #128 - - -0.5.0 (2019-04-18) -++++++++++++++++++ -Project is now added to https://www.codeshelter.co/ - -* Add gridfs support, thanks to @chengguangnan -* Add dynamodb support, thanks to @ar90n -* Add response filter #104, thanks to @christopher-dG -* Fix bulk_commit #78 -* Fix remove_expired_responses missed in __init__.py #93 -* Fix deprecation warnings #122, thanks to mbarkhau - - -0.4.13 (2016-12-23) -+++++++++++++++++++ -* Support PyMongo3, thanks to @craigls #72 -* Fix streaming releate issue #68 - - -0.4.12 (2016-03-19) -+++++++++++++++++++ -* Fix ability to pass backend instance in ``install_cache`` #61 - - -0.4.11 (2016-03-07) -+++++++++++++++++++ -* ``ignore_parameters`` feature, thanks to @themiurgo and @YetAnotherNerd (#52, #55) -* More informative message for missing backend dependencies, thanks to @Garrett-R (#60) - - -0.4.10 (2015-04-28) -+++++++++++++++++++ -* Better transactional handling in sqlite #50, thanks to @rgant -* Compatibility with streaming in requests >= 2.6.x - - -0.4.9 (2015-01-17) -++++++++++++++++++ - -* ``expire_after`` now also accepts ``timedelta``, thanks to @femtotrader -* Added Ability to include headers to cache key (``include_get_headers`` option) -* Added string representation for ``CachedSession`` - - -0.4.8 (2014-12-13) -++++++++++++++++++ - -* Fix bug in reading cached streaming response - - -0.4.7 (2014-12-06) -++++++++++++++++++ - -* Fix compatibility with Requests > 2.4.1 (json arg, response history) - - -0.4.6 (2014-10-13) -++++++++++++++++++ - -* Monkey patch now uses class instead lambda (compatibility with rauth) -* Normalize (sort) parameters passed as builtin dict - - -0.4.5 (2014-08-22) -++++++++++++++++++ - -* Requests==2.3.0 compatibility, thanks to @gwillem - - -0.4.4 (2013-10-31) -++++++++++++++++++ - -* Check for backend availability in install_cache(), not at the first request -* Default storage fallbacks to memory if ``sqlite`` is not available - - -0.4.3 (2013-09-12) -++++++++++++++++++ - -* Fix ``response.from_cache`` not set in hooks - - - -0.4.2 (2013-08-25) -++++++++++++++++++ - -* Fix ``UnpickleableError`` for gzip responses - - - -0.4.1 (2013-08-19) -++++++++++++++++++ - -* ``requests_cache.enabled()`` context manager -* Compatibility with Requests 1.2.3 cookies handling - - -0.4.0 (2013-04-25) -++++++++++++++++++ - -* Redis backend. Thanks to @michaelbeaumont -* Fix for changes in Requests 1.2.0 hooks dispatching - - -0.3.0 (2013-02-24) -++++++++++++++++++ - -* Support for ``Requests`` 1.x.x -* ``CachedSession`` -* Many backward incompatible changes - -0.2.1 (2013-01-13) -++++++++++++++++++ - -* Fix broken PyPi package - -0.2.0 (2013-01-12) -++++++++++++++++++ - -* Last backward compatible version for ``Requests`` 0.14.2 - - -0.1.3 (2012-05-04) -++++++++++++++++++ - -* Thread safety for default ``sqlite`` backend -* Take into account the POST parameters when cache is configured - with 'POST' in ``allowable_methods`` - - -0.1.2 (2012-05-02) -++++++++++++++++++ - -* Reduce number of ``sqlite`` database write operations -* ``fast_save`` option for ``sqlite`` backend - - -0.1.1 (2012-04-11) -++++++++++++++++++ - -* Fix: restore responses from response.history -* Internal refactoring (``MemoryCache`` -> ``BaseCache``, ``reduce_response`` - and ``restore_response`` moved to ``BaseCache``) -* ``connection`` option for ``MongoCache`` - - -0.1.0 (2012-04-10) -++++++++++++++++++ - -* initial PyPI release @@ -1,6 +1,4 @@ # requests-cache -Requests-cache is a transparent persistent cache for the [requests](http://python-requests.org) library (version 2+). - [![Build](https://github.com/reclosedev/requests-cache/actions/workflows/build.yml/badge.svg)](https://github.com/reclosedev/requests-cache/actions/workflows/build.yml) [![Coverage](https://coveralls.io/repos/github/reclosedev/requests-cache/badge.svg?branch=master)](https://coveralls.io/github/reclosedev/requests-cache?branch=master) [![Documentation](https://img.shields.io/readthedocs/requests-cache/latest)](https://requests-cache.readthedocs.io/en/latest/) @@ -9,51 +7,145 @@ Requests-cache is a transparent persistent cache for the [requests](http://pytho [![PyPI - Format](https://img.shields.io/pypi/format/requests-cache?color=blue)](https://pypi.org/project/requests-cache) [![Code Shelter](https://www.codeshelter.co/static/badges/badge-flat.svg)](https://www.codeshelter.co/) +## Summary +**Requests-cache** is a transparent persistent HTTP cache for the python [requests](http://python-requests.org) +library. It is especially useful for web scraping, consuming REST APIs, slow or rate-limited +sites, or any other scenario in which you're making lots of requests that are likely to be sent +more than once. + +Several storage backends are included: **SQLite**, **Redis**, **MongoDB**, and **DynamoDB**. + +See full project documentation at: https://requests-cache.readthedocs.io + ## Installation Install with pip: -``` +```bash pip install requests-cache ``` -## Usage example +Requirements: +Requires python 3.6+. +You may need additional dependencies depending on which backend you want to use. +To install with extra dependencies for all supported backends: +```bash +pip install requests-cache[backends] +``` + +See [Contributing Guide](https://github.com/reclosedev/requests-cache/blob/master/CONTRIBUTING.md) +for setup info for local development. + +## Full Examples +* You can find a working example at Real Python: +[Caching External API Requests](https://realpython.com/blog/python/caching-external-api-requests) +* There are some additional examples in the [examples/](https://github.com/reclosedev/requests-cache/tree/master/examples) folder + +## General Usage +There are two main ways of using `requests-cache`: +* Using [CachedSession](https://requests-cache.readthedocs.io/en/latest/api.html#requests_cache.core.CachedSession) (recommended) +* Globally patching `requests` using [install_cache](https://requests-cache.readthedocs.io/en/latest/api.html#requests_cache.core.install_cache) + +### Sessions +`CachedSession` wraps `requests.Session` with caching features, and otherwise behaves the same as a +normal session. + +Basic example: ```python -import requests_cache +from requests_cache import CachedSession -requests_cache.install_cache('demo_cache') +session = CachedSession('demo_cache', backend='sqlite') +for i in range(100): + session.get('http://httpbin.org/delay/1') ``` +The URL in this example adds a delay of 1 second, but all 100 requests will complete in just over 1 +second. The response will be fetched once, saved to `demo_cache.sqlite`, and subsequent requests +will return the cached response near-instantly. -And all responses with headers and cookies will be transparently cached to `demo_cache.sqlite`. -For example, following the code will take only 1-2 seconds instead of 10, and will run instantly on next launch: +There are many ways to customize caching behavior; see +[Advanced Usage](https://requests-cache.readthedocs.io/en/latest/advanced_usage.html) for details. +### Patching +Patching with `requests_cache.install_cache()` will add caching to all `requests` functions: ```python import requests +import requests_cache -for i in range(10): - requests.get('http://httpbin.org/delay/1') +requests_cache.install_cache() +requests.get('http://httpbin.org/get') +session = requests.Session() +session.get('http://httpbin.org/get') ``` -It can be useful when you are creating some simple data scraper with constantly -changing parsing logic or data format, and don't want to redownload pages or -write complex error handling and persistence. +`install_cache()` takes all the same parameters as `CachedSession`. It can be temporarily disabled +with `disabled()`, and completely removed with `uninstall_cache()`: +```python +# Neither of these requests will use the cache +with requests_cache.disabled(): + requests.get('http://httpbin.org/get') -For more complex workflows, it is possible to cache different requests with different expiration times, or disable caching for a specific request completely: +requests_cache.uninstall_cache() +requests.get('http://httpbin.org/get') +``` + +**Limitations:** +Like any other utility that uses global patching, there are some scenarios where you won't want to +use this: +* In a multi-threaded or multiprocess applications +* In an application that uses other packages that extend or modify `requests.Session` +* In a package that will be used by other packages or applications -For more complex workflows, it is possible to cache different requests with different expiry times, or disable caching for a specific request completely: +### Cache Backends +Several cache backends are included, which can be selected with the `backend` parameter to +`CachedSession` or `install_cache()`: +* `'memory'` : Not persistent, just stores responses with an in-memory dict +* `'sqlite'` : [SQLite](https://www.sqlite.org) database (**default**) +* `'redis'` : [Redis](https://redis.io/) cache (requires `redis`) +* `'mongodb'` : [MongoDB](https://www.mongodb.com/) database (requires `pymongo`) +* `'dynamodb'` : [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) database (requires `boto3`) + +### Cache Expiration +By default, cached responses will be stored indefinitely. There are a number of ways you can handle +cache expiration. The simplest is using the `expire_after` param with a value in seconds: ```python -import time -import requests -import requests_cache +# Expire after 30 seconds +session = CachedSession(expire_after=30) +``` -requests_cache.install_cache('demo_cache') +Or a `timedelta`: +```python +from datetime import timedelta -# Hits the URL only 2 times -for i in range(10): - requests.get('http://httpbin.org/delay/1', expire_after=1) - time.sleep(0.2) +# Expire after 30 days +session = CachedSession(expire_after=timedelta(days=30)) ``` +You can also set expiration on a per-request basis, which will override any session settings: +```python +# Expire after 6 minutes +session.get('http://httpbin.org/get', expire_after=360) +``` + +If a per-session expiration is set but you want to temporarily disable it, use ```-1```: +```python +# Never expire +session.get('http://httpbin.org/get', expire_after=-1) +``` + +For better performance, expired responses won't be removed immediately, but will be removed +(or replaced) the next time they are accessed. To manually clear all expired responses: +```python +session.remove_expired_responses() +``` +Or, when using patching: +```python +requests_cache.remove_expired_responses() +``` + +Or, to revalidate the cache with a new expiration: +```python +session.remove_expired_responses(expire_after=360) +``` ## Related Projects If `requests-cache` isn't quite what you need, you can help make it better! See the @@ -72,8 +164,3 @@ You can also check out these other python cache projects: file-based cache built on SQLite * [aiocache](https://github.com/aio-libs/aiocache): General-purpose (not HTTP-specific) async cache backends - -## Links -- **Documentation** at [readthedocs](https://requests-cache.readthedocs.io) -- **Source code and issue tracking** at [GitHub](https://github.com/reclosedev/requests-cache) -- **Working example** at [Real Python](https://realpython.com/blog/python/caching-external-api-requests) diff --git a/docs/advanced_usage.rst b/docs/advanced_usage.rst new file mode 100644 index 0000000..cf55778 --- /dev/null +++ b/docs/advanced_usage.rst @@ -0,0 +1,349 @@ +.. _advanced-usage: + +Advanced Usage +============== +.. contents:: + :local: + +CachedSession Options +--------------------- +See :py:class:`requests_cache.CachedSession` for a full list of parameters. + +Cache Name +~~~~~~~~~~ +The ``cache_name`` parameter will be used as follows depending on the backend: + +* ``sqlite``: Cache filename, e.g ``my_cache.sqlite`` +* ``mongodb``: Database name +* ``redis``: Namespace, meaning all keys will be prefixed with ``'cache_name:'`` + +Cache Keys +~~~~~~~~~~ +The cache key is a hash created from request information, and is used as an index for cached +responses. There are a couple ways you can customize what information is used to create this key: + +* Use ``include_get_headers`` if you want headers to be included in the cache key. In other + words, this will create separate cache items for responses with different headers. +* Use ``ignored_parameters`` to exclude specific request params from the cache key. This is + useful, for example, if you request the same resource with different credentials or access + tokens. + +HTTP methods and status codes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +You can choose which request HTTP methods and response status codes you want to cache using the +parameters ``allowable_methods`` and ``allowable_codes``, respectively. By default, only GET and HEAD +requests and 200 responses are cached. Example: + + >>> from requests_cache import CachedSession + >>> + >>> session = CachedSession( + >>> allowable_methods=('GET', 'POST'), + >>> allowable_codes=(200, 418), + >>> ) + +Custom response filter +~~~~~~~~~~~~~~~~~~~~~~ +If you need more advanced behaviour for determining what to cache, you can provide a custom filtering +function via the ``filter_fn`` param. This function that takes a :py:class:`requests.Response` object +and returns a boolean indicating whether or not that response should be cached. It will be applied to +both new responses (on write) and previously cached responses (on read). Example: + + >>> from sys import getsizeof + >>> from requests_cache import CachedSession + >>> + >>> def filter_by_size(response): + >>> """Don't cache responses with a body over 1 MB""" + >>> return getsizeof(response.content) <= 1024 * 1024 + >>> + >>> session = CachedSession(filter_fn=filter_by_size) + +Cache Expiration +~~~~~~~~~~~~~~~~ +Use ``expire_after`` to specify how long responses will be cached. This can be: + +* A positive number (in seconds) +* ``-1`` (to never expire) +* A :py:class:`.timedelta` +* A :py:class:`datetime` + +This will only apply to responses cached in the current session; to apply a different expiration +to previously cached responses, see :py:meth:`remove_expired_responses`. + +Expiration can also be set on a per-URL or per request basis. The following order of precedence +is used: + +1. Per-request expiration (``expire_after`` argument for :py:meth:`.request`) +2. Per-URL expiration (``urls_expire_after`` argument for ``CachedSession``) +3. Per-session expiration (``expire_after`` argument for ``CachedSession``) + +URL Patterns +~~~~~~~~~~~~ +You can use ``expire_after_urls`` to set different expiration times for different requests, based on +URL glob patterns. This allows you to customize caching based on what you know about the resources +you're requesting. For example, you might request one resource that gets updated frequently, another +that changes infrequently, and another that never changes. Example: + + >>> urls_expire_after = { + >>> '*.site_1.com': 30, + >>> 'site_2.com/resource_1': 60 * 2, + >>> 'site_2.com/resource_2': 60 * 60 * 24, + >>> 'site_2.com/static': -1, + >>> } + +**Notes:** + +* ``urls_expire_after`` should be a dict in the format ``{'pattern': expire_after}`` +* ``expire_after`` accepts the same types as ``CachedSession.expire_after`` +* Patterns will match request **base URLs**, so the pattern ``site.com/resource/`` is equivalent to + ``http*://site.com/resource/**`` +* If there is more than one match, the first match will be used in the order they are defined +* If no patterns match a request, ``expire_after`` will be used as a default. + +Cache Inspection +---------------- +Here are some ways to get additional information out of the cache session, backend, and responses: + +Response Attributes +~~~~~~~~~~~~~~~~~~~ +The following attributes are available on responses: +* ``from_cache``: indicates if the response came from the cache +* ``created_at``: ``datetime`` of when the cached response was created or last updated +* ``expires``: ``datetime`` after which the cached response will expire +* ``is_expired``: indicates if the cached response is expired (if an old response was returned due to a request error) + +Examples: + + >>> from requests_cache import CachedSession + >>> session = CachedSession(expire_after=timedelta(days=1)) + + >>> # Placeholders are added for non-cached responses + >>> r = session.get('http://httpbin.org/get') + >>> print(r.from_cache, r.created_at, r.expires, r.is_expired) + False None None None + + >>> # Values will be populated for cached responses + >>> r = session.get('http://httpbin.org/get') + >>> print(r.from_cache, r.created_at, r.expires, r.is_expired) + True 2021-01-01 18:00:00 2021-01-02 18:00:00 False + +Cache Contents +~~~~~~~~~~~~~~ +You can use :py:meth:`.CachedSession.cache.urls` to see all URLs currently in the cache: + + >>> session = CachedSession() + >>> print(session.urls) + ['https://httpbin.org/get', 'https://httpbin.org/stream/100'] + +If needed, you can get more details on cached responses via ``CachedSession.cache.responses``, which +is a dict-like interface to the cache backend. See :py:class:`requests.Response` and +:py:class:`.CachedResponse` for a full list of attributes available. + +For example, if you wanted to to see all URLs requested with a specific method: + + >>> post_urls = [ + >>> response.url for response in session.cache.responses.values() + >>> if response.request.method == 'POST' + >>> ] + +You can also inspect ``CachedSession.cache.redirects``, which maps redirect URLs to keys of the +responses they redirect to. + +Custom Backends +--------------- +If the built-in :ref:`cache_backends` don't suit your needs and you want to create your own, you can create +subclasses of :py:class:`.BaseCache` and :py:class:`.BaseStorage`: + + >>> from requests_cache import CachedSession + >>> from requests_cache.backends import BaseCache, BaseStorage + >>> + >>> class MyCache(BaseCache): + >>> """Wrapper for higher-level cache operations""" + >>> def __init__(self, **kwargs): + >>> super().__init__(**kwargs) + >>> self.redirects = MyStorage(**kwargs) + >>> self.responses = MyStorage(**kwargs) + >>> + >>> class MyStorage(BaseStorage): + >>> """Lower-level backend storage operations""" + +You can then use your custom backend in a ``CachedSession`` with the ``backend`` parameter: + + >>> session = CachedSession(backend=MyCache()) + +Security +-------- +The python ``pickle`` module has some +`known security vulnerabilities <https://docs.python.org/3/library/pickle.html>`_, +meaning it should only be used to serialize and deserialize data you trust. Since this isn't always +possible, requests-cache can optionally use `itsdangerous <https://itsdangerous.palletsprojects.com>`_ +to add a layer of security around these operations. + +It works by signing serialized data with a secret key that you control. Then, if the data is tampered +with, the signature check fails and raises an error. To enable this behavior, first install the extra package:: + + $ pip install itsdangerous + +Then create your key and set it in your application. A common pattern for this (or any other +application credentials) is to store it wherever you store the rest of your credentials +(system keyring, password database, etc.) and set it in an environment variable. + +Then, initialize your cache with your key: + + >>> import os + >>> from requests_cache import CachedSession + >>> session = CachedSession(secret_key=os.environ['SECRET_KEY']) + >>> session.get('https://httpbin.org/get') + +You can verify that it's working by modifying the cached item (*without* your key): + + >>> session_2 = CachedSession(secret_key='a different key') + >>> cache_key = list(session_2.cache.responses.keys())[0] + >>> session_2.cache.responses[cache_key] = 'exploit!' + +Then, if you try to get that cached response again (*with* your key), you will get an error: + + >>> session.get('https://httpbin.org/get') + BadSignature: Signature b'iFNmzdUOSw5vqrR9Cb_wfI1EoZ8' does not match + + +Usage with other requests features +---------------------------------- + +Request Hooks +~~~~~~~~~~~~~ +Requests has an `Event Hook <https://requests.readthedocs.io/en/master/user/advanced/#event-hooks>`_ +system that can be used to add custom behavior into different parts of the request process. +It can be used, for example, for request throttling: + + >>> import time + >>> import requests + >>> from requests_cache import CachedSession + >>> + >>> def make_throttle_hook(timeout=1.0): + >>> """Make a request hook function that adds a custom delay for non-cached requests""" + >>> def hook(response, *args, **kwargs): + >>> if not getattr(response, 'from_cache', False): + >>> print('sleeping') + >>> time.sleep(timeout) + >>> return response + >>> return hook + >>> + >>> session = CachedSession() + >>> session.hooks['response'].append(make_throttle_hook(0.1)) + >>> # The first (real) request will have an added delay + >>> session.get('http://httpbin.org/get') + >>> session.get('http://httpbin.org/get') + +Streaming Requests +~~~~~~~~~~~~~~~~~~ +If you use `streaming requests <https://2.python-requests.org/en/master/user/advanced/#id9>`_, you +can use the same code to iterate over both cached and non-cached requests. A cached request will, +of course, have already been read, but will use a file-like object containing the content. +Example:: + + from requests_cache import CachedSession + + session = CachedSession() + for i in range(2): + r = session.get('https://httpbin.org/stream/20', stream=True) + for chunk in r.iter_lines(): + print(chunk.decode('utf-8')) + + +.. _library_compatibility: + +Usage with other requests-based libraries +----------------------------------------- +This library works by patching and/or extending ``requests.Session``. Many other libraries out there +do the same thing, making it potentially difficult to combine them. For that scenario, a mixin class +is provided, so you can create a custom class with behavior from multiple Session-modifying libraries:: + + from requests import Session + from requests_cache import CacheMixin + from some_other_lib import SomeOtherMixin + + class CustomSession(CacheMixin, SomeOtherMixin ClientSession): + """Session class with features from both requests-html and requests-cache""" + +Requests-HTML +~~~~~~~~~~~~~ +Example with `requests-html <https://github.com/psf/requests-html>`_:: + + import requests + from requests_cache import CacheMixin, install_cache + from requests_html import HTMLSession + + class CachedHTMLSession(CacheMixin, HTMLSession): + """Session with features from both CachedSession and HTMLSession""" + + session = CachedHTMLSession() + r = session.get("https://github.com/") + print(r.from_cache, r.html.links) + +Or, using the monkey-patch method:: + + install_cache(session_factory=CachedHTMLSession) + r = requests.get("https://github.com/") + print(r.from_cache, r.html.links) + +The same approach can be used with other libraries that subclass ``requests.Session``. + +Requests-futures +~~~~~~~~~~~~~~~~ +Example with `requests-futures <https://github.com/ross/requests-futures>`_: + +Some libraries, including `requests-futures`, support wrapping an existing session object. + + session = FutureSession(session=CachedSession()) + +In this case, ``FutureSession`` must wrap ``CachedSession`` rather than the other way around, since +``FutureSession`` returns (as you might expect) futures rather than response objects. +See `issue #135 <https://github.com/reclosedev/requests-cache/issues/135>`_ for more notes on this. + +Requests-mock +~~~~~~~~~~~~~ +Example with `requests-mock <https://github.com/jamielennox/requests-mock>`_: + +Requests-mock works a bit differently. It has multiple methods of mocking requests, and the +method most compatible with requests-cache is attaching its +`adapter <https://requests-mock.readthedocs.io/en/latest/adapter.html>`_ to a CachedSession:: + + import requests + from requests_mock import Adapter + from requests_cache import CachedSession + + # Set up a CachedSession that will make mock requests where it would normally make real requests + adapter = Adapter() + adapter.register_uri( + 'GET', + 'mock://some_test_url', + headers={'Content-Type': 'text/plain'}, + text='mock response', + status_code=200, + ) + session = CachedSession() + session.mount('mock://', adapter) + + session.get('mock://some_test_url', text='mock_response') + response = session.get('mock://some_test_url') + print(response.text) + +Internet Archive +~~~~~~~~~~~~~~~~ +Example with `internetarchive <https://github.com/jjjake/internetarchive>`_: + +Usage is the same as other libraries that subclass `requests.Session`:: + + from requests_cache import CacheMixin + from internetarchive.session import ArchiveSession + + class CachedArchiveSession(CacheMixin, ArchiveSession): + """Session with features from both CachedSession and ArchiveSession""" + + +Potential Issues +---------------- + +.. warning:: Version updates of ``requests``, ``urllib3`` or ``requests_cache`` itself may not be + compatible with previously cached data (see `issue #56 <https://github.com/reclosedev/requests-cache/issues/56>`_). + To prevent this, you can use a virtualenv and pin your dependency versions. diff --git a/docs/api.rst b/docs/api.rst index ad13a49..cae87cf 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -1,52 +1,41 @@ API === +This section covers all the public interfaces of ``requests-cache`` -This part of the documentation covers all the interfaces of `requests-cache` - - -Public api +Public API ---------- +.. Explicitly show inherited method docs on CachedSession instead of CachedMixin +.. autoclass:: requests_cache.core.CachedSession + :members: send, request, cache_disabled, remove_expired_responses + :show-inheritance: + +.. autoclass:: requests_cache.core.CacheMixin .. automodule:: requests_cache.core - :members: + :members: + :exclude-members: CachedSession, CacheMixin ----------------------------------------------- +.. automodule:: requests_cache.response + :members: -.. _cache_backends: +---------------------------------------------- -Cache backends +Cache Backends -------------- - .. automodule:: requests_cache.backends.base :members: -.. _backends_sqlite: - .. automodule:: requests_cache.backends.sqlite :members: -.. _backends_mongo: - .. automodule:: requests_cache.backends.mongo :members: -.. _backends_redis: - -.. automodule:: requests_cache.backends.redis +.. automodule:: requests_cache.backends.gridfs :members: ----------------------------------------------- - -Internal modules which can be used outside ------------------------------------------- - -.. _backends_dbdict: - -.. automodule:: requests_cache.backends.storage.dbdict - :members: - -.. automodule:: requests_cache.backends.storage.mongodict +.. automodule:: requests_cache.backends.redis :members: -.. automodule:: requests_cache.backends.storage.redisdict +.. automodule:: requests_cache.backends.dynamodb :members: diff --git a/docs/conf.py b/docs/conf.py index d152386..f2c2ae8 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -21,19 +21,21 @@ templates_path = ['_templates'] # Sphinx extension modules extensions = [ 'sphinx.ext.autodoc', + 'sphinx.ext.autosectionlabel', 'sphinx.ext.intersphinx', 'sphinx.ext.napoleon', 'sphinx.ext.viewcode', 'sphinx_autodoc_typehints', 'sphinx_copybutton', # 'sphinxcontrib.apidoc', - # 'm2r2', + 'm2r2', ] # Enable automatic links to other projects' Sphinx docs intersphinx_mapping = { 'python': ('https://docs.python.org/3', None), 'requests': ('https://requests.readthedocs.io/en/master/', None), + 'urllib3': ('http://urllib3.readthedocs.org/en/latest', None), } # Enable Google-style docstrings diff --git a/docs/contributing.rst b/docs/contributing.rst new file mode 100644 index 0000000..4fc5016 --- /dev/null +++ b/docs/contributing.rst @@ -0,0 +1 @@ +.. mdinclude:: ../CONTRIBUTING.md diff --git a/docs/history.rst b/docs/history.rst new file mode 100644 index 0000000..d26e5be --- /dev/null +++ b/docs/history.rst @@ -0,0 +1 @@ +.. mdinclude:: ../HISTORY.md diff --git a/docs/index.rst b/docs/index.rst index ad390d1..78d1d14 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,26 +1,21 @@ -.. requests-cache documentation master file, created by - sphinx-quickstart on Sun Apr 08 20:51:24 2012. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - Requests-cache documentation ============================ -`Requests-cache <http://pypi.python.org/pypi/requests-cache>`_ -is a transparent persistent cache for requests_ (version >= 1.1.0) library. - -Source code and issue tracking can be found at -`GitHub <https://github.com/reclosedev/requests-cache>`_. - +.. Include Readme contents (except for the link to readthedocs, since we're already on readthedocs!) +.. mdinclude:: ../README.md + :end-line: 17 -**Contents:** +.. mdinclude:: ../README.md + :start-line: 19 .. toctree:: - :maxdepth: 2 - - user_guide - api + :caption: Contents + :maxdepth: 2 + advanced_usage + api + contributing + history Indices and tables ================== @@ -28,5 +23,3 @@ Indices and tables * :ref:`genindex` * :ref:`modindex` * :ref:`search` - -.. _requests: http://docs.python-requests.org/
\ No newline at end of file diff --git a/docs/user_guide.rst b/docs/user_guide.rst index 23d4d45..7ab8737 100644 --- a/docs/user_guide.rst +++ b/docs/user_guide.rst @@ -1,228 +1,5 @@ -.. _user_guide: +.. This is just a placeholder in case anyone had this page bookmarked -User guide +User Guide ========== - - -Installation ------------- - -Install with pip_ or easy_install_:: - - pip install --upgrade requests-cache - -or download latest version from version control:: - - git clone git://github.com/reclosedev/requests-cache.git - cd requests-cache - python setup.py install - - -.. warning:: Version updates of ``requests``, ``urllib3`` or ``requests_cache`` itself can break existing - cache database (see https://github.com/reclosedev/requests-cache/issues/56 ). - So if your code relies on cache, or is expensive in terms of time and traffic, please be sure to use - something like ``virtualenv`` and pin your requirements. - -.. _pip: http://pypi.python.org/pypi/pip/ -.. _easy_install: http://pypi.python.org/pypi/setuptools - -Usage ------ - -.. currentmodule:: requests_cache.core - -There is two ways of using :mod:`requests_cache`: - - - Using :class:`CachedSession` instead ``requests.Session`` - - Monkey patching ``requests`` to use :class:`CachedSession` by default - -Monkey-patching allows to add caching to existent program by adding just two lines: - -Import :mod:`requests_cache` and call :func:`install_cache` -:: - - import requests - import requests_cache - - requests_cache.install_cache() - -And you can use ``requests``, all responses will be cached transparently! - -For example, following code will take only 1-2 seconds instead 10:: - - for i in range(10): - requests.get('http://httpbin.org/delay/1') - -Cache can be configured with some options, such as cache filename, backend -(sqlite, mongodb, redis, memory), expiration time, etc. E.g. cache stored in sqlite -database (default format) named ``'test_cache.sqlite'`` with expiration -set to 300 seconds can be configured as:: - - requests_cache.install_cache('test_cache', backend='sqlite', expire_after=300) - -.. seealso:: - Full list of options can be found in - :func:`requests_cache.install_cache() <requests_cache.core.install_cache>` reference - - -Transparent caching is achieved by monkey-patching ``requests`` library -It is possible to uninstall this patch with :func:`requests_cache.uninstall_cache() <requests_cache.core.uninstall_cache>`. - -Also, you can use :func:`requests_cache.disabled() <requests_cache.core.disabled>` -context manager for temporary disabling caching:: - - with requests_cache.disabled(): - print(requests.get('http://httpbin.org/ip').text) - - -If ``Response`` is taken from cache, ``from_cache`` attribute will be ``True``: -:: - - >>> import requests - >>> import requests_cache - >>> requests_cache.install_cache() - >>> requests_cache.clear() - >>> r = requests.get('http://httpbin.org/get') - >>> r.from_cache - False - >>> r = requests.get('http://httpbin.org/get') - >>> r.from_cache - True - -It can be used, for example, for request throttling with help of ``requests`` hook system:: - - import time - import requests - import requests_cache - - def make_throttle_hook(timeout=1.0): - """ - Returns a response hook function which sleeps for `timeout` seconds if - response is not cached - """ - def hook(response, *args, **kwargs): - if not getattr(response, 'from_cache', False): - print('sleeping') - time.sleep(timeout) - return response - return hook - - if __name__ == '__main__': - requests_cache.install_cache('wait_test') - requests_cache.clear() - - s = requests_cache.CachedSession() - s.hooks['response'].append(make_throttle_hook(0.1)) - s.get('http://httpbin.org/delay/get') - s.get('http://httpbin.org/delay/get') - -.. seealso:: `example.py <https://github.com/reclosedev/requests-cache/blob/master/example.py>`_ - -.. note:: requests_cache prefetchs response content, be aware if your code uses streaming requests. - - -.. _request_caching: - -Request caching ---------------- - -If you want to cache on a per request basis, all requests support the ``expire_after`` keyword argument, which takes an expiration time in seconds:: - - requests.get('https://httpbin.org/get', expire_after=10) - -Alternatively, ``expire_after`` accepts :class:`None`, ``'default'``, or a :class:`~datetime.timedelta`. :class:`None` enables infinite caching, ``'default'`` uses the expiry defined in :func:`install_cache`, and the timedelta is used as is. - -.. note:: Internally, numbers are converted to :class:`~datetime.timedelta` instances as well. - -.. seealso:: `example_request.py <https://github.com/reclosedev/requests-cache/blob/master/example_request.py>`_ - - -.. _persistence: - -Persistence ------------ - -:mod:`requests_cache` designed to support different backends for persistent storage. -By default it uses ``sqlite`` database. Type of storage can be selected with ``backend`` argument of :func:`install_cache`. - -List of available backends: - -- ``'sqlite'`` - sqlite database (**default**) -- ``'memory'`` - not persistent, stores all data in Python ``dict`` in memory -- ``'mongodb'`` - (**experimental**) MongoDB database (``pymongo < 3.0`` required) -- ``'redis'`` - stores all data on a redis data store (``redis`` required) - -You can write your own and pass instance to :func:`install_cache` or :class:`CachedSession` constructor. -See :ref:`cache_backends` API documentation and sources. - -.. _expiration: - -Expiration ----------- - -If you are using cache with ``expire_after`` parameter set, responses are removed from the storage only when the same -request is made. Since the store sizes can get out of control pretty quickly with expired items -you can remove them using :func:`remove_expired_responses` -or :meth:`BaseCache.remove_old_entries(created_before) <requests_cache.backends.base.BaseCache.remove_old_entries>`. -:: - - expire_after = timedelta(hours=1) - requests_cache.install_cache(expire_after=expire_after) - ... - requests_cache.core.remove_expired_responses() - # or - remove_old_entries.get_cache().remove_old_entries(datetime.utcnow() - expire_after) - # when used as session - session = CachedSession(..., expire_after=expire_after) - ... - session.cache.remove_old_entries(datetime.utcnow() - expire_after) - - -For more information see :doc:`API reference <api>`. - -Usage with other libraries that modify requests.Session -------------------------------------------------------- -This library works by patching ``requests.Session``. Many other libraries out there do the same -thing, making it difficult to combine them. For that case, a mixin class is provided, so you can -create a custom class with behavior from multiple Session-modifying libraries:: - - from requests import Session - from requests_cache import CacheMixin - from some_other_lib import SomeOtherMixin - - class CustomSession(CacheMixin, SomeOtherMixin ClientSession): - """Session class with features from both requests-html and requests-cache""" - - -Example with `requests-html <https://github.com/psf/requests-html>`_:: - - import requests - from requests_cache import CacheMixin, install_cache - from requests_html import HTMLSession - - class CachedHTMLSession(CacheMixin, HTMLSession): - """Session with features from both CachedSession and HTMLSession""" - - session = CachedHTMLSession() - r = session.get("https://github.com/") - print(r.from_cache, r.html.links) - -Or, using the monkey-patch method:: - - install_cache(session_factory=CachedHTMLSession) - r = requests.get("https://github.com/") - print(r.from_cache, r.html.links) - -The same approach can be used with other libraries that subclass ``requests.Session``. - -Example with `requests-mock <https://github.com/jamielennox/requests-mock>`_:: - - import requests - from requests_mock import Mocker - from requests_cache import CachedSession - - session = CachedSession() - with Mocker(session=session) as m: - m.get('http://test.com', text='mock_response') - response = session.get('http://test.com') - print(response.text) +This page has moved. Please see :ref:`usage` and :ref:`advanced-usage` sections. diff --git a/example.py b/example.py deleted file mode 100644 index f1e8211..0000000 --- a/example.py +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env python -import time - -import requests - -import requests_cache - -requests_cache.install_cache('example_cache') - - -def main(): - # Once cached, delayed page will be taken from cache - # redirects also handled - for i in range(5): - requests.get('http://httpbin.org/delay/2') - r = requests.get('http://httpbin.org/redirect/5') - print(r.text) - - # And if we need to get fresh page or don't want to cache it? - with requests_cache.disabled(): - print(requests.get('http://httpbin.org/ip').text) - - # Debugging info about cache - print(requests_cache.get_cache()) - - -if __name__ == "__main__": - t = time.time() - main() - print('Elapsed: %.3f seconds' % (time.time() - t)) diff --git a/example_request.py b/example_request.py deleted file mode 100644 index ead525d..0000000 --- a/example_request.py +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env python -import time - -import requests - -import requests_cache - -requests_cache.install_cache('example_cache', backend='sqlite') - - -def main(): - # Normal caching forever - response = requests.get('https://httpbin.org/get') - assert not response.from_cache - response = requests.get('https://httpbin.org/get') - assert response.from_cache - - # Caching with expiration - requests_cache.clear() - response = requests.get('https://httpbin.org/get', expire_after=1) - response = requests.get('https://httpbin.org/get') - assert response.from_cache - # After > 1 second, it's cached value is expired - time.sleep(1.2) - response = requests.get('https://httpbin.org/get') - assert not response.from_cache - - -if __name__ == "__main__": - t = time.perf_counter() - main() - print('Elapsed: %.3f seconds' % (time.perf_counter() - t)) diff --git a/examples/basic_usage.py b/examples/basic_usage.py new file mode 100644 index 0000000..3888681 --- /dev/null +++ b/examples/basic_usage.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python +"""A Simple example using requests-cache with httpbin""" +import time + +from requests_cache import CachedSession + + +def main(): + session = CachedSession('example_cache', backend='sqlite') + + # The real request will only be made once; afterward, the cached response is used + for i in range(5): + response = session.get('http://httpbin.org/get') + + # This is more obvious when calling a slow endpoint + for i in range(5): + response = session.get('http://httpbin.org/delay/2') + + # Caching can be disabled if we want to get a fresh page and not cache it + with session.cache_disabled(): + print(session.get('http://httpbin.org/ip').text) + + # Get some debugging info about the cache + print(session.cache) + print('Cached URLS:', session.cache.urls) + + +if __name__ == "__main__": + t = time.time() + main() + print('Elapsed: %.3f seconds' % (time.time() - t)) diff --git a/examples/expiration.py b/examples/expiration.py new file mode 100644 index 0000000..4afa5fa --- /dev/null +++ b/examples/expiration.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python +"""An example of setting expiration for individual requests""" +import time + +from requests_cache import CachedSession + + +def main(): + session = CachedSession('example_cache', backend='sqlite') + + # By default, cached responses never expire + response = session.get('https://httpbin.org/get') + assert not response.from_cache + response = session.get('https://httpbin.org/get') + assert response.from_cache + assert not response.expires + + # We can set default expiration for the session using expire_after + session = CachedSession('example_cache', backend='sqlite', expire_after=60) + session.cache.clear() + response = session.get('https://httpbin.org/get') + response = session.get('https://httpbin.org/get') + print('Expiration time:', response.expires) + + # This can also be overridden for individual requests + session.cache.clear() + response = session.get('https://httpbin.org/get', expire_after=1) + response = session.get('https://httpbin.org/get') + assert response.from_cache + print('Expiration time:', response.expires) + + # After 1 second, the cached value will expired + time.sleep(1.2) + assert response.is_expired + response = session.get('https://httpbin.org/get') + assert not response.from_cache + + +if __name__ == "__main__": + t = time.perf_counter() + main() + print('Elapsed: %.3f seconds' % (time.perf_counter() - t)) diff --git a/examples/session_patch.py b/examples/session_patch.py new file mode 100644 index 0000000..8ae900b --- /dev/null +++ b/examples/session_patch.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python +"""The same as `basic_usage.py`, but using global session patching""" +import time + +import requests +import requests_cache + +# After installation, all requests functions and Session methods will be cached +requests_cache.install_cache('example_cache', backend='sqlite') + + +def main(): + # The real request will only be made once; afterward, the cached response is used + for i in range(5): + response = requests.get('http://httpbin.org/get') + + # This is more obvious when calling a slow endpoint + for i in range(5): + response = requests.get('http://httpbin.org/delay/2') + + # Caching can be disabled if we want to get a fresh page and not cache it + with requests_cache.disabled(): + print(requests.get('http://httpbin.org/ip').text) + + # Get some debugging info about the cache + print(requests_cache.get_cache()) + print('Cached URLS:', requests_cache.get_cache().urls) + + # Uninstall to remove caching from all requests functions + requests_cache.uninstall_cache() + + +if __name__ == "__main__": + t = time.time() + main() + print('Elapsed: %.3f seconds' % (time.time() - t)) diff --git a/pyproject.toml b/pyproject.toml index b2bafe1..aa18291 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,7 +16,7 @@ source = ['requests_cache'] profile = "black" line_length = 100 skip_gitignore = true -skip = ['requests_cache/__init__.py'] +skip = ['examples/', 'requests_cache/__init__.py'] known_first_party = ['test'] # Things that are common enough they may as well be grouped with stdlib imports extra_standard_library = ['pytest', 'setuptools'] @@ -3,9 +3,9 @@ universal = 1 [metadata] description = Persistent cache for requests library -long_description = file: README.rst, HISTORY.rst -long_description_content_type = text/x-rst -keywords = requests, cache, persistence +long_description = file: README.md, HISTORY.md +long_description_content_type = text/markdown +keywords = requests, http, cache, persistence, sqlite, redis, mongodb, gridfs, dynamodb license = BSD License license_files = LICENSE python_requires = >=3.6 |