diff options
Diffstat (limited to 'README.md')
-rw-r--r-- | README.md | 180 |
1 files changed, 43 insertions, 137 deletions
@@ -8,164 +8,70 @@ [![Code Shelter](https://www.codeshelter.co/static/badges/badge-flat.svg)](https://www.codeshelter.co/) ## Summary -**requests-cache** is a transparent persistent HTTP cache for the python [requests](http://python-requests.org) -library. It is especially useful for web scraping, consuming REST APIs, slow or rate-limited -sites, or any other scenario in which you're making lots of requests that are likely to be sent -more than once. - -Several storage backends are included: **SQLite**, **Redis**, **MongoDB**, and **DynamoDB**. +**requests-cache** is a transparent, persistent HTTP cache for the python [requests](http://python-requests.org) +library. It's a convenient tool to use with web scraping, consuming REST APIs, slow or rate-limited +sites, or any other scenario in which you're making lots of requests that are expensive and/or +likely to be sent more than once. See full project documentation at: https://requests-cache.readthedocs.io -## Installation -Install with pip: +## Features +* **Ease of use:** Use as a [drop-in replacement](https://requests-cache.readthedocs.io/en/latest/api.html#sessions) + for `requests.Session`, or [install globally](https://requests-cache.readthedocs.io/en/latest/user_guide.html#patching) + to add caching to all `requests` functions +* **Customization:** Works out of the box with zero config, but with plenty of options available + for customizing cache + [expiration](https://requests-cache.readthedocs.io/en/latest/user_guide.html#cache-expiration) + and other [behavior](https://requests-cache.readthedocs.io/en/latest/user_guide.html#cache-options) +* **Persistence:** Includes several [storage backends](https://requests-cache.readthedocs.io/en/latest/user_guide.html#cache-backends): + SQLite, Redis, MongoDB, and DynamoDB. +* **Compatibility:** Can be used alongside + [other popular libraries based on requests](https://requests-cache.readthedocs.io/en/latest/advanced_usage.html#library-compatibility) + +# Quickstart +First, install with pip: ```bash pip install requests-cache ``` -**Requirements:** -* Requires python 3.6+. -* You may need additional dependencies depending on which backend you want to use. To install with - extra dependencies for all supported backends: - - ```bash - pip install requests-cache[backends] - ``` - -**Optional Setup Steps:** -* See [Security](https://requests-cache.readthedocs.io/en/latest/security.html) for recommended - setup steps for more secure cache serialization. -* See [Contributing Guide](https://requests-cache.readthedocs.io/en/latest/contributing.html) - for setup info for local development. - -## General Usage -There are two main ways of using requests-cache: -* [Sessions](https://requests-cache.readthedocs.io/en/latest/api.html#sessions): - Use `requests_cache.CachedSession` in place of - [requests.Session](https://requests.readthedocs.io/en/master/user/advanced/#session-objects) (recommended) -* [Patching](https://requests-cache.readthedocs.io/en/latest/api.html#patching): - Globally patch `requests` using `requests_cache.install_cache()`. - -### Sessions -The `CachedSession` class is a drop-in replacement for `requests.Session` that adds caching features. - -Basic example: -```python -from requests_cache import CachedSession - -session = CachedSession('demo_cache', backend='sqlite') -for i in range(100): - session.get('http://httpbin.org/delay/1') -``` -The URL in this example adds a delay of 1 second, but all 100 requests will complete in just over 1 -second. The response will be fetched once, saved to `demo_cache.sqlite`, and subsequent requests -will return the cached response near-instantly. +Next, use [requests_cache.CachedSession](https://requests-cache.readthedocs.io/en/latest/api.html#sessions) +to send and cache requests. To quickly demonstrate how to use it: -### Patching -Using `requests_cache.install_cache()` will add caching to all `requests` functions: +**This takes ~1 minute:** ```python import requests -import requests_cache -requests_cache.install_cache() -requests.get('http://httpbin.org/get') session = requests.Session() -session.get('http://httpbin.org/get') -``` - -`install_cache()` takes all the same parameters as `CachedSession`. It can be temporarily disabled -with `disabled()`, and completely removed with `uninstall_cache()`: -```python -# Neither of these requests will use the cache -with requests_cache.disabled(): - requests.get('http://httpbin.org/get') - -requests_cache.uninstall_cache() -requests.get('http://httpbin.org/get') -``` - -**Limitations:** - -Like any other utility that uses global patching, there are some scenarios where you won't want to -use this: -* In a multi-threaded or multiprocess application -* In an application that uses other packages that extend or modify `requests.Session` -* In a package that will be used by other packages or applications - -### Cache Backends -Several [cache backends](https://requests-cache.readthedocs.io/en/latest/modules/requests_cache.backends.html) -are included, which can be selected with the `backend` parameter to `CachedSession` or `install_cache()`: - -* `'memory'` : Not persistent, just stores responses with an in-memory dict -* `'sqlite'` : [SQLite](https://www.sqlite.org) database (**default**) -* `'redis'` : [Redis](https://redis.io/) cache (requires `redis`) -* `'mongodb'` : [MongoDB](https://www.mongodb.com/) database (requires `pymongo`) -* `'dynamodb'` : [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) database (requires `boto3`) - -### Cache Expiration -By default, cached responses will be stored indefinitely. There are a number of ways you can handle -cache expiration. The simplest is using the `expire_after` param with a value in seconds: -```python -# Expire after 30 seconds -session = CachedSession(expire_after=30) +for i in range(60): + session.get('http://httpbin.org/delay/1') ``` -Or a `timedelta`: +**This takes ~1 second:** ```python -from datetime import timedelta - -# Expire after 30 days -session = CachedSession(expire_after=timedelta(days=30)) -``` +import requests_cache -You can also set expiration on a per-request basis, which will override any session settings: -```python -# Expire after 6 minutes -session.get('http://httpbin.org/get', expire_after=360) +session = requests_cache.CachedSession('demo_cache') +for i in range(60): + session.get('http://httpbin.org/delay/1') ``` -If a per-session expiration is set but you want to temporarily disable it, use `-1`: -```python -# Never expire -session.get('http://httpbin.org/get', expire_after=-1) -``` +The URL in this example adds a delay of 1 second, simulating a slow or rate-limited website. +With caching, the response will be fetched once, saved to `demo_cache.sqlite`, and subsequent +requests will return the cached response near-instantly. -For better performance, expired responses won't be removed immediately, but will be removed -(or replaced) the next time they are accessed. To manually clear all expired responses: +If you don't want to manage a session object, requests-cache can also be installed globally: ```python -session.remove_expired_responses() -``` -Or, when using patching: -```python -requests_cache.remove_expired_responses() +requests_cache.install_cache('demo_cache') +requests.get('http://httpbin.org/delay/1') ``` -Or, to revalidate the cache with a new expiration: -```python -session.remove_expired_responses(expire_after=360) -``` +## Next Steps +To find out more about what you can do with requests-cache, see: -## More Features & Examples -* You can find a working example at Real Python: +* The + [User Guide](https://requests-cache.readthedocs.io/en/latest/user_guide.html) and + [Advanced Usage](https://requests-cache.readthedocs.io/en/latest/advanced_usage.html) sections +* A working example at Real Python: [Caching External API Requests](https://realpython.com/blog/python/caching-external-api-requests) -* There are some additional examples in the [examples/](https://github.com/reclosedev/requests-cache/tree/master/examples) folder -* See [Advanced Usage](https://requests-cache.readthedocs.io/en/latest/advanced_usage.html) for - details on customizing cache behavior and other features beyond the basics. - -## Related Projects -If `requests-cache` isn't quite what you need, you can help make it better! See the -[Contributing Guide](https://requests-cache.readthedocs.io/en/latest/contributing.html) -for details. - -You can also check out these other python cache projects: - -* [CacheControl](https://github.com/ionrock/cachecontrol): An HTTP cache for `requests` that caches - according to HTTP headers -* [diskcache](https://github.com/grantjenks/python-diskcache): A general-purpose (not HTTP-specific) - file-based cache built on SQLite -* [aiohttp-client-cache](https://github.com/JWCook/aiohttp-client-cache): An async HTTP cache for - `aiohttp`, based on `requests-cache` -* [aiohttp-cache](https://github.com/cr0hn/aiohttp-cache): A server-side async HTTP cache for the - `aiohttp` web server -* [aiocache](https://github.com/aio-libs/aiocache): General-purpose (not HTTP-specific) async cache - backends +* More examples in the + [examples/](https://github.com/reclosedev/requests-cache/tree/master/examples) folder |