summaryrefslogtreecommitdiff
path: root/t
Commit message (Collapse)AuthorAgeFilesLines
* proxy: add tests for ustatsFei Hu2023-05-112-0/+258
|
* proxy: add more tests around request APIs to proxyunits.tFei Hu2023-05-092-20/+123
|
* proxy: add proxy_test() and test scenarios to proxyunits.tFei Hu2023-05-082-28/+213
|
* proxy: print info of test groups in proxyunits.t.Fei Hu2023-05-081-0/+26
|
* proxy: add check_sanity() to check proxy and backends at boarder sections.Fei Hu2023-05-072-16/+39
| | | | Migrate check_version() to check_sanity() to make sure both proxy and backends buffers are clean between sections.
* proxy: comment proxyunits.t test moredormando2023-04-281-0/+19
|
* proxy: remove redundancy in test codedormando2023-04-282-102/+41
| | | | re-accepting backends was really common.
* proxy: return 'readvalidate' on be read timeoutdormando2023-04-281-1/+1
| | | | | | | New backend connections returned 'conntimeout' whether it timed out establishing the TCP connection or if it died waiting for the "version\r\n" response. Now gives a 'readvalidate' if it's already properly connected.
* proxy: let tests use unix socket for proxy daemondormando2023-04-285-8/+7
| | | | | | | | | | | A long sleep in the unix startup code made backends hit the connection timeout before the backends were configured. Make all the proxy tests use the unix socket instead of listening on a hardcoded port. Proxy code is completely equivalent from the client standpoint. This fix should make the whole test suite run a bit faster too.
* proxy: fix t/proxy.tdormando2023-04-191-1/+1
| | | | | | Apparently I don't typically run this one much. I think it should be deprecated for the newer style used in proxyunits.t/etc, but needs to be a concerted effort.
* proxy: fix flaky test in proxyconfig.tdormando2023-04-172-6/+7
| | | | | | | | | | | | The connect timeout won't fire when blocking a backend from connecting in these tests; it will connect, send a version command to validate, then time out on read. With the read timeout set to 0.1 it would sometimes fail before the restart finished, clogging log lines and causing test failures. Now we wait for the watcher and remove a sleep, with a longer read timeout.
* proxy: fix backend leak with 2-arg mcp.pool()dormando2023-04-101-0/+6
| | | | | | | | | | | | When mcp.pool() is called in its two argument form, ie: mcp.pool({b1, b2}, { foo = bar }), backend objects would not be properly cached internally, causing objects to leak. Further, it was settings the objects into the cache table indexed by the object itself, so they would not be cleaned up by garbage collection. Bug was introduced as part of 6442017c (allow workers to run IO optionally)
* proxy: send CLIENT_ERROR when necessarystagingdormando2023-04-031-0/+5
| | | | | A few code paths were returning SERVER_ERROR (a retryable error) when it should have been CLIENT_ERROR (bad protocol syntax).
* proxy: overhaul backend error handlingdormando2023-03-262-8/+147
| | | | | | | | | | Cleans up logic around response handling in general. Allows returning server-sent error messages upstream for handling. In general SERVER_ERROR means we can keep the connection to the backend. The rest of the errors are protocol errors, and while some are perfectly safe to whitelist, clients should not be causing those sorts of errors and we should cycle the backend regardless.
* proxy: fix reversal of pipelined backend queriesdormando2023-03-261-10/+6
| | | | | | | | | | | | | | | | | | If a client sends multiple requests in the same packet, the proxy would reverse the requests before sending them to the backend. They would return to client in the correct order because top level responses are sent in the order they were created. In practice I guess this is rarely noticed. If a client sends a series of commands where the first one generates a syntax error, all prior commands would still succeed. It would also trip people up if they test pipelining commands as read-your-write would fail as the write gets ordered after the read. Did run into this before, but I thought it was just the ascii multiget code reversing keys, which would be harmless as the whole command has to complete regardless of key order.
* proxy: add request and buffer memory limitsdormando2023-03-262-5/+228
| | | | | | | | | | | | | | | | | | Adds: mcp.active_req_limit(count) mcp.buffer_memory_limit(kilobytes) Divides by the number of worker threads and creates a per-worker-thread limit for the number of concurrent proxy requests, and how many bytes used specifically for value bytes. This does not represent total memory usage but will be close. Buffer memory for inbound set requests is not accounted for until after the object has been read from the socket; to be improved in a future update. This should be fine unless clients send just the SET request and then hang without sending further data. Limits should be live-adjustable via configuration reloads.
* proxy: restrict functions for lua config vs routedormando2023-03-263-7/+6
| | | | | | | | | | Also changes the way the global context and thread contexts are fetched from lua; via the VM extra space instead of upvalues, which is a little faster and more universal. It was always erroneous to run a lot of the config functions from routes and vice versa, but there was no consistent strictness so users could get into trouble.
* proxy: add conntimeout errordormando2023-03-211-1/+1
| | | | | use a specific error when timeouts happen during connection stage vs read/write stage. it even had a test!
* log: Add a new watcher to watch for deletions.Hemal Shah2023-03-151-1/+48
| | | | | | | `watch deletions`: would log all keys which are deleted using either `delete` or `md` command. The log line would contain the command used, the key, the clsid and size of the deleted item. Items which result in delete miss or are marked as stale wouldn't show up in the logs
* meta: N flag changes append/prepend. ms s flag.dormando2023-03-111-0/+37
| | | | | | | | | Sending 's' flag to metaset now returns the size of the item stored. Useful if you want to know how large an append/prepended item now is. If the 'N' flag is supplied while in append/prepend mode, allows autovivifying (with exptime supplied from N) for append/prepend style keys that don't need headers created first.
* proxy: repair t/proxyconfig.tdormando2023-03-091-0/+1
| | | | somehow missed from earlier change with marking dead backends.
* core: fix another dtrace compilation issue1.6.19dormando2023-03-081-4/+4
|
* proxy: mcp.internal fixes and testsdormando2023-03-062-0/+239
| | | | | | | | | | | - Refcount leak on sets - Move the response elapsed timer back closer to when the response was processed as to not clobber the wrong IO object data - Restores error messages from set/ms - Adds start of unit tests Requests will look like they run a tiiiiny bit faster than they do, but I need to get the elapsed time there for a later change.
* proxy: more await unit testsdormando2023-03-012-1/+113
| | | | test FASTGOOD and some set scenarios
* proxy: fix trailingdata error with ascii multigetdormando2023-02-281-0/+15
| | | | | | | | One of the side effects of pre-warming all of the tests I did with multiget, and not having done a second round on the unit tests, is that we somehow never tried an ascii multiget against a damn miss. Easy to test, easy to fix.
* proxy: allow workers to run IO optionallydormando2023-02-242-0/+114
| | | | | | | | | | | | | | | | | `mcp.pool(p, { dist = etc, iothread = true }` By default the IO thread is not used; instead a backend connection is created for each worker thread. This can be overridden by setting `iothread = true` when creating a pool. `mcp.pool(p, { dist = etc, beprefix = "etc" }` If a `beprefix` is added to pool arguments, it will create unique backend connections for this pool. This allows you to create multiple sockets per backend by making multiple pools with unique prefixes. There are legitimate use cases for sharing backend connections across different pools, which is why that is the default behavior.
* proxy: redo libevent handling codedormando2023-02-221-4/+6
| | | | | | | | | | | | | | | | | | The event handling code was unoptimized and temporary; it was slated for a rewrite for performance and non-critical bugs alone. However the old code may be causing critical bugs so it's being rewritten now. Fixes: - backend disconnects are detected immediately instead of on the next time they are used. - backend reconnects happen _after_ the retry timeout, not before - use a persistent read handler and a temporary write handler to avoid constantly calling epoll_ctl syscalls for potential performance boost. Updated some tests for proxyconfig.t as it was picking up the disconnects immediately. Unrelated to a timing issue I resolved to the benchmark.
* proxy: fix "missingend" error on reading responsesdormando2023-02-171-0/+55
| | | | | | | | | | | | | | If the backend handler reads an incomplete response from the network, it changes state to wait for more data. The want_read state was considering the data completed if "data read" was bigger than "value length", but it should have been "value + result line". This means if the response buffer landed in a bullseye where it has read more than the size of the value but less than the total size of the request (typically a span of 200 bytes or less), it would consider the request complete and look for the END\r\n marker. This change has been... here forever.
* proxy: add read buffer data to backend errorsdormando2023-02-151-0/+26
| | | | | | Errors like "trailing data" or "missingend" or etc are only useful if you're in a debugger and can break and inspect. This adds detail in uriencoding into the log message when applicable.
* proxy: fix partial responses on backend timeoutsdormando2023-02-142-6/+23
| | | | | | | Response object error conditions were not being checked before looking at the response buffer. If a response was partially filled then the backend timed out, a partial response could be sent intead of the proper backend error.
* proxy: add mcp.backend(t) for more overridesdormando2023-02-012-2/+53
| | | | | | | | | | ie: local b1 = mcp.backend({ label = "b1", host = "127.0.0.1", port = 11511, connecttimeout = 1, retrytimeout = 0.5, readtimeout = 0.1, failurelimit = 11 }) ... to allow for overriding connect/retry/etc tunables on a per-backend basis. If not passed in the global settings are used.
* proxy: add mcp.await_logerrors()dormando2023-01-272-0/+39
| | | | | | | Logs any backgrounded requests that resulted in an error. Note that this may be a temporary interface, and could be deprecated in the future.
* proxy: new integration tests.dormando2023-01-255-0/+933
| | | | | | | | uses mocked backend servers so we can test: - end to end client to backend proxying - lua API functions - configuration reload - various error conditions
* core: remove *conn object from cache commandsdormando2023-01-112-2/+6
| | | | | | | | | We want to start using cache commands in contexts without a client connection, but the client object has always been passed to all functions. In most cases we only need the worker thread (LIBEVENT_THREAD *t), so this change adjusts the arguments passed in.
* proxy: add mcp.AWAIT_BACKGROUNDdormando2022-12-011-4/+5
| | | | | | mcp.await(request, pools, 0, mcp.AWAIT_BACKGROUND) will, instead of waiting on any request to return, simply return an empty table as soon as the background requests are dispatched.
* meta: allow mg without flags + reflect O/k on ENdormando2022-10-201-1/+1
| | | | | | | | "mg" required at least one flag. now "mg key" returns a bare HD on hit if you don't care about the value. HD modes would reflect O and k flags in the response, but EN didn't. This is now fixed for full coverage.
* meta: meta arithmetic command had excess spacesdormando2022-10-201-5/+5
| | | | for HD/NF/etc responses but not VA.
* meta: remove excess spaces from meta responsesdormando2022-10-201-1/+9
| | | | was returning "HD \r\n" and "HD Oetc\r\n" - not to protocol spec.
* Find perl via /usr/bin/env instead of directlyDavid CARLIER2022-08-2583-83/+83
| | | | | At least FreeBSD has perl in /usr/local/bin/perl and no symlink by default.
* proxy: mcp.attach(CMD, r, "tag")dormando2022-08-241-0/+3
| | | | | | | | allows using tagged listeners (ex; `-l tag[test]:127.0.0.1:11212`) to select a top level route for a function. expects there to not be dozens of listeners, but for a handful will be faster than a hash table lookup.
* proxy: req:flag_token("F", "Freplacement")dormando2022-08-091-0/+2
| | | | | | returns (exists, previous_token) optional second argument will replace the flag/token with supplied flag/token, or nothing if "" is passed.
* proxy: add req:flag_token("F")dormando2022-08-031-0/+4
| | | | | | function accepts a flag, returns (bool, token|nil). bool indicates if the flag exists, and if the flag has a token it is returned instead of nil as the second value.
* proxy: mcp.response code and rline APIdormando2022-08-031-0/+15
| | | | | | | | | Adds resp:code(), which returns code you can compare with mcp.MCMC_CODE_* Adds resp:line(), which returns the raw response line after the command. This can be used in lua while the flag/token API is missing for the response object.
* proxy: add r:has_flag(), fix r:token() lengthdormando2022-08-031-0/+14
| | | | | | | | | Add mcp.request function for quick checking if a flag exists in a request string. Also updates internal code for checking the length of a token to use the endcap properly, and uses that for the r:token(n) requets as well, which fixes a subtle bug of the token length being too long.
* proxy: mcp.request improvementsdormando2022-08-031-0/+7
| | | | | | | | | | - errors if a string value is missing the "\r\n" terminator - properly uses a value from a response object - allows passing in a request object for the value as well - also adds r:vlen() for recovering the value length of a response think this still needs r:flags() or similar?
* tls: Add switch to opt-in to kernel TLS on OpenSSL 3.0.0+Kevin Lin2022-07-031-0/+1
|
* proxy: mcp.log_req* API interfacedormando2022-04-081-5/+14
| | | | | | Lua level API for logging full context of a request/response. Provides log_req() for simple logging and log_reqsample() for conditional logging.
* proxy: allow await() to be called recursivelydormando2022-03-021-3/+7
| | | | | | | | previously mcp.await() only worked if it was called before any other dispatches. also fixes a bug if the supplied pool table was key=value instead of an array-type table.
* proxy: add ring_hash builtindormando2022-02-242-31/+28
| | | | | | | | | | | | | this is ketama-based, with options for minor compat changes with major libraries. does _not_ support weights. The weights bits in the original ketama broke the algorithm, as changing the number of points would shift unrelated servers when the list changes. this also changes backends to take a "name" specifically, instead of an "ip address". Though note if supplying a hostname instead of an IP there might be inline DNS lookups on reconnects.
* storage: parameterize the compaction thread sleepdormando2022-02-215-5/+5
| | | | | | allows tests to run faster, let users make it sleep longer/less time. Also cuts the sleep time down when actively compacting and coming from high idle.
View Lorry Controller Status