| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
| |
This fixes issue: https://github.com/apache/couchdb/issues/3786
In addition, add few _all_dbs limit tests since we didn't seem to have
any previously to catch such issues. Plus, test some of the corner
cases which should be caught by the BDU and should return a 403 error
code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This means `update_seq` values from `GET $db` `last_seq` returned from ` GET
$db/_changes?since=now&limit=` will be more resilient to change feed rewinds.
Besides, those sequences will now be more consistent and users won't have to
wonder why one opaque sequence works slightly differently than another opaque
update sequence.
Previously, when the sequences were returned only as numeric values, it was
impossible to calculate replacements and change feeds had to always rewind back
to 0 for those ranges. With uuids and epochs in play, it is possible to figure
out that some shards might have moved to new nodes or find internal replication
checkpoints to avoid streaming changes feeds from 0 on those ranges.
Some replication Elixir tests decode update sequences, so those were updated to
handle the new uuid and epoch format as well.
Fixes: https://github.com/apache/couchdb/issues/3787
Co-author: Adam Kocoloski kocolosk@apache.org
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
couch_icu_driver is only used for binary strings comparison in
couch_ejson_compare when expression depth becomes greater than 10.
The logic for string comparison is identical to what couch_ejson_compare uses,
so opt to just use couch_ejson_compare instead of keeping a whole other binary
collation driver around.
To avoid a possible infinite loop if couch_ejson_compare nif fails to
load, throw a nif loading error as is common for nif modules.
To avoid another case of a possible infinite retry from of badarg
generated by max depth, and/or an actual bad ejson term, use a
specific max depth error so we don't have to guess when we catch it
and retry term traversal in erlang.
There was another uncodumented case when badarg was thrown besides max
depth or an invalid arg. It was when a prop value was compared with
any other supported type. In erlang it would be handled in these
clauses:
```
less_erl({A},{B}) when is_list(A), is_list(B) -> less_props(A,B);
less_erl({A},_) when is_list(A) -> -1;
less_erl(_,{B}) when is_list(B) -> 1.
```
However, in C we threw a badarg for the last two clauses and relied on
erlang to do all the work. This case was a potential performance issue
as well since that is a common comparison for mango where we may
compare keys against the max json object value (<<255,255,255,255>>).
Add a few property tests in order to validate collation behavior. The two main
ones are:
1) Given an expected sort order of some test values, assert that both the
erlang and nif collators would correctly order any of those test values.
2) In general, the nif collator would sort any json the same way as the
erlang one.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`infinity` it turns out is a valid configuration value for fabric
request_timeout. We can pass that to Erlang `receive` statement, any arithmetic
with it would fail.
To guard against the crash use the max small int value (60 bits). With enough
shards, due to the exponential nature of the algorithm, we still get a nice
progression from the minimum 100 msec all the way up to the large int value.
This case is illustrated in the test.
Issue: https://github.com/apache/couchdb/issues/3789
|
| |
|
| |
|
|
|
|
|
|
|
|
| |
Prevent this race condition:
*** context setup failed ***
**in function couch_replicator_doc_processor:setup/0 (src/couch_replicator_doc_processor.erl, line 872)
**error:{badmatch,{error,{already_started,<0.4946.0>}}}
|
|
|
|
|
|
| |
- Prepend unused variable with underscore
- Add nowarn_export_all compiler option
- Use STACKTRACE macro
|
|
|
|
| |
Delete unused function and remove unused variable.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In the previous attempt [1] we improved the logic by spawning workers on the
matching target shards only. However, that wasn't enough as workers might still
reject the passed in sequence from the old node when it asserts ownership
locally on each shard.
Re-use the already existing replacement clause, where after uuid is matched, we
try harder to find the highest viable sequence. To use the unit test setup as
an example, if the shard moved from node1 to node2, and recorded epoch `{node2,
10}` on the new node, then a sequence generated on node1 before the move, for
example 12, would rewind down to 10 only when calculated on new location on
node2, instead of being rewound all the way down to 0.
[1] https://github.com/apache/couchdb/commit/e83935c7f8c3e47b47f07f22ece327f6529d4da0
|
|
|
|
| |
To include another changelog entry: https://github.com/apache/couchdb-documentation/commit/4f00da0b0cedf63ebf391e43b1a56bb36f7d0f96
|
|
|
| |
Missed file in f85cff669f20cee0a54da7bb8c645dfc4d2de5c9
|
|
|
|
|
|
| |
Based off of the upstream 1.0.9 + CouchDB clone changes
https://github.com/apache/couchdb-jiffy/releases/tag/CouchDB-1.0.9-1
|
| |
|
|
|
|
|
|
| |
Fauxton was failing so we backported the fix from main
https://github.com/apache/couchdb/commit/f85cff669f20cee0a54da7bb8c645dfc4d2de5c9
for it.
|
| |
|
|
|
|
|
| |
These two tests exercise the same assertions as the individual
`sandbox_doc_attachments` test case in chttpd_csp_tests.erl.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We do that by matching the comparator function behavior used during row merging
[1] in with the comparison function used when sorting the rows on view
shards [2]. This goes along with the constraint in the lists:merge/3 docs which
indicates that the input lists should be sorted according to the same
comparator [3] as the one passed to the lists:merge/3 call.
The stability of returned rows results from the case when both keys match as
equal. Now `lists:merge/3` will favor the element in the existing rows list
instead of always replacing [4] the older matching row with the last arriving
row, since now `less(A, A)` will be `false`, while previously it was `true`.
The fix was found by Adam when discussing issue #3750
https://github.com/apache/couchdb/issues/3750#issuecomment-920947424
Co-author: Adam Kocoloski <kocolosk@apache.org>
[1] https://github.com/apache/couchdb/blob/3.x/src/fabric/src/fabric_view_map.erl#L244-L248
[2] https://github.com/apache/couchdb/blob/3.x/src/couch_mrview/src/couch_mrview_util.erl#L1103-L1107
[3] https://erlang.org/doc/man/lists.html#merge-3
[4] https://github.com/erlang/otp/blob/master/lib/stdlib/src/lists.erl#L2668-L2675
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Results should now be returned in descending {key, doc_id} order.
The idea is to reverse the key list before sending it to the workers, so they
will emit rows in reverse order. Also, we are using the same reversed list when
building the KeyDict structure on the coordinator. That way the order of the
sent rows and the expected coordinator sorting order will match.
For testing, enhance an existing multi-key Elixir view test to test both
ascending and descending cases and actually check that the rows are in the
correct order each time.
|
|\
| |
| | |
Port 3286 - Add ability to control which Elixir integration tests to run
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New `elixir-suite` Makefile target is added. It runs a predefined set of elixir
integration tests.
The feature is controlled by two files:
- test/elixir/test/config/suite.elixir - contains list of all available tests
- test/elixir/test/config/skip.elixir - contains list of tests to skip
In order to update the `test/elixir/test/config/suite.elixir` when new tests
are added. The one would need to run the following command:
```
MIX_ENV=integration mix suite > test/elixir/test/config/suite.elixir
```
|
|\
| |
| | |
Fix limit0 for views again
|
| |
| |
| |
| |
| |
| |
| |
| | |
The limit=0 clause was introduced in commit 4e0c97bf which added
sorted=false support. It accidentally matches when the user specifies
limit=0 and causes us not to apply the logic that ensures we collect a
{meta, Meta} message from each shard range and then send the
total_rows and offset fields.
|
|/
|
|
| |
This reverts commit a36e7308ab4a2cfead6da64a9f83b7776722382d.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, as soon as one row returned, we immediately stopped, erroneously
assuming that meta for all ranges have already been received. However, it was
possible that we'd get meta from range 00-7f, then a row from 00-7f before
getting meta from 7f-ff and thus we'd return an empty result.
To fix the issue we simply re-use the already existing limit=0 clause from the
fabric_view:maybe_send_row/1 function which will wait until there is a complete
ring before returning. That relies on updating the counters (the ring) only
with meta return and not with view rows, so if the ring is complete, we know we
only completed with meta.
The other issue with limit=0 clause was that it wasn't properly ack-ing the
received row. Rows are acked for sorted=false case below and for the regular
limit>0, sorted=true case in fabric_view:get_next_row/1.
Issue: https://github.com/apache/couchdb/issues/3750
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, if the source db purge sequence > `purge_infos_limit`, shard
splitting would crash with the `{{invalid_start_purge_seq,0},
[{couch_bt_engine,fold_purge_infos,5...` error. That was because purge
sequences were always copied starting from 0. That would only work as long as
the total number of purges stayed below the purge_infos_limit threshold. In
order to correctly gather the purge sequences, the start sequence must be
based off of the actual oldest sequence currently available.
An example of how it should be done is in the `mem_rpc` module, when loading
purge infos [0], so here we do exactly the same. The `MinSeq - 1` logic is also
evident by inspecting the fold_purge_infos [1] function.
The test sets up the exact scenario as described above: reduces the purge info
limit to 10 then purges 20 documents. By purging more than the limit, we ensure
the starting sequence is now != 0. However, the purge sequence btree is
actually trimmed down during compaction. That is why there are a few extra
helper functions to ensure compaction runs and finishes before shard splitting
starts.
Fixes: https://github.com/apache/couchdb/issues/3738
[0] https://github.com/apache/couchdb/blob/4ea9f1ea1a2078162d0e281948b56469228af3f7/src/mem3/src/mem3_rpc.erl#L206-L207
[1] https://github.com/apache/couchdb/blob/3.x/src/couch/src/couch_bt_engine.erl#L625-L637
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, users with low {Q, N} dbs often got the `"No DB shards could be
opened."` error when the cluster is overloaded. The hard-coded 100 msec timeout
was too low to open the few available shards and the whole request would crash
with a 500 error.
Attempt to calculate an optimal timeout value based on the number of shards and
the max fabric request timeout limit.
The sequence of doubling (by default) timeouts forms a geometric progression.
Use the well known closed form formula for the sum [0], and the maximum request
timeout, to calculate the initial timeout. The test case illustrates a few
examples with some default Q and N values.
Because we don't want the timeout value to be too low, since it takes time to
open shards, and we don't want to quickly cycle through a few initial shards
and discard the results, the minimum inital timeout is clipped to the
previously hard-coded 100 msec timeout. Unlike previously however, this minimum
value can now also be configured.
[0] https://en.wikipedia.org/wiki/Geometric_series
Fixes: https://github.com/apache/couchdb/issues/3733
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This introduces CSP settings for attachments and show/list funs and
streamlines the configuration with the existing Fauxton CSP options.
Deprecates the old `[csp] enable` and `[csp] header_value` config
options, but they are honoured going forward.
They are replaced with `[csp] utils_enable` and `[csp] utils_header_value`
respectively. The funcitonality and default values remain the same.
In addition, these new config options are added, along with their
default values:
```
[csp]
attachments_enable = true
attachments_header_value = sandbox
showlist_enable = true
showlist_header_value = sandbox
```
These add `Content-Security-Policy` headers to all attachment requests
and to all non-JSON show and all list function responses.
Co-authored-by: Nick Vatamaniuc <vatamane@gmail.com>
Co-authored-by: Robert Newson <rnewson@apache.org>
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When shards are moved to new nodes, and the user supplies a change sequence
from the old shard map configuration, attempt to match missing nodes and ranges
by inspecting current shard uuids in order to avoid rewinds.
Previously, if a node and range was missing, we randomly picked a node in the
appropriate range, so 1/3 of the time we might have hit the exact node, but 2/3
of the time we would end up with a complete changes feed rewind to 0.
Unfortunately, this involves a fabric worker scatter gather operation to all
shard copies. This should only happen when we get an old sequence. We rely on
that happening rarely, mostly right after the shards moved, then users would
get new sequence from the recent shard map.
|
|
|
|
|
|
| |
This module was kept around since 2.2.0 only to facilitate cluster upgrades
after we switched the receiver logic to not closures around between nodes
https://github.com/apache/couchdb/commit/fe53e437ca5ec9d23aa1b55d7934daced157a9e3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This only applies to databases that have an n > [cluster] n.
Our `middleman()` function that proxies attachment streams from
the incoming HTTP socket on the coordinating node to the target
shard-bearing nodes used the server config to determine whether
it should start dropping chunks from the stream.
If a database was created with a larger `n`, the `middleman()`
function could have started to drop attachment chunks before
all attached nodes had a chance to receive it.
This fix uses a database’s concrete `n` value rather than the
server config default value.
Co-Authored-By: James Coglan <jcoglan@gmail.com>
Co-Authored-By: Robert Newson <rnewson@apache.org>
|
|
|
|
|
|
|
|
|
|
|
| |
While including a payload within a DELETE request is not forbidden by RFC7231
its presence on a delete attachment request leaves a mochiweb acceptor
in a semi-opened state since mochiweb's using lazy load for the request bodies.
This makes a next immediate request to the same acceptor to hung
until previous request's receive timeout.
This PR adds a step to explicitly "drain" and discard an entity body on a
delete attachment request to prevent that.
|
|
|
|
|
|
|
|
| |
Rebar mustache templating engine has a bug when handling the }}} brackets in a
case like {...{{var}}}. So we work around the issue by using a separate
variable.
This is an alternate fix for issue: https://github.com/apache/couchdb/pull/3617
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
They used to be disabled before the last major ibrowse upgrade.
On MacOS and FreeBSD the following tests fails periodically:
```
ibrowse_tests: running_server_fixture_test_ (Pipeline too small signals retries)...*failed*
in function ibrowse_tests:'-small_pipeline/0-fun-5-'/1 (test/ibrowse_tests.erl, line 150)
in call from ibrowse_tests:small_pipeline/0 (test/ibrowse_tests.erl, line 150)
**error:{assertEqual,[{module,ibrowse_tests},
{line,150},
{expression,"Counts"},
{expected,"\n\n\n\n\n\n\n\n\n\n"},
{value,"\t\n\n\n\n\t\t\n\n\t"}]}
output:<<"Load Balancer Pid : <0.494.0>
```
But seems to pass more reliable on Linux for some reson. It would be nice to
run the tests of course but having a passing full platsform suite is more
important.
|
| |
|
| |
|
| |
|
| |
|