summaryrefslogtreecommitdiff
path: root/storage
Commit message (Collapse)AuthorAgeFilesLines
* MDEV-28363 remove #ifdef SPIDER_use_LEX_CSTRING_for_Field_blob_constructorbb-11.1-mdev-28363Yuchen Pei2023-04-146-74/+0
|
* MDEV-21921 Make transaction_isolation and transaction_read_only into system ↵Junqi Xie2023-04-121-3/+3
| | | | | | | | | | | | | | variables In MariaDB, we have a confusing problem where: * The transaction_isolation option can be set in a configuration file, but it cannot be set dynamically. * The tx_isolation system variable can be set dynamically, but it cannot be set in a configuration file. Therefore, we have two different names for the same thing in different contexts. This is needlessly confusing, and it complicates the documentation. The same thing applys for transaction_read_only. MySQL 5.7 solved this problem by making them into system variables. https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-20.html This commit takes a similar approach by adding new system variables and marking the original ones as deprecated. This commit also resolves some legacy problems related to SET STATEMENT and transaction_isolation.
* Merge 11.0 into 11.1Marko Mäkelä2023-03-2919-97/+143
|\
| * Merge 10.11 into 11.0Marko Mäkelä2023-03-2919-97/+143
| |\
| | * Merge 10.10 into 10.11Marko Mäkelä2023-03-2916-94/+140
| | |\
| | | * Merge 10.9 into 10.10Marko Mäkelä2023-03-2916-94/+140
| | | |\
| | | | * Merge 10.8 into 10.9Marko Mäkelä2023-03-2916-94/+140
| | | | |\
| | | | | * Merge 10.6 into 10.8Marko Mäkelä2023-03-2916-94/+140
| | | | | |\
| | | | | | * Merge 10.5 into 10.6Marko Mäkelä2023-03-2810-46/+20
| | | | | | |\
| | | | | | | * MDEV-30936 fixupMarko Mäkelä2023-03-282-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fil_space_t::~fil_space_t(): Invoke ut_free(name) because doing so in the callers would trip MSAN_OPTIONS=poison_in_dtor=1
| | | | | | | * MDEV-30936 clang 15.0.7 -fsanitize=memory fails massivelyMarko Mäkelä2023-03-2815-59/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | handle_slave_io(), handle_slave_sql(), os_thread_exit(): Remove a redundant pthread_exit(nullptr) call, because it would cause SIGSEGV. mysql_print_status(): Add MEM_MAKE_DEFINED() to work around some missing instrumentation around mallinfo2(). que_graph_free_stat_list(): Invoke que_node_get_next(node) before que_graph_free_recursive(node). That is the logical and MSAN_OPTIONS=poison_in_dtor=1 compatible way of freeing memory. ins_node_t::~ins_node_t(): Invoke mem_heap_free(entry_sys_heap). que_graph_free_recursive(): Rely on ins_node_t::~ins_node_t(). fts_t::~fts_t(): Invoke mem_heap_free(fts_heap). fts_free(): Replace with direct calls to fts_t::~fts_t(). The failures in free_root() due to MSAN_OPTIONS=poison_in_dtor=1 will be covered in MDEV-30942.
| | | | | | * | MDEV-26071: rpl.rpl_perfschema_applier_status_by_worker failed in bb …Andrei2023-03-241-27/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | …with: Test assertion failed Problem: ======= Assertion text: 'Value returned by SSS and PS table for Last_Error_Number should be same.' Assertion condition: '"1146" = "0"' Assertion condition, interpolated: '"1146" = "0"' Assertion result: '0' Analysis: ======== In parallel replication when slave is started the worker pool gets activated and it gets cleared when slave stops. Each time the worker pool gets activated a backup worker pool also gets created to store worker specific perforance schema information in case of errors. On error, all relevant information is copied from rpl_parallel_thread to rli and it gets cleared from thread. Then server waits for all workers to complete their work, during this stage performance schema table specific worker info is stored into the backup pool and finally the actual pool gets cleared. If users query the performance schema table to know the status of workers the information from backup pool will be used. The test simulates ER_NO_SUCH_TABLE error and verifies the worker information in pfs table. Test works fine if execution occurs in following order. Step 1. Error occurred 'worker information is copied to backup pool'. Step 2. handle_slave_sql invokes 'rpl_parallel_resize_pool_if_no_slaves' to deactivate worker pool, it marks the pool->count=0 Step 3. PFS table is queried, since actual pool is deactivated backup pool information is read. If the Step 3 happens prior to Step2 the pool is yet to be deactivated and the actual pool is read, which doesn't have any error details as they were cleared. Hence test ocasionally fails. Fix: === Upon error mark the back pool as being active so that if PFS table is quried since the backup pool is flagged as valid its information will be read, in case it is not flagged regular pool will be read. This work is one of the last pieces created by the late Sujatha Sivakumar.
| | | | | | * | MDEV-29545 InnoDB: Can't find record during replace stmtThirunarayanan Balathandayuthapani2023-03-241-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: ======== - InnoDB replace statement returns can't find record as result during bulk insert operation. InnoDB returns DB_END_OF_INDEX blindly when bulk transaction is visible to current transaction even though the search tuple is inserted as a part of current replace statement. Solution: ========= row_search_mvcc(): InnoDB should allow the transaction to read all the rows when innodb intends to do any locking on the record even though bulk insert transaction changes are visible to the current transaction
| | | | | | * | MDEV-30900 Crash on macOS due to zero-initialized buf_dblwr.write_condMarko Mäkelä2023-03-241-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | buf_dblwr_t::init(), buf_dblwr_t::close(): Cover also write_cond, which was added in commit a55b951e6082a4ce9a1f2ed5ee176ea7dbbaf1f2 without explicit initialization. On GNU/Linux, PTHREAD_COND_INITIALIZER is a zero-initializer. That is why the default zero initialization happened to work on that platform.
| | | | | | * | Merge 10.5 into 10.6Marko Mäkelä2023-03-222-17/+88
| | | | | | |\ \ | | | | | | | |/
| | | | | | | * MDEV-30882 Crash on ROLLBACK in a ROW_FORMAT=COMPRESSED tableMarko Mäkelä2023-03-222-17/+89
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | btr_cur_upd_rec_in_place(): Avoid calling page_zip_write_rec() if we are not modifying any fields that are stored in compressed format. btr_cur_update_in_place_zip_check(): New function to check if a ROW_FORMAT=COMPRESSED record can actually be updated in place. btr_cur_pessimistic_update(): If the BTR_KEEP_POS_FLAG is not set (we are in a ROLLBACK and cannot write any BLOBs), ignore the potential overflow and let page_zip_reorganize() or page_zip_compress() handle it. This avoids a failure when an attempted UPDATE of an NULL column to 0 is rolled back. During the ROLLBACK, we would try to move a non-updated long column to off-page storage in order to avoid a compression failure of the ROW_FORMAT=COMPRESSED page. page_zip_write_trx_id_and_roll_ptr(): Remove an assertion that would fail in row_upd_rec_in_place() because the uncompressed page would already have been modified there. This is a 10.5 version of commit ff3d4395d808b6421d2e0714e10d48c7aa2f3c3a (different because of commit 08ba388713946c03aa591899cd3a446a6202f882).
| | | | | | * | MDEV-30400 fixup: Fix the UNIV_ZIP_DEBUG buildMarko Mäkelä2023-03-202-3/+3
| | | | | | | |
| | * | | | | | Fix trivial spelling errorsOtto Kekalainen2023-03-243-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - agressively -> aggressively - exising -> existing - occured -> occurred - releated -> related - seperated -> separated - sucess -> success - use use -> use All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
* | | | | | | | MDEV-30920 Remove need_lock and table from spider_close_sys_table()bb-11.1-mdev-30920Yuchen Pei2023-03-287-280/+173
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | They became obsolete after commit cfd145faedd: commit cfd145faedd68e27d4ac8f1e5f95ec1a6dcd8c8d Author: Nayuta Yanagisawa <nayuta.yanagisawa@hey.com> Date: Fri Jan 28 01:03:06 2022 +0900 MDEV-27641 Spider: remove #if MYSQL_VERSION_ID < ${VERSION}
* | | | | | | | Fix ColumnStore againMarko Mäkelä2023-03-271-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes up commit e371b1e2648f043f1bd27f416b0c52f083cbb410 that accidentally reverted d77aaa6994b30660bd8788d3415ae4a44f55d9a0.
* | | | | | | | MDEV-28883 fixup: clang -Winconsistent-missing-overrideMarko Mäkelä2023-03-241-0/+0
| | | | | | | |
* | | | | | | | Merge 11.0 into 11.1Marko Mäkelä2023-03-231-8/+43
|\ \ \ \ \ \ \ \ | |/ / / / / / /
| * | | | | | | MDEV-30911 Multi-batch recovery of ROW_FORMAT=COMPRESSED table hangsMarko Mäkelä2023-03-231-8/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In commit d6aed21621e438e6135415c0c09fc6adfc0be368 a condition at the start of buf_read_ahead_random() was refactored. Only the caller buf_read_recv_pages() was adjusted for this. We must in fact adjust every caller and make sure that spare blocks will be allocated while crash recovery is in progress. This is the simplest fix; ideally recovery would operate on the compressed page frame. The observed recovery hang occurred because pages 0 and 3 of a tablespace were being read due to buf_page_get_gen() calls by trx_resurrect_table_locks() before the log records for these pages had been applied. In buf_page_t::read_complete() we would skip the call to recv_recover_page() because no uncompressed page frame had been allocated for the block.
* | | | | | | | MDEV-28522 Delete constant SPIDER_SQL_TYPE_*_HSbb-11.1-mdev-28522Nayuta Yanagisawa2023-03-233-137/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The HandlerSocket support of Spider has been deleted by MDEV-26858. Thus, the constants, SPIDER_SQL_TYPE_*_HS, are no longer necessary.
* | | | | | | | Merge 11.0 into 11.1Marko Mäkelä2023-03-201-1/+0
|\ \ \ \ \ \ \ \ | |/ / / / / / /
| * | | | | | | Merge 10.11 into 11.0Marko Mäkelä2023-03-201-1/+0
| |\ \ \ \ \ \ \ | | |/ / / / / /
| | * | | | | | Merge 10.10 into 10.11Marko Mäkelä2023-03-201-1/+0
| | |\ \ \ \ \ \ | | | |/ / / / /
| | | * | | | | Merge 10.9 into 10.10Marko Mäkelä2023-03-201-1/+0
| | | |\ \ \ \ \ | | | | |/ / / /
| | | | * | | | Merge 10.8 into 10.9Marko Mäkelä2023-03-201-1/+0
| | | | |\ \ \ \ | | | | | |/ / /
| | | | | * | | Merge 10.6 into 10.8Marko Mäkelä2023-03-201-1/+0
| | | | | |\ \ \ | | | | | | |/ /
| | | | | | * | MDEV-26827 fixup: Remove a bogus assertionMarko Mäkelä2023-03-201-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can have dirty_blocks=0 when buf_flush_page_cleaner() is being woken up to write out or evict pages from the buf_pool.LRU list.
* | | | | | | | Merge 11.0 into 11.1Marko Mäkelä2023-03-1736-1262/+1392
|\ \ \ \ \ \ \ \ | |/ / / / / / /
| * | | | | | | Merge 10.11 into 11.0Marko Mäkelä2023-03-1736-1262/+1392
| |\ \ \ \ \ \ \ | | |/ / / / / /
| | * | | | | | Merge 10.10 into 10.11Marko Mäkelä2023-03-173-11/+33
| | |\ \ \ \ \ \ | | | |/ / / / /
| | | * | | | | Merge 10.9 into 10.10Marko Mäkelä2023-03-173-11/+33
| | | |\ \ \ \ \ | | | | |/ / / /
| | | | * | | | Merge 10.8 into 10.9Marko Mäkelä2023-03-173-11/+33
| | | | |\ \ \ \ | | | | | |/ / /
| | | | | * | | Merge 10.6 into 10.8Marko Mäkelä2023-03-173-11/+33
| | | | | |\ \ \ | | | | | | |/ /
| | | | | | * | MDEV-30870 Undo tablespace name displays wrongly for I_S queriesThirunarayanan Balathandayuthapani2023-03-171-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - INNODB_SYS_TABLESPACES in information schema should display innodb_undo001, innodb_undo002 etc as tablespace name for undo tablespaces
| | | | | | * | MDEV-29975 InnoDB fails to release savepoint during bulk insertThirunarayanan Balathandayuthapani2023-03-171-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - InnoDB does rollback the whole transaction and discards the savepoint when there is a failure happens during bulk insert operation. When server request to release the savepoint, InnoDB should return DB_SUCCESS when it deals with bulk insert operation
| | * | | | | | Merge 10.10 into 10.11Marko Mäkelä2023-03-1735-1311/+1418
| | |\ \ \ \ \ \ | | | |/ / / / /
| | | * | | | | Merge 10.9 into 10.10Marko Mäkelä2023-03-1735-1311/+1418
| | | |\ \ \ \ \ | | | | |/ / / /
| | | | * | | | Merge 10.8 into 10.9Marko Mäkelä2023-03-1735-1311/+1418
| | | | |\ \ \ \ | | | | | |/ / /
| | | | | * | | Merge 10.6 into 10.8Marko Mäkelä2023-03-1634-1300/+1396
| | | | | |\ \ \ | | | | | | |/ /
| | | | | | * | MDEV-26827 Make page flushing even fasterMarko Mäkelä2023-03-1619-668/+703
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For more convenient monitoring of something that could greatly affect the volume of page writes, we add the status variable Innodb_buffer_pool_pages_split that was previously only available via information_schema.innodb_metrics as "innodb_page_splits". This was suggested by Axel Schwenke. buf_flush_page_count: Replaced with buf_pool.stat.n_pages_written. We protect buf_pool.stat (except n_page_gets) with buf_pool.mutex and remove unnecessary export_vars indirection. buf_pool.flush_list_bytes: Moved from buf_pool.stat.flush_list_bytes. Protected by buf_pool.flush_list_mutex. buf_pool_t::page_cleaner_status: Replaces buf_pool_t::n_flush_LRU_, buf_pool_t::n_flush_list_, and buf_pool_t::page_cleaner_is_idle. Protected by buf_pool.flush_list_mutex. We will exclusively broadcast buf_pool.done_flush_list by the buf_flush_page_cleaner thread, and only wait for it when communicating with buf_flush_page_cleaner. There is no need to keep a count of pending writes by the buf_pool.flush_list processing. A single flag suffices for that. Waits for page write completion can be performed by simply waiting on block->page.lock, or by invoking buf_dblwr.wait_for_page_writes(). buf_LRU_block_free_non_file_page(): Broadcast buf_pool.done_free and set buf_pool.try_LRU_scan when freeing a page. This would be executed also as part of buf_page_write_complete(). buf_page_write_complete(): Do not broadcast buf_pool.done_flush_list, and do not acquire buf_pool.mutex unless buf_pool.LRU eviction is needed. Let buf_dblwr count all writes to persistent pages and broadcast a condition variable when no outstanding writes remain. buf_flush_page_cleaner(): Prioritize LRU flushing and eviction right after "furious flushing" (lsn_limit). Simplify the conditions and reduce the hold time of buf_pool.flush_list_mutex. Refuse to shut down or sleep if buf_pool.ran_out(), that is, LRU eviction is needed. buf_pool_t::page_cleaner_wakeup(): Add the optional parameter for_LRU. buf_LRU_get_free_block(): Protect buf_lru_free_blocks_error_printed with buf_pool.mutex. Invoke buf_pool.page_cleaner_wakeup(true) to to ensure that buf_flush_page_cleaner() will process the LRU flush request. buf_do_LRU_batch(), buf_flush_list(), buf_flush_list_space(): Update buf_pool.stat.n_pages_written when submitting writes (while holding buf_pool.mutex), not when completing them. buf_page_t::flush(), buf_flush_discard_page(): Require that the page U-latch be acquired upfront, and remove buf_page_t::ready_for_flush(). buf_pool_t::delete_from_flush_list(): Remove the parameter "bool clear". buf_flush_page(): Count pending page writes via buf_dblwr. buf_flush_try_neighbors(): Take the block of page_id as a parameter. If the tablespace is dropped before our page has been written out, release the page U-latch. buf_pool_invalidate(): Let the caller ensure that there are no outstanding writes. buf_flush_wait_batch_end(false), buf_flush_wait_batch_end_acquiring_mutex(false): Replaced with buf_dblwr.wait_for_page_writes(). buf_flush_wait_LRU_batch_end(): Replaces buf_flush_wait_batch_end(true). buf_flush_list(): Remove some broadcast of buf_pool.done_flush_list. buf_flush_buffer_pool(): Invoke also buf_dblwr.wait_for_page_writes(). buf_pool_t::io_pending(), buf_pool_t::n_flush_list(): Remove. Outstanding writes are reflected by buf_dblwr.pending_writes(). buf_dblwr_t::init(): New function, to initialize the mutex and the condition variables, but not the backing store. buf_dblwr_t::is_created(): Replaces buf_dblwr_t::is_initialised(). buf_dblwr_t::pending_writes(), buf_dblwr_t::writes_pending: Keeps track of writes of persistent data pages. buf_flush_LRU(): Allow calls while LRU flushing may be in progress in another thread. Tested by Matthias Leich (correctness) and Axel Schwenke (performance)
| | | | | | * | MDEV-26055: Improve adaptive flushingMarko Mäkelä2023-03-165-354/+360
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Adaptive flushing is enabled by setting innodb_max_dirty_pages_pct_lwm>0 (not default) and innodb_adaptive_flushing=ON (default). There is also the parameter innodb_adaptive_flushing_lwm (default: 10 per cent of the log capacity). It should enable some adaptive flushing even when innodb_max_dirty_pages_pct_lwm=0. That is not being changed here. This idea was first presented by Inaam Rana several years ago, and I discussed it with Jean-François Gagné at FOSDEM 2023. buf_flush_page_cleaner(): When we are not near the log capacity limit (neither buf_flush_async_lsn nor buf_flush_sync_lsn are set), also try to move clean blocks from the buf_pool.LRU list to buf_pool.free or initiate writes (but not the eviction) of dirty blocks, until the remaining I/O capacity has been consumed. buf_flush_LRU_list_batch(): Add the parameter bool evict, to specify whether dirty least recently used pages (from buf_pool.LRU) should be evicted immediately after they have been written out. Callers outside buf_flush_page_cleaner() will pass evict=true, to retain the existing behaviour. buf_do_LRU_batch(): Add the parameter bool evict. Return counts of evicted and flushed pages. buf_flush_LRU(): Add the parameter bool evict. Assume that the caller holds buf_pool.mutex and will invoke buf_dblwr.flush_buffered_writes() afterwards. buf_flush_list_holding_mutex(): A low-level variant of buf_flush_list() whose caller must hold buf_pool.mutex and invoke buf_dblwr.flush_buffered_writes() afterwards. buf_flush_wait_batch_end_acquiring_mutex(): Remove. It is enough to have buf_flush_wait_batch_end(). page_cleaner_flush_pages_recommendation(): Avoid some floating-point arithmetics. buf_flush_page(), buf_flush_check_neighbor(), buf_flush_check_neighbors(), buf_flush_try_neighbors(): Rename the parameter "bool lru" to "bool evict". buf_free_from_unzip_LRU_list_batch(): Remove the parameter. Only actual page writes will contribute towards the limit. buf_LRU_free_page(): Evict freed pages of temporary tables. buf_pool.done_free: Broadcast whenever a block is freed (and buf_pool.try_LRU_scan is set). buf_pool_t::io_buf_t::reserve(): Retry indefinitely. During the test encryption.innochecksum we easily run out of these buffers for PAGE_COMPRESSED or ENCRYPTED pages. Tested by Matthias Leich and Axel Schwenke
| | | | | | * | MDEV-30357 Performance regression in locking reads from secondary indexesMarko Mäkelä2023-03-164-36/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | lock_sec_rec_some_has_impl(): Remove a harmful condition that caused the performance regression and should not have been added in commit b6e41e38720d1e8d33b2abec0d1109615133bc2b in the first place. Locking transactions that have not modified any persistent tables can carry the transaction identifier 0. trx_t::max_inactive_id: A cache for trx_sys_t::find_same_or_older(). The value is not reset on transaction commit so that previous results can be reused for subsequent transactions. The smallest active transaction ID can only increase over time, not decrease. trx_sys_t::find_same_or_older(): Remember the maximum previous id for which rw_trx_hash.iterate() returned false, to avoid redundant iterations. lock_sec_rec_read_check_and_lock(): Add an early return in case we are already holding a covering table lock. lock_rec_convert_impl_to_expl(): Add a template parameter to avoid a redundant run-time check on whether the index is secondary. lock_rec_convert_impl_to_expl_for_trx(): Move some code from lock_rec_convert_impl_to_expl(), to reduce code duplication due to the added template parameter. Reviewed by: Vladislav Lesin Tested by: Matthias Leich
| | | | | | * | MDEV-29835 InnoDB hang on B-tree split or mergeMarko Mäkelä2023-03-167-160/+156
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a follow-up to commit de4030e4d49805a7ded5c0bfee01cc3fd7623522 (MDEV-30400), which fixed some hangs related to B-tree split or merge. btr_root_block_get(): Use and update the root page guess. This is just a minor performance optimization, not affecting correctness. btr_validate_level(): Remove the parameter "lockout", and always acquire an exclusive dict_index_t::lock in CHECK TABLE without QUICK. This is needed in order to avoid latching order violation in btr_page_get_father_node_ptr_for_validate(). btr_cur_need_opposite_intention(): Return true in case btr_cur_compress_recommendation() would hold later during the mini-transaction, or if a page underflow or overflow is possible. If we return true, our caller will escalate to aqcuiring an exclusive dict_index_t::lock, to prevent a latching order violation and deadlock during btr_compress() or btr_page_split_and_insert(). btr_cur_t::search_leaf(), btr_cur_t::open_leaf(): Also invoke btr_cur_need_opposite_intention() on the leaf page. btr_cur_t::open_leaf(): When escalating to exclusive index locking, acquire exclusive latches on all pages as well. innobase_instant_try(): Return an error code if the root page cannot be retrieved. In addition to the normal stress testing with Random Query Generator (RQG) this has been tested with ./mtr --mysqld=--loose-innodb-limit-optimistic-insert-debug=2 but with the injection in btr_cur_optimistic_insert() for non-leaf pages adjusted so that it would use the value 3. (Otherwise, infinite page splits could occur in some mtr tests.) Tested by: Matthias Leich
| | | | | | * | Merge 10.5 into 10.6Marko Mäkelä2023-03-163-19/+93
| | | | | | |\ \ | | | | | | | |/
| | | | | | | * MDEV-30860 Race condition between buffer pool flush and log file deletion in ↵Marko Mäkelä2023-03-161-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | mariadb-backup --prepare srv_start(): If we are going to close the log file in mariadb-backup --prepare, call buf_flush_sync() before calling recv_sys.debug_free() to ensure that the log file will not be accessed. This fixes a rather rare failure in the test mariabackup.innodb_force_recovery where buf_flush_page_cleaner() would invoke log_checkpoint_low() because !recv_recovery_is_on() would hold due to the fact that recv_sys.debug_free() had already been called. Then, the log write for the checkpoint would fail because srv_start() had invoked log_sys.log.close_file().
| | | | | | | * MDEV-30775 Performance regression in fil_space_t::try_to_close() introduced ↵Vlad Lesin2023-03-103-14/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | in MDEV-23855 fil_node_open_file_low() tries to close files from the top of fil_system.space_list if the number of opened files is exceeded. It invokes fil_space_t::try_to_close(), which iterates the list searching for the first opened space. Then it just closes the space, leaving it in the same position in fil_system.space_list. On heavy files opening, like during 'SHOW TABLE STATUS ...' execution, if the number of opened files limit is reached, fil_space_t::try_to_close() iterates more and more closed spaces before reaching any opened space for each fil_node_open_file_low() call. What causes performance regression if the number of spaces is big enough. The fix is to keep opened spaces at the top of fil_system.space_list, and move closed files at the end of the list. For this purpose fil_space_t::space_list_last_opened pointer is introduced. It points to the last inserted opened space in fil_space_t::space_list. When space is opened, it's inserted to the position just after the pointer points to in fil_space_t::space_list to preserve the logic, inroduced in MDEV-23855. Any closed space is added to the end of fil_space_t::space_list. As opened spaces are located at the top of fil_space_t::space_list, fil_space_t::try_to_close() finds opened space faster. There can be the case when opened and closed spaces are mixed in fil_space_t::space_list if fil_system.freeze_space_list was set during fil_node_open_file_low() execution. But this should not cause any error, as fil_space_t::try_to_close() still iterates spaces in the list. There is no need in any test case for the fix, as it does not change any functionality, but just fixes performance regression.