MDEV-24953: 10.5.9 crashes with large IN() listbb-10.5-mdev24953

The problem was in and_all_keys(), the code of MDEV-9759 which calculates
the new tree weight:

First, it didn't take into account the case when

(next->next_key_part=tmp) == NULL

and dereferenced a NULL pointer when getting tmp->weight.

Second, "if (param->alloced_sel_args > SEL_ARG::MAX_SEL_ARGS) break"
could leave the loop with incorrect value of weight.

Fixed by introducing SEL_ARG::update_weight_locally() and calling it
at the end of the function. This allows to avoid caring about all the
above cases.

1 files changed, 15 insertions, 1 deletions

diff --git a/mysql-test/main/range_notembedded.result b/mysql-test/main/range_notembedded.result
index 87fa85d3ac6..0ecf47c892e 100644
--- a/mysql-test/main/range_notembedded.result
+++ b/mysql-test/main/range_notembedded.result
@@ -159,7 +159,6 @@ left(@json, 2500)
     ]
 ]
 ## Repeat the above with a bit higher max_weight:
-set @tmp9750_weight=@@optimizer_max_sel_arg_weight;
 set optimizer_max_sel_arg_weight=120;
 explain select * from t1 where 
 kp1 in (1,2,3,4,5,6,7,8,9,10) and
@@ -225,3 +224,18 @@ SELECT *
 FROM mysql.help_relation ignore index (help_topic_id)
 WHERE (help_topic_id = 8 OR help_keyword_id = 0) AND help_keyword_id != 2 AND help_topic_id >= 1900;
 help_topic_id	help_keyword_id
+#
+# MDEV-24953: 10.5.9 crashes with large IN() list
+#
+CREATE TABLE t1 (
+notification_type_id smallint(4) unsigned NOT NULL DEFAULT 0,
+item_id int(10) unsigned NOT NULL DEFAULT 0,
+item_parent_id int(10) unsigned NOT NULL DEFAULT 0,
+user_id int(10) unsigned NOT NULL DEFAULT 0,
+PRIMARY KEY (notification_type_id,item_id,item_parent_id,user_id)
+);
+insert into t1 values (1,1,1,1), (2,2,2,2), (3,3,3,3);
+# Run crashing query
+id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
+1	SIMPLE	t1	range	PRIMARY	PRIMARY	2	NULL	3	Using where
+drop table t1;