Update cost for hash and cached joins

The old code did not't correctly add TIME_FOR_COMPARE to rows that are
part of the scan that will be compared with the attached where clause.

Now the cost calculation for hash join and full join cache join are
identical except for HASH_FANOUT (10%)

The cost for a join with keys is now also uniform.
The total cost for a using a key for lookup is calculated in one place as:

(cost_of_finding_rows_through_key(records) + records/TIME_FOR_COMPARE)*
record_count_of_previous_row_combinations + startup_cost

startup_cost is the cost of a creating a temporary table (if needed)

Best_cost now includes the cost of comparing all WHERE clauses and also
cost of joining with previous row combinations.

Other things:
- Optimizer trace is now printing the total costs, including testing the
  WHERE clause (TIME_FOR_COMPARE) and comparing with all previous rows.
- In optimizer trace, include also total cost of query together with the
  final join order. This makes it easier to find out where the cost was
  calculated.
- Old code used filter even if the cost for it was higher than not using a
  filter. This is not corrected.
- When rebasing on 10.11, I noticed some changes to access_cost_factor
  calculation. These changes was not picked as the coming changes
  to filtering will make that code obsolete.

1 files changed, 1 insertions, 1 deletions

diff --git a/mysql-test/main/subselect3.result b/mysql-test/main/subselect3.result
index 2820775cb4f..28187e0ffdd 100644
--- a/mysql-test/main/subselect3.result
+++ b/mysql-test/main/subselect3.result
@@ -166,7 +166,7 @@ from t3;
 id	select_type	table	type	possible_keys	key	key_len	ref	rows	filtered	Extra
 1	PRIMARY	t3	ALL	NULL	NULL	NULL	NULL	3	100.00	
 2	DEPENDENT SUBQUERY	t1	ref_or_null	a	a	5	func	4	100.00	Using where; Full scan on NULL key
-2	DEPENDENT SUBQUERY	t2	ALL	a	NULL	NULL	NULL	10	75.00	Using where; Using join buffer (flat, BNL join)
+2	DEPENDENT SUBQUERY	t2	ref	a	a	5	test.t1.b	1	100.00	Using where
 Warnings:
 Note	1276	Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1
 Note	1003	/* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3`