passes.texi (Tree-SSA passes): Document SLP pass.

	* doc/passes.texi (Tree-SSA passes): Document SLP pass.
	* tree-pass.h (pass_slp_vectorize): New pass.
	* params.h (SLP_MAX_INSNS_IN_BB): Define.
	* timevar.def (TV_TREE_SLP_VECTORIZATION): Define.
	* tree-vectorizer.c (timevar.h): Include.
	(user_vect_verbosity_level): Declare.
	(vect_location): Fix comment.
	(vect_set_verbosity_level): Update user_vect_verbosity_level
	instead of vect_verbosity_level.
	(vect_set_dump_settings): Add an argument. Ignore user defined
	verbosity if dump flags require higher level of verbosity. Print to
	stderr only for loop vectorization.
	(vectorize_loops): Update call to vect_set_dump_settings.
	(execute_vect_slp): New function.
	(gate_vect_slp): Likewise.
	(struct gimple_opt_pass pass_slp_vectorize): New.
	* tree-vectorizer.h (struct _bb_vec_info): Define along macros to
	access its members.
	(vec_info_for_bb): New function.
	(struct _stmt_vec_info): Add bb_vinfo and a macro for its access.
	(VECTORIZATION_ENABLED): New macro.
	(SLP_ENABLED, SLP_DISABLED): Likewise.
	(vect_is_simple_use): Add bb_vec_info argument.
	(new_stmt_vec_info, vect_analyze_data_ref_dependences,
	vect_analyze_data_refs_alignment, vect_verify_datarefs_alignment,
	vect_analyze_data_ref_accesses, vect_analyze_data_refs,
	vect_schedule_slp, vect_analyze_slp): Likewise.
	(vect_analyze_stmt): Add slp_tree argument.
	(find_bb_location): Declare.
	(vect_slp_analyze_bb, vect_slp_transform_bb): Likewise.
	* tree-vect-loop.c (new_loop_vec_info): Adjust function calls.
	(vect_analyze_loop_operations, vect_analyze_loop,
	get_initial_def_for_induction, vect_create_epilog_for_reduction,
	vect_finalize_reduction, vectorizable_reduction,
	vectorizable_live_operation, vect_transform_loop): Likewise.
	* tree-data-ref.c (dr_analyze_innermost): Update comment,
	skip evolution analysis if analyzing a basic block.
	(dr_analyze_indices): Likewise.
	(initialize_data_dependence_relation): Skip the test whether the
	object is invariant for basic blocks.
	(compute_all_dependences): Skip dependence analysis for data
	references in basic blocks.
	(find_data_references_in_stmt): Don't fail in case of invariant
	access in basic block.
	(find_data_references_in_bb): New function.
	(find_data_references_in_loop): Move code to
	find_data_references_in_bb    and add a call to it.
	(compute_data_dependences_for_bb): New function.
	* tree-data-ref.h (compute_data_dependences_for_bb): Declare.
	* tree-vect-data-refs.c (vect_check_interleaving): Adjust to the case
	that STEP is 0.
	(vect_analyze_data_ref_dependence): Check for interleaving in case of
	unknown dependence in basic block and fail in case of dependence in
	basic block.
	(vect_analyze_data_ref_dependences): Add bb_vinfo argument, get data
	dependence instances from either loop or basic block vectorization
	info.
	(vect_compute_data_ref_alignment): Check if it is loop vectorization
	before calling nested_in_vect_loop_p.
	(vect_compute_data_refs_alignment): Add bb_vinfo argument, get data
	dependence instances from either loop or basic block vectorization
	info.
	(vect_verify_datarefs_alignment): Likewise.
	(vect_enhance_data_refs_alignment): Adjust function calls.
	(vect_analyze_data_refs_alignment): Likewise.
	(vect_analyze_group_access): Fix printing. Skip different checks if
	DR_STEP is 0. Keep strided stores either in loop or basic block
	vectorization data structure. Fix indentation.
	(vect_analyze_data_ref_access): Fix comments, allow zero step in
	basic blocks.
	(vect_analyze_data_ref_accesses): Add bb_vinfo argument, get data
	dependence instances from either loop or basic block vectorization
	info.
	(vect_analyze_data_refs): Update comment. Call
	compute_data_dependences_for_bb to analyze basic blocks.
	(vect_create_addr_base_for_vector_ref): Check for outer loop only in
	case of loop vectorization. In case of basic block vectorization use
	data-ref itself   as  a base.
	(vect_create_data_ref_ptr): In case of basic block vectorization:
	don't advance the pointer, add new statements before the current
	statement.  Adjust function calls.
	(vect_supportable_dr_alignment): Support only aligned accesses in
	basic block vectorization.
	* common.opt (ftree-slp-vectorize): New flag.
	* tree-vect-patterns.c (widened_name_p): Adjust function calls.
	(vect_pattern_recog_1): Likewise.
	* tree-vect-stmts.c (process_use): Likewise.
	(vect_init_vector): Add new statements in the beginning of the basic
	block in case of basic block SLP.
	(vect_get_vec_def_for_operand): Adjust function calls.
	(vect_finish_stmt_generation): Likewise.
	(vectorizable_call): Add assert that it is loop vectorization, adjust
	function calls.
	(vectorizable_conversion, vectorizable_assignment): Likewise.
	(vectorizable_operation): In case of basic block SLP, take
	vectorization factor from statement's type and skip the relevance
	check. Adjust function calls.
	(vectorizable_type_demotion): Add assert that it is loop
	vectorization, adjust function calls.
	(vectorizable_type_promotion): Likewise.
	(vectorizable_store): Check for outer loop only in case of loop
	vectorization. Adjust function calls. For basic blocks, skip the
	relevance check and don't advance pointers.
	(vectorizable_load): Likewise.
	(vectorizable_condition): Add assert that it is loop vectorization,
	adjust function calls.
	(vect_analyze_stmt): Add argument. In case of basic block SLP, check
	that it is not reduction, get vector type, call only supported
	functions, skip loop    specific parts.
	(vect_transform_stmt): Check for outer loop only in case of loop
	vectorization.
	(new_stmt_vec_info): Add new argument and initialize bb_vinfo.
	(vect_is_simple_use): Fix comment, add new argument, fix conditions
	for external definition.
	* passes.c (pass_slp_vectorize): New pass.
	* tree-vect-slp.c (find_bb_location): New function.
	(vect_get_and_check_slp_defs): Add argument, adjust function calls,
	check for patterns only in loops.
	(vect_build_slp_tree): Add argument, adjust function calls, fail in
	case of multiple types in basic block SLP.
	(vect_mark_slp_stmts_relevant): New function.
	(vect_supported_load_permutation_p): Fix comment.
	(vect_analyze_slp_instance): Add argument. In case of basic block
	SLP, take vectorization factor from statement's type, check that
	unrolling factor is 1. Adjust function call. Save SLP instance in
	either loop or basic block vectorization structure. Return FALSE,
	if SLP failed.
	(vect_analyze_slp): Add argument. Get strided stores groups from
	either loop or basic block vectorization structure. Return FALSE
	if basic block SLP failed.
	(new_bb_vec_info): New function.
	(destroy_bb_vec_info, vect_slp_analyze_node_operations,
	vect_slp_analyze_operations, vect_slp_analyze_bb): Likewise.
	(vect_schedule_slp): Add argument. Get SLP instances from either
	loop or basic block vectorization structure. Set vectorization factor
	to be 1 for basic block SLP.
	(vect_slp_transform_bb): New function.
	* params.def (PARAM_SLP_MAX_INSNS_IN_BB): Define.

From-SVN: r147829

1 files changed, 204 insertions, 82 deletions

diff --git a/gcc/tree-vect-stmts.c b/gcc/tree-vect-stmts.c
index 06d51e2d318..9ac733a76be 100644
--- a/gcc/tree-vect-stmts.c
+++ b/gcc/tree-vect-stmts.c
@@ -248,7 +248,7 @@ process_use (gimple stmt, tree use, loop_vec_info loop_vinfo, bool live_p,
   if (!exist_non_indexing_operands_for_use_p (use, stmt))
      return true;
 
-  if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
+  if (!vect_is_simple_use (use, loop_vinfo, NULL, &def_stmt, &def, &dt))
     { 
       if (vect_print_dump_info (REPORT_UNVECTORIZED_LOCATIONS))
         fprintf (vect_dump, "not vectorized: unsupported use in stmt.");
@@ -336,7 +336,7 @@ process_use (gimple stmt, tree use, loop_vec_info loop_vinfo, bool live_p,
         {
         case vect_unused_in_scope:
           relevant = (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def) ?
-                        vect_used_in_outer_by_reduction : vect_unused_in_scope;
+                      vect_used_in_outer_by_reduction : vect_unused_in_scope;
           break;
 
         case vect_used_in_outer_by_reduction:
@@ -813,13 +813,29 @@ vect_init_vector (gimple stmt, tree vector_var, tree vector_type,
   else
     {
       loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
-      struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
-
-      if (nested_in_vect_loop_p (loop, stmt))
-        loop = loop->inner;
-      pe = loop_preheader_edge (loop);
-      new_bb = gsi_insert_on_edge_immediate (pe, init_stmt);
-      gcc_assert (!new_bb);
+      
+      if (loop_vinfo)
+        {
+          struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
+
+          if (nested_in_vect_loop_p (loop, stmt))
+            loop = loop->inner;
+          
+	  pe = loop_preheader_edge (loop);
+          new_bb = gsi_insert_on_edge_immediate (pe, init_stmt);
+          gcc_assert (!new_bb);
+	}
+      else
+       {
+          bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_vinfo);
+          basic_block bb;
+          gimple_stmt_iterator gsi_bb_start;
+
+          gcc_assert (bb_vinfo);
+          bb = BB_VINFO_BB (bb_vinfo);
+          gsi_bb_start = gsi_start_bb (bb);
+          gsi_insert_before (&gsi_bb_start, init_stmt, GSI_SAME_STMT);
+       }
     }
 
   if (vect_print_dump_info (REPORT_DETAILS))
@@ -832,6 +848,7 @@ vect_init_vector (gimple stmt, tree vector_var, tree vector_type,
   return vec_oprnd;
 }
 
+
 /* Function vect_get_vec_def_for_operand.
 
    OP is an operand in STMT. This function returns a (vector) def that will be
@@ -869,7 +886,8 @@ vect_get_vec_def_for_operand (tree op, gimple stmt, tree *scalar_def)
       print_generic_expr (vect_dump, op, TDF_SLIM);
     }
 
-  is_simple_use = vect_is_simple_use (op, loop_vinfo, &def_stmt, &def, &dt);
+  is_simple_use = vect_is_simple_use (op, loop_vinfo, NULL, &def_stmt, &def, 
+                                      &dt);
   gcc_assert (is_simple_use);
   if (vect_print_dump_info (REPORT_DETAILS))
     {
@@ -1122,12 +1140,14 @@ vect_finish_stmt_generation (gimple stmt, gimple vec_stmt,
 {
   stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
   loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
+  bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info);
 
   gcc_assert (gimple_code (stmt) != GIMPLE_LABEL);
 
   gsi_insert_before (gsi, vec_stmt, GSI_SAME_STMT);
 
-  set_vinfo_for_stmt (vec_stmt, new_stmt_vec_info (vec_stmt, loop_vinfo));
+  set_vinfo_for_stmt (vec_stmt, new_stmt_vec_info (vec_stmt, loop_vinfo, 
+                                                   bb_vinfo));
 
   if (vect_print_dump_info (REPORT_DETAILS))
     {
@@ -1191,6 +1211,9 @@ vectorizable_call (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt)
   enum { NARROW, NONE, WIDEN } modifier;
   size_t i, nargs;
 
+  /* FORNOW: unsupported in basic block SLP.  */
+  gcc_assert (loop_vinfo);
+    
   if (!STMT_VINFO_RELEVANT_P (stmt_info))
     return false;
 
@@ -1232,7 +1255,7 @@ vectorizable_call (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt)
 	}
       rhs_type = TREE_TYPE (op);
 
-      if (!vect_is_simple_use (op, loop_vinfo, &def_stmt, &def, &dt[i]))
+      if (!vect_is_simple_use (op, loop_vinfo, NULL, &def_stmt, &def, &dt[i]))
 	{
 	  if (vect_print_dump_info (REPORT_DETAILS))
 	    fprintf (vect_dump, "use not simple.");
@@ -1509,6 +1532,9 @@ vectorizable_conversion (gimple stmt, gimple_stmt_iterator *gsi,
 
   /* Is STMT a vectorizable conversion?   */
 
+  /* FORNOW: unsupported in basic block SLP.  */
+  gcc_assert (loop_vinfo);
+  
   if (!STMT_VINFO_RELEVANT_P (stmt_info))
     return false;
 
@@ -1575,7 +1601,7 @@ vectorizable_conversion (gimple stmt, gimple_stmt_iterator *gsi,
   gcc_assert (ncopies >= 1);
 
   /* Check the operands of the operation.  */
-  if (!vect_is_simple_use (op0, loop_vinfo, &def_stmt, &def, &dt[0]))
+  if (!vect_is_simple_use (op0, loop_vinfo, NULL, &def_stmt, &def, &dt[0]))
     {
       if (vect_print_dump_info (REPORT_DETAILS))
 	fprintf (vect_dump, "use not simple.");
@@ -1759,6 +1785,7 @@ vectorizable_assignment (gimple stmt, gimple_stmt_iterator *gsi,
   int i;
   VEC(tree,heap) *vec_oprnds = NULL;
   tree vop;
+  bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info);
 
   /* Multiple types in SLP are handled by creating the appropriate number of
      vectorized stmts for each SLP node. Hence, NCOPIES is always 1 in
@@ -1772,7 +1799,7 @@ vectorizable_assignment (gimple stmt, gimple_stmt_iterator *gsi,
   if (ncopies > 1)
     return false; /* FORNOW */
 
-  if (!STMT_VINFO_RELEVANT_P (stmt_info))
+  if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo)
     return false;
 
   if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def)
@@ -1792,7 +1819,7 @@ vectorizable_assignment (gimple stmt, gimple_stmt_iterator *gsi,
   else
     return false;
 
-  if (!vect_is_simple_use (op, loop_vinfo, &def_stmt, &def, &dt[0]))
+  if (!vect_is_simple_use (op, loop_vinfo, bb_vinfo, &def_stmt, &def, &dt[0]))
     {
       if (vect_print_dump_info (REPORT_DETAILS))
         fprintf (vect_dump, "use not simple.");
@@ -1875,6 +1902,14 @@ vectorizable_operation (gimple stmt, gimple_stmt_iterator *gsi,
   unsigned int k;
   bool shift_p = false;
   bool scalar_shift_arg = false;
+  bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info);
+  int vf;
+
+  if (loop_vinfo)
+    vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
+  else
+    /* FORNOW: multiple types are not supported in basic block SLP.  */
+    vf = nunits_in;
 
   /* Multiple types in SLP are handled by creating the appropriate number of
      vectorized stmts for each SLP node. Hence, NCOPIES is always 1 in
@@ -1886,7 +1921,7 @@ vectorizable_operation (gimple stmt, gimple_stmt_iterator *gsi,
 
   gcc_assert (ncopies >= 1);
 
-  if (!STMT_VINFO_RELEVANT_P (stmt_info))
+  if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo)
     return false;
 
   if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def)
@@ -1924,7 +1959,7 @@ vectorizable_operation (gimple stmt, gimple_stmt_iterator *gsi,
     }
 
   op0 = gimple_assign_rhs1 (stmt);
-  if (!vect_is_simple_use (op0, loop_vinfo, &def_stmt, &def, &dt[0]))
+  if (!vect_is_simple_use (op0, loop_vinfo, bb_vinfo, &def_stmt, &def, &dt[0]))
     {
       if (vect_print_dump_info (REPORT_DETAILS))
         fprintf (vect_dump, "use not simple.");
@@ -1934,7 +1969,8 @@ vectorizable_operation (gimple stmt, gimple_stmt_iterator *gsi,
   if (op_type == binary_op)
     {
       op1 = gimple_assign_rhs2 (stmt);
-      if (!vect_is_simple_use (op1, loop_vinfo, &def_stmt, &def, &dt[1]))
+      if (!vect_is_simple_use (op1, loop_vinfo, bb_vinfo, &def_stmt, &def, 
+                               &dt[1]))
 	{
 	  if (vect_print_dump_info (REPORT_DETAILS))
 	    fprintf (vect_dump, "use not simple.");
@@ -2016,8 +2052,7 @@ vectorizable_operation (gimple stmt, gimple_stmt_iterator *gsi,
 	fprintf (vect_dump, "op not supported by target.");
       /* Check only during analysis.  */
       if (GET_MODE_SIZE (vec_mode) != UNITS_PER_WORD
-          || (LOOP_VINFO_VECT_FACTOR (loop_vinfo)
-	      < vect_min_worthwhile_factor (code)
+	  || (vf < vect_min_worthwhile_factor (code)
               && !vec_stmt))
         return false;
       if (vect_print_dump_info (REPORT_DETAILS))
@@ -2026,8 +2061,7 @@ vectorizable_operation (gimple stmt, gimple_stmt_iterator *gsi,
 
   /* Worthwhile without SIMD support? Check only during analysis.  */
   if (!VECTOR_MODE_P (TYPE_MODE (vectype))
-      && LOOP_VINFO_VECT_FACTOR (loop_vinfo)
-	 < vect_min_worthwhile_factor (code)
+      && vf < vect_min_worthwhile_factor (code)
       && !vec_stmt)
     {
       if (vect_print_dump_info (REPORT_DETAILS))
@@ -2333,6 +2367,9 @@ vectorizable_type_demotion (gimple stmt, gimple_stmt_iterator *gsi,
   VEC (tree, heap) *vec_dsts = NULL, *interm_types = NULL, *tmp_vec_dsts = NULL;
   tree last_oprnd, intermediate_type;
 
+  /* FORNOW: not supported by basic block SLP vectorization.  */
+  gcc_assert (loop_vinfo);
+
   if (!STMT_VINFO_RELEVANT_P (stmt_info))
     return false;
 
@@ -2371,7 +2408,6 @@ vectorizable_type_demotion (gimple stmt, gimple_stmt_iterator *gsi,
     ncopies = 1;
   else
     ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits_out;
-
   gcc_assert (ncopies >= 1);
 
   if (! ((INTEGRAL_TYPE_P (TREE_TYPE (scalar_dest))
@@ -2382,7 +2418,7 @@ vectorizable_type_demotion (gimple stmt, gimple_stmt_iterator *gsi,
     return false;
 
   /* Check the operands of the operation.  */
-  if (!vect_is_simple_use (op0, loop_vinfo, &def_stmt, &def, &dt[0]))
+  if (!vect_is_simple_use (op0, loop_vinfo, NULL, &def_stmt, &def, &dt[0]))
     {
       if (vect_print_dump_info (REPORT_DETAILS))
         fprintf (vect_dump, "use not simple.");
@@ -2562,7 +2598,7 @@ vect_create_vectorized_promotion_stmts (VEC (tree, heap) **vec_oprnds0,
                                               prev_stmt_info);
     }
 }
-
+    
 
 /* Function vectorizable_type_promotion
 
@@ -2600,6 +2636,9 @@ vectorizable_type_promotion (gimple stmt, gimple_stmt_iterator *gsi,
   VEC (tree, heap) *vec_oprnds0 = NULL, *vec_oprnds1 = NULL;
   VEC (tree, heap) *vec_dsts = NULL, *interm_types = NULL, *tmp_vec_dsts = NULL;
   
+  /* FORNOW: not supported by basic block SLP vectorization.  */
+  gcc_assert (loop_vinfo);
+    
   if (!STMT_VINFO_RELEVANT_P (stmt_info))
     return false;
 
@@ -2650,7 +2689,7 @@ vectorizable_type_promotion (gimple stmt, gimple_stmt_iterator *gsi,
     return false;
 
   /* Check the operands of the operation.  */
-  if (!vect_is_simple_use (op0, loop_vinfo, &def_stmt, &def, &dt[0]))
+  if (!vect_is_simple_use (op0, loop_vinfo, NULL, &def_stmt, &def, &dt[0]))
     {
       if (vect_print_dump_info (REPORT_DETAILS))
 	fprintf (vect_dump, "use not simple.");
@@ -2661,7 +2700,7 @@ vectorizable_type_promotion (gimple stmt, gimple_stmt_iterator *gsi,
   if (op_type == binary_op)
     {
       op1 = gimple_assign_rhs2 (stmt);
-      if (!vect_is_simple_use (op1, loop_vinfo, &def_stmt, &def, &dt[1]))
+      if (!vect_is_simple_use (op1, loop_vinfo, NULL, &def_stmt, &def, &dt[1]))
         {
 	  if (vect_print_dump_info (REPORT_DETAILS))
 	    fprintf (vect_dump, "use not simple.");
@@ -2806,7 +2845,7 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
   struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info), *first_dr = NULL;
   tree vectype = STMT_VINFO_VECTYPE (stmt_info);
   loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
-  struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
+  struct loop *loop = NULL;
   enum machine_mode vec_mode;
   tree dummy;
   enum dr_alignment_support alignment_support_scheme;
@@ -2827,6 +2866,10 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
   bool slp = (slp_node != NULL);
   stmt_vec_info first_stmt_vinfo;
   unsigned int vec_num;
+  bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info);
+
+  if (loop_vinfo)
+    loop = LOOP_VINFO_LOOP (loop_vinfo);
 
   /* Multiple types in SLP are handled by creating the appropriate number of
      vectorized stmts for each SLP node. Hence, NCOPIES is always 1 in
@@ -2839,14 +2882,14 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
   gcc_assert (ncopies >= 1);
 
   /* FORNOW. This restriction should be relaxed.  */
-  if (nested_in_vect_loop_p (loop, stmt) && ncopies > 1)
+  if (loop && nested_in_vect_loop_p (loop, stmt) && ncopies > 1)
     {
       if (vect_print_dump_info (REPORT_DETAILS))
         fprintf (vect_dump, "multiple types in nested loop.");
       return false;
     }
 
-  if (!STMT_VINFO_RELEVANT_P (stmt_info))
+  if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo)
     return false;
 
   if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def)
@@ -2865,7 +2908,7 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
 
   gcc_assert (gimple_assign_single_p (stmt));
   op = gimple_assign_rhs1 (stmt);
-  if (!vect_is_simple_use (op, loop_vinfo, &def_stmt, &def, &dt))
+  if (!vect_is_simple_use (op, loop_vinfo, bb_vinfo, &def_stmt, &def, &dt))
     {
       if (vect_print_dump_info (REPORT_DETAILS))
         fprintf (vect_dump, "use not simple.");
@@ -2907,7 +2950,8 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
             {
 	      gcc_assert (gimple_assign_single_p (next_stmt));
 	      op = gimple_assign_rhs1 (next_stmt);
-              if (!vect_is_simple_use (op, loop_vinfo, &def_stmt, &def, &dt))
+              if (!vect_is_simple_use (op, loop_vinfo, bb_vinfo, &def_stmt, 
+                                       &def, &dt))
                 {
                   if (vect_print_dump_info (REPORT_DETAILS))
                     fprintf (vect_dump, "use not simple.");
@@ -2935,7 +2979,7 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
       DR_GROUP_STORE_COUNT (vinfo_for_stmt (first_stmt))++;
 
       /* FORNOW */
-      gcc_assert (!nested_in_vect_loop_p (loop, stmt));
+      gcc_assert (!loop || !nested_in_vect_loop_p (loop, stmt));
 
       /* We vectorize all the stmts of the interleaving group when we
 	 reach the last stmt in the group.  */
@@ -3063,7 +3107,7 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
 	  dataref_ptr = vect_create_data_ref_ptr (first_stmt, NULL, NULL_TREE, 
 						  &dummy, &ptr_incr, false, 
 						  &inv_p);
-	  gcc_assert (!inv_p);
+	  gcc_assert (bb_vinfo || !inv_p);
 	}
       else 
 	{
@@ -3077,7 +3121,8 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
 	  for (i = 0; i < group_size; i++)
 	    {
 	      op = VEC_index (tree, oprnds, i);
-	      vect_is_simple_use (op, loop_vinfo, &def_stmt, &def, &dt);
+	      vect_is_simple_use (op, loop_vinfo, bb_vinfo, &def_stmt, &def, 
+	                          &dt);
 	      vec_oprnd = vect_get_vec_def_for_stmt_copy (dt, op); 
 	      VEC_replace(tree, dr_chain, i, vec_oprnd);
 	      VEC_replace(tree, oprnds, i, vec_oprnd);
@@ -3162,9 +3207,9 @@ vectorizable_load (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
   stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
   stmt_vec_info prev_stmt_info; 
   loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
-  struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
+  struct loop *loop = NULL;
   struct loop *containing_loop = (gimple_bb (stmt))->loop_father;
-  bool nested_in_vect_loop = nested_in_vect_loop_p (loop, stmt);
+  bool nested_in_vect_loop = false;
   struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info), *first_dr;
   tree vectype = STMT_VINFO_VECTYPE (stmt_info);
   tree new_temp;
@@ -3192,6 +3237,18 @@ vectorizable_load (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
   bool slp = (slp_node != NULL);
   bool slp_perm = false;
   enum tree_code code;
+  bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info);
+  int vf;
+
+  if (loop_vinfo)
+    {
+      loop = LOOP_VINFO_LOOP (loop_vinfo);
+      nested_in_vect_loop = nested_in_vect_loop_p (loop, stmt);
+      vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
+    }
+  else
+    /* FORNOW: multiple types are not supported in basic block SLP.  */
+    vf = nunits;
 
   /* Multiple types in SLP are handled by creating the appropriate number of
      vectorized stmts for each SLP node. Hence, NCOPIES is always 1 in
@@ -3211,10 +3268,7 @@ vectorizable_load (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
       return false;
     }
 
-  if (slp && SLP_INSTANCE_LOAD_PERMUTATION (slp_node_instance))
-    slp_perm = true;
- 
-  if (!STMT_VINFO_RELEVANT_P (stmt_info))
+  if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo)
     return false;
 
   if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def)
@@ -3300,7 +3354,9 @@ vectorizable_load (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
 	{
 	  strided_load = false;
 	  vec_num = SLP_TREE_NUMBER_OF_VEC_STMTS (slp_node);
-	}
+          if (SLP_INSTANCE_LOAD_PERMUTATION (slp_node_instance))
+            slp_perm = true;
+    	}
       else
 	vec_num = group_size;
 
@@ -3419,7 +3475,7 @@ vectorizable_load (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
      This can only occur when vectorizing memory accesses in the inner-loop
      nested within an outer-loop that is being vectorized.  */
 
-  if (nested_in_vect_loop_p (loop, stmt)
+  if (loop && nested_in_vect_loop_p (loop, stmt)
       && (TREE_INT_CST_LOW (DR_STEP (dr))
 	  % GET_MODE_SIZE (TYPE_MODE (vectype)) != 0))
     {
@@ -3551,7 +3607,7 @@ vectorizable_load (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
 	    }
 
 	  /* 4. Handle invariant-load.  */
-	  if (inv_p)
+	  if (inv_p && !bb_vinfo)
 	    {
 	      gcc_assert (!strided_load);
 	      gcc_assert (nested_in_vect_loop_p (loop, stmt));
@@ -3598,8 +3654,7 @@ vectorizable_load (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt,
 
       if (slp_perm)
         {
-          if (!vect_transform_slp_perm_load (stmt, dr_chain, gsi,
-                                   LOOP_VINFO_VECT_FACTOR (loop_vinfo),
+          if (!vect_transform_slp_perm_load (stmt, dr_chain, gsi, vf,
                                              slp_node_instance, false))
             {
               VEC_free (tree, heap, dr_chain);
@@ -3659,7 +3714,8 @@ vect_is_simple_cond (tree cond, loop_vec_info loop_vinfo)
   if (TREE_CODE (lhs) == SSA_NAME)
     {
       gimple lhs_def_stmt = SSA_NAME_DEF_STMT (lhs);
-      if (!vect_is_simple_use (lhs, loop_vinfo, &lhs_def_stmt, &def, &dt))
+      if (!vect_is_simple_use (lhs, loop_vinfo, NULL, &lhs_def_stmt, &def, 
+                               &dt))
 	return false;
     }
   else if (TREE_CODE (lhs) != INTEGER_CST && TREE_CODE (lhs) != REAL_CST
@@ -3669,7 +3725,8 @@ vect_is_simple_cond (tree cond, loop_vec_info loop_vinfo)
   if (TREE_CODE (rhs) == SSA_NAME)
     {
       gimple rhs_def_stmt = SSA_NAME_DEF_STMT (rhs);
-      if (!vect_is_simple_use (rhs, loop_vinfo, &rhs_def_stmt, &def, &dt))
+      if (!vect_is_simple_use (rhs, loop_vinfo, NULL, &rhs_def_stmt, &def, 
+                               &dt))
 	return false;
     }
   else if (TREE_CODE (rhs) != INTEGER_CST  && TREE_CODE (rhs) != REAL_CST
@@ -3709,6 +3766,9 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi,
   int ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits;
   enum tree_code code;
 
+  /* FORNOW: unsupported in basic block SLP.  */
+  gcc_assert (loop_vinfo);
+    
   gcc_assert (ncopies >= 1);
   if (ncopies > 1)
     return false; /* FORNOW */
@@ -3757,7 +3817,7 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi,
   if (TREE_CODE (then_clause) == SSA_NAME)
     {
       gimple then_def_stmt = SSA_NAME_DEF_STMT (then_clause);
-      if (!vect_is_simple_use (then_clause, loop_vinfo, 
+      if (!vect_is_simple_use (then_clause, loop_vinfo, NULL, 
 			       &then_def_stmt, &def, &dt))
 	return false;
     }
@@ -3769,7 +3829,7 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi,
   if (TREE_CODE (else_clause) == SSA_NAME)
     {
       gimple else_def_stmt = SSA_NAME_DEF_STMT (else_clause);
-      if (!vect_is_simple_use (else_clause, loop_vinfo, 
+      if (!vect_is_simple_use (else_clause, loop_vinfo, NULL,
 			       &else_def_stmt, &def, &dt))
 	return false;
     }
@@ -3819,11 +3879,14 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi,
 /* Make sure the statement is vectorizable.  */
 
 bool
-vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
+vect_analyze_stmt (gimple stmt, bool *need_to_vectorize, slp_tree node)
 {
   stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
-  enum vect_relevant relevance = STMT_VINFO_RELEVANT (stmt_info);
+  bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info);
+  enum vect_relevant relevance = STMT_VINFO_RELEVANT (stmt_info);  
   bool ok;
+  HOST_WIDE_INT dummy;
+  tree scalar_type, vectype;
 
   if (vect_print_dump_info (REPORT_DETAILS))
     {
@@ -3831,15 +3894,15 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
       print_gimple_stmt (vect_dump, stmt, 0, TDF_SLIM);
     }
 
-  /* Skip stmts that do not need to be vectorized. In loops this is expected
+  /* Skip stmts that do not need to be vectorized. In loops this is expected 
      to include:
      - the COND_EXPR which is the loop exit condition
      - any LABEL_EXPRs in the loop
-     - computations that are used only for array indexing or loop control.
+     - computations that are used only for array indexing or loop control.  
      In basic blocks we only analyze statements that are a part of some SLP
      instance, therefore, all the statements are relevant.  */
 
-  if (!STMT_VINFO_RELEVANT_P (stmt_info)
+  if (!STMT_VINFO_RELEVANT_P (stmt_info) 
       && !STMT_VINFO_LIVE_P (stmt_info))
     {
       if (vect_print_dump_info (REPORT_DETAILS))
@@ -3854,9 +3917,9 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
         break;
 
       case vect_reduction_def:
-         gcc_assert (relevance == vect_used_in_outer
+         gcc_assert (!bb_vinfo && (relevance == vect_used_in_outer
                      || relevance == vect_used_in_outer_by_reduction
-                     || relevance == vect_unused_in_scope);
+                     || relevance == vect_unused_in_scope));
          break;
 
       case vect_induction_def:
@@ -3867,6 +3930,37 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
         gcc_unreachable ();
     }
 
+  if (bb_vinfo)
+    {
+      gcc_assert (PURE_SLP_STMT (stmt_info));
+
+      scalar_type = vect_get_smallest_scalar_type (stmt, &dummy, &dummy);
+      if (vect_print_dump_info (REPORT_DETAILS))
+        {
+          fprintf (vect_dump, "get vectype for scalar type:  ");
+          print_generic_expr (vect_dump, scalar_type, TDF_SLIM);
+        }
+
+      vectype = get_vectype_for_scalar_type (scalar_type);
+      if (!vectype)
+        {
+          if (vect_print_dump_info (REPORT_DETAILS))
+            {
+               fprintf (vect_dump, "not SLPed: unsupported data-type ");
+               print_generic_expr (vect_dump, scalar_type, TDF_SLIM);
+            }
+          return false;
+        }
+
+      if (vect_print_dump_info (REPORT_DETAILS))
+        {
+          fprintf (vect_dump, "vectype:  ");
+          print_generic_expr (vect_dump, vectype, TDF_SLIM);
+        }
+
+      STMT_VINFO_VECTYPE (stmt_info) = vectype;
+   }
+
   if (STMT_VINFO_RELEVANT_P (stmt_info))
     {
       gcc_assert (!VECTOR_MODE_P (TYPE_MODE (gimple_expr_type (stmt))));
@@ -3875,8 +3969,9 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
     }
 
    ok = true;
-   if (STMT_VINFO_RELEVANT_P (stmt_info)
-       || STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def)
+   if (!bb_vinfo 
+       && (STMT_VINFO_RELEVANT_P (stmt_info)
+           || STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def))
       ok = (vectorizable_type_promotion (stmt, NULL, NULL, NULL)
             || vectorizable_type_demotion (stmt, NULL, NULL, NULL)
             || vectorizable_conversion (stmt, NULL, NULL, NULL)
@@ -3887,6 +3982,14 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
             || vectorizable_store (stmt, NULL, NULL, NULL)
             || vectorizable_condition (stmt, NULL, NULL)
             || vectorizable_reduction (stmt, NULL, NULL));
+    else
+      {
+        if (bb_vinfo)
+          ok = (vectorizable_operation (stmt, NULL, NULL, node)
+                || vectorizable_assignment (stmt, NULL, NULL, node)
+                || vectorizable_load (stmt, NULL, NULL, node, NULL)
+                || vectorizable_store (stmt, NULL, NULL, node));
+      }	    
 
   if (!ok)
     {
@@ -3896,10 +3999,13 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
           fprintf (vect_dump, "supported: ");
           print_gimple_stmt (vect_dump, stmt, 0, TDF_SLIM);
         }
-
+    
       return false;
     }
 
+  if (bb_vinfo)
+    return true;
+
   /* Stmts that are (also) "live" (i.e. - that are used out of the loop)
       need extra handling, except for vectorizable reductions.  */
   if (STMT_VINFO_LIVE_P (stmt_info)
@@ -3914,16 +4020,16 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
           fprintf (vect_dump, "supported: ");
           print_gimple_stmt (vect_dump, stmt, 0, TDF_SLIM);
         }
-
+   
        return false;
     }
 
   if (!PURE_SLP_STMT (stmt_info))
     {
-      /* Groups of strided accesses whose size is not a power of 2 are not
-         vectorizable yet using loop-vectorization. Therefore, if this stmt
-         feeds non-SLP-able stmts (i.e., this stmt has to be both SLPed and
-         loop-based vectorized), the loop cannot be vectorized.  */
+      /* Groups of strided accesses whose size is not a power of 2 are not 
+         vectorizable yet using loop-vectorization. Therefore, if this stmt 
+	 feeds non-SLP-able stmts (i.e., this stmt has to be both SLPed and 
+	 loop-based vectorized), the loop cannot be vectorized.  */
       if (STMT_VINFO_STRIDED_ACCESS (stmt_info)
           && exact_log2 (DR_GROUP_SIZE (vinfo_for_stmt (
                                         DR_GROUP_FIRST_DR (stmt_info)))) == -1)
@@ -3938,7 +4044,7 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vectorize)
           return false;
         }
     }
-
+     
   return true;
 }
 
@@ -3957,8 +4063,6 @@ vect_transform_stmt (gimple stmt, gimple_stmt_iterator *gsi,
   stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
   gimple orig_stmt_in_pattern;
   bool done;
-  loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
-  struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
 
   switch (STMT_VINFO_TYPE (stmt_info))
     {
@@ -4045,19 +4149,23 @@ vect_transform_stmt (gimple stmt, gimple_stmt_iterator *gsi,
   /* Handle inner-loop stmts whose DEF is used in the loop-nest that
      is being vectorized, but outside the immediately enclosing loop.  */
   if (vec_stmt
-      && nested_in_vect_loop_p (loop, stmt)
+      && STMT_VINFO_LOOP_VINFO (stmt_info)
+      && nested_in_vect_loop_p (LOOP_VINFO_LOOP (
+                                STMT_VINFO_LOOP_VINFO (stmt_info)), stmt)
       && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type
       && (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_outer
-          || STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_outer_by_reduction))
+          || STMT_VINFO_RELEVANT (stmt_info) == 
+                                           vect_used_in_outer_by_reduction))
     {
-      struct loop *innerloop = loop->inner;
+      struct loop *innerloop = LOOP_VINFO_LOOP (
+                                STMT_VINFO_LOOP_VINFO (stmt_info))->inner;
       imm_use_iterator imm_iter;
       use_operand_p use_p;
       tree scalar_dest;
       gimple exit_phi;
 
       if (vect_print_dump_info (REPORT_DETAILS))
-       fprintf (vect_dump, "Record the vdef for outer-loop vectorization.");
+        fprintf (vect_dump, "Record the vdef for outer-loop vectorization.");
 
       /* Find the relevant loop-exit phi-node, and reord the vec_stmt there
         (to be used when vectorizing outer-loop stmts that use the DEF of
@@ -4137,7 +4245,8 @@ vect_remove_stores (gimple first_stmt)
    Create and initialize a new stmt_vec_info struct for STMT.  */
 
 stmt_vec_info
-new_stmt_vec_info (gimple stmt, loop_vec_info loop_vinfo)
+new_stmt_vec_info (gimple stmt, loop_vec_info loop_vinfo, 
+                   bb_vec_info bb_vinfo)
 {
   stmt_vec_info res;
   res = (stmt_vec_info) xcalloc (1, sizeof (struct _stmt_vec_info));
@@ -4145,6 +4254,7 @@ new_stmt_vec_info (gimple stmt, loop_vec_info loop_vinfo)
   STMT_VINFO_TYPE (res) = undef_vec_info_type;
   STMT_VINFO_STMT (res) = stmt;
   STMT_VINFO_LOOP_VINFO (res) = loop_vinfo;
+  STMT_VINFO_BB_VINFO (res) = bb_vinfo;
   STMT_VINFO_RELEVANT (res) = vect_unused_in_scope;
   STMT_VINFO_LIVE_P (res) = false;
   STMT_VINFO_VECTYPE (res) = NULL;
@@ -4267,23 +4377,30 @@ get_vectype_for_scalar_type (tree scalar_type)
 /* Function vect_is_simple_use.
 
    Input:
-   LOOP - the loop that is being vectorized.
-   OPERAND - operand of a stmt in LOOP.
+   LOOP_VINFO - the vect info of the loop that is being vectorized.
+   BB_VINFO - the vect info of the basic block that is being vectorized.
+   OPERAND - operand of a stmt in the loop or bb.
    DEF - the defining stmt in case OPERAND is an SSA_NAME.
 
    Returns whether a stmt with OPERAND can be vectorized.
-   Supportable operands are constants, loop invariants, and operands that are
-   defined by the current iteration of the loop. Unsupportable operands are 
-   those that are defined by a previous iteration of the loop (as is the case
-   in reduction/induction computations).  */
+   For loops, supportable operands are constants, loop invariants, and operands 
+   that are defined by the current iteration of the loop. Unsupportable 
+   operands are those that are defined by a previous iteration of the loop (as 
+   is the case in reduction/induction computations).
+   For basic blocks, supportable operands are constants and bb invariants.
+   For now, operands defined outside the basic block are not supported.  */
 
 bool
-vect_is_simple_use (tree operand, loop_vec_info loop_vinfo, gimple *def_stmt,
+vect_is_simple_use (tree operand, loop_vec_info loop_vinfo, 
+                    bb_vec_info bb_vinfo, gimple *def_stmt,
 		    tree *def, enum vect_def_type *dt)
 { 
   basic_block bb;
   stmt_vec_info stmt_vinfo;
-  struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
+  struct loop *loop = NULL;
+  
+  if (loop_vinfo)
+    loop = LOOP_VINFO_LOOP (loop_vinfo);
 
   *def_stmt = NULL;
   *def = NULL_TREE;
@@ -4299,6 +4416,7 @@ vect_is_simple_use (tree operand, loop_vec_info loop_vinfo, gimple *def_stmt,
       *dt = vect_constant_def;
       return true;
     }
+    
   if (is_gimple_min_invariant (operand))
     {
       *def = operand;
@@ -4312,6 +4430,7 @@ vect_is_simple_use (tree operand, loop_vec_info loop_vinfo, gimple *def_stmt,
         fprintf (vect_dump, "non-associatable copy.");
       operand = TREE_OPERAND (operand, 0);
     }
+    
   if (TREE_CODE (operand) != SSA_NAME)
     {
       if (vect_print_dump_info (REPORT_DETAILS))
@@ -4343,7 +4462,10 @@ vect_is_simple_use (tree operand, loop_vec_info loop_vinfo, gimple *def_stmt,
     }
 
   bb = gimple_bb (*def_stmt);
-  if (!flow_bb_inside_loop_p (loop, bb))
+
+  if ((loop && !flow_bb_inside_loop_p (loop, bb))
+      || (!loop && bb != BB_VINFO_BB (bb_vinfo))
+      || (!loop && gimple_code (*def_stmt) == GIMPLE_PHI)) 
     *dt = vect_external_def;
   else
     {