2006-01-18 Richard Henderson <rth@redhat.com>

	    Jakub Jelinek  <jakub@redhat.com>
            Diego Novillo  <dnovillo@redhat.com>

	* libgomp: New directory.
	* Makefile.def: Add target_module libgomp.
	* Makefile.in: Regenerate.
	* configure.in (target_libraries): Add target-libgomp.
	* configure: Regenerate.


contrib/

2006-01-18  Richard Henderson  <rth@redhat.com>
            Diego Novillo  <dnovillo@redhat.com>

	* gcc_update (files_and_dependencies): Add libgomp files.


gcc/

2006-01-18  Richard Henderson  <rth@redhat.com>
            Aldy Hernandez  <aldyh@redhat.com>
	    Jakub Jelinek  <jakub@redhat.com>
            Diego Novillo  <dnovillo@redhat.com>

	* omp-low.c: New file.
	* c-omp.c: New file.

2006-01-18  Richard Henderson  <rth@redhat.com>
	    Jakub Jelinek  <jakub@redhat.com>
            Diego Novillo  <dnovillo@redhat.com>

	* doc/invoke.texi: Document -fopenmp.
	* tree-dump.h (debug_function): Declare.

	* hooks.c (hook_bool_tree_bool_false): New function.
	(hook_tree_tree_null): Remove.
	(hook_tree_tree_tree_null): New.
	* hooks.h: Update to match.

	* tree-pretty-print.c (debug_tree_chain): New.
	(print_generic_expr): Handle TDF_CHAIN.
	(dump_generic_node): Handle BLOCK.
	Do not abort with incomplete SWITCH_EXPRs.
	Do not dump body of an OpenMP directive if TDF_SLIM is given.
	<case OMP_PARALLEL, OMP_FOR, OMP_SECTIONS>: Don't
	print space after directive name.
	<OMP_FOR>: Handle printing OMP_FOR_PRE_BODY.
	Handle OMP_MASTER and OMP_ORDERED.
	Handle printing of OMP_BODY just in one place, goto
	dump_omp_body in the rest of OMP_* nodes that have
	OMP_BODY.
	Don't handle clause nodes here.  Update omp statements to
	use dump_omp_clauses.
	Handle OMP_SINGLE, OMP_SECTIONS, OMP_SECTION,
	OMP_CLAUSE_ORDERED, OMP_CLAUSE_SCHEDULE, OMP_ATOMIC,
	OMP_CRITICAL, OMP_CLAUSE_NOWAIT, GOMP_CLAUSE_IF,
	GOMP_CLAUSE_NUM_THREADS, GOMP_FOR, GOMP_CLAUSE_SHARED,
	GOMP_CLAUSE_FIRSTPRIVATE, GOMP_CLAUSE_LASTPRIVATE,
	GOMP_CLAUSE_COPYIN and GOMP_CLAUSE_COPYPRIVATE.
	Adjust output for GOMP_PARALLEL.
	(dump_omp_clauses): New.
	(print_declaration): Dump DECL_VALUE_EXPR.
	(op_symbol_1): Split out of op_symbol.
	(dumping_stmts): Remove.  Update all users.

	* cgraph.c (cgraph_analyze_queue): New.
	(cgraph_add_new_function): New.
	* cgraph.h (cgraph_analyze_queue): Declare.
	(cgraph_add_new_function): Declare.
	(cgraph_lower_function): Remove.

	* tree.c (walk_tree): Walk OMP_CLAUSE_CHAIN of OMP_CLAUSE_*
	nodes.  Use switch for all nodes, handle most of IS_EXPR_CODE_CLASS
	and TYPE_P nodes in its default clause.
	(empty_body_p): New.
	(tree_range_check_failed): New.
	(build5_stat): New.

	* tree.h (OMP_CLAUSE_REDUCTION_INIT,
	OMP_CLAUSE_REDUCTION_MERGE,
	OMP_CLAUSE_REDUCTION_PLACEHOLDER,
	OMP_CLAUSE_PRIVATE_DEBUG,
	OMP_CLAUSE_LASTPRIVATE_FIRSTPRIVATE, OMP_FOR_PRE_BODY,
	OMP_MASTER_BODY, OMP_ORDERED_BODY OMP_BODY,
	OMP_CLAUSES, OMP_CLAUSE_DECL, OMP_CLAUSE_DEFAULT_KIND,
	OMP_CLAUSE_CHAIN, OMP_CLAUSE_OUTER_DECL,
	OMP_CLAUSE_INNER_DECL, OMP_CLAUSE_NUM_THREADS_EXPR,
	OMP_CLAUSE_IF_EXPR, OMP_CLAUSE_SCHEDULE_CHUNK_EXPR,
	OMP_CLAUSE_SCHEDULE_CHUNK_SIZE. OMP_PARALLEL_VAR_INIT,
	OMP_PARALLEL_VAR_REDUC, OMP_FOR_VAR_INIT,
	OMP_FOR_VAR_LAST, OMP_FOR_VAR_REDUC,
	OMP_SECTIONS_VAR_INIT, OMP_SECTIONS_VAR_LAST,
	OMP_SECTIONS_VAR_REDUC, OMP_CLAUSE_REDUCTION_CODE
	OMP_SINGLE_CLAUSES, OMP_SINGLE_BODY,
	OMP_CLAUSE_SCHEDULE_CHUNK_SIZE, OMP_SECTION_BODY,
	OMP_CRITICAL_NAME, OMP_CRITICAL_BODY): New.
	(TREE_RANGE_CHECK): New.
	(empty_body_p): Declare.
	(enum omp_clause_default_kind): New.
	(build_string_literal): Declare.
	(enum omp_clause_schedule_kind, OMP_CLAUSE_SCHEDULE_KIND): New.
	(build5_stat, build5): Declare.

	* tree-pass.h (TDF_CHAIN): Define.
	* tree-pass.h (PROP_gimple_lomp): Define.
	(pass_lower_omp): Declare.

	* diagnostic.h (debug_tree_chain): Declare.

	* builtins.c (get_builtin_sync_mode): Use 0 as last argument to
	mode_for_size.
	(expand_builtin): Handle sync BUILT_IN_*_16 builtins.
	* builtins.c (build_string_literal): Make extern.

	* gcc.c (include_spec_function): New.
	(static_spec_functions): Add it.
	(main): Move load of libgomp.spec ...
	(LINK_COMMAND_SPEC): ... here.
	(link_gomp_spec): New.
	(static_specs): Include it.
	(LINK_COMMAND_SPEC): Add link_gomp.
	(GOMP_SELF_SPECS): New.
	(driver_self_specs): Include it.
	(switch_matches): Don't mark inline.
	(main): Load libgomp.spec.

	* tree-gimple.c (is_gimple_stmt): True for OMP_MASTER,
	OMP_ORDERED, OMP_CRITICAL, OMP_SECTIONS, OMP_SECTION,
	and OMP_SINGLE, OMP_FOR and OMP_PARALLEL.

	* tree-gimple.h (enum omp_parallel): Declare.
	(determine_parallel_type): Declare.
	(omp_firstprivatize_variable): Declare.
	(omp_reduction_init): Declare.
	(diagnose_omp_structured_block_errors): Declare.
	(struct walk_stmt_info): Add want_return_expr.
	(struct walk_stmt_info): Add want_bind_expr, want_locations.
	(find_omp_clause): Declare.
	(insert_field_into_struct): Declare.
	(struct walk_stmt_info): Move from tree-nested.c
	(walk_stmts): Declare.

	* c-cppbuiltin.c (c_cpp_builtins): If -fopenmp, #define _OPENMP
	to 200505.

	* cgraphunit.c (cgraph_lower_function): Make static.
	(cgraph_finalize_pending_functions): New.
	(cgraph_finalize_function): Call it.
	(cgraph_finalize_compilation_unit): Likewise.

	* builtin-types.def (BT_I16, BT_FN_I16_VPTR_I16,
	BT_FN_BOOL_VPTR_I16_I16, BT_FN_I16_VPTR_I16_I16): Add.
	(BT_FN_UINT_UINT): New.
	(DEF_FUNCTION_TYPE_6, DEF_FUNCTION_TYPE_7,
	DEF_FUNCTION_TYPE_VAR_4): Document.
	(BT_PTR_LONG, BT_PTR_PTR, BT_FN_BOOL, BT_FN_INT,
	BT_FN_VOID_PTRPTR, BT_PTR_FN_VOID_PTR,
	BT_FN_BOOL_LONGPTR_LONGPTR, BT_FN_VOID_OMPFN_PTR_UINT,
	BT_FN_VOID_OMPFN_PTR_UINT_UINT,
	BT_FN_BOOL_LONG_LONG_LONG_LONGPTR_LONGPTR,
	BT_FN_BOOL_LONG_LONG_LONG_LONG_LONGPTR_LONGPTR,
	BT_FN_VOID_OMPFN_PTR_UINT_LONG_LONG_LONG,
	BT_FN_VOID_OMPFN_PTR_UINT_LONG_LONG_LONG_LONG): New.

	* builtins.def: Update DEF_BUILTIN comment to include COND argument.
	Move all DEF_SYNC_BUILTIN () and DEF_GOMP_BUILTIN () builtins
	into separate files.
	(DEF_GOMP_BUILTIN): New.
	(BUILT_IN_OMP_GET_THREAD_NUM, BUILT_IN_GOMP_BARRIER,
	BUILT_IN_GOMP_CRITICAL_START, BUILT_IN_GOMP_CRITICAL_END,
	BUILT_IN_GOMP_CRITICAL_NAME_START, BUILT_IN_GOMP_CRITICAL_NAME_END,
	BUILT_IN_GOMP_LOOP_STATIC_START, BUILT_IN_GOMP_LOOP_DYNAMIC_START,
	BUILT_IN_GOMP_LOOP_GUIDED_START, BUILT_IN_GOMP_LOOP_RUNTIME_START,
	BUILT_IN_GOMP_LOOP_ORDERED_STATIC_START,
	BUILT_IN_GOMP_LOOP_ORDERED_DYNAMIC_START,
	BUILT_IN_GOMP_LOOP_ORDERED_GUIDED_START,
	BUILT_IN_GOMP_LOOP_ORDERED_RUNTIME_START,
	BUILT_IN_GOMP_LOOP_STATIC_NEXT, BUILT_IN_GOMP_LOOP_DYNAMIC_NEXT,
	BUILT_IN_GOMP_LOOP_GUIDED_NEXT, BUILT_IN_GOMP_LOOP_RUNTIME_NEXT,
	BUILT_IN_GOMP_LOOP_ORDERED_STATIC_NEXT,
	BUILT_IN_GOMP_LOOP_ORDERED_DYNAMIC_NEXT,
	BUILT_IN_GOMP_LOOP_ORDERED_GUIDED_NEXT,
	BUILT_IN_GOMP_LOOP_ORDERED_RUNTIME_NEXT,
	BUILT_IN_GOMP_PARALLEL_LOOP_STATIC_START,
	BUILT_IN_GOMP_PARALLEL_LOOP_DYNAMIC_START,
	BUILT_IN_GOMP_PARALLEL_LOOP_GUIDED_START,
	BUILT_IN_GOMP_PARALLEL_LOOP_RUNTIME_START,
	BUILT_IN_GOMP_LOOP_END, BUILT_IN_GOMP_LOOP_END_NOWAIT,
	BUILT_IN_GOMP_ORDERED_START, BUILT_IN_GOMP_ORDERED_END,
	BUILT_IN_GOMP_PARALLEL_START, BUILT_IN_GOMP_PARALLEL_END,
	BUILT_IN_GOMP_SECTIONS_START, BUILT_IN_GOMP_SECTIONS_NEXT,
	BUILT_IN_GOMP_PARALLEL_SECTIONS_START, BUILT_IN_GOMP_SECTIONS_END,
	BUILT_IN_GOMP_SECTIONS_END_NOWAIT, BUILT_IN_GOMP_SINGLE_START,
	BUILT_IN_GOMP_SINGLE_COPY_START, BUILT_IN_GOMP_SINGLE_COPY_END): New.
	* sync-builtins.def: New file, moved from builtins.def.
	* omp-builtins.def: New file, moved from builtins.def.

	* c-objc-common.h (LANG_HOOKS_OMP_PREDETERMINED_SHARING): Redefine.

	* gimple-low.c (lower_function_body): Clear data.
	(lower_stmt): Do not handle COMPOUND_EXPR.
	Remove call to print_node_brief.

	* c-tree.h (c_finish_omp_clauses): New prototype.
	(C_DECL_THREADPRIVATE_P): Define.
	(lookup_name_no_remap, c_omp_remap_private): Remove
	(c_begin_omp_parallel, c_finish_omp_parallel): Update.
	(check_for_loop_decls): Update decl.
	(lookup_name_no_remap, c_omp_remap_private): Declare.
	(build_indirect_ref, build_modify_expr, pushdecl,
	pushdecl_top_level): Move to c-common.h.

	* dwarf2out.c (loc_descriptor_from_tree_1): Don't set unsignedp
	before the switch, but just in the 2 places that need it.

	* c-decl.c (diagnose_mismatched_decls): Do not check for
	mismatched thread-local attributes when OLDDECL is marked
	threadprivate and NEWDECL has no thread-local attributes.
	(merge_decls): Merge C_DECL_THREADPRIVATE_P.
	(c_gimple_diagnostics_recursively): Rename from
	c_warn_unused_result_recursively.  Invoke
	diagnose_omp_structured_block_errors.
	(check_for_loop_decls): Return a singular decl found.

	* langhooks.c (lhd_omp_predetermined_sharing): Return
	OMP_CLAUSE_DEFAULT_SHARED for DECL_ARTIFICIAL decls.
	(lhd_omp_firstprivatize_type_sizes): New.
	(lhd_omp_assignment): New.
	(lhd_omp_predetermined_sharing): New.

	* langhooks.h (struct gimplify_omp_ctx): Forward declare.
	(struct lang_hooks_for_types): Add
	omp_firstprivatize_type_sizes, omp_privatize_by_reference,
	omp_predetermined_sharing, omp_disregard_value_expr,
	omp_private_debug_clause, omp_clause_default_ctor,
	omp_clause_copy_ctor, omp_clause_assign_op, omp_clause_dtor.

	(c_finish_omp_clauses): New.
	(c_finish_bc_stmt): Diagnose break within omp for.
	(c_begin_omp_parallel, c_finish_omp_parallel): New.
	(build_unary_op): Return error_mark after reporting
	a readonly_error.
	(build_modify_expr): Likewise.

	* gimplify.c: Include optabs.h and pointer-set.h.
	(enum gimplify_omp_var_data): Declare.
	(struct gimplify_omp_ctx): Declare.
	(struct gimplify_ctx): Add fields prev_context, combined_pre_p
	and combined_ctxp.
	(gimplify_ctxp, gimplify_omp_ctxp): New local variables.
	(push_gimplify_context, pop_gimplify_context): Allow nesting.
	(splay_tree_compare_decl_uid): New.
	(new_omp_context): New.
	(delete_omp_context): New.
	(gimple_add_tmp_var): Call omp_add_variable.
	(gimplify_bind_expr): Likewise.
	(gimplify_var_or_parm_decl): If omp_notice_variable returned
	true, disregard DECL_VALUE_EXPR on the decl if any.
	(gimplify_expr_in_ctx): New.
	(omp_firstprivatize_variable, omp_firstprivatize_type_sizes
	omp_add_variable, omp_notice_variable, omp_is_private
	gimplify_scan_omp_clauses, gimplify_adjust_omp_clauses_1
	gimplify_adjust_omp_clauses, gimplify_omp_parallel
	gimplify_omp_for, gimplify_omp_workshare, goa_lhs_expr_p
	gimplify_omp_atomic_fetch_op, goa_stabilize_expr
	gimplify_omp_atomic_pipeline, gimplify_omp_atomic_mutex
	gimplify_omp_atomic): New.
	(gimplify_expr): Handle OMP_PARALLEL, OMP_FOR, OMP_SECTIONS,
	OMP_SINGLE, OMP_SECTION, OMP_MASTER, OMP_ORDERED,
	OMP_CRITICAL and OMP_ATOMIC.
	(gimplify_body): Verify gimplify_ctxp is empty after gimplification.

	* c-pragma.h (enum pragma_kind): Add
	PRAGMA_OMP_ATOMIC, PRAGMA_OMP_BARRIER,
	PRAGMA_OMP_CRITICAL, PRAGMA_OMP_FLUSH, PRAGMA_OMP_FOR,
	PRAGMA_OMP_MASTER, PRAGMA_OMP_ORDERED,
	PRAGMA_OMP_PARALLEL, PRAGMA_OMP_PARALLEL_FOR,
	PRAGMA_OMP_PARALLEL_SECTIONS, PRAGMA_OMP_SECTION,
	PRAGMA_OMP_SECTIONS, PRAGMA_OMP_SINGLE,
	PRAGMA_OMP_THREADPRIVATE.

	* tree.def (OMP_PARALLEL, OMP_FOR, OMP_SECTIONS,
	OMP_SINGLE, OMP_SECTION, OMP_MASTER, OMP_ORDERED,
	OMP_CRITICAL, OMP_ATOMIC, OMP_CLAUSE_PRIVATE,
	OMP_CLAUSE_SHARED, OMP_CLAUSE_FIRSTPRIVATE,
	OMP_CLAUSE_LASTPRIVATE, OMP_CLAUSE_REDUCTION,
	OMP_CLAUSE_COPYIN, OMP_CLAUSE_COPYPRIVATE,
	OMP_CLAUSE_IF, OMP_CLAUSE_NUM_THREADS,
	OMP_CLAUSE_SCHEDULE, OMP_CLAUSE_NOWAIT,
	OMP_CLAUSE_ORDERED, OMP_CLAUSE_DEFAULT): Define.

	* print-tree.c (print_node): Dump DECL_VALUE_EXPR.

	* tree-ssa-dce.c (find_control_dependence): Do not assume that
	ENTRY_BLOCK_PTR->next_bb == single_succ (ENTRY_BLOCK_PTR).

	* tree-nested.c (convert_call_expr): Call walk_body on OMP_BODY for
	OpenMP directives.
	(struct nesting_info): Add field_map,
	suppress_expansion, debug_var_chain.
	(create_nesting_tree): Initialize them.
	(lookup_field_for_decl): Use field_map.
	(get_nonlocal_debug_decl, get_local_debug_decl): New.
	(convert_local_omp_clauses): New.
	(finalize_nesting_tree_1): Add debug_var_chain to toplevel block.
	(walk_body): Split out of walk_function.
	(convert_nonlocal_omp_clauses, convert_local_omp_clauses): New.
	(convert_nonlocal_reference): Handle omp statements.
	(convert_local_reference): Likewise.
	(unnest_nesting_tree_1): Split out of finalize_nesting_tree_1.
	(unnest_nesting_tree): New.
	(lower_nested_functions): Call it.
	(insert_field_into_struct): Make extern.
	(struct walk_stmt_info): Move to tree-gimple.h.
	(walk_stmts): Make extern.

	* omp-builtins.def: New file.

	* tree-iterator.c (expr_only): Clarify comment.

	* c-common.h (pushdecl_top_level, pushdecl,
	build_modify_expr, build_indirect_ref,
	c_finish_omp_master, c_finish_omp_critical,
	c_finish_omp_ordered, c_finish_omp_barrier,
	c_finish_omp_atomic, c_finish_omp_flush,
	c_finish_omp_for, c_split_parallel_clauses,
	omp_clause_default_kind, c_omp_sharing_predetermined,
	c_omp_remap_decl): Declare.

	* Makefile.in (BUILTINS_DEF): Add omp-builtins.def.
	(OBJS-common): Add omp-low.o.
	(c-omp.o, omp-low.o): Add.
	(gimplify.o): Add dependency on $(OPTABS_H).
	(GTFILES): Add omp-low.c.
	(gt-stringpool.h): Add.

	* tree-cfg.c (set_bb_for_stmt): Do not update the
	block-to-labels map if we are currently expanding to RTL.
	(tree_node_can_be_shared): Remove unnecessary CONSTANT_CLASS_P
	checks.
	Handle IDENTIFIER_NODE.
	(tree_verify_flow_info): Do not ICE when emitting error
	messages about invalid labels.
	(dump_function_to_file): Reset CFUN before emitting the body
	of the function.
	(debug_function): New.

	* passes.c (init_optimization_passes): Schedule
	pass_lower_omp.

	* langhooks-def.h (lhd_omp_predetermined_sharing,
	lhd_omp_assignment, lhd_omp_firstprivatize_type_sizes):
	Declare.
	(LANG_HOOKS_OMP_FIRSTPRIVATIZE_TYPE_SIZES): Define.
	(LANG_HOOKS_FOR_TYPES_INITIALIZER): Use it.
	(LANG_HOOKS_OMP_PRIVATIZE_BY_REFERENCE,
	LANG_HOOKS_OMP_PREDETERMINED_SHARING,
	LANG_HOOKS_OMP_DISREGARD_VALUE_EXPR,
	LANG_HOOKS_OMP_PRIVATE_DEBUG_CLAUSE,
	LANG_HOOKS_OMP_CLAUSE_DEFAULT_CTOR,
	LANG_HOOKS_OMP_CLAUSE_COPY_CTOR,
	LANG_HOOKS_OMP_CLAUSE_ASSIGN_OP,
	LANG_HOOKS_OMP_CLAUSE_DTOR): Define.
	(LANG_HOOK_DECLS): Use them.


2006-01-18  Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
	    Richard Henderson  <rth@redhat.com>
	    Jakub Jelinek  <jakub@redhat.com>
            Diego Novillo  <dnovillo@redhat.com>

	* c-parser.c (pragma_omp_clause): Define.
	(c_parser_declaration_or_fndef): Document OpenMP syntax.
	(c_parser_compound_statement): Likewise.
	(c_parser_statement): Likewise.
	(c_parser_pragma): Handle omp pragmas.
	(OMP_FOR_CLAUSE_MASK, OMP_SECTIONS_CLAUSE_MASK,
	OMP_PARALLEL_CLAUSE_MASK, OMP_SINGLE_CLAUSE_MASK): Define.
	(c_parser_omp_clause_name, check_no_duplicate_clause,
	c_parser_omp_variable_list,
	c_parser_omp_var_list_parens, c_parser_omp_clause_copyin,
	c_parser_omp_clause_copyprivate,
	c_parser_omp_clause_default,
	c_parser_omp_clause_firstprivate, c_parser_omp_clause_if,
	c_parser_omp_clause_lastprivate,
	c_parser_omp_clause_nowait,
	c_parser_omp_clause_num_threads,
	c_parser_omp_clause_ordered, c_parser_omp_clause_private,
	c_parser_omp_clause_reduction,
	c_parser_omp_clause_schedule, c_parser_omp_clause_shared,
	c_parser_omp_all_clauses, c_parser_omp_structured_block,
	c_parser_omp_atomic, c_parser_omp_barrier,
	c_parser_omp_critical, c_parser_omp_flush,
	c_parser_omp_for_loop, c_parser_omp_for,
	c_parser_omp_master, c_parser_omp_ordered,
	c_parser_omp_sections_scope, c_parser_omp_sections,
	c_parser_omp_parallel, c_parser_omp_single,
	c_parser_omp_construct, c_parser_omp_threadprivate): New.
	* c-pragma.c (init_pragma): Do omp pragma registration here.
	* c.opt (fopenmp): New flag.


2006-01-18  Eric Christopher  <echristo@apple.com>

	* gcc.c (GOMP_SELF_SPECS): Bracket in #ifndef/#endif.
	* config/darwin.h (GOMP_SELF_SPECS): Define.


testsuite/

2006-01-18  Richard Henderson  <rth@redhat.com>
            Aldy Hernandez  <aldyh@redhat.com>
	    Jakub Jelinek  <jakub@redhat.com>
            Diego Novillo  <dnovillo@redhat.com>
            Uros Bizjak  <uros@kss-loka.si>

	* testsuite/gcc.dg/gomp: New directory.




git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk@109902 138bc75d-0d04-0410-961f-82ee72b054a4

1 files changed, 1208 insertions, 24 deletions

diff --git a/gcc/gimplify.c b/gcc/gimplify.c
index 8783dc65df8..acd0468a338 100644
--- a/gcc/gimplify.c
+++ b/gcc/gimplify.c
@@ -46,21 +46,67 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA
 #include "ggc.h"
 #include "toplev.h"
 #include "target.h"
+#include "optabs.h"
+#include "pointer-set.h"
 
-static struct gimplify_ctx
+
+enum gimplify_omp_var_data
+{
+  GOVD_SEEN = 1,
+  GOVD_EXPLICIT = 2,
+  GOVD_SHARED = 4,
+  GOVD_PRIVATE = 8,
+  GOVD_FIRSTPRIVATE = 16,
+  GOVD_LASTPRIVATE = 32,
+  GOVD_REDUCTION = 64,
+  GOVD_LOCAL = 128,
+  GOVD_DEBUG_PRIVATE = 256,
+  GOVD_DATA_SHARE_CLASS = (GOVD_SHARED | GOVD_PRIVATE | GOVD_FIRSTPRIVATE
+			   | GOVD_LASTPRIVATE | GOVD_REDUCTION | GOVD_LOCAL)
+};
+
+struct gimplify_omp_ctx
 {
+  struct gimplify_omp_ctx *outer_context;
+  splay_tree variables;
+  struct pointer_set_t *privatized_types;
+  location_t location;
+  enum omp_clause_default_kind default_kind;
+  bool is_parallel;
+};
+
+struct gimplify_ctx
+{
+  struct gimplify_ctx *prev_context;
+
   tree current_bind_expr;
   tree temps;
   tree conditional_cleanups;
   tree exit_label;
   tree return_temp;
+  
   VEC(tree,heap) *case_labels;
   /* The formal temporary table.  Should this be persistent?  */
   htab_t temp_htab;
+
   int conditions;
   bool save_stack;
   bool into_ssa;
-} *gimplify_ctxp;
+
+  /* When gimplifying combined omp parallel directives (omp parallel
+     loop and omp parallel sections), any prefix code needed to setup
+     the associated worksharing construct needs to be emitted in the
+     pre-queue of its parent parallel, otherwise the lowering process
+     will move that code to the child function.  Similarly, we need to
+     move up to the gimplification context of the parent parallel
+     directive so temporaries are declared in the right context.  */
+  tree *combined_pre_p;
+  struct gimplify_ctx *combined_ctxp;
+};
+
+static struct gimplify_ctx *gimplify_ctxp;
+static struct gimplify_omp_ctx *gimplify_omp_ctxp;
+
 
 
 /* Formal (expression) temporary table handling: Multiple occurrences of
@@ -116,14 +162,14 @@ gimple_tree_eq (const void *p1, const void *p2)
 void
 push_gimplify_context (void)
 {
-  gcc_assert (!gimplify_ctxp);
-  gimplify_ctxp
-    = (struct gimplify_ctx *) xcalloc (1, sizeof (struct gimplify_ctx));
+  struct gimplify_ctx *c;
+
+  c = (struct gimplify_ctx *) xcalloc (1, sizeof (struct gimplify_ctx));
+  c->prev_context = gimplify_ctxp;
   if (optimize)
-    gimplify_ctxp->temp_htab
-      = htab_create (1000, gimple_tree_hash, gimple_tree_eq, free);
-  else
-    gimplify_ctxp->temp_htab = NULL;
+    c->temp_htab = htab_create (1000, gimple_tree_hash, gimple_tree_eq, free);
+
+  gimplify_ctxp = c;
 }
 
 /* Tear down a context for the gimplifier.  If BODY is non-null, then
@@ -133,28 +179,23 @@ push_gimplify_context (void)
 void
 pop_gimplify_context (tree body)
 {
+  struct gimplify_ctx *c = gimplify_ctxp;
   tree t;
 
-  gcc_assert (gimplify_ctxp && !gimplify_ctxp->current_bind_expr);
+  gcc_assert (c && !c->current_bind_expr);
+  gimplify_ctxp = c->prev_context;
 
-  for (t = gimplify_ctxp->temps; t ; t = TREE_CHAIN (t))
+  for (t = c->temps; t ; t = TREE_CHAIN (t))
     DECL_GIMPLE_FORMAL_TEMP_P (t) = 0;
 
   if (body)
-    declare_tmp_vars (gimplify_ctxp->temps, body);
+    declare_tmp_vars (c->temps, body);
   else
-    record_vars (gimplify_ctxp->temps);
-
-#if 0
-  if (!quiet_flag && optimize)
-    fprintf (stderr, " collisions: %f ",
-	     htab_collisions (gimplify_ctxp->temp_htab));
-#endif
+    record_vars (c->temps);
 
   if (optimize)
-    htab_delete (gimplify_ctxp->temp_htab);
-  free (gimplify_ctxp);
-  gimplify_ctxp = NULL;
+    htab_delete (c->temp_htab);
+  free (c);
 }
 
 static void
@@ -214,6 +255,48 @@ gimple_pop_condition (tree *pre_p)
     }
 }
 
+/* A stable comparison routine for use with splay trees and DECLs.  */
+
+static int
+splay_tree_compare_decl_uid (splay_tree_key xa, splay_tree_key xb)
+{
+  tree a = (tree) xa;
+  tree b = (tree) xb;
+
+  return DECL_UID (a) - DECL_UID (b);
+}
+
+/* Create a new omp construct that deals with variable remapping.  */
+
+static struct gimplify_omp_ctx *
+new_omp_context (bool is_parallel)
+{
+  struct gimplify_omp_ctx *c;
+
+  c = XCNEW (struct gimplify_omp_ctx);
+  c->outer_context = gimplify_omp_ctxp;
+  c->variables = splay_tree_new (splay_tree_compare_decl_uid, 0, 0);
+  c->privatized_types = pointer_set_create ();
+  c->location = input_location;
+  c->is_parallel = is_parallel;
+  c->default_kind = OMP_CLAUSE_DEFAULT_SHARED;
+
+  return c;
+}
+
+/* Destroy an omp construct that deals with variable remapping.  */
+
+static void
+delete_omp_context (struct gimplify_omp_ctx *c)
+{
+  splay_tree_delete (c->variables);
+  pointer_set_destroy (c->privatized_types);
+  XDELETE (c);
+}
+
+static void omp_add_variable (struct gimplify_omp_ctx *, tree, unsigned int);
+static bool omp_notice_variable (struct gimplify_omp_ctx *, tree, bool);
+
 /* A subroutine of append_to_statement_list{,_force}.  T is not NULL.  */
 
 static void
@@ -601,6 +684,16 @@ gimple_add_tmp_var (tree tmp)
     {
       TREE_CHAIN (tmp) = gimplify_ctxp->temps;
       gimplify_ctxp->temps = tmp;
+
+      /* Mark temporaries local within the nearest enclosing parallel.  */
+      if (gimplify_omp_ctxp)
+	{
+	  struct gimplify_omp_ctx *ctx = gimplify_omp_ctxp;
+	  while (ctx && !ctx->is_parallel)
+	    ctx = ctx->outer_context;
+	  if (ctx)
+	    omp_add_variable (ctx, tmp, GOVD_LOCAL | GOVD_SEEN);
+	}
     }
   else if (cfun)
     record_vars (tmp);
@@ -933,6 +1026,16 @@ gimplify_bind_expr (tree *expr_p, tree temp, tree *pre_p)
 	DECL_COMPLEX_GIMPLE_REG_P (t) = 1;
     }
 
+  /* Mark variables seen in this bind expr as locals.  */
+  if (gimplify_omp_ctxp)
+    {
+      struct gimplify_omp_ctx *ctx = gimplify_omp_ctxp;
+
+      for (t = BIND_EXPR_VARS (bind_expr); t ; t = TREE_CHAIN (t))
+	if (TREE_CODE (t) == VAR_DECL && !is_global_var (t))
+	  omp_add_variable (ctx, t, GOVD_LOCAL | GOVD_SEEN);
+    }
+
   gimple_push_bind_expr (bind_expr);
   gimplify_ctxp->save_stack = false;
 
@@ -1282,9 +1385,16 @@ static enum gimplify_status
 gimplify_case_label_expr (tree *expr_p)
 {
   tree expr = *expr_p;
+  struct gimplify_ctx *ctxp;
+
+  /* Invalid OpenMP programs can play Duff's Device type games with
+     #pragma omp parallel.  At least in the C front end, we don't
+     detect such invalid branches until after gimplification.  */
+  for (ctxp = gimplify_ctxp; ; ctxp = ctxp->prev_context)
+    if (ctxp->case_labels)
+      break;
 
-  gcc_assert (gimplify_ctxp->case_labels);
-  VEC_safe_push (tree, heap, gimplify_ctxp->case_labels, expr);
+  VEC_safe_push (tree, heap, ctxp->case_labels, expr);
   *expr_p = build1 (LABEL_EXPR, void_type_node, CASE_LABEL (expr));
   return GS_ALL_DONE;
 }
@@ -1491,6 +1601,10 @@ gimplify_var_or_parm_decl (tree *expr_p)
       return GS_ERROR;
     }
 
+  /* When within an OpenMP context, notice uses of variables.  */
+  if (gimplify_omp_ctxp && omp_notice_variable (gimplify_omp_ctxp, decl, true))
+    return GS_ALL_DONE;
+
   /* If the decl is an alias for another expression, substitute it now.  */
   if (DECL_HAS_VALUE_EXPR_P (decl))
     {
@@ -4017,6 +4131,1049 @@ gimplify_to_stmt_list (tree *stmt_p)
     }
 }
 
+/* Gimplify *EXPR_P as if it had been used inside the gimplification
+   context CTX_P.  The other arguments are as in gimplify_expr.  */
+
+static enum gimplify_status
+gimplify_expr_in_ctx (tree *expr_p, tree *pre_p, tree *post_p, 
+		      bool (* gimple_test_f) (tree), fallback_t fallback,
+		      struct gimplify_ctx *ctx_p,
+		      struct gimplify_omp_ctx *omp_ctx_p)
+{
+  enum gimplify_status ret;
+  struct gimplify_ctx *prev_ctxp;
+  struct gimplify_omp_ctx *prev_omp_ctxp;
+  
+  prev_ctxp = gimplify_ctxp;
+  gimplify_ctxp = ctx_p;
+  prev_omp_ctxp = gimplify_omp_ctxp;
+  gimplify_omp_ctxp = omp_ctx_p;
+  ret = gimplify_expr (expr_p, pre_p, post_p, gimple_test_f, fallback);
+  gimplify_ctxp = prev_ctxp;
+  gimplify_omp_ctxp = prev_omp_ctxp;
+
+  return ret;
+}
+
+/* Add FIRSTPRIVATE entries for DECL in the OpenMP the surrounding parallels
+   to CTX.  If entries already exist, force them to be some flavor of private.
+   If there is no enclosing parallel, do nothing.  */
+
+void
+omp_firstprivatize_variable (struct gimplify_omp_ctx *ctx, tree decl)
+{
+  splay_tree_node n;
+
+  if (decl == NULL || !DECL_P (decl))
+    return;
+
+  do
+    {
+      n = splay_tree_lookup (ctx->variables, (splay_tree_key)decl);
+      if (n != NULL)
+	{
+	  if (n->value & GOVD_SHARED)
+	    n->value = GOVD_FIRSTPRIVATE | (n->value & GOVD_SEEN);
+	  else
+	    return;
+	}
+      else if (ctx->is_parallel)
+	omp_add_variable (ctx, decl, GOVD_FIRSTPRIVATE);
+
+      ctx = ctx->outer_context;
+    }
+  while (ctx);
+}
+
+/* Similarly for each of the type sizes of TYPE.  */
+
+static void
+omp_firstprivatize_type_sizes (struct gimplify_omp_ctx *ctx, tree type)
+{
+  if (type == NULL || type == error_mark_node)
+    return;
+  type = TYPE_MAIN_VARIANT (type);
+
+  if (pointer_set_insert (ctx->privatized_types, type))
+    return;
+
+  switch (TREE_CODE (type))
+    {
+    case INTEGER_TYPE:
+    case ENUMERAL_TYPE:
+    case BOOLEAN_TYPE:
+    case CHAR_TYPE:
+    case REAL_TYPE:
+      omp_firstprivatize_variable (ctx, TYPE_MIN_VALUE (type));
+      omp_firstprivatize_variable (ctx, TYPE_MAX_VALUE (type));
+      break;
+
+    case ARRAY_TYPE:
+      omp_firstprivatize_type_sizes (ctx, TREE_TYPE (type));
+      omp_firstprivatize_type_sizes (ctx, TYPE_DOMAIN (type));
+      break;
+
+    case RECORD_TYPE:
+    case UNION_TYPE:
+    case QUAL_UNION_TYPE:
+      {
+	tree field;
+	for (field = TYPE_FIELDS (type); field; field = TREE_CHAIN (field))
+	  if (TREE_CODE (field) == FIELD_DECL)
+	    {
+	      omp_firstprivatize_variable (ctx, DECL_FIELD_OFFSET (field));
+	      omp_firstprivatize_type_sizes (ctx, TREE_TYPE (field));
+	    }
+      }
+      break;
+
+    case POINTER_TYPE:
+    case REFERENCE_TYPE:
+      omp_firstprivatize_type_sizes (ctx, TREE_TYPE (type));
+      break;
+
+    default:
+      break;
+    }
+
+  omp_firstprivatize_variable (ctx, TYPE_SIZE (type));
+  omp_firstprivatize_variable (ctx, TYPE_SIZE_UNIT (type));
+  lang_hooks.types.omp_firstprivatize_type_sizes (ctx, type);
+}
+
+/* Add an entry for DECL in the OpenMP context CTX with FLAGS.  */
+
+static void
+omp_add_variable (struct gimplify_omp_ctx *ctx, tree decl, unsigned int flags)
+{
+  splay_tree_node n;
+  unsigned int nflags;
+  tree t;
+
+  if (decl == error_mark_node || TREE_TYPE (decl) == error_mark_node)
+    return;
+
+  /* Never elide decls whose type has TREE_ADDRESSABLE set.  This means
+     there are constructors involved somewhere.  */
+  if (TREE_ADDRESSABLE (TREE_TYPE (decl))
+      || TYPE_NEEDS_CONSTRUCTING (TREE_TYPE (decl)))
+    flags |= GOVD_SEEN;
+
+  n = splay_tree_lookup (ctx->variables, (splay_tree_key)decl);
+  if (n != NULL)
+    {
+      /* We shouldn't be re-adding the decl with the same data
+	 sharing class.  */
+      gcc_assert ((n->value & GOVD_DATA_SHARE_CLASS & flags) == 0);
+      /* The only combination of data sharing classes we should see is
+	 FIRSTPRIVATE and LASTPRIVATE.  */
+      nflags = n->value | flags;
+      gcc_assert ((nflags & GOVD_DATA_SHARE_CLASS)
+		  == (GOVD_FIRSTPRIVATE | GOVD_LASTPRIVATE));
+      n->value = nflags;
+      return;
+    }
+
+  /* When adding a variable-sized variable, we have to handle all sorts
+     of additional bits of data: the pointer replacement variable, and 
+     the parameters of the type.  */
+  if (!TREE_CONSTANT (DECL_SIZE (decl)))
+    {
+      /* Add the pointer replacement variable as PRIVATE if the variable
+	 replacement is private, else FIRSTPRIVATE since we'll need the
+	 address of the original variable either for SHARED, or for the
+	 copy into or out of the context.  */
+      if (!(flags & GOVD_LOCAL))
+	{
+	  nflags = flags & GOVD_PRIVATE ? GOVD_PRIVATE : GOVD_FIRSTPRIVATE;
+	  nflags |= flags & GOVD_SEEN;
+	  t = DECL_VALUE_EXPR (decl);
+	  gcc_assert (TREE_CODE (t) == INDIRECT_REF);
+	  t = TREE_OPERAND (t, 0);
+	  gcc_assert (DECL_P (t));
+	  omp_add_variable (ctx, t, nflags);
+	}
+
+      /* Add all of the variable and type parameters (which should have
+	 been gimplified to a formal temporary) as FIRSTPRIVATE.  */
+      omp_firstprivatize_variable (ctx, DECL_SIZE_UNIT (decl));
+      omp_firstprivatize_variable (ctx, DECL_SIZE (decl));
+      omp_firstprivatize_type_sizes (ctx, TREE_TYPE (decl));
+
+      /* The variable-sized variable itself is never SHARED, only some form
+	 of PRIVATE.  The sharing would take place via the pointer variable
+	 which we remapped above.  */
+      if (flags & GOVD_SHARED)
+	flags = GOVD_PRIVATE | GOVD_DEBUG_PRIVATE
+		| (flags & (GOVD_SEEN | GOVD_EXPLICIT));
+
+      /* We're going to make use of the TYPE_SIZE_UNIT at least in the 
+	 alloca statement we generate for the variable, so make sure it
+	 is available.  This isn't automatically needed for the SHARED
+	 case, since we won't be allocating local storage then.  */
+      else
+	omp_notice_variable (ctx, TYPE_SIZE_UNIT (TREE_TYPE (decl)), true);
+    }
+  else if (lang_hooks.decls.omp_privatize_by_reference (decl))
+    {
+      gcc_assert ((flags & GOVD_LOCAL) == 0);
+      omp_firstprivatize_type_sizes (ctx, TREE_TYPE (decl));
+
+      /* Similar to the direct variable sized case above, we'll need the
+	 size of references being privatized.  */
+      if ((flags & GOVD_SHARED) == 0)
+	{
+	  t = TYPE_SIZE_UNIT (TREE_TYPE (TREE_TYPE (decl)));
+	  if (!TREE_CONSTANT (t))
+	    omp_notice_variable (ctx, t, true);
+	}
+    }
+
+  splay_tree_insert (ctx->variables, (splay_tree_key)decl, flags);
+}
+
+/* Record the fact that DECL was used within the OpenMP context CTX.
+   IN_CODE is true when real code uses DECL, and false when we should
+   merely emit default(none) errors.  Return true if DECL is going to
+   be remapped and thus DECL shouldn't be gimplified into its
+   DECL_VALUE_EXPR (if any).  */
+
+static bool
+omp_notice_variable (struct gimplify_omp_ctx *ctx, tree decl, bool in_code)
+{
+  splay_tree_node n;
+  unsigned flags = in_code ? GOVD_SEEN : 0;
+  bool ret = false, shared;
+
+  if (decl == error_mark_node || TREE_TYPE (decl) == error_mark_node)
+    return false;
+
+  /* Threadprivate variables are predetermined.  */
+  if (is_global_var (decl))
+    {
+      if (DECL_THREAD_LOCAL_P (decl))
+	return false;
+
+      if (DECL_HAS_VALUE_EXPR_P (decl))
+	{
+	  tree value = get_base_address (DECL_VALUE_EXPR (decl));
+
+	  if (value && DECL_P (value) && DECL_THREAD_LOCAL_P (value))
+	    return false;
+	}
+    }
+
+  n = splay_tree_lookup (ctx->variables, (splay_tree_key)decl);
+  if (n == NULL)
+    {
+      enum omp_clause_default_kind default_kind, kind;
+
+      if (!ctx->is_parallel)
+	goto do_outer;
+
+      /* ??? Some compiler-generated variables (like SAVE_EXPRs) could be
+	 remapped firstprivate instead of shared.  To some extent this is
+	 addressed in omp_firstprivatize_type_sizes, but not effectively.  */
+      default_kind = ctx->default_kind;
+      kind = lang_hooks.decls.omp_predetermined_sharing (decl);
+      if (kind != OMP_CLAUSE_DEFAULT_UNSPECIFIED)
+	default_kind = kind;
+
+      switch (default_kind)
+	{
+	case OMP_CLAUSE_DEFAULT_NONE:
+	  error ("%qs not specified in enclosing parallel",
+		 IDENTIFIER_POINTER (DECL_NAME (decl)));
+	  error ("%Henclosing parallel", &ctx->location);
+	  /* FALLTHRU */
+	case OMP_CLAUSE_DEFAULT_SHARED:
+	  flags |= GOVD_SHARED;
+	  break;
+	case OMP_CLAUSE_DEFAULT_PRIVATE:
+	  flags |= GOVD_PRIVATE;
+	  break;
+	default:
+	  gcc_unreachable ();
+	}
+
+      omp_add_variable (ctx, decl, flags);
+
+      shared = (flags & GOVD_SHARED) != 0;
+      ret = lang_hooks.decls.omp_disregard_value_expr (decl, shared);
+      goto do_outer;
+    }
+
+  shared = ((flags | n->value) & GOVD_SHARED) != 0;
+  ret = lang_hooks.decls.omp_disregard_value_expr (decl, shared);
+
+  /* If nothing changed, there's nothing left to do.  */
+  if ((n->value & flags) == flags)
+    return ret;
+  flags |= n->value;
+  n->value = flags;
+
+ do_outer:
+  /* If the variable is private in the current context, then we don't
+     need to propagate anything to an outer context.  */
+  if (flags & GOVD_PRIVATE)
+    return ret;
+  if (ctx->outer_context
+      && omp_notice_variable (ctx->outer_context, decl, in_code))
+    return true;
+  return ret;
+}
+
+/* Verify that DECL is private within CTX.  If there's specific information
+   to the contrary in the innermost scope, generate an error.  */
+
+static bool
+omp_is_private (struct gimplify_omp_ctx *ctx, tree decl)
+{
+  splay_tree_node n;
+
+  n = splay_tree_lookup (ctx->variables, (splay_tree_key)decl);
+  if (n != NULL)
+    {
+      if (n->value & GOVD_SHARED)
+	{
+	  if (ctx == gimplify_omp_ctxp)
+	    error ("iteration variable %qs should be private",
+		   IDENTIFIER_POINTER (DECL_NAME (decl)));
+	  n->value = GOVD_PRIVATE;
+	}
+      return true;
+    }
+
+  if (ctx->outer_context)
+    return omp_is_private (ctx->outer_context, decl);
+  else if (ctx->is_parallel)
+    return false;
+  else
+    return !is_global_var (decl);
+}
+
+/* Scan the OpenMP clauses in *LIST_P, installing mappings into a new
+   and previous omp contexts.  */
+
+static void
+gimplify_scan_omp_clauses (tree *list_p, tree *pre_p, bool in_parallel)
+{
+  struct gimplify_omp_ctx *ctx, *outer_ctx;
+  tree c;
+
+  ctx = new_omp_context (in_parallel);
+  outer_ctx = ctx->outer_context;
+
+  while ((c = *list_p) != NULL)
+    {
+      enum gimplify_status gs;
+      bool remove = false;
+      bool notice_outer = true;
+      unsigned int flags;
+      tree decl;
+
+      switch (TREE_CODE (c))
+	{
+	case OMP_CLAUSE_PRIVATE:
+	  flags = GOVD_PRIVATE | GOVD_EXPLICIT;
+	  notice_outer = false;
+	  goto do_add;
+	case OMP_CLAUSE_SHARED:
+	  flags = GOVD_SHARED | GOVD_EXPLICIT;
+	  goto do_add;
+	case OMP_CLAUSE_FIRSTPRIVATE:
+	  flags = GOVD_FIRSTPRIVATE | GOVD_EXPLICIT;
+	  goto do_add;
+	case OMP_CLAUSE_LASTPRIVATE:
+	  flags = GOVD_LASTPRIVATE | GOVD_SEEN | GOVD_EXPLICIT;
+	  goto do_add;
+	case OMP_CLAUSE_REDUCTION:
+	  flags = GOVD_REDUCTION | GOVD_SEEN | GOVD_EXPLICIT;
+	  goto do_add;
+
+	do_add:
+	  decl = OMP_CLAUSE_DECL (c);
+	  if (decl == error_mark_node || TREE_TYPE (decl) == error_mark_node)
+	    {
+	      remove = true;
+	      break;
+	    }
+	  omp_add_variable (ctx, decl, flags);
+	  if (TREE_CODE (c) == OMP_CLAUSE_REDUCTION
+	      && OMP_CLAUSE_REDUCTION_PLACEHOLDER (c))
+	    {
+	      omp_add_variable (ctx, OMP_CLAUSE_REDUCTION_PLACEHOLDER (c),
+				GOVD_LOCAL);
+	      gimplify_omp_ctxp = ctx;
+	      push_gimplify_context ();
+	      gimplify_stmt (&OMP_CLAUSE_REDUCTION_INIT (c));
+	      pop_gimplify_context (OMP_CLAUSE_REDUCTION_INIT (c));
+	      push_gimplify_context ();
+	      gimplify_stmt (&OMP_CLAUSE_REDUCTION_MERGE (c));
+	      pop_gimplify_context (OMP_CLAUSE_REDUCTION_MERGE (c));
+	      gimplify_omp_ctxp = outer_ctx;
+	    }
+	  if (notice_outer)
+	    goto do_notice;
+	  break;
+
+	case OMP_CLAUSE_COPYIN:
+	case OMP_CLAUSE_COPYPRIVATE:
+	  decl = OMP_CLAUSE_DECL (c);
+	  if (decl == error_mark_node || TREE_TYPE (decl) == error_mark_node)
+	    {
+	      remove = true;
+	      break;
+	    }
+	do_notice:
+	  if (outer_ctx)
+	    omp_notice_variable (outer_ctx, decl, true);
+	  break;
+
+	case OMP_CLAUSE_SCHEDULE:
+	  if (gimplify_ctxp->combined_pre_p)
+	    {
+	      gcc_assert (gimplify_omp_ctxp == outer_ctx);
+	      gs = gimplify_expr_in_ctx (&OMP_CLAUSE_SCHEDULE_CHUNK_EXPR (c),
+					 gimplify_ctxp->combined_pre_p, NULL,
+					 is_gimple_val, fb_rvalue,
+					 gimplify_ctxp->combined_ctxp,
+					 outer_ctx->outer_context);
+	      if (gs == GS_ERROR)
+		remove = true;
+	      break;
+	    }
+	  /* FALLTHRU */
+	case OMP_CLAUSE_IF:
+	case OMP_CLAUSE_NUM_THREADS:
+	  gs = gimplify_expr (&TREE_OPERAND (c, 0), pre_p, NULL,
+			      is_gimple_val, fb_rvalue);
+	  if (gs == GS_ERROR)
+	    remove = true;
+	  break;
+
+	case OMP_CLAUSE_NOWAIT:
+	case OMP_CLAUSE_ORDERED:
+	  break;
+
+	case OMP_CLAUSE_DEFAULT:
+	  ctx->default_kind = OMP_CLAUSE_DEFAULT_KIND (c);
+	  break;
+
+	default:
+	  gcc_unreachable ();
+	}
+
+      if (remove)
+	*list_p = OMP_CLAUSE_CHAIN (c);
+      else
+	list_p = &OMP_CLAUSE_CHAIN (c);
+    }
+
+  gimplify_omp_ctxp = ctx;
+}
+
+/* For all variables that were not actually used within the context,
+   remove PRIVATE, SHARED, and FIRSTPRIVATE clauses.  */
+
+static int
+gimplify_adjust_omp_clauses_1 (splay_tree_node n, void *data)
+{
+  tree *list_p = (tree *) data;
+  tree decl = (tree) n->key;
+  unsigned flags = n->value;
+  enum tree_code code;
+  tree clause;
+  bool private_debug;
+
+  if (flags & (GOVD_EXPLICIT | GOVD_LOCAL))
+    return 0;
+  if ((flags & GOVD_SEEN) == 0)
+    return 0;
+  if (flags & GOVD_DEBUG_PRIVATE)
+    {
+      gcc_assert ((flags & GOVD_DATA_SHARE_CLASS) == GOVD_PRIVATE);
+      private_debug = true;
+    }
+  else
+    private_debug
+      = lang_hooks.decls.omp_private_debug_clause (decl,
+						   !!(flags & GOVD_SHARED));
+  if (private_debug)
+    code = OMP_CLAUSE_PRIVATE;
+  else if (flags & GOVD_SHARED)
+    {
+      if (is_global_var (decl))
+	return 0;
+      code = OMP_CLAUSE_SHARED;
+    }
+  else if (flags & GOVD_PRIVATE)
+    code = OMP_CLAUSE_PRIVATE;
+  else if (flags & GOVD_FIRSTPRIVATE)
+    code = OMP_CLAUSE_FIRSTPRIVATE;
+  else
+    gcc_unreachable ();
+
+  clause = build1 (code, void_type_node, decl);
+  OMP_CLAUSE_CHAIN (clause) = *list_p;
+  if (private_debug)
+    OMP_CLAUSE_PRIVATE_DEBUG (clause) = 1;
+  *list_p = clause;
+
+  return 0;
+}
+
+static void
+gimplify_adjust_omp_clauses (tree *list_p)
+{
+  struct gimplify_omp_ctx *ctx = gimplify_omp_ctxp;
+  tree c, decl;
+
+  while ((c = *list_p) != NULL)
+    {
+      splay_tree_node n;
+      bool remove = false;
+
+      switch (TREE_CODE (c))
+	{
+	case OMP_CLAUSE_PRIVATE:
+	case OMP_CLAUSE_SHARED:
+	case OMP_CLAUSE_FIRSTPRIVATE:
+	  decl = OMP_CLAUSE_DECL (c);
+	  n = splay_tree_lookup (ctx->variables, (splay_tree_key) decl);
+	  remove = !(n->value & GOVD_SEEN);
+	  if (! remove)
+	    {
+	      bool shared = TREE_CODE (c) == OMP_CLAUSE_SHARED;
+	      if ((n->value & GOVD_DEBUG_PRIVATE)
+		  || lang_hooks.decls.omp_private_debug_clause (decl, shared))
+		{
+		  gcc_assert ((n->value & GOVD_DEBUG_PRIVATE) == 0
+			      || ((n->value & GOVD_DATA_SHARE_CLASS)
+				  == GOVD_PRIVATE));
+		  TREE_SET_CODE (c, OMP_CLAUSE_PRIVATE);
+		  OMP_CLAUSE_PRIVATE_DEBUG (c) = 1;
+		}
+	    }
+	  break;
+
+	case OMP_CLAUSE_LASTPRIVATE:
+	  /* Make sure OMP_CLAUSE_LASTPRIVATE_FIRSTPRIVATE is set to
+	     accurately reflect the presence of a FIRSTPRIVATE clause.  */
+	  decl = OMP_CLAUSE_DECL (c);
+	  n = splay_tree_lookup (ctx->variables, (splay_tree_key) decl);
+	  OMP_CLAUSE_LASTPRIVATE_FIRSTPRIVATE (c)
+	    = (n->value & GOVD_FIRSTPRIVATE) != 0;
+	  break;
+	  
+	case OMP_CLAUSE_REDUCTION:
+	case OMP_CLAUSE_COPYIN:
+	case OMP_CLAUSE_COPYPRIVATE:
+	case OMP_CLAUSE_IF:
+	case OMP_CLAUSE_NUM_THREADS:
+	case OMP_CLAUSE_SCHEDULE:
+	case OMP_CLAUSE_NOWAIT:
+	case OMP_CLAUSE_ORDERED:
+	case OMP_CLAUSE_DEFAULT:
+	  break;
+
+	default:
+	  gcc_unreachable ();
+	}
+
+      if (remove)
+	*list_p = OMP_CLAUSE_CHAIN (c);
+      else
+	list_p = &OMP_CLAUSE_CHAIN (c);
+    }
+
+  /* Add in any implicit data sharing.  */
+  splay_tree_foreach (ctx->variables, gimplify_adjust_omp_clauses_1, list_p);
+  
+  gimplify_omp_ctxp = ctx->outer_context;
+  delete_omp_context (ctx);
+}
+
+/* Gimplify the contents of an OMP_PARALLEL statement.  This involves
+   gimplification of the body, as well as scanning the body for used
+   variables.  We need to do this scan now, because variable-sized
+   decls will be decomposed during gimplification.  */
+
+static enum gimplify_status
+gimplify_omp_parallel (tree *expr_p, tree *pre_p)
+{
+  tree expr = *expr_p;
+
+  gimplify_scan_omp_clauses (&OMP_PARALLEL_CLAUSES (expr), pre_p, true);
+
+  push_gimplify_context ();
+
+  if (determine_parallel_type (expr) == IS_COMBINED_PARALLEL)
+    {
+      gimplify_ctxp->combined_pre_p = pre_p;
+      gimplify_ctxp->combined_ctxp = gimplify_ctxp->prev_context;
+    }
+
+  gimplify_stmt (&OMP_PARALLEL_BODY (expr));
+  pop_gimplify_context (OMP_PARALLEL_BODY (expr));
+
+  gimplify_ctxp->combined_pre_p = NULL;
+  gimplify_ctxp->combined_ctxp = NULL;
+
+  gimplify_adjust_omp_clauses (&OMP_PARALLEL_CLAUSES (expr));
+
+  return GS_ALL_DONE;
+}
+
+/* Gimplify the gross structure of an OMP_FOR statement.  */
+
+static enum gimplify_status
+gimplify_omp_for (tree *expr_p, tree *pre_p)
+{
+  tree for_stmt, decl, t;
+  enum gimplify_status ret = 0;
+  struct gimplify_omp_ctx *outer_combined_omp_ctxp = NULL;
+
+  for_stmt = *expr_p;
+
+  if (gimplify_ctxp->combined_pre_p)
+    outer_combined_omp_ctxp = gimplify_omp_ctxp->outer_context;
+
+  gimplify_scan_omp_clauses (&OMP_FOR_CLAUSES (for_stmt), pre_p, false);
+
+  t = OMP_FOR_INIT (for_stmt);
+  gcc_assert (TREE_CODE (t) == MODIFY_EXPR);
+  decl = TREE_OPERAND (t, 0);
+  gcc_assert (DECL_P (decl));
+  gcc_assert (INTEGRAL_TYPE_P (TREE_TYPE (decl)));
+  gcc_assert (!TYPE_UNSIGNED (TREE_TYPE (decl)));
+
+  /* Make sure the iteration variable is private.  */
+  if (omp_is_private (gimplify_omp_ctxp, decl))
+    omp_notice_variable (gimplify_omp_ctxp, decl, true);
+  else
+    omp_add_variable (gimplify_omp_ctxp, decl, GOVD_PRIVATE | GOVD_SEEN);
+
+  /* Gimplify inside our parent's context if this is part of a combined
+     parallel+workshare directive.  */
+  if (gimplify_ctxp->combined_pre_p)
+    ret |= gimplify_expr_in_ctx (&TREE_OPERAND (t, 1),
+				 gimplify_ctxp->combined_pre_p, NULL,
+				 is_gimple_val, fb_rvalue,
+				 gimplify_ctxp->combined_ctxp,
+				 outer_combined_omp_ctxp);
+  else
+    ret |= gimplify_expr (&TREE_OPERAND (t, 1), &OMP_FOR_PRE_BODY (for_stmt),
+			  NULL, is_gimple_val, fb_rvalue);
+
+  t = OMP_FOR_COND (for_stmt);
+  gcc_assert (COMPARISON_CLASS_P (t));
+  gcc_assert (TREE_OPERAND (t, 0) == decl);
+
+  /* Gimplify inside our parent's context if this is part of a combined
+     parallel+workshare directive.  */
+  if (gimplify_ctxp->combined_pre_p)
+    ret |= gimplify_expr_in_ctx (&TREE_OPERAND (t, 1),
+				 gimplify_ctxp->combined_pre_p, NULL,
+				 is_gimple_val, fb_rvalue,
+				 gimplify_ctxp->combined_ctxp,
+				 outer_combined_omp_ctxp);
+  else
+    ret |= gimplify_expr (&TREE_OPERAND (t, 1), &OMP_FOR_PRE_BODY (for_stmt),
+			  NULL, is_gimple_val, fb_rvalue);
+
+  t = OMP_FOR_INCR (for_stmt);
+  switch (TREE_CODE (t))
+    {
+    case PREINCREMENT_EXPR:
+    case POSTINCREMENT_EXPR:
+      t = build_int_cst (TREE_TYPE (decl), 1);
+      goto build_modify;
+    case PREDECREMENT_EXPR:
+    case POSTDECREMENT_EXPR:
+      t = build_int_cst (TREE_TYPE (decl), -1);
+      goto build_modify;
+    build_modify:
+      t = build2 (PLUS_EXPR, TREE_TYPE (decl), decl, t);
+      t = build2 (MODIFY_EXPR, void_type_node, decl, t);
+      OMP_FOR_INCR (for_stmt) = t;
+      break;
+      
+    case MODIFY_EXPR:
+      gcc_assert (TREE_OPERAND (t, 0) == decl);
+      t = TREE_OPERAND (t, 1);
+      switch (TREE_CODE (t))
+	{
+	case PLUS_EXPR:
+	  if (TREE_OPERAND (t, 1) == decl)
+	    {
+	      TREE_OPERAND (t, 1) = TREE_OPERAND (t, 0);
+	      TREE_OPERAND (t, 0) = decl;
+	      break;
+	    }
+	case MINUS_EXPR:
+	  gcc_assert (TREE_OPERAND (t, 0) == decl);
+	  break;
+	default:
+	  gcc_unreachable ();
+	}
+
+      /* Gimplify inside our parent's context if this is part of a
+	 combined parallel+workshare directive.  */
+      if (gimplify_ctxp->combined_pre_p)
+	ret |= gimplify_expr_in_ctx (&TREE_OPERAND (t, 1),
+				     gimplify_ctxp->combined_pre_p, NULL,
+				     is_gimple_val, fb_rvalue,
+				     gimplify_ctxp->combined_ctxp,
+				     outer_combined_omp_ctxp);
+      else
+	ret |= gimplify_expr (&TREE_OPERAND (t, 1),
+			      &OMP_FOR_PRE_BODY (for_stmt), NULL,
+			      is_gimple_val, fb_rvalue);
+      break;
+
+    default:
+      gcc_unreachable ();
+    }
+
+  gimplify_to_stmt_list (&OMP_FOR_BODY (for_stmt));
+  gimplify_adjust_omp_clauses (&OMP_FOR_CLAUSES (for_stmt));
+
+  return ret == GS_ALL_DONE ? GS_ALL_DONE : GS_ERROR;
+}
+
+/* Gimplify the gross structure of other OpenMP worksharing constructs.
+   In particular, OMP_SECTIONS and OMP_SINGLE.  */
+
+static enum gimplify_status
+gimplify_omp_workshare (tree *expr_p, tree *pre_p)
+{
+  tree stmt = *expr_p;
+
+  gimplify_scan_omp_clauses (&OMP_CLAUSES (stmt), pre_p, false);
+  gimplify_to_stmt_list (&OMP_BODY (stmt));
+  gimplify_adjust_omp_clauses (&OMP_CLAUSES (stmt));
+
+  return GS_ALL_DONE;
+}
+
+/* A subroutine of gimplify_omp_atomic.  The front end is supposed to have
+   stabilized the lhs of the atomic operation as *ADDR.  Return true if 
+   EXPR is this stabilized form.  */
+
+static bool
+goa_lhs_expr_p (tree expr, tree addr)
+{
+  /* Also include casts to other type variants.  The C front end is fond
+     of adding these for e.g. volatile variables.  This is like 
+     STRIP_TYPE_NOPS but includes the main variant lookup.  */
+  while ((TREE_CODE (expr) == NOP_EXPR
+          || TREE_CODE (expr) == CONVERT_EXPR
+          || TREE_CODE (expr) == NON_LVALUE_EXPR)
+         && TREE_OPERAND (expr, 0) != error_mark_node
+         && (TYPE_MAIN_VARIANT (TREE_TYPE (expr))
+             == TYPE_MAIN_VARIANT (TREE_TYPE (TREE_OPERAND (expr, 0)))))
+    expr = TREE_OPERAND (expr, 0);
+
+  if (TREE_CODE (expr) == INDIRECT_REF && TREE_OPERAND (expr, 0) == addr)
+    return true;
+  if (TREE_CODE (addr) == ADDR_EXPR && expr == TREE_OPERAND (addr, 0))
+    return true;
+  return false;
+}
+
+/* A subroutine of gimplify_omp_atomic.  Attempt to implement the atomic
+   operation as a __sync_fetch_and_op builtin.  INDEX is log2 of the
+   size of the data type, and thus usable to find the index of the builtin
+   decl.  Returns GS_UNHANDLED if the expression is not of the proper form.  */
+
+static enum gimplify_status
+gimplify_omp_atomic_fetch_op (tree *expr_p, tree addr, tree rhs, int index)
+{
+  enum built_in_function base;
+  tree decl, args, itype;
+  enum insn_code *optab;
+
+  /* Check for one of the supported fetch-op operations.  */
+  switch (TREE_CODE (rhs))
+    {
+    case PLUS_EXPR:
+      base = BUILT_IN_FETCH_AND_ADD_N;
+      optab = sync_add_optab;
+      break;
+    case MINUS_EXPR:
+      base = BUILT_IN_FETCH_AND_SUB_N;
+      optab = sync_add_optab;
+      break;
+    case BIT_AND_EXPR:
+      base = BUILT_IN_FETCH_AND_AND_N;
+      optab = sync_and_optab;
+      break;
+    case BIT_IOR_EXPR:
+      base = BUILT_IN_FETCH_AND_OR_N;
+      optab = sync_ior_optab;
+      break;
+    case BIT_XOR_EXPR:
+      base = BUILT_IN_FETCH_AND_XOR_N;
+      optab = sync_xor_optab;
+      break;
+    default:
+      return GS_UNHANDLED;
+    }
+
+  /* Make sure the expression is of the proper form.  */
+  if (goa_lhs_expr_p (TREE_OPERAND (rhs, 0), addr))
+    rhs = TREE_OPERAND (rhs, 1);
+  else if (commutative_tree_code (TREE_CODE (rhs))
+	   && goa_lhs_expr_p (TREE_OPERAND (rhs, 1), addr))
+    rhs = TREE_OPERAND (rhs, 0);
+  else
+    return GS_UNHANDLED;
+
+  decl = built_in_decls[base + index + 1];
+  itype = TREE_TYPE (TREE_TYPE (decl));
+
+  if (optab[TYPE_MODE (itype)] == CODE_FOR_nothing)
+    return GS_UNHANDLED;
+
+  args = tree_cons (NULL, fold_convert (itype, rhs), NULL);
+  args = tree_cons (NULL, addr, args);
+  *expr_p = build_function_call_expr (decl, args);
+  return GS_OK;
+}
+
+/* A subroutine of gimplify_omp_atomic_pipeline.  Walk *EXPR_P and replace
+   appearences of *LHS_ADDR with LHS_VAR.  If an expression does not involve
+   the lhs, evaluate it into a temporary.  Return 1 if the lhs appeared as
+   a subexpression, 0 if it did not, or -1 if an error was encountered.  */
+
+static int
+goa_stabilize_expr (tree *expr_p, tree *pre_p, tree lhs_addr, tree lhs_var)
+{
+  tree expr = *expr_p;
+  int saw_lhs;
+
+  if (goa_lhs_expr_p (expr, lhs_addr))
+    {
+      *expr_p = lhs_var;
+      return 1;
+    }
+  if (is_gimple_val (expr))
+    return 0;
+ 
+  saw_lhs = 0;
+  switch (TREE_CODE_CLASS (TREE_CODE (expr)))
+    {
+    case tcc_binary:
+      saw_lhs |= goa_stabilize_expr (&TREE_OPERAND (expr, 1), pre_p,
+				     lhs_addr, lhs_var);
+    case tcc_unary:
+      saw_lhs |= goa_stabilize_expr (&TREE_OPERAND (expr, 0), pre_p,
+				     lhs_addr, lhs_var);
+      break;
+    default:
+      break;
+    }
+
+  if (saw_lhs == 0)
+    {
+      enum gimplify_status gs;
+      gs = gimplify_expr (expr_p, pre_p, NULL, is_gimple_val, fb_rvalue);
+      if (gs != GS_ALL_DONE)
+	saw_lhs = -1;
+    }
+
+  return saw_lhs;
+}
+
+/* A subroutine of gimplify_omp_atomic.  Implement the atomic operation as:
+
+	oldval = *addr;
+      repeat:
+	newval = rhs;	// with oldval replacing *addr in rhs
+	oldval = __sync_val_compare_and_swap (addr, oldval, newval);
+	if (oldval != newval)
+	  goto repeat;
+
+   INDEX is log2 of the size of the data type, and thus usable to find the
+   index of the builtin decl.  */
+
+static enum gimplify_status
+gimplify_omp_atomic_pipeline (tree *expr_p, tree *pre_p, tree addr,
+			      tree rhs, int index)
+{
+  tree oldval, oldival, oldival2, newval, newival, label;
+  tree type, itype, cmpxchg, args, x, iaddr;
+
+  cmpxchg = built_in_decls[BUILT_IN_VAL_COMPARE_AND_SWAP_N + index + 1];
+  type = TYPE_MAIN_VARIANT (TREE_TYPE (TREE_TYPE (addr)));
+  itype = TREE_TYPE (TREE_TYPE (cmpxchg));
+
+  if (sync_compare_and_swap[TYPE_MODE (itype)] == CODE_FOR_nothing)
+    return GS_UNHANDLED;
+
+  oldval = create_tmp_var (type, NULL);
+  newval = create_tmp_var (type, NULL);
+
+  /* Precompute as much of RHS as possible.  In the same walk, replace
+     occurrences of the lhs value with our temporary.  */
+  if (goa_stabilize_expr (&rhs, pre_p, addr, oldval) < 0)
+    return GS_ERROR;
+
+  x = build_fold_indirect_ref (addr);
+  x = build2 (MODIFY_EXPR, void_type_node, oldval, x);
+  gimplify_and_add (x, pre_p);
+
+  /* For floating-point values, we'll need to view-convert them to integers
+     so that we can perform the atomic compare and swap.  Simplify the 
+     following code by always setting up the "i"ntegral variables.  */
+  if (INTEGRAL_TYPE_P (type) || POINTER_TYPE_P (type))
+    {
+      oldival = oldval;
+      newival = newval;
+      iaddr = addr;
+    }
+  else
+    {
+      oldival = create_tmp_var (itype, NULL);
+      newival = create_tmp_var (itype, NULL);
+
+      x = build1 (VIEW_CONVERT_EXPR, itype, oldval);
+      x = build2 (MODIFY_EXPR, void_type_node, oldival, x);
+      gimplify_and_add (x, pre_p);
+      iaddr = fold_convert (build_pointer_type (itype), addr);
+    }
+
+  oldival2 = create_tmp_var (itype, NULL);
+
+  label = create_artificial_label ();
+  x = build1 (LABEL_EXPR, void_type_node, label);
+  gimplify_and_add (x, pre_p);
+
+  x = build2 (MODIFY_EXPR, void_type_node, newval, rhs);
+  gimplify_and_add (x, pre_p);
+
+  if (newval != newival)
+    {
+      x = build1 (VIEW_CONVERT_EXPR, itype, newval);
+      x = build2 (MODIFY_EXPR, void_type_node, newival, x);
+      gimplify_and_add (x, pre_p);
+    }
+
+  x = build2 (MODIFY_EXPR, void_type_node, oldival2, oldival);
+  gimplify_and_add (x, pre_p);
+
+  args = tree_cons (NULL, fold_convert (itype, newival), NULL);
+  args = tree_cons (NULL, fold_convert (itype, oldival), args);
+  args = tree_cons (NULL, iaddr, args);
+  x = build_function_call_expr (cmpxchg, args);
+  if (oldval == oldival)
+    x = fold_convert (type, x);
+  x = build2 (MODIFY_EXPR, void_type_node, oldival, x);
+  gimplify_and_add (x, pre_p);
+
+  /* For floating point, be prepared for the loop backedge.  */
+  if (oldval != oldival)
+    {
+      x = build1 (VIEW_CONVERT_EXPR, type, oldival);
+      x = build2 (MODIFY_EXPR, void_type_node, oldval, x);
+      gimplify_and_add (x, pre_p);
+    }
+
+  /* Note that we always perform the comparison as an integer, even for
+     floating point.  This allows the atomic operation to properly 
+     succeed even with NaNs and -0.0.  */
+  x = build3 (COND_EXPR, void_type_node,
+	      build2 (NE_EXPR, boolean_type_node, oldival, oldival2),
+	      build1 (GOTO_EXPR, void_type_node, label), NULL);
+  gimplify_and_add (x, pre_p);
+
+  *expr_p = NULL;
+  return GS_ALL_DONE;
+}
+
+/* A subroutine of gimplify_omp_atomic.  Implement the atomic operation as:
+
+	GOMP_atomic_start ();
+	*addr = rhs;
+	GOMP_atomic_end ();
+
+   The result is not globally atomic, but works so long as all parallel
+   references are within #pragma omp atomic directives.  According to
+   responses received from omp@openmp.org, appears to be within spec.
+   Which makes sense, since that's how several other compilers handle
+   this situation as well.  */
+
+static enum gimplify_status
+gimplify_omp_atomic_mutex (tree *expr_p, tree *pre_p, tree addr, tree rhs)
+{
+  tree t;
+
+  t = built_in_decls[BUILT_IN_GOMP_ATOMIC_START];
+  t = build_function_call_expr (t, NULL);
+  gimplify_and_add (t, pre_p);
+
+  t = build_fold_indirect_ref (addr);
+  t = build2 (MODIFY_EXPR, void_type_node, t, rhs);
+  gimplify_and_add (t, pre_p);
+  
+  t = built_in_decls[BUILT_IN_GOMP_ATOMIC_END];
+  t = build_function_call_expr (t, NULL);
+  gimplify_and_add (t, pre_p);
+
+  *expr_p = NULL;
+  return GS_ALL_DONE;
+}
+
+/* Gimplify an OMP_ATOMIC statement.  */
+
+static enum gimplify_status
+gimplify_omp_atomic (tree *expr_p, tree *pre_p)
+{
+  tree addr = TREE_OPERAND (*expr_p, 0);
+  tree rhs = TREE_OPERAND (*expr_p, 1);
+  tree type = TYPE_MAIN_VARIANT (TREE_TYPE (TREE_TYPE (addr)));
+  HOST_WIDE_INT index;
+
+  /* Make sure the type is one of the supported sizes.  */
+  index = tree_low_cst (TYPE_SIZE_UNIT (type), 1);
+  index = exact_log2 (index);
+  if (index >= 0 && index <= 4)
+    {
+      enum gimplify_status gs;
+      unsigned int align;
+
+      if (DECL_P (TREE_OPERAND (addr, 0)))
+	align = DECL_ALIGN_UNIT (TREE_OPERAND (addr, 0));
+      else if (TREE_CODE (TREE_OPERAND (addr, 0)) == COMPONENT_REF
+	       && TREE_CODE (TREE_OPERAND (TREE_OPERAND (addr, 0), 1))
+		  == FIELD_DECL)
+	align = DECL_ALIGN_UNIT (TREE_OPERAND (TREE_OPERAND (addr, 0), 1));
+      else
+	align = TYPE_ALIGN_UNIT (type);
+
+      /* __sync builtins require strict data alignment.  */
+      if (exact_log2 (align) >= index)
+	{
+	  /* When possible, use specialized atomic update functions.  */
+	  if (INTEGRAL_TYPE_P (type) || POINTER_TYPE_P (type))
+	    {
+	      gs = gimplify_omp_atomic_fetch_op (expr_p, addr, rhs, index);
+	      if (gs != GS_UNHANDLED)
+		return gs;
+	    }
+
+	  /* If we don't have specialized __sync builtins, try and implement
+	     as a compare and swap loop.  */
+	  gs = gimplify_omp_atomic_pipeline (expr_p, pre_p, addr, rhs, index);
+	  if (gs != GS_UNHANDLED)
+	    return gs;
+	}
+    }
+
+  /* The ultimate fallback is wrapping the operation in a mutex.  */
+  return gimplify_omp_atomic_mutex (expr_p, pre_p, addr, rhs);
+}
 
 /*  Gimplifies the expression tree pointed to by EXPR_P.  Return 0 if
     gimplification failed.
@@ -4441,6 +5598,30 @@ gimplify_expr (tree *expr_p, tree *pre_p, tree *post_p,
 	  ret = GS_ALL_DONE;
 	  break;
 
+	case OMP_PARALLEL:
+	  ret = gimplify_omp_parallel (expr_p, pre_p);
+	  break;
+
+	case OMP_FOR:
+	  ret = gimplify_omp_for (expr_p, pre_p);
+	  break;
+
+	case OMP_SECTIONS:
+	case OMP_SINGLE:
+	  ret = gimplify_omp_workshare (expr_p, pre_p);
+	  break;
+
+	case OMP_SECTION:
+	case OMP_MASTER:
+	case OMP_ORDERED:
+	case OMP_CRITICAL:
+	  gimplify_to_stmt_list (&OMP_BODY (*expr_p));
+	  break;
+
+	case OMP_ATOMIC:
+	  ret = gimplify_omp_atomic (expr_p, pre_p);
+	  break;
+
 	default:
 	  switch (TREE_CODE_CLASS (TREE_CODE (*expr_p)))
 	    {
@@ -4880,6 +6061,8 @@ gimplify_body (tree *body_p, tree fndecl, bool do_parms)
   tree body, parm_stmts;
 
   timevar_push (TV_TREE_GIMPLIFY);
+
+  gcc_assert (gimplify_ctxp == NULL);
   push_gimplify_context ();
 
   /* Unshare most shared trees in the body and in that of any nested functions.
@@ -4933,6 +6116,7 @@ gimplify_body (tree *body_p, tree fndecl, bool do_parms)
   *body_p = body;
 
   pop_gimplify_context (body);
+  gcc_assert (gimplify_ctxp == NULL);
 
 #ifdef ENABLE_CHECKING
   walk_tree (body_p, check_pointer_types_r, NULL, NULL);