pmda: Initial ctdb pmda check-in

The CTDB Performance Metrics Domain Agent (PMDA) is compiled when
Performance Co-Pilot (PCP) header files are present.

The CTDB PMDA periodically requests runtime counters from ctdbd (similar
to ctdb statistics) and exports these values via PCP for capture and
charting etc.

(This used to be ctdb commit fe7f69d6fc37661c2f1caa11de4ed3a7940d0a2f)

8 files changed, 821 insertions, 0 deletions

diff --git a/ctdb/utils/pmda/Install b/ctdb/utils/pmda/Install
new file mode 100644
index 00000000000..a56a63506a8
--- /dev/null
+++ b/ctdb/utils/pmda/Install
@@ -0,0 +1,36 @@
+#! /bin/sh
+#
+# Copyright (c) 1997 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
+#
+# Install the ctdb PMDA and/or PMNS
+#
+
+. $PCP_DIR/etc/pcp.env
+. $PCP_SHARE_DIR/lib/pmdaproc.sh
+
+iam=ctdb
+pmda_interface=2
+
+# runs as daemon and only supports pipe IPC
+daemon_opt=true
+dso_opt=false
+pipe_opt=true
+socket_opt=false
+
+pmdaSetup
+pmdaInstall
+exit 0
diff --git a/ctdb/utils/pmda/README b/ctdb/utils/pmda/README
new file mode 100644
index 00000000000..cd262de70cd
--- /dev/null
+++ b/ctdb/utils/pmda/README
@@ -0,0 +1,64 @@
+CTDB PMDA
+===========
+
+This PMDA extracts metrics from the locally running ctdbd daemon for
+export to PMCD.
+
+Note:
+	This PMDA may be remade from source and hence requires IDO (or
+	more specifically a C compiler) to be installed.
+
+	Uses of make(1) may fail (without removing or clobbering files)
+	if the C compiler cannot be found.  This is most likely to
+	happen when running the PMDA ./Install script.
+
+	The only remedial action is to install the C compiler, or
+	hand-craft changes to the Makefile.
+
+Metrics
+=======
+
+The file ./help contains descriptions for all of the metrics exported
+by this PMDA.
+
+Once the PMDA has been installed, the following command will list all
+the available metrics and their explanatory "help" text:
+
+	$ pminfo -fT ctdb
+
+Installation
+============
+
+ +  # cd $PCP_PMDAS_DIR/ctdb
+
+ +  Check that there is no clash in the Performance Metrics Domain
+    defined in ./domain.h and the other PMDAs currently in use (see
+    $PCP_PMCDCONF_PATH).  If there is, edit ./domain.h to choose another
+    domain number.
+
+ +  Then simply use
+
+	# ./Install
+
+    and choose both the "collector" and "monitor" installation
+    configuration options.
+
+    You will be prompted to choose either a daemon implementation
+    or a DSO implementation of the PMDA, and in the case of the daemon
+    variant to select an IPC method -- everything else is automated
+
+De-installation
+===============
+
+ +  Simply use
+
+	# cd $PCP_PMDAS_DIR/ctdb
+	# ./Remove
+
+Troubleshooting
+===============
+
+ +  After installing or restarting the agent, the PMCD log file
+    ($PCP_LOG_DIR/pmcd/pmcd.log) and the PMDA log file
+    ($PCP_LOG_DIR/pmcd/ctdb.log) should be checked for any warnings
+    or errors.
diff --git a/ctdb/utils/pmda/Remove b/ctdb/utils/pmda/Remove
new file mode 100644
index 00000000000..7d1c509e50e
--- /dev/null
+++ b/ctdb/utils/pmda/Remove
@@ -0,0 +1,29 @@
+#! /bin/sh
+#
+# Copyright (c) 1997 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
+#
+# Remove the ctdb PMDA
+#
+
+. $PCP_DIR/etc/pcp.env
+. $PCP_SHARE_DIR/lib/pmdaproc.sh
+
+iam=ctdb
+
+pmdaSetup
+pmdaRemove
+exit 0
diff --git a/ctdb/utils/pmda/domain.h b/ctdb/utils/pmda/domain.h
new file mode 100644
index 00000000000..d9b316f66cc
--- /dev/null
+++ b/ctdb/utils/pmda/domain.h
@@ -0,0 +1,19 @@
+/* domain.h
+ *
+ * Copyright (c) 2004-2009 Silicon Graphics, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+#define CTDB 155
diff --git a/ctdb/utils/pmda/help b/ctdb/utils/pmda/help
new file mode 100644
index 00000000000..4066fde3ade
--- /dev/null
+++ b/ctdb/utils/pmda/help
@@ -0,0 +1,104 @@
+#
+# Copyright (c) 2000-2004 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
+#
+# ctdb PMDA help file in the ASCII format
+#
+# lines beginning with a # are ignored
+# lines beginning @ introduce a new entry of the form
+#  @ metric_name oneline-text
+#  help test goes
+#  here over multiple lines
+#  ...
+#
+# the metric_name is decoded against the default PMNS -- as a special case,
+# a name of the form NNN.MM (for numeric NNN and MM) is interpreted as an
+# instance domain identification, and the text describes the instance domain
+#
+# blank lines before the @ line are ignored
+#
+
+@ ctdb.num_clients number of clients connected to ctdbd
+
+@ ctdb.frozen whether any databases are frozen
+
+@ ctdb.recovering whether recovery is active
+
+@ ctdb.client_packets_sent number of packets sent to all clients
+
+@ ctdb.client_packets_recv number of packets received from all clients
+
+@ ctdb.node_packets_sent number of packets sent to other nodes
+
+@ ctdb.node_packets_recv number of packets received from other nodes
+
+@ ctdb.keepalive_packets_sent number of keepalive packets sent to other nodes
+
+@ ctdb.keepalive_packets_recv number of keepalive packets received from other nodes
+
+@ ctdb.node.req_call number of node CTDB_REQ_CALL packets handled
+
+@ ctdb.node.reply_call number of node CTDB_REPLY_CALL packets handled
+
+@ ctdb.node.req_dmaster number of node CTDB_REQ_DMASTER packets handled
+
+@ ctdb.node.reply_dmaster number of node CTDB_REPLY_DMASTER packets handled
+
+@ ctdb.node.reply_error number of node CTDB_REPLY_ERROR packets handled
+
+@ ctdb.node.req_message number of node CTDB_REQ_MESSAGE packets handled
+
+@ ctdb.node.req_control number of node CTDB_REQ_CONTROL packets handled
+
+@ ctdb.node.reply_control number of node CTDB_REPLY_CONTROL packets handled
+
+@ ctdb.client.req_call number of client CTDB_REQ_CALL packets handled
+
+@ ctdb.client.req_message number of client CTDB_REQ_MESSAGE packets handled
+
+@ ctdb.client.req_control number of client CTDB_REQ_CONTROL packets handled
+
+@ ctdb.timeouts.call (counter not implemented) number of call timeouts
+
+@ ctdb.timeouts.control number of node control message request timeouts awaiting reply
+
+@ ctdb.timeouts.traverse number of database traversal timeouts
+
+@ ctdb.total_calls total number of client ctdb request calls received
+
+@ ctdb.pending_calls total number of client ctdb request calls in progress
+
+@ ctdb.lockwait_calls number of tdb chainlock lockwait calls
+
+@ ctdb.pending_lockwait_calls number of lockwait calls waiting for a lock
+
+@ ctdb.childwrite_calls number of childwrite calls
+
+@ ctdb.pending_childwrite_calls number of childwrite calls in progress
+
+@ ctdb.memory_used total size of the ctdbd null talloc pool
+
+@ ctdb.max_hop_count maximum hops performed by a CTDB_REQ_CALL packet
+
+@ ctdb.max_reclock_ctdbd maximum recovery lock latency during setrecmode
+
+@ ctdb.max_reclock_recd maximum recovery lock latency as reported by the recovery process
+
+@ ctdb.max_call_latency maximum time spent handling a client request call
+
+@ ctdb.max_lockwait_latency maximum time spent waiting for a tdb chainlock
+
+@ ctdb.max_childwrite_latency maximum time spent performing a childwrite
diff --git a/ctdb/utils/pmda/pmda_ctdb.c b/ctdb/utils/pmda/pmda_ctdb.c
new file mode 100644
index 00000000000..40efb28af31
--- /dev/null
+++ b/ctdb/utils/pmda/pmda_ctdb.c
@@ -0,0 +1,487 @@
+/*
+ * CTDB PMDA
+ *
+ * Copyright (c) 1995,2004 Silicon Graphics, Inc.  All Rights Reserved.
+ * Copyright (c) 2011 David Disseldorp
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
+ */
+
+#include <pcp/pmapi.h>
+#include <pcp/impl.h>
+#include <pcp/pmda.h>
+#include "../../include/includes.h"
+#include "../../lib/events/events.h"
+#include "../../include/ctdb.h"
+#include "../../include/ctdb_private.h"
+#include "domain.h"
+
+/*
+ * CTDB PMDA
+ *
+ * This PMDA connects to the locally running ctdbd daemon and pulls
+ * statistics for export via PCP.
+ */
+
+/*
+ * list of instances
+ */
+
+
+/*
+ * All metrics supported in this PMDA - one table entry for each.
+ * The 4th field specifies the serial number of the instance domain
+ * for the metric, and must be either PM_INDOM_NULL (denoting a
+ * metric that only ever has a single value), or the serial number
+ * of one of the instance domains declared in the instance domain table
+ * (i.e. in indomtab, above).
+ */
+
+static pmdaMetric metrictab[] = {
+	/* num_clients */
+	{ NULL, { PMDA_PMID(0,0), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,0,0,0,0,0) }, },
+	/* frozen */
+	{ NULL, { PMDA_PMID(1,2), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,0,0,0,0,0) }, },
+	/* recovering */
+	{ NULL, { PMDA_PMID(3,3), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,0,0,0,0,0) }, },
+	/* client_packets_sent */
+	{ NULL, { PMDA_PMID(4,4), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* client_packets_recv */
+	{ NULL, { PMDA_PMID(5,5), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* node_packets_sent */
+	{ NULL, { PMDA_PMID(6,6), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* node_packets_recv */
+	{ NULL, { PMDA_PMID(7,7), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* keepalive_packets_sent */
+	{ NULL, { PMDA_PMID(8,8), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* keepalive_packets_recv */
+	{ NULL, { PMDA_PMID(9,9), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* req_call */
+	{ NULL, { PMDA_PMID(10,10), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* reply_call */
+	{ NULL, { PMDA_PMID(10,11), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* req_dmaster */
+	{ NULL, { PMDA_PMID(10,12), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* reply_dmaster */
+	{ NULL, { PMDA_PMID(10,13), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* reply_error */
+	{ NULL, { PMDA_PMID(10,14), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* req_message */
+	{ NULL, { PMDA_PMID(10,15), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* req_control */
+	{ NULL, { PMDA_PMID(10,16), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* reply_control */
+	{ NULL, { PMDA_PMID(10,17), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* req_call */
+	{ NULL, { PMDA_PMID(11,18), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* req_message */
+	{ NULL, { PMDA_PMID(11,19), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* req_control */
+	{ NULL, { PMDA_PMID(11,20), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* call */
+	{ NULL, { PMDA_PMID(12,21), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,0) }, },
+	/* control */
+	{ NULL, { PMDA_PMID(12,22), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,0) }, },
+	/* traverse */
+	{ NULL, { PMDA_PMID(12,23), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,0) }, },
+	/* total_calls */
+	{ NULL, { PMDA_PMID(13,24), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* pending_calls */
+	{ NULL, { PMDA_PMID(14,25), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,0,0,0,0,0) }, },
+	/* lockwait_calls */
+	{ NULL, { PMDA_PMID(15,27), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* pending_lockwait_calls */
+	{ NULL, { PMDA_PMID(16,27), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,0,0,0,0,0) }, },
+	/* childwrite_calls */
+	{ NULL, { PMDA_PMID(17,28), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+		PMDA_PMUNITS(0,0,1,0,0,PM_COUNT_ONE) }, },
+	/* pending_childwrite_calls */
+	{ NULL, { PMDA_PMID(18,29), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,0,0,0,0,0) }, },
+	/* memory_used */
+	{ NULL, { PMDA_PMID(19,30), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(1,0,0,PM_SPACE_BYTE,0,0) }, },
+	/* max_hop_count */
+	{ NULL, { PMDA_PMID(20,31), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,0,0,0,0,0) }, },
+	/* max_reclock_ctdbd */
+	{ NULL, { PMDA_PMID(21,32), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,1,0,0,PM_TIME_SEC,0) }, },
+	/* max_reclock_recd */
+	{ NULL, { PMDA_PMID(22,33), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,1,0,0,PM_TIME_SEC,0) }, },
+	/* max_call_latency */
+	{ NULL, { PMDA_PMID(23,34), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,1,0,0,PM_TIME_SEC,0) }, },
+	/* max_lockwait_latency */
+	{ NULL, { PMDA_PMID(24,35), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,1,0,0,PM_TIME_SEC,0) }, },
+	/* max_childwrite_latency */
+	{ NULL, { PMDA_PMID(25,36), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_INSTANT,
+		PMDA_PMUNITS(0,1,0,0,PM_TIME_SEC,0) }, },
+};
+
+static struct ctdb_context *ctdb;
+static struct event_context *ev;
+
+static int
+fill_node(unsigned int item, struct ctdb_statistics *stats, pmAtomValue *atom)
+{
+	switch (item) {
+	case 10:
+		atom->ul = stats->node.req_call;
+		break;
+	case 11:
+		atom->ul = stats->node.reply_call;
+		break;
+	case 12:
+		atom->ul = stats->node.req_dmaster;
+		break;
+	case 13:
+		atom->ul = stats->node.reply_dmaster;
+		break;
+	case 14:
+		atom->ul = stats->node.reply_error;
+		break;
+	case 15:
+		atom->ul = stats->node.req_message;
+		break;
+	case 16:
+		atom->ul = stats->node.req_control;
+		break;
+	case 17:
+		atom->ul = stats->node.reply_control;
+		break;
+	default:
+		return PM_ERR_PMID;
+	}
+
+	return 0;
+}
+
+static int
+fill_client(unsigned int item, struct ctdb_statistics *stats, pmAtomValue *atom)
+{
+	switch (item) {
+	case 18:
+		atom->ul = stats->client.req_call;
+		break;
+	case 19:
+		atom->ul = stats->client.req_message;
+		break;
+	case 20:
+		atom->ul = stats->client.req_control;
+		break;
+	default:
+		return PM_ERR_PMID;
+	}
+
+	return 0;
+}
+
+static int
+fill_timeout(unsigned int item, struct ctdb_statistics *stats, pmAtomValue *atom)
+{
+	switch (item) {
+	case 21:
+		atom->ul = stats->timeouts.call;
+		break;
+	case 22:
+		atom->ul = stats->timeouts.control;
+		break;
+	case 23:
+		atom->ul = stats->timeouts.traverse;
+		break;
+	default:
+		return PM_ERR_PMID;
+	}
+
+	return 0;
+}
+
+/*
+ * callback provided to pmdaFetch
+ */
+static int
+pmda_ctdb_fetch_cb(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
+{
+	struct ctdb_statistics stats;
+	int ret;
+	__pmID_int *id = (__pmID_int *)&(mdesc->m_desc.pmid);
+
+	if (inst != PM_IN_NULL)
+		return PM_ERR_INST;
+
+	ret = ctdb_ctrl_statistics(ctdb, ctdb->pnn, &stats);
+	if (ret) {
+		ret = PM_ERR_VALUE;
+		goto err_out;
+	}
+
+	switch (id->cluster) {
+	case 0:
+		atom->ul = stats.num_clients;
+		break;
+	case 1:
+		atom->ul = stats.frozen;
+		break;
+	case 3:
+		atom->ul = stats.recovering;
+		break;
+	case 4:
+		atom->ul = stats.client_packets_sent;
+		break;
+	case 5:
+		atom->ul = stats.client_packets_recv;
+		break;
+	case 6:
+		atom->ul = stats.node_packets_sent;
+		break;
+	case 7:
+		atom->ul = stats.node_packets_recv;
+		break;
+	case 8:
+		atom->ul = stats.keepalive_packets_sent;
+		break;
+	case 9:
+		atom->ul = stats.keepalive_packets_recv;
+		break;
+	case 10:
+		ret = fill_node(id->item, &stats, atom);
+		if (ret)
+			goto err_out;
+		break;
+	case 11:
+		ret = fill_client(id->item, &stats, atom);
+		if (ret)
+			goto err_out;
+		break;
+	case 12:
+		ret = fill_timeout(id->item, &stats, atom);
+		if (ret)
+			goto err_out;
+		break;
+	case 13:
+		atom->ul = stats.total_calls;
+		break;
+	case 14:
+		atom->ul = stats.pending_calls;
+		break;
+	case 15:
+		atom->ul = stats.lockwait_calls;
+		break;
+	case 16:
+		atom->ul = stats.pending_lockwait_calls;
+		break;
+	case 17:
+		atom->ul = stats.childwrite_calls;
+		break;
+	case 18:
+		atom->ul = stats.pending_childwrite_calls;
+		break;
+	case 19:
+		atom->ul = stats.memory_used;
+		break;
+	case 20:
+		atom->ul = stats.max_hop_count;
+		break;
+	case 21:
+		atom->d = stats.reclock.ctdbd;
+		break;
+	case 22:
+		atom->d = stats.reclock.recd;
+		break;
+	case 23:
+		atom->d = stats.max_call_latency;
+		break;
+	case 24:
+		atom->d = stats.max_lockwait_latency;
+		break;
+	case 25:
+		atom->d = stats.max_childwrite_latency;
+		break;
+	default:
+		return PM_ERR_PMID;
+	}
+
+	ret = 0;
+err_out:
+	return ret;
+}
+
+/*
+ * This routine is called once for each pmFetch(3) operation, so is a
+ * good place to do once-per-fetch functions, such as value caching or
+ * instance domain evaluation.
+ */
+static int
+pmda_ctdb_fetch(int numpmid, pmID pmidlist[], pmResult **resp, pmdaExt *pmda)
+{
+	return pmdaFetch(numpmid, pmidlist, resp, pmda);
+}
+
+static int
+pmda_ctdb_daemon_connect(void)
+{
+	const char *socket_name;
+	int ret;
+
+	ev = event_context_init(NULL);
+	if (ev == NULL) {
+		fprintf(stderr, "Failed to init event ctx\n");
+		return -1;
+	}
+
+	ctdb = ctdb_init(ev);
+	if (ctdb == NULL) {
+		fprintf(stderr, "Failed to init ctdb\n");
+		return -1;
+	}
+
+	socket_name = getenv("CTDB_SOCKET");
+	if (socket_name == NULL) {
+		socket_name = "/var/lib/ctdb/ctdb.socket";
+	}
+
+	ret = ctdb_set_socketname(ctdb, socket_name);
+	if (ret == -1) {
+		fprintf(stderr, "ctdb_set_socketname failed - %s\n",
+				ctdb_errstr(ctdb));
+		talloc_free(ctdb);
+		return -1;
+	}
+
+	ret = ctdb_socket_connect(ctdb);
+	if (ret != 0) {
+		fprintf(stderr, "Failed to connect to daemon\n");
+		talloc_free(ctdb);
+		return -1;
+	}
+
+	ctdb->pnn = ctdb_ctrl_getpnn(ctdb, timeval_current_ofs(3, 0),
+				     CTDB_CURRENT_NODE);
+	if (ctdb->pnn == (uint32_t)-1) {
+		fprintf(stderr, "Failed to get ctdb pnn\n");
+		talloc_free(ctdb);
+		return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * Initialise the agent
+ */
+void
+pmda_ctdb_init(pmdaInterface *dp)
+{
+	int ret;
+
+	if (dp->status != 0)
+		return;
+
+	ret = pmda_ctdb_daemon_connect();
+	if (ret < 0)
+		return;
+
+	dp->version.two.fetch = pmda_ctdb_fetch;
+	pmdaSetFetchCallBack(dp, pmda_ctdb_fetch_cb);
+
+	pmdaInit(dp, NULL, 0, metrictab, sizeof(metrictab)/sizeof(metrictab[0]));
+}
+
+static char *
+helpfile(void)
+{
+	static char buf[MAXPATHLEN];
+
+	if (!buf[0]) {
+		snprintf(buf, sizeof(buf), "%s/ctdb/help",
+			 pmGetConfig("PCP_PMDAS_DIR"));
+	}
+	return buf;
+}
+
+static void
+usage(void)
+{
+	fprintf(stderr, "Usage: %s [options]\n\n", pmProgname);
+	fputs("Options:\n"
+	  "  -d domain	use domain (numeric) for metrics domain of PMDA\n"
+	  "  -l logfile   write log into logfile rather than using default log name\n"
+	  "\nExactly one of the following options may appear:\n"
+	  "  -i port	  expect PMCD to connect on given inet port (number or name)\n"
+	  "  -p		   expect PMCD to supply stdin/stdout (pipe)\n"
+	  "  -u socket	expect PMCD to connect on given unix domain socket\n",
+	  stderr);
+	exit(1);
+}
+
+/*
+ * Set up the agent if running as a daemon.
+ */
+int
+main(int argc, char **argv)
+{
+	int err = 0;
+	char log_file[] = "pmda_ctdb.log";
+	pmdaInterface dispatch;
+
+	__pmSetProgname(argv[0]);
+
+	pmdaDaemon(&dispatch, PMDA_INTERFACE_2, pmProgname, CTDB,
+		   log_file, helpfile());
+
+	if (pmdaGetOpt(argc, argv, "d:i:l:pu:?", &dispatch, &err) != EOF)
+		err++;
+
+	if (err)
+		usage();
+
+	pmdaOpenLog(&dispatch);
+	pmda_ctdb_init(&dispatch);
+	pmdaConnect(&dispatch);
+	pmdaMain(&dispatch);
+
+	exit(0);
+}
+
diff --git a/ctdb/utils/pmda/pmns b/ctdb/utils/pmda/pmns
new file mode 100644
index 00000000000..000dee53c79
--- /dev/null
+++ b/ctdb/utils/pmda/pmns
@@ -0,0 +1,72 @@
+/*
+ * Metrics for CTDB PMDA
+ *
+ * Copyright (c) 2000-2004 Silicon Graphics, Inc.  All Rights Reserved.
+ * Copyright (c) 2011 David Disseldorp
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
+ */
+
+ctdb {
+	num_clients		CTDB:0:0
+	frozen			CTDB:1:2
+	recovering		CTDB:3:3
+	client_packets_sent	CTDB:4:4
+	client_packets_recv	CTDB:5:5
+	node_packets_sent	CTDB:6:6
+	node_packets_recv	CTDB:7:7
+	keepalive_packets_sent	CTDB:8:8
+	keepalive_packets_recv	CTDB:9:9
+	node
+	client
+	timeouts
+	total_calls		CTDB:13:24
+	pending_calls		CTDB:14:25
+	lockwait_calls		CTDB:15:27
+	pending_lockwait_calls	CTDB:16:27
+	childwrite_calls	CTDB:17:28
+	pending_childwrite_calls CTDB:18:29
+	memory_used		CTDB:19:30
+	max_hop_count		CTDB:20:31
+	max_reclock_ctdbd	CTDB:21:32
+	max_reclock_recd	CTDB:22:33
+	max_call_latency	CTDB:23:34
+	max_lockwait_latency	CTDB:24:35
+	max_childwrite_latency	CTDB:25:36
+}
+
+ctdb.node {
+	req_call	CTDB:10:10
+	reply_call	CTDB:10:11
+	req_dmaster	CTDB:10:12
+	reply_dmaster	CTDB:10:13
+	reply_error	CTDB:10:14
+	req_message	CTDB:10:15
+	req_control	CTDB:10:16
+	reply_control	CTDB:10:17
+}
+
+ctdb.client {
+	req_call	CTDB:11:18
+	req_message	CTDB:11:19
+	req_control	CTDB:11:20
+}
+
+ctdb.timeouts {
+	call		CTDB:12:21
+	control		CTDB:12:22
+	traverse	CTDB:12:23
+}
+
diff --git a/ctdb/utils/pmda/root b/ctdb/utils/pmda/root
new file mode 100644
index 00000000000..ff036ed7fbd
--- /dev/null
+++ b/ctdb/utils/pmda/root
@@ -0,0 +1,10 @@
+/*
+ * fake "root" for validating the local PMNS subtree
+ */
+
+#include <stdpmid>
+
+root { ctdb }
+
+#include "pmns"
+