# Design note: HA with Transactions.

Clients can use transactions (TX or DTX) with the current HA module but:

- the results are not guaranteed to be comitted atomically on the backups.
- the backups do not record the transaction in the store for possible recovery.

## Requirements

1. Atomic: Transaction results must be atomic on primary and backups.

2. Consistent: Backups and primary must agree on whether the transaction comitted.

3. Isolated: Concurrent transactions must not interfere with each other.

4. Durable: Transactional state is written to the store.

5. Recovery: If a cluster crashes while a DTX is in progress, it can be
   re-started and participate in DTX recovery with a transaction co-ordinater
   (currently via JMS)

## TX and DTX

Both TX and DTX transactions require a DTX-like `prepare` phase to synchronize
the commit or rollback across multiple brokers. This design note applies to both.

For DTX the transaction is identified by the `xid`. For TX the HA module generates
a unique identifier.

## Intercepting transactional operations

Introduce  2 new interfaces on the Primary to intercept transactional operations:

    TransactionObserverFactory {
      TransactionObserver create(...);
    }

    TransactionObserver {
      publish(queue, msg);
      accept(accumulatedAck, unacked) # NOTE: translate queue positions to replication ids
      prepare()
      commit()
      rollback()
    }

The primary will register a `TransactionObserverFactory` with the broker to hook
into transactions. On the primary, transactional events are processed as normal
and additionally are passed to the `TransactionObserver` which replicates the
events to the backups.

## Replicating transaction content

The primary creates a specially-named queue for each transaction (the tx-queue)

TransactionObserver operations:

- publish(queue, msg): push enqueue event onto tx-queue
- accept(accumulatedAck, unacked) push dequeue event onto tx-queue
  (NOTE: must translate queue positions to replication ids)

The tx-queue is replicated like a normal queue with some extensions for transactions.

- Backups create a `TransactionReplicator` (extends `QueueReplicator`)
  - `QueueReplicator` builds up a `TxBuffer` or `DtxBuffer` of transaction events.
- Primary has `TxReplicatingSubscription` (extends `ReplicatingSubscription`


Using a tx-queue has the following benefits:

- Already have the tools to replicate it.
- Handles async fan-out of transactions to multiple Backups.
- keeps the tx events in linear order even if the tx spans multiple queues.
- Keeps tx data available for new backups until the tx is closed
- By closing the queue (see next section) its easy to establish what backups are
   in/out of the transaction.

## DTX Prepare

Primary receives dtx.prepare:

- "closes" the tx-queue
  - A closed queue rejects any attempt to subscribe.
  - Backups subscribed before the close are in the transaction.
- Puts prepare event on tx-queue, and does local prepare
- Returns ok if outcome of local and all backup prepares is ok, fail otherwise.

*TODO*: this means we need asynchronous completion of the prepare control
so we complete only when all backups have responded (or time out.)

Backups receiving prepare event do a local prepare.  Outcome is communicated to
the TxReplicatingSubscription on the primary as follows:

- ok: Backup acknowledges prepare event message on the tx-queue
- fail: Backup cancels tx-queue subscription

## DTX Commit/Rollback

Primary receives dtx.commit/rollback

- Primary puts commit/rollback event on the tx-queue and does local commit/rollback.
- Backups commit/rollback as instructed and unsubscribe from tx-queue.
- tx-queue auto deletes when last backup unsubscribes.

## TX Commit/Rollback

Primary receives tx.commit/rollback

- Do prepare phase as for DTX.
- Do commit/rollback phase as for DTX.

## De-duplication

When tx commits, each broker (backup & primary) pushes tx messages to the local queue.

On the primary, ReplicatingSubscriptions will see the tx messages on the local
queue. Need to avoid re-sending to backups that already have the messages from
the transaction.

ReplicatingSubscriptions has a "skip set" of messages already on the backup,
add tx messages to the skip set before Primary commits to local queue.

## Failover

Backups abort all open tx if the primary fails.

## Atomicity

Keep tx atomic when backups catch up while a tx is in progress.
A `ready` backup should never contain a proper subset of messages in a transaction.

Scenario:

- Guard placed on Q for an expected backup.
- Transaction with messages A,B,C on Q is prepared, tx-queue is closed.
- Expected backup connects and is declared ready due to guard.
- Transaction commits, primary publishes A, B to Q and crashes before adding C.
- Backup is ready so promoted to new primary with a partial transaction A,B.

*TODO*: Not happy with the following solution.

Solution: Primary delays `ready` status till full tx is replicated.

- When a new backup joins it is considered 'blocked' by any TX in progress.
  - create a TxTracker per queue, per backup at prepare, before local commit.
  - TxTracker holds set of enqueues & dequeues in the tx.
  - ReplicatingSubscription removes enqueus & dequeues from TxTracker as they are replicated.
- Backup starts to catch up as normal but is not granted ready till:
  - catches up on it's guard (as per normal catch-up) AND
  - all TxTrackers are clear.

NOTE: needs thougthful locking to correctly handle

- multiple queues per tx
- multiple tx per queue
- multiple TxTrackers per queue per per backup
- synchronize backups in/out of transaction wrt setting trackers.

*TODO*: A blocking TX eliminates the benefit of the queue guard for expected backups.

- It won't be ready till it has replicated all guarded positions up to the tx.
- Could severely delay backups becoming ready after fail-over if queues are long.

## Recovery

*TODO*

## Testing

New tests:

- TX transaction tests in python.
- TX failover stress test (c.f. `test_failover_send_receive`)

Existing tests:

- JMS transaction tests (DTX & TX)
- `qpid/tests/src/py/qpid_tests/broker_0_10/dtx.py,tx.py` run against HA broker.

