> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kombify.io/llms.txt
> Use this file to discover all available pages before exploring further.

# High Availability Kit

> High-availability multi-node homelab infrastructure

<Warning>
  **Vision-only** -- This StackKit is not yet implemented. The documentation below describes the planned architecture and capabilities. Use the [Base Kit](https://docs.kombify.io/stackkits/kits/base-kit) for production deployments today.
</Warning>

The **High Availability Kit** provides high-availability infrastructure for homelabs that can't afford downtime. It supports 2-5 nodes with automatic failover.

## Overview

```mermaid theme={null}
graph TB
    subgraph Cluster["HA Cluster"]
        subgraph Node1["Node 1 (Primary)"]
            T1["Traefik"]
            A1["Apps"]
            D1["Database<br/>Primary"]
        end
        
        subgraph Node2["Node 2 (Replica)"]
            T2["Traefik"]
            A2["Apps"]
            D2["Database<br/>Replica"]
        end
        
        subgraph Node3["Node 3 (Optional)"]
            T3["Traefik"]
            A3["Apps"]
            D3["Database<br/>Replica"]
        end
    end
    
    VIP["Virtual IP<br/>Keepalived"]
    Internet["Internet"]
    
    Internet --> VIP
    VIP --> Node1
    VIP --> Node2
    VIP --> Node3
    
    D1 <-.->|Replication| D2
    D2 <-.->|Replication| D3
    
    style Cluster fill:#1e293b,stroke:#38bdf8,color:#f8fafc
    style Node1 fill:#1e293b,stroke:#4ade80,color:#f8fafc
    style Node2 fill:#1e293b,stroke:#fbbf24,color:#f8fafc
    style Node3 fill:#1e293b,stroke:#a78bfa,color:#f8fafc
```

## Key Features

<CardGroup cols={3}>
  <Card title="Auto Failover" icon="rotate">
    Automatic failover when primary node fails
  </Card>

  <Card title="Load Balancing" icon="scale-balanced">
    Distribute traffic across healthy nodes
  </Card>

  <Card title="Data Replication" icon="clone">
    Synchronous database replication
  </Card>
</CardGroup>

## Requirements

| Resource    | Per Node  | Total (2 nodes) |
| ----------- | --------- | --------------- |
| **CPU**     | 2 cores   | 4 cores         |
| **RAM**     | 8 GB      | 16 GB           |
| **Storage** | 50 GB SSD | 100 GB          |
| **Network** | 1 Gbps    | Same subnet     |

## Architecture

### Failover Mechanism

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant VIP as Virtual IP
    participant N1 as Node 1 (Primary)
    participant N2 as Node 2 (Backup)
    participant Keepalived
    
    Note over N1,N2: Normal Operation
    Client->>VIP: Request
    VIP->>N1: Forward (Primary owns VIP)
    N1->>Client: Response
    
    Note over N1: Node 1 Fails
    N1-xN1: Crash/Unreachable
    
    Keepalived->>Keepalived: Detect failure (VRRP)
    Keepalived->>N2: Promote to Primary
    N2->>VIP: Take ownership
    
    Note over N2: Node 2 Now Primary
    Client->>VIP: Request
    VIP->>N2: Forward (N2 owns VIP)
    N2->>Client: Response
```

### Database Replication

```mermaid theme={null}
graph LR
    subgraph Primary["Primary Node"]
        PG1["PostgreSQL<br/>Read/Write"]
        WAL1["WAL Files"]
    end
    
    subgraph Replica1["Replica Node 1"]
        PG2["PostgreSQL<br/>Read Only"]
        WAL2["WAL Receiver"]
    end
    
    subgraph Replica2["Replica Node 2"]
        PG3["PostgreSQL<br/>Read Only"]
        WAL3["WAL Receiver"]
    end
    
    PG1 --> WAL1
    WAL1 -->|Streaming| WAL2
    WAL1 -->|Streaming| WAL3
    WAL2 --> PG2
    WAL3 --> PG3
    
    style Primary fill:#1e293b,stroke:#4ade80,color:#f8fafc
    style Replica1 fill:#1e293b,stroke:#fbbf24,color:#f8fafc
    style Replica2 fill:#1e293b,stroke:#a78bfa,color:#f8fafc
```

## Quick Start

<Steps>
  <Step title="Define your nodes">
    ```yaml stack-spec.yaml theme={null}
    stackkit: ha-kit

    domain: homelab.local
    email: you@example.com

    nodes:
      - name: node-1
        ip: 192.168.1.10
        role: primary
        
      - name: node-2
        ip: 192.168.1.11
        role: replica

    # Virtual IP for failover
    failover:
      strategy: keepalived
      virtual_ip: 192.168.1.100
      interface: eth0
    ```
  </Step>

  <Step title="Configure services">
    ```yaml theme={null}
    services:
      traefik:
        enabled: true
        replicas: all         # Run on all nodes
        
      authelia:
        enabled: true
        replicas: all
        
      postgres:
        enabled: true
        mode: ha              # Enable replication
        synchronous: true     # Zero data loss
        
      redis:
        enabled: true
        mode: sentinel        # Redis Sentinel for HA
    ```
  </Step>

  <Step title="Validate and generate">
    ```bash theme={null}
    stackkit validate
    stackkit generate
    ```
  </Step>

  <Step title="Deploy to all nodes">
    ```bash theme={null}
    # Deploy to primary first
    stackkit deploy --node node-1

    # Then replicas
    stackkit deploy --node node-2
    ```
  </Step>
</Steps>

## Configuration Reference

### Failover Strategies

<Tabs>
  <Tab title="Keepalived (Recommended)">
    ```yaml theme={null}
    failover:
      strategy: keepalived
      virtual_ip: 192.168.1.100
      interface: eth0
      priority:
        node-1: 100  # Higher = preferred primary
        node-2: 90
    ```

    Uses VRRP protocol for fast failover (\~3 seconds).
  </Tab>

  <Tab title="Corosync/Pacemaker">
    ```yaml theme={null}
    failover:
      strategy: corosync
      quorum: majority
      resources:
        - virtual_ip: 192.168.1.100
        - postgres_primary
    ```

    Enterprise-grade with resource management.
  </Tab>
</Tabs>

### Database HA Modes

| Mode     | Description       | RPO     | RTO   |
| -------- | ----------------- | ------- | ----- |
| `async`  | Async replication | Seconds | \~30s |
| `sync`   | Sync replication  | Zero    | \~30s |
| `quorum` | Quorum commit     | Zero    | \~30s |

```yaml theme={null}
services:
  postgres:
    enabled: true
    mode: ha
    replication:
      type: sync        # async, sync, quorum
      max_replicas: 2
```

### Service Distribution

```yaml theme={null}
services:
  # Run on all nodes (stateless)
  traefik:
    replicas: all
    
  # Run on specific nodes
  home-assistant:
    replicas: 1
    preferred_node: node-1
    
  # Run on N nodes
  plex:
    replicas: 2
```

## Network Architecture

```mermaid theme={null}
graph TB
    subgraph External["External Network"]
        Internet["Internet"]
        Router["Router"]
    end
    
    subgraph Internal["Internal Network (192.168.1.0/24)"]
        VIP["Virtual IP<br/>192.168.1.100"]
        
        subgraph N1["Node 1 - 192.168.1.10"]
            T1["Traefik"]
            K1["Keepalived<br/>MASTER"]
        end
        
        subgraph N2["Node 2 - 192.168.1.11"]
            T2["Traefik"]
            K2["Keepalived<br/>BACKUP"]
        end
    end
    
    Internet --> Router
    Router -->|Port Forward| VIP
    VIP --> T1
    VIP --> T2
    K1 <-.->|VRRP| K2
    
    style External fill:#1e293b,stroke:#ef4444,color:#f8fafc
    style Internal fill:#1e293b,stroke:#38bdf8,color:#f8fafc
```

## Monitoring

Built-in health checks and monitoring:

```yaml theme={null}
monitoring:
  enabled: true
  
  healthchecks:
    interval: 10s
    timeout: 5s
    
  alerts:
    - type: node_down
      notify: email
    - type: failover_triggered
      notify: [email, slack]
    - type: replication_lag
      threshold: 10s
      notify: slack
```

## Testing Failover

<Warning>
  Always test failover in a maintenance window first!
</Warning>

```bash theme={null}
# Simulate node failure
ssh node-1 "sudo systemctl stop docker"

# Watch failover
stackkit status --watch

# Expected output:
# [12:00:01] node-1: UNREACHABLE
# [12:00:03] Failover triggered: node-1 → node-2
# [12:00:04] VIP 192.168.1.100 moved to node-2
# [12:00:05] Services healthy on node-2

# Restore node-1
ssh node-1 "sudo systemctl start docker"

# node-1 becomes backup (no auto-failback by default)
```

## Constraints

| Constraint  | Value    | Reason                |
| ----------- | -------- | --------------------- |
| Min nodes   | 2        | Need backup for HA    |
| Max nodes   | 5        | Complexity ceiling    |
| Same subnet | Required | VRRP/VIP requirement  |
| Network     | 1 Gbps+  | Replication bandwidth |

## Troubleshooting

<AccordionGroup>
  <Accordion title="Split-brain scenario">
    When nodes can't communicate but both think they're primary:

    1. Check network connectivity between nodes
    2. Review Keepalived logs: `journalctl -u keepalived`
    3. Consider adding a third node for quorum
  </Accordion>

  <Accordion title="Replication lag">
    If replica falls behind:

    1. Check network bandwidth: `iperf3 -c node-1`
    2. Review PostgreSQL logs for errors
    3. Consider async replication for high-write workloads
  </Accordion>

  <Accordion title="Failover not triggering">
    1. Verify Keepalived is running on both nodes
    2. Check VRRP traffic: `tcpdump -i eth0 vrrp`
    3. Review interface configuration
  </Accordion>
</AccordionGroup>

## Migration from the Base Kit

<Steps>
  <Step title="Add second node hardware">
    Install same OS and Docker version as your current node.
  </Step>

  <Step title="Update stack-spec.yaml">
    ```yaml theme={null}
    stackkit: ha-kit  # Changed from base-kit!

    nodes:
      - name: node-1      # Existing
        ip: 192.168.1.10
        role: primary
      - name: node-2      # New
        ip: 192.168.1.11
        role: replica

    failover:
      strategy: keepalived
      virtual_ip: 192.168.1.100
    ```
  </Step>

  <Step title="Migrate data">
    ```bash theme={null}
    stackkit migrate --from base-kit --to ha-kit
    ```
  </Step>
</Steps>

## Next Steps

<CardGroup cols={2}>
  <Card title="Monitoring Setup" icon="chart-line" href="https://docs.kombify.io/stackkits/reference/monitoring">
    Set up comprehensive monitoring
  </Card>
</CardGroup>