Proxmox + Ceph Integration and Distributed Storage Deployment

🔰 Introduction

In any virtualization infrastructure, storage reliability and scalability are critical for maintaining system stability.
As the number of servers grows and administrators require seamless live migration and high availability (HA),
traditional centralized storage (such as NAS or iSCSI) often becomes a performance bottleneck or single point of failure.

To address this, Proxmox VE natively integrates the Ceph distributed storage system,
enabling a self-healing, linearly scalable, and highly available storage backend across multiple nodes.

This article explains:
1️⃣ The core architecture and principles of Ceph
2️⃣ How to deploy Ceph within a Proxmox cluster
3️⃣ Practical optimization and reliability strategies

🧩 1. What Is Ceph?

1️⃣ Definition

Ceph is an open-source, software-defined distributed storage system that provides:

Block storage (RBD) for VMs and containers
Object storage (RGW, compatible with S3 APIs)
File system storage (CephFS, a distributed file system)

Ceph ensures data durability by distributing data across multiple nodes using replication or erasure coding,
so even if hardware fails, your storage remains available and consistent.

🧱 Architecture Overview

           ┌──────────────────────────────┐
           │          Proxmox VE          │
           │     (KVM / LXC / HA)         │
           └──────────────┬───────────────┘
                          │
                   Ceph Client (RBD)
                          │
          ┌──────────────────────────────────┐
          │            Ceph Cluster          │
          │ ┌──────────┬──────────┬────────┐ │
          │ │   MONs   │   OSDs   │   MGRs │ │
          │ └──────────┴──────────┴────────┘ │
          │        (Distributed Storage)     │
          └──────────────────────────────────┘

2️⃣ Core Components

Component	Full Name	Function
MON	Monitor	Maintains cluster membership and health information.
OSD	Object Storage Daemon	Stores actual data and handles replication.
MGR	Manager	Provides metrics, dashboards, and API services.
MDS	Metadata Server	Manages file system metadata (used for CephFS).

⚙️ 2. Advantages of Proxmox + Ceph Integration

Feature	Benefit
Native Integration	Ceph is built directly into Proxmox VE’s Web GUI and CLI tools.
No Single Point of Failure	Data is distributed across nodes rather than stored centrally.
Self-Healing	Automatically replicates and rebuilds data when disks or nodes fail.
Scalable Architecture	Add disks or nodes dynamically without downtime.
Live Migration Support	All nodes share the same Ceph storage pool, enabling smooth VM migration.

🧰 3. Example Deployment Architecture

Environment Overview

Node	IP Address	Role
pve-node01	10.0.0.11	Proxmox + Ceph MON + OSD
pve-node02	10.0.0.12	Proxmox + Ceph MON + OSD
pve-node03	10.0.0.13	Proxmox + Ceph MON + OSD

Recommendations:

At least three nodes are required for quorum.
Use two network interfaces per node:
- One for management and VM traffic (Public Network)
- One dedicated to Ceph replication (Cluster Network)

Network Design

Public Network  : 10.0.0.0/24   (Client I/O)
Cluster Network : 192.168.100.0/24 (Replication / Heartbeat)

💡 Keep Ceph replication traffic separate from VM traffic to avoid latency or performance degradation.

🧭 4. Step-by-Step Deployment

Step 1 – Enable Ceph Repository

apt update
apt install ceph ceph-common ceph-fuse

Proxmox VE 9.x ships with Ceph Reef or newer Ceph Squid builds by default.

Step 2 – Initialize Ceph Cluster

In the Proxmox Web UI:

Go to Datacenter → Ceph → Install Ceph
After installation, click Create Cluster
Define both Public and Cluster Networks

CLI method:

pveceph init --cluster-network 192.168.100.0/24 \
             --network 10.0.0.0/24

Step 3 – Add MON and MGR Nodes

pveceph mon create
pveceph mgr create

Repeat for all participating nodes or use Add Monitor in the Web UI.

Step 4 – Create OSDs (Object Storage Daemons)

Select available disks for Ceph storage:

pveceph osd create /dev/sdb

Or use the Web GUI: Ceph → OSD → Create

💡 Use SSD or NVMe drives for Ceph’s DB/WAL partitions to enhance I/O performance.

Step 5 – Create a Storage Pool

pveceph pool create ceph-pool --size 3 --min-size 2

Then add it as a Proxmox storage backend:

pvesh create /storage --storage ceph-rbd \
--type rbd --pool ceph-pool \
--monhost 10.0.0.11,10.0.0.12,10.0.0.13

🧠 5. Performance and Reliability Design

Setting	Recommended Configuration
Replica Count (size)	3 replicas for production reliability
Min. Active Replicas (min-size)	2 for safe I/O operations
Disk Layout	Use SSD/NVMe for DB/WAL; HDD for bulk data
Network Design	Separate Ceph cluster and public networks
Monitoring	Use Ceph Dashboard or `ceph -s` for real-time health checks

🗄️ 6. Integration with HA and PBS

With High Availability (HA):
All Proxmox nodes share the same Ceph storage pool, allowing seamless VM migration between nodes.
With Proxmox Backup Server (PBS):
Ceph RBD pools can be mounted as a backend datastore for incremental backups.

Architecture Example:

[Proxmox Cluster]───┐
                     │  (Ceph RBD)
                     ▼
              [Ceph Storage Pool]
                     │
             [Proxmox Backup Server]

⚙️ 7. Cross-Site Replication and DR Extension

Use Ceph RBD Mirror for real-time block-level replication between data centers.
Combine with Proxmox Backup Server (PBS) for snapshot-based offsite backups.
For long-distance or high-latency environments, use Active/Passive replication with periodic synchronization.

✅ Conclusion

By integrating Proxmox VE with Ceph, enterprises can build a fully distributed, self-healing, and scalable storage layer
that eliminates the single points of failure common in traditional architectures.

This unified design delivers:

Built-in high availability
Linear scalability
Automated recovery
Simplified storage management

Together, Proxmox and Ceph form a powerful open-source foundation for enterprise-grade virtualization and cloud infrastructure.

💬 In the next article, we’ll explore
“Proxmox Cloud Management and Hybrid Architecture Integration”,
demonstrating how to extend your Proxmox environment into hybrid and multi-cloud deployments.