๐ฐ Introduction
In any virtualization infrastructure, storage reliability and scalability are critical for maintaining system stability.
As the number of servers grows and administrators require seamless live migration and high availability (HA),
traditional centralized storage (such as NAS or iSCSI) often becomes a performance bottleneck or single point of failure.
To address this, Proxmox VE natively integrates the Ceph distributed storage system,
enabling a self-healing, linearly scalable, and highly available storage backend across multiple nodes.
This article explains:
1๏ธโฃ The core architecture and principles of Ceph
2๏ธโฃ How to deploy Ceph within a Proxmox cluster
3๏ธโฃ Practical optimization and reliability strategies
๐งฉ 1. What Is Ceph?
1๏ธโฃ Definition
Ceph is an open-source, software-defined distributed storage system that provides:
- Block storage (RBD) for VMs and containers
- Object storage (RGW, compatible with S3 APIs)
- File system storage (CephFS, a distributed file system)
Ceph ensures data durability by distributing data across multiple nodes using replication or erasure coding,
so even if hardware fails, your storage remains available and consistent.
๐งฑ Architecture Overview
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Proxmox VE โ
โ (KVM / LXC / HA) โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ
โ
Ceph Client (RBD)
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Ceph Cluster โ
โ โโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโ โ
โ โ MONs โ OSDs โ MGRs โ โ
โ โโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโ โ
โ (Distributed Storage) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
2๏ธโฃ Core Components
| Component | Full Name | Function |
|---|---|---|
| MON | Monitor | Maintains cluster membership and health information. |
| OSD | Object Storage Daemon | Stores actual data and handles replication. |
| MGR | Manager | Provides metrics, dashboards, and API services. |
| MDS | Metadata Server | Manages file system metadata (used for CephFS). |
โ๏ธ 2. Advantages of Proxmox + Ceph Integration
| Feature | Benefit |
|---|---|
| Native Integration | Ceph is built directly into Proxmox VEโs Web GUI and CLI tools. |
| No Single Point of Failure | Data is distributed across nodes rather than stored centrally. |
| Self-Healing | Automatically replicates and rebuilds data when disks or nodes fail. |
| Scalable Architecture | Add disks or nodes dynamically without downtime. |
| Live Migration Support | All nodes share the same Ceph storage pool, enabling smooth VM migration. |
๐งฐ 3. Example Deployment Architecture
Environment Overview
| Node | IP Address | Role |
|---|---|---|
| pve-node01 | 10.0.0.11 | Proxmox + Ceph MON + OSD |
| pve-node02 | 10.0.0.12 | Proxmox + Ceph MON + OSD |
| pve-node03 | 10.0.0.13 | Proxmox + Ceph MON + OSD |
Recommendations:
- At least three nodes are required for quorum.
- Use two network interfaces per node:
- One for management and VM traffic (Public Network)
- One dedicated to Ceph replication (Cluster Network)
Network Design
Public Network : 10.0.0.0/24 (Client I/O)
Cluster Network : 192.168.100.0/24 (Replication / Heartbeat)
๐ก Keep Ceph replication traffic separate from VM traffic to avoid latency or performance degradation.
๐งญ 4. Step-by-Step Deployment
Step 1 โ Enable Ceph Repository
apt update
apt install ceph ceph-common ceph-fuse
Proxmox VE 9.x ships with Ceph Reef or newer Ceph Squid builds by default.
Step 2 โ Initialize Ceph Cluster
In the Proxmox Web UI:
- Go to Datacenter โ Ceph โ Install Ceph
- After installation, click Create Cluster
- Define both Public and Cluster Networks
CLI method:
pveceph init --cluster-network 192.168.100.0/24 \
--network 10.0.0.0/24
Step 3 โ Add MON and MGR Nodes
pveceph mon create
pveceph mgr create
Repeat for all participating nodes or use Add Monitor in the Web UI.
Step 4 โ Create OSDs (Object Storage Daemons)
Select available disks for Ceph storage:
pveceph osd create /dev/sdb
Or use the Web GUI: Ceph โ OSD โ Create
๐ก Use SSD or NVMe drives for Cephโs DB/WAL partitions to enhance I/O performance.
Step 5 โ Create a Storage Pool
pveceph pool create ceph-pool --size 3 --min-size 2
Then add it as a Proxmox storage backend:
pvesh create /storage --storage ceph-rbd \
--type rbd --pool ceph-pool \
--monhost 10.0.0.11,10.0.0.12,10.0.0.13
๐ง 5. Performance and Reliability Design
| Setting | Recommended Configuration |
|---|---|
| Replica Count (size) | 3 replicas for production reliability |
| Min. Active Replicas (min-size) | 2 for safe I/O operations |
| Disk Layout | Use SSD/NVMe for DB/WAL; HDD for bulk data |
| Network Design | Separate Ceph cluster and public networks |
| Monitoring | Use Ceph Dashboard or ceph -s for real-time health checks |
๐๏ธ 6. Integration with HA and PBS
- With High Availability (HA):
All Proxmox nodes share the same Ceph storage pool, allowing seamless VM migration between nodes. - With Proxmox Backup Server (PBS):
Ceph RBD pools can be mounted as a backend datastore for incremental backups.
Architecture Example:
[Proxmox Cluster]โโโโ
โ (Ceph RBD)
โผ
[Ceph Storage Pool]
โ
[Proxmox Backup Server]
โ๏ธ 7. Cross-Site Replication and DR Extension
- Use Ceph RBD Mirror for real-time block-level replication between data centers.
- Combine with Proxmox Backup Server (PBS) for snapshot-based offsite backups.
- For long-distance or high-latency environments, use Active/Passive replication with periodic synchronization.
โ Conclusion
By integrating Proxmox VE with Ceph, enterprises can build a fully distributed, self-healing, and scalable storage layer
that eliminates the single points of failure common in traditional architectures.
This unified design delivers:
- Built-in high availability
- Linear scalability
- Automated recovery
- Simplified storage management
Together, Proxmox and Ceph form a powerful open-source foundation for enterprise-grade virtualization and cloud infrastructure.
๐ฌ In the next article, weโll explore
โProxmox Cloud Management and Hybrid Architecture Integrationโ,
demonstrating how to extend your Proxmox environment into hybrid and multi-cloud deployments.