๐ฐ Introduction
As enterprise virtualization and cloud adoption continue to accelerate,
the demand for high-availability, scalable distributed storage has become a cornerstone of modern infrastructure design.
Ceph, the leading open-source distributed storage system, provides unified support for:
- Block storage (RBD)
- Object storage (RGW)
- File storage (CephFS)
Within Proxmox VE clusters, Ceph is deeply integrated โ
enabling administrators to deploy, configure, and monitor distributed storage directly through the Proxmox Web GUI or CLI tools.
This article outlines:
- Cephโs architecture within Proxmox clusters
- Node and network design recommendations
- Step-by-step deployment
- Performance tuning and optimization
- Monitoring and maintenance strategies
๐งฉ 1. Ceph Architecture Overview
Core Components
| Component | Description |
|---|---|
| MON (Monitor) | Maintains cluster maps and ensures quorum; at least three nodes recommended for HA. |
| OSD (Object Storage Daemon) | Manages data on physical disks; each drive typically corresponds to one OSD daemon. |
| MDS (Metadata Server) | Manages directory and metadata operations for CephFS. |
| RGW (RADOS Gateway) | Provides an S3/Swift-compatible object storage interface. |
| MGR (Manager) | Provides monitoring, metrics, and external API interfaces (e.g., Prometheus). |
In Proxmox, Ceph components like MON, OSD, and MGR can be deployed directly from the Web GUI โ
tightly integrating compute and storage management within a single cluster.
โ๏ธ 2. Cluster Architecture and Node Design
Recommended Topology
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Proxmox Cluster โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Node1 (Compute + MON + OSD + MGR) โ
โ Node2 (Compute + MON + OSD + MGR) โ
โ Node3 (Compute + MON + OSD + MGR) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Ceph Public Network (10GbE) โ
โ Ceph Cluster Network (10GbE) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Network Planning
| Network Type | Function | Recommended Bandwidth | Example CIDR |
|---|---|---|---|
| Public Network | VM โ Ceph communication | 10 GbE or higher | 172.16.10.0/24 |
| Cluster Network | OSD replication and backfill | 10 GbE dedicated | 172.16.20.0/24 |
| Management Network (optional) | SSH / GUI / control traffic | 1 GbE | 172.16.5.0/24 |
๐ Separate network interfaces for public and cluster traffic are strongly recommended to prevent I/O congestion and ensure stability.
๐ง 3. Ceph Deployment Steps (Proxmox VE Example)
1๏ธโฃ Install Ceph Packages on All Nodes
apt update
apt install ceph ceph-common ceph-mgr ceph-mon ceph-osd
Or use the Proxmox GUI:
Datacenter โ Ceph โ Install Ceph
2๏ธโฃ Initialize MON and MGR Services
pveceph init --cluster-network 172.16.20.0/24 --network 172.16.10.0/24
pveceph createmon
pveceph createmgr
Verify status:
ceph -s
3๏ธโฃ Create OSDs
pveceph createosd /dev/sdb
pveceph createosd /dev/sdc
Check status:
ceph osd tree
4๏ธโฃ Create a Storage Pool
ceph osd pool create vm-pool 128 128
Register it as a Proxmox storage target:
pvesm add rbd vmstore --pool vm-pool --monhost 172.16.10.11
5๏ธโฃ Enable Compression and Balancing
ceph osd pool set vm-pool compression_algorithm lz4
ceph balancer status
โก 4. Performance Optimization Guidelines
1๏ธโฃ Hardware Recommendations
| Category | Recommended Configuration |
|---|---|
| OSD Disks | SSD / NVMe preferred; journals on faster media |
| MON / MGR Nodes | Deploy on SSDs |
| Network | Dual 10 GbE+ links with Jumbo Frames enabled |
| CPU / RAM | Minimum 8 cores and 32 GB RAM per node |
2๏ธโฃ Key Ceph Parameters to Tune
| Parameter | Recommended Value | Description |
|---|---|---|
osd_max_backfills | 2โ3 | Controls number of simultaneous backfills |
osd_recovery_max_active | 3โ4 | Balances recovery load with active I/O |
osd_op_queue | wpq | Enables Write Priority Queue for better latency |
bluestore_cache_size | 4โ8 GB | Improves metadata performance |
filestore_max_sync_interval | 10 | Increases write buffer interval to boost throughput |
3๏ธโฃ Proxmox Integration Optimizations
- Use CephFS or RBD for VM storage.
- Enable Writeback cache (ensure UPS-backed power).
- Use IO Threads to leverage multi-core performance.
- Disable unnecessary automatic snapshot jobs.
๐ 5. Monitoring and Maintenance
1๏ธโฃ Proxmox GUI Monitoring
Navigate to:
Datacenter โ Ceph โ Status
Provides real-time cluster health, capacity usage, and OSD performance graphs.
2๏ธโฃ Common CLI Monitoring Commands
ceph df
ceph osd perf
ceph health detail
3๏ธโฃ Prometheus + Grafana Integration
Enable the Prometheus module in Ceph:
ceph mgr module enable prometheus
Visualize performance metrics (IOPS, latency, recovery speed) using Grafana dashboards.
๐ 6. Fault Tolerance and High Availability Strategies
- Deploy at least three MON nodes to maintain quorum.
- Use triple replication or erasure coding (k=2, m=1) for fault tolerance.
- Sync critical data to a remote Ceph or PBS backup cluster.
- Use cephadm or Ansible for automated upgrades and rolling maintenance.
- Enable Ceph Dashboard for visual cluster management.
โ Conclusion
Ceph is one of the most resilient and scalable open-source storage platforms for enterprise virtualization.
In a Proxmox cluster, it not only provides native integration for virtual machines and containers
but also forms the foundation for high-availability, cross-site redundancy, and disaster recovery.
With proper node planning, dual-network segmentation, tiered storage, and performance tuning,
Ceph can evolve into the enterpriseโs distributed storage backbone โ delivering:
High Availability (HA) ยท Scale-Out Performance ยท Operational Resilience
๐ฌ Coming next:
โCephFS vs. RBD โ Performance and Application Use Case Comparisonโ
A detailed analysis of which storage type fits best in VM, container, and hybrid workloads.