Brents IT Blog

My project goal was to create a production cluster ( after creating several dev versions to get the hang of the setup problems and questions ) on four storage hosts(physical) and four virtual machines.  My environment called for a DNS server as well but i dont count that as part of the cluster as you can also do the DNS part with DNSmasq locally and you only need it if you are going to use the RADOS gateway.

So after building and burning several dev environments, i finally had a configuration that would work for me and the project.  Let me tell you, i very much appreciate all the online documentation and help from IRC, but seriously, the online documentation needs some edits that contain information on where you do what and when.  If you are building a dev environment on one server, its not an issue, but if you scale beyond that, the references are crucial.  Perhaps thats why they want you to use their consulting services?  I dont begrudge them for that though!  If you need an expert, pay them, its not like a continuing fee is required.

So without further ado, here is my version of what their documents should look like:  

Ceph Install / Deployment in a Production Environment:  Ceph Install.pdf

Note: I have removed the web version because i keep changing the document and updating both is annoying.


My Current setup:

4 Ceph servers
10 4TB drives for ceph storage ( Raid 0 )
2 1TB drives for OS ( raid 1 )
2 500GB SSDs ( direct sata, 1 drive per 5 spinning disks for log )
48GB memory
2 Quad Core 3Ghz
6 1GB connections ( 2 for public, 4 for private/ceph ) Bonded

3 Virtual Monitors
3 Virtual Rados Gateways
2 HAProxy servers handling requests from public users and internal IIS Servers ( we found that internal DNS round robin wasnt working for the rados gateway servers, one server always seemed to get backed up under heavy website loads ).