Brents IT Blog

Random thoughts by an IT GOAT


Ceph Install and Deployment in a production environment


My project goal was to create a production cluster ( after creating several dev versions to get the hang of the setup problems and questions ) on four storage hosts(physical) and four virtual machines.  My environment called for a DNS server as well but i dont count that as part of the cluster as you can also do the DNS part with DNSmasq locally and you only need it if you are going to use the RADOS gateway.

So after building and burning several dev environments, i finally had a configuration that would work for me and the project.  Let me tell you, i very much appreciate all the online documentation and help from IRC, but seriously, the online documentation needs some edits that contain information on where you do what and when.  If you are building a dev environment on one server, its not an issue, but if you scale beyond that, the references are crucial.  Perhaps thats why they want you to use their consulting services?  I dont begrudge them for that though!  If you need an expert, pay them, its not like a continuing fee is required.

So without further ado, here is my version of what their documents should look like:  

Ceph Install / Deployment in a Production Environment:  Ceph Install.pdf

Note: I have removed the web version because i keep changing the document and updating both is annoying.

My Current setup:

4 Ceph servers
10 4TB drives for ceph storage ( Raid 0 )
2 1TB drives for OS ( raid 1 )
2 500GB SSDs ( direct sata, 1 drive per 5 spinning disks for log )
48GB memory
2 Quad Core 3Ghz
6 1GB connections ( 2 for public, 4 for private/ceph ) Bonded

3 Virtual Monitors
3 Virtual Rados Gateways
2 HAProxy servers handling requests from public users and internal IIS Servers ( we found that internal DNS round robin wasnt working for the rados gateway servers, one server always seemed to get backed up under heavy website loads ).

Comments (2) -

Thanks for nice article.
In this article, you have mentioned to run ""ceph-authtool"on admin node. (Section 5 , b 1). Bit ceph-authtool is not availble on admin pc, it is on client pc.
Please clarify.


The ceph admin node is going to be the node that the cluster was created with.  Generally this is the first monitor instance that was created.  For the purposes of this article, it was the server created in section 1 starting at step 1.  I note in section 1 step 10 that the server you are working on is the Admin node.  Hope this helps clarify.

Further explanation:
You have to create the keyring on the Main Ceph admin node( its the node that is authorized to create and distribute keyrings ).  By default only the main Ceph admin node is authorized to create keyrings.  This is a security feature, you can add additional nodes as authorized to make keyrings, but its not recommended unless you have a large cluster.


Add comment