So, it seems that prior to vCenter 4.0 it is not possible to use an RSA key length longer than 1024 bits for SSL.
I came to this discovery because for the last few days I’ve been replacing the default SSL certs for each of the ESX (3.5u4) hosts with certs that are signed by our local CA (and therefore trusted automatically). Our security office requires a key length of 2048 bits. Nothing less.
So, when I replaced the cert and tried reconnecting my hosts I got an error.
Not being one to be deterred by “a database schema limitation”, I decided to fix the problem. The kb article here tells us what the error is: “string too large for database”, and there is also some references to the password’s inability to be decrypted.
In my test environment I decided to do a test…modifying the database schema so that it can hold the encrypted password in the field. The default schema (for vCenter 2.5 anyway) has the VPX_HOST.PASSWORD field as an NVARCHAR(255). Using a 1024 bit RSA key the encrypted passwords are 174 characters long. The same password encrypted with a 2048 bit RSA key is just over 340 characters. So, we see what the problem is.
Before I go any further, backup your database. This page isn’t going anywhere, go ahead and do it now.
So, a simple SQL command:
ALTER TABLE VPX_HOST
ALTER COLUMN password NVARCHAR(512)
You may or may not get a warning if you are using SSMS (depending on your version I believe) about operations that cause a table to be dropped. The reason you may see this error (even if you don’t see the error it still operates this way) is that in order to change the column’s data type the table is dropped and rebuilt. If you are doing this on a very large table it can take some time to complete, during which time the table is unavailable. At most, I would assume that the hosts table will have a couple hundred rows (the number of rows is the same as the number of ESX hosts you have in vCenter). Even with a few thousand rows, this operation will only take a few seconds to complete.
If you want to be safe, turn off the vCenter service prior to the operation, but I don’t think it’s necessary.
Of course, doing this will probably cause VMware to laugh at you if you open a support issue about the database being broken. Additionally, should you update vCenter 2.5 (say from update 4 to update 5), the change could be undone.
Once you’ve made the change, go about the normal way of replacing vCenter’s certs with your own, and when you reconnect the hosts you shouldn’t get any “string too long” errors.
I haven’t tried replacing the certs for my vCenter 4.0 instance with 2048 bit, so I don’t know if this is still needed, but from reading the documentation it seems that it shouldn’t be.