[Crowbar] Support for multiple Nova zones

Andi_Abes at Dell.com Andi_Abes at Dell.com
Thu Jan 5 15:02:59 CST 2012

You're right - if you lose a nova-compute node, then you lose only the VM's that are running on it.
The impact of losing other services (network, api, schedulers etc) depends on on your deployment options.

If you deploy nova-network in HA mode (with the crowbar scheme), effectively you get multiple nova-api and nova-network instances (1 per compute node). So there's no central dependency.
We're working on making other infrastructure services (mysql, rabbit etc) be more HA.

Re: automatic recovery of VM's running on a failed compute node, openstack offers various options, but they'll need to be customized to your specific situation.
This might be a good start:


From: Chris Sanburn [mailto:csanburn at redwoodit.com]
Sent: Thursday, January 05, 2012 12:10 PM
To: Abes, Andi
Subject: RE: Support for multiple Nova zones

Great explanation Andi, thanks! I know it's not actually a crowbar issue but when you said:
"if you lose that zone, and whatever  is running in that VM doesn't have application level high-availability measures - you're hosed."
If just one nova node in the zone goes down then it's just the VMs that were assigned to that node that are hosed, assuming they had no application level HA,  and the rest of the nodes continue on with their assigned VMs?

I'm just trying to confirm what I suspect is true. I believe my boss would like to, ideally, have us deploy an openstack cloud that can continue running when one or two nodes goes down. But it appears to me that it doesn't support that in all aspects. I haven't seen any evidence that you can have a nova compute node drop and the VMs it was hosting automatically get activated on a remaining nova compute node.

If your novel is about openstack & crowbar I'll have to get a copy :) Four months ago I'd never worked with either one, so I've got a lot to learn yet.


From: Andi_Abes at Dell.com [mailto:Andi_Abes at Dell.com]
Sent: Thursday, January 05, 2012 11:31 AM
To: Chris Sanburn
Cc: crowbar at lists.us.dell.com
Subject: RE: Support for multiple Nova zones

You are right that in swift, zones are the basic unit of availability - swift will guarantee that a different replicas of a file is present in different zones (so if you have 3 replicas, each copy will be in a different zone). So, if you have a zone represent a rack and you have 5 racks, and swift has 3 replicas - a file will exist on 3 racks. If you lose a whole rack, you will still have 2 separate copies on the surviving racks.

Nova works a bit differently....
A VM will be running on 1 compute node.  If that compute node fails, you've lost the VM (there some stuff around live-migration and various discussions on getting high-availability at the VM level - but that's WIP).
What availability zones in nova allow you to do is choose which zone you put your VM in. if you lose that zone, and whatever  is running in that VM doesn't have application level high-availability measures - you're hosed.

But lets assume for a second that you're running a cluster of web-servers. If you setup your cloud in a way that lets you put different instances of the web servers in different zones (i.e. racks). If you lose a rack, there're still servers running in the other one. For this setup, you'd use nova zones. Each zone is

If you're looking just to have high availability for a single VM, you don't really need separate zones. You'd need to make your 1 nova zone as resilient as possible. i.e. deploy each of the components in a highly available fashion - e.g. 2 nova API, nova-network in HA mode, Rabbit MQ, Mysql and such. You will still not be protected for the failure of a compute node, but your overall cluster will be resilient.

During the Essex design session, there was an interesting discussion about nova-ha - generalizing the description above into 2 classes of cloud users, and their expectations:

-          Legacy workloads - expect the same HA that high quality servers provide.

-          Cloud Friendly workloads -  e.g. web-server clusters or swift - where the application is designed to deal with failure

Nova currently doesn't fully handle the legacy workloads too nicely.  There were various arguments for and against handling them better and some folks felt strongly that better support should be provided. There are options out there, and at an extreme you can setup some hypervisors to execute the same VM on more than 1 physical machine (VMWare calls it fault-tolerance, in some other contexts it's referred  to as lockstep execution).

I think I'll go write a great American novel next....it'd be shorter.
Hope this helps.

From: Chris Sanburn [mailto:csanburn at redwoodit.com]<mailto:[mailto:csanburn at redwoodit.com]>
Sent: Thursday, January 05, 2012 9:44 AM
To: Abes, Andi
Subject: RE: Support for multiple Nova zones

Thatnks for the information. Basically I think my boss is wanting to do this test to see if zones will give us redundancy when some of the compute nodes fail or are taken offline. Our end goal is to build a cloud that is highly available and continues to work in the event that a node fails or needs to be taken offline for some other reason. Perhaps we're mistaken in our assumption that you need zones to accomplish this?
For instance, I've already tested my Swift by copying files to it and then removing one of my 3 nodes, one at a time, and confirming I could still access the files stored on it.

Chris Sanburn

From: Andi_Abes at Dell.com<mailto:Andi_Abes at Dell.com> [mailto:Andi_Abes at Dell.com]<mailto:[mailto:Andi_Abes at Dell.com]>
Sent: Wednesday, January 04, 2012 8:52 PM
To: Chris Sanburn; crowbar at lists.us.dell.com<mailto:crowbar at lists.us.dell.com>
Subject: RE: Support for multiple Nova zones

You are right that there's no explicit support for  nova zones  - zones in nova can be used for lots of different reasons (as the page you're pointing to lists). And as opposed to swift, where zones are integral to the operation of the system, for nova they're mostly optional.

I'm curious what you're trying to achieve w/ zones in nova?

All that said and done, you could probably achieve a multi-zone nova deployment with crowbar. Crowbar allows you to deploy multiple clusters of any openstack component - you would deploy 2 independent nova clusters, and then link them in a parent-child relationship.

You would need to futz with some of the configuration to add capabilities and use a zone aware scheduler..... all based on the page you referenced.
I'd be interested to hear of your experience.

From: crowbar-bounces On Behalf Of Chris Sanburn
Sent: Wednesday, January 04, 2012 1:14 PM
To: crowbar
Subject: [Crowbar] Support for multiple Nova zones

I was asked to setup child zones and test that on my crowbar openstack test deployment. But when I read the Zone documentation here:
It states:
"At the very least a Zone requires an API node, a Scheduler node, a database and RabbitMQ."

It occurs to me that I've not seen anything in the Nova barclamp to setup zones, like there is in the Swift barclamp. Has anyone done this? Is it possible with Crowbar?

-Chris Sanburn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/crowbar/attachments/20120105/74868eb9/attachment-0001.html 

More information about the Crowbar mailing list