[Crowbar] How to deploy Swift?

Andi_Abes at Dell.com Andi_Abes at Dell.com
Wed Jan 11 08:55:22 CST 2012


It seems that the ring-compute doesn’t correctly find the storage nodes, and hence doesn’t add them to the rings – causing the error (which basically says – you have no disks).
This can be caused by various reasons (some already covered in this thread).  A few others to look at:

·         Network setup – do you have the storage network configured to use vlans (the default)? Is your network equipment capable of handling vlans, and correctly configured ? if not storage node configuration might fail, leading to it not being discovered by the ring-compute code.

·         How are the 12 drives configured – are they RAID’d in any which way? JBOD? – the swift recipes require the disks to be visible as separate devices to correctly be identified (i.e. /dev/sda, /dev/sdb etc) [hence they should be configured as JBOD) [see below *]

·         Where there prior errors in the chef execution on the storage nodes?


For further troubleshooting it would be useful to look at errors for all the nodes involved (you can find the logs from all the chef-client executions that are triggered by crowbar in /opt/dell/crowbar_framework/log/<node-name>.log



* (the “full” version of crowbar has code to configure Dell RAID controllers to the “right” config based on the role)

From: crowbar-bounces On Behalf Of i3D.net - Tristan van Bokkem
Sent: Wednesday, January 11, 2012 9:11 AM
To: Paul Regan
Cc: crowbar
Subject: Re: [Crowbar] How to deploy Swift?

Thanks, but I am not sure if I am facing the same problem. I have 3 nodes with each 12 1TB disks. There should plenty of space available.

Best regards,

Tristan van Bokkem
Datacenter Operations

Contact:
E-mail Personal: tristanvanbokkem at i3d.net<mailto:tristanvanbokkem at i3d.net>
E-mail Support: info at i3d.net<mailto:info at i3d.net>
E-mail NOC: noc at i3d.net<mailto:noc at i3d.net>
Website: http://www.i3d.net Office:
Interactive 3D B.V.
Meent 93b
3011 JG Rotterdam
The Netherlands

Visit www.smartdc.net<http://www.smartdc.net> – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP.

Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies.
________________________________
From: Paul Regan [mailto:me at paulregan.co.uk]
To: i3D.net - Tristan van Bokkem [mailto:tristanvanbokkem at i3d.nl]
Cc: crowbar at lists.us.dell.com
Sent: Wed, 11 Jan 2012 14:47:23 +0100
Subject: Re: [Crowbar] How to deploy Swift?
Hi

May or may not help but I had the same error and it was due to not enough storage on the nodes.  A small lab and I'd only connected one disk on each node, which the OS had grabbed.  Once I added more storage I still had barclamp deploy failures and ended up starting the crowbar install from the beginning.

I'm now looking at all green lights and reading the OpenStack manuals

On 11 January 2012 13:05, i3D.net - Tristan van Bokkem <tristanvanbokkem at i3d.nl<mailto:tristanvanbokkem at i3d.nl>> wrote:
Coming back to this. I am still unable to deploy swift correctly. I am now testing with actual hardware instead of vmware but the problem remains. The made sure the storage nodes are only 2 and tried both with keystone and swauth as Authentication method.

Some log output of one of the clients:

[Wed, 11 Jan 2012 07:41:11 -0600] ERROR: Running exception handlers
[Wed, 11 Jan 2012 07:41:11 -0600] FATAL: Saving node information to /var/cache/chef/failed-run-data.json
[Wed, 11 Jan 2012 07:41:11 -0600] ERROR: Exception handlers complete
[Wed, 11 Jan 2012 07:41:11 -0600] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
[Wed, 11 Jan 2012 07:41:11 -0600] FATAL: Chef::Exceptions::ShellCommandFailed: execute[rebalance ring for /etc/swift/account.builder] (/var/cache/chef/cookbooks/swift/providers/ringfile.rb line 211) had and an error: Expected process to exit with [0, 1], but received '2'
---- Begin output of swift-ring-builder /etc/swift/account.builder rebalance ----
STDOUT: -------------------------------------------------------------------------------
An error has occurred during ring validation. Common
causes of failure are rings that are empty or do not
have enough devices to accommodate the replica count.
Original exception message:
 There are no devices in this ring, or all devices have been deleted
-------------------------------------------------------------------------------
STDERR: /usr/bin/swift-ring-builder:534: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  "Original exception message:\n %s" % e.message
---- End output of swift-ring-builder /etc/swift/account.builder rebalance ----
Ran swift-ring-builder /etc/swift/account.builder rebalance returned 2




Best regards,

Tristan van Bokkem
Datacenter Operations

Contact:
E-mail Personal: tristanvanbokkem at i3d.net<mailto:tristanvanbokkem at i3d.net>
E-mail Support: info at i3d.net<mailto:info at i3d.net>
E-mail NOC: noc at i3d.net<mailto:noc at i3d.net>
Website: http://www.i3d.net Office:
Interactive 3D B.V.
Meent 93b
3011 JG Rotterdam
The Netherlands

Visit www.smartdc.net<http://www.smartdc.net> – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP.

Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies.
________________________________
From: Andi_Abes at Dell.com<mailto:Andi_Abes at Dell.com>
To: tristanvanbokkem at i3d.nl<mailto:tristanvanbokkem at i3d.nl>, crowbar at lists.us.dell.com<mailto:crowbar at lists.us.dell.com>
Sent: Fri, 09 Dec 2011 20:17:53 +0100

Subject: RE: [Crowbar] How to deploy Swift?
The current head has 1.4.4, and AFAIK, it is working.
But we’re still hammering at it.
The Diablo crowbar has all kinds of other goodies in it, and should be blessed soon.

From: i3D.net - Tristan van Bokkem [mailto:tristanvanbokkem at i3d.nl<mailto:tristanvanbokkem at i3d.nl>]
Sent: Friday, December 09, 2011 9:45 AM
To: Abes, Andi; crowbar
Subject: RE: [Crowbar] How to deploy Swift?

Shall I wait for Swift 1.4.4 then?

Best regards,

Tristan van Bokkem
Datacenter Operations

Contact:
E-mail Personal: tristanvanbokkem at i3d.net<mailto:tristanvanbokkem at i3d.net>
E-mail Support: info at i3d.net<mailto:info at i3d.net>
E-mail NOC: noc at i3d.net<mailto:noc at i3d.net>
Website: http://www.i3d.net Office:
Interactive 3D B.V.
Meent 93b
3011 JG Rotterdam
The Netherlands

Visit www.smartdc.net<http://www.smartdc.net> – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP.

Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies.
________________________________
From: Andi_Abes at Dell.com<mailto:Andi_Abes at Dell.com>
To: tristanvanbokkem at i3d.nl<mailto:tristanvanbokkem at i3d.nl>, crowbar at lists.us.dell.com<mailto:crowbar at lists.us.dell.com>
Sent: Fri, 09 Dec 2011 15:00:45 +0100
Subject: RE: [Crowbar] How to deploy Swift?
Crowbar will take care of discovering the disks, partitioning them and all the tidbits that need to be taken care of.
We recently updated to swift 1.4.4, and one of the utilities we use changed its return values (swift-ring-builder in particular) – this was causing failures to deploy swift in builds during last week.
I just pushed a change last evening (US-Eastern time) to fix that. The joy of being on the bleeding edge… out QA folks are still hammering on this, but it seems to work ok in my environment.


A few trouble shooting notes:

•         When things go bad, it’s time to hit the logs:

o   /opt/dell/crowbar_framework/log/production.log has the crowbar admin app outout

o   In the same directory there are _chef_client.log files, which record the output from chef during the application of proposals.

Look in both to see if there are any “interesting” errors, like chef exceptions. Those would be very useful to identify the issue


•         The swift proxy has the ring files in /etc/swift/<>.builder. You can use the swift-ring-builder utility to inspect the rings, and make sure they contain what you’d expect – i.e. all the drives to be used from all the storage nodes


In theory, swift is simple… so there aren’t that many crowbar –swift specific docs… I’ll see what I can add.

Hth,
a.




_______________________________________________
Crowbar mailing list
Crowbar at dell.com<mailto:Crowbar at dell.com>
https://lists.us.dell.com/mailman/listinfo/crowbar
For more information: https://github.com/dellcloudedge/crowbar/wiki

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/crowbar/attachments/20120111/3cebc23f/attachment-0001.html 


More information about the Crowbar mailing list