[Crowbar] crowbar <service> proposal create racey / chef index not populated fast enough

Victor_Lowther at Dell.com Victor_Lowther at Dell.com
Fri Jul 13 08:16:19 CDT 2012

I would be interested in patches that decrease the latency of chef-solr and chef-expander -- their lag is a source of lots of anguish and annoying workarounds in our current codebase.

-----Original Message-----
From: crowbar-bounces On Behalf Of Ralf Haferkamp
Sent: Friday, July 13, 2012 7:13 AM
To: crowbar
Subject: [Crowbar] crowbar <service> proposal create racey / chef index not populated fast enough


we recently ran into trouble when creating multiple proposals on the command line. Doing

crowbar "mysql" proposal create default; (in our setup is "database" instead for "mysql" but that doesn't make a

crowbar "keystone" proposal create default;

directly after another on the commandline the keystone proposal will not find the mysql proposal and fallback to sqlite instead. I figured out that this is because the chef search for mysql proposals that KeystoneService does in create_proposal() does not return anything because the updates to solr's search indexes that creating the "mysql" 
proposal would cause, weren't committed yet. This seems to be because chef-expander doesn't explicitly commit it's changes to solr, but relies solr's autocommit settings, currently solr is configured to leave changes uncommitted for at most 10 seconds, this gives a pretty big window for failures in crowbar. And I was even able to trigger the problem by clicking fast enough in the WebUI :).

There are different solutions to this:
- insert sleeps everywhere in script calling the crowbar cli. Or alternatively to manual knife searches until the desired chef object show up in the results (I would like to avoid this kind of hacks)
- lower the autocommit settings in the solr configuration to do commits much faster
- patch chef-expander to commit updates immediatly, the patch for that is very simple, but I don't know what negative consequence this might have for solr, and the IO system of the admin node :)


Crowbar mailing list
Crowbar at dell.com
For more information: https://github.com/dellcloudedge/crowbar/wiki

More information about the Crowbar mailing list