[Crowbar] crowbar <service> proposal create racey / chef index not populated fast enough

Ralf Haferkamp rhafer at suse.de
Fri Jul 13 07:13:25 CDT 2012


we recently ran into trouble when creating multiple proposals on the 
command line. Doing

crowbar "mysql" proposal create default;
(in our setup is "database" instead for "mysql" but that doesn't make a 

crowbar "keystone" proposal create default;

directly after another on the commandline the keystone proposal will not 
find the mysql proposal and fallback to sqlite instead. I figured out 
that this is because the chef search for mysql proposals that 
KeystoneService does in create_proposal() does not return anything 
because the updates to solr's search indexes that creating the "mysql" 
proposal would cause, weren't committed yet. This seems to be because 
chef-expander doesn't explicitly commit it's changes to solr, but relies 
solr's autocommit settings, currently solr is configured to leave changes 
uncommitted for at most 10 seconds, this gives a pretty big window for 
failures in crowbar. And I was even able to trigger the problem by 
clicking fast enough in the WebUI :).

There are different solutions to this:
- insert sleeps everywhere in script calling the crowbar cli. Or 
alternatively to manual knife searches until the desired chef object show 
up in the results (I would like to avoid this kind of hacks)
- lower the autocommit settings in the solr configuration to do commits 
much faster
- patch chef-expander to commit updates immediatly, the patch for that is 
very simple, but I don't know what negative consequence this might have 
for solr, and the IO system of the admin node :)


More information about the Crowbar mailing list