[Crowbar] crowbar <service> proposal create racey / chef index not populated fast enough

Ralf Haferkamp rhafer at suse.de
Fri Jul 13 08:37:45 CDT 2012


Hi,

find a very simple one attached. The patch causes chef-expander to tell 
solr to commit index immediately after every update. As I already 
mentioned I am unsure what impact this will have on chef's and solr's 
overall performance.

Another possiblity is to tweak solr's autocommit settings in 
solrconfig.xml. The current default is:
    <autoCommit> 
      <maxDocs>100</maxDocs>
      <maxTime>10000</maxTime> 
    </autoCommit>

Reducing those numbers should decrease the latency as well.

regards,
	Ralf

On Freitag 13 Juli 2012 08:16:19 Victor_Lowther at Dell.com wrote:
> I would be interested in patches that decrease the latency of chef-solr
> and chef-expander -- their lag is a source of lots of anguish and
> annoying workarounds in our current codebase.
> 
> -----Original Message-----
> From: crowbar-bounces On Behalf Of Ralf Haferkamp
> Sent: Friday, July 13, 2012 7:13 AM
> To: crowbar
> Subject: [Crowbar] crowbar <service> proposal create racey / chef index
> not populated fast enough
> 
> Hi,
> 
> we recently ran into trouble when creating multiple proposals on the
> command line. Doing
> 
> crowbar "mysql" proposal create default; (in our setup is "database"
> instead for "mysql" but that doesn't make a difference)
> and
> 
> crowbar "keystone" proposal create default;
> 
> directly after another on the commandline the keystone proposal will
> not find the mysql proposal and fallback to sqlite instead. I figured
> out that this is because the chef search for mysql proposals that
> KeystoneService does in create_proposal() does not return anything
> because the updates to solr's search indexes that creating the "mysql"
> proposal would cause, weren't committed yet. This seems to be because
> chef-expander doesn't explicitly commit it's changes to solr, but
> relies solr's autocommit settings, currently solr is configured to
> leave changes uncommitted for at most 10 seconds, this gives a pretty
> big window for failures in crowbar. And I was even able to trigger the
> problem by clicking fast enough in the WebUI :).
> 
> There are different solutions to this:
> - insert sleeps everywhere in script calling the crowbar cli. Or
> alternatively to manual knife searches until the desired chef object
> show up in the results (I would like to avoid this kind of hacks) -
> lower the autocommit settings in the solr configuration to do commits
> much faster - patch chef-expander to commit updates immediatly, the
> patch for that is very simple, but I don't know what negative
> consequence this might have for solr, and the IO system of the admin
> node :)
> 
> regards,
> 	Ralf
> 
> _______________________________________________
> Crowbar mailing list
> Crowbar at dell.com
> https://lists.us.dell.com/mailman/listinfo/crowbar
> For more information: https://github.com/dellcloudedge/crowbar/wiki
-- 
SUSE LINUX Products GmbH,
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer
HRB 16746 (AG Nuernberg)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: expander.dif
Type: text/x-patch
Size: 442 bytes
Desc: not available
Url : http://lists.us.dell.com/pipermail/crowbar/attachments/20120713/2686f900/attachment.bin 


More information about the Crowbar mailing list