memcached opinions

[Luke Macken]

On Wed, Sep 09, 2009 at 04:33:19PM -0400, Luke Macken wrote:
> On Wed, Jul 01, 2009 at 05:50:12PM -0700, Toshio Kuratomi wrote:
> > On 07/01/2009 05:39 PM, Mike McGrath wrote:
> > > On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
> > > 
> > >> On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrath<mmcgrath at redhat.com> wrote:
> > >>> On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
> > >>>
> > >>>> On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrath<mmcgrath at redhat.com> wrote:
> > >>>>> I'm not sure if we have any memcached experience on the list but I thought
> > >>>>> I'd ask.  Can anyone explain this:
> > >>>>>
> > >>>>> http://pastebin.ca/1481219
> > >>>>>
> > >>>>> Notice how memcached1 has a much higher hit rate and memcached2 has a much
> > >>>>> lower hit rate?
> > >>>>>
> > >>>>
> > >>>> The time for memcached1 is 5x less than memcached2 being up. That can
> > >>>> have an effect on caching as right after a system comes up its rates
> > >>>> are usually much higher and then over time fall off (iirc). I think it
> > >>>> would take bringing both up at the same time to figure out if there is
> > >>>> a true disparity over caching.
> > >>>>
> > >>>
> > >>> I thought that exact same thing :)
> > >>>
> > >>> memcahed1:
> > >>> STAT uptime 9143
> > >>> STAT get_hits 311736
> > >>> STAT get_misses 11255
> > >>>
> > >>> memcached2:
> > >>> STAT uptime 9144
> > >>> STAT get_hits 49679
> > >>> STAT get_misses 11116
> > >>>
> > >>
> > >> Now that shows something not kosher. My guess is some app is not
> > >> talking to both? What apps use memcached for what?
> > >>
> > > 
> > > I was just talking to ricky about this a bit in IRC.  So here's the scoop.
> > > 
> > > We've got mediawiki using memcached for a couple of things, including
> > > session data (which is weird and wrong but fast).
> > > 
> > > The recent addition to the group is Fedora community, specifically in it's
> > > implementation of beaker.  I'm going to get ahold of luke tomorrow to
> > > verify and test some stuff but I think this line in the config:
> > > 
> > > beaker.cache.url = memcached1;memcached
> > > 
> > > I'm not sure how beaker reads that, but I suspect it might be only sending
> > > information to memcached1 and ignoring memcached2 altogether.  If this
> > > theory holds it'd explain why memcached1 not only has a higher request
> > > rate but also a higher hit rate because I suspect fedoracommunity requests
> > > some of the same info over and over again compared to the wiki which
> > > probably has a broader data pool it pulls from.
> > > 
> > easy test: reverse that:
> > 
> > beaker.cache.url = memcached2;memcached1
> > 
> > and see if the cache hit ratio reverses itself.
> > 
> > -Toshio
> 
> So, I've been poking at this a little bit lately, and I'm thinking there are
> some problems with the way Fedora Community is utilizing it's Beaker cache &
> memcached.
> 
> The fact that something is wrong with the caching setup becomes obvious when
> playing with the Bugzilla grid *should* cache the first 5 pages, and be very
> snappy, as it is in my local instance.  However, that is not the case, which
> makes me think it's not hitting our memcached servers.
> 
> I wrote up a little test script on app1::
> 
>     from beaker.cache import CacheManager
> 
>     memcached1 = CacheManager(type='ext:memcached', url='memcached1', lock_dir='.')
>     memcached2 = CacheManager(type='ext:memcached', url='memcached2', lock_dir='.')
> 
>     def createfunc(*args):
>             print "createfunc(%s)" % (args,)
> 
>     def get_value(cache, value):
>             cache_1 = memcached1.get_cache(cache)
>             cache_2 = memcached2.get_cache(cache)
>             result1 = cache_1.get_value(key=value, createfunc=createfunc)
>             result2 = cache_2.get_value(key=value, createfunc=createfunc)
>             print "memcached1[%s][%s] = %s" % (cache, value, result1)
>             print "memcached2[%s][%s] = %s" % (cache, value, result2)
>             return result1, result2
> 
>     get_value('fedoracommunity_alerts_global', 'today')
>     get_value('bodhi', 'dashboard_None')
> 
> Which produces::
> 
>     memcached1[fedoracommunity_alerts_global][today] = None
>     memcached2[fedoracommunity_alerts_global][today] = None
>     memcached1[bodhi][dashboard_None] = None
>     memcached2[bodhi][dashboard_None] = None
> 
> I also tried using netcat to query for these by hand, to no avail.
> 
> So, it looks like we need to look a bit deeper into what is going on here.
> Either I'm Doing It Wrong with Beaker, or we're hitting a bug somewhere.

I did a little more testing with this today.

On app1 I was unable to get fedoracommunity to put any values in
memcached, and I was not seeing any errors either.  I tried this with
both the mod_wsgi deployment, and running paster locally.  I then ran
`curl http://localhost:8080/apps/fedoracommunity.alerts/` to trigger the
cache, and then ran the script I mentioned earlier to see if I could
retrieve those values.

I was able to reproduce this failure on a local RHEL5 VM.  However,
after upgrading to python-beaker-1.4 *and* restarting memcached, my
script *worked*. (the app just silently returned None for all requests
if I didn't restart memcached after upgrading Beaker)

Looking at the ChangeLog for Beaker 1.4, I'm thinking this could have
done the trick:

    * Fixed bug with CacheMiddleware overwriting configuration with default
      arguments despite prior setting.

It'll be tough to test this new package in staging, as earlier today I
couldn't get any of the cached apps to load due to koji/fas problems in
staging.  When we deploy this production, we'll need to make sure to
restart our memcache daemons.

luke