Uploaded image for project: 'BrainzUtils'
  1. BrainzUtils
  2. BU-25

Cache namespace versions don't work in docker or with distributed hosts



    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Python
    • Labels:


      We have some functionality in BU cache for "cache namespace versions", where each namespace can have a version:
      namespace:v:key = value
      where v is an increasing integer. We can increase a namespace version to quickly invalidate all cache items for a namespace.
      The current namespace version is stored on disk. This is a holdback from when we used memcache, and we didn't want to lose this value.

      The problem with storing the namespace versions on disk is that if we have a service running on two different physical hosts, they have no way of synchronising their namespace version numbers, so one process may be working with version 1, and another process on another host could be with version 2.
      This is also an issue in docker if we don't share the namespace versions between containers using a shared volume, or something.

      Only critiquebrainz uses the namespace versions to invalidate caches. LB and AB don't use it. Perhaps there are better ways of doing this without using files on disk to track namespace versions:

      • Now that we use redis, ensure that redis serialises to disk and store the version here. This should work because even if redis restarts, we'll have access to the old data
      • See if there's some functionality in redis already to replicate this functionality




            • Assignee:
              iliekcomputers Param Singh
              alastairp Alastair Porter
            • Votes:
              0 Vote for this issue
              3 Start watching this issue


              • Created:


                Version Package