Bug #112760 for CHI-Driver-Redis: Key listing set is never cleaned

It sounds like expiring members of set in Redis is a common problem. I found a few pages about different approaches: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis/ https://github.com/antirez/redis/issues/135 https://groups.google.com/forum/#!topic/redis-db/rXXMCLNkNSs The main suggested approach is to use sorted set with expiration time as score, ZRANGEBYSCORE to retrieve unexpired elements, and ZREMRANGEBYSCORE to expire elements. It looks like there is a purge() method in CHI for doing the expiration. Would it work for you to call purge() regularly? We could implement a more efficient purge() that handles the Redis keys expiring. I wonder if it would be possible to set expiration on the set to the latest expiration time and then the set of keys will expire if all keys expire. But I'm guessing that doesn't happen for you.

Hi Ian, I've extended CHI::Driver::Redis and implemented the main approach of the fixes you've mentioned on your previous message —by keeping inventory on a sorted set in contrast to a set as there would otherwise be no way to discern which keys have already expired. For the meantime, I've released this modification as a separate module under CHI::Driver::Redis::SortedSet <https://metacpan.org/pod/CHI::Driver::Redis::SortedSet> but would like to check if you'd be interested to incorporate the modifications I've made into your module so the community does not to have to deal with multiple drivers which would essentially be doing the same thing. Some implementation notes BTW: 1. Upon calling store(), instead of adding to a set to keep track of namespace members, the same key is added to a sorted set with the score set to either: a. IF key has a finite expiration — now + $expires_at b. ELSE — "+inf" (as key will never expire) 2. Likewise, when store() gets called, "cleanup" of expired keys via ZREMRANGEBYSCORE is executed. This can be costly for sparsely-used namespaces with blasts in between as the number of expired entries can be huge. That said, I still would not imagine this would have huge costs as sorted sets are automatically "house-kept" with every usage. 3. chinamespaces is still left intact as a set (and not migrated as a sorted set) as this should not be a concern either way. However, if it did, the same strategy can be applied. 4. For existing Redis datasets pre-populated with entries using CHI::Driver::Redis, switching the driver to using the new one will fail (albeit silently as one of the design goals of the original module) as the namespace inventory list data type is now different. A transparent "migration" schema should not be too difficult to implement as it will only involve: a. check if TYPE "<ns>" eq "zset", if it is, no-op (i.e. bail-out) b. foreach SMEMBERS, ZADD to "<ns_temp>", with SCORE = now + TTL c. DEL "<ns>" d. RENAME "<ns_temp>" to "<ns>" I hope this provides an outline of my thoughts. Looking forward to hearing from you. Cheers, Arnold On Tue Mar 08 13:58:37 2016, jsemaan@inverse.ca wrote: Show quoted text