Handle mmap failures for redblack tree cache

The redblack tree cache is totally optional, so if we can't allocate
room for the cache, then just pretend as if the cache is full if mmap
fails

Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>