Annoying detail for DEFLATE + dictionary is that the dictionary has to be provided within every reader (setDictionary in java).
The output from the DEFLATE stream is not standalone, which makes it really hard to use - a more ideal approach would've been to provide training to the Huffman tree & guess best window sizing from a sample.
But SDCH solves that problem by storing an external named dictionary - clients will read it off a common HTTP location if it's missing in the cache.
Within that, there's a pretty neat standard which SDCH uses - RFC 3284.
I ran into it trying to optimize holding a few billion serialized blogs within an in-memory store with transactional updates - the issue was mostly the in-memory sizes of serialized objects.
I ended up using a versioned external dictionary for compression, which was an extension to VCDIFF with 1 additional instruction in its streams - COPY_DICT.
The output from the DEFLATE stream is not standalone, which makes it really hard to use - a more ideal approach would've been to provide training to the Huffman tree & guess best window sizing from a sample.
But SDCH solves that problem by storing an external named dictionary - clients will read it off a common HTTP location if it's missing in the cache.
Within that, there's a pretty neat standard which SDCH uses - RFC 3284.
I ran into it trying to optimize holding a few billion serialized blogs within an in-memory store with transactional updates - the issue was mostly the in-memory sizes of serialized objects.
I ended up using a versioned external dictionary for compression, which was an extension to VCDIFF with 1 additional instruction in its streams - COPY_DICT.
That concept can be pushed further, to get to something more generic like femtozip - https://github.com/gtoubassi/femtozip/wiki/How-femtozip-work...