Perhaps I'm mistaken because I thought blingfire was mostly a tokenizer. But now I realize it might also be a language model.

How are tokens/phrases/documents represented, computationaly? Theoretically, do they live in a vector space?