refactor: introduce shared tokenizer abstraction and split implementations#1423

leejet merged 1 commit intomasterleejet/stable-diffusion.cpp:masterfrom

tokenizer_refactorleejet/stable-diffusion.cpp:tokenizer_refactorCopy head branch name to clipboard