https://sidneyfletcher.com/posts/Recent content in Posts on Sidney FletcherHugoen-usThu, 20 Aug 2026 00:00:00 +0000https://sidneyfletcher.com/understanding-llms-from-scratch-tokenizers/Thu, 20 Aug 2026 00:00:00 +0000https://sidneyfletcher.com/understanding-llms-from-scratch-tokenizers/<p>You wake up, Rip Van Winkle-like, from a deep sleep. It&rsquo;s 2026 and LLM&rsquo;s are taking over the world. Time to learn something about how these work, you think (and write up your notes).</p> <p>Best to start at the very beginning of the LLM processing pipeline - what is a tokenizer and how does it work?</p> <h2 id="what-are-tokenizers">What are Tokenizers?</h2> <p>At its most basic, a tokenizer is a way to take a string of text and turn it into a list of numbers that a computer can understand.</p>