Ali Shaikh

AI

How AI Models Process Text: Understanding Tokens in AI and LLMs with TikTokenizer

In the world of artificial intelligence (AI) and large language models (LLMs), tokens are the essential building blocks that allow these systems to process and understand text. But what exactly are tokens, and how can a tool like TikTokenizer help us explore them? Let’s dive in. What Are Tokens?

AI

Run a Local LLM with LM Studio

Run Llama|Mistral|Phi|Gemma|DeepSeek|Qwen 2.5 locally on your computer! Picture this: your own AI buddy, running right on your computer, ready to chat, code, or solve problems—no internet, no fees, just you and your machine. That’s what running a Large Language Model (LLM) locally