Start with the simple explanation, see a concrete example, then go deeper when you want the technical definition. Every term also explains why it matters when you run a model on a phone or Mac.
38 terms, explained from simple to technical
0–9
1-bit quantization
Also: 1-bit model · one-bit model
A very aggressive way to shrink a model so that it uses much less storage and memory.
Example
A 27B model that would normally be far too large for a phone may become small enough to try when released in a purpose-built 1-bit format.
Technical explanation
A quantization scheme that represents each model weight with roughly one bit on average. Real file sizes are usually larger than the theoretical minimum because formats also store scales, metadata, and sometimes higher-precision values.
Why it matters on your phone
It can make unusually large models fit on phones, but compatibility, speed, and output quality depend heavily on the specific format and runtime.
The model can work with roughly 4,000 pieces of text from the current conversation at once.
Example
A short chat or article may fit inside 4K context. A long document may not, or the oldest parts of the conversation may need to be removed.
Technical explanation
A 4K context window usually means a limit near 4,096 tokens, although the exact usable amount and handling of input versus output vary by app and model.
Why it matters on your phone
Longer context consumes more memory through the KV cache, so AICanRun uses 4K as a consistent baseline for compatibility estimates.
A model with about seven billion adjustable values inside it. The number describes model size, not its download size.
Example
The same 7B model might be around 4–5 GB in a Q4 version and much larger in Q8.
Technical explanation
The B suffix means billion parameters. Parameter count is a rough measure of model scale, but architecture, training, and quantization also strongly affect capability and memory use.
Why it matters on your phone
More parameters usually require a larger model file and more memory, but the quantization determines how many bytes those parameters occupy.
The part of a model that is actually used for one piece of text. Some models keep most of their parameters idle at each step.
Example
A mixture-of-experts model may contain 30B total parameters but use only 3B of them to generate each token.
Technical explanation
Active parameters are the subset of weights evaluated for a particular token or forward pass, most commonly discussed with sparse mixture-of-experts architectures.
Why it matters on your phone
Active parameter count can affect generation speed, while the full model file still has to fit in storage and usually in addressable memory.
Apple's family of chips for modern Macs, combining the main processors and shared memory system in one package.
Example
An M1, M2, M3, or later M-series Mac is an Apple-silicon Mac; an older Intel Mac is not.
Technical explanation
Apple silicon refers to Apple's ARM-based systems on a chip for Macs. They integrate CPU and GPU cores, media and machine-learning accelerators, and a unified-memory architecture.
Why it matters on your phone
A Mac app or model runtime may require Apple silicon, and the exact chip, memory capacity, memory bandwidth, and cooling affect whether a model fits and how quickly it runs.
The phone's general-purpose processor. It handles everyday work and can also run parts—or all—of a local AI model.
Example
A local AI app may use the CPU for model operations that are not supported or efficient on the phone's GPU.
Technical explanation
The central processing unit executes general-purpose instructions. Mobile inference runtimes use optimized vector instructions and multiple CPU cores for tensor operations.
Why it matters on your phone
CPU design affects speed and power use, but large-model generation on phones is often limited by memory bandwidth rather than raw CPU peak performance.
How quickly the model writes its answer after it has started responding.
Example
At 8 tokens per second, a roughly 300-word answer may take about 50 seconds to finish.
Technical explanation
Generation or decode speed is the autoregressive output rate, commonly reported in tokens per second. Each new token is generated using the tokens that came before it.
Why it matters on your phone
It describes the waiting experience after the first token, but it does not include model loading or prompt-processing delay.
GGML Universal File — GGML is the library that runs the math, named after its author Georgi Gerganov (GG) + ML
A common model-file format used by apps that run AI locally.
Example
When PocketPal asks you to choose a model file, you may see names ending in .gguf, with separate files for Q4, Q8, or other versions.
Technical explanation
GGUF is a binary format that stores model tensors and metadata for inference runtimes in the llama.cpp ecosystem. GGML is the underlying machine-learning tensor library (its author's initials plus ML); GGUF is that library's model file format. It supports multiple architectures and quantization types.
Why it matters on your phone
The GGUF file and quantization you download determine storage use, model-memory use, and whether your app can load the model.
A processor built to do many similar calculations at the same time. AI apps can use it to speed up model work.
Example
An app may offload model layers to the phone's GPU instead of calculating every layer on the CPU.
Technical explanation
The graphics processing unit is a highly parallel processor. Inference runtimes can use compute APIs such as Metal or Vulkan to execute supported tensor operations on it.
Why it matters on your phone
GPU support can improve performance, but real speed also depends on the runtime, model format, memory bandwidth, and thermal limits.
The model's working memory for the conversation so it does not have to reread everything from scratch for every new token.
Example
A longer conversation produces a larger KV cache. Doubling the context setting can roughly double this part of memory use for the same model and configuration.
Technical explanation
The key-value cache stores attention keys and values from earlier tokens during autoregressive inference. Its size depends on context length, architecture, precision, and batch settings.
Why it matters on your phone
The KV cache uses RAM in addition to the model file, which is why download size alone cannot tell you whether a model fits.
An AI model trained to understand and generate text.
Example
Llama, Gemma, and Qwen are families of language models. Some versions are small enough to run directly on a phone.
Technical explanation
A large language model is a neural network trained on large text datasets to predict tokens, enabling tasks such as generation, summarization, extraction, and question answering.
Why it matters on your phone
Model family alone does not tell you whether it fits; you also need the parameter count, model file, quantization, and available memory.
Memory left over after the model and app have taken what they need.
Example
If a model needs 6.5 GB and the device has a 9 GB working budget, the estimated headroom is 2.5 GB.
Technical explanation
Memory headroom is the difference between estimated usable memory and the combined model weights, KV cache, and runtime overhead.
Why it matters on your phone
More headroom gives the operating system and other apps room to work. Very low headroom increases the chance of failed loading, memory pressure, or the inference app being closed.
How quickly a device can move data between memory and its processors.
Example
If two devices can both fit the same model, the one with higher usable memory bandwidth may generate tokens faster.
Technical explanation
Memory bandwidth is the rate at which data can be read from or written to memory, commonly expressed in gigabytes per second. Large-model decoding often repeatedly reads model weights and can be bandwidth-bound.
Why it matters on your phone
AICanRun uses memory bandwidth as one input to speed estimates, then labels those results as estimates unless a real measurement exists.
A way for an app to let the operating system load pieces of a model file as they are needed.
Example
Instead of copying an entire GGUF file into a separate buffer at once, a runtime can map the file and let the operating system manage its pages.
Technical explanation
Memory mapping associates a file with a process's virtual address space. File-backed pages are brought into physical memory on demand and managed by the operating system.
Why it matters on your phone
Mapping reduces unnecessary copying, but the model still needs substantial address space and physical memory while it is being used.
Apple's system for letting apps use the GPU efficiently.
Example
An inference app on an iPhone or Mac may enable a Metal backend so supported model operations run on the Apple GPU instead of only on the CPU.
Technical explanation
Metal is Apple's low-overhead graphics and compute API for iPhone, iPad, and Mac hardware. Inference runtimes can provide Metal kernels for supported tensor operations.
Why it matters on your phone
Metal support can improve local inference performance, but it does not remove the model's memory requirement or the operating system's practical memory limits.
A model made of several specialist sections that only turns on some of them for each piece of text.
Example
A model may contain 30B parameters in total but activate only 3B parameters for each token.
Technical explanation
A mixture-of-experts model uses a router to select a sparse subset of expert feed-forward networks for each token, reducing computation relative to activating every parameter.
Why it matters on your phone
MoE models can be faster than their total size suggests, but the complete set of experts still makes the model file and memory footprint large.
Apple's machine-learning framework for running and developing models efficiently on Apple-silicon Macs.
Example
A model repository may provide a separate MLX download for a Mac and a GGUF download for llama.cpp-based apps; they are not interchangeable files.
Technical explanation
MLX is an open-source array and machine-learning framework designed for Apple silicon. Its execution model uses the chip's unified memory and supports model implementations and tooling built for the MLX ecosystem.
Why it matters on your phone
A Mac may have enough memory for a model but still need an MLX-compatible model conversion, architecture implementation, and app version before that download can run.
Also: total needed · total memory needed · working memory requirement
The estimated memory a model and its running software need together before the operating system keeps its own reserve.
Example
If loaded weights use 4.2 GB, the KV cache uses 0.5 GB, and the runtime uses 0.8 GB, the model memory requirement is about 5.5 GB.
Technical explanation
AICanRun estimates model memory requirement as resident model weights plus the context-dependent KV cache plus runtime and compute-buffer overhead. It does not include the operating-system reserve already removed from the working budget.
Why it matters on your phone
The model-file download size alone understates memory use. AICanRun compares this total against the device's usable-memory working budget to decide whether the selected quant fits.
The billions of learned numbers inside a model that shape how it responds.
Example
Quantization stores the same model weights with fewer bits, making the model file smaller.
Technical explanation
Weights are learned numerical parameters in a neural network. During inference, matrix operations apply those values to the model's intermediate representations.
Why it matters on your phone
Weights usually account for most of a local model's file size and memory use.
A processor designed specifically for AI calculations.
Example
A phone may advertise a powerful NPU, but a GGUF app cannot benefit from it unless the app and model format support that hardware path.
Technical explanation
A neural processing unit is a specialized accelerator optimized for neural-network operations, often with vendor-specific supported data types, compilers, and execution frameworks.
Why it matters on your phone
NPU specifications alone do not predict local LLM speed. Runtime access and model compatibility matter just as much.
The text or instructions you give the model before it answers.
Example
‘Summarize these meeting notes in three bullets’ is a prompt. The pasted meeting notes are also part of the prompt.
Technical explanation
A prompt is the input token sequence supplied to a generative model, often including system instructions, conversation history, user content, and formatting templates.
Why it matters on your phone
Longer prompts take more time to process and use more of the available context window.
Labels for differently compressed versions of a model. Lower numbers are usually smaller; higher numbers usually preserve more detail.
Example
A Q4 file may fit on a phone when Q8 does not. Q8 may preserve more of the original model but takes much more space.
Technical explanation
The Q number broadly indicates the target bits per weight, while suffixes such as K_M describe a particular quantization method or quality-size tradeoff. Exact formats and effective bit rates vary.
Why it matters on your phone
Choosing the right quant is often the difference between a model fitting comfortably, barely fitting, or not loading at all.
A way to make a model smaller so it needs less memory, usually with some tradeoff in output quality.
Example
An 8 GB Q8 version might be reduced to roughly 5 GB in Q4, allowing Q4 to fit on a phone where Q8 cannot.
Technical explanation
Quantization represents model values with lower numerical precision, such as 8-bit, 4-bit, or lower formats, often using per-block scales and format-specific encodings.
Why it matters on your phone
It directly changes download size, memory use, generation speed, and potentially output quality.
A device's short-term working space. Apps and models use it while they are running.
Example
A device may advertise 12 GB of memory, but its operating system and other processes need part of it before a model can load.
Technical explanation
Random-access memory is volatile working memory used for active code and data. It is different from persistent storage used for downloaded files and photos.
Why it matters on your phone
A model can fit in storage yet still fail to run if its weights, KV cache, and app need more memory than the system makes available.
How long you wait before the model begins showing its answer.
Example
An app may wait five seconds to load and read your prompt, then generate the rest of the reply at 10 tokens per second.
Technical explanation
Time to first token measures latency from request start until the first output token, including relevant loading, prompt processing, scheduling, and initial decode work.
Why it matters on your phone
A high output speed can still feel slow when model loading or prompt processing creates a long initial pause.
Memory shared by a device's CPU, GPU, and other processors instead of giving each one a completely separate pool.
Example
On an Apple-silicon Mac, the CPU and GPU use the same physical memory pool, but macOS and other apps still need part of it.
Technical explanation
A unified memory architecture exposes shared physical memory to multiple processors, reducing explicit copies while retaining operating-system and per-process allocation limits.
Why it matters on your phone
Unified memory can reduce copying during local inference, but advertised capacity is still not the same as the conservative amount available to one model app.
Also: usable RAM · available memory · memory available for the model · working budget
The part of a device's memory that a model app can realistically use after the operating system keeps its share.
Example
A 12 GB Android phone might have a roughly 9 GB working budget, while a 36 GB Mac might use a conservative 32.4 GB budget instead of the full advertised capacity.
Technical explanation
Usable RAM is an estimate of memory available to the inference process after system-resident usage, operating-system behavior, and practical app limits are accounted for.
Why it matters on your phone
AICanRun compares the model memory requirement against this working budget rather than advertised capacity to avoid an unrealistically optimistic fit result.