Start with the simple explanation, see a concrete example, then go deeper when you want the technical definition. Every term also explains why it matters when you run a model on a phone.
35 terms, explained from simple to technical
0–9
1-bit quantization
Also: 1-bit model · one-bit model
A very aggressive way to shrink a model so that it uses much less storage and memory.
Example
A 27B model that would normally be far too large for a phone may become small enough to try when released in a purpose-built 1-bit format.
Technical explanation
A quantization scheme that represents each model weight with roughly one bit on average. Real file sizes are usually larger than the theoretical minimum because formats also store scales, metadata, and sometimes higher-precision values.
Why it matters on your phone
It can make unusually large models fit on phones, but compatibility, speed, and output quality depend heavily on the specific format and runtime.
The model can work with roughly 4,000 pieces of text from the current conversation at once.
Example
A short chat or article may fit inside 4K context. A long document may not, or the oldest parts of the conversation may need to be removed.
Technical explanation
A 4K context window usually means a limit near 4,096 tokens, although the exact usable amount and handling of input versus output vary by app and model.
Why it matters on your phone
Longer context consumes more memory through the KV cache, so AICanRun uses 4K as a consistent baseline for compatibility estimates.
A model with about seven billion adjustable values inside it. The number describes model size, not its download size.
Example
The same 7B model might be around 4–5 GB in a Q4 version and much larger in Q8.
Technical explanation
The B suffix means billion parameters. Parameter count is a rough measure of model scale, but architecture, training, and quantization also strongly affect capability and memory use.
Why it matters on your phone
More parameters usually require a larger model file and more memory, but the quantization determines how many bytes those parameters occupy.
The part of a model that is actually used for one piece of text. Some models keep most of their parameters idle at each step.
Example
A mixture-of-experts model may contain 30B total parameters but use only 3B of them to generate each token.
Technical explanation
Active parameters are the subset of weights evaluated for a particular token or forward pass, most commonly discussed with sparse mixture-of-experts architectures.
Why it matters on your phone
Active parameter count can affect generation speed, while the full model file still has to fit in storage and usually in addressable memory.
The phone's general-purpose processor. It handles everyday work and can also run parts—or all—of a local AI model.
Example
A local AI app may use the CPU for model operations that are not supported or efficient on the phone's GPU.
Technical explanation
The central processing unit executes general-purpose instructions. Mobile inference runtimes use optimized vector instructions and multiple CPU cores for tensor operations.
Why it matters on your phone
CPU design affects speed and power use, but large-model generation on phones is often limited by memory bandwidth rather than raw CPU peak performance.
How quickly the model writes its answer after it has started responding.
Example
At 8 tokens per second, a roughly 300-word answer may take about 50 seconds to finish.
Technical explanation
Generation or decode speed is the autoregressive output rate, commonly reported in tokens per second. Each new token is generated using the tokens that came before it.
Why it matters on your phone
It describes the waiting experience after the first token, but it does not include model loading or prompt-processing delay.
GGML Universal File — GGML is the library that runs the math, named after its author Georgi Gerganov (GG) + ML
A common model-file format used by apps that run AI locally.
Example
When PocketPal asks you to choose a model file, you may see names ending in .gguf, with separate files for Q4, Q8, or other versions.
Technical explanation
GGUF is a binary format that stores model tensors and metadata for inference runtimes in the llama.cpp ecosystem. GGML is the underlying machine-learning tensor library (its author's initials plus ML); GGUF is that library's model file format. It supports multiple architectures and quantization types.
Why it matters on your phone
The GGUF file and quantization you download determine storage use, model-memory use, and whether your app can load the model.
A processor built to do many similar calculations at the same time. AI apps can use it to speed up model work.
Example
An app may offload model layers to the phone's GPU instead of calculating every layer on the CPU.
Technical explanation
The graphics processing unit is a highly parallel processor. Inference runtimes can use compute APIs such as Metal or Vulkan to execute supported tensor operations on it.
Why it matters on your phone
GPU support can improve performance, but real speed also depends on the runtime, model format, memory bandwidth, and thermal limits.
The model's working memory for the conversation so it does not have to reread everything from scratch for every new token.
Example
A longer conversation produces a larger KV cache. Doubling the context setting can roughly double this part of memory use for the same model and configuration.
Technical explanation
The key-value cache stores attention keys and values from earlier tokens during autoregressive inference. Its size depends on context length, architecture, precision, and batch settings.
Why it matters on your phone
The KV cache uses RAM in addition to the model file, which is why download size alone cannot tell you whether a model fits.
An AI model trained to understand and generate text.
Example
Llama, Gemma, and Qwen are families of language models. Some versions are small enough to run directly on a phone.
Technical explanation
A large language model is a neural network trained on large text datasets to predict tokens, enabling tasks such as generation, summarization, extraction, and question answering.
Why it matters on your phone
Model family alone does not tell you whether it fits; you also need the parameter count, model file, quantization, and available memory.
How quickly the phone can move data between memory and its processors.
Example
If two phones can both fit the same model, the phone with higher usable memory bandwidth may generate tokens faster.
Technical explanation
Memory bandwidth is the rate at which data can be read from or written to memory, commonly expressed in gigabytes per second. Large-model decoding often repeatedly reads model weights and can be bandwidth-bound.
Why it matters on your phone
AICanRun uses memory bandwidth as one input to speed estimates, then labels those results as estimates unless a real measurement exists.
A way for an app to let the operating system load pieces of a model file as they are needed.
Example
Instead of copying an entire GGUF file into a separate buffer at once, a runtime can map the file and let the operating system manage its pages.
Technical explanation
Memory mapping associates a file with a process's virtual address space. File-backed pages are brought into physical memory on demand and managed by the operating system.
Why it matters on your phone
Mapping reduces unnecessary copying, but the model still needs substantial address space and physical memory while it is being used.
Apple's system for letting apps use the GPU efficiently.
Example
An iPhone inference app may enable a Metal backend so supported model operations run on the Apple GPU instead of only on the CPU.
Technical explanation
Metal is Apple's low-overhead graphics and compute API for iPhone, iPad, and Mac hardware. Inference runtimes can provide Metal kernels for supported tensor operations.
Why it matters on your phone
Metal support can improve local inference performance, but it does not remove the model's memory requirements or iOS app-memory limits.
A model made of several specialist sections that only turns on some of them for each piece of text.
Example
A model may contain 30B parameters in total but activate only 3B parameters for each token.
Technical explanation
A mixture-of-experts model uses a router to select a sparse subset of expert feed-forward networks for each token, reducing computation relative to activating every parameter.
Why it matters on your phone
MoE models can be faster than their total size suggests, but the complete set of experts still makes the model file and memory footprint large.
The billions of learned numbers inside a model that shape how it responds.
Example
Quantization stores the same model weights with fewer bits, making the model file smaller.
Technical explanation
Weights are learned numerical parameters in a neural network. During inference, matrix operations apply those values to the model's intermediate representations.
Why it matters on your phone
Weights usually account for most of a local model's file size and memory use.
A processor designed specifically for AI calculations.
Example
A phone may advertise a powerful NPU, but a GGUF app cannot benefit from it unless the app and model format support that hardware path.
Technical explanation
A neural processing unit is a specialized accelerator optimized for neural-network operations, often with vendor-specific supported data types, compilers, and execution frameworks.
Why it matters on your phone
NPU specifications alone do not predict local LLM speed. Runtime access and model compatibility matter just as much.
The text or instructions you give the model before it answers.
Example
‘Summarize these meeting notes in three bullets’ is a prompt. The pasted meeting notes are also part of the prompt.
Technical explanation
A prompt is the input token sequence supplied to a generative model, often including system instructions, conversation history, user content, and formatting templates.
Why it matters on your phone
Longer prompts take more time to process and use more of the available context window.
Labels for differently compressed versions of a model. Lower numbers are usually smaller; higher numbers usually preserve more detail.
Example
A Q4 file may fit on a phone when Q8 does not. Q8 may preserve more of the original model but takes much more space.
Technical explanation
The Q number broadly indicates the target bits per weight, while suffixes such as K_M describe a particular quantization method or quality-size tradeoff. Exact formats and effective bit rates vary.
Why it matters on your phone
Choosing the right quant is often the difference between a model fitting comfortably, barely fitting, or not loading at all.
A way to make a model smaller so it needs less memory, usually with some tradeoff in output quality.
Example
An 8 GB Q8 version might be reduced to roughly 5 GB in Q4, allowing Q4 to fit on a phone where Q8 cannot.
Technical explanation
Quantization represents model values with lower numerical precision, such as 8-bit, 4-bit, or lower formats, often using per-block scales and format-specific encodings.
Why it matters on your phone
It directly changes download size, memory use, generation speed, and potentially output quality.
The phone's short-term working space. Apps and models use it while they are running.
Example
A phone may have 12 GB of RAM, but Android or iOS and other processes need part of it before a model can load.
Technical explanation
Random-access memory is volatile working memory used for active code and data. It is different from persistent storage used for downloaded files and photos.
Why it matters on your phone
A model can be stored on a phone yet still fail to run if its weights, KV cache, and app need more RAM than the system makes available.
How long you wait before the model begins showing its answer.
Example
An app may wait five seconds to load and read your prompt, then generate the rest of the reply at 10 tokens per second.
Technical explanation
Time to first token measures latency from request start until the first output token, including relevant loading, prompt processing, scheduling, and initial decode work.
Why it matters on your phone
A high output speed can still feel slow when model loading or prompt processing creates a long initial pause.
Also: available memory · memory available for the model
The part of the phone's RAM that a model app can realistically use after the operating system takes its share.
Example
A 12 GB Android phone may have roughly 9 GB available for a large model in AICanRun's estimate, not the full advertised 12 GB.
Technical explanation
Usable RAM is an estimate of memory available to the inference process after system-resident usage, operating-system behavior, and practical app limits are accounted for.
Why it matters on your phone
AICanRun compares total model memory against usable RAM rather than advertised RAM to avoid an unrealistically optimistic fit result.