Foundation Models and LLMs

Definition

Foundation Models and LLMs

Foundation models are large pre-trained models that learn broad representations from large-scale data and are adapted to many downstream tasks through prompting, retrieval, tool use, fine-tuning or system integration. Large language models are foundation models specialised for text and code; production use requires comparing capability, latency, context length, modality support, cost, safety behaviour, data policy and operational reliability.

Why It Matters

A foundation model is trained broadly enough to be adapted to many tasks. Adaptation can occur through prompting, tool use, retrieval, fine-tuning or specialised wrappers.

A large language model processes and generates sequences of tokens. Practical understanding includes tokenisation, embeddings, attention, context windows, inference, sampling, structured output, reasoning effort, caching, tool calling and model limitations.

Multimodal models process or generate more than one modality, such as text, images, audio or video. The engineer must decide how modalities are represented, combined, validated and evaluated.

Model selection is a system decision based on capability, reliability, latency, cost, privacy, context size, modality and deployment constraints. The largest model is not automatically the best production choice.

Failure mode

The recurring risk is treating this concept as a label rather than an engineering decision with trade-offs, evidence needs and operational consequences.

Evaluation check

Ask what observable evidence would show that this concept has been applied well in a real system, repository or learning exercise.

Related Concepts

• Context engineering
• Cost and latency

Further study: Stanford CRFM, On the Opportunities and Risks of Foundation Models, and OpenAI, Models.