Document Loading, Splitting & Embeddings

← Back to LLM Engineering & RAG course

Before a language model can answer questions about your documents, those documents must be ingested, split into appropriately sized pieces, and converted into vector representations that enable semantic search. This lesson covers LangChain's document loaders, text splitters, chunk strategy decisions, and embedding generation — the foundation of every RAG pipeline.

Previous: Lesson 5 — LangChain Memory & Conversation Management

The Ingestion Pipeline

Every RAG system has the same three-stage ingestion pipeline:

text

Documents → Load → Split → Embed → Store

Load: read raw content from files, URLs, databases
Split: cut documents into chunks that fit within the model's context window
Embed: convert each chunk into a dense vector using an embedding model

LangChain provides components for each stage that compose directly into LCEL chains.

Document Loaders

A Document is LangChain's standard unit: a page_content string plus a metadata dict.

PDF Loader

bash

pip install pypdf

python

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("technical_report.pdf")
docs   = loader.load()

print(f"Pages loaded: {len(docs)}")
print(f"First page content (preview): {docs[0].page_content[:300]}")
print(f"Metadata: {docs[0].metadata}")
# → {'source': 'technical_report.pdf', 'page': 0}

Text and Markdown Files

python

from langchain_community.document_loaders import TextLoader, UnstructuredMarkdownLoader

# Plain text
text_loader = TextLoader("README.txt", encoding="utf-8")
text_docs   = text_loader.load()

# Markdown (preserves heading structure)
md_loader = UnstructuredMarkdownLoader("documentation.md")
md_docs   = md_loader.load()

Web Pages

bash

pip install beautifulsoup4

python

from langchain_community.document_loaders import WebBaseLoader
import bs4

loader = WebBaseLoader(
    web_paths=["https://docs.python.org/3/library/functions.html"],
    bs_kwargs={"parse_only": bs4.SoupStrainer(class_=["body", "section"])}
)
docs = loader.load()
print(f"Characters loaded: {len(docs[0].page_content)}")

Directory Loader — Bulk Ingestion

python

from langchain_community.document_loaders import DirectoryLoader

# Load all markdown files in a directory tree
loader = DirectoryLoader(
    path="./docs/",
    glob="**/*.md",
    loader_cls=UnstructuredMarkdownLoader,
    show_progress=True,
    use_multithreading=True
)
all_docs = loader.load()
print(f"Documents loaded: {len(all_docs)}")

Text Splitters

LLMs have context window limits and embedding models have token limits. Splitting documents into smaller chunks ensures each piece fits and retrieves with precision.

RecursiveCharacterTextSplitter

The default choice. It tries to split on paragraph boundaries first, then sentences, then words, to preserve semantic coherence:

python

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1_000,       # target characters per chunk
    chunk_overlap=200,      # overlap to preserve cross-chunk context
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""]  # priority order
)

chunks = splitter.split_documents(docs)
print(f"Chunks created: {len(chunks)}")
print(f"Average chunk size: {sum(len(c.page_content) for c in chunks) // len(chunks)} chars")

The chunk_overlap is critical: without it, a sentence that straddles a chunk boundary will be lost from retrieval. 200 characters (~50 tokens) is a reasonable default.

Token-Based Splitting

Character-based sizing is imprecise because token count varies by content. For exact token budgets use TokenTextSplitter:

python

from langchain_text_splitters import TokenTextSplitter

token_splitter = TokenTextSplitter(
    encoding_name="cl100k_base",  # GPT-4 / GPT-4o tokeniser
    chunk_size=256,               # tokens per chunk
    chunk_overlap=32
)
token_chunks = token_splitter.split_documents(docs)

Markdown Header Splitter

For structured documents, split on headings to preserve section boundaries and inject heading metadata:

python

from langchain_text_splitters import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#",  "h1"),
    ("##", "h2"),
    ("###","h3"),
]

md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
md_chunks   = md_splitter.split_text(md_docs[0].page_content)

# Each chunk carries its heading hierarchy in metadata
print(md_chunks[0].metadata)
# → {'h1': 'Introduction', 'h2': 'Overview'}
print(md_chunks[0].page_content[:200])

Chunk Strategy

Choosing the right chunk parameters is one of the most impactful decisions in a RAG pipeline:

Parameter	Too Small	Too Large	Recommended
`chunk_size`	Low recall — answer may span multiple chunks	Low precision — noise dilutes answer	512–1024 chars
`chunk_overlap`	Context lost at boundaries	Duplicate retrieval, inflated cost	10–20% of chunk size
Splitter	N/A	N/A	`RecursiveCharacterTextSplitter` for prose; `MarkdownHeaderTextSplitter` for structured docs

Metadata enrichment: always add source, section, and page number metadata to each chunk. This enables source attribution in answers and filtered retrieval:

python

from langchain_core.documents import Document

def enrich_chunks(chunks: list[Document], source_name: str) -> list[Document]:
    for i, chunk in enumerate(chunks):
        chunk.metadata.update({
            "source":    source_name,
            "chunk_id":  i,
            "char_count": len(chunk.page_content)
        })
    return chunks

chunks = enrich_chunks(splitter.split_documents(docs), source_name="technical_report.pdf")

OpenAI Embeddings

Embeddings are dense vector representations of text. Semantically similar text produces geometrically similar vectors, enabling similarity search without keyword matching.

bash

pip install langchain-openai

python

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Embed a single string
vector = embeddings.embed_query("What is retrieval-augmented generation?")
print(f"Embedding dimensions: {len(vector)}")
# → 1536

# Embed a batch of documents
texts  = [chunk.page_content for chunk in chunks[:10]]
vectors = embeddings.embed_documents(texts)
print(f"Batch size: {len(vectors)}, dimensions: {len(vectors[0])}")

Choosing an Embedding Model

Model	Dimensions	Cost (per 1M tokens)	Notes
`text-embedding-3-small`	1536	$0.02	Best cost/performance ratio — recommended default
`text-embedding-3-large`	3072	$0.13	Higher accuracy on complex retrieval tasks
`text-embedding-ada-002`	1536	$0.10	Legacy — use `3-small` instead

For most RAG applications, text-embedding-3-small provides excellent retrieval quality at minimal cost.

Semantic Similarity

Understanding the embedding space helps you reason about retrieval quality:

python

import numpy as np

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# Similar concepts → high similarity
v1 = embeddings.embed_query("transformer attention mechanism")
v2 = embeddings.embed_query("how does self-attention work in LLMs")
v3 = embeddings.embed_query("Python list comprehensions")

print(f"Similar: {cosine_similarity(v1, v2):.3f}")    # → ~0.92
print(f"Different: {cosine_similarity(v1, v3):.3f}")  # → ~0.65

This is the exact computation that vector databases perform at scale when you run a similarity query.

Complete Ingestion Pipeline

A reusable function that loads, splits, enriches, and returns chunks ready for embedding:

python

from pathlib import Path
from langchain_community.document_loaders import PyPDFLoader, TextLoader, DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.documents import Document

def ingest_documents(
    source: str | Path,
    chunk_size: int = 800,
    chunk_overlap: int = 150,
) -> list[Document]:
    """
    Load documents from a file or directory, split into chunks, and enrich metadata.
    Supports: .pdf, .txt, .md and directories containing them.
    """
    source = Path(source)
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap
    )

    if source.is_dir():
        loader = DirectoryLoader(str(source), glob="**/*.{pdf,txt,md}", show_progress=True)
        raw_docs = loader.load()
    elif source.suffix == ".pdf":
        raw_docs = PyPDFLoader(str(source)).load()
    else:
        raw_docs = TextLoader(str(source), encoding="utf-8").load()

    chunks = splitter.split_documents(raw_docs)

    for i, chunk in enumerate(chunks):
        chunk.metadata["chunk_id"]   = i
        chunk.metadata["char_count"] = len(chunk.page_content)

    return chunks


# Usage
chunks = ingest_documents("./knowledge_base/", chunk_size=800, chunk_overlap=150)
print(f"Ready to embed: {len(chunks)} chunks")
print(f"Sample: {chunks[0].page_content[:200]}")
print(f"Metadata: {chunks[0].metadata}")