Python File Handling: Read, Write & Manage Files

What is Python File Handling?

Python file handling refers to the set of operations for creating, reading, writing, and closing files using Python's built-in open() function and file object methods. The with statement (context manager) is the recommended approach — it automatically closes the file even if an exception occurs, preventing resource leaks and data corruption.

Introduction to File Handling

File handling is a critical skill for any developer. Whether you're saving user preferences, processing large datasets, or logging system events, you need to know how to interact with the file system.

In this guide, we'll cover:

• Opening and closing files correctly.
• Understanding different file modes (read, write, append).
• The Pythonic way to handle files using the with statement.
• Practical examples of reading and writing data.

1. Opening a File: The open() Function

Everything starts with the open() function. It creates a file object that acts as a bridge between your Python script and the file on your disk.

# Basic Syntax
file_object = open("example.txt", "r")

Essential Parameters

• file: The path to your file (relative or absolute).
• mode: How you want to open the file (default is "r" for read).
• encoding: Recommended to use "utf-8" for text files to avoid character errors.

2. Comprehensive File Modes

Choosing the right mode is crucial to avoid accidentally deleting data.

3. Reading and Writing Data

Reading Content

You can read a file all at once, line by line, or into a list.

with open("data.txt", "r") as f:
    content = f.read()       # Entire file
    lines = f.readlines()    # List of lines
    # or loop through
    for line in f:
        print(line.strip())

Writing Content

Use "w" to start fresh or "a" to add to the end.

with open("log.txt", "a") as f:
    f.write("New entry added!\n")

4. The Pythonic Way: Using with

Manually closing files with f.close() is risky. If an error occurs before that line, the file stays open, leading to memory leaks or corrupted data.

# The BEST way to open files
with open("notes.txt", "r") as f:
    data = f.read()
# File is automatically closed here!

5. Handling Binary Files

For images, videos, or compiled files, you must add a "b" to your mode (e.g., "rb" or "wb").

with open("image.png", "rb") as f:
    binary_data = f.read()

6. Using the os and pathlib Modules

Python's standard library provides two powerful modules for file system operations beyond basic read/write.

The os Module

import os

# Check if a file exists before opening
if os.path.exists("data.txt"):
    os.remove("data.txt")  # Delete a file

# Create a directory
os.makedirs("output/reports", exist_ok=True)

# List files in a directory
for filename in os.listdir("."):
    print(filename)

The pathlib Module (Recommended for Modern Python)

pathlib provides an object-oriented approach to file system paths that is cleaner and more readable than string manipulation.

from pathlib import Path

# Create a path object
config_path = Path("config") / "settings.json"

# Check existence, read, write
if config_path.exists():
    content = config_path.read_text(encoding="utf-8")
    
# Write text directly
output = Path("output.txt")
output.write_text("Hello, pathlib!\n", encoding="utf-8")

# Iterate over all CSV files in a directory
for csv_file in Path("data").glob("*.csv"):
    print(csv_file.name)

pathlib is the modern, preferred approach for new Python code. It handles path separator differences between Windows and Unix automatically.

7. Handling Exceptions in File Operations

File operations are a prime location for errors — the file might not exist, permissions might be wrong, or the disk might be full. Always pair file handling with exception handling:

from pathlib import Path

def safe_read(filepath: str) -> str:
    """Read a file, returning an empty string on failure."""
    try:
        return Path(filepath).read_text(encoding="utf-8")
    except FileNotFoundError:
        print(f"Error: {filepath} not found.")
        return ""
    except PermissionError:
        print(f"Error: No permission to read {filepath}.")
        return ""

For a deep dive on the exception handling patterns demonstrated here, see our Python exception handling guide.

8. Practical Example: CSV Log Writer

Here is a real-world example that combines everything — writing structured log data to a CSV file safely:

import csv
from datetime import datetime
from pathlib import Path

def log_event(event_type: str, message: str, logfile: str = "events.csv"):
    """Append a timestamped event to a CSV log file."""
    log_path = Path(logfile)
    file_exists = log_path.exists()
    
    with open(log_path, "a", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["timestamp", "event_type", "message"])
        
        if not file_exists:
            writer.writeheader()  # Write header only on first use
        
        writer.writerow({
            "timestamp": datetime.utcnow().isoformat(),
            "event_type": event_type,
            "message": message
        })

# Usage
log_event("LOGIN", "User alice logged in from 192.168.1.1")
log_event("ERROR", "Database connection timeout after 30s")

This pattern — using append mode with a header guard — is the foundation of simple file-based logging in Python.

Working with Large Files Efficiently

Reading an entire large file into memory with f.read() can exhaust RAM. For large files, always process line by line:

# Memory-efficient: processes one line at a time
with open("large_dataset.txt", "r", encoding="utf-8") as f:
    for line in f:
        process(line.strip())

For even larger workloads, the pandas library provides optimised file reading with chunking. See our pandas read CSV guide for processing large datasets. For working with tar archives and compressed files in Python, the Python tarfile module guide covers that use case.

External Resources

• Python official docs: Reading and Writing Files
• Python pathlib documentation
• Python csv module documentation

Conclusion

Mastering file handling allows your programs to persist data beyond a single execution. Remember to always use the with statement and be careful with the "w" mode to avoid losing data!

The progression to follow: start with open() and the with statement, graduate to pathlib for modern path handling, and always wrap file operations in try/except blocks for production code. Once comfortable, explore csv and json modules for structured data formats.

Common Mistakes with Python File Handling

1. Not using a with statement Calling f = open("file.txt") without a with block means you must remember to call f.close() manually. If an exception occurs before close(), the file handle leaks. Always use with open("file.txt") as f: — the with statement guarantees the file is closed when the block exits, even if an exception is raised. See the Python file I/O documentation.

2. Forgetting to specify the encoding open("file.txt", "r") uses the system's default encoding, which varies by platform (UTF-8 on macOS/Linux, cp1252 on Windows). Files written on one platform may fail to read on another. Always specify encoding="utf-8" explicitly: open("file.txt", "r", encoding="utf-8"). This is especially important for files containing non-ASCII characters.

3. Reading a large file all at once f.read() loads the entire file into memory as a single string. For files larger than available RAM, this causes MemoryError. Iterate over the file line by line with for line in f: (which uses a buffer internally) or read in chunks using f.read(chunk_size).

4. Writing text when binary mode is needed Opening a file in text mode ("w") applies newline translation on Windows (\n → \r\n). For binary files (images, PDFs, pickled data), always use binary mode: open("image.png", "wb"). Mixing text and binary mode corrupts non-text files silently.

5. Not handling file-not-found errors Calling open("config.json") without error handling raises FileNotFoundError if the file doesn't exist, crashing the program. Use try/except FileNotFoundError: or check with pathlib.Path("config.json").exists() before opening. The pathlib module provides a more ergonomic file-system API than os.path.

Frequently Asked Questions

What is the difference between read modes "r", "rb", "r+", and "a" in Python? "r" opens for reading text (default). "rb" opens for reading in binary mode. "r+" opens for reading and writing without truncating. "w" opens for writing and truncates the file (or creates it). "a" opens for appending — writes go to the end of the file and the original content is preserved. "b" can be appended to any mode for binary access. The Python open() documentation lists all valid mode combinations.

How do I read a JSON file in Python? Open the file in text mode and pass the file object to json.load(): with open("data.json", "r", encoding="utf-8") as f: data = json.load(f). This returns a Python dict or list. To write JSON, use json.dump(data, f, indent=2) inside a with open("data.json", "w") as f: block. The json module documentation covers all serialization options.

How do I work with file paths across Windows and macOS/Linux in Python? Use pathlib.Path instead of string concatenation for paths. Path("data") / "subdir" / "file.txt" produces the correct separator for the current platform automatically. Path.home() returns the user's home directory. Path.cwd() returns the current working directory. path.resolve() returns the absolute path. Avoid os.path.join() in new code — pathlib is more readable and fully cross-platform. See the pathlib documentation.