Python File I/O: Reading, Writing, Modes, and Encoding

Python file I/O comes down to three habits: open files with a with statement so they always close, pass encoding='utf-8' explicitly so your code behaves the same on every machine, and iterate line by line instead of loading whole files into memory. Get those three right and most file bugs disappear.

Why should you always use with open() in Python?

Because with guarantees the file is closed the moment the block exits — whether it exits normally, via return, or because an exception was raised. Without it, an exception between open() and close() leaks the file handle, which can lock files on Windows, lose buffered writes, and exhaust OS file descriptors in long-running processes.

# Fragile: file stays open if process() raises ❌
f = open('data.txt', 'r')
content = f.read()
process(content)
f.close()

# Guaranteed cleanup ✅
with open('data.txt', 'r', encoding='utf-8') as f:
    content = f.read()
# f is closed here, no matter what happened

with works because file objects are context managers — the same protocol you can build yourself; see context managers and the with statement.

What do the file modes r, w, a, x, and b mean?

The mode string tells open() what you're allowed to do and what happens to existing content. The dangerous one is 'w': it truncates the file to zero bytes the instant it opens, even if you never write. Use 'x' when overwriting would be a bug, and add 'b' for anything that isn't text.

Mode	Meaning	If file exists	If file missing
'r'	Read (default)	Reads from start	FileNotFoundError
'w'	Write	Erased immediately	Created
'a'	Append	Writes at end	Created
'x'	Exclusive create	FileExistsError	Created
'r+'	Read and write	Kept, cursor at start	FileNotFoundError
'b' suffix	Binary (e.g. 'rb', 'wb')	Bytes, no encoding/newline translation	—

# 'w' overwrites, 'a' appends
with open('output.txt', 'w', encoding='utf-8') as f:
    f.write('Hello, World!\n')

with open('log.txt', 'a', encoding='utf-8') as f:
    f.write('New log entry\n')

# Binary: images, PDFs, pickles
with open('image.png', 'rb') as f:
    binary_data = f.read()
with open('copy.png', 'wb') as f:
    f.write(binary_data)

What encoding does open() use by default?

Not necessarily UTF-8. Through Python 3.14, text-mode open() defaults to the platform's locale encoding — often cp1252 on Windows — so a file written on Linux can raise UnicodeDecodeError (or silently mangle characters) on Windows. PEP 686 flips the default to UTF-8 in Python 3.15. Until everyone is there, pass encoding='utf-8' explicitly, every time.

# Explicit encoding: same behavior on every OS ✅
with open('data.txt', 'r', encoding='utf-8') as f:
    content = f.read()

# Diagnose the classic failure
# UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d...
#   → the file is UTF-8 but Windows opened it as cp1252

# Last-resort handling of genuinely messy files
with open('messy.txt', 'r', encoding='utf-8', errors='replace') as f:
    content = f.read()   # bad bytes become U+FFFD instead of crashing

# Spot mixed-encoding trouble early:
# run python with  -X warn_default_encoding  to flag implicit defaults

How do you read a large file without running out of memory?

Iterate over the file object itself: for line in f: reads one buffered line at a time, so a 50 GB log file uses only a few KB of memory. Avoid f.read() and f.readlines() on big files — both load everything at once. For binary or line-less data, read fixed-size chunks in a loop.

# Line by line — constant memory
with open('large_file.txt', 'r', encoding='utf-8') as f:
    for line in f:
        process(line.strip())

# Fixed-size chunks for binary / no newlines
def read_in_chunks(file_path, chunk_size=1024 * 1024):
    with open(file_path, 'rb') as f:
        while True:
            chunk = f.read(chunk_size)
            if not chunk:
                break
            yield chunk

for chunk in read_in_chunks('huge_file.bin'):
    process(chunk)

Both patterns are generators under the hood — the same lazy-evaluation idea covered in generators and yield.

Should you use pathlib instead of open()?

For path manipulation, yes — pathlib.Path replaces error-prone string concatenation with / joins that work on every OS, and bundles existence checks, globbing, and quick read/write helpers. For small files, read_text()/write_text() are the shortest safe idiom. For large files, still use open() (or Path.open()) to stream.

from pathlib import Path

file = Path('data') / 'reports' / 'q3.txt'   # cross-platform join

# Quick one-shot read/write (opens and closes for you)
content = file.read_text(encoding='utf-8')
file.write_text('Hello, World!', encoding='utf-8')
binary = file.read_bytes()

# Everyday operations
file.exists()
file.suffix                 # '.txt'
file.parent.mkdir(parents=True, exist_ok=True)
for p in Path('data').glob('*.csv'):
    print(p.name)

Quick Patterns: CSV and JSON Files

Two file formats cover most data work. For CSV, always pass newline='' when writing (otherwise Windows inserts blank rows), and prefer DictReader/DictWriter for named columns. For JSON, json.load/json.dump work directly on file objects.

import csv, json

# CSV read/write
with open('data.csv', 'r', encoding='utf-8', newline='') as f:
    for row in csv.DictReader(f):
        print(row['name'], row['age'])

with open('output.csv', 'w', encoding='utf-8', newline='') as f:
    writer = csv.DictWriter(f, fieldnames=['name', 'age'])
    writer.writeheader()
    writer.writerow({'name': 'Alice', 'age': 30})

# JSON read/write
with open('config.json', 'r', encoding='utf-8') as f:
    config = json.load(f)

with open('config.json', 'w', encoding='utf-8') as f:
    json.dump(config, f, indent=2)

For data analysis, pandas is usually the better CSV tool — see how to read CSV files in Python.

Best Practices Checklist

• Always use the with statement for automatic closing
• Specify encoding='utf-8' explicitly in text mode
• Iterate line by line (or in chunks) for large files
• Use pathlib for paths; read_text() for small files
• Use mode 'x' when overwriting would be a bug
• Pass newline='' when writing CSV files