Reading and Writing Files in Python Without Losing Data

Reading a file in Python is three lines, right up to the moment the encoding is wrong, the file is larger than memory, or the program dies halfway through writing and leaves half a file behind. This post covers the patterns that handle those cases, in the order you tend to meet them.

Always open with a context manager

with open('notes.txt', encoding='utf-8') as f:
    text = f.read()

The with block closes the file when it ends, including when an exception is raised inside it. Without it, the file stays open until the garbage collector gets around to it, which on some platforms means a file you cannot delete or rename, and a write that has not been flushed to disk yet.

Paths are worth the same care. pathlib handles separators for you and reads well:

from pathlib import Path

data_dir = Path('data')
text = (data_dir / 'notes.txt').read_text(encoding='utf-8')

Building a path by joining strings with slashes works until the code runs on Windows, or until a directory name contains a character you did not expect.

Encoding is not optional

Text on disk is bytes; an encoding says which bytes mean which characters. Leave it unspecified and Python uses a platform default, which differs between machines. That is the source of the classic bug where a file with accented characters reads fine on your laptop and raises a UnicodeDecodeError on the server.

State encoding='utf-8' on every text open, for reading and for writing. Two extra notes. Files exported from some Windows tools begin with a byte order mark; reading them with utf-8-sig strips it, where plain utf-8 leaves an invisible character at the start of your first field. And if a file genuinely is in another encoding and you cannot change the source, decode it explicitly rather than guessing, then write your own output as UTF-8.

When the content is not text at all, open in binary mode with 'rb' or 'wb' and do not think about encoding at all.

Large files: stream instead of loading

f.read() pulls the whole file into memory. For a log file or an export that is larger than comfortable, iterate instead:

with open('access.log', encoding='utf-8') as f:
    for line in f:
        if 'ERROR' in line:
            handle(line)

Iterating a file yields one line at a time, so memory use stays flat whether the file is two megabytes or two gigabytes. For binary formats, read in fixed-size chunks with f.read(65536) in a loop.

The same logic applies to writing: build output incrementally rather than assembling a huge string and writing it once.

CSV: use the module

Splitting a line on commas works until a field contains a comma, a quote or a newline, all of which are legal inside a quoted CSV field. The csv module handles the quoting rules, and DictReader gives you named columns:

import csv

with open('orders.csv', newline='', encoding='utf-8') as f:
    for row in csv.DictReader(f):
        print(row['order_id'], row['total'])

The newline='' argument is not decoration. Without it, the module's own newline handling collides with the platform's and you get blank lines between records on Windows.

Two more things that bite. Every value arrives as a string, so numbers and dates need converting, and a column of identifiers with leading zeros must stay a string or you lose the zeros. And files from spreadsheets in some regions use a semicolon as the separator and a comma as the decimal mark; pass the delimiter explicitly rather than assuming.

JSON: shapes and dates

import json

with open('config.json', encoding='utf-8') as f:
    config = json.load(f)

with open('out.json', 'w', encoding='utf-8') as f:
    json.dump(result, f, ensure_ascii=False, indent=2)

ensure_ascii=False keeps accented characters readable instead of escaping them. indent makes the output diff-friendly, which matters for anything in version control.

JSON has no date type, so datetimes need converting on the way out and parsing on the way back in. ISO format strings are the least surprising choice. And when a file holds one JSON object per line, which is common for logs and exports, parse it line by line rather than trying to load the file as a single document.

Writing safely

Opening a file with 'w' truncates it immediately. If the program fails between that moment and the end of the write, the original is gone and the replacement is incomplete. For anything you care about, write to a temporary file in the same directory and replace the original at the end:

import os, tempfile
from pathlib import Path

def write_atomic(path: Path, text: str) -> None:
    fd, tmp = tempfile.mkstemp(dir=path.parent, suffix='.tmp')
    try:
        with os.fdopen(fd, 'w', encoding='utf-8') as f:
            f.write(text)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)      # atomic on the same filesystem
    except Exception:
        os.unlink(tmp)
        raise

Replacing a file on the same filesystem is atomic: readers see either the old file or the new one, never a half-written one. The temporary file has to be in the same directory for that guarantee to hold, which is why the helper takes the parent rather than using the system temp directory.

Use 'a' when you want to append rather than replace, and remember that appending from several processes at once needs more care than the mode alone provides.

The habits worth keeping

• Every open is inside a with block, with an explicit encoding.
• Paths are Path objects, not concatenated strings.
• Files that might be large are streamed, not read whole.
• CSV goes through the csv module, with newline='' and an explicit delimiter.
• Anything you would be sorry to lose is written to a temporary file and moved into place.

Five habits, and they remove most of the file-handling bugs that reach production.