Pandas Chunked CSV

How it works

Chunking the parser does not automatically make every later operation streaming or memory efficient. A global sort, join or duplicate-identity check may still need retained state or another system. The fixture uses known valid numeric rows and three uneven chunks, making the correct total and count easy to inspect. Real ingestion also needs a strategy for invalid rows and identities spanning chunks.

Run the code

Select Try in playground to load the complete editable source. Nothing executes until you choose Run. The browser uses Pyodide 0.27.7, Pandas 2.2.3 and NumPy 2.0.2. Each ordinary execution starts a fresh isolated runtime. The tables are fictional and do not use personal files.

For local work, use a separate virtual environment and run python chunked-input.py. Download the complete source. The reviewed local environment uses Python 3.12.14, Pandas 3.0.6 and NumPy 2.5.4; the browser uses Pyodide 0.27.7 with Pandas 2.2.3 and NumPy 2.0.2. Compatible examples are checked in both. See the environment setup. No Apache or database server is required.

Example

import pandas as pd
import numpy as np

from io import StringIO
source = StringIO("kg\n12\n8\n6\n10\n4\n")
total, count = 0., 0
with pd.read_csv(source, chunksize=2, dtype={"kg":"float64"}) as reader:
    for chunk in reader:
        total += float(chunk["kg"].sum())
        count += int(chunk["kg"].count())
print(total, count)
print(total / count)

Expected result

40.0 5
8.0

The five observations sum to forty kilograms and average eight. Each valid observation contributes once even though the final chunk is shorter. That is different from assigning every chunk equal weight.