Read CSV Data Without Changing Its Meaning

For dependency-free, row-by-row processing, read a CSV in Python with the built-in csv module. For filtering, explicit type conversion, chunking and table-level validation, use pandas.read_csv(). In either case, initially preserve fields as text, parse quoted fields with a CSV parser and convert only columns whose meanings you know.

Choose your task and dependency constraint; the picker recommends a reader.

Assume observations.csv contains this sample dataset:

dataset_id,place,date,value
A001,North,2026-09-01,12.5
A002,"Central, East",2026-09-02,
A003,South,2026-09-03,9.0

The quoted place name is deliberate. A conforming CSV parser returns Central, East as one value rather than treating its comma as a column separator. The empty final field in the second record is also deliberate: its meaning should be determined from the data specification, not guessed during import.

Read Rows With Python’s Built-In csv Module

Python’s csv module requires no third-party package. DictReader is a convenient choice when the file has a header and the processing code refers to columns by name:

import csv

with open(
    "observations.csv",
    mode="r",
    encoding="utf-8",
    newline="",
) as file:
    reader = csv.DictReader(file)

    for row in reader:
        print(
            row["dataset_id"],
            row["place"],
            row["value"],
        )

Expected output:

A001 North 12.5
A002 Central, East
A003 South 9.0

DictReader uses the first row as field names and returns each subsequent record as a dictionary. That makes row["place"] clearer and less fragile than relying on a hard-coded position such as row[1].

Python’s CSV documentation recommends opening file objects with newline=''. It also states that the reader returns strings rather than automatically converting fields to numbers or dates (Python csv documentation). In this example, 12.5 and 2026-09-01 are therefore text until the program deliberately converts them.

That behavior is useful for identifiers. A value that looks numeric may actually be a code whose formatting matters. Importing it as text prevents a conversion rule from silently changing its representation.

Select the File Encoding Explicitly

Specify an encoding instead of relying on the computer’s locale-dependent default. Use the encoding documented by the data provider, commonly UTF-8:

with open(
    "observations.csv",
    mode="r",
    encoding="utf-8",
    newline="",
) as file:
    reader = csv.DictReader(file)

If a UTF-8 file has a byte-order mark, the first header may otherwise appear as \ufeffdataset_id. Reopen that file with encoding="utf-8-sig". Python’s utf_8_sig decoder skips an optional UTF-8 BOM at the beginning of the data (Python codecs documentation).

Do not switch encodings merely to suppress a decoding error. The correct encoding should come from the provider or a reliable file specification. Decoding with the wrong character set can produce readable-looking but incorrect text.

Validate Headers and Record Widths Before Using Rows

DictReader is concise, but publication workflows often need explicit structural checks before records are accepted. The following version rejects an empty file, blank column names, duplicate column names and records whose field count differs from the header:

import csv

path = "observations.csv"
records = []

with open(
    path,
    mode="r",
    encoding="utf-8-sig",
    newline="",
) as file:
    reader = csv.reader(file, strict=True)

    try:
        header = next(reader)
    except StopIteration:
        raise ValueError("CSV is empty")

    if any(name.strip() == "" for name in header):
        raise ValueError("CSV has a blank column name")

    if len(header) != len(set(header)):
        raise ValueError(
            "CSV has duplicate column names"
        )

    for row in reader:
        if len(row) != len(header):
            raise ValueError(
                f"Physical line {reader.line_num}: "
                f"expected {len(header)} fields, "
                f"found {len(row)}"
            )

        records.append(dict(zip(header, row)))

print(f"Read {len(records)} records")

The field-count check runs before zip constructs the dictionary. That ordering matters because pairing a short row with the header would otherwise omit data, while pairing a long row would leave trailing values unused.

The parser’s line_num counts physical lines, not logical records. One quoted record can span multiple physical lines (Python reader-object documentation). An error reported at a particular physical line therefore does not necessarily identify a record by its one-based data-row position.

Do not validate CSV structure with line.split(","). That approach mishandles quoted commas, escaped quotes and multiline fields. In the sample file, it would split "Central, East" even though the comma belongs inside one field.

Structural validation also does not establish that the data is semantically valid. A row can have the correct number of fields while containing an invalid date, a duplicate identifier or text where a numeric measurement is required. Those checks belong after parsing because they depend on each column’s documented meaning.

Read the Whole Table With pandas

Install pandas once if it is not already available:

python -m pip install pandas

Then read the CSV into a DataFrame while preserving the initial field text:

import pandas as pd

expected_columns = [
    "dataset_id",
    "place",
    "date",
    "value",
]

df = pd.read_csv(
    "observations.csv",
    encoding="utf-8-sig",
    dtype=str,
    keep_default_na=False,
    on_bad_lines="error",
)

if df.columns.tolist() != expected_columns:
    raise ValueError(
        f"Unexpected columns: {df.columns.tolist()}"
    )

print(df.head())
print(df.shape)

This initial import deliberately keeps every field as text. It protects identifiers such as 00123 from becoming 123. The keep_default_na=False argument prevents strings such as NA and NULL from being interpreted as missing automatically.

Pandas normally recognizes empty strings and several common markers as missing values. Its documentation explains how dtype, na_values and keep_default_na control that behavior (pandas read_csv documentation). Preserving the original text first avoids making one global missing-value decision for unrelated columns.

The exact column-order comparison is intentional. It catches missing, additional and reordered columns. Whether reordering should be rejected depends on the publication contract, but checking for it explicitly prevents a changed source layout from passing unnoticed.

Convert Only Columns With Known Types

After the raw values and columns have been checked, convert only fields whose definitions support conversion:

df["value"] = pd.to_numeric(
    df["value"].replace("", pd.NA),
    errors="raise",
)

df["date"] = pd.to_datetime(
    df["date"],
    format="%Y-%m-%d",
    errors="raise",
)

if df["dataset_id"].eq("").any():
    raise ValueError(
        "dataset_id contains blanks"
    )

if df["dataset_id"].duplicated().any():
    raise ValueError(
        "dataset_id contains duplicates"
    )

Here, a blank is converted to missing only because the sample’s value field has that documented meaning. The same blank in dataset_id is rejected instead. A blank in another dataset might have a different defined meaning, so that policy should not be copied without checking its specification.

Both conversion functions raise on invalid input when errors="raise" (pandas numeric conversion; pandas datetime conversion). The explicit date format also states what the date strings are expected to look like rather than asking the importer to infer a format.

Do not apply numeric conversion to every column that contains digits. Postal codes, account codes and other identifiers can consist only of digits while still being text. See the CSV leading-zero preservation checklist when fields must retain fixed widths.

A value of zero also should not be treated as equivalent to a blank. Zero is a numeric value; a blank may represent an unavailable, unreported or inapplicable measurement depending on the dataset. Preserve that distinction throughout cleaning and export.

Process Large CSV Files in Chunks

A DataFrame normally gives convenient table-level operations, but loading the entire file is not always appropriate. For a large CSV that does not fit comfortably in memory, pandas can return an iterator that yields DataFrames:

for chunk in pd.read_csv(
    "large.csv",
    dtype=str,
    keep_default_na=False,
    chunksize=100_000,
):
    print(len(chunk))

Each chunk can be validated, transformed or aggregated before the next one is read. Checks that depend on the entire dataset need state outside the loop. For example, detecting duplicate identifiers across chunks requires retaining previously encountered identifiers or using another process that can enforce uniqueness globally.

The read_csv API documents both chunksize and usecols. Selecting only required columns with usecols can reduce parsing time and memory use. Chunking changes how rows are delivered; it does not remove the need to specify encoding, missing-value behavior and column types.

Export a Checked Publication File

After cleaning, write a new file instead of overwriting the source. Keeping the original makes it possible to investigate a failed conversion or compare the published output with the delivered data.

df.to_csv(
    "observations_publish.csv",
    index=False,
    encoding="utf-8",
    lineterminator="\n",
    date_format="%Y-%m-%d",
)

These arguments omit the DataFrame index, use UTF-8, standardize line endings and format datetime values as dates (pandas DataFrame.to_csv documentation). Omitting the index prevents pandas from adding an unintended extra column to the public file.

Treat the exported file as a new input rather than assuming that a successful write proves correctness. Read observations_publish.csv back with the intended parser and repeat the checks against the actual artifact that will be uploaded.

Confirm that its header and column order match the publication schema and that its data-record count matches the checked DataFrame. Recheck identifier uniqueness and inspect identifiers that require leading zeros. Verify that blanks, zeros and missing values still have their documented meanings after serialization.

Compare minimum and maximum dates and numeric values with the checked data. Sample text fields that contain commas, quotation marks or line breaks so the read-back test exercises CSV quoting rather than only simple records. A round trip is useful because publication errors can arise from export options even when the in-memory DataFrame is correct.

If readers may download the public CSV and open it in spreadsheet software, also check text cells for formula injection. CSV quoting handles delimiters and embedded text, but it is not a security policy for spreadsheet interpretation.

Publish only fields intended for public access. Remove personal, confidential and security-sensitive columns before uploading the final table. That decision must be made from the dataset’s access rules; neither csv nor pandas can determine which correctly parsed fields are appropriate to disclose.