Append CSV files without repeated headers or shifted columns

To append CSV files, stack their data rows into one table and write the header only once. For monthly exports with the same columns, the result should contain every input observation—not repeated headers or values under the wrong labels.

This workflow uses Python’s standard library to build a new combined file, leaving the originals untouched. It also adds the source filename to each row so you can trace observations before publishing.

Appending is not joining: if you need to add columns by matching an ID, use a key-based CSV join instead.

Start with compatible files

Here are two fictional input files.

january.csv

month,region_id,applications
2026-01,00123,12
2026-01,00456,0

february.csv

month,region_id,applications
2026-02,00123,15
2026-02,00456,

The intended output has four data rows and one header:

month,region_id,applications,source_file
2026-01,00123,12,january.csv
2026-01,00456,0,january.csv
2026-02,00123,15,february.csv
2026-02,00456,,february.csv

Before combining your own files, confirm:

• Each file has a header. This script treats the first record as column names; it is not suitable for headerless inputs without modification.
• Columns have the same names and order. The script rejects reordered columns rather than guessing how to align them. Use unique, nonblank column names.
• The meaning matches too. Identical labels do not prove that units, date conventions or missing-value definitions are compatible.
• The batches belong together. Separate monthly observations can be appended. Overlapping full-dataset snapshots usually need comparison or reconciliation first.
• The format is known. The example expects comma-delimited UTF-8 files with conventional double-quote quoting. Normalize other formats first; see CSV delimiter detection if the separator is uncertain.

Do not simply concatenate whole files. That retains later headers, and a missing final newline can run two records together. Nor should you blindly remove the first physical line from each later file: CSV permits quoted fields containing line breaks, so physical lines are not always complete records. These cases are described in RFC 4180’s CSV format guidance.

Append locally with Python

Put the two input files and this script in one working folder. Save the script as append_csv.py. For your own dataset, edit inputs, expected and output. List inputs in the order you want them appended, and exclude previous combined outputs.

Python’s CSV reader returns strings by default, so it does not turn 00123 into a number or reinterpret dates. Its reader and writer should use newline="" so the CSV module handles line endings. See the Python CSV documentation.

import csv
from pathlib import Path

inputs = [Path("january.csv"), Path("february.csv")]
expected = ["month", "region_id", "applications"]
output = Path("combined.csv")

if not inputs:
    raise ValueError("No input files selected")
if not expected or any(not name.strip() for name in expected):
    raise ValueError("Expected column names must be nonblank")
if len(set(expected)) != len(expected):
    raise ValueError("Expected column names must be unique")
if len({p.resolve() for p in inputs}) != len(inputs):
    raise ValueError("The input list contains the same file twice")
if output.resolve() in {p.resolve() for p in inputs}:
    raise ValueError("Output must not be an input file")
if "source_file" in expected:
    raise ValueError("source_file is reserved for provenance")

counts = []
out = output.open("x", encoding="utf-8", newline="")
try:
    with out:
        writer = csv.writer(out)
        writer.writerow(expected + ["source_file"])

        for path in inputs:
            count = 0
            with path.open(
                encoding="utf-8-sig", newline=""
            ) as src:
                reader = csv.reader(src, strict=True)
                header = next(reader, None)
                if header != expected:
                    raise ValueError(
                        f"{path}: expected {expected!r}, got {header!r}"
                    )

                for record_no, row in enumerate(reader, start=2):
                    if len(row) != len(expected):
                        raise ValueError(
                            f"{path}: record {record_no} has "
                            f"{len(row)} fields; expected {len(expected)}"
                        )
                    if row == expected:
                        raise ValueError(
                            f"{path}: repeated header at record {record_no}"
                        )
                    writer.writerow(row + [path.name])
                    count += 1
            counts.append((path.name, count))
except BaseException:
    output.unlink(missing_ok=True)
    raise

for name, count in counts:
    print(f"{name}: {count} data rows")
print(f"{output}: {sum(count for _, count in counts)} data rows")

Open a terminal in that working folder and run it with your Python 3 installation:

python append_csv.py

Use a supported Python 3 release. The script requires at least Python 3.8 because its cleanup uses Path.unlink(missing_ok=True). It processes one record at a time rather than loading the whole dataset into memory. Blank records fail the field-count check; they are not silently skipped.

If processing raises an exception, the script attempts to remove this run’s incomplete output. After a failed or interrupted run, check for a leftover file before trying again.

The input encoding utf-8-sig accepts UTF-8 and skips an initial UTF-8 byte-order mark when present, as documented in Python’s encoding reference. It does not detect arbitrary encodings. If decoding fails, check the source’s documented encoding rather than ignoring invalid bytes.

The output uses exclusive creation: Python’s open() documentation specifies that mode "x" fails when the file already exists. For another run, choose a new output name or deliberately remove the old combined file after reviewing it.

Check the result before publication

For the sample, the expected printed counts are two rows per input and four combined rows. For your dataset, re-import the output with a CSV-aware tool and check:

1. Row count: output data rows equal the sum of input data rows, excluding headers. Count parsed records, not text lines.
2. Structure: one header, expected columns plus source_file, and no shifted fields.
3. Values: spot-check identifiers such as 00123, dates, quoted text, zero and empty fields. The sample’s blank application count stays blank; it is not converted to zero. The writer may change quoting and record line endings, so compare field values rather than expecting a byte-for-byte copy.
4. Overlap: inspect the dataset’s record key. This script retains duplicate observations; appending is not deduplication.
5. Provenance and disclosure: keep a local record of source URLs, release dates, units and transformations. Review filenames too, since they become public values if you publish source_file.

Header and width checks catch structural mismatches, not incorrect totals or incompatible definitions.

Once the combined file is reviewed, upload only data cleared for public release. TablePage accepts CSV uploads and generates a public dataset page with a shareable link and filterable table. Check the published row count and representative values again—especially identifiers—before distributing the link. Never upload sensitive data.