Load a domain snapshot into SQLite with Python

Python example: Data Engineering

Requires Python 3.9 or later. Uses the standard library only, processes local files, and makes no network requests. Save the script beside your downloaded inputs, or provide full paths.

Download data-engineering.py

Run the example

python3 data-engineering.py com.txt.gz --database domains.sqlite --snapshot-date 2026-09-01

Results are printed to the terminal. Redirect standard output to a file if you want to save the report. For the SQLite example, the database is saved at the path you specify.

Complete script

import argparse
import csv
import gzip
import sys


def domains(path):
    with gzip.open(path, "rt", encoding="utf-8") as source:
        for line in source:
            domain = line.strip().lower().rstrip(".")
            if domain:
                yield domain

import datetime
import sqlite3

parser = argparse.ArgumentParser(description="Import a dated domain snapshot into SQLite")
parser.add_argument("file")
parser.add_argument("--database", required=True)
parser.add_argument("--snapshot-date", required=True)
args = parser.parse_args()
try:
    date = datetime.date.fromisoformat(args.snapshot_date).isoformat()
except ValueError:
    parser.error("snapshot-date must be YYYY-MM-DD")
connection = sqlite3.connect(args.database)
try:
    with connection:
        connection.execute("CREATE TABLE IF NOT EXISTS snapshots "
                           "(domain TEXT, observed_date TEXT, "
                           "PRIMARY KEY (domain, observed_date))")
        batch = []
        for domain in domains(args.file):
            batch.append((domain, date))
            if len(batch) == 1000:
                connection.executemany("INSERT OR IGNORE INTO snapshots VALUES (?, ?)", batch)
                batch.clear()
        connection.executemany("INSERT OR IGNORE INTO snapshots VALUES (?, ?)", batch)
    print("Rows for date:", connection.execute(
        "SELECT COUNT(*) FROM snapshots WHERE observed_date = ?", (date,)).fetchone()[0])
finally:
    connection.close()

To automate input downloads, follow the API documentation for tokens, supported endpoints, and historical dates. Keep API tokens out of shared scripts.