Which years of our data are complete enough to sell? Five FEC cycles rated for four uses

Step 3. Measure every cycle in one query

One query computes every share per cycle. A date counts as in the cycle if it parses as MMDDYYYY, the format the FEC documents, and falls between January 1 of the year before the cycle and December 31 of the cycle year. A date that is empty or does not parse is counted as unusable, and one that parses but falls outside the cycle is counted as outside, so the three date shares add to 100%. The recipient ID counts as present when either CAND_ID or OTHER_ID is filled, because a contribution to a committee has no candidate ID.

def measure(con):
    con.sql("CREATE TABLE tt (code VARCHAR, meaning VARCHAR)")
    con.executemany("INSERT INTO tt VALUES (?, ?)", list(TRANSACTION_TYPES.items()))
    nb = lambda c: f"NULLIF(TRIM({c}), '')"
    return con.sql(f"""
    WITH d AS (
      SELECT p.*, TRY_STRPTIME({nb('TRANSACTION_DT')}, '%m%d%Y')::DATE AS dt,
             TRY_CAST({nb('TRANSACTION_AMT')} AS DECIMAL(14, 2)) AS amt,
             COUNT(*) OVER (PARTITION BY cycle, {nb('SUB_ID')}) AS sub_id_n
      FROM pas2 p
    ), m AS (
      SELECT cycle, COUNT(*) AS n, COUNT(*) FILTER (WHERE dt >= MAKE_DATE(cycle - 1, 1, 1) AND dt <= MAKE_DATE(cycle, 12, 31)) AS in_cycle,
             COUNT(*) FILTER (WHERE dt IS NOT NULL AND NOT (dt >= MAKE_DATE(cycle - 1, 1, 1) AND dt <= MAKE_DATE(cycle, 12, 31))) AS outside,
             COUNT(*) FILTER (WHERE dt IS NULL) AS unusable,
             COUNT(DISTINCT DATE_TRUNC('month', dt)) FILTER (WHERE dt >= MAKE_DATE(cycle - 1, 1, 1) AND dt <= MAKE_DATE(cycle, 12, 31)) AS months_present,
             COUNT(*) FILTER (WHERE {nb('CAND_ID')} IS NOT NULL OR {nb('OTHER_ID')} IS NOT NULL) AS recipient,
             COUNT(*) FILTER (WHERE {nb('CAND_ID')} IS NOT NULL) AS cand,
             COUNT(*) FILTER (WHERE {nb('STATE')} IS NOT NULL) AS st,
             COUNT(*) FILTER (WHERE REGEXP_MATCHES(TRIM(ZIP_CODE), '^[0-9]{{5}}([0-9]{{4}})?$')) AS zip5,
             COUNT(*) FILTER (WHERE amt IS NOT NULL) AS amt_ok,
             COUNT(*) FILTER (WHERE amt < 0) AS amt_neg,
             COUNT(*) FILTER (WHERE {nb('SUB_ID')} IS NOT NULL AND sub_id_n = 1) AS sub_ok,
             COUNT(*) FILTER (WHERE {nb('TRAN_ID')} IS NOT NULL) AS tran,
             COUNT(*) FILTER (WHERE {nb('ENTITY_TP')} IS NOT NULL) AS ent,
             COUNT(*) FILTER (WHERE TRANSACTION_TP IN (SELECT code FROM tt)) AS mapped
      FROM d GROUP BY cycle
    )
    SELECT cycle, n, in_cycle, outside, unusable, months_present, mapped,
           100.0 * recipient / n, 100.0 * cand / n, 100.0 * in_cycle / n, 100.0 * outside / n, 100.0 * unusable / n,
           100.0 * st / n, 100.0 * zip5 / n, 100.0 * amt_ok / n, 100.0 * amt_neg / n, 100.0 * sub_ok / n,
           100.0 * tran / n, 100.0 * ent / n, 100.0 * mapped / n
    FROM m ORDER BY cycle""").fetchall()

The rating function applies the thresholds from step 2:

def rate(row):
    """Return {use: rating} for one cycle's measures."""
    out = {}
    for use, (_label, needs) in USES.items():
        rating = "Sell"
        for measure, (ok, floor) in needs.items():
            sign = -1 if measure in LESS_IS_BETTER else 1
            v = sign * row[measure]
            if v < sign * floor:
                rating = "Fix first"
                break
            if v < sign * ok:
                rating = "Sell with a caveat"
        out[use] = rating
    return out

Step 4. Run 1: the measurement

We ran the script on 2026-10-18 with DuckDB 1.5.6. The self-check passed: each cycle’s loaded row count equals the number of lines in its source file, the date shares add to 100%, every share is between 0 and 100, and every transaction type is in the mapping table. The full output of the run follows.

SQL and Python: run 1

python3 -I fec_cycle_readiness.py --data-dir <a new empty folder> --out guides/which-years-of-data-are-sellable/fec_cycle_readiness.csv

Output from our run 1

duckdb 1.5.6, python 3.14.8
source  https://www.fec.gov/files/bulk-downloads/data_dictionaries/pas2_header_file.csv  bytes=212  last-modified=Wed, 10 Jan 2018 18:30:12 GMT
source  https://www.fec.gov/files/bulk-downloads/2014/pas214.zip  bytes=11947422  last-modified=Sun, 30 Dec 2018 16:47:04 GMT
source  https://www.fec.gov/files/bulk-downloads/2016/pas216.zip  bytes=15450184  last-modified=Tue, 18 Oct 2022 15:03:02 GMT
source  https://www.fec.gov/files/bulk-downloads/2018/pas218.zip  bytes=12556308  last-modified=Sun, 25 Dec 2022 21:49:49 GMT
source  https://www.fec.gov/files/bulk-downloads/2020/pas220.zip  bytes=28978598  last-modified=Sun, 29 Dec 2024 22:08:51 GMT
source  https://www.fec.gov/files/bulk-downloads/2022/pas222.zip  bytes=25677940  last-modified=Sun, 11 Oct 2026 16:58:13 GMT
total downloaded: 94610664 bytes

cycle      rows  %recipient_id  %cand_id  %date_in_cycle  %date_outside_cycle  %date_unusable  months_present  %state  %zip5  %amount_parsed  %amount_negative  %sub_id_unique  %tran_id  %entity_tp  %tp_mapped
2014    350272         100.00     99.78           99.60                 0.04            0.36              24   98.73  98.73          100.00              2.08          100.00     99.97       97.20      100.00
2016    516385         100.00     99.69           99.60                 0.01            0.39              24   99.46  99.44          100.00              1.36          100.00    100.00       98.54      100.00
2018    349820         100.00     99.58           99.39                 0.03            0.58              24   99.51  99.47          100.00              2.33          100.00    100.00       98.38      100.00
2020    887829         100.00     99.79           99.36                 0.01            0.63              24   99.84  99.83          100.00              5.23          100.00     99.95       99.42      100.00
2022    749055         100.00     99.82           98.87                 0.42            0.71              24   99.37  99.37          100.00             35.23          100.00     99.52       99.16      100.00

cycle  link_to_recipient  monthly_trend  state_zip_map  dollar_totals
2014  Sell               Sell           Sell with a caveat  Sell         
2016  Sell               Sell           Sell           Sell         
2018  Sell               Sell           Sell           Sell         
2020  Sell               Sell           Sell           Sell         
2022  Sell               Sell with a caveat  Sell           Fix first    

SELF-CHECK PASSED: 5 cycles, row counts equal the source files, shares add up, every transaction type is mapped
wrote guides/which-years-of-data-are-sellable/fec_cycle_readiness.csv

The same numbers as tables, with the ratings:

Download the CSV (fec_cycle_readiness.csv)

What the numbers say

• Every cycle can be linked to its recipient. 100.00% of rows in all five cycles carry a candidate or committee ID, and every cycle has rows in all 24 months.
• 2014 gets a caveat for state and ZIP maps. 98.73% of its rows have a state, and our threshold for Sell is 99%. A buyer who accepts 98% would rate it Sell. That is a 0.27-point miss, and the threshold decides it.
• 2022 gets a caveat for the monthly trend. 98.87% of its dates fall inside the cycle, against 99.36% to 99.60% for the other four. That misses our 99% threshold by 0.13 points. The cycle also has the most dates outside it: 0.42%, against 0.01% to 0.04%.
• 2022 is Fix first for dollar totals. 35.23% of its amounts are negative. The other cycles run from 1.36% to 5.23%. We did not find out why. Before anyone sells a 2022 total, someone has to explain it, and the buyer will ask the same question.
• 2016, 2018 and 2020 are Sell for all four uses. Every record ID is unique in every cycle, and every amount parses, to two decimals.

So “is this dataset complete enough to sell?” has no single answer. Five cycles of one file came out with three different ratings across four uses.

Method

• Dataset: FEC “contributions from committees to candidates and other committees” (pas2), cycles 2014, 2016, 2018, 2020 and 2022, from https://www.fec.gov/files/bulk-downloads/<year>/pas2<yy>.zip. Column names from https://www.fec.gov/files/bulk-downloads/data_dictionaries/pas2_header_file.csv. It is US government data, public. The FEC restricts the sale or commercial use of contributor names and addresses, and their use to solicit contributions. We publish counts and shares only, and none of this data is offered for sale.
• Recorded by size and date: the byte size and Last-Modified of every file are in the run 1 output. The 2022 file was last modified on 2026-10-11, a week before our run, so rerun it and compare those two lines first if your numbers differ.
• One script, one query: every share is a count over all rows of the cycle divided by the cycle’s row count, shown to two decimals. There is no sampling and no random seed.
• The mapping and thresholds are written out in step 2. They are our judgment.
• Self-checks: row count equals source lines; the three date classes add up to the row count and to 100%; every share is between 0 and 100; every transaction type is mapped; the CSV is written only if all pass.

Limits

• This is not a client result and not a data set for sale. It measures a public government file. A company’s own data has different fields, different owners and different rules.
• Completeness is not accuracy. A filled-in ZIP can be the wrong ZIP. We counted filled values, and checked only the ZIP’s form (5 or 9 digits). We did not check any value against anything.
• The thresholds are ours, and two ratings turn on a small gap: 2014 state (98.73% against our 99% threshold, 0.27 points) and 2022 monthly trend (98.87% against 99%, 0.13 points). We did not tune them against any buyer’s requirements.
• Four uses are a sample. We did not rate uniqueness of people, freshness, lineage, licensing or price, which a buyer also checks.
• We did not explain the 2022 negative amounts. We know the share, not the cause.
• Amendments are not resolved. The FEC says a transaction ID stays the same across amendments (FEC: file description (read 2026-10-18)). We did not collapse amended records, so a buyer who does may see different shares.
• The source can change. The FEC republishes its bulk files, and our numbers are for the files as they stood on 2026-10-18.

Tested on

When this is not enough

• Your data is not this file. You need your own fields, your own uses and thresholds that match what your buyers have asked for.
• A rating is a prompt for a question. It says which year and use to look at. It cannot say what went wrong or whether you may sell the data at all.
• Rights and privacy come first. Whether you hold the right to sell, and whether personal data is in the set, decide the question before any completeness number does.