When running the fastfec from Python, some errors piped to stderr such as
fprintf(stderr, "Unknown type (%c) in %s\n", type, ctx->formType); will exit 1 and cause the program to crash.
For errors in fec.c, it would be a better user experience if those errors bubbled up to python where they could be caught and give a more informative error message.

Per #35, fastfec now does not support system installed pcre, can we add option of recovering that? Thanks!

relates to Homebrew/homebrew-core#107476

Thanks for this excellent utility!!

If you pip install fastfec, then you get a python package, and that python code use s dynamically linked lib. You DON'T get a standalone binary. So you can only interact with fastfec with the python bindings. You can't call fastfec from the command line.

I'm looking for a scriptable way to install the CLI interface, and the version on homebrew is still stuck at 0.0.4, which doesn't work on v8.4 of .fec files. I can manually install the CLI executable from the releases, but this can't be automated.

I can help with this if you point me in the right direction.

As I'm sure you know, PCRE is no longer under development. Using PCRE2 instead would help ensure your users are less vulnerable to security problems. Moreover, older software tends to get trickier to build on newer systems as time goes by, so switching will probably make maintenance simpler in the long run.

This might be helpful: PCRE2Project/pcre2#51

👋 it looks like the 0.1.9 release build does not build against zig 0.11

==> zig build -Dvendored-pcre=false
/private/tmp/fastfec-20240112-70052-um5wxm/FastFEC-0.1.9/build.zig:18:6: error: no field or member function named 'setPreferredReleaseMode' in 'Build'

Hi! Thanks for this great utility.

I only care about the Schedule A itemizations. In some cases of multi gig .FEC files, the non-schedule A entries can take up more than half of the file, and so really slow down parsing.

Can we add some options to only parse particular itemizations?

In the meantime, I do this, do you see any problems with it? Like are schedule A itemizations always going to come before other schedules?

# filter_fec.sh

# We only want the individual contributions from an FEC file. We don't want
# the other itemizations, they can be gigabytes and slow parsings

# From the FEC file format documentation:

# The first record of every electronic file that is submitted to the FEC must be an
# HDR record that precedes the main body of the ASCII CSV (comma separated values) data.
# The second record will be a "cover" record for the particular filing, (for example,
# a F3 or and F3X record for a FEC-3 or FEC-3X electronic report). An unlimited number
# of Schedule records (examples: SA, SB, SC/ ...) can follow the first two records of
# an FEC electronic report file. (Electronic fi les are usually assigned the file
# suffix ".fec".)

# So as soon as we see a line starting with "SB", "SC", or "SD", we stop.
# From https://stackoverflow.com/a/8940829/5156887
awk '{if(/^SB|^SC|^SD/)exit;else print}'

and use it as curl https://docquery.fec.gov/dcdev/posted/13360.fec | filter_fec.sh | fastfec 13360

What if we made it so the .c stuff only ever compiles to a library, and the only way to access it was through python? The current CLI stuff in c is awkward and hard to test. If we implemented the CLI in python it would be way easier and way more testable.

The downside is that users would have to have python installed. Currently they can just download the binary and it works. But I'm not sure how common that is, I feel like mot users would have python??? IDK though.

It seems like calls of context->customWriteFunction are going amiss. I'm seeing SEGFAULTs coming without any evidence that the custom open function or write callback are ever called.

I've tried to demonstrate that the custom function passed in is called via print statements followed by stdout flushes, and bybreakpoints, but have seen no evidence that the FFI is behaving as one would hope. The problem persists with calls to parse_as_files, as well.

To recreate, one can run:

import smart_open
from fastfec import *


if __name__ == "__main__":
    headers = {'headers': {'User-Agent': 'Mozilla/5.0'}}
    with smart_open.open(f'http://docquery.fec.gov/dcdev/posted/1606847.fec', 'rb', transport_params=headers) as f:
        with FastFEC() as fastfec:
            fastfec.parse_as_files(f, "some_output_directory", include_filing_id='1606847')

On at least revision 460d0c4, built on MacOS and run on MacOS.

As I understand it, somewhere in writer.c's call of the custom function being handed in from the python client, there seems to be something going wrong.

The issue presented here is the smallest bit I could get to fail easily without getting rid of, for example, the use of smart_open (in the event that that's causing problems), but I'd ideally be able to use fastfec.parse_as_files_custom in a more general case, with other file-like objects. This is simply the smallest failing case I could demonstrate.

I'll keep looking at this as time and priority allow, but I figured a GH issue might be helpful in this instance.

FastFEC export seems to be missing a trailing comma in lines that have one or more empty items at the end of a row.

Using homebrew version of fastfec on a M1 MacBook Pro running macOS Montery 12.4.

For example, you can reproduce this by running fastfec 876050 fastfec_output/ and checking the header.csv (should be an additional trailing comma after report_number 002), SB28A.csv (42 fields in line items vs 43 in header) or SB23.csv (43 fields in line items vs 44 in header)

header.csv:

record_type,ef_type,fec_version,soft_name,soft_ver,report_id,report_number,comment
HDR,FEC,8.0,Microsoft Navision 3.60 - AVF Consulting,1.00,FEC-840327,002

Attempting to build from source with brew install --build-from-source fastfec no longer links with Homebrew PCRE, but the system-provided one instead.

This seems to be due to a change in Zig 0.9.0 (Homebrew/homebrew-core@72b36e9).

❯ brew install --quiet --build-from-source fastfec
==> zig build -Dvendored-pcre=false
🍺  /usr/local/Cellar/fastfec/0.0.4: 6 files, 982.8KB, built in 11 seconds
==> Running `brew cleanup fastfec`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
❯ brew linkage fastfec
System libraries:
  /usr/lib/libSystem.B.dylib
  /usr/lib/libcurl.4.dylib
  /usr/lib/libpcre.0.dylib

I've poked at this for a bit, but I don't use Zig so I'm unsure how to get this to ignore the system libpcre. Passing --search-prefix doesn't work. I'd appreciate it if you could take a look. Thanks!

While upgrading fastfec to 0.1.6, I found that the test behavior got changed a bit.

The following test no longer works:

$ /opt/homebrew/Cellar/fastfec/0.1.6/bin/fastfec --no-stdin 13425
About to parse filing ID 13425
Trying filename: (null)
Couldn't open file: (null)

This is the old working test:

$ /opt/homebrew/Cellar/fastfec/0.0.4/bin/fastfec --no-stdin 13425
About to parse filing ID 13425
Trying URL: https://docquery.fec.gov/dcdev/posted/13425.fec
Done; parsing successful!

Per #16 we have an issue compiling Curl with Zig for multiple platforms, which has made it difficult/impossible to release new versions of the CLI automatically. We can still update the Python library, which most downstream tooling has seemingly adopted, but for the health of the project, I am proposing removing Curl support from the CLI.

Instead of fastfec [filing_id] to download and parse a filing from FEC, you'd have to have Curl installed locally (or use wget or similar) and run curl [filing_url] | fastfec [filing_id], which admittedly is not as convenient.

I think it's ultimately for the best for the project to move forward though. When the Zig Homebrew issue is resolved we can think about adding internal curl back in, but this should let us rethink and significantly refine the release process.

I've created this issue to track it, and when it's resolved it should be quick and easy to get the other issues closed.

Not sure if this is a bug or not. If I run fastfec 878160 and I look at the resulting output/878160/SA11D.csv, then I see this:

form_type,filer_committee_id_number,transaction_id,back_reference_tran_id_number,back_reference_sched_name,entity_type,contributor_organization_name,contributor_last_name,contributor_first_name,contributor_middle_name,contributor_prefix,contributor_suffix,contributor_street_1,contributor_street_2,contributor_city,contributor_state,contributor_zip_code,election_code,election_other_description,contribution_date,contribution_amount,contribution_aggregate,contribution_purpose_descrip,contributor_employer,contributor_occupation,donor_committee_fec_id,donor_committee_name,donor_candidate_fec_id,donor_candidate_last_name,donor_candidate_first_name,donor_candidate_middle_name,donor_candidate_prefix,donor_candidate_suffix,donor_candidate_office,donor_candidate_state,donor_candidate_district,conduit_name,conduit_street1,conduit_street2,conduit_city,conduit_state,conduit_zip_code,memo_code,memo_text_description,reference_code
SA11D,C00477828,C7168136,,,CAN,,Clarke,Hansen,,,,2900 E Jefferson Ave,Apt C4,Detroit,MI,482074242,P2012,,2013-06-30,565.73,565.73,,,,,,H0MI13398,Clarke,Hansen,,,,H,MI,13,,,,,,,,"* In-Kind: In-kind, web hosting and phone services, to be reimbursed"

It looks to me that this is missing the required trailing comma that separates the memo_text_description and (the missing) reference_code value. If I try to load this with a pyarrow csv reader with the given 45 column names, it gets mad because it only sees 44 values in the row. You can replicate with pd.read_csv(path, engine="pyarrow"). Other CSV parsers such as vanilla pandas (pd.read_csv(path)) and vaex are more forgiving and just fill in NA for the missing reference_code values, so perhaps that is why this hasn't been caught before.

If I look at at the resulting output/878160/SB17.csv, it's a similar story: there is one less trailing comma than there should be to separate the missing last value.

However, if I look at output/878160/F3S.csv, then this looks correct. I'd guess this is because the last value in that row are non-missing:

form_type,filer_committee_id_number,date_general_election,date_day_after_general_election,a_total_contributions_no_loans,b_total_contribution_refunds,c_net_contributions,a_total_operating_expenditures,b_total_offsets_to_operating_expenditures,c_net_operating_expenditures,a_i_individuals_itemized,a_ii_individuals_unitemized,a_iii_individuals_total,b_political_party_committees,c_all_other_political_committees_pacs,d_the_candidate,e_total_contributions,transfers_from_other_auth_committees,a_loans_made_or_guarn_by_the_candidate,b_all_other_loans,c_total_loans,offsets_to_operating_expenditures,other_receipts,total_receipts,operating_expenditures,transfers_to_other_auth_committees,a_loan_repayment_by_candidate,b_loan_repayments_all_other_loans,c_total_loan_repayments,a_refund_individuals_other_than_pol_cmtes,b_refund_political_party_committees,c_refund_other_political_committees,d_total_contributions_refunds,other_disbursements,total_disbursements
F3S,C00477828,2012-11-06,2012-11-07,3120.73,0.00,3120.73,2153.17,3340.65,-1187.48,1500.00,55.00,1555.00,0.00,1000.00,565.73,3120.73,0.00,0.00,0.00,0.00,3340.65,0.00,6461.38,2153.17,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,2153.17