Giter Club home page Giter Club logo

fpds's Introduction

fpds

A no-frills parser for the Federal Procurement Data System (FPDS) found here.

Motivation

The only programmatic access to this data via an ATOM feed limits each request to 10 records, which forces users to deal with pagination. Additonally, data is exported as XML, which proves annoying for most developers. fpds will handle all pagination and data transformation to provide users with a nice JSON representation of the equivalent XML data.

Setup

To install this package for development, create a virtual environment and install dependencies.

$ python3.8 -m venv venv
$ source venv/bin/activate
$ pip install -e .

Usage

For a list of valid search criteria parameters, consult FPDS documentation found here. Parameters will follow the URL String format shown in the link above, with the following exceptions:

  • Colons (:) will be replaced by equal signs (=)
  • Certain parameters enclose their value in quotations. fpds will automatically determine if quotes are needed, so simply enclose your entire criteria string in quotes.

For example, AGENCY_CODE:”3600” should be used as "AGENCY_CODE=3600".

Via CLI:

$  fpds parse "LAST_MOD_DATE=[2022/01/01, 2022/05/01]" "AGENCY_CODE=7504"

By default, data will be dumped into an .fpds folder at the user's $HOME directory. If you wish to override this behavior, provide the -o option. The directory will be created if it doesn't exist:

$  fpds parse "LAST_MOD_DATE=[2022/01/01, 2022/05/01]" "AGENCY_CODE=7504" -o {some-directory}

Same request via python interpreter:

from fpds import fpdsRequest

request = fpdsRequest(
    LAST_MOD_DATE="[2022/01/01, 2022/05/01]",
    AGENCY_CODE="7504"
)
data = request()

# or conversely, you can call the explicit `process_records` method
data = request.process_records()


# if you wish to bypass `multiprocessing`
request = fpdsRequest(
    LAST_MOD_DATE="[2022/01/01, 2022/05/01]",
    AGENCY_CODE="7504"
)
data = request.run_asyncio_loop()
records = [xml.jsonified_entries() for xml in data]

For linting and formatting, we use flake8 and black.

$ make lint
$ make formatters

Lastly, you can clean the clutter and unwanted noise.

$ make clean

Testing

$ make test

What's New

As of 06/05/2024, v1.3.2 patches a bug that was caching attributes due to a misuse of a mutable default argument.

fpds now supports asynchronous requests! As of v1.3.0, users can instantiate the class as usual, but will now need to call the process_records method to get records as JSON. Note: due to some recursive function calls in the XML parsing, users might experience some high completion times for this function call. Recommendation is to limit the number of results.

Timing Benchmarks (in seconds):

v1.2.1 v.1.3.0
188.46 29.40
190.38 28.14
187.20 27.66

Using v.1.2.1, the average completion time is 188.68 seconds (~3min). Using v.1.3.0, the average completion time is 28.40 seconds.

This equates to a 84.89% decrease in completion time!

As of v1.3.0, fpds now supports the use of over 100 keyword tags when searching for contracts using the v1.5.3 ATOM feed.

fpds's People

Contributors

dherincx92 avatar vdavez avatar

Stargazers

Dan P avatar Deepak avatar Mike Endale avatar Jonathan "Yoni" Knoll avatar Mitchell Bregman avatar Leon Kozlowski avatar Captain_America avatar Mark Baird avatar  avatar Andy avatar

Watchers

Mitchell Bregman avatar  avatar Leon Kozlowski avatar  avatar

fpds's Issues

Pydantic model?

Hi @dherincx92—I have been wondering whether it might be worth creating Pydantic models for FPDS data. The use case I'm envisioning is to allow for easier export from FPDS into a database that matches the USASpending database.

See the USASpending model here: https://github.com/fedspendingtransparency/usaspending-api/blob/master/usaspending_api/transactions/models/source_procurement_transaction.py

I'm not sure if it's worth it or not, but thought I'd float it and see if you had any reactions. I'd be happy to take a swing at doing some of the modeling / creating a PR if you think it's useful. Thanks!

Strange behavior with IDVs

I can't tell if this is a local bug, but when I run the following, I get a strange thing going on...

from fpds import fpdsRequest
request = fpdsRequest(
    SIGNED_DATE="[2023/09/18, 2023/09/18]",
    CONTRACTING_OFFICE_ID="12505B"
)
records = request()
[rec for rec in records if rec['contract_type'] == 'IDV'][0]

I get the following result:

{'title': 'New IDC 12505B23D0010 awarded to BISONTE INDUSTRIAL USA LLC for the amount of $0', 'contract_type': 'IDV', 'link__rel': 'alternate', 'link__type': 'text/html', 'link__href': 'https://www.fpds.gov/ezsearch/search.do?s=FPDS&indexName=awardfull&templateName=1.5.3&q=12505B23D0010+12H2+', 'modified': '2023-09-18 14:47:30', 'content': '\n      ', 'content__type': 'application/xml', 'award': '\n        ', 'award__version': '1.5', 'awardID': '\n          ', 'awardID__awardContractID': '\n            ', 'awardID__awardContractID__agencyID': '12H2', 'awardID__awardContractID__agencyID__name': 'AGRICULTURAL RESEARCH SERVICE', 'awardID__awardContractID__PIID': '12505B23C0009', 'awardID__awardContractID__modNumber': '0', 'awardID__awardContractID__transactionNumber': '0', 'relevantContractDates': '\n          ', 'relevantContractDates__signedDate': '2023-09-18 00:00:00', 'relevantContractDates__effectiveDate': '2023-09-18 00:00:00', 'relevantContractDates__currentCompletionDate': '2024-09-24 00:00:00', 'relevantContractDates__ultimateCompletionDate': '2028-09-24 00:00:00', 'dollarValues': '\n          ', 'dollarValues__obligatedAmount': '0.00', 'dollarValues__baseAndExercisedOptionsValue': '2346.12', 'dollarValues__baseAndAllOptionsValue': '124290.00', 'totalDollarValues': '\n          ', 'totalDollarValues__totalObligatedAmount': '0.00', 'totalDollarValues__totalBaseAndExercisedOptionsValue': '2346.12', 'totalDollarValues__totalBaseAndAllOptionsValue': '124290.00', 'purchaserInformation': '\n          ', 'purchaserInformation__contractingOfficeAgencyID': '12H2', 'purchaserInformation__contractingOfficeAgencyID__name': 'AGRICULTURAL RESEARCH SERVICE', 'purchaserInformation__contractingOfficeAgencyID__departmentID': '1200', 'purchaserInformation__contractingOfficeAgencyID__departmentName': 'AGRICULTURE, DEPARTMENT OF', 'purchaserInformation__contractingOfficeID': '12505B', 'purchaserInformation__contractingOfficeID__name': 'USDA ARS MWA AAO ACQ/PER PROP', 'purchaserInformation__contractingOfficeID__country': 'USA', 'purchaserInformation__fundingRequestingAgencyID': '12H2', 'purchaserInformation__fundingRequestingAgencyID__name': 'AGRICULTURAL RESEARCH SERVICE', 'purchaserInformation__fundingRequestingAgencyID__departmentID': '1200', 'purchaserInformation__fundingRequestingAgencyID__departmentName': 'AGRICULTURE, DEPARTMENT OF', 'purchaserInformation__fundingRequestingOfficeID': '126125', 'purchaserInformation__fundingRequestingOfficeID__name': 'USDA ARS MWA NADC', 'purchaserInformation__foreignFunding': 'X', 'purchaserInformation__foreignFunding__description': 'NOT APPLICABLE', 'contractMarketingData': '\n          ', 'contractMarketingData__feePaidForUseOfService': '0.00', 'contractData': '\n          ', 'contractData__contractActionType': 'B', 'contractData__contractActionType__description': 'IDC', 'contractData__typeOfContractPricing': 'J', 'contractData__typeOfContractPricing__description': 'FIRM FIXED PRICE', 'contractData__nationalInterestActionCode': 'NONE', 'contractData__nationalInterestActionCode__description': 'NONE', 'contractData__costOrPricingData': 'N', 'contractData__costOrPricingData__description': 'No', 'contractData__solicitationID': '12505B23Q0274', 'contractData__costAccountingStandardsClause': 'X', 'contractData__costAccountingStandardsClause__description': 'NOT APPLICABLE EXEMPT FROM CAS', 'contractData__descriptionOfContractRequirement': 'GRASS-ALFALFA AND TIMOTHY HAY CUBES IDIQ', 'contractData__inherentlyGovernmentalFunction': 'OT        ', 'contractData__inherentlyGovernmentalFunction__description': 'OTHER FUNCTIONS', 'contractData__GFE-GFP': 'N', 'contractData__GFE-GFP__description': 'Transaction does not use GFE/GFP', 'contractData__undefinitizedAction': 'X', 'contractData__undefinitizedAction__description': 'NO', 'contractData__consolidatedContract': 'D', 'contractData__consolidatedContract__description': 'NOT CONSOLIDATED', 'contractData__performanceBasedServiceContract': 'X', 'contractData__performanceBasedServiceContract__description': 'NOT APPLICABLE', 'contractData__multiYearContract': 'N', 'contractData__multiYearContract__description': 'NO', 'contractData__contingencyHumanitarianPeacekeepingOperation': 'X', 'contractData__contingencyHumanitarianPeacekeepingOperation__description': 'NOT APPLICABLE', 'contractData__purchaseCardAsPaymentMethod': 'N', 'contractData__purchaseCardAsPaymentMethod__description': 'NO', 'contractData__numberOfActions': '1', 'legislativeMandates': '\n          ', 'legislativeMandates__ClingerCohenAct': 'N', 'legislativeMandates__ClingerCohenAct__description': 'NO', 'legislativeMandates__materialsSuppliesArticlesEquipment': 'X', 'legislativeMandates__materialsSuppliesArticlesEquipment__description': 'NOT APPLICABLE', 'legislativeMandates__laborStandards': 'X', 'legislativeMandates__laborStandards__description': 'NOT APPLICABLE', 'legislativeMandates__constructionWageRateRequirements': 'X', 'legislativeMandates__constructionWageRateRequirements__description': 'NOT APPLICABLE', 'legislativeMandates__listOfAdditionalReportingValues': '\n            ', 'legislativeMandates__listOfAdditionalReportingValues__additionalReportingValue': 'NONE', 'legislativeMandates__listOfAdditionalReportingValues__additionalReportingValue__description': 'NONE OF THE ABOVE', 'legislativeMandates__interagencyContractingAuthority': 'X', 'legislativeMandates__interagencyContractingAuthority__description': 'NOT APPLICABLE', 'productOrServiceInformation': '\n          ', 'productOrServiceInformation__productOrServiceCode': '8710', 'productOrServiceInformation__productOrServiceCode__description': 'FORAGE AND FEED', 'productOrServiceInformation__productOrServiceCode__productOrServiceType': 'PRODUCT', 'productOrServiceInformation__contractBundling': 'H', 'productOrServiceInformation__contractBundling__description': 'NOT BUNDLED', 'productOrServiceInformation__principalNAICSCode': '111940', 'productOrServiceInformation__principalNAICSCode__description': 'HAY FARMING', 'productOrServiceInformation__recoveredMaterialClauses': 'C', 'productOrServiceInformation__recoveredMaterialClauses__description': 'NO CLAUSES INCLUDED AND NO SUSTAINABILITY INCLUDED', 'productOrServiceInformation__manufacturingOrganizationType': 'A', 'productOrServiceInformation__manufacturingOrganizationType__description': 'U.S. OWNED BUSINESS', 'productOrServiceInformation__useOfEPADesignatedProducts': 'E', 'productOrServiceInformation__useOfEPADesignatedProducts__description': 'NOT REQUIRED', 'productOrServiceInformation__countryOfOrigin': 'USA', 'productOrServiceInformation__countryOfOrigin__name': 'UNITED STATES', 'productOrServiceInformation__placeOfManufacture': 'C', 'productOrServiceInformation__placeOfManufacture__description': 'NOT A MANUFACTURED END PRODUCT', 'vendor': '\n          ', 'vendor__vendorHeader': '\n            ', 'vendor__vendorHeader__vendorName': 'BISONTE INDUSTRIAL USA LLC', 'vendor__vendorSiteDetails': '\n            ', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators': '\n              ', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isAlaskanNativeOwnedCorporationOrFirm': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isAmericanIndianOwned': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isIndianTribe': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isNativeHawaiianOwnedOrganizationOrFirm': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isTriballyOwnedFirm': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isSmallBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isVeteranOwned': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isServiceRelatedDisabledVeteranOwnedBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isWomenOwned': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__minorityOwned': '\n                ', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__minorityOwned__isMinorityOwned': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__minorityOwned__isSubContinentAsianAmericanOwnedBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__minorityOwned__isAsianPacificAmericanOwnedBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__minorityOwned__isBlackAmericanOwnedBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__minorityOwned__isHispanicAmericanOwnedBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__minorityOwned__isNativeAmericanOwnedBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__minorityOwned__isOtherMinorityOwned': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isVerySmallBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isWomenOwnedSmallBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isEconomicallyDisadvantagedWomenOwnedSmallBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isJointVentureWomenOwnedSmallBusiness': 'false', 'vendor__vendorSiteDetails__vendorSocioEconomicIndicators__isJointVentureEconomicallyDisadvantagedWomenOwnedSmallBusiness': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes': '\n              ', 'vendor__vendorSiteDetails__vendorBusinessTypes__isCommunityDevelopedCorporationOwnedFirm': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__isLaborSurplusAreaFirm': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__federalGovernment': '\n                ', 'vendor__vendorSiteDetails__vendorBusinessTypes__federalGovernment__isFederalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__federalGovernment__isFederallyFundedResearchAndDevelopmentCorp': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__federalGovernment__isFederalGovernmentAgency': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__isStateGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment': '\n                ', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment__isLocalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment__isCityLocalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment__isCountyLocalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment__isInterMunicipalLocalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment__isLocalGovernmentOwned': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment__isMunicipalityLocalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment__isSchoolDistrictLocalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__localGovernment__isTownshipLocalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__isTribalGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__isForeignGovernment': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__businessOrOrganizationType': '\n                ', 'vendor__vendorSiteDetails__vendorBusinessTypes__businessOrOrganizationType__isCorporateEntityNotTaxExempt': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__businessOrOrganizationType__isCorporateEntityTaxExempt': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__businessOrOrganizationType__isPartnershipOrLimitedLiabilityPartnership': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__businessOrOrganizationType__isSolePropreitorship': 'true', 'vendor__vendorSiteDetails__vendorBusinessTypes__businessOrOrganizationType__isSmallAgriculturalCooperative': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__businessOrOrganizationType__isInternationalOrganization': 'false', 'vendor__vendorSiteDetails__vendorBusinessTypes__businessOrOrganizationType__isUSGovernmentEntity': 'false', 'vendor__vendorSiteDetails__vendorLineOfBusiness': '\n              ', 'vendor__vendorSiteDetails__vendorLineOfBusiness__isCommunityDevelopmentCorporation': 'false', 'vendor__vendorSiteDetails__vendorLineOfBusiness__isDomesticShelter': 'false', 'vendor__vendorSiteDetails__vendorLineOfBusiness__isEducationalInstitution': 'false', 'vendor__vendorSiteDetails__vendorLineOfBusiness__isFoundation': 'false', 'vendor__vendorSiteDetails__vendorLineOfBusiness__isHospital': 'false', 'vendor__vendorSiteDetails__vendorLineOfBusiness__isManufacturerOfGoods': 'false', 'vendor__vendorSiteDetails__vendorLineOfBusiness__isVeterinaryHospital': 'false', 'vendor__vendorSiteDetails__vendorLineOfBusiness__isHispanicServicingInstitution': 'false', 'vendor__vendorSiteDetails__vendorRelationshipWithFederalGovernment': '\n              ', 'vendor__vendorSiteDetails__vendorRelationshipWithFederalGovernment__receivesContracts': 'false', 'vendor__vendorSiteDetails__vendorRelationshipWithFederalGovernment__receivesGrants': 'false', 'vendor__vendorSiteDetails__vendorRelationshipWithFederalGovernment__receivesContractsAndGrants': 'true', 'vendor__vendorSiteDetails__typeOfGovernmentEntity': '\n              ', 'vendor__vendorSiteDetails__typeOfGovernmentEntity__isAirportAuthority': 'false', 'vendor__vendorSiteDetails__typeOfGovernmentEntity__isCouncilOfGovernments': 'false', 'vendor__vendorSiteDetails__typeOfGovernmentEntity__isHousingAuthoritiesPublicOrTribal': 'false', 'vendor__vendorSiteDetails__typeOfGovernmentEntity__isInterstateEntity': 'false', 'vendor__vendorSiteDetails__typeOfGovernmentEntity__isPlanningCommission': 'false', 'vendor__vendorSiteDetails__typeOfGovernmentEntity__isPortAuthority': 'false', 'vendor__vendorSiteDetails__typeOfGovernmentEntity__isTransitAuthority': 'false', 'vendor__vendorSiteDetails__vendorOrganizationFactors': '\n              ', 'vendor__vendorSiteDetails__vendorOrganizationFactors__isSubchapterSCorporation': 'false', 'vendor__vendorSiteDetails__vendorOrganizationFactors__isLimitedLiabilityCorporation': 'true', 'vendor__vendorSiteDetails__vendorOrganizationFactors__isForeignOwnedAndLocated': 'false', 'vendor__vendorSiteDetails__vendorOrganizationFactors__profitStructure': '\n                ', 'vendor__vendorSiteDetails__vendorOrganizationFactors__profitStructure__isForProfitOrganization': 'true', 'vendor__vendorSiteDetails__vendorOrganizationFactors__profitStructure__isNonprofitOrganization': 'false', 'vendor__vendorSiteDetails__vendorOrganizationFactors__profitStructure__isOtherNotForProfitOrganization': 'false', 'vendor__vendorSiteDetails__vendorOrganizationFactors__isShelteredWorkshop': 'false', 'vendor__vendorSiteDetails__vendorOrganizationFactors__stateOfIncorporation': 'MD', 'vendor__vendorSiteDetails__vendorOrganizationFactors__stateOfIncorporation__name': 'MARYLAND', 'vendor__vendorSiteDetails__vendorOrganizationFactors__countryOfIncorporation': 'USA', 'vendor__vendorSiteDetails__vendorOrganizationFactors__countryOfIncorporation__name': 'UNITED STATES', 'vendor__vendorSiteDetails__vendorOrganizationFactors__organizationalType': 'SOLE PROPRIETORSHIP', 'vendor__vendorSiteDetails__typeOfEducationalEntity': '\n              ', 'vendor__vendorSiteDetails__typeOfEducationalEntity__is1862LandGrantCollege': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__is1890LandGrantCollege': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__is1994LandGrantCollege': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isHistoricallyBlackCollegeOrUniversity': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isMinorityInstitution': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isPrivateUniversityOrCollege': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isSchoolOfForestry': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isStateControlledInstitutionofHigherLearning': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isTribalCollege': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isVeterinaryCollege': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isAlaskanNativeServicingInstitution': 'false', 'vendor__vendorSiteDetails__typeOfEducationalEntity__isNativeHawaiianServicingInstitution': 'false', 'vendor__vendorSiteDetails__vendorCertifications': '\n              ', 'vendor__vendorSiteDetails__vendorCertifications__isDOTCertifiedDisadvantagedBusinessEnterprise': 'false', 'vendor__vendorSiteDetails__vendorCertifications__isSelfCertifiedSmallDisadvantagedBusiness': 'true', 'vendor__vendorSiteDetails__vendorCertifications__isSBACertifiedSmallDisadvantagedBusiness': 'false', 'vendor__vendorSiteDetails__vendorCertifications__isSBACertified8AProgramParticipant': 'false', 'vendor__vendorSiteDetails__vendorCertifications__isSelfCertifiedHUBZoneJointVenture': 'false', 'vendor__vendorSiteDetails__vendorCertifications__isSBACertifiedHUBZone': 'false', 'vendor__vendorSiteDetails__vendorCertifications__isSBACertified8AJointVenture': 'false', 'vendor__vendorSiteDetails__vendorLocation': '\n              ', 'vendor__vendorSiteDetails__vendorLocation__streetAddress': '2452 DRUID HILL AVE', 'vendor__vendorSiteDetails__vendorLocation__city': 'BALTIMORE', 'vendor__vendorSiteDetails__vendorLocation__state': 'MD', 'vendor__vendorSiteDetails__vendorLocation__state__name': 'MARYLAND', 'vendor__vendorSiteDetails__vendorLocation__ZIPCode': '212171826', 'vendor__vendorSiteDetails__vendorLocation__ZIPCode__city': 'BALTIMORE', 'vendor__vendorSiteDetails__vendorLocation__countryCode': 'USA', 'vendor__vendorSiteDetails__vendorLocation__countryCode__name': 'UNITED STATES', 'vendor__vendorSiteDetails__vendorLocation__phoneNo': '4107799552', 'vendor__vendorSiteDetails__vendorLocation__congressionalDistrictCode': '07', 'vendor__vendorSiteDetails__vendorLocation__entityDataSource': 'E&Y', 'vendor__vendorSiteDetails__vendorAlternateSiteCode': '21217', 'vendor__vendorSiteDetails__entityIdentifiers': '\n              ', 'vendor__vendorSiteDetails__entityIdentifiers__vendorUEIInformation': '\n                ', 'vendor__vendorSiteDetails__entityIdentifiers__vendorUEIInformation__UEI': 'LHSCQQP3WSH7', 'vendor__vendorSiteDetails__entityIdentifiers__vendorUEIInformation__UEILegalBusinessName': 'BISONTE INDUSTRIAL USA LLC', 'vendor__vendorSiteDetails__entityIdentifiers__vendorUEIInformation__ultimateParentUEI': 'LHSCQQP3WSH7', 'vendor__vendorSiteDetails__entityIdentifiers__vendorUEIInformation__ultimateParentUEIName': 'BISONTE INDUSTRIAL USA LLC', 'vendor__vendorSiteDetails__entityIdentifiers__cageCode': '8FWT7', 'vendor__vendorSiteDetails__ccrRegistrationDetails': '\n              ', 'vendor__vendorSiteDetails__ccrRegistrationDetails__registrationDate': '2019-12-11 00:00:00', 'vendor__vendorSiteDetails__ccrRegistrationDetails__renewalDate': '2024-05-18 00:00:00', 'vendor__contractingOfficerBusinessSizeDetermination': 'S', 'vendor__contractingOfficerBusinessSizeDetermination__description': 'SMALL BUSINESS', 'placeOfPerformance': '\n          ', 'placeOfPerformance__principalPlaceOfPerformance': '\n            ', 'placeOfPerformance__principalPlaceOfPerformance__stateCode': 'IA', 'placeOfPerformance__principalPlaceOfPerformance__stateCode__name': 'IOWA', 'placeOfPerformance__principalPlaceOfPerformance__countryCode': 'USA', 'placeOfPerformance__principalPlaceOfPerformance__countryCode__name': 'UNITED STATES', 'placeOfPerformance__placeOfPerformanceZIPCode': '500109602', 'placeOfPerformance__placeOfPerformanceZIPCode__county': 'STORY', 'placeOfPerformance__placeOfPerformanceZIPCode__city': 'AMES', 'placeOfPerformance__placeOfPerformanceCongressionalDistrict': '04', 'competition': '\n          ', 'competition__extentCompeted': 'F', 'competition__extentCompeted__description': 'COMPETED UNDER SAP', 'competition__solicitationProcedures': 'SP1', 'competition__solicitationProcedures__description': 'SIMPLIFIED ACQUISITION', 'competition__typeOfSetAside': 'SBA', 'competition__typeOfSetAside__description': 'SMALL BUSINESS SET ASIDE - TOTAL', 'competition__typeOfSetAsideSource': 'F', 'competition__typeOfSetAsideSource__description': 'This Action', 'competition__evaluatedPreference': 'NONE', 'competition__evaluatedPreference__description': 'NO PREFERENCE USED', 'competition__reasonNotCompeted': 'SP2', 'competition__reasonNotCompeted__description': 'SAP NON-COMPETITION (FAR 13)', 'competition__numberOfOffersReceived': '1', 'competition__numberOfOffersSource': 'F', 'competition__numberOfOffersSource__description': 'This Action', 'competition__commercialItemAcquisitionProcedures': 'A', 'competition__commercialItemAcquisitionProcedures__description': 'COMMERCIAL PRODUCTS/SERVICES', 'competition__commercialItemTestProgram': 'N', 'competition__commercialItemTestProgram__description': 'NO', 'competition__fedBizOpps': 'Y', 'competition__fedBizOpps__description': 'YES', 'competition__localAreaSetAside': 'N', 'competition__localAreaSetAside__description': 'NO', 'competition__priceEvaluationPercentDifference': '0.00', 'preferencePrograms': '\n          ', 'preferencePrograms__subcontractPlan': 'B', 'preferencePrograms__subcontractPlan__description': 'PLAN NOT REQUIRED ', 'transactionInformation': '\n          ', 'transactionInformation__createdBy': '[email protected]', 'transactionInformation__createdDate': '2023-09-13 13:16:42', 'transactionInformation__lastModifiedBy': '[email protected]', 'transactionInformation__lastModifiedDate': '2023-09-18 14:47:30', 'transactionInformation__status': 'F', 'transactionInformation__status__description': 'FINAL', 'transactionInformation__approvedBy': '[email protected]', 'transactionInformation__approvedDate': '2023-09-18 14:47:30', 'transactionInformation__closedStatus': 'N', 'genericTags': '\n          ', 'genericTags__genericStrings': '\n            ', 'genericTags__genericStrings__genericString01': '2023-08-09 00:00:00', 'genericTags__genericStrings__genericString02': 'LHSCQQP3WSH7', 'genericTags__genericStrings__genericString06': 'LPTA', 'genericTags__genericBooleans': '\n            ', 'genericTags__genericBooleans__genericBoolean01': 'false', 'genericTags__genericBooleans__genericBoolean02': 'false', 'genericTags__genericBooleans__genericBoolean03': 'false', 'genericTags__genericBooleans__genericBoolean04': 'false', 'IDV': '\n        ', 'IDV__version': '1.5', 'contractID': '\n          ', 'contractID__IDVID': '\n            ', 'contractID__IDVID__agencyID': '12H2', 'contractID__IDVID__agencyID__name': 'AGRICULTURAL RESEARCH SERVICE', 'contractID__IDVID__PIID': '12505B23D0010', 'contractID__IDVID__modNumber': '0', 'relevantContractDates__lastDateToOrder': '2024-09-17 00:00:00', 'dollarValues__totalEstimatedOrderValue': '124290.00', 'contractMarketingData__whoCanUse': 'ONLY MY AGENCY', 'contractMarketingData__individualOrderLimit': '165', 'contractData__typeOfIDC': 'B', 'contractData__typeOfIDC__description': 'INDEFINITE DELIVERY / INDEFINITE QUANTITY', 'contractData__multipleOrSingleAwardIDC': 'S', 'contractData__multipleOrSingleAwardIDC__description': 'SINGLE AWARD'}

What's strange is that the dict seems to have the awardID (12505B23C0009) from a previous entry. But when I go to the equivalent entry on the FPDS Atom feed (https://www.fpds.gov/ezsearch/FEEDS/ATOM?s=FPDS&FEEDNAME=PUBLIC&VERSION=1.5.3&q=SIGNED_DATE%3A[2023%2F09%2F18%2C2023%2F09%2F18]+AGENCY_CODE%3A12H2&start=20), there's no AwardID...

It seems like the script is persisting the AwardID from the previous entry, but I have no idea what might be causing that...

Consider a generator?

This might be me thinking about this all wrong, so bear with me. But one thing I am currently doing is taking the FPDS requests, taking the JSON, and loading them into a DB. For large FPDS requests, this is quite memory intensive. Right now, I've written a script that intercepts at the multiprocess_jsonified_entries step. But that still means all of the FPDS requests happen in the asyncio loop before and the memory builds up while processing it.

All of which is a long way of saying: I wonder if there's a way to be even lazier around the FPDS request and/or expose the JSON files in more of a streaming way so that I don't overheat my computer in the meantime lol.

Or, I could be doing it wrong. 😂

awardContractID versus referencedIDVID

If I correctly understand how this works, right now the parser does not differentiate between the award and the referenced IDV, so that it presents only agencyID, PIID, modNumber without presenting them as awardContractID__agencyID awardContractID__PIID, and awardContractID__modNumber, etc...

It would be amazing if you could distinguish between them. I'll try and figure out what it would take to do this, but if you know off the top of your head and/or have a suggestion about where to look, I'd be happy to submit another PR.

Thanks!

<ns1:awardID>
    <ns1:awardContractID>
        <ns1:agencyID name="ENVIRONMENTAL PROTECTION AGENCY">6800</ns1:agencyID>
        <ns1:PIID>0002</ns1:PIID>
        <ns1:modNumber>P00018</ns1:modNumber>
        <ns1:transactionNumber>0</ns1:transactionNumber>
    </ns1:awardContractID>
    <ns1:referencedIDVID>
        <ns1:agencyID name="ENVIRONMENTAL PROTECTION AGENCY">6800</ns1:agencyID>
        <ns1:PIID>EPS31703</ns1:PIID>
        <ns1:modNumber>0</ns1:modNumber>
    </ns1:referencedIDVID>
</ns1:awardID>

Dependency updates

👋 I love this package! Thank you for creating it! Would you be open to me sending a pull request that updates some of the dependencies? I am having a dependency conflict with requests 2.28.1 and ^2.28.2. I tested locally with removing the pinned dependencies in requirements.txt and setup.cfg and everything seems to work. Thanks again!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.