singer-io / tap-adwords Goto Github PK

We are experiencing an issue when we try to export any report.
The library tries to get a start date for stream and fail because of the date format.

Here is the traceback:

CRITICAL ('Unknown string format:', '4Y-06-02T00:00:00.000000Z')
Traceback (most recent call last):
  File "/Users/guest/anaconda3/envs/singer_adwords/bin/tap-adwords", line 10, in <module>
    sys.exit(main())
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 1061, in main
    raise exc
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 1058, in main
    main_impl()
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 1051, in main_impl
    do_sync_all_customers(customer_ids, args.properties)
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 1037, in do_sync_all_customers
    do_sync(properties, sdk_client)
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 860, in do_sync
    sync_stream(stream_name, stream_metadata, sdk_client)
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 851, in sync_stream
    sync_report(stream_name, stream_metadata, sdk_client)
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 230, in sync_report
    sync_report_for_day(stream_name, stream_schema, sdk_client, start_date, field_list)
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 362, in sync_report_for_day
    if start > get_start_for_stream(sdk_client.client_customer_id, stream_name):
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/tap_adwords/__init__.py", line 149, in get_start_for_stream
    bk_start_date = utils.strptime_with_tz(bk_value or CONFIG['start_date'])
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/singer/utils.py", line 21, in strptime_with_tz
    d_object = dateutil.parser.parse(dtime)
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1358, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/Users/guest/anaconda3/envs/singer_adwords/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 649, in parse
    raise ValueError("Unknown string format:", timestr)
ValueError: ('Unknown string format:', '4Y-06-02T00:00:00.000000Z')

The date displayed in the STATE row has the same problem:

{"type": "STATE", "value": {"bookmarks": {"CAMPAIGN_PERFORMANCE_REPORT_XXXXXXXXXX": {"last_attribution_window_date": "4Y-06-02T00:00:00.000000Z", "date": "4Y-06-02T00:00:00.000000Z"}}}}

According to this page of Google's documentation, "The Account Performance report includes all statistics aggregated by default at the account level, one row per account. If other segment fields are used, you may get more than one row per account. See segmentation for more information."

This report is currently unavailable through tap-adwords and is commented out: https://github.com/singer-io/tap-adwords/blob/master/tap_adwords/__init__.py#L61

Below Link Giving 404 error: https://github.com/singer-io/getting-started/blob/master/SPEC.md
schema discovery and property selection: https://github.com/singer-io/getting-started/blob/master/BEST_PRACTICES.md#schema-discovery-and-property-selection

Can anyone help with updated link.

Instead of only using OAuth, I want to enable service account authorization as well. I am using the service account authorization for big query and google analytics will make it easier for me to maintain my code

According to this page of Google's documentation, "The Display Topics Performance report includes all Display Network and YouTube Network statistics aggregated at the topic level, one row per topic. If other segment fields are used, you may get more than one row per topic. See segmentation for more information. This is a multiple attribution report."

This report is currently unavailable through tap-adwords and is commented out: https://github.com/singer-io/tap-adwords/blob/master/tap_adwords/__init__.py#L85

I tried to fetch data for my adwords account for CLICK_PERFORMANCE_REPORT but I get only the following info.

{ "type": "RECORD", "stream": "CLICK_PERFORMANCE_REPORT", "record": { "day": "2019-11-13T00:00:00.000000Z", "_sdc_customer_id": "4444444444", "_sdc_report_datetime": "2020-02-05T09:59:59.050964Z" }, "time_extracted": "2020-02-05 10:00:17.506047+00:00" }

I tried with the default one which is generated by discovery (after marking selected as true). I also tried making the fieldExclusions array empty.
Can someone share their properties file which will give all the data?

According to the Google's website(https://developers.google.com/adwords/api/docs/guides/start), adwords api as has been depreacted(The AdWords API will sunset on April 27, 2022. Migrate to the Google Ads API to take advantage of the latest Google Ads features.) and there is new google ads api in place of it. Till when can we expect this singer tap to implement this update?

I am receiving error while running tap adwords currently as a new user which asks me to update to google ads api

TLDR; The static schemas in this tap have aged, and seem to be missing a lot of (potentially important data). If you are missing data, this is how you can verify your change is working before submitting a PR.

The Issue

Singer-python logs fields not present in the schema as they are transformed in this sort of fashion:

2019-06-21 21:01:14,459Z    tap - WARNING Removed 10 paths during transforms:
2019-06-21 21:01:14,459Z    tap -       advertisingChannelSubType
2019-06-21 21:01:14,459Z    tap -       biddingStrategyConfiguration
2019-06-21 21:01:14,460Z    tap -       budget
2019-06-21 21:01:14,460Z    tap -       campaignGroupId
2019-06-21 21:01:14,460Z    tap -       finalUrlSuffix
2019-06-21 21:01:14,460Z    tap -       forwardCompatibilityMap
2019-06-21 21:01:14,460Z    tap -       selectiveOptimization
2019-06-21 21:01:14,460Z    tap -       trackingUrlTemplate
2019-06-21 21:01:14,461Z    tap - WARNING Removed paths list: ['advertisingChannelSubType', 'biddingStrategyConfiguration', 'budget', 'campaignGroupId', 'finalUrlSuffix', 'forwardCompatibilityMap', 'selectiveOptimization', 'trackingUrlTemplate']

This is done to ensure the quality of the data from the source by requiring an explicit data typing to be specified in the schema. Data typing changes are challenging for ETL, since the destination would need to be backfilled with historical data that is typed correctly.

The Ask

It's challenging to find representative examples for every field that can come through this tap, due to the nature of ads. So, if you see fields removed in this fashion and want to get the data, please submit a PR with the information below included, so that we can verify the correctness and successfully merge them.

The Process

A schema change is deceptively simple (given that it's just a JSON schema), but this can cause real problems down the line if not verified correctly. Including this information with a PR helps us verify that the data is correct using the tools Singer has to offer.

1. Make the change to the schema and its metadata (the AdWords documentation on the service being modified can be a good guide for a rough start). Here is an example change set.
2. Run the tap using target-stitch in verify mode, without a state file. (e.g., /path/to/env/tap-adwords --config config.json --catalog catalog.json | /path/to/env/target-stitch --config config.json --dry-run -o target-output.json).

• The --dry-run flag will add local schema validation while not posting to Stitch and -o will save the records locally to the specified file for manual inspection.
• Not using a state file will ensure that enough data has passed through to catch any issues with inconsistent data types.
• To select the stream you want, you can use this tool with the output of /path/to/env/tap-adwords --config config.json --discover

3. Open a PR with the Schema change on this repo with the following information:

• A log snippet (~20 lines) with a "Removed paths" log line with and without the field removed
• A log snippet with a "Batch is valid" message from the target (e.g., 2019-06-21 21:01:13,527Z target - INFO ad_groups (2345): Batch is valid
• (Optional) A redacted RECORD message from the output file containing non-null data for the intended field(s). For information security's sake, do not post clear data or PII on Github. Adwords will frequently return null in the event of a lack of data, so seeing actual data come through is a HUGE help.

This is a new idea, so the requirements might change as we learn what is needed to ensure correctness, but this should be a good start. Thank you for helping us improve SInger

The catalog produced by running --discover function is missing the selected property, called fromtap-adwords --config config.json --discover > catalog.json , see example below.

Stream is skipped when running tap if the selected property is not true

{
    "stream": "campaigns",
    "tap_stream_id": "campaigns",
    "schema": {
    ...
    },
    "metadata": [
      {
        "metadata": {
          "inclusion": "available",
        },
        "breadcrumb": []
      }
    ]
}

"The Shopping Performance report provides shopping campaign statistics aggregated at the product (OfferId field) level. Product dimension values from Merchant Center such as brand, category, custom attributes, product condition and product type will reflect the state of each dimension as of the date and time when AdWords recorded the corresponding event."

This report is currently unavailable through tap-adwords, and is commented out:
https://github.com/singer-io/tap-adwords/blob/master/tap_adwords/__init__.py#L102

Google's docs for the report can be found here:
https://developers.google.com/adwords/api/docs/appendix/reports/shopping-performance-report

"The Call Metrics Call Details report includes data for call tracking of call-only ads or call extensions."

This report is currently unavailable through tap-adwords, and is commented out:
https://github.com/singer-io/tap-adwords/blob/master/tap_adwords/__init__.py#L70

Google's docs for the report can be found here:
https://developers.google.com/adwords/api/docs/appendix/reports/call-metrics-call-details-report

Hi guys, I am sending data from Tap-AdWords but the tap is not properly working with the state.json file or the config.json file. The report starts fetching data one month in advance from the date I put in the config file and state.json doesn't impact it also.

Here is my state file

{"KEYWORDS_PERFORMANCE_REPORT_64XXXXXX555": "2020-03-20T00:00:00.000000Z"}
I get this error also

INFO not updating bookmark: 2020-03-15 00:00:00+00:00 <= 2020-03-20 00:00:00+00:00

I am working with the keyword_performance_report

message: "Version v11 is deprecated. Requests to this version will be blocked."

i am unable to fetch the leads using singer is there any CODE

"The Video Performance report includes statistics for your account's upgraded AdWords for Video campaigns.

This report summarizes statistics by videos in your campaign. If you want statistics summarized by where your ad was shown, use the Placement Performance Report or the Url Performance Report."

This report is currently unavailable through tap-adwords, and is commented out:
https://github.com/singer-io/tap-adwords/blob/master/tap_adwords/__init__.py#L107

Google's docs for the report can be found here:
https://developers.google.com/adwords/api/docs/appendix/reports/video-performance-report

Hi there - recently pulled data from the Keywords Performance Report using the Stitch integration and it looks like finalURL didn't populate as a column ... when I looked at the Google AdWords API documentation, they mention finalURLs - so I'm thinking this needs to be renamed in the integration in order to get that data pulled over?

https://developers.google.com/adwords/api/docs/appendix/reports/keywords-performance-report#finalurls

I believe this use of DATETIME_FMT and here is causing errors. Seems that some C code backing strftime doesn't behave well across all platforms. I was able to get this working locally using the DATETIME_FMT_MAC instead of DATETIME_FMT. Current singer-python has a DATETIME_FMT_SAFE but it looks like that's the same format, just a more generic name.

LocationCriterion data appears to be retrievable via this endpoint, which can then be used in conjunction with the GEO_PERFORMANCE_REPORT to map output from the geo report.

According to this page from Google's documentation, "The Display Keyword Performance report includes all Display Network and YouTube Network statistics aggregated at the keyword level, one row per keyword. If other segment fields are used, you may get more than one row per keyword. See segmentation for more information. This is a multiple attribution report.

By default, this report will only include rows where IsNegative = false. If you want to include negative keywords, add a predicate for IsNegative IN [true, false] or IsNegative = true."

This report is not currently supported by tap-adwords, and is commented out: https://github.com/singer-io/tap-adwords/blob/master/tap_adwords/__init__.py#L84

Hi all,

This is an issue which I've noticed across a few taps but first here so I'll raise the issue here and take it from there.

In my use case I am pulling static reports from various taps and using a target to store them in a database. Because the reports are static I know what the primary key (PK) should be. I had anticipated that I should be able to set the PK for the Singer target to use by either manually setting key_properties in the CatalogEntry's schema or probably more sensibly (as the schema should probably remain static/read-only) in the metadata of the stream in table-key-properties as documented here. This seems sensible to me as the metadata is the place for user defined behaviour if I understand the intention correctly.

However, the way this tap (and others) is written precludes this.

When a sync is run this tap seems to regenerate all of the schemas from the API. This means that any change such as adding key_properties to the schema will not be passed onto the target. This is, I suppose, reasonable in case the API has changed.

It is also not possible to set table-key-properties as the metadata is not passed to the target. However, this could be solved by adjusting this line to instead read the table-key-properties if they exist and use them. Something like:

primary_keys = metadata.get(stream_metadata, (), 'table-key-properties') or []

I've also noticed that the same issue applies to other taps including tap-bing-ads. Please do let me know what your thought's on this issue are.

"The Keywordless Query report includes keywordless ads (dynamic search ads) statistics based on search terms."

This report is currently unavailable through tap-adwords, and is commented out:
https://github.com/singer-io/tap-adwords/blob/master/tap_adwords/__init__.py#L90

Google's docs for the report can be found here:
https://developers.google.com/adwords/api/docs/appendix/reports/keywordless-query-report