singer-io / tap-bronto Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 2.0 58 KB

License: GNU Affero General Public License v3.0

Python 100.00%

tap-bronto's People

Contributors

Stargazers

Watchers

Forkers

notonthehighstreet isabella232

tap-bronto's Issues

Historical data won't replicate

I've used the settings on my Bronto integration in Stitch Data to set the historical sync start date as 1/1/17. Previously, I'd had it set to 09/20/17. Regardless of what value I have set for my historical sync start date, the data only goes as far back as 9/23/18. I believe this may be the date that my team first signed up with Stitch and began using the Bronto integration.

There are no extraction logs available in Stitch to show any attempted extractions or related errors.

How can I get the data to actually go back to the start date?

Issues replicating historical data

We're trying to replicate historical data from our Bronto instance dating back to 2014. We started using Stitch, but were experiencing severe data quality issues and outbound_activity and inbound_activity events that only dated back to 2018-12-15 when they should go back to 2014. When I looked at this repo and Bronto API it looks like inbound_activity data is limited to 30 days and outbound_activity is limited to 1 day. I see that the inbound_activity endpoint throws an error if the start date is greater than 30 days before execution date. However the outbount_activity endpoint does not throw an error if start date is greater than 1 day from execution date (it does for 30 days). I'm trying to understand the limitations of the Bronto api and this tap when it comes to replicating historical data. Am I limited to past 30 days for inbound_activity and last day for outbound_activity? Thank you for any help or guidance in advance.

Examples how to use?

Is it possible to post some examples how to use this Contacts with Filters?

can't replicate non XML compatible data

Summary

Encountered an error while attempting to replicate data from . The message was:

All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

Logs

2018-10-23 17:03:06,479Z   main - INFO Running tap-bronto version 1.0.2 and target-stitch version 1.7.4
2018-10-23 17:03:06,560Z   main - INFO Starting tap to discover schemas: tap-env/bin/tap-bronto --config /tmp/tap_discover_config.json --discover
2018-10-23 17:03:06,806Z    tap - INFO Starting discovery.
2018-10-23 17:03:06,842Z   main - INFO Tap exited normally.
2018-10-23 17:03:06,847Z   main - INFO Saving list of discovered streams
2018-10-23 17:03:06,921Z   main - INFO Saving structure of stream contact
2018-10-23 17:03:07,014Z   main - INFO Saving structure of stream inbound_activity
2018-10-23 17:03:07,102Z   main - INFO Saving structure of stream list
2018-10-23 17:03:07,122Z   main - INFO Saving structure of stream outbound_activity
2018-10-23 17:03:07,171Z   main - INFO Saving structure of stream unsubscribe
2018-10-23 17:03:07,318Z   main - INFO Getting list of selected properties
2018-10-23 17:03:07,368Z   main - DEBUG Getting initial state
2018-10-23 17:03:07,374Z   main - INFO Starting tap: tap-env/bin/tap-bronto --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json
2018-10-23 17:03:07,384Z   main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json
2018-10-23 17:03:07,753Z    tap - INFO Starting sync.
2018-10-23 17:03:07,756Z    tap - INFO Logging in
2018-10-23 17:03:07,811Z target - INFO Using Stitch import URL https://api.stitchdata.com/v2/import/batch
2018-10-23 17:03:08,120Z    tap - ERROR All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
2018-10-23 17:03:08,120Z    tap - ERROR Failed to sync endpoint, moving on!
2018-10-23 17:03:08,120Z    tap - CRITICAL All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
2018-10-23 17:03:08,123Z    tap - Traceback (most recent call last):
2018-10-23 17:03:08,123Z    tap -   File "tap-env/bin/tap-bronto", line 11, in <module>
2018-10-23 17:03:08,123Z    tap -     sys.exit(main())
2018-10-23 17:03:08,123Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_bronto/__init__.py", line 179, in main
2018-10-23 17:03:08,123Z    tap -     raise exception
2018-10-23 17:03:08,123Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_bronto/__init__.py", line 175, in main
2018-10-23 17:03:08,124Z    tap -     do_sync(args)
2018-10-23 17:03:08,124Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_bronto/__init__.py", line 124, in do_sync
2018-10-23 17:03:08,124Z    tap -     raise exception
2018-10-23 17:03:08,124Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_bronto/__init__.py", line 117, in do_sync
2018-10-23 17:03:08,124Z    tap -     stream_accessor.login()
2018-10-23 17:03:08,124Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_bronto/stream.py", line 66, in login
2018-10-23 17:03:08,125Z    tap -     self.config.get('token'))
2018-10-23 17:03:08,125Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/client.py", line 45, in __call__
2018-10-23 17:03:08,125Z    tap -     self._op_name, args, kwargs)
2018-10-23 17:03:08,125Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/wsdl/bindings/soap.py", line 110, in send
2018-10-23 17:03:08,125Z    tap -     options=options)
2018-10-23 17:03:08,125Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/wsdl/bindings/soap.py", line 68, in _create
2018-10-23 17:03:08,126Z    tap -     serialized = operation_obj.create(*args, **kwargs)
2018-10-23 17:03:08,126Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/wsdl/definitions.py", line 197, in create
2018-10-23 17:03:08,126Z    tap -     return self.input.serialize(*args, **kwargs)
2018-10-23 17:03:08,126Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/wsdl/messages/soap.py", line 64, in serialize
2018-10-23 17:03:08,126Z    tap -     self.body.render(body, body_value)
2018-10-23 17:03:08,126Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/xsd/elements/element.py", line 191, in render
2018-10-23 17:03:08,126Z    tap -     self._render_value_item(parent, value, render_path)
2018-10-23 17:03:08,127Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/xsd/elements/element.py", line 215, in _render_value_item
2018-10-23 17:03:08,127Z    tap -     return self.type.render(node, value, None, render_path)
2018-10-23 17:03:08,127Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/xsd/types/complex.py", line 253, in render
2018-10-23 17:03:08,127Z    tap -     element.render(parent, element_value, child_path)
2018-10-23 17:03:08,127Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/xsd/elements/indicators.py", line 241, in render
2018-10-23 17:03:08,127Z    tap -     element.render(parent, element_value, child_path)
2018-10-23 17:03:08,128Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/xsd/elements/element.py", line 191, in render
2018-10-23 17:03:08,128Z    tap -     self._render_value_item(parent, value, render_path)
2018-10-23 17:03:08,128Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/xsd/elements/element.py", line 215, in _render_value_item
2018-10-23 17:03:08,128Z    tap -     return self.type.render(node, value, None, render_path)
2018-10-23 17:03:08,129Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/zeep/xsd/types/simple.py", line 74, in render
2018-10-23 17:03:08,129Z    tap -     parent.text = self.xmlvalue(value)
2018-10-23 17:03:08,129Z    tap -   File "src/lxml/etree.pyx", line 1020, in lxml.etree._Element.text.__set__
2018-10-23 17:03:08,129Z    tap -   File "src/lxml/apihelpers.pxi", line 711, in lxml.etree._setNodeText
2018-10-23 17:03:08,129Z    tap -   File "src/lxml/apihelpers.pxi", line 699, in lxml.etree._createTextNode
2018-10-23 17:03:08,130Z    tap -   File "src/lxml/apihelpers.pxi", line 1439, in lxml.etree._utf8
2018-10-23 17:03:08,130Z    tap - ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
2018-10-23 17:03:08,173Z target - INFO Exiting normally
2018-10-23 17:03:08,208Z   main - INFO Target exited normally with status 0
2018-10-23 17:03:08,210Z   main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters". Target succeeded.

Data not loading incrementally

Issue

Historical data seems to be replicating over and over again, rapidly accumulating an enormous number of rows due to duplication.

Issue Description

We are using Stitch to load Bronto historical data into BigQuery on a daily basis. When investigating a single contact's outbound activities, we find that the exact same rows are being duplicated on every sync. All values except for id are the same.

Not sure where the problem is coming from 🤔

No module named 'tap_bronto.endpoints'

Step 4 in the quick start guide:

tap-bronto -c config.json --discover > catalog.json

Is returning a module not found error.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.