singer-io / tap-asana Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 21.0 208 KB

Singer.io tap for extracting Asana data

License: GNU Affero General Public License v3.0

Python 100.00%

tap-asana's People

Contributors

Stargazers

Watchers

Forkers

wynot datamill-co smoghul jeffhuth-bytecode adswerve smeltser j-oflaherty oltjonad isabella232 pathlight goes-funky lucyken psanker hotgluexyz z3z1ma railsware zellers

tap-asana's Issues

Custom Fields - Currently Selected

I really appreciate the addition of custom fields! We are having trouble understanding which value in a custom field is selected for a task. In the example below "Code as NLA" is selected from a dropdown in the asana UI but I'm not sure how to tell which was selected from here.

[ { "enabled": true, "enum_options": [ { "color": "red", "enabled": true, "gid": "1171586570372210", "name": "Digital Content edit", "resource_type": "enum_option" }, { "color": "orange", "enabled": true, "gid": "1171586570372211", "name": "Code as NLA", "resource_type": "enum_option" }]

How are tasks tied to projects?

I'm using a stitch connector that uses this code to dump data from asana into redshift. I end up with a tasks__projects table that just has a guid column and a resource_type column. With just those two columns i don't see how the table relates a task to a project?

I'm new to singer/stitch so apologize if this question is in the wrong place.

Project custom fields not coming in

Hi I've encountered an issue where updates to project custom fields are not being extracted to my dwh. Can anyone please help with this?

Adding Sub-tasks

It seems like only 1st level tasks are pulled into the task table. Most projects have more sub-tasks than first level tasks, so I'm hoping I'm just missing something. Are there any plans to add sub-tasks in the extraction either in it's own schema or as part of the tasks schema?

Task Updates Not Coming In

I'm often encountering issues where a user marks an Asana task as completed but the update never flows into my data warehouse. I checked the logs but there's no error. Any idea on why this is happening so often (happens to at least 20% of the tasks).

Getting a server time out or your pagination token has expired error.

This seems to only happen when I attempt to extract the stories table. Any suggestions on what to do?

Add "start_on" for Tasks

There is a field called start_on that exists when tasks are pulled via the API. Can we have that added to the list of fields that we pull from tasks?

Sample JSON response when calling curl -X GET https://app.asana.com/api/1.0/tasks/{task_gid}

..."start_on":"2020-05-12"...

Task custom fields

As a suggestion/request, it would be great to have the task's custom fields flow through.

Thanks,
Dave

Bearer token has expired

The tasks stream takes over an hour to replicate so the token expires. Is there any way to fix this. Here is the error:

CRITICAL No Authorization: The bearer token has expired. If you have a refresh token, please use it to request a new bearer token, otherwise allow the user to re-authenticate.
Traceback (most recent call last):
  File ".virtualenvs/tap-asana/bin/tap-asana", line 8, in <module>
    sys.exit(main())
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/singer/utils.py", line 229, in wrapped
    return fnc(*args, **kwargs)
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/tap_asana/__init__.py", line 185, in main
    sync()
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/tap_asana/__init__.py", line 137, in sync
    for rec in stream.sync():
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/tap_asana/streams/base.py", line 157, in sync
    for obj in self.get_objects():
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/tap_asana/streams/tasks.py", line 58, in get_objects
    for task in self.call_api("tasks", project=project["gid"], opt_fields=opt_fields, modified_since=modified_since):
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/asana/page_iterator.py", line 58, in items
    for page in self:
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/asana/page_iterator.py", line 40, in __next__
    result = self.get_initial()
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/asana/page_iterator.py", line 69, in get_initial
    return self.client.get(self.path, self.query, **self.options)
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/asana/client.py", line 173, in get
    return self.request('get', path, params=query, **options)
  File "[working_directory]/.virtualenvs/tap-asana/lib/python3.8/site-packages/asana/client.py", line 91, in request
    raise STATUS_MAP[response.status_code](response)
asana.error.NoAuthorizationError: No Authorization: The bearer token has expired. If you have a refresh token, please use it to request a new bearer token, otherwise allow the user to re-authenticate.

Teams data

It would be great to add the teams data to the tap:

https://asana.com/developers/api-reference/teams

Filtering deleted tasks

We are working with this tasks but are noticing deleted tasks are being loaded as well - any way to filter these out?

login fails to recognize Business account status

I've logged in to the integration with my Asana Admin account, which definitely shows we're billing for a Business license but the Extraction continues to fail with this error message:
Payment Required: Portfolios are only available for users in an Enterprise or Business plan.
Is anyone else encountering this?

Adding Custom Fields for Projects

Would it be possible to add access to custom fields at the project level as has recently been done for tasks?

Stories Sync - Never Finishing - Time Out

I am currently running a sync that is setup to ONLY sync the 'stories' object. There is a second sync that is setup that syncs all of the other objects. Not sure why it was setup like that originally, but just trying to diagnose the stories sync.

Currently, it is failing every day with a "The process was terminated. Most often this occurs because the job has exceeded its allotted runtime. The next job will run based on the integration's schedule." error.

Every hour it runs, it seems to pick up some records, but never finishes.

For example:

2023-12-18 06:00:41,285Z tap - INFO Received invalid or expired token error, refreshing access token
2023-12-18 07:00:41,030Z tap - INFO Backing off get_initial(...) for 0.1s (asana.error.NoAuthorizationError: No Authorization: The bearer token has expired. If you have a refresh token, please use it to request a new bearer token, otherwise allow the user to re-authorize.)
2023-12-18 07:00:41,037Z tap - INFO Received invalid or expired token error, refreshing access token
2023-12-18 07:23:02,365Z target - INFO Serializing batch with 6 messages for table stories
2023-12-18 07:23:02,366Z target - INFO Sending batch of 17230 bytes to
2023-12-18 07:23:02,368Z target - INFO replicated 6 records from "stories" endpoint
2023-12-18 08:00:42,096Z tap - INFO Backing off get_initial(...) for 0.0s (TokenExpiredError: (token_expired) )
2023-12-18 08:00:42,096Z tap - INFO Received invalid or expired token error, refreshing access token

Failing sub-task replication

A number of sub-tasks within Asana are failing to replicate resulting in incomplete data.

This does not seem to be tracked within the closed issues but I am surprised I am the first to recognise this.

Any ideas on what could be causing this or any historic records of this happening to other users?

Missing access token parameter

Randomly getting CRITICAL (missing_token) Missing access token parameter. on subsequent runs with the same config. Creating an issue in case anyone else notices the same. Please let me know if this isn't the right way to handle this issue.

(These runs are via stitchdata.com).

Full log of errored run:

2021-01-11 13:46:32,740Z   main - INFO Running tap-asana version 2.1.1 and target-stitch version 3.2.0
2021-01-11 13:46:32,881Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobStarted ), use_ssl( true )
2021-01-11 13:46:32,885Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobFinished ), use_ssl( true )
2021-01-11 13:46:32,885Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.streamRecordCount ), use_ssl( true )
2021-01-11 13:46:33,893Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobStarted [44] at offset None
2021-01-11 13:46:33,896Z   main - INFO Starting tap to discover schemas: tap-env/bin/tap-asana --config /tmp/tap_discover_config.json --discover
2021-01-11 13:48:34,226Z    tap - CRITICAL (missing_token) Missing access token parameter.
2021-01-11 13:48:34,226Z    tap - Traceback (most recent call last):
2021-01-11 13:48:34,226Z    tap -   File "tap-env/bin/tap-asana", line 33, in <module>
2021-01-11 13:48:34,227Z    tap -     sys.exit(load_entry_point('tap-asana==2.1.1', 'console_scripts', 'tap-asana')())
2021-01-11 13:48:34,227Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2021-01-11 13:48:34,227Z    tap -     return fnc(*args, **kwargs)
2021-01-11 13:48:34,227Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_asana/__init__.py", line 172, in main
2021-01-11 13:48:34,227Z    tap -     Context.asana = Asana(**creds)
2021-01-11 13:48:34,227Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_asana/asana.py", line 18, in __init__
2021-01-11 13:48:34,227Z    tap -     self.refresh_access_token()
2021-01-11 13:48:34,227Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_asana/asana.py", line 35, in refresh_access_token
2021-01-11 13:48:34,227Z    tap -     return self._client.session.refresh_token(self._client.session.token_url, client_id=self.client_id, client_secret=self.client_secret, refresh_token=self.refresh_token)
2021-01-11 13:48:34,227Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/requests_oauthlib/oauth2_session.py", line 446, in refresh_token
2021-01-11 13:48:34,227Z    tap -     self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
2021-01-11 13:48:34,227Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 421, in parse_request_body_response
2021-01-11 13:48:34,227Z    tap -     self.token = parse_token_response(body, scope=scope)
2021-01-11 13:48:34,227Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 431, in parse_token_response
2021-01-11 13:48:34,227Z    tap -     validate_token_parameters(params)
2021-01-11 13:48:34,227Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 441, in validate_token_parameters
2021-01-11 13:48:34,227Z    tap -     raise MissingTokenError(description="Missing access token parameter.")
2021-01-11 13:48:34,227Z    tap - oauthlib.oauth2.rfc6749.errors.MissingTokenError: (missing_token) Missing access token parameter.
2021-01-11 13:48:34,266Z   main - INFO Tap exited abnormally with status 1
2021-01-11 13:48:34,907Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [40] at offset None
2021-01-11 13:48:34,908Z   main - INFO No tunnel subprocess to tear down
2021-01-11 13:48:34,908Z   main - INFO Exit status is: Discovery failed with code 1 and error message: "(missing_token) Missing access token parameter.".

Full log of subsequent successful run:

2021-01-11 14:46:32,545Z   main - INFO Running tap-asana version 2.1.1 and target-stitch version 3.2.0
2021-01-11 14:46:32,681Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobStarted ), use_ssl( true )
2021-01-11 14:46:32,685Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobFinished ), use_ssl( true )
2021-01-11 14:46:32,685Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.streamRecordCount ), use_ssl( true )
2021-01-11 14:46:33,694Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobStarted [30] at offset None
2021-01-11 14:46:33,696Z   main - INFO Starting tap to discover schemas: tap-env/bin/tap-asana --config /tmp/tap_discover_config.json --discover
2021-01-11 14:46:34,714Z    tap - INFO Starting discover
2021-01-11 14:46:34,716Z    tap - INFO Finished discover
2021-01-11 14:46:34,758Z   main - INFO Tap exited normally.
2021-01-11 14:46:34,760Z   main - INFO Saving list of discovered streams
2021-01-11 14:46:34,779Z   main - INFO Saving structure of stream projects (tap_stream_id: projects)
2021-01-11 14:46:34,823Z   main - INFO Saving structure of stream portfolios (tap_stream_id: portfolios)
2021-01-11 14:46:34,842Z   main - INFO Saving structure of stream tasks (tap_stream_id: tasks)
2021-01-11 14:46:34,900Z   main - INFO Saving structure of stream stories (tap_stream_id: stories)
2021-01-11 14:46:34,934Z   main - INFO Saving structure of stream tags (tap_stream_id: tags)
2021-01-11 14:46:34,949Z   main - INFO Saving structure of stream users (tap_stream_id: users)
2021-01-11 14:46:35,002Z   main - INFO Saving structure of stream workspaces (tap_stream_id: workspaces)
2021-01-11 14:46:35,019Z   main - INFO Saving structure of stream teams (tap_stream_id: teams)
2021-01-11 14:46:35,037Z   main - INFO Saving structure of stream sections (tap_stream_id: sections)
2021-01-11 14:46:35,057Z   main - INFO Writing catalog to file
2021-01-11 14:46:35,083Z   main - INFO Selected fields for stream workspaces: name, resource_type, gid, is_organization
2021-01-11 14:46:35,083Z   main - INFO Selected fields for stream projects: current_status, notes, owner, due_date, name, archived, members, custom_fields, created_at, modified_at, followers, team, color, public, workspace, gid
2021-01-11 14:46:35,083Z   main - INFO Selected fields for stream users: email, name, workspaces, photo, gid
2021-01-11 14:46:35,083Z   main - INFO Selected fields for stream tasks: memberships, completed, num_hearts, start_on, completed_at, assignee, parent, projects, name, notes, assignee_status, custom_fields, created_at, modified_at, followers, hearts, due_on, tags, gid, due_at, hearted, workspace, external
2021-01-11 14:46:35,083Z   main - INFO Selected fields for stream tags: color, followers, name, created_at, workspace, notes, gid
2021-01-11 14:46:35,087Z   main - INFO Current state: {"bookmarks": {"projects": {"modified_at": "2021-01-08T19:31:56.479000"}, "tasks": {"modified_at": "2021-01-11T12:30:31.757000"}, "tags": {"created_at": "2020-12-02T21:44:23.859000"}}}
2021-01-11 14:46:35,087Z   main - INFO Starting tap: tap-env/bin/tap-asana --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json
2021-01-11 14:46:35,091Z   main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json
2021-01-11 14:46:35,525Z    tap - INFO Syncing stream: projects
2021-01-11 14:46:35,543Z target - INFO Using batch_size_prefernces of {'batch_size_preference': None, 'user_batch_size_preference': None, 'full_table_streams': []}
2021-01-11 14:47:05,079Z    tap - INFO Skipping stream: portfolios
2021-01-11 14:47:05,079Z    tap - INFO Syncing stream: tasks
2021-01-11 14:53:02,875Z target - INFO Serializing batch with 32 messages for table tasks
2021-01-11 14:53:02,883Z target - INFO Sending batch of 249061 bytes to https://api.stitchdata.com/v2/import/batch
2021-01-11 14:53:03,213Z   main - INFO State update: adding bookmarks.projects.modified_at = "2021-01-11T14:01:10.940000"
2021-01-11 14:53:03,213Z   main - INFO State update: adding bookmarks.tags.created_at = "2020-12-02T21:44:23.859000"
2021-01-11 14:53:03,213Z   main - INFO State update: adding bookmarks.tasks.modified_at = "2021-01-11T12:30:31.757000"
2021-01-11 14:53:03,717Z target - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [1] at offset None
2021-01-11 14:53:03,717Z target - INFO replicated 32 records from "tasks" endpoint
2021-01-11 14:53:03,718Z target - INFO Serializing batch with 3 messages for table projects
2021-01-11 14:53:03,718Z target - INFO Sending batch of 7787 bytes to https://api.stitchdata.com/v2/import/batch
2021-01-11 14:53:03,723Z target - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [1] at offset None
2021-01-11 14:53:03,723Z target - INFO replicated 3 records from "projects" endpoint
2021-01-11 14:55:53,228Z    tap - INFO Skipping stream: stories
2021-01-11 14:55:53,229Z    tap - INFO Syncing stream: tags
2021-01-11 14:56:10,099Z    tap - INFO Syncing stream: users
2021-01-11 14:56:19,001Z    tap - INFO Syncing stream: workspaces
2021-01-11 14:56:20,858Z    tap - INFO Skipping stream: teams
2021-01-11 14:56:20,858Z    tap - INFO Skipping stream: sections
2021-01-11 14:56:20,858Z    tap - INFO ----------------------
2021-01-11 14:56:20,859Z    tap - INFO tasks: 83
2021-01-11 14:56:20,859Z    tap - INFO workspaces: 1
2021-01-11 14:56:20,859Z    tap - INFO tags: 2
2021-01-11 14:56:20,859Z    tap - INFO projects: 3
2021-01-11 14:56:20,859Z    tap - INFO users: 75
2021-01-11 14:56:20,859Z    tap - INFO ----------------------
2021-01-11 14:56:20,894Z target - INFO Serializing batch with 75 messages for table users
2021-01-11 14:56:20,895Z target - INFO Sending batch of 57577 bytes to https://api.stitchdata.com/v2/import/batch
2021-01-11 14:56:20,909Z target - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [8] at offset None
2021-01-11 14:56:20,909Z target - INFO replicated 75 records from "users" endpoint
2021-01-11 14:56:20,909Z target - INFO Serializing batch with 1 messages for table workspaces
2021-01-11 14:56:20,910Z target - INFO Sending batch of 857 bytes to https://api.stitchdata.com/v2/import/batch
2021-01-11 14:56:20,919Z target - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [3] at offset None
2021-01-11 14:56:20,919Z target - INFO replicated 1 records from "workspaces" endpoint
2021-01-11 14:56:20,919Z target - INFO Serializing batch with 51 messages for table tasks
2021-01-11 14:56:20,919Z target - INFO Sending batch of 535520 bytes to https://api.stitchdata.com/v2/import/batch
2021-01-11 14:56:20,925Z target - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [7] at offset None
2021-01-11 14:56:20,925Z target - INFO replicated 51 records from "tasks" endpoint
2021-01-11 14:56:20,925Z target - INFO Serializing batch with 2 messages for table tags
2021-01-11 14:56:20,925Z target - INFO Sending batch of 2167 bytes to https://api.stitchdata.com/v2/import/batch
2021-01-11 14:56:20,931Z target - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [5] at offset None
2021-01-11 14:56:20,931Z target - INFO replicated 2 records from "tags" endpoint
2021-01-11 14:56:21,334Z   main - INFO State update: changing bookmarks.tags.created_at from "2020-12-02T21:44:23.859000" to "2021-01-11T13:43:52.589000"
2021-01-11 14:56:21,334Z   main - INFO State update: changing bookmarks.tasks.modified_at from "2021-01-11T12:30:31.757000" to "2021-01-11T14:51:28.948000"
2021-01-11 14:56:21,336Z target - INFO Requests complete, stopping loop
2021-01-11 14:56:21,405Z   main - INFO Target exited normally with status 0
2021-01-11 14:56:21,766Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [15] at offset None
2021-01-11 14:56:21,767Z   main - INFO No tunnel subprocess to tear down
2021-01-11 14:56:21,767Z   main - INFO Exit status is: Discovery succeeded. Tap succeeded. Target succeeded.

Adding Section endpoint

Hello!
It would be really helpful to add the 'section' endpoint to tasks! Would anybody else find this useful?

Stories table can not be request the endpoint be supported by the integration

Membership column in tasks is empty

The membership column in my tasks table is empty and I want it have populated so I can link the tasks to their respective sections. Can anyone help with this?

Tasks w/o projects

I tried this on Stitch > BigQuery upsert. I can't find tasks without any project assigned to them.

Add resource_subtype to Tasks

The resource_subtype field contains information on if the task is a Milestone or not. We need to be able to filter by whether a task is a Milestone or not. Since no other field seems to contain this info please add the resource_subtype to tasks. Thanks!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.