grai-io / grai-core Goto Github PK

License: Other

Dockerfile 0.09% Python 69.77% Makefile 0.13% Shell 0.27% HTML 5.96% CSS 0.01% TypeScript 23.72% TSQL 0.01% JavaScript 0.03%

hacktoberfest data datalineage dbt open-source postgresql python mssql mysql snowflake

grai-core's People

Contributors

Stargazers

Watchers

grai-core's Issues

Django USE_L10N setting is deprecated

The django USE_L10N setting is being deprecated and needs to be removed from settings.

[GRA-1072] [GRA-1073] Example docker compose files out of date

null

_GRA-1073

Integrate pull request preview environments

I would like to support Grai by implementing Uffizzi preview environments.
Disclaimer: I work on Uffizzi.

Uffizzi is a Open Source full stack previews engine and our platform is available completely free for Grai (and all open source projects). This will provide maintainers with preview environments of every PR in the cloud, which enables faster iterations and reduces time to merge. You can see the open source repos which are currently using Uffizzi over here

Uffizzi is purpose-built for the task of previewing PRs and it integrates with your workflow to deploy preview environments in the background without any manual steps for maintainers or contributors.

We can go ahead and create an Initial PoC for you right away if you think there is value in this proposal.

Add a default ordering on Django ViewSets

The current Django ViewSets do not include ordering or ordering_fields settings which can affect paginated endpoints.

These should be set on

When complete, tests will no longer generate UnorderedObjectListWarning's e.g.

workspaces/tests/test_views.py::TestMemberships::test_get_memberships
workspaces/tests/test_views.py::TestMemberships::test_get_memberships_filter_by_is_active
  /Users/ian/Library/Caches/pypoetry/virtualenvs/the-guide-y4fToIIm-py3.11/lib/python3.11/site-packages/rest_framework/pagination.py:200: UnorderedObjectListWarning: Pagination may yield inconsistent results with an unordered object_list: <class 'workspaces.models.Membership'> QuerySet.
    paginator = self.django_paginator_class(queryset, page_size)

[GRA-454] Organisation names with slash break routing

There is nothing to stop a user creating an organisation with one or more slashes in it. These break the Web App routing, which relies on slashes.

The same issue would occur with workspace names.

_GRA-454

Change `NGROK_URL` environment variable

This should be server_url or similar

test

Failed transactions in the client don't roll back

If the client is tasked with updating a batch of resource but fails midway through the previously executed transactions remain unchanged.

[GRA-464] Lineage to multiple foreign keys is omitted

We are currently bypassing foreign keys in the RDBMS integrations which key to multiple columns like can be found here: https://dba.stackexchange.com/questions/240160/postgresql-multiple-column-foreign-key

We should update integrations to handle this scenario.

_GRA-464

twisted.web has disordered HTTP pipeline response

This issue refers to Dependabot alert #155 https://github.com/grai-io/grai-core/security/dependabot/155

Tests using timezone naive datetime for timezone aware field

The following warning messages are being emitted for timezone naive test values:

connections/tests/test_tasks.py::TestEventsAllTests::test_dbt_cloud
  /Users/ian/Library/Caches/pypoetry/virtualenvs/the-guide-y4fToIIm-py3.11/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1535: RuntimeWarning: DateTimeField Event.date received a naive datetime (2023-10-09 00:00:00) while time zone support is active.
    warnings.warn(

auth/tests/test_mutations.py::test_verify_email
  /Users/ian/Library/Caches/pypoetry/virtualenvs/the-guide-y4fToIIm-py3.11/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1595: RuntimeWarning: DateTimeField User.verified_at received a naive datetime (2023-10-09 14:25:20.943758) while time zone support is active.
    warnings.warn(

These tests should be modified so that running tests no longer emits the respective warning message.

`'NoneType' object has no attribute 'encode'` on homepage search bar

Hi,

Running Grai locally and after creating a few nodes/edges, I still see this error as soon as I click the search bar on the homepage.

Here are the logs:

grai-server-frontend-1            | 172.19.0.1 - - [04/May/2023:21:16:52 +0000] "GET /default/default HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
grai-server-frontend-1            | 172.19.0.1 - - [04/May/2023:21:16:52 +0000] "GET /favicon.ico HTTP/1.1" 304 0 "http://localhost:3000/default/default" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
grai-server-the_guide-1           | 'NoneType' object has no attribute 'encode'
grai-server-the_guide-1           |
grai-server-the_guide-1           | GraphQL request:4:5
grai-server-the_guide-1           | 3 |     id
grai-server-the_guide-1           | 4 |     search_key
grai-server-the_guide-1           |   |     ^
grai-server-the_guide-1           | 5 |     __typename
grai-server-the_guide-1           | Traceback (most recent call last):
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/graphql/execution/execute.py", line 528, in await_result
grai-server-the_guide-1           |     return_type, field_nodes, info, path, await result
grai-server-the_guide-1           |                                           ^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry_django_plus/utils/aio.py", line 89, in resolve_async
grai-server-the_guide-1           |     ret = resolver(await value)
grai-server-the_guide-1           |                    ^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__
grai-server-the_guide-1           |     ret = await asyncio.wait_for(future, timeout=None)
grai-server-the_guide-1           |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/tasks.py", line 442, in wait_for
grai-server-the_guide-1           |     return await fut
grai-server-the_guide-1           |            ^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 22, in run
grai-server-the_guide-1           |     result = self.fn(*self.args, **self.kwargs)
grai-server-the_guide-1           |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 490, in thread_handler
grai-server-the_guide-1           |     return func(*args, **kwargs)
grai-server-the_guide-1           |            ^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/types/fields/resolver.py", line 187, in __call__
grai-server-the_guide-1           |     return self.wrapped_func(*args, **kwargs)
grai-server-the_guide-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/src/app/api/types.py", line 580, in search_key
grai-server-the_guide-1           |     return client.generate_secured_api_key(
grai-server-the_guide-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/src/app/api/search.py", line 6, in generate_secured_api_key
grai-server-the_guide-1           |     return SearchClient.generate_secured_api_key(parent_api_key, restrictions)
grai-server-the_guide-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/algoliasearch/search_client.py", line 275, in generate_secured_api_key
grai-server-the_guide-1           |     parent_api_key.encode("utf-8"),
grai-server-the_guide-1           |     ^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           | AttributeError: 'NoneType' object has no attribute 'encode'
grai-server-the_guide-1           | Stack (most recent call last):
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap
grai-server-the_guide-1           |     self._bootstrap_inner()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
grai-server-the_guide-1           |     self.run()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/threading.py", line 975, in run
grai-server-the_guide-1           |     self._target(*self._args, **self._kwargs)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
grai-server-the_guide-1           |     work_item.run()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
grai-server-the_guide-1           |     result = self.fn(*self.args, **self.kwargs)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 248, in _run_event_loop
grai-server-the_guide-1           |     loop.run_until_complete(coro)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
grai-server-the_guide-1           |     self.run_forever()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
grai-server-the_guide-1           |     self._run_once()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once
grai-server-the_guide-1           |     handle._run()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
grai-server-the_guide-1           |     self._context.run(self._callback, *self._args)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 306, in main_wrap
grai-server-the_guide-1           |     result = await self.awaitable(*args, **kwargs)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/django/views.py", line 254, in dispatch
grai-server-the_guide-1           |     return await self.run(request=request)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/http/async_base_view.py", line 176, in run
grai-server-the_guide-1           |     result = await self.execute_operation(
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/http/async_base_view.py", line 115, in execute_operation
grai-server-the_guide-1           |     return await self.schema.execute(
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/schema/schema.py", line 248, in execute
grai-server-the_guide-1           |     result = await execute(
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/schema/execute.py", line 156, in execute
grai-server-the_guide-1           |     process_errors(result.errors, execution_context)

Postgres tests fail on empty database

Tests currently rely upon the assumption of a populated database but it should be possible to run tests against an empty database.

[GRA-937] Add html to password verification email

Update password verification email to send both plain text and html content.

_GRA-937

CLI command produces recursion error

The command grai config view currently generates a RecursionError in the config's __getattr__.

[GRA-258] Pressing run on an existing run with a RunFile fails

To reproduce:

Add a source which uses a file, for example dbt or yaml file.
You should get a screen showing the run, as the file is processed.
On this screen press the Run button and you will get an error.

The bug relates to the fact that the file is associated with an individual run, as a RunFile Django model. Pressing the button causes a new Run to be created, but it isn't associated with the file. The solution is to copy the file and create a new RunFile for the new Run.

_{From SyncLinear.com | GRA-258}

ClickHouse support?

Thanks for the great tool, would you like to add ClickHouse support?

https://clickhouse.com/docs/en/

it have lot of connectors
https://github.com/xzkostyan/clickhouse-sqlalchemy.git
https://github.com/mymarilyn/clickhouse-driver

https://github.com/ClickHouse/clickhouse-connect

Create documentation for demo environment configs

Various docker deployment configs can be found in the examples folder alongside the cli command grai demo start. However these are largely undocumented. We should include more details in the various doc pages including

a copy of the docker scripts
A page about the cli demo commands

Redshift integration not detecting late-binding view columns

Description

Running the Redshift integration against databases with late-binding views fails to generate columns associated with late-binding views. Although referenced tables appear in information_schema.tables their associated columns do not appear in information_schema.columns resulting in an error here.

Proposed Solution

It appears possible to collect metadata from late binding views via pg_get_late_binding_view_cols which should be included alongside information_schema.columns.

Originally reported on slack

gunicorn worker timeout

Bug report was issued by Vignesh Hari. He's seeing worker timeout issues usually on boot but sporadically afterwards as well. He's running on iOS under apple silicon.

[GRA-909] Add html to email for verification

Current email template is at https://github.com/grai-io/grai-core/blob/af7a53ff85e0cd8b47698aca55f44897d9f583b6/grai-server/app/templates/auth/email_verification.txt but is text only, would be nice to add an html template.

I presume that Django has an off the shelf way to do this.

The current email could be improved with more text, sign off and so on.

_GRA-909

GraphQL APIs need Documentation

Although the REST API's have both swagger and redoc pages, the GraphQL API's are undocumented. It would be nice to include something like graphql playground but other options are possible.

The set of graphql implementations are available here

[GRA-908] [GRA-548] Improved test failures email

Currently just says "Test Failures"

_{From SyncLinear.com | GRA-548}

_GRA-908

grai-io / grai-core Goto Github PK

grai-core's People

Contributors

Stargazers

Watchers

Forkers

grai-core's Issues

Description

Proposed Solution

Recommend Projects

Recommend Topics

Recommend Org