curts0 / pytabular Goto Github PK

View Code? Open in Web Editor NEW

68.0 68.0 11.0 89.71 MB

Connect to Tabular Models via Python

Home Page: https://curts0.github.io/PyTabular/

License: MIT License

Python 99.88% Batchfile 0.12%

aas dax devops-tools gen2 python tabular windows

pytabular's People

Contributors

Stargazers

Watchers

Forkers

kevarnold972 madhumathi2 pablojmoreno bugzey daandamhuis mattburnham gustavoleo ustdoestech grenzi hoanghce jamestiotio

pytabular's Issues

Implement pyenv to pytest multiple python versions

https://github.com/pyenv/pyenv

need models specifically for pytest not local models

Then can run pytest in github workflow

Refresh the Model

Hi,

Is there a way to first process Data Only and the Calculate?

import pytabular
model = pytabular.Tabular(CONNECTION_STR)
tables = model.Tables
if len(tables) > 0:
    tables.refresh(RefreshType.DataOnly)

model.refresh(RefreshType.Calculate)

I'm getting a Recursion after merging some changes from the master branch. Did anything change? I'm testing it on a production model, so maybe the adventure works model doesn't have enough translations to generate a recursion error.

self.Cultures = PyCultures(
    [
        PyCulture(culture, self)
        for culture in self.Model.Cultures.GetEnumerator()
    ]
)

PyTabular/pytabular/pytabular.py

Line 119 in 18210eb

self.Cultures = PyCultures(

Complete updating with google docstring

import error in Azure Functions

I am using python-tabular to refresh partitions in SSAS model through python script. While deploying the code in Azure Functions, I tried to trigger the azure functions, getting the import error in the import statement of this particular library.

Azure Function:
Operating System : Linux
Runtime version: 4.25.2.2
Language used : Python
Python version: 3.10

Full Exception :
 Exception while executing function
 /Functions.httpTrigger ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException 
 /Result
 /Failure Exception
 /RuntimeError
 /Could not find libmono 

 /  
 '/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/dispatcher.py', line 380, in _handle__function_load_request     func = loader.load_function(   
 '/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/utils/wrappers.py', line 44, in call     return func(*args, **kwargs)   
 '/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/loader.py', line 132, in load_function     mod = importlib.import_module(fullmodname)   
 '/usr/local/lib/python3.10/importlib/__init__.py', line 126, in import_module     return _bootstrap._gcd_import(name[level:], package, level)   
 '<frozen importlib._bootstrap>', line 1050, in _gcd_import   
 '<frozen importlib._bootstrap>', line 1027, in _find_and_load   
 '<frozen importlib._bootstrap>', line 1006, in _find_and_load_unlocked   
 '<frozen importlib._bootstrap>', line 688, in _load_unlocked   
 '<frozen importlib._bootstrap_external>', line 883, in exec_module   
 '<frozen importlib._bootstrap>', line 241, in _call_with_frames_removed   
 '/home/site/wwwroot/httpTrigger/__init__.py', line 9, in <module>     from .refresh_automation import main_func   
 '/home/site/wwwroot/httpTrigger/refresh_automation.py', line 6, in <module>     import pytabular   
 '/home/site/wwwroot/.python_packages/lib/site-packages/pytabular/__init__.py', line 45, in <module>     import clr   
 '/home/site/wwwroot/.python_packages/lib/site-packages/clr.py', line 6, in <module>     load()   
 '/home/site/wwwroot/.python_packages/lib/site-packages/pythonnet/__init__.py', line 36, in load     set_default_runtime()   
 '/home/site/wwwroot/.python_packages/lib/site-packages/pythonnet/__init__.py', line 22, in set_default_runtime     set_runtime(clr_loader.get_mono())   
 '/home/site/wwwroot/.python_packages/lib/site-packages/clr_loader/__init__.py', line 21, in get_mono     libmono = find_libmono(sgen)   
 '/home/site/wwwroot/.python_packages/lib/site-packages/clr_loader/util/find.py', line 67, in find_libmono     raise RuntimeError('Could not find libmono')     
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) 
/src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs 
 /101    
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) 
/src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs 
 /82    
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) 
/src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs 
 /225    
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs 
 /52    
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /581    
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /527    
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /306    End of inner exception    
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /352    
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /108

[Question] Documenting a Model

Hi,

I was looking into the PyTabular package, and first of thanks, finally a worthy implementation! On top of this package, I started "building" a Documenter for our models, because we use Docusaurus with generated markdown files to create our documentation site, which users can use to see how things are calculated for instance.

It seems a bit "strange" to build it on top of the PyTabular package, instead of including it in the package. Is this something you have envisioned for PyTabular? Otherwise, I'll continue building it on top of PyTabular.

Simple Example

Thanks!

Refresh by table string not working

subclass TE2 and BPA with namedtuples

more robust bpa_analyze() to handle all arguments passible through command line

Need to handle class initialization with model changes

Fix Datatype Conversion Issues from Pandas -> M & Dax

Linux Compatibility - Currently Windows Only

Describe the bug
Running the provided test suite produces an error with importing clr with Python 3.10 and Manjaro Linux. If we try importing the package from an interactive Python shell, the error is more specific about the clr package not having an AddReference attribute. Logs attached

To Reproduce

Create and activate Python a virtual environment
Install the package to the virtual environment
run pytest
Pytest gets interrupted during collection

Expected behavior
Tests load correctly and start running.

Screenshots
(None)

Desktop (please complete the following information):

OS: Linux 5.19.7-1-MANJARO #1 SMP PREEMPT_DYNAMIC Mon Sep 5 10:16:57 UTC 2022 x86_64 GNU/Linux
Python Version: Python 3.10.6 (main, Aug 3 2022, 17:39:45) [GCC 12.1.1 20220730] on linux

Additional context
Seeing that the module comes bundled with .dll files led me to believe that this package was never meant to run under anything other than Microsoft Windows even though any automation of data processes would likely be carried out by data engineers, developed and tested under Linux or the Windows Subsystem for Linux (WSL), and deployed to some cloud service such as functions, app services, containers or virtual machines. All of those predominantly run Linux. The current package should either try to support and test on such systems or explicitly state that they are unsupported in the README file.

Pytest log:

=================================================================================== test session starts ====================================================================================
platform linux -- Python 3.10.6, pytest-7.1.1, pluggy-1.0.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/radi/Code/PyTabular, configfile: pyproject.toml
plugins: anyio-3.6.1
collected 0 items / 1 error                                                                                                                                                                

========================================================================================== ERRORS ==========================================================================================
__________________________________________________________________________ ERROR collecting test/test_tabular.py ___________________________________________________________________________
ImportError while importing test module '/home/radi/Code/PyTabular/test/test_tabular.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.local/lib/python3.10/site-packages/_pytest/python.py:608: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
../../.local/lib/python3.10/site-packages/_pytest/pathlib.py:533: in import_path
    importlib.import_module(module_name)
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
../../.local/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
    exec(co, module.__dict__)
test/test_tabular.py:1: in <module>
    import pytabular
pytabular/__init__.py:32: in <module>
    import clr
E   ModuleNotFoundError: No module named 'clr'
================================================================================= short test summary info ==================================================================================
ERROR test/test_tabular.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 1 error in 0.08s =====================================================================================

Import log:

Python 3.10.6 (main, Aug  3 2022, 17:39:45) [GCC 12.1.1 20220730] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytabular
22/10/01 10:06:51 +0300::__init__::<module>::INFO::Logging configured...
22/10/01 10:06:51 +0300::__init__::<module>::INFO::To update PyTabular logger...
22/10/01 10:06:51 +0300::__init__::<module>::INFO::>>> import logging
22/10/01 10:06:51 +0300::__init__::<module>::INFO::>>> pytabular.logger.setLevel(level=logging.INFO)
22/10/01 10:06:51 +0300::__init__::<module>::INFO::See https://docs.python.org/3/library/logging.html#logging-levels
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Python Version::3.10.6 (main, Aug  3 2022, 17:39:45) [GCC 12.1.1 20220730]
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Python Location::/home/radi/Code/PyTabular/venv
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Package Location::/home/radi/Code/PyTabular/pytabular/__init__.py
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Working Directory::/home/radi/Code/PyTabular
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Platform::linux-5.19.7-1-MANJARO
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Beginning CLR references...
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Adding Reference Microsoft.AnalysisServices.AdomdClient
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/radi/Code/PyTabular/pytabular/__init__.py", line 35, in <module>
    clr.AddReference("Microsoft.AnalysisServices.AdomdClient")
AttributeError: module 'clr' has no attribute 'AddReference'

Documentation Generation

I've added a ModelDocumenter class which enables generating markdown files based on the Docusaurus notation for rendering the pages. It's not neat yet, but I will address that in the coming week. Is it easier to put the documentation in a submodule/subfolder?

@Curts0 What is easier, keep everything in the root folder of pytabular or create a documentation subfolder with all specific doc generation functions and classes?

import pytabular
import logging

logger = logging.getLogger("PyTabular")
model = pytabular.Tabular(f"{SERVER};Catalog={INITIAL_CATALOG}")

docs = pytabular.ModelDocumenter(model)
docs.save_documentation()

To-do:

Get translations per object
Link related objects with custom Id's ## 'Date'[Date] {#date_date}
Show all available attributes per object.
Cleanup class for the strings, without the "strange" indention.
Alternative documentation sites (e.g. mkdocs)
Make page setup dynamic -> Choose which fields needs to be display.

Integration Open Metadata

I've been looking into Open Metadata and seeing if it's possible to add lineage from SQL Server to Power BI datasets.

So I'll be trying to add that to PyTabular or at the very least leverage PyTabular.

@Curts0; How likely is Linux support?

Better PEP-8 Compliance

Skimming through the code, I see that we are violating quite a lot of standard Python conventions both in the object names, project structure, and imports. Considering that the project is considered in early alpha, would it be OK to take steps to make backwards-incompatible changes at this point in time and apply as many conventions as possible?

Most notably, I'd like to focus on the following:

Object naming:
- snake_case: function_name, method_name, global_variable_name
- CapitalCase: ClassName, ExceptionName
- SCREAMING_SNAKE_CASE: GLOBAL_CONSTANT
Line length - while PEP-8 recommends 72 characters, I can recommend 100 from my personal experience. This goes doubly so for docstrings
Standardized docstring format - you seem to be using the Google Python guide on this matter, although line lengths and indentation are off. This could be fixed easily Reference
Using only absolute imports from the project:
- from pytabular.sub_module import whatever (You seem to have fixed that recently?)
Using a tests folder (plural)

Implementing the above would probably also remove some flake8 exceptions the project has identified.