Giter Club home page Giter Club logo

pbi-tools's Introduction

pbi-tools

pbi-tools is a command-line tool bringing source-control features to Power BI. It works alongside Power BI Desktop and enables mature enterprise workflows for Power BI projects.

An example project is available here: https://github.com/pbi-tools/adventureworksdw2020-pbix

GitHub release (latest by date) Twitter Follow Join the chat at https://gitter.im/pbi-tools/general

User Notes

Developer Notes

Build System

Prerequisites

  • Visual Studio 2022 17.2 or later (for MSBuild dependencies), incl C# 11 Preview
  • .Net 4.7.2 Targeting Pack
  • .Net 6.0 SDK
  • Power BI Desktop x64 (Must be installed in default location for local development: C:\Program Files\Microsoft Power BI Desktop\)

List Build Targets

dotnet fake build --list

Versioning

The project strictly adheres to SemVer v2 for release versioning. The build system uses the first entry in RELEASE_NOTES.md to inject version numbers into build artifacts.

Diagnostics

  • Log output can be controlled using the environment variable PBITOOLS_LogLevel.
  • Allowed values are:
    • Verbose
    • Debug
    • Information
    • Warning
    • Error
    • Fatal
  • The default is Information, which is also effective when an unknown/invalid option has been specified.

Build

.\build.cmd Build

Run Tests

.\build.cmd Test

Run All Targets (Build, Publish, Test, UsageDocs, Pack)

.\build.cmd Pack

Run only the specified build target

.\build.cmd UsageDocs -s
dotnet fake build -s -t SmokeTest

Install Dependencies

dotnet paket install

That is generally not needed as the build.cmd script takes care of fetching dependencies. However, it could be useful to run this manually on a fresh clone or after making changes in the paket.dependencies file.

Update Specific Dependency to latest version (ex: AMO)

dotnet paket update Microsoft.AnalysisServices.retail.amd64
dotnet paket update Microsoft.AnalysisServices.AdomdClient.retail.amd64

Updating All Dependencies (NuGet)

dotnet paket update
dotnet paket update -g Fake-Build

Find outdated dependencies

dotnet paket outdated -g Main

Invoke Build script directly

dotnet fake {...}
dotnet fake -t Build
dotnet fake --version

Extract embedded sample PBIX with local build version and using default settings

.\pbi-tools.local.cmd extract '.\data\Samples\Adventure Works DW 2020.pbix'

Extract embedded sample PBIX with local build version and 'Raw' serialization mode, into custom output folder

.\pbi-tools.local.cmd extract '.\data\Samples\Adventure Works DW 2020.pbix' -extractFolder '.\data\Samples\Adventure Works DW 2020 - Raw' -modelSerialization Raw

Enable Debug logging (PowerShell)

$env:PBITOOLS_LogLevel = "Debug"

Fast local build (no clean)

.\build.cmd Publish -s
.\build.cmd Pack -s

Git Submodules

Clone with submodules

git clone --recurse-submodules https://github.com/pbi-tools/pbi-tools.git

Pulling in Upstream Changes

git submodule update --remote

Clone specific single branch into named folder, with submodules

git clone -b Release/1.0.0-beta.9 --single-branch --recurse-submodules https://github.com/pbi-tools/pbi-tools.git ./1.0.0-beta.9

pbi-tools's People

Contributors

gitter-badger avatar mthierba avatar mthierba-ms avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pbi-tools's Issues

FEATURE: pbi-tools Core Version

A stripped-down version of pbi-tools, without the Power BI Desktop dependency, would extend the application of the tool to CI/CD environments, but would need to sacrifice some features:

Features possible without PBID

  • export-bim from sources, without data source generation, i.e. fine to target Power BI Premium/XMLA.
  • extract PBIX/PBIT: PBIT with data model and PBIX report with live connection possible.
  • compile-pbix: V3 model possible w/o Mashup part

Additional features possible

  • Deploy to PBI Premium/XMLA: Model sources from folder/repo, credentials to be provided

Feature: Settings inheritance

Inherit project settings from a parent folder or the user profile, allowing system-wide or solution-wide defaults.

  • Parameters
  • Environments
  • Options
  • Authentication

In the most ideal case, a pbixproj manifest for a given project would only declare mode and source, and otherwise inherit team/enterprise defaults from a referenced template.

CLI Action Names

Introduce some better aliases?

  • extract-data > export-data (too close to the 'extract' action)
  • export-bim > generate-bim (more appropriate verb)

Suggest pbi-tools info parameter to just return sessions

When calling pbi-tools info from external tool, we are only looking for the pbiSessions info.

If there is any delay caused by getting the other data amoVersion, pbiInstalls, effectivePbiInstallDir etc then it would be good to have a switch to just retrive the pbiSessions

However, I'm guessing that the pbiSessions might be the longest step, so only retrieving pbiSessions may not be appreciably faster. If this is the case then ignore the request.

Unhandled exception at extract

I have tried to run the beta 6 on a small local PBIX.

C:\Program Files\pbi-tools\pbi-tools.exe' extract '.\dataModel.pbix' -extractFolder './data'

I got the following error:

Using Power BI Desktop install: 2.98.882.0 (21.10) at C:\Program Files\WindowsApps\Microsoft.MicrosoftPowerBIDesktop_2.98.882.0_x64__8wekyb3d8bbwe\bin
An unhandled exception occurred.
System.IO.FileFormatException: File contains corrupted data.
   at MS.Internal.IO.Zip.ZipIOEndOfCentralDirectoryBlock.FindPosition(Stream archiveStream)
   at MS.Internal.IO.Zip.ZipIOEndOfCentralDirectoryBlock.SeekableLoad(ZipIOBlockManager blockManager)
   at MS.Internal.IO.Zip.ZipArchive..ctor(Stream archiveStream, FileMode mode, FileAccess access, Boolean streaming, Boolean ownStream)
   at MS.Internal.IO.Zip.ZipArchive.OpenOnStream(Stream stream, FileMode mode, FileAccess access, Boolean streaming)
   at System.IO.Packaging.ZipPackage..ctor(Stream s, FileMode mode, FileAccess access, Boolean streaming)
   at System.IO.Packaging.Package.Open(Stream stream, FileMode packageMode, FileAccess packageAccess, Boolean streaming)
   at Microsoft.PowerBI.Packaging.PowerBIPackager.Open(Stream storageStream, Boolean& requiresConversionToReportLayout, Byte[]& securityBindings, Version& pbixFileVersion, Boolean skipValidation)
   at PbiTools.PowerBI.PbixReader..ctor(String pbixPath, IDependenciesResolver resolver) in E:\pbi-tools\pbi-tools-cli\src\PBI-Tools\PowerBI\PbixReader.cs:line 42
   at PbiTools.CmdLineActions.Extract(String pbixPath, Int32 pbiPort, String extractFolder, ExtractActionCompatibilityMode mode, ModelSerializationMode modelSerialization, MashupSerializationMode mashupSerialization) in E:\pbi-tools\pbi-tools-cli\src\PBI-Tools\CmdLineActions.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at PowerArgs.ArgAction.Invoke()
   at PbiTools.Program.Main(String[] args) in E:\pbi-tools\pbi-tools-cli\src\PBI-Tools\Program.cs:line 76

Ignoring certain properties at scripting

I have been using pbi-tools for a few days now, and I find it super good - a real game changer IMHO.

What I have seen is that I get a lot of "noise" changes when I script the PBIX model, and I was thinking what would be the impact of suppressing scripting certain properties.

Example:

  • lastModified property: every time I save the file (like refresh data), this property will change although there is no structural change in the PBIX file
  • I have seen a bunch of properties (I guess PBI internal stuff), like PBI_ChangedProperties. Like changing a field from visible to invisible is not just adding isHidden: true (which is OK and expected), but also
   {
      "name": "PBI_ChangedProperties",
      "value": "[\"IsHidden\"]"
    } 

So are these properties really needed to compile a new PBIX from the scripted model, or would it make sense NOT to script them?

pbi-tools info misreportes file path

Two instances where info action returns the folder instead of the actual path to the pbix file - I believe it could be when the folder name has ".pbix" in it. You might need to ensure that the match being returned is actually a file and not a folder.

[extract-data] fails when -outPath has trailing back-slash

pbi-tools extract-data -pbixPath '.\Dataset.pbix' -outPath '.\Output\'
Using Power BI Desktop install: 2.87.762.0 (20.11) at C:\Program Files\WindowsApps\Microsoft.MicrosoftPowerBIDesktop_2.87.762.0_x64__8wekyb3d8bbwe\bin
MSMDSRV.EXE found at C:\Users\mt\AppData\Local\pbi-tools\Microsoft.MicrosoftPowerBIDesktop_2.87.762.0_x64__8wekyb3d8bbwe\msmdsrv.exe
Working directory: C:\Users\mt\AppData\Local\Temp\tmp3F58.tmp
Tabular Server process launched successfully. Port: 57177
Model image from PBIX loaded successfully.
An unhandled exception occurred.
System.ArgumentException: Illegal characters in path.
   at System.IO.LongPathHelper.Normalize(String path, UInt32 maxPathLength, Boolean checkInvalidCharacters, Boolean expandShortPaths)
   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
   at System.IO.Path.GetFullPathInternal(String path)
   at System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost)
   at PbiTools.TabularModel.TabularDataReader.ExtractTableData(String outPath) in E:\temp\PBIX-Tools\src\PBI-Tools\TabularModel\TabularDataReader.cs:line 58
   at PbiTools.CmdLineActions.ExtractData(Int32 port, String pbixPath, String outPath) in E:\temp\PBIX-Tools\src\PBI-Tools\CmdLineActions.cs:line 229
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at PowerArgs.ArgAction.Invoke()
   at PbiTools.Program.Main(String[] args) in E:\temp\PBIX-Tools\src\PBI-Tools\Program.cs:line 73
  • Error occurs on Directory.CreateDirectory()

Extract report-level measures into separate files

  • Currently held in /Report/config.json (modelExtensions[])

Proposed folder structure

/Report
  config.json
  /measures
    /{table}
      {measure}.dax
      {measure}.json

Example

"modelExtensions": [
    {
      "name": "extension",
      "entities": [
        {
          "name": "Info",
          "extends": "Info",
          "measures": [
            {
              "name": "_Version",
              "dataType": 3,
              "expression": "\"v2.19\"",
              "errorMessage": null,
              "hidden": false,
              "formulaOverride": null,
              "formatInformation": {
                "formatString": "G",
                "format": "General",
                "thousandSeparator": false,
                "currencyFormat": null,
                "dateTimeCustomFormat": null
              }
            }
          ]
        },
        {
          "name": "Currency",
          "extends": "Currency",
          "measures": [
            {
              "name": "_Report Currency",
              "dataType": 1,
              "expression": "\nIF (\n    'Currency'[Report Currency] = \"N/A\",\n    \"Multiple CUR\",\n    'Currency'[Report Currency]\n)",
              "errorMessage": null,
              "hidden": false,
              "references": {
                "unrecognizedReferences": false,
                "measures": [
                  {
                    "schema": "",
                    "entity": "Currency",
                    "name": "Report Currency"
                  }
                ]
              },
              "formulaOverride": null,
              "formatInformation": {
                "formatString": "G",
                "format": "Text",
                "thousandSeparator": false,
                "currencyFormat": null,
                "dateTimeCustomFormat": null
              }
            }
          ]
        }
]

pbi-tools compile-pbix -folder argument

When you try to execute the compile-pbix action with a folder string without the trailing "" it fails

c:\MyFolder\MySubfolder

You need to correct the folder string to have the trailing "" for it to work:
c:\MyFolder\MySubfolder\

Would it be possible to accept either input?

Find SSRS Installers

Hi, we are using the Power BI Report Server (on-premise). This brings it's own flavour of the PowerBI Desktop Tools with it. Most of it is seemlingly just the same, there is just a few options to deploy from the desktop tools to the report server.

However: The installation is a little different in terms of folder names and thus the installation cannot be found currently by the pbi-tools.

This is the registry path
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Power BI Desktop SSRS\Installer

This is the settings path:
C:\Users\{UserName}\AppData\Local\Microsoft\Power BI Desktop SSRS

The actual installation path seems to be this C:\Program Files\Microsoft Power BI Desktop RS\, however this value can be identified via the regvalue anyways.

Is there any chance you would add this lookup to the tool?

Build failed. 2 CS0173 Errors - type of conditional expression cannot be determined because there is no implicit conversion between...

VS Code, Windows 10.
.\build.cmd Build

2 errors
CmdLineActions.cs(93,93): error CS0173: Type of conditional expression cannot be determined because there is no implicit conversion between 'int' and ''

FileSystem\ProjectFolder.cs(179,13): error CS0173: Type of conditional expression cannot be determined because there is no implicit conversion between 'System.Collections.Generic.IEnumerable<PbiTools.FileSystem.ProjectFile>' and 'PbiTools.FileSystem.IProjectFile[]'

PBI Desktop version check

Once runtime PBI Desktop install is determined, check version and warn if detected version is too far behind current version.

User focussed file and folder naming conventions in report repository

Suggest we consider user-focussed naming throughout the report repository. It is really going to help disambiguate and provide cues to the user when they are viewing the report in source control.

sections folder to pages

  • pageconfig.json
  • pagefilters.json
  • page.json

visualContainers to visuals

  • visualconfig.json
  • visualfilters.json
  • visual.json

report

  • reportconfig.json
  • reportfilters.json
  • report.json

pods to drillthrough?
bookmarks?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.