navikt / dataverk Goto Github PK

View Code? Open in Web Editor NEW

3.0 6.0 0.0 1.47 MB

Dataverk er et python bibliotek som forenkler gjennomføring av kodebasert ELT & ETL

License: MIT License

Python 100.00%

data etl-pipeline etl

dataverk's Introduction

Dataverk

Verktøykasse for kodebasert serverless ELT/ETL på NAIS med tilgangsstyring i Vault.

Getting started

pip install dataverk

Docs

About

Usage

dataverk's People

Contributors

Stargazers

Watchers

dataverk's Issues

Add support for dask in addition to pandas

Update docs to match current version

lage "dataverk init -open" funksjonalitet for bruk av dataverk i åpne projekter

Create functionality to start and run a dataverk project locally without NAV dependencies.

Felles akseptansetest som del av deploy pipeline

dataverk-cli delete should delete cronjob in addition to jenkins job

Error on dataverk-cli init -i on windows

(base) C:\projects\dataverktest\datasett-test>dataverk-cli init -i
dataverk-cli init completed
Traceback (most recent call last):
  File "c:\apps\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\apps\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\apps\Anaconda3\Scripts\dataverk-cli.exe\__main__.py", line 9, in <module>
  File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\dataverk_cli_entrypoint.py", line 40, in main
    settings_dict, env_store = get_package_configuration(args, initialize=True)
  File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\cli\cli_utils\package_config_handler.py", line 17, in get_package_configuration
    settings_dict = setting_store_functions.create_settings_dict(args=args, env_store=env_store)
  File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\cli\cli_utils\setting_store_functions.py", line 22, in create_settings_dict
    settings_dict = settings_loader.load_settings_file_from_resource(resource_url)
  File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\cli\cli_utils\settings_loader.py", line 16, in load_settings_file_from_resource
    settings_dict = _get_settings_dict_from_git_repo(url)
  File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\cli\cli_utils\settings_loader.py", line 64, in _get_settings_dict_from_git_repo
    return settings_dict
  File "c:\apps\anaconda3\lib\tempfile.py", line 807, in __exit__
    self.cleanup()
  File "c:\apps\anaconda3\lib\tempfile.py", line 811, in cleanup
    _shutil.rmtree(self.name)
  File "c:\apps\anaconda3\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\apps\anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\apps\anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\apps\anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\apps\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\apps\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Ingen tilgang: 'C:\\Users\\g153850\\AppData\\Local\\Temp\\tmph5te_rac\\.git\\objects\\pack\\pack-309141ef0ec03ac6e96825dc4e0fcfc9e8f7dd66.idx'

dataverk-cli init rework

Remove dataverk factory step
Remove dependency on remote and local repo
create open-source settings repo

Rette feil/mangler i henhold til krav i navikt/utvikling (funnet av roboten repo-linter)

Dette er en autogenerert issue, laget av et skript som går gjennom alle NAV sine kodebaser på Github og gjør diverse sjekker. Her er en liste over ting som må endres.

Beskrivelse mangler

På Github kan man gi hver kodebase en kort beskrivelse. Denne bør fortelle hva kodebasen heter, og litt om hva den brukes til. (Eksempel: kodebasen "veilarbportefoljeflatefs" har beskrivelse "Oversikt for veiledere over oppfølgingsbrukere".)
NB! Dette gjelder ikke beskrivelse i en README-fil, det gjelder beskrivelse i metadataen til selve Github-repoet.

Når alle endringer er gjort, så kan denne saken lukkes.

Spørsmål og svar

Jeg har meninger om disse rådene - kan jeg komme med tilbakemeldinger?

Skriv i vei, på Slack-kanalen #open-source.

Kodebasen vår er ikke open source, derfor er det ikke noe poeng

Selv om koden i dag ikke er åpen for innsyn, så ta høyde for at den kan komme til å bli det i fremtiden. Uansett så vil forbedringene være til hjelp, enten kodebasen er åpen eller ei!

Hvem har ansvaret for å fikse det her?

Det er i utgangspunktet den/de/teamet som eier kodebasen som må fikse.

Det er en feil i rådene

Alle roboter gjør jo feil, denne også. Lag en issue på https://github.com/navikt/repo-linter.

Støtte for xls(x) i add_resource

Implement WindowsSafeTemporaryDirectory class

Postgres connector similar to oracle

Integrasjon mot S3

Jeg savner Koala-funksjonaliteten som lar meg laste ned/opp pickles fra S3 med en oneliner i Dataverk. Slik det er nå så må jeg kjøre en request, bruke BytesIO m.m. for å pickle og laste opp en pandas dataframe. Hadde vært smud om dette kunne blitt gjort enklere.

Støtte for dsv-filer (konfigurerbar deliminator) i add_resource

handle new lines in secrets from vault

gcloud publish bug: no priv_key for gcloud in settings object

Publishing package tester-med-ny-dv-versjon
The private_key field was not found in the service account info.
dataverk-cli publish completed
Traceback (most recent call last):
File "/usr/local/bin/dataverk-cli", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_cli_entrypoint.py", line 67, in main
publish_datapackage()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_publish.py", line 95, in publish_datapackage
datapackage.publish()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_publish.py", line 87, in publish
self.package_settings["package_name"]))
File "/usr/local/lib/python3.6/site-packages/dataverk/utils/publish_data.py", line 6, in upload_to_storage_bucket
conn.upload_blob(os.path.join(dir_path, 'datapackage.json'), datapackage_key_prefix + 'datapackage.json')
File "/usr/local/lib/python3.6/site-packages/dataverk/connectors/google_storage.py", line 72, in upload_blob
blob = self.bucket.blob(destination_blob_name)
AttributeError: 'GoogleStorageConnector' object has no attribute 'bucket'

Error when trying to create new local project

(IndexError: tuple index out of range) is thrown when trying to execute dataverk-cli init from a new local project with no matching github repository

location: line 68 in _getssh_url /dataverk_base.py

throw and handle exception when writing to es index fails

Improve vault secrets handling per datapackage

Functionality for writing dataframe to oracle

dataverk init -i feiler med ValueError 'dataverk_s3' is not a valid BucketStorage

/home/G153850/projects/datapakke-arbeidsledighet $ dataverk-cli init -i
Skriv inn ønsket pakkenavn: arbeidsledighet
Do you want to create the datapackage (arbeidsledighet)? [y/n] y
dataverk-cli init completed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 29, in run
    self._create()
  File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 40, in _create
    self._edit_package_metadata()
  File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 90, in _edit_package_metadata
    package_metadata['path'] = self._determine_bucket_path()
  File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 105, in _determine_bucket_path
    if BucketStorage(bucket_type) == BucketStorage.GITHUB:
  File "/usr/local/lib/python3.6/enum.py", line 291, in __call__
    return cls.__new__(cls, value)
  File "/usr/local/lib/python3.6/enum.py", line 533, in __new__
    return cls._missing_(value)
  File "/usr/local/lib/python3.6/enum.py", line 546, in _missing_
    raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: 'dataverk_s3' is not a valid BucketStorage

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/dataverk-cli", line 11, in <module>
    load_entry_point('dataverk==0.0.17', 'console_scripts', 'dataverk-cli')()
  File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_cli_entrypoint.py", line 46, in main
    dp.run()
  File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 32, in run
    raise Exception(f'Klarte ikke generere datapakken {self._settings_store["package_name"]}')
Exception: Klarte ikke generere datapakken arbeidsledighet

View the update logs.

/usr/local/lib/python3.7/site-packages/dask/dataframe/utils.py:15: FutureWarning:

pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.

INFO:numexpr.utils:Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:numexpr.utils:NumExpr defaulting to 8 threads.