Dataverk
Verktøykasse for kodebasert serverless ELT/ETL på NAIS med tilgangsstyring i Vault.
Getting started
pip install dataverk
Dataverk er et python bibliotek som forenkler gjennomføring av kodebasert ELT & ETL
License: MIT License
Create functionality to start and run a dataverk project locally without NAV dependencies.
(base) C:\projects\dataverktest\datasett-test>dataverk-cli init -i
dataverk-cli init completed
Traceback (most recent call last):
File "c:\apps\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\apps\anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\apps\Anaconda3\Scripts\dataverk-cli.exe\__main__.py", line 9, in <module>
File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\dataverk_cli_entrypoint.py", line 40, in main
settings_dict, env_store = get_package_configuration(args, initialize=True)
File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\cli\cli_utils\package_config_handler.py", line 17, in get_package_configuration
settings_dict = setting_store_functions.create_settings_dict(args=args, env_store=env_store)
File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\cli\cli_utils\setting_store_functions.py", line 22, in create_settings_dict
settings_dict = settings_loader.load_settings_file_from_resource(resource_url)
File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\cli\cli_utils\settings_loader.py", line 16, in load_settings_file_from_resource
settings_dict = _get_settings_dict_from_git_repo(url)
File "c:\apps\anaconda3\lib\site-packages\dataverk_cli\cli\cli_utils\settings_loader.py", line 64, in _get_settings_dict_from_git_repo
return settings_dict
File "c:\apps\anaconda3\lib\tempfile.py", line 807, in __exit__
self.cleanup()
File "c:\apps\anaconda3\lib\tempfile.py", line 811, in cleanup
_shutil.rmtree(self.name)
File "c:\apps\anaconda3\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\apps\anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\apps\anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\apps\anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\apps\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "c:\apps\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Ingen tilgang: 'C:\\Users\\g153850\\AppData\\Local\\Temp\\tmph5te_rac\\.git\\objects\\pack\\pack-309141ef0ec03ac6e96825dc4e0fcfc9e8f7dd66.idx'
Remove dataverk factory step
Remove dependency on remote and local repo
create open-source settings repo
Dette er en autogenerert issue, laget av et skript som går gjennom alle NAV sine kodebaser på Github og gjør diverse sjekker. Her er en liste over ting som må endres.
På Github kan man gi hver kodebase en kort beskrivelse. Denne bør fortelle hva kodebasen heter, og litt om hva den brukes til. (Eksempel: kodebasen "veilarbportefoljeflatefs" har beskrivelse "Oversikt for veiledere over oppfølgingsbrukere".)
NB! Dette gjelder ikke beskrivelse i en README-fil, det gjelder beskrivelse i metadataen til selve Github-repoet.
Når alle endringer er gjort, så kan denne saken lukkes.
Skriv i vei, på Slack-kanalen #open-source.
Selv om koden i dag ikke er åpen for innsyn, så ta høyde for at den kan komme til å bli det i fremtiden. Uansett så vil forbedringene være til hjelp, enten kodebasen er åpen eller ei!
Det er i utgangspunktet den/de/teamet som eier kodebasen som må fikse.
Alle roboter gjør jo feil, denne også. Lag en issue på https://github.com/navikt/repo-linter.
Jeg savner Koala-funksjonaliteten som lar meg laste ned/opp pickles fra S3 med en oneliner i Dataverk. Slik det er nå så må jeg kjøre en request, bruke BytesIO m.m. for å pickle og laste opp en pandas dataframe. Hadde vært smud om dette kunne blitt gjort enklere.
Publishing package tester-med-ny-dv-versjon
The private_key field was not found in the service account info.
dataverk-cli publish completed
Traceback (most recent call last):
File "/usr/local/bin/dataverk-cli", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_cli_entrypoint.py", line 67, in main
publish_datapackage()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_publish.py", line 95, in publish_datapackage
datapackage.publish()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_publish.py", line 87, in publish
self.package_settings["package_name"]))
File "/usr/local/lib/python3.6/site-packages/dataverk/utils/publish_data.py", line 6, in upload_to_storage_bucket
conn.upload_blob(os.path.join(dir_path, 'datapackage.json'), datapackage_key_prefix + 'datapackage.json')
File "/usr/local/lib/python3.6/site-packages/dataverk/connectors/google_storage.py", line 72, in upload_blob
blob = self.bucket.blob(destination_blob_name)
AttributeError: 'GoogleStorageConnector' object has no attribute 'bucket'
(IndexError: tuple index out of range) is thrown when trying to execute dataverk-cli init from a new local project with no matching github repository
location: line 68 in _getssh_url /dataverk_base.py
/home/G153850/projects/datapakke-arbeidsledighet $ dataverk-cli init -i
Skriv inn ønsket pakkenavn: arbeidsledighet
Do you want to create the datapackage (arbeidsledighet)? [y/n] y
dataverk-cli init completed
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 29, in run
self._create()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 40, in _create
self._edit_package_metadata()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 90, in _edit_package_metadata
package_metadata['path'] = self._determine_bucket_path()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 105, in _determine_bucket_path
if BucketStorage(bucket_type) == BucketStorage.GITHUB:
File "/usr/local/lib/python3.6/enum.py", line 291, in __call__
return cls.__new__(cls, value)
File "/usr/local/lib/python3.6/enum.py", line 533, in __new__
return cls._missing_(value)
File "/usr/local/lib/python3.6/enum.py", line 546, in _missing_
raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: 'dataverk_s3' is not a valid BucketStorage
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/dataverk-cli", line 11, in <module>
load_entry_point('dataverk==0.0.17', 'console_scripts', 'dataverk-cli')()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_cli_entrypoint.py", line 46, in main
dp.run()
File "/usr/local/lib/python3.6/site-packages/dataverk_cli/dataverk_init.py", line 32, in run
raise Exception(f'Klarte ikke generere datapakken {self._settings_store["package_name"]}')
Exception: Klarte ikke generere datapakken arbeidsledighet
Datapackage-functionality should be a separate package. Should be usable without cli tool or complex settings handling.
Dependabot couldn't authenticate with https://pypi.python.org/simple/.
You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.
/usr/local/lib/python3.7/site-packages/dask/dataframe/utils.py:15: FutureWarning:
pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
INFO:numexpr.utils:Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:numexpr.utils:NumExpr defaulting to 8 threads.
gitignore: settings.json
Default: read from offset to now
Optional: read defined window
Optional: read all messages
settings_loader fungerer ikke på windows (mappeseparator for stier)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.