Comments (4)
Hi @esjewett just flagging that I am available for any questions, both here and in our Frictionless Data chat: https://gitter.im/frictionlessdata/chat 👍
from breve.
I think we'd certainly be interested in adding support to these formats to Palladio's capabilities and we have looked at them before, but there are a few questions. It's worth noting that Breve uses Palladio's data processing engine internally and doesn't expose all of Palladio's capabilities. If you can provide some insight into these issues, that would be helpful:
- Is there a single-file format available? The ability for users to port data using a single file is fairly integral to the approach of Palladio and Breve, not least because the ability to download the file from the browser client-side is limited to one file at a time.
- How much flexibility is there in the types? Here are some examples of where the JSON Table Schema types and the Palladio types don't agree, as far as I recall:
- No URL/URI type in JSON Table Schema
- Palladio supports heterogenous data columns (e.g. a column with both 1234-12-12 YYYY-MM-DD and 1234 YYYY formats) and we hope to support even more in the future. The goal is to be able to support different types of fuzzy dates and durations.
- Breve supports ordinal/nominal indicators and such a concept doesn't seem to exist in JSON data table.
- The Palladio format stores multiple tables in a single data file along with information about the user-defined mapping/join between the tables. Would this be possible with the Data Packages format?
- Related to 2 above, the Palladio format stores further user-supplied information about dimensions. Does the format allow storing this type of additional information. A couple of examples:
- If the user has selected the dimension as displayed or not.
- If the dimension uses multi-value delimiters internally
I don't mean to imply that the Palladio format is some sort of superior format. It is really just based on Palladio's internal representation. If we could switch to a format based on Data Packages, that would probably be ideal, but as a research project we also need to maintain the ability to be flexible and expressive in areas where Data Packages has made the sort of decisions to limit expression that make perfect sense in a standard format.
Thanks!
from breve.
Hi Ethan, thanks for your quick, thoughtful, and thorough response!
-
Data Packages can support a single-file use case similar to Palladio's export format. A given "resource" in the
resources
array (equivalent to a "file" in the Palladio export JSONfiles
array) in a Data Package can have one of "url", "path", or "data"; the "data" attribute can be used to store in-line data in a JSON array exactly equivalent to the "data" attribute in Palladio's export format. -
Types
-
Each field type in JSON Table Schema has a set of
format
options. For strings, this does, in fact, include auri
format: http://specs.frictionlessdata.io/json-table-schema/#string -
For a field type of
date
(ortime
ordatetime
), there actually is aformat
option "any" which is specified like so:any: Any parsable representation of the type. The implementing library can attempt to parse the datetime via a range of strategies. An example is dateutil.parser.parse from the python-dateutils library.
I'm not sure if this fully supports your use case, so let me know.
-
No explicit support for "ordinal" or "nominal" indicators on a field.
-
-
The Tabular Data Package format supports multiple tables and user-defined relations between them. See this section for details:
http://specs.frictionlessdata.io/json-table-schema/#foreign-keys
-
The Data Package format does allow for extra fields.
NOTE: A Data Package author MAY add any number of additional fields beyond those listed in the specification here.
http://specs.frictionlessdata.io/data-packages/#optional-fields
cc'ing @rgrp @pwalsh as they might have further thoughts on the above
from breve.
Hi Dan,
This is really encouraging. Given all this, I think it may be possible to simply move Palladio's data format to Data Packages, which would solve this problem for Palladio as well as Breve. It's going to a process, but I'll probably start prototyping in a branch of the Palladio repo soon.
Thanks,
Ethan
from breve.
Related Issues (15)
- Beginning of Breve design doc added HOT 1
- Edit dimension verify special chars does not filter
- Add metrics download
- In dimension edit view, values do not appear in the list
- Once a data type has been specified, that should apply to all values in the dimension
- Long dimensions push symbols out of the picture
- special char pills HOT 3
- Edit Dimension "Sort" field offers search, too
- Sort by value/frequency is broken again
- Feedback link?
- "Tweet @hdstanford"
- Use grey background from dimension for the full row
- Fix situation in the "About" box
- Default (opening view) should be gaps & errors HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from breve.