Comments (8)
good (terrible) example: https://www.ssa.gov/foia/html/FY08CSV.csv
from cultivar.
another great one: http://www.planecrashinfo.com/1920/1920.htm
from cultivar.
another example: https://catalog.data.gov/dataset/veterans-health-administration-2008-hospital-report-card-patient-satisfaction
from cultivar.
Great examples - I definitely noticed all the error messages that came up as you were experimenting! The actual error seems to be a Unicode decoding error, which is actually potentially more serious. It leads to the question of whether or not these files are actually unicode encoded or if they have some other scheme (making things way more difficult).
from cultivar.
Hmm, sounds like it's potentially related to my #43 then?
from cultivar.
Potentially, though encoding detection is more of an annoying task that's tough to figure out. You could use the file
command in your terminal to see if your computer knows the encoding. It's definitely something I'll take a look at.
from cultivar.
Found some more test data that might help with this issue, see: https://github.com/okfn/messytables/tree/7e4f12abef257a4d70a8020e0d024df6fbb02976/horror
from cultivar.
Ok, so specifically for the files that @rebeccabilbro first linked, in terms of unicode decode errors, this problem has been solved, assumedly from the python 3.x upgrade in which the default python encoding is utf-8 instead of ascii so these utf-8 (or utf-8 subset) encoded files no longer caused unicode errors. Specifically I today tested the CSVs from https://www.ssa.gov/foia/html/FY08CSV.csv and https://catalog.data.gov/dataset/veterans-health-administration-2008-hospital-report-card-patient-satisfaction, and the HTML from http://www.planecrashinfo.com/1920/1920.htm, with both file storage and S3 backend and none of them caused an error.
How we want to deal with files in encodings that are not utf-8 encoded is a much broader question. For example a utf-16le encoded file (i.e. https://github.com/okfn/messytables/blob/7e4f12abef257a4d70a8020e0d024df6fbb02976/horror/utf-16le_encoded.csv) won't work right now since utf-16le isn't a subset of utf-8, but I'm not sure yet if we really care. IMHO, it was unreasonable to expect ascii encoding, but is not unreasonable to expect utf-8/utf-8-subset encoding.
So, given all of that I am going to close this issue in terms of the scope of the initial bug. However I will make a note on the roadmap issue for consideration more generally about how we want to deal with non-utf-8-subset encodings in this project for the future. cc @rebeccabilbro @ojedatony1616 @bbengfort @looselycoupled
from cultivar.
Related Issues (20)
- Better Licensing HOT 2
- Citation HOT 2
- Create Trinket Python Library
- Download Bundle HOT 7
- Update readme/docs to document use of ENV variable for database settings HOT 3
- Update/ Edit Dataset Info 400 HOT 2
- star a dataset HOT 2
- Interactive slider for hyperparameter tuning HOT 1
- Handling various data types and models. HOT 1
- Project Codename HOT 17
- Resolve missing migration for starred datasets HOT 1
- Set up demo site on AWS HOT 3
- Examine existing documentation & develop roadmap HOT 1
- Update Celery to 4.0.1 (or latest at the time)
- OAUTH2 signup/login is erroring HOT 1
- Make use of local storage (vs S3) an environmental variable in .env HOT 1
- Remove old requirements.txt file HOT 4
- Dataset Bundle Error when no README
- Uploading a file that exists in another dataset produces a duplicate file error
- Update dev files to make Vagrant and Docker startup work HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cultivar.