Should document with parameters what args are required or optional and then pass argparse results into those functions. How to handle output?

Add line numbers (or "primary key") to a csv. Supplants "-l" option to csvcut. Useful for piping to unix utils or to csvsql.

(csvkit)sk@scarlet:~/src/csvkit$ ./csvclean -n examples/bad.csv
Line 3: Expected 3 columns, found 4 columns
Line 4: Expected 3 columns, found 2 columns
No errors.

Convert the king of tabular data formats into the king of hierarchical data formats for serialization across the wire.

Useful for datasets like this (see Appointment column): http://data.cityofchicago.org/Government/Public-Technology-Resources/nen3-vcxj

"value: count" (R-style)

Instead of a random sample. This is more useful.

Either convert input csv to .rdata format or open an R session with data already loaded into a local dataframe.

Should probably coerce to string in the case of differing types... could also be an error case.

Head, tail, etc. don't work correctly with embedded newlines

Related to #16. No need for a 7 MB zip file in everyone's checkout.

Cartesian product!

Should behave as SQL does.

Note: calling this one a feature and not a bug since it does actually "work" fine and the error is intelligible.

Counts the number of rows in a csv file (wc -l does not work with embedded newlines)

Stack the rows from multiple csv files on top of one other, optionally adding a common grouping value to each set of rows.

Should be able to do:

csvsql --insert test.csv | psql

To create a table and load data.

This is very difficult as sqlalchemy dialects only generate parameterized insert statements. Would need to either substitute actual values in (different for each dialect) or find another way of creating insert statements. (Or not use sqlalchemy.)

Strip duplicate rows.

As with csvsort, to make up for the fact that internal linebreaks mess with the unix utils.

Sort entire rows or by columns

To make up for the fact that unix sort chokes on internal linebreaks.

Like unix grep, but allows specifying single columns to filter on (and handles in-row newlines correctly)

Identify rows/cells which are different between two spreadsheets

I'm sketched out by the append-in-for-loops inside a try-catch block.

It doesn't look atomic to me. What if there were an exception midway-through adding a row? come columns which already had a value would have a None also appended, no?

I don't have time to write a test case tonight, so maybe I'm missing something or maybe there's a test for it already and you should just tell me to RTFC.

anyway, it might be nice to abstract an _outer_join method. I was starting down that path, passing in keys and some rules about when to allow for None values. I'm reading this code trying to decide how to handle the one-to-many case.

On OSX I get this when testing out one of the command-line utils:

(csvkit)Derek-Willis-4901:csvkit 200025$ ./in2csv
Traceback (most recent call last):
File "./in2csv", line 7, in
from csvkit import convert
File "/Users/200025/code/csvkit/csvkit/csvkit/convert/init.py", line 6, in
from xlsx import xlsx2csv
ImportError: No module named xlsx

The pip install went fine, but the tests also fail because of the missing library.

Serialize keys using jsonpipe schema:

https://github.com/dvxhouse/jsonpipe

Include -h output on each page.

Produces a CSV file which maximizes Excel compatibility. Uses the lessons learned here:

http://simonwillison.net/2008/Aug/20/excel/

Should be consistent with csvcut.

Strips inline newlines from fields so they the output can be piped through standard unix utilities such as head, tail, sort, grep, uniq, etc. Should work with all tools, but this doesn't deserve its own utility.

Its too hard to keep this value in sync when columns are modified.

def count_rows(self): return max([len(c) for c in self])

Open python shell with CSV data already loaded into a Table or csv reader.

I kind of think csvcut -n should be fast. If inference were not default, you could stop after reading the first row. What do you think to making it only on request, or barring that, having an option that suppresses it?

Its good to be explicit about these things (esp. linebreaks)

Attempted this but no available xlsx library for python gracefully handles Windows, Mac and OpenOffice produced files. openpyxl is the closest, but needs to be patched to support both OO-produced files and Mac date formats.