On recent versions of Ubuntu, the line 10 'elementtree.ElementTree' fails as described at http://fingergeek.blogspot.com/2009/06/python-elementtree-fix-for-ubuntu.html

I changed it to xml.etree.ElementTree on my system to get it to work. Could this be added in as a platform check?

Hi,

I have an xml-file that looks like this

<?xml version="1.0" standalone="yes"?>
<DATAPACKET Version="2.0">
    <METADATA>
        <FIELDS>
            <FIELD attrname="test_attribute" fieldtype="string" WIDTH="6" />
        </FIELDS>
        <PARAMS />
    </METADATA>
    <ROWDATA>
        <ROW test_attribute="value" />
    </ROWDATA>
</DATAPACKET>

Can one adjust your code to apply it to such a file?

Best,

Henning

In my xml, I have some nodes in the chosen tag that have no values, I would like to skip them:

A typical node is like this

<source>Yahoo Business</source><url>http://us.rd.yahoo.com/dailynews/rss/business/*http://story.news.yahoo.com/news?tmpl=story2 u=/nm/20040814/bs_nm/column_stocks_week_dc</url><title>Wall St. Pullback Reflects Tech Blowout (Reuters)</title><image>none</image><category>Business</category><description>Reuters - Wall Street's long-playing drama,\"Waiting for Google," is about to reach its final act, but its\stock market debut is ending up as more of a nostalgia event\than the catalyst for a new era.</description><rank>5</rank><pubdate>0000-00-00 00:00:00</pubdate>

and for some of them, I have that the category is "" or it has another value to skip. When I convert to csv doing like

xml2csv --input data/newsspace200.xml --output data/newsspace200_.csv --delimiter $'\t' --tag 'source'

I will get the rows with "" category columni.n the csv file. Is it possibile to skip then those columns?

Excellent package, Kailash. Thank you for sharing it. I'm trying to convert the massive 40GB wikipedia dumps from XML to JSON to use in another program, and most converters I've tried are erroring out.

When I try xml2json, however, I find that it is only converting a single record, so I'm not sure what I'm doing wrong. So, for example, I run...

xml2json --input "enwiki-latest-pages-articles.xml" --output "wiki.json" &

And it only outputs the first JSON record. How would I iterate through all the items in <page>brackets</page> and append them to a single file, either from the command line, or from within a python script (that I can run overnight)?

They're listed as...

<page>
  <field1></field1>
  <field2></field2>
</page>
<page>
  <field1></field1>
  <field2></field2>
</page>
<page>
  <field1></field1>
  <field2></field2>
</page>

When I enter the command prompt as modelled in the Documentation ----->

(xmlutils.xml2csv
from xmlutils.xml2csv import xml2csv
converter = xml2csv("samples/fruits.xml", "samples/fruits.csv", encoding="utf-8")
converter.convert(tag="item"))

I get the following message. The details of the error are below. Please advise on a solution.

$ python
Python 3.4.3 |Anaconda 2.2.0 (64-bit)| (default, Mar 6 2015, 12:06:10) [MSC v.1
600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

from xmlutils.xml2csv import xmlcsv
Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'xmlcsv'
from xmlutils.xml2csv import xml2csv
converter = xml2csv("samples/fruits.xml", "samples/fruits.csv" , encoding =
"utf-8")
coverter.convert(tag = "item")
Traceback (most recent call last):
File "", line 1, in
NameError: name 'coverter' is not defined
converter.convert(tag = "item")
Traceback (most recent call last):
File "", line 1, in
File "c\Python Folder\xmlutils.py\xmlutils\xml2csv.py",
line 56, in convert
event, root = self.context.next()
AttributeError: '_IterParseIterator' object has no attribute 'next'
converter.convert(tag = "item")
Traceback (most recent call last):
File "", line 1, in
File "c\Python Folder\xmlutils.py\xmlutils\xml2csv.py",
line 56, in convert
# event, root = self.context.next()
AttributeError: '_IterParseIterator' object has no attribute 'next'
converter.convert(tag = "item")
Traceback (most recent call last):
File "", line 1, in
File "c\Python Folder\xmlutils.py\xmlutils\xml2csv.py",
line 56, in convert
# event, root = self.context.next()
AttributeError: '_IterParseIterator' object has no attribute 'next'

A XML node will have fix number of elements and they can be in order from one node to another node. If during conversion a fixed order is not used then the output csv will have mis-matched values. if XML don't have XSD then well formed XML can have elements in any order among different node. During conversion, we can control the output order of elements and hence the csv will write values under header fields. for example, below example have same elements but order are different but in a output csv value must match with appropriate header field.

<files>

<file name="com.nineleaf.tp-1.apk" source="original">
     <format>Android Package Archive</format>
     <mtime>1407440118</mtime>
      <size>27723910</size>
      <md5>2221c4f14bd4178380df53428bc83cd6</md5>
    <crc32>7dedaf45</crc32>
     <sha1>cbb5ecbf3477ad0def0264bf294fbf22271ca313</sha1>
</file>

<file name="com.striker.lalosfeliz-6.apk" source="original">
        <mtime>1413362341</mtime>
       <size>2705561</size>
       <md5>53033d768f5fbc8ec2913d42fce7c9e2</md5>
        <crc32>e6f36b15</crc32>
       <sha1>ae9c4bc0730f67cba372f09c03da27fc954fa5e6</sha1>
       <format>Android Package Archive</format>
</file>
</files>

Hi Kailash, Please Push the latest changes that includes support for python 3.x for xml2csv script, as the package available in pypi.org doesn't handle the attribute error.

Thanks!

The documentation on README keeps getting out of sync with the documentation provided by the utility itself. E.g., Issue #18 highlights a problem with the --header option on xml2csv, Here's the relevant documentation as spat out by xml2csv -h:

--header              print csv header (default=True)

and that contained in the README:

--header    Whether to print the CSV header (list of fields) in the first line; 1=yes, 0=no. Default is 1.

I would argue that the documentation spat out by the command line tool is always going to be correct. Either we should continuously rework the online (README) documentation (the preferred option) or drop it in favour of something along the lines of:

xml2csv

Basic usage:

xml2csv --input INPUT_FILE --output OUTPUT_FILE --tag TAG

More detailed usage documentation can be accessed by running xml2csv --help.

Currently the header line is prefixed with the # character, e.g.,

#id,name,price,quantity

Is the prefix strictly-speaking necessary? The closest I can find in a spec for CSV is the RFC 4180, in particular Section 2.3, which might suggest a header line of:

id,name,price,quantity

It's a bit of a niggle maybe, but would it be reasonable to remove the prefix or alternatively add a boolean argument (defaulting to false) to tell xml2csv not to include it? (I'm happy to submit a pull-request if there is general support for either of these suggestions).

I have a script that loops through different high school sports and grabs an xml schedule for each one.

https://gist.github.com/dangayle/6127265

If I use the function to grab a single sport, it works perfectly. If I loop through a simple dict of sports, it creates a new csv file and adds all of the sports data from the preceding loops onto the csv. (You'll want to run the code to see what I mean).

I don't have any idea why it's not working, since Python's garbage collector should be destroying the data from the prior loops.

Thoughts?

Hi,

I tried using
--delimiter "\t"
for specifying TAB as the delimiter but that didnt work. Tried escaping also
"\\t"
but it didn't work. Resulted CSV prints \t as text and TAB character is not created.

Tried converting simple XML (below) but get nothing in the output file. Output file is created though. I am using Python v2.7

<?xml version="1.0" encoding="utf-8"?>
<rows>
  <row Id="2" PostId="35314"/> 
</rows>

Tried command line tool which didn't work
xml2csv --input "subset.xml" --output "subset.csv" --tag "row"

Error

Tried it through the code which didn't work either

from xmlutils.xml2csv import xml2csv
converter = xml2csv("subset.xml", "subset.csv", encoding="utf-8")
converter.convert(tag="row")

When I run xmltable2csv I get the following error message"

File "/usr/lib/python3.4/site-packages/xmlutils/console.py", line 20
"""
^
SyntaxError: invalid syntax

The offending code is the triple-double quoted string literal as follows

def run_xml2sql():
print """xml2sql by Kailash Nadh (http://nadh.in)
--help for help

"""

What am I doing wrong here? My python is version 3.4, as you can see. I feel I must be making some basic compatibility mistake or some such, as this can not possibly be a real syntax errror, can it?

It would be helpful to support --limit 0. This would allow the user to request zero records output, and consequently the user could use this option to get the column headers, and only the column headers.

Currently --limit 0 behaves identically to --limit -1 (the default) which outputs all records. This behavior (outputting all records when limit is zero) is actually a by-product of another minor bug, which is the following test:

if n == limit:
    break

This test happens after n has been incremented and consequently will never break when limit is zero.

This is a very useful tool for an XML file containing homogenous records. However, for XML files that contain various types of records, parsing all the record types requires scanning the file multiple times. For small XML files this is trivial but for larger GB sized XML files this is very slow.

Leveraging what you already have built, could you add a convert_multiple() method that would accept a list of tags and use the output path as the base path with the tag name appended. Something like this:

converter = xml2csv('some/path/my_xml.xml', 'some/path/my_csv_', encoding='utf8')
converter.convert_multiple(tags=('tag1', 'tag2', 'tag3'))

Resulting in output files like this:

my_csv_tag1.csv
my_csv_tag2.csv
my_csv_tag3.csv

line 59 of xml2csv:

if started and event == 'start' ....

checks for 'start' events. this does not guarantee that the element is actually read in its entirety, and can result in None type elem.text, which causes blank fields in the ouptut.

see,
http://infix.se/2009/05/10/text-safe-xml-processing-with-iterparse

this happened in practice for me. I believe building on 'end' events fixes this problem, i.e.:

if started and event == 'end' ....

with everything else the same.

Rather than assuming the xml tag structure, the script needs to essentially scan the xml file BEFORE outputting the contents into CSV so that the column headers can be created first. This will ensure that all tags within the CSV file are accounted for. I have some files you can use for testing this if needs be.

If an element is missing in the source XML, then the corresponding cell is erroneously omitted from the resulting CSV. This causes values in the resulting CSV to appear in the wrong columns.

This command:

xml2csv --input in.xml --output out.csv --tag "row"

On this input file, in.xml:

<?xml version="1.0"?>
<top>
  <row>
    <a>r1_col_a</a>
    <b>r1_col_b</b>
  </row>
  <row>
    <b>r2_col_b</b>
  </row>
</top>

Produced this output file, out.csv:

a,b
"r1_col_a","r1_col_b"
"r2_col_b"

Notice that r2_col_b is in column a, when it should be in column b.

Refer to example XML below. Those elements under App_Data App="SVOD" are mostly missing when converted using xmlutils.py

`

The default separator set for xml2csv in console.py is ", " (a comma and a space). Perhaps it should be "," (just a comma)?
In this way we get files with lines like:

#id,name,price,quantity
"1000","Apple","4","133"

rather than

#id, name, price, quantity
"1000", "Apple", "4", "133"

by default. I'm submitting a pull request now, if this is worth changing.
Great work on xmlutils :-)

Example:
<root> <c> <field attr="a">hello</field> <field attr="b">hello</field> </c> </root>
is converted to:
INSERT INTO c (field,field)
VALUES
("hello", "hello");

Proposed fix attached (xml2sql.py)

fix.txt

I'm trying to convert the data from this resource: http://www.drugbank.ca/downloads and only seem to be getting the first record (DB00001, "Lepirudin") in the conversion. However, if I use DrungBank's on-line browser to search on "Lepirudin" it finds it but throws an error when trying to view the record so this may be an XML and not xml2json problem.

For example if a XML file have below node,
<file name="com.nineleaf.tp-1.apk" source="original"> <format>Android Package Archive</format> <mtime>1407440118</mtime> <size>27723910</size> <md5>2221c4f14bd4178380df53428bc83cd6</md5> <crc32>7dedaf45</crc32> <sha1>cbb5ecbf3477ad0def0264bf294fbf22271ca313</sha1> </file>

Now, the resulting csv should have file-name and file-source (node name + attribute name) as a field and these values should go under these fields.

So, resulting csv header should be like.

<file-name, file-source, format, mtime, size, md5, crc32, sha1>

I built the environment successfully, and trying to run the command from directory xmlutils.py
xml2json --input "/sample/logs.xml" --output "/sample/logs.json"

getting this error:

Traceback (most recent call last): File "/home/xalteddev/anaconda3/bin/xml2json", line 33, in <module> sys.exit(load_entry_point('xmlutils==1.4.1', 'console_scripts', 'xml2json')()) File "/home/xalteddev/anaconda3/lib/python3.8/site-packages/xmlutils/console.py", line 109, in run_xml2json parser.add_argument('--input', type=file, dest='input_file', required=True, help='input xml filename') NameError: name 'file' is not defined

i want to converter xml on the fly like

converter = xml2json(message, encoding="utf-8")

I wanted to convert xml from a variable rather than file as explained in the examples.

I wanted a header. Documentation said "yes" was default, but I used --header to make sure and got no header. Then tried --header 1 and that was an error. Finally ran it without the --header and got the result I was looking for. I don't think this result matches the result I was expecting based on the documentation.

My suggestion: two options, --header
--no-header

with the obvious effect applied...

I am a new user of this repository, and I was wondering how I can convert the xml files from the IRS website to csv file and save it.