Giter Club home page Giter Club logo

xmlconvert's Introduction

Package xmlconvert

xmlconvert logo

What xmlconvert does

The xmlconvert package is made to easily and comfortably convert XML data to R dataframes and the other way around. It provides a lot of options to control the conversion process and to export the results to CSV or Excel files, if desired.

How you install xmlconvert

Just execute install.packages("xmlconvert", dependencies = TRUE) in the R console to install the package including all packages it depends on. By default, the xlsx package for writing Excel files is not installed together with xmlconvert. xmlconvert will tell you when you need xlsx and will ask you to install it then.

How you work with xmlconvert

The functions xml_to_df() and df_to_xml() are the two workhorses of the package.

The functions assume that each data record (e.g. a customer) in the XML data is stored in one XML element/tag; the individual fields (e.g. name, address, e-mail), however, can be represented in different ways.

An XML data file could like this:

<datarecord>
  <field1>value1-1</field1>
  <field2>value1-2</field2>
</datarecord>

<datarecord>
  <field1>value2-1</field1>
  <field2>value2-2</field2>
</datarecord>

When working with the xml_to_df() and df_to_xml() functions, we would first of all use the records.tags argument to provide the tag name of the XML element that represents the records, in this example datarecord. Alternatively, the xml_to_df()’s records.xpath argument can be used to supply an XPath expression describing the location of the data records.

In the next step, we need to specify how to find the fields. In the example above, each field within a data record has its own XML element, and the tag name of this element is the field name. In this case, we would use fields = "tags" to make it clear to the xmlconvert functions that the fields are represented by XML elements. Not always will the tag names be the field names. The XML could also look like this:

<datarecord>
  <field name="field1">value1-1</field>
  <field name="field2">value1-2</field>
</datarecord>

<datarecord>
  <field name="field1">value2-1</field>
  <field name="field2">value2-2</field>
</datarecord>

Here, the XML elements representing the fields all have the tag field. The name of the field is not given by the tag name but is an attribute (called name in our case) of the field element. This attribute name can be supplied to the xmlconvert functions using the fields.names argument.

In this case, xml_to_df() could be called like this: xml_to_df("mydata.xml", records.tag = "datarecord", fields = "tags", fields.names = "name")

However, instead of being XML elements, the data fields could also be represented by individual attributes, as in the following example:

<datarecord field1="value1-1" field2="value1-2" />
<datarecord field1="value2-1" field2="value2-2" />

In this case, we use fields = "attributes" instead of fields = "tags" to adjust to this XML data structure. Here, a call of xml_to_df() could look like this: xml_to_df("mydata.xml", records.tag = "datarecord", fields = "attributes")

The xmlconvert functions offer a lot of options to specify which data fields are to be used, how to deal with missing values, how to treat different data types and how to export the results. Please consult the online help by executing ?xml_to_df in the R console for more information on these options.

Contact the author

Joachim Zuckarelli

Twitter: [@jsugarelli](https://twitter.com/jsugarelli)

GitHub: https://github.com/jsugarelli/xmlconvert

xmlconvert's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

xmlconvert's Issues

xml_to_df ends up ignoring encoding

I am trying to open a xml file encoded in ISO-8859-1 (aka latin-1) using xmlconvert, yet even if I specify xml_encoding it still claims my input isn't proper UTF-8. My call and traceback are as follows:

> carbu_df <- xmlconvert::xml_to_df("./PrixCarburants_instantane.xml",
+                                   xml.encoding = "latin-1",
+                                   records.xpath = "//pdv | //prix",
+                                   fields = "attributes")
Error in read_xml.raw(charToRaw(enc2utf8(x)), "UTF-8", ..., as_html = as_html,  : 
  Input is not proper UTF-8, indicate encoding !
Bytes: 0xE8 0x73 0x2D 0x4C [9]
> traceback()
4: read_xml.raw(charToRaw(enc2utf8(x)), "UTF-8", ..., as_html = as_html, 
       options = options)
3: read_xml.character(text, encoding = xml.encoding)
2: xml2::read_xml(text, encoding = xml.encoding)
1: xmlconvert::xml_to_df("./PrixCarburants_instantane.xml", xml.encoding = "latin-1", 
       records.xpath = "//pdv | //prix", fields = "attributes")

Loading the file using xml2::read_xml("./PrixCarburants_instantane.xml", encoding="latin-1") does work, and so does opening the file using Notepad and saving it as UTF-8 (which is a bit tedious). It appears to me that enc2utf8 and charToRaw somehow isn't doing its job when being confronted with direct latin-1 input.

My dataset can be found here

Parsing large file with various tags

I tried the following command but got the error message Error in recs[[i]] : subscript out of bounds. I couldn't see right away where it went wrong, but I'm sure it's almost trivial.

test_df <- xmlconvert::xml_to_df("https://www.sec.gov/Archives/edgar/data/1477336/000176379920000063/ex102nov2020aart20191_1.xml", 
                                 records.tags = "assets",
                                 fields = "tags")

Thanks for your work!

Reading XML with both attributes and fields elements

Hi,

I am trying to use xmlconvert function xml_to_df to read an XML file that contains properties in both attributes and XML own elements. A simplified example :

<xml>
  <country id="US" name="United States of America">
    <property name="currency" value="USD"/>
  </country>
</xml>

I manage to recover the properties (id, name) for each country, but not to list in addition currency.

To further complete my example, I also have nested (hierarchical) XML elements which I am not interested to extract (but they seem to interfere with the extraction. Completing my previous example :

<xml>
  <country id="US" name="United States of America">
    <property name="currency" value="USD"/>
    <city id ="NYC" name = "New York">
      <property name="area" value="121260"/>
      <property name="population" value="8175133"/>
    </city>
  </country>
</xml>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.