Package python to remove common ugliness from real-world csv-like file.

When working with lots of CSVs from different companies or even worse

This package is to fix that kind of issues.

As usual, just download it using pip:

pip install csv_trimming

The package is very simple to use, just load your CSV and pass it to the trimmer.

from csv_trimming import CSVTrimmer

# Load your csv
csv = pd.read_csv("path/to/csv.csv")
# Instantiate the trimmer
trimmer = CSVTrimmer()
# And trim it
trimmed_csv = trimmer.trim(csv)
# That's it!

For instance, your input CSV to clean up may look like this at the beginning:

And after the trimming, it will look like this:

Magic!

Sometimes, the CSVs you are working with may have a row correlation, meaning part of a given row is inserted in the next row. Such cases are common when the data-entry clerk wants to make the whole table fit in their screen, and in order to do so, they split the row in two. While this is clearly an extremely bad practice, it happens in the real world and the CSV Trimmer can handle it with a little help.

You just need to provide a function that defines which rows are correlated, and the CSV Trimmer will take care of the rest. While in this example we are using a rather simple function and a relatively clean CSV, the package can handle more complex cases.

def simple_correlation_callback(
    current_row: pd.Series, next_row: pd.Series
) -> Tuple[bool, pd.Series]:
    """Return the correlation between two rows."""

    # All of the rows that have a subsequent correlated row are
    # non-empty, and the subsequent correlated rows are always
    # with the first cell empty.
    if pd.isna(next_row.iloc[0]) and all(pd.notna(current_row)):
        return True, pd.concat(
            [
                current_row,
                pd.Series({"surname": next_row.iloc[-1]}),
            ]
        )

    return False, current_row


trimmer = CSVTrimmer(simple_correlation_callback)
result = trimmer.trim(csv)

In this case, our CSV looked like this at the beginning:

And after the trimming, it will look like this:

Here follow some examples of the package in action.

Sometimes, when chaining multiple CSVs in a poor manner, you may end up with duplicated schemas. The CSV Trimmer detects rows that match the detected header, and it can (optionally) remove them.

from csv_trimming import CSVTrimmer

# Load your csv
csv = pd.read_csv("path/to/csv.csv")
# Instantiate the trimmer
trimmer = CSVTrimmer(drop_duplicated_schema=True)
# And trim it
trimmed_csv = trimmer.trim(csv)
# That's it!

For instance, your input CSV to clean up may look like this at the beginning:

And after the trimming, it will look like this:

Sometimes, the data entry clerk may start filling a table offsetted from the top-left corner, and export it with also empty cells all around. We call such cells "padding". The CSV Trimmer can detect and remove them.

from csv_trimming import CSVTrimmer

# Load your csv
csv = pd.read_csv("path/to/csv.csv")

# Instantiate the trimmer
trimmer = CSVTrimmer(drop_padding=True)

# And trim it
trimmed_csv = trimmer.trim(csv)

For instance, your input CSV to clean up may look like this at the beginning:

And after the trimming, it will look like this:

If you have identified some new corner case that the package does not handle, or you have a suggestion for a new feature, feel free to open an issue. If you want to contribute with code, open an issue describing the feature you intend to add and submit a pull request.

This package is released under MIT license.