Giter Club home page Giter Club logo

beanio's People

Contributors

bigloupe avatar garcia-jj avatar kevinseim avatar nicoschl avatar saxicek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

beanio's Issues

Enable XML mapping file to identify and ignore Byte Order Mark (BOM)

I use the 2.1.0 release of beanio to extract data from .csv files.

If the file comes from Windows, the first three bytes of the file contain a “Byte Order Mark” (in my case ef bb bf) that will crash the beanio reader.

In order to process Windows files I have to first run them through some external program to remove the BOM.

I’ve searched the documentation and unless I’ve missed something it appears that the ability to do this is not available in this release.

XML snippet below:

<stream name=“inputFile" format="csv">
   <parser>
      <property name="delimiter" value="," />
      <!--property name="quote" value="\" /-->
      <property name="comments" value="HeaderLineBegin" />
   </parser>
    …
</stream>

👉👉👉 BeanIO is alive! A fork is available at beanio/beanio 👈👈👈

Hi everyone,

I'm pleased to announce that I started resurrecting BeanIO after several years of silence from its original author.
Someone created a GitHub org a few years ago, but that repo was also inactive. I offered my help, and things have been in motion again for the past week or so.

What's been done so far:

  • I migrated the old SVN repo to a Git repo and applied the initial V3 that was started by Kevin Seim a long time ago
  • I also applied a few patches that I found in various forks on GitHub (including the thread-safety fixes for dates and numbers)
  • I started working on a new version of the website, hosted on GitHub Pages
  • I added a CI build on GitHub Actions. The project is now built with a JDK 11, but is still targeting bytecode version 1.7
  • I set up SonarCloud to monitor code quality

So, what's next?
Well, I'm open to suggestions, ideas and contributions. Feel free to reopen your issues and PRs on the new repo.

I think I'll select a few urgent things to fix/implement (like maybe support for java.time?), and start working on a release.

Important point: the Maven coordinates will change from org.beanio:beanio to com.github.beanio:beanio. The rest should be compatible with version 2.1 (including package names).

New repo: https://github.com/beanio/beanio
New website: https://beanio.github.io/
To start a discussion: https://github.com/beanio/beanio/discussions

Release Candidate

I am using BeanIo in a project that needs the ability to validate the length on marshaling fields. This feature was added in the commit c603758 but there hasn't been a formal release since then. Is there a plan to release a new version? I would feel more comfortable using a formal release than using a build based on a specific git commit.

Calin

How BeanIO to fix length with read byte characters (LITTLE ENDIAN)

As I need to marshal and unmarshal message that is fixed length with 2 little endian bytes "2A 00" as below example, how to configure BeanIO mapping file and object in order to can do that?

RAW-DATA:
30 30 30 31 2A 00 0A 00 31 32 - 33 20 20 20 20 20 20 20 14 00 1 [0001* 123 ]
41 41 41 41 41 41 20 20 20 20 - 20 20 20 20 20 20 20 20 20 20 21 [AAAAAA ]
00 00 42 [ ]

Packet Details: Code=0001 len=42
[001: 10]:[123 ]
[002: 20]:[AAAAAA ]
[003: 0]:[]

https://code.google.com/archive/p/beanio/issues/131

hat steps will reproduce the problem? 1. Create multiple Unmarshallers (one per thread) from the same StreamFactory 2. Unmarshall records with a date in each of the threads

What is the expected output? What do you see instead? All records get unmarshalled correctely. Instead, Exceptions get thrown.

What version of BeanIO are you using? What JDK version? BeanIO 2.1.0

Please provide any additional information below. While the documentation claims that "All included BeanIO type handlers are thread safe", DateTypeHandlerSupport potentially uses the same SimpleDateFormat for multiple Unmarshaller instances. Since SimpleDateFormat is not thread-safe this causes the behaviour as shown. Switching to a custom TypeHandler that uses a thread-local SimpleDateFormat solves the problem.

Comment #1

Posted on Apr 27, 2015 by Helpful Camel
Same behaviour when using beanIO in a camel route.

Strange exceptions that trace back to SimpleDateFormat not being thread safe.

Problema with record ID

If I have a fixed length records like this

Xmessage
XDother message
XDanother message

Where X, XD and XR are the Ids, BeanIO cannot identify the record type correctly.

PGP encryption with beanio

BeanIO is used to generate the formatted files in one of our projects @ our org.
It is well-versed and helped to generate the all type of specifications such as .csv, delimited, fixed-length..etc

As a payment company, we have to stick with PCI compliance, the files should not be readable by anyone immediately, because it may contain Personal Identifiable Information. BeanIO is enhanced to generate the encrypted file. Encryption is based on PGP; bouncy-castle Open Source APIs are used to generated the PGP encrypted content.

We would like to contribute this commits into BeanIO framework. So that it will be available to public, if anyone wants to generated the PGP content, then it will be quite useful.

As part of this we have added few new classes and modified few classes. We would like to know if we can commit our changes and if yes, please let us know if there is any process to be followed

Using marshaller / unmarshaller are very buggy

I'm running into all kinds of bugs, NullPointers, ClassCast exceptions inside the framework, validation errors. I'm not able to get the following to work at all, although I'm already working around several bugs.

Following are the following components: annotated record class, fixed width JMS message converter config + typehandlers, and junit test class. The JMS parts can be removed.

I don't think it should be very difficult: map between fixed width string to annotated class and back. I couldn't find any junit tests also in your code base that touches this.

Annotated Record class

import org.beanio.annotation.Field;
import org.beanio.annotation.Record;

import java.math.BigDecimal;
import java.util.Date;

import static org.beanio.builder.Align.LEFT;
import static org.beanio.builder.Align.RIGHT;

@Record(minOccurs = 1)
public class InputVariablesDto {

    // INVOER-SEGMENT

    @Field(at = 0, length = 3, align = RIGHT, padding = '0')
    private Integer renteBerichtCode;

    @Field(at = 1, length = 8, format = "yyyyMMdd")
    private Date renteBerichtDat;

    @Field(at = 2, length = 3, align = RIGHT, padding = '0')
    private Integer renteVastPeriode;

    @Field(at = 3, length = 1, handlerName = "JaNeeTypeHandler")
    private Boolean nhgInd;

    @Field(at = 4, length = 12, align = RIGHT, handlerName = "LisDecimalTypeHandler", padding = '0')
    private BigDecimal marktwaardeFactor;

    // UITVOER-SEGMENT

    @Field(at = 5, length = 4, align = RIGHT, padding = '0')
    private String foutKode;

    @Field(at = 6, length = 120, align = LEFT, padding = ' ')
    private String foutOms;

    @Field(at = 7, length = 12, align = RIGHT, padding = '0', handlerName = "LisDecimalTypeHandler")
    private BigDecimal actHypRente;

    public Integer getRenteBerichtCode() {
        return renteBerichtCode;
    }

    public void setRenteBerichtCode(Integer renteBerichtCode) {
        this.renteBerichtCode = renteBerichtCode;
    }

    public Date getRenteBerichtDat() {
        return renteBerichtDat;
    }

    public void setRenteBerichtDat(Date renteBerichtDat) {
        this.renteBerichtDat = renteBerichtDat;
    }

    public Integer getRenteVastPeriode() {
        return renteVastPeriode;
    }

    public void setRenteVastPeriode(Integer renteVastPeriode) {
        this.renteVastPeriode = renteVastPeriode;
    }

    public Boolean getNhgInd() {
        return nhgInd;
    }

    public void setNhgInd(Boolean nhgInd) {
        this.nhgInd = nhgInd;
    }

    public BigDecimal getMarktwaardeFactor() {
        return marktwaardeFactor;
    }

    public void setMarktwaardeFactor(BigDecimal marktwaardeFactor) {
        this.marktwaardeFactor = marktwaardeFactor;
    }

    public String getFoutKode() {
        return foutKode;
    }

    public void setFoutKode(String foutKode) {
        this.foutKode = foutKode;
    }

    public String getFoutOms() {
        return foutOms;
    }

    public void setFoutOms(String foutOms) {
        this.foutOms = foutOms;
    }

    public BigDecimal getActHypRente() {
        return actHypRente;
    }

    public void setActHypRente(BigDecimal actHypRente) {
        this.actHypRente = actHypRente;
    }
}

Fixed width JMS message converter config and typehandlers

import InputVariablesDto;
import org.beanio.Marshaller;
import org.beanio.StreamFactory;
import org.beanio.Unmarshaller;
import org.beanio.builder.FixedLengthParserBuilder;
import org.beanio.builder.StreamBuilder;
import org.beanio.internal.parser.MarshallerImpl;
import org.beanio.types.TypeConversionException;
import org.beanio.types.TypeHandler;
import org.springframework.jms.support.converter.MessageConversionException;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.Session;
import javax.jms.TextMessage;
import java.math.BigDecimal;

import static org.apache.commons.lang.StringUtils.*;

public class FixedWithMessageConverter implements org.springframework.jms.support.converter.MessageConverter {

    private static final Marshaller marshaller;
    private static final Unmarshaller unmarshaller;

    static {
        StreamFactory factory = StreamFactory.newInstance();
        factory.define(new StreamBuilder("inputVariables")
                .format("fixedlength")
                .parser(new FixedLengthParserBuilder())
                .addTypeHandler("JaNeeTypeHandler", new JaNeeTypeHandler())
                .addTypeHandler("LisDecimalTypeHandler", new LisDecimalTypeHandler())
                .addRecord(InputVariablesDto.class));
        unmarshaller = factory.createUnmarshaller("inputVariables");
        marshaller = factory.createMarshaller("inputVariables");
    }

    @Override
    public Message toMessage(Object object, Session session) throws JMSException, MessageConversionException {
        if (!(object instanceof InputVariablesDto)) {
            throw new IllegalArgumentException("expected InputVariablesDto but was: " + object.getClass());
        }
        String[] strings = ((MarshallerImpl) marshaller.marshal(object)).toArray();
        return session.createTextMessage(join(strings));
    }

    @Override
    public Object fromMessage(Message message) throws JMSException, MessageConversionException {
        if (!(message instanceof TextMessage)) {
            throw new IllegalArgumentException("unknown message type: " + message.getClass());
        }
        return unmarshaller.unmarshal(((TextMessage) message).getText());
    }

    /**
     * Domain: J = With NHG (true), N = Without NHG (false)
     */
    private static class JaNeeTypeHandler implements TypeHandler {
        @Override
        public Object parse(String text) throws TypeConversionException {
            return "J".equals(text);
        }

        @Override
        public String format(Object value) {
            return ((Boolean) value) ? "J" : "N";
        }

        @Override
        public Class<?> getType() {
            return Boolean.class;
        }
    }

    /**
     * Format: ‘+00000,00000’ Example: ‘+00000,90000’
     */
    private static class LisDecimalTypeHandler implements TypeHandler {
        @Override
        public Object parse(String text) throws TypeConversionException {
            return new BigDecimal(text.replace(',', '.'));
        }

        @Override
        public String format(Object value) {
            BigDecimal bigDecimal = (BigDecimal) value;
            bigDecimal.setScale(5);
            String signum = bigDecimal.signum() == -1 ? "" : "+";
            String strValue = bigDecimal.toPlainString().replace('.', ',');
            String mainValue = strValue.substring(0, strValue.indexOf(','));
            String fractionValue = strValue.substring(strValue.indexOf(',') + 1, strValue.length());
            return String.format("%s%s,%s", signum, leftPad(mainValue, 5, '0'), rightPad(fractionValue, 5, '0'));
        }

        @Override
        public Class<?> getType() {
            return BigDecimal.class;
        }
    }
}

Junit test

import InputVariablesDto;
import org.joda.time.DateTime;
import org.junit.Test;
import org.mockito.ArgumentCaptor;

import javax.jms.JMSException;
import javax.jms.Session;
import javax.jms.TextMessage;
import java.math.BigDecimal;

import static org.assertj.core.api.Assertions.assertThat;
import static org.joda.time.format.DateTimeFormat.forPattern;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

public class FixedWithMessageConverterTest {

    private FixedWithMessageConverter converter = new FixedWithMessageConverter();

    private final DateTime VANDAAG = DateTime.now();

    @Test
    public void toMessage() throws JMSException {
        Session mockedSession = mock(Session.class);
        ArgumentCaptor<String> captor = ArgumentCaptor.forClass(String.class);
        when(mockedSession.createTextMessage(captor.capture())).thenReturn(null);
        converter.toMessage(createListObject(), mockedSession);

        String datumStr = forPattern("yyyyMMdd").print(VANDAAG);
        String oms = "omschrijving                                                                                                            ";
        assertThat(captor.getValue()).isEqualTo("001" + datumStr + "044" + "J" + "+00000,90000" + "0001" + oms + "+00005,75000");
    }

    @Test
    public void fromMessage() throws JMSException {
        String fixedWidthMsg = "00120160615044J+00000,900000001omschrijving                                                                                                            +00005,75000";
        TextMessage tm = mock(TextMessage.class);
        when(tm.getText()).thenReturn(fixedWidthMsg);
        String result = (String) converter.fromMessage(tm);
        assertThat(result).isNotEqualTo(createListObject());
    }

    private InputVariablesDto createListObject() {
        InputVariablesDto dto = new InputVariablesDto();
        dto.setRenteBerichtCode(1);
        dto.setRenteBerichtDat(VANDAAG.toDate());
        dto.setRenteVastPeriode(44);
        dto.setNhgInd(true);
        dto.setMarktwaardeFactor(new BigDecimal("0.9"));
        dto.setFoutKode("1");
        dto.setFoutOms("omschrijving");
        dto.setActHypRente(new BigDecimal("5.75"));
        return dto;
    }
}

Is this project still active?

At this time I see some important issues and pull requests waiting approval like thread-safe issues that can cause bug in several projects that uses this library, like #11 #14 .

Is this project still active? Contributions like this are welcome?

BeanIo can't read special caracters

I have a .txt file with ANSI (windows-1252) Encoding i'am able to read it on windows but not on unix.

here is the xml mapping file :

  <beanio>
    <stream name="empData" format="csv">
        <parser>
            <property name="delimiter" value=";"/>
            <property name="alwaysQuote" value="false"/>
            <!--<property name="quote" value='' />-->
        </parser>    

            <record name="emp" class="com.MyClass" >
            <field name="name" />
            <field name="job" />
            <field name="adress"/>  
       </record>
    </stream>
</beanio>

Java Side :

StreamFactory factory = StreamFactory.newInstance();

    InputStream in = this.getClass().getClassLoader()
            .getResourceAsStream("mapping.xml");

    Reader reader = new InputStreamReader(this.getClass().getClassLoader()
            .getResourceAsStream("countries.txt"));
    factory.load(in);

    BeanReader beanReader = factory.createReader("empData", reader);
    Gson gson = new Gson();
    /*Object bean =new Object();*/
    Object record = null;
    while ((record = beanReader.read()) != null) {
        System.out.println(beanReader.getRecordName() + ": "
                +((MyClass)record).getCountry());
    }

Result :
line : France

line : S??o Paulo should be (São Paulo) windows OK but unix is KO

line : USA

line : China

Any idea ?

FYI : i already tried to set Charset to UTF-8 java side .

new InputStreamReader(this.getClass().getClassLoader().getResourceAsStream("countries.txt"), Charset.forName("UTF-8"));

SQL Date type handler

Hey there,
There is some inconsistency with the Date objects currently, specially in the persistence part.
Imagine the scenario:

  • Reading a file and parsing a field to a java.util.Date object.
  • Insert the date into the database. (HERE IS THE PROBLEM)

The prepare statement object only accepts java.sql.Date and not the java.util.Date, shouldn't BeanIO had a SQLDateTypeHandler class to handle sql.Date classes? Could be really handy...

Ignore property by regex

In beanio we have a property regex that we can add for field validation, which causes to throw an Exception if is not valid...

We could add a new property, for example, regexIgnore or something like that in order to avoid the exception and read only the lines that respect the predicate of the regex.

Relative position (at = x)

When you use fixed length stream you need to calculate the at position based in the length of the and the position of the previous record.

I suggest to have a new annotation property called relativeAt or something like that to calculate the true position in the beanIO implementation it self.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.