kevinseim / beanio Goto Github PK
View Code? Open in Web Editor NEWA Java library for marshalling and unmarshalling bean objects from XML, CSV, delimited and fixed length stream formats.
License: Apache License 2.0
A Java library for marshalling and unmarshalling bean objects from XML, CSV, delimited and fixed length stream formats.
License: Apache License 2.0
I use the 2.1.0 release of beanio to extract data from .csv files.
If the file comes from Windows, the first three bytes of the file contain a “Byte Order Mark” (in my case ef bb bf) that will crash the beanio reader.
In order to process Windows files I have to first run them through some external program to remove the BOM.
I’ve searched the documentation and unless I’ve missed something it appears that the ability to do this is not available in this release.
XML snippet below:
<stream name=“inputFile" format="csv">
<parser>
<property name="delimiter" value="," />
<!--property name="quote" value="\" /-->
<property name="comments" value="HeaderLineBegin" />
</parser>
…
</stream>
Hi everyone,
I'm pleased to announce that I started resurrecting BeanIO after several years of silence from its original author.
Someone created a GitHub org a few years ago, but that repo was also inactive. I offered my help, and things have been in motion again for the past week or so.
What's been done so far:
So, what's next?
Well, I'm open to suggestions, ideas and contributions. Feel free to reopen your issues and PRs on the new repo.
I think I'll select a few urgent things to fix/implement (like maybe support for java.time
?), and start working on a release.
Important point: the Maven coordinates will change from org.beanio:beanio
to com.github.beanio:beanio
. The rest should be compatible with version 2.1 (including package names).
New repo: https://github.com/beanio/beanio
New website: https://beanio.github.io/
To start a discussion: https://github.com/beanio/beanio/discussions
A bug when handling a csv line ending with the last field having a missing quotation mark.
The bug can be fixed by, CsvReader.java line 279, adding recover(text);
https://github.com/kevinseim/beanio/blob/master/src/org/beanio/stream/csv/CsvReader.java
I am using BeanIo in a project that needs the ability to validate the length on marshaling fields. This feature was added in the commit c603758 but there hasn't been a formal release since then. Is there a plan to release a new version? I would feel more comfortable using a formal release than using a build based on a specific git commit.
Calin
As I need to marshal and unmarshal message that is fixed length with 2 little endian bytes "2A 00" as below example, how to configure BeanIO mapping file and object in order to can do that?
RAW-DATA:
30 30 30 31 2A 00 0A 00 31 32 - 33 20 20 20 20 20 20 20 14 00 1 [0001* 123 ]
41 41 41 41 41 41 20 20 20 20 - 20 20 20 20 20 20 20 20 20 20 21 [AAAAAA ]
00 00 42 [ ]
Packet Details: Code=0001 len=42
[001: 10]:[123 ]
[002: 20]:[AAAAAA ]
[003: 0]:[]
hat steps will reproduce the problem? 1. Create multiple Unmarshallers (one per thread) from the same StreamFactory 2. Unmarshall records with a date in each of the threads
What is the expected output? What do you see instead? All records get unmarshalled correctely. Instead, Exceptions get thrown.
What version of BeanIO are you using? What JDK version? BeanIO 2.1.0
Please provide any additional information below. While the documentation claims that "All included BeanIO type handlers are thread safe", DateTypeHandlerSupport potentially uses the same SimpleDateFormat for multiple Unmarshaller instances. Since SimpleDateFormat is not thread-safe this causes the behaviour as shown. Switching to a custom TypeHandler that uses a thread-local SimpleDateFormat solves the problem.
Comment #1
Posted on Apr 27, 2015 by Helpful Camel
Same behaviour when using beanIO in a camel route.
Strange exceptions that trace back to SimpleDateFormat not being thread safe.
If I have a fixed length records like this
Xmessage
XDother message
XDanother message
Where X, XD and XR are the Ids, BeanIO cannot identify the record type correctly.
BeanIO is used to generate the formatted files in one of our projects @ our org.
It is well-versed and helped to generate the all type of specifications such as .csv, delimited, fixed-length..etc
As a payment company, we have to stick with PCI compliance, the files should not be readable by anyone immediately, because it may contain Personal Identifiable Information. BeanIO is enhanced to generate the encrypted file. Encryption is based on PGP; bouncy-castle Open Source APIs are used to generated the PGP encrypted content.
We would like to contribute this commits into BeanIO framework. So that it will be available to public, if anyone wants to generated the PGP content, then it will be quite useful.
As part of this we have added few new classes and modified few classes. We would like to know if we can commit our changes and if yes, please let us know if there is any process to be followed
I'm running into all kinds of bugs, NullPointers, ClassCast exceptions inside the framework, validation errors. I'm not able to get the following to work at all, although I'm already working around several bugs.
Following are the following components: annotated record class, fixed width JMS message converter config + typehandlers, and junit test class. The JMS parts can be removed.
I don't think it should be very difficult: map between fixed width string to annotated class and back. I couldn't find any junit tests also in your code base that touches this.
import org.beanio.annotation.Field;
import org.beanio.annotation.Record;
import java.math.BigDecimal;
import java.util.Date;
import static org.beanio.builder.Align.LEFT;
import static org.beanio.builder.Align.RIGHT;
@Record(minOccurs = 1)
public class InputVariablesDto {
// INVOER-SEGMENT
@Field(at = 0, length = 3, align = RIGHT, padding = '0')
private Integer renteBerichtCode;
@Field(at = 1, length = 8, format = "yyyyMMdd")
private Date renteBerichtDat;
@Field(at = 2, length = 3, align = RIGHT, padding = '0')
private Integer renteVastPeriode;
@Field(at = 3, length = 1, handlerName = "JaNeeTypeHandler")
private Boolean nhgInd;
@Field(at = 4, length = 12, align = RIGHT, handlerName = "LisDecimalTypeHandler", padding = '0')
private BigDecimal marktwaardeFactor;
// UITVOER-SEGMENT
@Field(at = 5, length = 4, align = RIGHT, padding = '0')
private String foutKode;
@Field(at = 6, length = 120, align = LEFT, padding = ' ')
private String foutOms;
@Field(at = 7, length = 12, align = RIGHT, padding = '0', handlerName = "LisDecimalTypeHandler")
private BigDecimal actHypRente;
public Integer getRenteBerichtCode() {
return renteBerichtCode;
}
public void setRenteBerichtCode(Integer renteBerichtCode) {
this.renteBerichtCode = renteBerichtCode;
}
public Date getRenteBerichtDat() {
return renteBerichtDat;
}
public void setRenteBerichtDat(Date renteBerichtDat) {
this.renteBerichtDat = renteBerichtDat;
}
public Integer getRenteVastPeriode() {
return renteVastPeriode;
}
public void setRenteVastPeriode(Integer renteVastPeriode) {
this.renteVastPeriode = renteVastPeriode;
}
public Boolean getNhgInd() {
return nhgInd;
}
public void setNhgInd(Boolean nhgInd) {
this.nhgInd = nhgInd;
}
public BigDecimal getMarktwaardeFactor() {
return marktwaardeFactor;
}
public void setMarktwaardeFactor(BigDecimal marktwaardeFactor) {
this.marktwaardeFactor = marktwaardeFactor;
}
public String getFoutKode() {
return foutKode;
}
public void setFoutKode(String foutKode) {
this.foutKode = foutKode;
}
public String getFoutOms() {
return foutOms;
}
public void setFoutOms(String foutOms) {
this.foutOms = foutOms;
}
public BigDecimal getActHypRente() {
return actHypRente;
}
public void setActHypRente(BigDecimal actHypRente) {
this.actHypRente = actHypRente;
}
}
import InputVariablesDto;
import org.beanio.Marshaller;
import org.beanio.StreamFactory;
import org.beanio.Unmarshaller;
import org.beanio.builder.FixedLengthParserBuilder;
import org.beanio.builder.StreamBuilder;
import org.beanio.internal.parser.MarshallerImpl;
import org.beanio.types.TypeConversionException;
import org.beanio.types.TypeHandler;
import org.springframework.jms.support.converter.MessageConversionException;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.Session;
import javax.jms.TextMessage;
import java.math.BigDecimal;
import static org.apache.commons.lang.StringUtils.*;
public class FixedWithMessageConverter implements org.springframework.jms.support.converter.MessageConverter {
private static final Marshaller marshaller;
private static final Unmarshaller unmarshaller;
static {
StreamFactory factory = StreamFactory.newInstance();
factory.define(new StreamBuilder("inputVariables")
.format("fixedlength")
.parser(new FixedLengthParserBuilder())
.addTypeHandler("JaNeeTypeHandler", new JaNeeTypeHandler())
.addTypeHandler("LisDecimalTypeHandler", new LisDecimalTypeHandler())
.addRecord(InputVariablesDto.class));
unmarshaller = factory.createUnmarshaller("inputVariables");
marshaller = factory.createMarshaller("inputVariables");
}
@Override
public Message toMessage(Object object, Session session) throws JMSException, MessageConversionException {
if (!(object instanceof InputVariablesDto)) {
throw new IllegalArgumentException("expected InputVariablesDto but was: " + object.getClass());
}
String[] strings = ((MarshallerImpl) marshaller.marshal(object)).toArray();
return session.createTextMessage(join(strings));
}
@Override
public Object fromMessage(Message message) throws JMSException, MessageConversionException {
if (!(message instanceof TextMessage)) {
throw new IllegalArgumentException("unknown message type: " + message.getClass());
}
return unmarshaller.unmarshal(((TextMessage) message).getText());
}
/**
* Domain: J = With NHG (true), N = Without NHG (false)
*/
private static class JaNeeTypeHandler implements TypeHandler {
@Override
public Object parse(String text) throws TypeConversionException {
return "J".equals(text);
}
@Override
public String format(Object value) {
return ((Boolean) value) ? "J" : "N";
}
@Override
public Class<?> getType() {
return Boolean.class;
}
}
/**
* Format: ‘+00000,00000’ Example: ‘+00000,90000’
*/
private static class LisDecimalTypeHandler implements TypeHandler {
@Override
public Object parse(String text) throws TypeConversionException {
return new BigDecimal(text.replace(',', '.'));
}
@Override
public String format(Object value) {
BigDecimal bigDecimal = (BigDecimal) value;
bigDecimal.setScale(5);
String signum = bigDecimal.signum() == -1 ? "" : "+";
String strValue = bigDecimal.toPlainString().replace('.', ',');
String mainValue = strValue.substring(0, strValue.indexOf(','));
String fractionValue = strValue.substring(strValue.indexOf(',') + 1, strValue.length());
return String.format("%s%s,%s", signum, leftPad(mainValue, 5, '0'), rightPad(fractionValue, 5, '0'));
}
@Override
public Class<?> getType() {
return BigDecimal.class;
}
}
}
import InputVariablesDto;
import org.joda.time.DateTime;
import org.junit.Test;
import org.mockito.ArgumentCaptor;
import javax.jms.JMSException;
import javax.jms.Session;
import javax.jms.TextMessage;
import java.math.BigDecimal;
import static org.assertj.core.api.Assertions.assertThat;
import static org.joda.time.format.DateTimeFormat.forPattern;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class FixedWithMessageConverterTest {
private FixedWithMessageConverter converter = new FixedWithMessageConverter();
private final DateTime VANDAAG = DateTime.now();
@Test
public void toMessage() throws JMSException {
Session mockedSession = mock(Session.class);
ArgumentCaptor<String> captor = ArgumentCaptor.forClass(String.class);
when(mockedSession.createTextMessage(captor.capture())).thenReturn(null);
converter.toMessage(createListObject(), mockedSession);
String datumStr = forPattern("yyyyMMdd").print(VANDAAG);
String oms = "omschrijving ";
assertThat(captor.getValue()).isEqualTo("001" + datumStr + "044" + "J" + "+00000,90000" + "0001" + oms + "+00005,75000");
}
@Test
public void fromMessage() throws JMSException {
String fixedWidthMsg = "00120160615044J+00000,900000001omschrijving +00005,75000";
TextMessage tm = mock(TextMessage.class);
when(tm.getText()).thenReturn(fixedWidthMsg);
String result = (String) converter.fromMessage(tm);
assertThat(result).isNotEqualTo(createListObject());
}
private InputVariablesDto createListObject() {
InputVariablesDto dto = new InputVariablesDto();
dto.setRenteBerichtCode(1);
dto.setRenteBerichtDat(VANDAAG.toDate());
dto.setRenteVastPeriode(44);
dto.setNhgInd(true);
dto.setMarktwaardeFactor(new BigDecimal("0.9"));
dto.setFoutKode("1");
dto.setFoutOms("omschrijving");
dto.setActHypRente(new BigDecimal("5.75"));
return dto;
}
}
I have a .txt file with ANSI (windows-1252) Encoding i'am able to read it on windows but not on unix.
here is the xml mapping file :
<beanio> <stream name="empData" format="csv"> <parser> <property name="delimiter" value=";"/> <property name="alwaysQuote" value="false"/> <!--<property name="quote" value='' />--> </parser> <record name="emp" class="com.MyClass" > <field name="name" /> <field name="job" /> <field name="adress"/> </record> </stream> </beanio>
Java Side :
StreamFactory factory = StreamFactory.newInstance();
InputStream in = this.getClass().getClassLoader()
.getResourceAsStream("mapping.xml");
Reader reader = new InputStreamReader(this.getClass().getClassLoader()
.getResourceAsStream("countries.txt"));
factory.load(in);
BeanReader beanReader = factory.createReader("empData", reader);
Gson gson = new Gson();
/*Object bean =new Object();*/
Object record = null;
while ((record = beanReader.read()) != null) {
System.out.println(beanReader.getRecordName() + ": "
+((MyClass)record).getCountry());
}
Result :
line : France
line : S??o Paulo should be (São Paulo) windows OK but unix is KO
line : USA
line : China
Any idea ?
FYI : i already tried to set Charset to UTF-8 java side .
new InputStreamReader(this.getClass().getClassLoader().getResourceAsStream("countries.txt"), Charset.forName("UTF-8"));
Can i define the precision of a field that does not have a decimal delimiter?
Hey there,
There is some inconsistency with the Date objects currently, specially in the persistence part.
Imagine the scenario:
The prepare statement object only accepts java.sql.Date and not the java.util.Date, shouldn't BeanIO had a SQLDateTypeHandler class to handle sql.Date classes? Could be really handy...
In beanio we have a property regex that we can add for field validation, which causes to throw an Exception if is not valid...
We could add a new property, for example, regexIgnore or something like that in order to avoid the exception and read only the lines that respect the predicate of the regex.
When you use fixed length stream you need to calculate the at position based in the length of the and the position of the previous record.
I suggest to have a new annotation property called relativeAt or something like that to calculate the true position in the beanIO implementation it self.
BeanIO should handle LocalDate and LocalDateTime by default.
https://docs.oracle.com/javase/8/docs/api/java/time/LocalDate.html
https://docs.oracle.com/javase/8/docs/api/java/time/LocalDateTime.html
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.