Internal Issue/Enhancement

Description

If selected Read CSV attachment action with Emit Behavior - Emit Batch there is no Batch Size field in input metadata

Definition of Done

Batch Size field in input metadata when selected Emit Behavior - Emit Batch

Other Feature Request

Description

Currently, the Read CSV Attachment action has two options for the Emit Behavior config field: Fetch All & Emit Individually. It currently lacks a Emit Batch option like the option that is present in the MongoDB component.

Definition of Done

Ideally, there should be a third option: Emit in Batches. When this option is selected, a new input should be added to the metadata to store Batch Size. Then the component should produce a series of message where each message has an array of max length equal to the Batch Size. This behavior should be similar to the MongoDB component.

Below is the issue elasticio/csv-component#47
Need to check if it is still a bug

I've build a simple integration flow:

then using simple load-test tool loaded 2000 request to it:

#!/bin/bash
loadtest -c 5 -n 20000 "https://in.elastic.io/hook/5de5199b20ea507fdaf1a542?message=Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum"

It run all well:

[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Max requests:        20000
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Concurrency level:   5
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Agent:               none
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Completed requests:  20000
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Total errors:        0
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Total time:          837.860013091 s
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Requests per second: 24
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Mean latency:        209.4 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Percentage of the requests served within a certain time
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO   50%      201 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO   90%      213 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO   95%      235 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO   99%      400 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO  100%      723 ms (longest request)

at the end I expected a file to be uploaded to SFTP to have 20.000 rows in it:

[2019-12-02 14:34:33.287]: The resulting CSV file contains 20000 rows
[2019-12-02 14:34:33.287]: Closing the stream due to inactivity
[2019-12-02 14:34:33.290]: Emitting message {"id":"f705ff7c-b18a-4e91-83ac-5a879745d92b","attachments":{"f705ff7c-b18a-4e91-83ac-5a879745d92b.csv":{"content-type":"text/csv","url":"http://steward-service.platform.svc.cluster.local:8200/files/0375788e-5fcd-4931-8208-811c28d1ac3d"}},"body":{"rowCount":20000},"headers":{},"metadata":{}}

unfortunately resulting file was only 1 MB large and had 1795 rows in it, not 20000

Attaching result and log of the execution as ZIP file.

Archive.zip

Component Bug Report

Description

Description from the Infobip team:
Hi Team,
we are trying to use the Emit Batch configuration for the CSV component but it doesn't have a way to input the batch size and it sends a null to the next step, can you advise?

Component Version

3.1.2

Steps to Reproduce

Actual Result

[What actually happened]

Expected Result

[What you expected to happen]

Workaround(s)

[Any known workaround(s) to circumvent the above issue]

We are about to deprecate the CSV component
To make customers' life easier we should add this trigger to the new component. As that component has it, so should this one.

As the base for this trigger an existing action Read CSV attachment can be used. The URL should be moved from the input metadata to the config fields

Other Feature Request

Description

We got feedback from the poss client:
"Partially very limited connectors - e.g. CSV: hardly any setting options for date, separator formats, position coded text formats"

Definition of Done

Define new additional actions applicable to the component.

Other Feature Request

Description

Currently, the max supported attachment size is 100MB (by the platform). It would be nice if the component can handle files of up to 1GB in size without the need to increase the allocated component memory beyond 250 MB. Specifically, we should check the read action when the emit mode is either emit individually or emit batch (for reasonably small batch sizes).

Definition of Done

One of:

• The component can manipulate files of 1 GB without needing to alter the component code or the available memory (only change the max file size env var.)
• We make the needed code changes so that the component can manipulate files of 1 GB without needing to increase the available component memory
• We identify a reason why files of 1 GB can not be manipulated without increasing the component memory.