There is currently no support for MySQL-Databases. As most infrastructures do proved MySQL/MariaDB instead of PostgreSQL, it would be a good addition to support MySQL-based databases as well.

Hi there,

I'm using .csv files with encapsulated headers and values, like "Header", "Header2" etc,
But for some reason all the "" get stripped off. Any ideas why this happens?

I do have one requirement where i need to add a new entry after a transformation. The pseudo-code would be something like this.

 public function transform(Rows $rows): Rows
            {
                foreach ($rows as $row) {
                    $entries = $row->entries();

                    // This will return a new entries object
                    $newEntries = $entries->add(new Row\Entry\StringEntry('name', 'Blabla'));

                    // The newEntries variable can not be used within the Rows-object. It would be great to have the option
                    // to dynamically add a new entry to an existing row.
                }
                return $rows;
            }

It seems like this is not a supported case at the moment. What i do instead is to create a new Rows-object instead. So something like this:

   public function transform(Rows $rows): Rows
            {
                foreach ($rows as $row) {
                    return new Rows(Row::create(
                        Entry::integer('name', '')
                    ));
                }
            }

https://phpspreadsheet.readthedocs.io/en/latest/ brings huge potential into the FLow ETL for both, reading and writing data.
If anyone would be interested in taking this please let me know so I can prepare the repository or help to design that adapter.

Even if initially this adapter would only support only reading and writing into Xlsx it would be enough.

• PhpSpreadsheetXlsxExtractor
• PhpSpreadsheetXlsxLoader

I want to evaluate the following case:

1. Read data from one system and enrich it with data from another system. One system returns a json while the other one returns some strings.
2. The data requires heavy transformation. Basically applying some algorithms is required where the result is a new json payload for an external systen.
3. Afterwards the data needs to be grouped in bulks of for example 100 items and then published to another system.

I currently struggle with the transformation-part. I would build a class which runs all the transformations but do not understand, how this could be achieved in this library:

$flow ->read($externalSystem1) ->transform() // How would i pass a custom transformer? It would be a class. ->write(To::memory($array)) ->run();

Webmozart Assert dependency was used here in early POC version however now when the library is open we should reduce the number of dependencies as much as possible.

I was able to reproduce this error: If the rows-entries only contain one single element, then an exception will be thrown.

yield new Rows(
                Row::create(Entry::integer('id', 1), Entry::string('name', 'EN'))
            );

Exception

Throwmax(): Argument #1 ($value) must be of type array, int given

In AbstractErrorHandler.php line 38:

  [Exception]
  TypeError: max(): Argument #1 ($value) must be of type array, int given in /var/www/share/dev/htdocs/vendor/flow-php/etl/src/Flow/ETL/Formatter/ASCII/ASCIITable.php:176
  Stack trace:
  #0 /var/www/share/dev/htdocs/vendor/flow-php/etl/src/Flow/ETL/Formatter/ASCII/ASCIITable.php(176): max(1)
  #1 /var/www/share/dev/htdocs/vendor/flow-php/etl/src/Flow/ETL/Formatter/ASCII/ASCIITable.php(89): Flow\ETL\Formatter\ASCII\ASCIITable->getColWidths(Array, 20)
  #2 /var/www/share/dev/htdocs/vendor/flow-php/etl/src/Flow/ETL/Formatter/AsciiTableFormatter.php(31): Flow\ETL\Formatter\ASCII\ASCIITable->makeTable(Array, 20)
  #3 /var/www/share/dev/htdocs/vendor/flow-php/etl/src/Flow/ETL/Loader/StreamLoader.php(89): Flow\ETL\Formatter\AsciiTableFormatter->format(Object(Flow\ETL\Rows), 20)
  #4 /var/www/share/dev/htdocs/vendor/flow-php/etl/src/Flow/ETL/Pipeline/SynchronousPipeline.php(66): Flow\ETL\Loader\StreamLoader->load(Object(Flow\ETL\Rows))
  #5 /var/www/share/dev/htdocs/vendor/flow-php/etl/src/Flow/ETL/DataFrame.php(260): Flow\ETL\Pipeline\SynchronousPipeline->process(Object(Flow\ETL\Config))
  #6 /var/www/share/dev/htdocs/bundle/Mothership/YadiBundle/src/Command/AbstractFlowCommand.php(138): Flow\ETL\DataFrame->run()

Complete example:

        $flow  = new Flow();
        $countries = new Rows(
            Row::create(Entry::integer('id', 1), Entry::string('country', 'EN')),
        );

        $flow->read(From::rows($countries))
            ->write(To::stdout())
            ->run();

I want to be able to extract documents stored on S3, so I have registered the stream wrapper:

$client = $sdk->createS3();
$client->registerStreamWrapper();

And then I attempt to load the file:

(new Flow())
    ->extract(CSV::from_file(
        file_name: 's3://my-bucket-name/test.csv',
        header_offset: 0,
    ))
    // ...
    ->run();

But the following exception is thrown:

 [Flow\ETL\Exception\InvalidArgumentException]                                                    
  File s3://my-bucket-name/test.csv not found.

I think the check for file existence should be modified to allow remote URLs:

if (! (str_contains($file_name, needle: '://') || is_file($file_name))) { /* throw exception */ }

OR add another method to handle resources, eg:

$extractor = CSV::from_resource(
    resource: fopen('s3://my-bucket-name/test.csv'),
    header_offset: 0,
);

Is it possible run php 7?

In the PHP community, there is PSR-6 which defines common interfaces for the common use of cache implementation.

Its key concepts can be described as:

Item
A single unit of information is stored as a key/value pair, where the key is the unique identifier of the information and the value is its contents;

Pool
A logical repository of cache items. All cache operations (saving items, looking for items, etc.) are performed through the pool. Applications can define as many pools as needed.

Adapter
It implements the actual caching mechanism to store the information in the filesystem, in a database, etc. The component provides several ready-to-use adapters for common caching backends (Redis, APCu, PDO, etc.)

Implementing those will allow to use of more adapters and reduce the maintenance of the ETL library while improving the DX.