mc12345678 / easypopulate-4.0 Goto Github PK

View Code? Open in Web Editor NEW

This project forked from chaddro/easypopulate-4.0

10.0 4.0 8.0 2.2 MB

Data import and export module for Zencart 1.3.x and 1.5.x

License: GNU General Public License v2.0

PHP 100.00%

easypopulate-4.0's Introduction

EasyPopulate 4.0 Beta SBA_Stock

There is a Zen-cart.com Support Thread located here:
http://www.zen-cart.com/forum/showthread.php?t=190417

I recommend that you first enter a couple Categories and Products through the normal Zencart Admin. This will help you understand the exported file formats and how to import.

Summary of important things:
1) CSV Files
2) primary key to match to records
3) Categories and multi-language support
4) File size (splitting)
5) Filenaming as a filter.
6) Attributes (the type for products)

Things of Importance:

1) EP4 works only with CSV data files, and for best cross platform compatibility please use OpenOffice to Export your CSV file.
You should select the Comma as your field delimiter, and the Double Quote as your text delimiter. You may not get proper results using Excel.
At least one specific known issue is that dates must be formatted to be like: YYYY-MM-DD HH:MM:SS (or without the time at the end)

2) There are now two ways to address product that is imported. The default as installed remains the products_model.
When assigned as the primary key, the products_model must be used to distinguish your products for import. Any record with a
blank v_products_model entry will be skipped to be considered desired by design.

Also note that if you enter the same products_model twice, the latest (last) record
entry will over-write any previous entry. If the category is different in any later record, then all other new data will be replaced with
what the current record has and the product will be linked to the category identified in this record.
This means that if two rows of data have all the same v_products_model data with only a difference of the category name,
then the product will appear in both categories and a change to the data in one entry will appear in all categories where the product is linked.
See the previous sentence about effect of two products having the same products_model identifier.
"Duplicate" products with the same products_model number is not supported when using the v_products_model as the primary key, they will each end up with the same information.
If you have these entries in your database, you may get unpredictable results when importing data.

The second method of import is to use the products_id.

NOTE: Incorrect use of the products_id can severely corrupt your database and store.

It is always possible to export the products_id as a user defined field even if products_model is the chosen primary
field. Use of the products_model field allows for matching of data from perhaps an outside vendor to the data in the database.
Use of the products_id will allow export of the store, modification of that file and then again import of the applicable row(s) of
data to update the modified field(s) (This could include assigning a products_model to the products_id).
When working with the products_id as a primary key, there are two settings. One with
products_id only, where rows with a blank products_id will be skipped like with a blank products_model entry described above, the
other, blank_new, will assign a new products_id (next "available") for records that have a blank for the products_id field in the
applicable row.

3) Categories are handled differently from other versions of EP. Also, you can now import multilingual categories and related multilingual
data (like Products Names, descriptions, etc.).
To achieve this, each v_categories_names_1, v_categories_names_2, etc. with the number corresponding to a language installed in your system or
it is also now possible to use the language code (e.g., en, de, es) to reference the language. For example if english represents language id of 1,
then either v_categories_names_1 or v_categories_en could be used. However, the data to capture for that field is to have each
Individual category names for a product separated by the Carat "^" symbol ($category_delimiter).

For Example: Bar Supplies^Glass Washers^Brushes

"Bar Supplies" will be your Top level Category, with "Glass Washers" as a sub-category, and "Brushes" as a sub-category of "Glass Washers" (sub-sub-category of "Bar Supplies").

Be careful when creating your category names, the routine is case sensitive. "Bar Supplies" is not equal to "BAR Supplies" ... this
will result in TWO category entries and will effectively create a linked product entry (same product will appear in each category).

As of 4.0.17, your lowest defined language (by language id) is needed to be used when creating categories. In the future I'll change
this to the defined default language, but that will take some additional work.

4) Work flow is somewhat different than other versions of EP. With EP4 you first upload your file, then click on Import. Optionally
after the file is uploaded, you can also Split an import file if it is too large. The number of records on which to split the file
is controlled in the configuration settings. Default is 2000. If you have a powerful server, you can increase this significantly.
Testing on a VPS with an import file of 900,000 records, I split the file into 50,000 record segments. Import of each 50,000 records
took about 250 seconds.

Also there is NO streaming upload or download support. Sorry if you liked this feature. A lot of effort has been put into improving
the code's performance and streaming the data was a real memory hog. To download your exported file, you will need to
"right-click, save-as". You may be able to set your browser to automatically download csv/CSV files when clicking on the download link. I'll come up with a better solution in due time.

5) File names are important and act as a switch inside the script for Importing. This means that in order process data associated with
featured product, the filename must begin with the non-case sensitive name of Featured-EP. Anything that follows that name can be
modified to suit but must follow the file naming "rules" of your host and software. The filenames requiring unique file
naming are: PriceBreaks-EP, CategoryMeta-EP, Featured-EP, SBA-Stock-EP, and Attrib-Basic-EP/Attrib-Detailed-EP ... these
import files must have names that start with the applicable string.

For example, attempting to import a file to modify the featured part(s) of a product where the filename does not begin with
Featured-EP (case insensitive) data will not update the featured specific data of the product. Files that are not named using the
identified unique filename prefix will be processed as a full product with each field processed as might be expected for a full
product import.

6) Basic Attribute Import. Please read the notes below carefully about importing attributes and how to use this feature which is still in development. Currently, you can
at Import create your products_options_name (your attribute name, ie. color), products_options_type (checkbox, dropdown, etc), and
products_options_values_name (red, green, blue).

LASTLY but definitely not LEAST and applicable to all of EP4: Be sure to backup your data before importing your files. I have made every attempt to make this a solid bug free product, but occasionally new features require more testing. A lot of error
trapping has been added, but I'm sure I've missed things.

IMPORTING ATTRIBUTES
Including sample attribute input file.

I am currently able to correctly import basic attributes, and assign
the options to an associated product's model. It is possible to create
multiple sets of Option Names / Option Values and assign to a single
product, say "Size" and "Color".

The basic attributes CSV file currently has 4 columns:
1) v_products_model
a) The products model number must already exist, and should be
unique within the store as linked products have not been tested.

2) v_products_option_type
a) this is the type of attribute you want to create and should be
a number between 0 - 5 (for a default store or the number associated with your added software) and are defined as follows:
0 - Drop Down Box
1 - Text Box
2 - Radio Button
3 - Check Box
4 - File Upload
5 - Read Only
One way to identify the number associated with an additional option type is to navigate to Catalog->Option Names. Once there,
note the option type assigned to the product on screen and the list of option types in the dropdown list. Now view the source of
the page and search for one of the option types in the list that are not otherwise used on the screen. The found html option
values do/should match the list above and then show any new(er) option type values that can be used by the software.

b) for a given option_name (say "Color"), do not change the products_option_type
on subsequent entries, doing so will not give the results you want.
c) If you need a "Color" with both a drop down box and check box, you will need
to define two unique Options Names.

3) v_products_options_name_1
a) The option name you want to create or use in the language associated with the number at the end.
b) It is important to note that Zen Cart will allow you to create
from within the admin two identical Options names, and assign
unique options values to each. For example:
"Color" with "red,green,blue" as one option name/value pair
"Color" with "cyan,magenta,yellow" as another option name/value pair.
Internally, Zen Cart knows these are two distinct options names,
but this info is not available to the user. (It may have been
better to have a unique Options Name, and associated Options Display Name
which in turn could be language specific).
c) For this reason, the products_options_name_1 must be unique, meaning
you CAN have "Colors" but the associated options_values set will be
the sum of the example above: { red,green,blue,cyan,magenta,yellow }.
(This is information in the database. The individual product will still only show
the attributes assigned.)
d) It is generally easier to work with and understand the attributes if there
is one option name that has multiple option values associated with it, but
there is no requirement to setup your site this way.

4) v_products_options_values_names_1
a) these are the values names that are assigned to the products_options_name
b) enter the values_names, delimited with a comma for each value.
c) note that ONLY these products_options_values_names will be assigned
to the given products_model

A way to reproduce attributes in one store from another using EP4:

If store 1 has a unique model # for each product and the option names are unique (only one instance of the option name in all of the ZC option names), then the import of the attributes_basic file has populated store 2 with all of the attributes of the product from store 1. What is missing though is the detail associated with each attribute. So, to update store 2's attribute details, a file has to be generated such that each of the four primary detailed attributes related keys match an existing entry in the attributes table. So, how to accomplish this?

Here's what I see. It is possible at this point to export the detailed attributes from both store 1 and store 2. Each of these has text versions of the option names and option values names. The model number is whatever it is and the products_attributes_id is expected to be unique to each store.

So what I would do would be to sort the data in both detailed lists on three fields in the same "order" (either ascending or descending but make it the same on both spreadsheets) such that sorted by products_model, then option_name, and then option_value_name.

Then pick the method/location desired, but the goal is to eliminate from the list for store 2 any product that is not in store 1. This would be by a comparison of products_model.
For now, also eliminate from store 1 any row that doesn't have a model #. (will have to address that separately because really that product never got uploaded to store 2, but at least the list should be small.)

Then begin moving entire rows as necessary such that the row in store 1 lines up with the row of store 2 by first comparing option names then option value names. Provided nothing "new" has been done with store 2, these should line up exactly with no editing.

Then once all have been lined up, copy the primary field data from store 2 over the data of the same field for store 1. Once all four columns are copied, save the file as the csv file to be uploaded and then imported into store 2.

Obviously through this process you'll want to save a backup of the file(s) to minimize any rework. Keep in mind the filenaming convention needed by the plugin.

And with that, the new file when uploaded and imported to store 2 should cause store 2 to have the same attributes and details of attributes as store 1.

Other attribute guidance: Currently the detailed attributes unlike the detailed product information
for example are based on the specific database centric field record designations. Because of this,
to properly update the detailed information, one must first upload and import the basic information,
then download/export the detailed attribute data and after all of this, the CSV file processed
as a new imported file with the updated data to be stored in the database.

IMPORTANT NOTE:
When creating your CSV file, please use Open Office (it's free!). When you save
CSV files from Open Office (OO), it will properly encapsulate all fields for import. Excel will not
necessarily do this, depending on your data, and the export CSV option
you pick from within Excel, also dates and date formats may be revised by Excel and not readable upon import.

In the /examples directory is a sample input file: Attrib-Basic-EP-examples.csv You can create your own sample file by exporting the data file type of your choosing.

The "Detailed Products Attributes" shows all attribute details assigned to a given products_model,
with one line per option_name/value combination. So a product with a dropbox of 3 colors will result in 3 lines of
data exported. As you can see, there is a significant amount of data that is associated with
attributes.

The "Stock of Items with Attributes Including SBA" option is used to support performing an inventory of stock providing a list of products that
1) do not have any attributes,
2) have attributes but are not tracked by stock, and
3) are tracked by stock by attributes (have attributes and the quantity of items that have that attribute/set of attributes is maintained).

This functionality will only show when the appropriate version of stock by attributes (SBA) is detected to be installed to the cart.
If SBA is not installed, then this functionality will not be presented and the remaining instruction regarding this feature can be skipped.

Below is a brief summary of the report that is generated by the Easy Populate version 4 (EP4) "Stock of Items with Attributes
Including SBA" option and the method to access it. The file prefix has been set to: SBA-Stock-EP

The two primary characteristics to consider when using EP4 are the columns (headers) across the top of the spreadsheet and the items
being displayed. To explain, the headers from left to right offered by this feature are:

1. Model #
2. Status (product enabled or disabled)
3. Product Name in each language entered shifting the remaining columns to the right.
4. Product Description without html tags (no <br/>, <p>, etc.) (In each language entered shifting the remaining columns to the right.)
5. If potteryhouse's version of SBA is installed (or the customid field exists in addition to some other fields) then this column will
have the customid, otherwise, all of the columns below will move left to replace what would be in this column.
6. Whether the item listed in the row is tracked by Stock by Attributes (SBA) using a marker if yes and leaving the row's field blank if no.
7. A unique identifier associated with the data type of item in the row.
8. The attributes associated with the item in the row (if any exist) put together in the format
OptionName1: OptionValue1; OptionName2: OptionValue2; etc...
OptionName1 may be the same as OptionName2 and still will be listed as shown with a different OptionValueX at each entry.
9. The quantity of the item in the row.

On upload/import, the only field that will change in the Zen Cart database with this report is the quantity associated with the row's data.

Item 7 of the list was abbreviated to simplify the explanation. For an entry displayed that shows the total quantity associated with a
product, the value in that column is the products_id taken from the products table of the database. For a row that contains SBA
information, the value is the stock_id taken from the products_with_attributes_stock table. These values are provided to support
import of the data and they should not be revised for normal operation. The position of the column was chosen to not place it adjacent
to a field that is likely to be changed by the user. (Technically for EP4 to import this new file, the only columns important for the
import are the ones located in items 6, 7 and 9, all of the other columns were provided to help the individual performing the stock
inventory identify the product(s).) Ideally, column 7 would not exist and instead the program would determine the appropriate value for
that column based on other information in the table so that the spreadsheet would not be dependent on the current database but could be
applied to any database at any time. This will require a revision to accomplish.

This brings us to the arrangement of the rows. Every row of data has the potential of having attributes associated with the product, but
every product has a quantity associated with it. Below are two examples 1) of a product without attributes, the other with attributes as
described.

On export, every product will be listed, whether an active product or not. If a product is tracked by SBA, then the product's data will
be provided as the first row for that product, followed by rows of the SBA associated data. So if a product has two option
values (green and blue) for a single option name (Color) but is not tracked by SBA then in the row for that product the attributes
column (item 7 of above) and quantity (item 8 of above) assuming 70 of this product will show as:

Product has no attributes:
| Attribute(s) | Quantity
| | 70

Product has two attributes (2 option values, but one option name):
| Attribute(s) | Quantity
| Color: green;Color: blue | 70

If a second product had the same attributes with 70 total, but 40 were green and 30 were blue tracked by SBA, then the export would be
something like:

There are two links available for import. One that imports all of the data as entered (Import) (Import of the below followed by export
would provide the below) it would be considered acceptable and left as is. The other imports the data, but then synchronizes the total
with the sum of the SBA tracked quantity. Import of the below followed by export would provide the above.

The first example applies to the display of every product in the database:
- If a product does not have attributes, there is a total quantity of the product.
- If a product has attributes, then there are attributes associated with the product but that does not mean that quantities
are applied to each attribute: (for example if an attribute were a checkbox of "include or not include a generic note" with
the product, there is less likely a need to track a specific quantity of items with and without the note option, therefore the
row with an attribute would have a total number of items. The same could be said for the green and blue options if the green and
blue attributes had an unlimited supply, say if a marker is used to add a green stripe or a blue stripe. The number of stripes
is more dependent on the age of the pen, the quantity of ink and the time available of those applying the stripe than the number
of products that already have the stripe. So that product may not be tracked by SBA.) The data of the second example is similar to
this situation, or if the product has not yet been entered to be tracked by SBA.
- Lastly, if an attribute is applied to a product and the quantity of the product is tracked by SBA, then the database typically maintains
two sets of quantities. One is the total number of the product, the other is the number of the product that has the specific attributes
being tracked: ie, quantity 5 of a 10" diameter ring, made of silver, with a blue pendant as compared to quantity of 6 of a 5" diameter
ring, made of stainless steel, with a green pendant, and all the variations of these options. So in this case, the data provided in the
exported file would be similar to what was provided in the third example. The first row shows the total number of the product and all
possible attributes, each subsequent row shows the specific quantity of the product having the assigned attribute(s).

GENERAL OVERVIEW OF OPERATION:
So with regards to the way EP4 works. As you have found/desire, entry of multiple products in a single "swoop" is easier
with this plugin. That said, it still "follows" the same ZC process. First the product must exist, and as part of "creating" the product,
it must go into a category. Then, to apply attributes, the attribute(s) must exist (attributes basic), then the product is populated
with the attribute(s) (attributes detailed). In each step that relates to a product, EP4 generally requires a unique model_number; however,
with EP4 version 4.0.32+, it is possible to relate product by the products_id when the admin configuration settings are "properly" set.
A word of note/caution though is that it also depends on how you plan to use/automate EP4. If your import information will be from a
vendor in the form of some other file, then purely using the products_model (or possibly some other field as recoded in Ep4) would be suggested.
If the data to be imported will be "self generated" then use of the products_id might be more suitable. For those that don't yet have a
products_model associated, the same feature can be used to apply one to the product.
If you want to delete a product, then set the status of the product to the number 9.
Understand that this will completely remove the product from the store, not just disable it (status of 0).
To change the master_category_id for a product to be the category of the current row set the status to 7.
If there is no master_category_id assigned, then the current category on the row will become the master_category_id.
If there is already a master_category_id assigned to the product, then the category on the row will
become the master_category_id and the product will be removed from the previous category represented
by the master_category_id. The process is performed by always adding to the database before deleting anything.
Using this operation it is therefore possible to now remove the linked product through a sequence.
If there were three categories assigned to a product, then assigning the master_category_id to
each category that is not the current master_category will delete the product from the other
category(ies) and do this until the last entry for that product has the category to be the product's master_category_id.

easypopulate-4.0's People

Contributors

Stargazers

Watchers

Forkers

reetp zen4all-nl hamagawa zen4all daifu024 torvista brittainmark retched

easypopulate-4.0's Issues

Data file field separators

The default version of EP4 uses a comma separated format for exporting and importing a file. While it is possible to change this option from within the code, having the ability to modify the option from the admin panel would ease the process.

Recommend adding an admin configurable setting, possibly on the the admin usage page to simplify the overall process and be able to control the operation easier. If not on the admin page then within the configuration section.

PHP Warning: Undefined variable $file_location easypopulate_4_import.php on line 219 with incorrect column header

An import of Categories, with the v_products_model heading incorrectly named.
The corresponding error message output uses $file_location, but this was unset around line 81.

Import of detailed attributes does not offer update by "name"

Import of data using EP4 generally uses human readable information to support import of data. Examples are the products_model field for products and the combination of option_name and option_value_name for basic attributes.

Detailed attributes though "still" use the specific database provided primary keys to match the detailed attribute information with the applicable database entry. While this is or can be an important feature for updating a database's specific attribute information because of the constraints applied in using a specific option name that has the same content of another, it also can be a drawback if detailed attribute information is collected from an alternate source (different database) and attempted to be pushed against the current database.

Implementation of such a change though could become tricky as if two option names are generated for a product to have the same name (not exactly likely, but something with which to contend) then the wrong attribute appears could be updated with information. (More consideration is needed for that possibility.) At any rate, if applying this type of change, then when a query to the database identifies that a match can occur in more than the one primary key, then each option of that primary key's match needs to be cycled through to validate that no match exists before an "error" is generated. As such each possible combination within the database must be attempted when doing this review/comparison to identify the presence of the specific attribute entry. This certainly would be a point of identifying what goes best, more memory on a variable or more run time on one or more queries. Ideally some sort of balance can be found.

Currently the primary reason to support this is to make EP4 a better program to transfer information from one database to another instead of upgrading/importing the database.

split of file with 2 fields creates too many split files

Condition reported in the forum was that a file containing ~20,000 product records having the primary record and one additional field when split generated ~10,000 Smaller files instead of the expected 5 files of 5000. Cause to be identified, which should lead to a solution or other suggestion.

For those where something similar has occurred, ftp was used to remove the files instead of allowing them to clutter the screen/load.

This leads to some other potential option(s) to perhaps be desired such as some sort of filter to support a larger quantity of files to be deleted at once (with filter to prevent touching files outside of the storage path).

Skip record import if primary key should have data but is empty

Skips trying to process an import record if the primary key has no data but the primary key is expected to have data. I.e. blank_new supports having an empty primary_key so that a record would be imported; however, products_id does not and there should not be a value of 0 for a products_id.

filelayout $$v_categories_name is not getting set

For some reason, I'm facing a little situation here at import ( with PHP Version 7.1.14) :
The script is checking:
if (isset($filelayout[${$v_categories_name_var}]) || isset($filelayout[$v_categories_name_var_code])) {

However this is what I'm getting:

output $filelayout[${$v_categories_name_var}] -> 
output $filelayout[($$v_categories_name_var)] -> 
output $filelayout[$v_categories_name_var] -> 10

I can only go in the IF without the $$v_categories_name_var
Do we really need that variable variables here ?
Thanks

Support auto extension of field length during import

For the fields that where the length of the field is tracked, allow auto expansion of the field during import.

Featured dates default and operation

ZC stores blank dates as 0001-01-01 in the database. Currently EP 4 doesn't enforce this for importing featured product. Also, it appears that EP4 will disable a product's featured status that effectively has no end date when actually it should still be active to an unknown date.

"500error" occurred in importing attrib-basic-ep

Category assignments begin with language_id=1

In original development, the first language_id to be considered on import was assigned a value of 1 instead of whatever the first value was. This corrects that and begins with the ~~first~~ default language_id associated with the store installation.

English words for Constants

In incorporation of constants for language statements, the English words for the following two constants were not updated but instead copied from the line above:
EASYPOPULATE_4_DISPLAY_IMPORT_RESULTS_NUM_ERRORS
EASYPOPULATE_4_DISPLAY_IMPORT_RESULTS_NUM_WARNINGS
resolving to the text of:

New Imported records: %d

product name and description on import blank

During coding improvements to provide 4.0.32, a new method of capturing product description data was implemented to iterate through the languages of the store. In implementation the use of the variable $languages was used which is typically a global variable to represent the languages of the cart. EP4 was written to use $langcode to represent relatively equivalent information. In implementing the new code, $languages was used instead of $langcode. Installation of this plugin to a new clean ZC errors causing both a debug.txt file in the EP4 admin to be generated and for an admin error log file to be generated in the applicable ZC log folder identifying a warning associated with the line with that variable.

This issue was rectified by commit: bdde10e

Simplify update process

EP4 was written with the ability to remove the database data from the EP4 admin usage screen. Occasionally updates to the software require changes to the database information. To support this action, the existing entries had to be deleted and then re-installed. This evolution makes updating a little bit of a chore.

Recommend modifying the main page to offer an update statement related to the database that appears when applicable conditions exist to update versus install. Ideally this routine would share the sql that is used for initial install so that information need only be entered one time instead of more than one.

v4.0.37 - Basic attrib import primary key field references unknown table

In the latest revision to the basic import, the ability to choose a different primary key was incorporated.
The reference to the field includes a table alias (instead of the table name). Without referencing the table alias, mysql doesn't necessarily understand the relationship. So in this case p.products_id or p.products_model was not recognized in a query such as:

SELECT * FROM products WHERE p.products_id = 1

as the p. of p.products_id was not recognized.

To correct this and potentially similar issues where variables are used to fill in the blank of the primary key, select queries should include the table reference as part of the sql statement regardless of how simple the select query is.

No notification of uploading an empty file

When the user attempts to upload a file, there are a few verifications performed to validate that the file should be incorporated into the list of downloadable files; however, there was no check for an empty file if the received file happened to have a 0 size. This could be because of some issue such as running out of storage space on the destination computer, that the sent file was corrupt or empty, or some other event that could result in a filename but no size to the file.

Custom fields limited to Products Table

The verification of the custom fields is offered in admin/easypopulate_4.php where if the field is in the products table then it is added to the fields array. This process could be used to allow custom fields to come from other tables as well, if the table was identified in the custom fields entry.

To do this, would recommend a separator be used (ie colon ':') between the table and the field, then if a field is entered without a table it would default to the products table, or products could be used/identified as applicable.

While this could be incorporated, the downstream affect/change needed still needs to be evaluated in making it happen. For example two tables may not relate to one another on any primary key, but may relate to each other through some intermediate table. Without reference to the intermediate table/primary key, subsequent sql could provide conflicting information/sql error. Additionally, if there is an intermediate table, it may have another factor associated with it to provide a unique row such as a language identifier, etc... So some "growth" to be understood before just willy nilly adding the ability have other tables involved.

custom fields no longer a part of full file query

In improving the memory management of the plugin, an additional unset was added that affected the custom fields expected for full file operation.

ZC 1.5.5 download file path needs tweaking

Identified by wtashby in the ZC forum at: https://www.zen-cart.com/showthread.php?190417-EasyPopulate-4-0-Support-Thread&p=1306346#post1306346

I just installed EP 4 into zc 1.5.5, running on my windows pc with xampp. everything works fine until i try to download a generated file in the temp folder. I get this:
https://localhostdir_ws_https_admintemp/Full-EP2016Mar19-030356.csv

Response was:

That is intended to be generated by combining several parts of the definitions in the admin/includes/configure.php file (or possibly in the admin/includes/local folder if so used...

It would appear that DIR_WS_HTTPS_ADMIN is not defined on the admin side and that the page is loaded while using https:

As for use on ZC 1.5.5? Well, hasn't been tested yet, though was just about to do so.. Problem is that the current ZC 1.5.5 setup that I have is experiencing a problem that I have yet to be able to explain but it nearly seems like it is related to a database that I upgraded from 1.3.8 to 1.5.5...

Solution/temporary solution provided by DrByte in thread that begins with this post.

Need to further "integrate" the change(s) made to ZC 1.5.5 to support the possibility of the storage files being located either on the catalog or admin side and backwards compatibility without forcing user to follow best practices (when establishing SSL on admin that both the HTTP_SERVER and HTTPS_SERVER paths and associated DIR_WS paths are set and then set to the expected/desired state).

PHP error processing

One of the characteristics of EP4 is that an error log generated as part of processing the file(s) is presented to the user within the EP4 admin screen.

The way this has been enabled/performed has been basically through error catching when using standard PHP commands instead of using the ZC related functions. Support of multiple versions of PHP has required specifically formatted code to allow continued operation and to have/keep this EP4 "error log". Ultimately though, compatibility of EP4 with future versions of PHP and ZC arrangements would be improved if ZC functions could be used throughout instead of mysql_/mysqli_ functions.

Look into the way to "overtake" the ZC error processing for the time(s) when it would be desirable to have the EP4 log available/generated. This may mean that the method of error processing is swapped back and forth during operation as a possibility; however, should not do so during the actual processing to reduce the number of extraneous operations during large file processing.

Changing Activate Blank Featured doesn't work as expected

The use_function for EP4_ACTIVATE_BLANK_FEATURED incorrectly uses zen_cfg_select_options instead of zen_cfg_select_option.

Assign attribute import language dynamically based on DEFAULT_LANGUAGE

The file admin/includes/modules/easypopulate_4_attrib_detailed_ep.php includes several instances where it either in the notes or in the code references using the language_id of 1 as the default language id. Unfortunately, for stores where they have installed another language, removed the first language, and/or have identified a language_id other than 1 as the default, this characteristic is not carried forward. The result is as described below which also contains reference to the fix.

See: chaddro#34

Missing primary key on import not identified

When importing a product list while the primary key is not set to products_model, the applicable primary key that is missing is not identified.

Suggest modifying the error message for a missing primary key to reflect what primary key is being used.

Reduce some code complexity

When working on Easypopulate many years ago and to support the ability of the software to work on multiple systems and retain features that had originally been added, it was "necessary" to provide some sort of means to switch between the two types of mysql database handlers of the mysql_ and mysqli_ styles.

It would reduce the complexity of the code if the needed process to continue to support operation could do so with less extra/unique code. Ultimately reducing to Zen Cart code would make it as compatible as possible, but there remain a few features that would need to be reworked to support that in its entirety. Once those features are reworked, then the query style can be reverted to standard Zen Cart features.

Additional info needed on main screen

The primary key isn't displayed nor is the Import override option. This additional information could be helpful for support options and for user's understanding of operation.

Query logging as an admin option setting

There is an option within the main file that when set to true will log the queries that are executed, but it requires modifying the file. It would be nice if this could be made controllable from the admin. Seems that such a feature could be added to the debug logging configuration option.

SBA sorted Ascend Export is incomplete

Seems that an issue is preventing the export of SBA stock in an ascend sorted order.

name_max_len related variables

There are several specific variables declared to address the maximum length of characters permitted for the associated variable. Suggest that instead of having X variables that there be 1 variable with X parameters (array). This seems like it will/would make the code more compact and possibly easier to "read" as it were.

category(ies)_delimiter

The export and import files each contain a variable to address delimiting categories^sub_category^sub_sub_category.

This variable has a different name in each file (category_delimiter in one and categories_delimiter in the other).
Need to 1) pick a name,
2) move name into easypopulate_4.php
3) provide a means to "easily" modify the categories delimiter, although a similar issue may exist as with issue #10 and ensuring that the categories field is consistent across each use of the code ("one" person may prefer the ^ another on the same system may prefer a triple dash (---) and if one person uses one method on another's file, then there would be an issue. Part of addressing this is 1) again provide a means to preview the file, 2) leave the option hard-coded with instruction, 3) other options as suggested. :)

Products names and descriptions are mandatory on priceQty file ?

Importing the priceQty file, now it seems that the product names / description have to be present.
PHP Warning: array_key_exists() expects parameter 2 to be array, null given in C:\xampp\htdocs\vhosts\testes.local\admin\easypopulate_4_import.php on line 1946

 if (isset($v_products_name[$lang_id]) || array_key_exists($lang_id, $v_products_name)) {
                  $_POST['products_name'][$lang_id] = $v_products_name[$lang_id];
                }
                if (isset($v_products_name[$lang_id_code]) || array_key_exists($lang_id_code, $v_products_name)) {
                  $_POST['products_name'][$lang_id_code] = $v_products_name[$lang_id_code];
                }

Is this correct ?

Delete notification

The delete process doesn't include a confirmation step.

Having a delete verification step in the process would add a level of assurance that the right file was being deleted.

Recommend that when the selection to delete is made that the file list be presented again with the selected file to be deleted highlighted and the confirmation at the bottom of the list. By presenting in this fashion, it is easier for the individual to recognize that the correct file was selected. Further it seems that an option to download/view the file should be available. Not sure yet if it should only be the selected file, or all files that should be made so available.

Ideas?

Admin path recognition

There are entries that a user/store owner could make regarding the file storage location that appear are not fully addressed by the code that checks for the presence of the admin directory. Ie. the use of features like: ../.././up_a_path/up_a_path/admin are not "factored" into the equality check. There is code in ZC that will help address this and it ought to be addressed in the upcoming version to support ZC 1.5.5 and reduce the possibility of a user entering the admin path into the database.

Download by POST?

Currently files are downloaded by clicking on a hyperlink. As part of this, the directory needs to allow download of .csv files which is more of an issue when the files are placed in a folder in the admin directory because of the .htaccess in the admin directory than it is in the catalog side where no parent .htaccess is provided by ZC (Doesn't prevent store from including one and therefore needing to again have an .htaccess file in the EP4 sub-directory.

At any rate, it could be more beneficial to have a post version of the download as compared to a standard a tag link. Consider review/incorporation of the ZC catalog side for download processing as an option even if it is to directly link that header instead of something new/additional in the admin side. This way any modifications applied to that functionality are only needed once (ie. change of by link folder permissions from 0777 to 0755).

Export is not limited to just one product entry per export.

In development of EP4, @chaddro had gone above and beyond with the export data to be able to show and present all "variations" of a product (ie. to include the categories in which it was linked); however, this has also proven to some to be confusing because of the "extra" effort necessary to support import without the product "losing" its newly modified information. (Last entry for a product wins the place in the database.) It's a relatively simple concept, but requires more thought/planning.

Anyways, for those that would otherwise routinely remove the additional product listings, this is an extra step for them or if they go to update the product in some way they may make the edit in the first instance of the listing. It seems like it may be of benefit (though adding yet another option/capability) to export the product based on the product's master_categories_id. For those that have perhaps otherwise already figured it out, it isn't difficult to add, it's just another possible option from which to select.

Proposing an additional selection list item to click on (full export related) and an additional option in the dropdowns/dropdown sequence. (May want to use a checkbox instead of dropdown selection depending on where all that linked product play a factor in the export.)

install debug log: PHP Warning: opendir(ADMIN/temp/): The system cannot find the file specified

PHP Warning: opendir(ADMIN/temp/): The system cannot find the file specified in ADMIN\easypopulate_4.php on line 918.

New install, creating configuration, when no ADMIN/temp directory.

Spaces not ignored in custom fields entry

Users are permitted to add additional fields from the products table to be exporting/importing the full products related data (drop down of products included); however, the code would not necessarily provide a match to the field if a space existed between a comma and the field name. Recommend trimming trim($variable) each individual field between the separator ,.

Support export of product without manufacturers

It seems that it may be useful to be able to identify items that do not contain data assigned to a field about as much as it is useful to identify/export data that specifically meet a condition. Of course, one could always export everything, and then filter the data on the field/data that is desired and then correct/modify whatever is desired. But, might as well add an option to the menu to reduce some of the operations to help handle the data, right?

null imported when import string contains NULL

In an effort for ZC to provide equivalent operation, a level of sanitization, and be able to better handle/address conditions such as an individual having the last name of null, a new "datatype" was introduced of stringIgnoreNull where a string that only contained the word null would be prepared for the database to be stored as a word and not as a null value. The problem is that there is no direct test for the presence of that "datatype" within the store software, but there is a way to test against the functions, not cause an error and determine the condition of the database and it's support of the datatype.

Testing for this aspect can be done by testing against a particularly formatted string that is identified as a string. If the result is of a particular format, then the "datatype" stringIgnoreNull is available for use. The thing is that cases where it is desirable to return the value of null only, then some up front testing should be performed where the review likely would be to validate that the entire content was only the word and not that it just contains the word somewhere within. Anyways, some thought consideration is needed.

Discovered this as an issue when I exported a database, imported that data, and then again exported it. In comparison between the two databases, I was able to identify that a products_description was cleared because the content had the word NULL in it. This is even though there was far more content than just the one word. So, yes it would be good to benefit from the sanitization where/when needed but to be able to apply the stringIgnoreNull type intelligently.

Basic Attribute Import file needs additional sanitization

When updating the core EP4 files, queries were modified/replaced with the use of $db->bindVars to offer data type sanitization when preparing the data to be stored in the database. The easypopulate_4_attrib.php file was overlooked in this revision. The file needs to be updated/modified to incorporate the use of bindVars or as applicable to force the data to be of the expected datatype (ie. (int) where applicable)

Update Instructions

Instructions need to be updated for EP 4.0.32 which has been updated.

csv delimiter is hardcoded

Some have expressed interest in having the CSV delimiter be a different "character". This could be accomplished by providing options within the admin as a configuration. Bigger problem though is that if one method is chosen and a file is provided with a different method, then what "error" checking is to be performed? How can that be "self-detected" to switch to another style with all of the "variability" allowed in the import file such that the exact columns/fields present can be determined.

Options might include: 1) some form of preview option to then test/suggest the CSV delimiter based on viewed results, 2) testing of delimiter on the first row to see if at least two fields come back (minimum of two required to perform an update, one as the primary key the second a field to be updated), 3) provide some notification to the user that the number of fields in the first row is not as expected because of the chosen delimiter (again minimum 2 required), 4) do nothing other than ensure documentation identifies where the delimiter is defined.

Full product import missing primary key generates error

If the primary key is missing from the import file, then upon import a sql error was generated when attempting to select the existing data for each line of the import file. This skips processing the individual lines of the file if the primary key is not in the file. No backup primary key is provided/attempted.

make dual pricing mod field optional

The dual pricing mod operation supports the inclusion of a products_attributes field options_values_price_w and is populated via EP4 through the detailed attributes operations. When originally incorporated, to maintain field values in a database that had that field, the field had to be imported with each detailed attributes import. The additional forethought of not including the field with an import had not been considered. The result of not including the field in the file was that if the database had that field, then upon import it would become blank for each record imported.

Need to add a condition to prevent removing the data in the field when the field is not included in the import file.

easypopulate_4.php links and forms not using ZC standard

The clickable links to initiate exports and one form used hardcoded information instead of the admin side ZC builtin functions of zen_draw_form and zen_href_link. At least these should be updated.

Language file load does not fallback to English

An example of this is that line 100 of the admin/easypopulate_4.php file loads/requires the language file from the current language directory only and doesn't check for its existence before trying to do so. The proper way to do this is to check for the existence of the language file, and if not found to load the "default" language file (typically seen as english, but should use the same fallback functionality implemented in ZC.)

Improve Upload file header information

When a file is uploaded, information is presented in the header along with a button to try to import the file. Problem is that this button is non-discriminatory in that if the uploaded file has a problem, the user is not notified until after attempting to import the file and review the results. This is compared to the information that is presented in the file list which may indicate that there is a problem with the delimiters or the file type.

So, giving the user more information about the uploaded file would make use of this import button clearer.

Primary key not exported when not products_model

With the addition of an alternate primary key, the primary key does not automatically get added to the export list and requires manual entry as a user defined field.

Recommend adding primary keys other than products_model to the export list so that it will appear one time in the file if it is/was added as a user defined field also.

Rename deutsche language directory

The language directory for german was labeled deutsche in consideration that one may either change the language specifier or establish german as deutsche to suit native spelling. That said, a comment was made in issue chaddro#32 that it should be named german instead.

At next push, rename the admin/includes/languages/deutsche folder to admin/includes/languages/german.

Expand operation capability with language code

The original development of this plugin used the language_id (a numeric number that corresponds to a record in the database but not to a "real-world" human readable value). It means that if an export of one database is used as the import of another database that the data may not line up and in fact it may be possible that one language would overwrite another.

By using the language_code (ie. English - en, German - de, etc...) then the export of english from one store will map to the english of another store provided the language itself is installed/mapped to that letter "pairing".

v4.0.36.ZC no default categories_description on import

Identified in the EP4 forum as directly affecting the use of EP4, though encountered on an upgrade to ZC 1.5.6 that when importing a product to a new category that an error occurs as a result of the categories_description field in the categories table not having a default value. EP4 has not previously provided a "default" value, but generally speaking it hasn't been noticed/announced otherwise it would have been addressed. It is already incorporated into the next release, but there are other issues still to be addressed before releasing it.

Not all EP4 "core" filenames are defined

Files under consideration are:
easypopulate_4_export.php
easypopulate_4_import.php
easypopulate_4_split.php
easypopulate_4_attrib.php

These files are included/required and would expect that their definition would include .php unlike other cases/uses of a filename being defined.