jmunzer / bulk_tools Goto Github PK

All List Item Report from Talis Aspire contains a URL. We could remove a pre-processing step which is imposed on users by having the code allow and detect wither a list item URL or a bare GUID.

i.e. except input in the first column that looks like any of these:

http://tenancy.rl.talis.com/items/abcd1234-abcd-efgh-adbc5432
http://tenancy.rl.talis.com/items/abcd1234-abcd-efgh-adbc5432.html
http://tenancy.rl.talis.com/items/abcd1234-abcd-efgh-adbc5432.html?some=params
abcd1234-abcd-efgh-adbc5432

This could be done in the while loop that reads the line, having a function that checks the first column's value and detects whether it needs to extract a guid or not.

The same could be done for lists and any other Talis identifiers handled by the API which could exist in those different forms.

Basically this is about making the scripts more forgiving.

Hi,

I've got the tool set up in docker in linux mint, and I ran into file permission issues, when running the URL bulk change tool.

I'm not an expert docker user, but it looks as though the problem is that the docker image is running apache as apache2, and this user doesn't have write permission to the directories under the directory in which the git repository is cloned (they appear to be 755). The initial error looks like:

Starting...

Tenancy Shortcode set: ***
Client ID set: ***
User GUID to use: ***
Writing to live tenancy?: false
Logging Level Selected: DEBUG
File uploaded:
Warning: move_uploaded_file(../uploads/test.csv): failed to open stream: Permission denied in /var/www/html/url/src/comp.php on line 60

Warning: move_uploaded_file(): Unable to move '/tmp/phpVbQR67' to '../uploads/test.csv' in /var/www/html/url/src/comp.php on line 60
File is invalid and failed to upload - Please click back and try again.

When I changed the file permissions in the url/uploads directory to 777, the first error was fixed:

Starting...

Tenancy Shortcode set: ***
Client ID set: ***
User GUID to use: ***
Writing to live tenancy?: false
Logging Level Selected: DEBUG
File uploaded:
INFO: File is valid, and was successfully uploaded.../uploads/test.csv

Warning: fopen(../../report_files/url_output.csv): failed to open stream: Permission denied in /var/www/html/url/src/comp.php on line 72
Unable to open url_output.csv

And when I finally changed all the permissions of subdirectories to 777, the tool ran without an error.

Regards,
Simon

Would be useful to be able to update notes fields - we have been talking about trying to add notes for temporary ebook collections where the list editor hasn't thought to do so, and this would make sure that students or librarians using the list were aware (depending on which kind of note was added).

Originally posted by @smcleish in #24 (comment)

This would be handy when comparing debug log with the output log

Variable names are changed part way through the script to delete items. Line 88 and 90 and lines 107 and 108 are affected.

The error seen by users is:

Warning:  Undefined variable $etag in C:\xampp\htdocs\del\src\comp.php on line 107
Warning:  Undefined variable $assoc_listid in C:\xampp\htdocs\del\src\comp.php on line 108 

ERROR: There was an error deleting the item:
'{"errors":[{"status":"500","title":"Internal Server Error","detail":"CURIE localPart must be nonempty string"}]}'

It would be useful to have a blank line in the output log ahead of a new run. It would help separate runs in the combined log. It would be especially useful in the scenario where the script was terminated prematurely, I found the layout got messed up by this if the script was aborted in the middle of a row.

Be able to update:

• LCN
• DOI (Also need option to use DOI as online resource link)

Getting:
Notice: Trying to get property 'link' of non-object in S:\xampp\htdocs\bulk_tools\url_complete\src\comp.php on line 190
when running items with "online_resource": null in the json data.

Not sure if there is/was an intention to use the link property of "online_resource", but $online_resource isn't used anywhere else in the script, so this line could possibly be removed or updated to ignore property of non-object notice.

It would help to clean up the output.

Thanks,
Ian

shows List title, but could do with showing item title &/or student note for quick verification of item removed.

On the following two lines we are filtering out any input data which is not an URL:
https://github.com/jmunzer/bulk_tools/blob/master/url/src/comp.php#L139

If one of the inputs does not match the rules of FILTER_VALIDATE_URL then that value is modified to be false. This could lead to surprising results - if a row intended to be an "update" has its "old" value removed, then the script will instead perform an add - if it has the "new" value removed, then the script will perform a "delete".

We are beholden to the rules which FILTER_VALIDATE_URL have implemented here, and whilst it should work in most cases there's always the chance of an outlier.

I suggest removing this validation and instead trusting the input given by the user.

Optionally we could continue to filter the new value only if we wanted to ensure only valid URLs are uploaded, but in this case we should fail the row if a value for the new web address goes in, but false is returned - preventing the action silently being turned into a delete.

I don't know how to quantify the risk of FILTER_VALIDATE_URL filtering out things unexpectedly, but it feels like a moving part which isn't necessarily needed.

Warning: Undefined variable $importanceID in C:\xampp\htdocs\bulk_tools\new_acq\src\comp.php on line 177

Hi,

Using the bulk update tool to edit URLs recently, I found a couple of places where errors were thrown which were misleading.

When the tool contacts the tenancy to verify the information in user.config.php, it returns the error that the information in the file is incorrect if it receives any http response code other than 200, and also if the curl call fails because of network issues (which is what turned out to be the problem). It would be useful at least to distinguish between four cases:

a) issues which prevent curl from contacting the server (i.e. no HTTP response code received)
b) the return of http code 200
c) the return of a code which indicates that authentication has failed (presumably 401)
d) the return of some other http code (with a notification which includes the code to enable investigation)

I also had the error that the CSV file had the wrong number of columns, when it didn't. Most lines were processed correctly, but it stopped with this error on a line where the URL in the item was very long (over 500 characters). When this line was removed, the error disappeared. It would be useful to separate this case from the general column issue, assuming that the response from the API distinguishes this case from other reasons for not processing the line.

Cheers,
Simon

We have recently used the LCN Updater - which worked fine and saved us a lot of time.

We did get some error / warning text appearing at the bottom of the results screen (but not in the saved log). I'm not sure if this is just the tool getting confused by blank lines at the bottom of the csv; or something else needs updating?

Michael

Warning: Undefined array key 1 in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 175

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 175

Warning: Undefined array key 2 in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 177

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 177
no new LCN found. Removing from .
ERROR: There was an error getting the resource:
'{"error_code":405}'

Warning: Undefined property: stdClass::$data in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 160

Warning: Attempt to read property "relationships" on null in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 160

Warning: Attempt to read property "resource" on null in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 160

Warning: Attempt to read property "data" on null in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 160

Warning: Attempt to read property "id" on null in C:\Local\xampp\htdocs\bulk_tools-master\lcn_updater\src\comp.php on line 160
Resource:
ERROR: There was an error updating the LCN:
'{"error_code":405}'

It seems to be possible to set max_execution_time to be 0, and that will mean there will be no timeout for long running scripts.

On a live web server this would be dangerous, but as this will be running on people's local machines it's just down to how long they want to keep their computer switched on.

Background Reading:
https://stackoverflow.com/questions/7036767/how-can-i-get-infinite-maximum-execution-time-with-php
https://www.php.net/manual/en/info.configuration.php#ini.max-execution-time

For example, openURL retains web addresses. Does the oeprator want this?

Must rewrite tools to collate valid changes into an array of list IDs.

can be tracked via 'Review Jobs' in TARL.