lselzer / metageta Goto Github PK

Modify runtransform.py to use the new GetArgs class. Will require adding a
DropList type to the getargs module.

Running crawler/transform repeatedly from the commandline results in a "The
input line is too long." error

I'm not able to run the crawler in 1.3.6 release, no issues with 1.3.5

1. run metaggeta-shell.bat
2. from prompt run runcrawler.bat
3. python interpreter cmplains about import of formats modules , see attached 
picture for details.

Regards

Stefano

Overview generation from byte ESRI binary grids fails

NoData values are not always set on self._gdaldataset bands. This skews
stats and histograms causing poor overview image quality.

Don't close dialog box if "keep open" is checked when Ok is clicked on the
progress logger.

Build a proper pythonic setup.py

16 bit record length in xls may be limiting - openpyxl supports larger number 
of output records xlsx. For large inventories this would be an advantage.

To reproduce:
* Lock an existing shapefile (have it open in ArcGIS or another MetaGETA
process)
* Call runcrawler with the --gui arg and -s = above shapefile.
* runcrawler process crashes, progresslogger process keeps waiting.

Enable checking of crawl results (reference existing .xls) during walk 
process for new and modified datasets in target directory.

Results may be appended to the existing crawl result or created as a 
new .xls - so that changes are maintained - depending on the users 
requirements i.e. a checkbox 'append existing' or 'create new'

Add an option to export directly to XML/MEF, skipping the XLS spreadsheet.

The GUIDs generated may not be backwards compatible with those generated by
MetaGETA <= 1.2. This is because those GUIDs (which are meant to be
reproducible based on filepath) were based on paths that may not be
constant - such as non-UNC and mixed case filepaths in Windows.

This will be an issue if we want to update existing crawl results.  Though
I have implemented a little hack in runcrawler to work around this.

For example, assume U:\ is mapped to \\server\share and V:\ is mapped to
\\server\share\subfolder. U:\subfolder\test.tif is the same file as
V:\test.tif but a different GUID would be generated. Also, GUIDs based on
V:\test.tif are different to those based on v:\test.tif (and other case
variations).  

The GUIDs from now on should be constant on Windows as they are "normcased"
and converted to UNC (I don't know enough yet about mount points and links
on *nix to implement something similar). 

There may still be an issue with datsets stored on moveable media, etc. I
think we need to look into getting disk IDs.

[deleted issue]

When updating a crawl result, create overviews when requested if a dataset is 
not modified and does not already have overview images.

What steps will reproduce the problem?
1. runcrawler.bat <no args>
2. Browse to data dir
3. Browse to existing xls
4. Delete .xls and copy rest of path
5. Paste rest of path as shp and log files
6. error occurs in "for ds in Crawler"

Incorrect histograms for 8 bit greyscale images as binsize is hardcoded to
be one but histogram min/max is not always 0,255.

None of the following work:
./runcrawler.sh -d '/mount/dir/some dir' ...
./runcrawler.sh -d "/mount/dir/some dir" ...

Download is missing Python26\Lib\encodings\mbcs.py

Refers to <=1.2 usage

Problem: gui popup prevents command line execution of runtransform

Example execution: runtransform -x 
/g/data1/v10/metageta/rs0_FC/1986-08/FC25_V0.0_1986-08.xls -d 
/g/data1/v10/metageta/rs0_FC/mef --nogui

Additional to this the xsl is missing in (source version?) @ 
metageta/1.3.9/lib/python2.7/site-packages/metageta/transforms/

No filelist or filesize values for ESRI Grids

Histogram statistics for overview image stretch may be generated from 
subsample image rather than original scene at full resolution to increase 
throughput.

Create overview image without stretch (scale image values to 8bit to 
retain dynamic range)
Calculate values for histogram on overview
Generate overview and thumbnail

Run transform and examine results. Search for thumbnail reference. In other 
examples thumbnail and quicklook references appear below maintenance frequency 
info. Thumbnail and quicklook are included in mef - just not referenced so 
appear null in GN view.

The function that checks whether an existing metadata record needs updating 
will always return True if the quicklook is stored with a relative path in the 
spreadsheet. 

What steps will reproduce the problem?
1. Insert MEF to Geonetwork
2. Validate sample record in geonetwork
3. Note 4 errors due to date omission in data quality fields

What is the expected output? What do you see instead?
Field is null - use date of metadata transform if no date exists

overviews.stretch('NONE', ...) truncates data > 8 bits instead of rescaling

Add support for Pleiades HR DIMAP V2.0

This will be added once GDAL supports it - 
https://trac.osgeo.org/gdal/ticket/4826

Steps that will reproduce the problem?
1. crawl folder containing partially written ECW - typically tmp files with an 
ecw prefix are located in the write directory.

Once incomplete file in encountered crawler suspends work

Ideal is for crawler to identify partially written file and skip to the next 
file.

Support vector data.

No startup errors reported to gui - e.g. failure to set GDAL_DATA 

The "updated" check only tests last modified date of the image, not related
files.

This stops runcrawler.py from updating metadata for images that have had
modifications such as adding a projection definition in an external file
(aux, ers, prj, etc...).

Assigning a NoData value stuffs up assigning colour table entry to pixel value 
assignment.

GDAL segfaults when reading NITF images containing JPEG2000 compressed
subdatasets.

GDAL 1.6.3 (OSGeo4W) with gdal_ECW_JP2ECW.dll plugin.

This issue has only been reported on 32-bit Windows Vista. Can't reproduce
on XP.

The following error is generated when the GetArgs dialog is called by MetaGETA 
apps:

{{{_tkinter.TclError: unknown color name "{#c3c3c3}"}}}

This is a problem with the version of Tix that ships with Ubuntu:
https://bugs.launchpad.net/ubuntu/+source/tix/+bug/371720

The workaround is to upgrade your version of Tix to 8.4.3 from the following 
PPA:
https://launchpad.net/~portis25/+archive/ppa

Add ability to extract metadata from rasters in zip and tar.gz/bz2 archives now 
that this is handled transparently (for some formats...) in GDAL.

ValueError: row index (65536) not an int in range(65536)

ExcelWriter.__addsheet__ is called but the row count is not reset.

What steps will reproduce the problem?
1. Run crawler in update mode on an existing xls which contains records for 
datasets that have been deleted.

Standard pywin32 install puts a pywintypes25.dll in C:\Windows\System32. If
the build is different to that included in the MetaGETA package, a dll
conflict occurs.

To do:

Check if having a later version in C:\Windows\System32 than is included in
MetaGETA solves the problem. E.g try installing build 214
http://sourceforge.net/projects/pywin32/files/pywin32/Build%20214/pywin32-214.wi
n32-py2.5.exe/download
as normal and installing an earlier build to the MetaGETA python
site-packages dir.
Also, check what pywin32 build comes with ArcGIS.

In runcrawler.py the option for arg -r is set to True as default.
this precvent it to be used as if you omit the -r in the command line it takes 
the default value of True. Itf you insert the -r in the command line it is 
again set to True.
I think line 326 should be patched from

opt=parser.add_option("-r", "--recurse", action="store_true", 
dest="recurse",default=True,

to

opt=parser.add_option("-r", "--recurse", action="store_true", 
dest="recurse",default=False,


Regards 

Stefano

It seems like the gdal release included in MetaGETA 1.3.6 setup was compiled 
with MySQL support but libraries was not included.

Steps to reproduce issue:

1) run metageta-shell.bat
2) at prompt run gdalinfo
3) System shows error message added as screen capture (sorry it is in italian 
as my system setting but it should be clear enough).

Enable crawl of hard media in sequence without crawler shutdown

Ask user at the end of crawl if they wish to "crawl another?"

Provide text box for user to enter media id (default blank) - the values 
entered here should persist following completion of the crawl for easy 
update by the user. i.e. if user is cataloguing a sequence of CDs 101 to 
200 etc.

For tranform: -

Include MEDIA ID in supplemental information (or preferrably in 
appropriate metadata field i.e Offline Media - offlineMed "Name of media 
on which the resource can be found") Include the MEDIA ID in the ABSTRACT 
for searchability in Geonetwork.

Include medianame in transform xsl to populate MediumName in TransferOptions 
from xls spreadsheet. Defaults to DVD but should be able to accept TAPE and 
other media..currently hard coded.

update documentation to include mediaid - http://metageta.googlecode.com
/svn/trunk/doc/files/transforms-module.html

Some NetCDF subdataset bands GetStatistics == [0.0, 0.0, -1.#IND, -1.#IND]
GDAL error is "Failed to compute statistics, no valid pixels found in
sampling".
Calling GetHistogram on such bands causes gdal to segfault.

When moving crawl results around you have to update the file location for 
quicklooks and thumbnails in order for the transform to bundle the image in the 
mef. Maybe the file path is not even required for the bundling to work and the 
quicklooks are written with the xls?

Add temporal extent element to ANZLIC XSL

Current implementation is a complete kludge and hard to maintain.
NB: multiprocessing is only available in Python 2.6+

*What steps will reproduce the problem?*
1. Run crawler and generate a new XLS
2. Run crawler over a folder that contains a tif with a non-ascii character
in its metadata, updating the above XLS
3. Profit???

*What is the expected output? What do you see instead?*
UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 87:
ordinal not in range(128)

Remove remaining .bat/.sh env vars (i.e CURDIR) so users with their own
python/gdal install and appropriate environment can just run the python code.

Transformed xml with mediaid field does not validate in GN 

Compliance to metadata standard (XML Schema) (1 errors) 
 cvc-type.3.1.2: Element 'gco:CharacterString' is a simple type, so it must have no element information item [children]. (Element: gco:CharacterString with parent element: gmd:abstract)

Add script to extract metadata from single images.
Something simple like:


{{{
import sys,formats
indataset=sys.argv[1] #better arg parsing required
outfile=sys.argv[2]
#some sort of arg to specify xls or xml output
#some sort of arg to flag overview generation

ds = formats.Open(f)
md=ds.metadata
qlk=ds.getoverview(outfile+'.qlk.jpg', width=800)
md['quicklook']=qlk
#write metadata record to xml/xls...
}}}