Add http://pywps.org/feed.xml to http://planet.osgeo.org/

Hello,

I have a process I define the ComplexInputType as follow

    self.dataIn = self.addComplexInput(identifier = "variable",title="variable to average",formats = [{"mimeType":"application/json"}], minOccurs=1, maxOccurs=10)
    self.domain = self.addComplexInput(identifier = "domain",title="domain over which to average",formats = [{"mimeType":"application/json","encoding":"utf-8","schema":None},])

I would lime to send json

right now my execute does nothing

  def execute(self):
    self.status.set("Starting %i, %i" % (0,0),0)
    return

The url I'm sending is:

http://localhost:8000/wps/?version=1.0.0&service=wps&request=Execute&identifier=ensemble_averager&datainputs=[domain={'longitude': {'start': -180.0, 'end': 180.0}, 'time': {'start': '1980', 'end': '1982'}};variable={'url':'file://lgm/uvcdat/2015-01-26/sample_data/clt.nc','id':'clt'}]

The process errors with the following:

PyWPS [2015-02-04 00:50:00,386] WARNING: Could not import processes from 'processes': ImportError('No module named icclim',)
PyWPS [2015-02-04 00:50:00,388] INFO: Following processes are imported: ['ensemble_averager', 'test_process']
PyWPS [2015-02-04 00:50:00,396] INFO: Status [processpaused]: Getting input variable of process ensemble_averager
Traceback (most recent call last):
  File "/lgm/uvcdat/2015-01-26/lib/python2.7/site-packages/pywps-3.2.2-py2.7.egg/pywps/Wps/Execute/__init__.py", line 382, in
 __init__
    self.consolidateInputs()
  File "/lgm/uvcdat/2015-01-26/lib/python2.7/site-packages/pywps-3.2.2-py2.7.egg/pywps/Wps/Execute/__init__.py", line 530, in
 consolidateInputs
    resp = input.setValue(inp)
  File "/lgm/uvcdat/2015-01-26/lib/python2.7/site-packages/pywps-3.2.2-py2.7.egg/pywps/Process/InAndOutputs.py", line 393, in
 setValue
    self.storeData(input["value"])
  File "/lgm/uvcdat/2015-01-26/lib/python2.7/site-packages/pywps-3.2.2-py2.7.egg/pywps/Process/InAndOutputs.py", line 470, in
 storeData
    os.remove(fout.name+".base64")
OSError: [Errno 2] No such file or directory: './pywpsInputWPjsM5.base64'
PyWPS [2015-02-04 00:50:00,401] INFO: Status [processfailed]: [Errno 2] No such file or directory: './pywpsInputWPjsM5.base64
'

I did a tracking on this, it dies on the finally, but it really died in the try bit on this line:

                   base64.decode(f1, f2)

The exception says:

Incorrect padding

I'm not sure how to fix this, I'm hoping it's something in my url

Thanks

It would be valuable to setup something like demo.pywps.org to provide a canonical demo based on master, etc. in support of OGC compliance and as we strive to become a reference implementation.

datainputs=complex=href@http:// seems not to be parsed

Issue by jachym
Thursday Jan 28, 2016 at 13:51 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/84

make pywps.readthedocs.org run off the docs in https://github.com/geopython/pywps

Issue by jachym
Saturday Apr 04, 2015 at 10:46 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/48

Download (and later validate) complex and reference data

https://github.com/jachym/pywps-4/blob/master/pywps/app.py#L55

Implement live deployments listing that can be managed/edited over time by the community to help demonstrate adoption and implementation.

Issue by jachym
Sunday Jan 03, 2016 at 20:57 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/79

Currently, in PyWPS language "en-US" is hardcoded. For adding support for more languages we need:

• To define configuration of supported languages in configuration file
• Use possibly existing i18n python-based tool around and i18n gettext seems to be it https://docs.python.org/3/library/gettext.html#internationalizing-your-programs-and-modules

Should be relatively easy, guessed time: 40 hours

Add coverage testing metrics as part of CI testing.

Issue by jachym
Tuesday Jan 12, 2016 at 20:46 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/81

does not seem to be implemented anywhere, still useful config value

Add PyWPS as a technology entry in the OSGeo Service Provider directory.

Overview

There seem to be numerous places for documenting contributors:

1. http://pywps.org/contributors/
2. https://github.com/geopython/pywps/blob/master/CONTRIBUTORS.md
3. in the codebase in https://github.com/geopython/pywps. Running git log|grep "Author: "|sort -u shows a historical list (some duplicates)

Requirement

OSGeo Incubation will require us to get contributor agreements in place for present and past contributors to the project. Sorting this out also makes good sense to reduce duplication and manage a central contribution manifest.

Proposal

1. update http://pywps.org/contributors/#individuals to simply refer to https://github.com/geopython/pywps/blob/master/CONTRIBUTORS.md as the authoritative individual contributors list. Everything else (i.e. Institutions) stays the same
2. update https://github.com/geopython/pywps/blob/master/CONTRIBUTORS.md to have a structured table of contributors past and present. The list will be made up from grepping codebase git logs as well as https://github.com/geopython/pywps/graphs/contributors. https://github.com/geopython/pywps/blob/master/CONTRIBUTORS.md will then also form the basis for gathering contribution agreements over time

I am getting

OSError: [Errno 24] Too many open files

After leaving my pywps server running for a few days. I am running pywps under wsgi and calling it ~1500 times per day.

After investigating, it seems that everytime that pywps calls tempfile.mkstemp an open file handler is left lying around unclosed[1].

According to python's documentation, mkstemp returns a tuple with an already opened file and the path to the file. In pywps we are never using the file handlers (only the file paths) and we never close these open files!

In a wsgi application this becomes a serious problem, because wsgi does some caching of running processes.

When using apache's mod_wsgi, the process is eventually restarted, causing all file descriptors to be closed. However, if your usage happens to be intensive (as seems to be my use case) there is a chance that you'll run out of file handlers to use.

I'll provide a Pull Request for solving this in the coming days.

[1] - To reproduce the problem, and assuming you get pywps running under wsgi, try the following:

# reload apache in order to start fresh
sudo service reload apache2

# extract the pid of the wsgi process
ps aux | grep [w]sgi

# check that there are no tmp* files opened by the wsgi process
sudo lsof -p <wsgi_process_pid> | grep tmp

# now run one of your WPS processes

# recheck if there are any tmp* files opened by the wsgi process
sudo lsof -p <wsgi_process_pid> | grep tmp

# there should now be one more tmp* file still open
# everytime you re-run a WPS process another tmp* file will be opened
# until sometime when apache's mod_wsgi decides to clean its cache

Issue by jachym
Sunday Jan 12, 2014 at 02:03 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/10

ComplexData output via MapServer

add https://openhub.net/p/PyWPS to https://openhub.net/orgs/OSGeo on Open HUB.

@jachym as per http://lists.osgeo.org/pipermail/pywps-dev/2016-February/000469.html, can you upload PyWPS 3.2.4 to PyPI based on tag pywps-3.2.4 (i.e. python setup.py bdist_egg upload).

Thanks

cc @kalxas

Description

Add PyWPS to the OSGeo Live project. Requirements:

• implementation/packaging/setup script(s)/configuration(s)
• documentation (overview, quickstart)

Environment

• operating system:
• Python version:
• PyWPS version:
• source/distribution
  • git clone
  • Debian
  • PyPI
  • zip/tar.gz
  • other (please specify): OSGeo Live
• web server
  • Apache/mod_wsgi
  • CGI
  • other (please specify):

Steps to Reproduce

NA

Additional Information

Related issue #114, we can start with 3.x and move towards 4.0.0 when final/stable.

Initial implementation:

• overview: https://github.com/OSGeo/OSGeoLive-doc/blob/master/en/overview/pywps_overview.rst
• quickstart: https://github.com/OSGeo/OSGeoLive-doc/blob/master/en/quickstart/pywps_quickstart.rst
• project logo: https://github.com/OSGeo/OSGeoLive-doc/blob/master/images/project_logos/logo-pywps.png
• test installation: https://github.com/OSGeo/OSGeoLive/blob/master/bin/labs/install_pywps.sh

cc @kalxas, @khosrow, @mipilon

Description

Add PyWPS latest stable to UbuntuGIS lifecycle.

Additional Information

Initial progress: https://launchpad.net/~gcpp-kalxas/+archive/ubuntu/osgeolive/+packages

Issue by Noctalin
Monday May 11, 2015 at 13:49 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/56

During file creation for status documents, referenced input, ouputs, temp files etc PyWPS should check if enough storage space is available otherwise raise NotEnoughStorage exception.

With recent release of 2.0 standard, I am mainly interested in async support which is currently completely missing in 1.0. It is based on a statefull (persistent?) processing queue with query/status/result capabilities and a multi-processing/threads pattern. Is there anyone already worked/working on such an improvement?

As part of incubation we need to clarify contributions to PyWPS. Task list:

• clarify with OSGeo Incubation Committee what level of contribution agreements are required
• add a .github/CONTRIBUTING.md file to PyWPS source code repo (example: https://github.com/geopython/pycsw/blob/master/CONTRIBUTING.rst)
• if required, update .github/PULL_REQUEST_TEMPLATE.md with contribution agreement

when running an async process the logger does not log to the configured logfile.

In Pywps.setLogFile() method logging.basicConfig() is called according to the configuration. But in the async case somehow the logger has already a logging handler and calling basicConfig() has no effect.

Patch #49 provides a workaround to remove logging handlers in the async case before calling basicConfig()

See also #28.

Add templates to help improve issue and pull request workflow.

logging is made via

logging.error("bla")

I would suggest adding the capability to use "loggers" rather than logging directly.

I'm trying to use pywps with multiple instance of pywps on the server and it gets confised

I would recommend replacing the logFile in the init with a logger keyword that would the be used to log, allowing to log simultaneously to multiple logs.

A string with the logger name is actually sufficient you can retrieve the logger via

logging.getLogger(name)

Let me know if this makes sense. Thanks for the great work!

Issue by jachym
Thursday Sep 24, 2015 at 08:43 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/68

Scheduler of accepted processes and process queue.

Also balancer for horizontal scaling would be nice.

Hi,

PyWPS 3.x changes permission on referenced output file, see [1]. In the result the user who runs the process (usually www-data) cannot delete this file.

 -r--r--r-- 2 www-data www-data 57590190 Aug 25 14:22 /tmp/basin_20252/basin.zip

Why PyWPS behaves like that? Thanks, Martin

[1] https://github.com/geopython/PyWPS/blob/master/pywps/Wps/Execute/__init__.py#L1135

Given 3.2.2 is the current latest stable release, looks like PyWPS' PyPI presence is out of date.

• upload 3.2.2 to PyPI
• update http://pywps.org/download with PyPI link
• ensure release process includes PyPI upload

It seems to me that 52379c5 completely breaks pywps. It produces

Traceback (most recent call last):
  File "/home/landamar/src/PyWPS/wps.py", line 86, in <module>
    wps = pywps.Pywps(method)
  File "/home/landamar/src/PyWPS/pywps/__init__.py", line 179, in __init__
    config.loadConfiguration(configFiles, environ)
  File "/home/landamar/src/PyWPS/pywps/config.py", line 90, in loadConfiguration
    _overrideConfigFromConfigDict({k.split("_",2)[-1]:v for k,v in environ.iteritems() if k.startswith('PYWPS_CONFIG_')})
AttributeError: 'str' object has no attribute 'iteritems'

Issue by jachym
Thursday Sep 24, 2015 at 08:45 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/69

Each process execute method should get private safe container before executing. With limited RAM, CPU cores, disk space, limited network communication and other. Possibly using Docker or Vagrant or even something better.

Description

Unit tests in a Python 3 environment yield ResourceWarning warnings like:

/home/travis/build/geopython/pywps/tests/test_inout.py:64: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpwbpvbq0c' mode='rb'>

Environment

• operating system: Ubuntu
• Python version: 3.4
• PyWPS version: master
• source/distribution
  • git clone
  • Debian
  • PyPI
  • zip/tar.gz
  • other (please specify):
• web server
  • Apache/mod_wsgi
  • CGI
  • other (please specify): unit test runner

Steps to Reproduce

# in a Python 3 virtualenv
python -m unittest tests

Additional Information

Initial discussion in #106.

Example can be found in current travis builds: https://travis-ci.org/geopython/pywps/jobs/111190874#L360

When going to http://pywps.org/download/ and clicking All Releases 3.2.3 does not show up on GitHub releases.

I believe this is because there is no release tag per se against 3.2.3, and that the download links:

https://github.com/geopython/pywps/archive/pywps-3.2.3.tar.gz
https://github.com/geopython/pywps/archive/pywps-3.2.3.zip

are actually based off the 3.2.3 BRANCH, not TAG.

Can we tag 3.2.3 off the 3.2.3 branch to make it a true GitHub release?

Moving forward, once we merge pywps-4 branch into master, we can branch pywps-4.0 once 4.0.0 is final.

reimplement process discovery as it was in pywps-3

Issue by jachym
Wednesday Jun 18, 2014 at 20:25 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/22

More advanced process control: PAUSE, RESUME, CANCEL

Make GDAL optional

this basically requires slight modification of validating code

Lang.setCode is not used, process does not know anything about language, the client is requiring

https://github.com/geopython/pywps/blob/pywps-3.2/pywps/Parser/Execute.py#L593 throws error

on

python wps.py "request=Execute&service=WPS&version=1.0.0&identifier=r.getslope&datainputs=dem=Reference@xlink:href=http%3A%2F%2Flocalhost%3A8080%2Fgeoserver%2Fwcs%3FSERVICE%3DWCS%26VERSION%3D2.0.1%26REQUEST%3DGetCoverage%26COVERAGEID%3Dtiger__demrepro%26FORMAT%3Dimage%2Ftiff&responsedocument=slope=@asreference=True"

Implementation of http://opengeospatial.org/standards/wps 2.0.0

Issue by jachym
Monday Mar 31, 2014 at 22:42 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/13

It should also be used in the future for new WPS requests, like "pause", "stop" etc.

Update Google+ presence with the following updates:

• logo should be new PyWPS logo
• website should be http://pywps.org
• description should be PyWPS is an implementation of the Web Processing Service standard from the Open Geospatial Consortium. PyWPS is written in Python. (as per https://github.com/geopython/pywps/wiki/Templates#project-description)

Issue by SiggyF
Sunday Nov 08, 2015 at 21:36 GMT
Originally opened as https://github.com/PyWPS/pywps-4/pull/72

I have formatted the code according to pep8 and removed unused imports. I used 132 as maximum line length (set in tox.ini). Lines with # noqa are ignored (long lines that should be long, imports for namespace flattening). Code now gives no warnings when using flake8 (pep8 + pyflakes).

SiggyF included the following code: https://github.com/PyWPS/pywps-4/pull/72/commits

Issue by jachym
Tuesday Jan 12, 2016 at 20:37 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/80

maxoperations config value does not seem to be implemented in the code

suggest to implement it as "maxparaleloperations" and prepare for future "maxstackoperations"

https://github.com/geopython/PyWPS/blob/master/RELEASE.howto

Tried building the HTML docs with "make html", but got errors. Changing line 20 of
https://github.com/geopython/PyWPS/blob/master/doc/source/_templates/layout.html from

<li><a href="{{ parent.link|e }}" {% ifloop.last %}{{ accesskey("U") }}{% endif %}>{{

to

<li><a href="{{ parent.link|e }}" {% if loop.last %}{{ accesskey("U") }}{% endif %}>{{

(note "ifloop") fixes this and the docs build. Could make a pull but this is really minor.

Issue by jachym
Monday Jan 25, 2016 at 11:34 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/83

http://docs.pycsw.org/en/latest/configuration.html

https://trac.osgeo.org/osgeo/ticket/1625

Issue by mgax
Monday Oct 21, 2013 at 18:50 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/6

I'm experimenting with handling the contents of an Execute request in an iterative fashion, so that we can handle requests with a large embedded data file. My idea is to parse the request iteratively, save any ComplexInput in a temporary file, remove it fron the XML, and build an in-memory document out of everything else.

I've hit on this roadblock: how to write XML files efficiently, as a stream, from Python? lxml has nice support for iterative parsing, but not for writing. There's the built-in xml.sax.saxutils.XMLGenerator but it's incredibly slow, writing just over 1MB/s on an Intel I7 machine.

Issue by jachym
Saturday Dec 26, 2015 at 19:18 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/75

LiteralData do support allowedValue, anyValue (implemented)

but also

noValues and valuesReference - OGC OWS 06-121rc spec, table 38, page 90

Wiki URL has duplicated protocol part: http://http://wiki.rsg.pml.ac.uk/pywps/Main_Page

Issue by jorgejesus
Tuesday Apr 01, 2014 at 06:55 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/16

PyWPS-4 can handle WSDL/SOAP interface, which will probably be ported from 3.x branch

Issue by Noctalin
Monday May 11, 2015 at 13:53 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/57

According to OGC 05-007r7 the server needs listen to a request for WSDL document. (Annex E)

http://hostname/WPSname?WSDL