Feature request: Extension of regex filtering to extract data about urlwatch HOT 7 CLOSED

grep and grepi can be used directly, so you could do something like:

filter:
  - grepi: 'price: <span>.*</span>'
  - re.sub:
      pattern: '^.*(price: <span>.*</span>).*$'
      repl: '\1'

from urlwatch.

findall might be easier though, what are you thinking for the output, just put each match on a new line?

from urlwatch.

findall might be easier though, what are you thinking for the output, just put each match on a new line?

yes, that's what I was thinking too.

I didn't check in the source yet, how it is implemented, but it felt like, it could be easier integrated. But maybe the similar names of re.sub in urlwatch and the re package fooled me.

from urlwatch.

Yes, its not hard to add it. A little more than re.sub because you have to do something with the matches, where re.sub will just give you the string to return.

https://github.com/thp/urlwatch/blob/master/lib/urlwatch/filters.py#L831

from urlwatch.

I actually have a couple of places that this would simplify my filters, so have put in an implementation in #805. See what you think.

from urlwatch.

For filtering out HTML elements, use the CSS or XPath filters. Never use regex.

from urlwatch.

For filtering out HTML elements, use the CSS or XPath filters. Never use regex.

For me, this was not the intention here. It's more that you want to extract certain data. I just tried to make my problem more clear by taking the previous example from the urlwatch docs. The scenario is more, that you have an p element and want to extract some data from there. E.g [...] and therefore for blablabla we set the price of 2.39€.[...]. The idea is to only grab the data 2.39€ without the whole text.

With re.sub you always have to build a regex which catches the whole paragraph which is error prone. And the grep solution only works on full lines.

I actually got inspired by changedetection.io which I tried recently, because of their GUI and they have this nice data extraction feature. However their scripting is much more troublesome so I would like to stick with urlwatch.

OT: It's just a bit frustrating, why open source often has to invent new wheels instead of joining forces. It would be amazing to see, if changedetection.io would have used urlwatch under the hood, to build a more powerful solution.

from urlwatch.