jakopako / goskyr Goto Github PK

https://github.com/araddon/dateparse

https://www.mascotte.ch/nu/events/event_list/

In some cases it might be necessary to filter certain events, such as in the case of Backstage. There we don't want the CORONA TESTZENTRUM events in our list.

Therefore, add a filter for text fields (currently title & comment) that is based on a regex.

https://walcheturm.ch/agenda/

It is better to separate the code development from the use case 'concert scraping'. Like that we would have a repository solely for managing which concert websites get crawled and could also accept pull requests for this.

Add comments to this example config file to explain the config options in detail.

It would be good to 'purify' this repo by only providing a generic crawler that extracts any structured data from websites. But since Github actions are very handy to execute a regular crawl with a specific configuration there could be a separate branch only for this purpose.

the respective post request in the github action results in a 400

Use fmt.Errorf instead of errMsg := fmt.Sprintf and then errors.New(errMsg)

https://www.locomotivclub.it/eventtype/calendario/

https://www.volkshaus.ch/aktuell

https://strom-muc.de/

url: https://exil.cl/programm

https://www.muffatwerk.de/de/events/concert

Add info which crawler logs which log string.

url: https://www.heldenbar.ch/programm/

https://www.hetdepot.be/programma

Not all crawlers have filters to remove canceled or postponed concerts. Add filters where necessary.

https://www.papiersaal.ch/

https://botanique.be/en/concerts

https://kaufleuten.ch/events/kultur/konzerte/

https://www.x-tra.ch/en/agenda/concerts/

If there are two filters with match: true then we’d expect the item to be included in the results if at least one of those filters returns true. Currently, both have to return true for the item to be included.

Add a way to cope with different formats. Issue #35 possibly solves this as well.

I think it would be better to have separate field filters per crawler like so:

filters:
  - field: "title"
    regex: "some regex"
  - field: "comment"
    regex: "some other regex"

This would make the implementation easier since we could simply loop over the array of filter items and apply the filter to the corresponding field. On the other hand we would still need sth like a switch case to map the field string to the corresponding field in the crawler struct... Think about this and improve the implementation.

Currently, those two config keys logically do very similar things. So it makes sense to merge their functionality into one config snippet.

http://www.lacigale.fr/en/

Example: location 'Strom' There the description can be found on the event specific subpage but does not always have the same selector. Probably other locations have the same issue. Just look at the config.yml and check locations that don't have the comment field defined.

This was fixed already but with refactoring had to be removed. The problem is smart-guessing the year of a date if it is not provided by the crawled website. An example is the Helsinki crawler. During the year this does not really matter but in between two years the dates would be wrong as the year would always be the 'old' one and not increased by one for dates that lie in the new year.

currently, only existing fields can be used for filtering. Sometimes however it is necessary to filter on other text elements that are not used for any fields. One example would be the location 'LaCigale'

https://gds.fm/SENDER#main-content

https://goreportcard.com/report/github.com/jakopako/goskyr

https://backstage.info/veranstaltungen-2/alle-veranstaltungen

https://www.tonhalle-orchester.ch/en/home-tz/konzerte/kalender/

eg https://www.hetdepot.be/programma?page=x

Make field names customizable. Add a field type that defines what parameters are needed to extract this field. Currently, there would be a text type a url type and a date type.