mathewmeconry / tv7_epg_parser Goto Github PK

View Code? Open in Web Editor NEW

17.0 8.0 12.0 68 KB

Parsers EPG information of different sources and prepares them for TV7 (Init7) in XMLTV format

License: MIT License

Python 100.00%

python3 epg tv7 init7 xmltv

tv7_epg_parser's Introduction

TV7_EPG_Parser

Parses tele.ch and teleboy.ch for the EPG-Information and prepares them for TV7 (Init7) in XMLTV format

Ready XMLTV Files

https://github.com/mathewmeconry/TV7_EPG_Data

Matching rate

~143 channels

Requirements

python3 >=3.7.3
modules listed in requirements.txt. (can be installed with python3 -m pip install -r requirements.txt)

Discord Server

This server is used for idea discussions and some small out of topic stuff.
Please report issues or complete feature requests with the github issues feature.
https://discord.gg/4FMXY3c

Contributing

Fork it!
Create your feature branch: git checkout -b my-new-feature
Commit your changes: git commit -am 'Add some feature'
Push to the branch: git push origin my-new-feature
Submit a pull request :D

License

MIT

tv7_epg_parser's People

Contributors

Stargazers

Watchers

Forkers

nbarrientos alexiri phwissmann bbkz honsa hostho manbehindmoon jokay zvea marwin79 rbkn mime01

tv7_epg_parser's Issues

IndexError: list index out of range tele.ch

Reported by Sascha Candella by mail

From time to time some threads fail at parsing the tele page.

[*] Getting EPG data from tele.ch
[*] Downloading from 2019-06-13T19:19:49.453393 until 2019-06-13T23:18:49.453393
[*] Downloading from 2019-06-13T23:18:49.453393 until 2019-06-14T03:17:49.453393
[*] Downloading from 2019-06-14T03:17:49.453393 until 2019-06-14T07:16:49.453393
[*] Downloading from 2019-06-14T07:16:49.453393 until 2019-06-14T11:15:49.453393
[*] Downloading from 2019-06-14T11:15:49.453393 until 2019-06-14T15:14:49.453393
Exception in thread Thread-36:
Traceback (most recent call last):
   File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
     self.run()
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 161, in run
     self.process_data(self.name, self.q)
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 170, in process_data
     self.enriched_data.append(tele.__enrich_show__(data))
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 128, in __enrich_show__
     tele.__api__+"/show/"+item["assetId"]).text)[0]
IndexError: list index out of range

[*] Downloading from 2019-06-14T15:14:49.453393 until 2019-06-14T19:13:49.453393
Exception in thread Thread-2:
Traceback (most recent call last):
   File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
     self.run()
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 161, in run
     self.process_data(self.name, self.q)
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 170, in process_data
     self.enriched_data.append(tele.__enrich_show__(data))
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 128, in __enrich_show__
     tele.__api__+"/show/"+item["assetId"]).text)[0]
IndexError: list index out of range

Exception in thread Thread-34:
Traceback (most recent call last):
   File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
     self.run()
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 161, in run
     self.process_data(self.name, self.q)
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 170, in process_data
     self.enriched_data.append(tele.__enrich_show__(data))
   File "/TV7_EPG_Parser/epg_sources/tele.py", line 128, in __enrich_show__

EPG parser is broken because cfscrape was broken by urllib3 update

cfscrape (https://github.com/Anorov/cloudflare-scrape) doesn't look like it's being developed any longer. Meanwhile, urllib3 has continued changing, and in version 2.0.0 the removed support for DEFAULT_CIPHERS, which cfscrape relies on.

The result is that trying to run the EPG parser fails with

$ python3 main.py 
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    from epg_sources.teleboy import teleboy
  File "epg_sources/teleboy.py", line 7, in <module>
    import cfscrape
  File "lib/python3.11/site-packages/cfscrape/__init__.py", line 19, in <module>
    from urllib3.util.ssl_ import create_urllib3_context, DEFAULT_CIPHERS
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (TV7_EPG_Parser/lib/python3.11/site-packages/urllib3/util/ssl_.py)

The best solution would be to no longer depend on cfscrape. I am not familiar with it, but among the 118 open issues there seems to be quite a few mentioning that it's broken and doesn't really bypass Cloudflare anymore (like 455, 458 and 459).

As a temporary measure I was able to make the parser work with a pip3 install urllib3==1.26.16 (the last release before 2.0.0); this may be a good addition to requirements.txt (urllib3 is installed by requests, but it doesn't seem to create a problem. Yet), but I expect this to break once and for all when requests starts requiring urllib3>=2.0.0 :(

Hope this helps!

Selective generation of EPG data

It would be great to be able to choose the used EPG providers (this could save some time, e.g. only Teleboy).

sub_title sometimes None

Reported by boardend over discord:

No EPG data available for RTS 1 HD nor RTS 2 HD

Hi @mathewmeconry,

At least at the time of writing these lines there does not seem to be any EPG data in the full dump for these two channels. Haven't checked the other (smaller) dumps.

Tele.ch seems to have data, though. Perhaps there's some extra channel id mapping to do, I haven't checked the code in detail. Please let me know if I can help :)

Teleboy sorting "excludes" some channels

I noticed that some channels are always empty when using the teleboy.ch EPG despites the website having full program informations.

Fiddling a little bit with the code I think that passing sort=station to the call retrieving the broadcasts (https://github.com/mathewmeconry/TV7_EPG_Parser/blob/master/epg_sources/teleboy.py#L61) causes some stations to always be excluded because of their name towards the end of the alphabet :)

Here's an example when running with the "standard" 7 days:

With sort=station

[✓] Matched 36 teleboy.ch channels
[✓] Matched 36 teleboy.ch icons
[✓] Total 36 icons

Without sort=station

[✓] Matched 103 teleboy.ch channels
[✓] Matched 103 teleboy.ch icons
[✓] Total 103 icons

I don't really see any drawbacks of not having the broadcasts sorted by station especially since we request date sorting of the data from teleboy's API.

(one-line PR incoming 😛 )

Gaps in Teleboy EPG data

There seem to be huge gaps in the Teleboy EPG data. This is an excerpt from all broadcasts on SRF1.

<programme start="20190614063000 +0200" stop="20190614073000 +0200" channel="srf1">
<programme start="20190614073000 +0200" stop="20190614090000 +0200" channel="srf1">
<programme start="20190614141000 +0200" stop="20190614150000 +0200" channel="srf1">
<programme start="20190614150000 +0200" stop="20190614150500 +0200" channel="srf1">
<programme start="20190614150500 +0200" stop="20190614155000 +0200" channel="srf1">
<programme start="20190614210000 +0200" stop="20190614215000 +0200" channel="srf1">
<programme start="20190614215000 +0200" stop="20190614221500 +0200" channel="srf1">
<programme start="20190615040500 +0200" stop="20190615044000 +0200" channel="srf1">
<programme start="20190615044000 +0200" stop="20190615052000 +0200" channel="srf1">
<programme start="20190615103500 +0200" stop="20190615121000 +0200" channel="srf1">
<programme start="20190615181000 +0200" stop="20190615184500 +0200" channel="srf1">
<programme start="20190615184500 +0200" stop="20190615191500 +0200" channel="srf1">
<programme start="20190616001500 +0200" stop="20190616020000 +0200" channel="srf1">
<programme start="20190616020000 +0200" stop="20190616034000 +0200" channel="srf1">
<programme start="20190616073000 +0200" stop="20190616093000 +0200" channel="srf1">
<programme start="20190616152500 +0200" stop="20190616162500 +0200" channel="srf1">
<programme start="20190616222500 +0200" stop="20190616225500 +0200" channel="srf1">
<programme start="20190616225500 +0200" stop="20190616230500 +0200" channel="srf1">
<programme start="20190616230500 +0200" stop="20190616231500 +0200" channel="srf1">
<programme start="20190616231500 +0200" stop="20190617004000 +0200" channel="srf1">
<programme start="20190617051000 +0200" stop="20190617054500 +0200" channel="srf1">
<programme start="20190617054500 +0200" stop="20190617060500 +0200" channel="srf1">
<programme start="20190617121500 +0200" stop="20190617124500 +0200" channel="srf1">
<programme start="20190617124500 +0200" stop="20190617125500 +0200" channel="srf1">
<programme start="20190617125500 +0200" stop="20190617130500 +0200" channel="srf1">
<programme start="20190617192500 +0200" stop="20190617193000 +0200" channel="srf1">
<programme start="20190617193000 +0200" stop="20190617195500 +0200" channel="srf1">
<programme start="20190617195500 +0200" stop="20190617200500 +0200" channel="srf1">
<programme start="20190618011000 +0200" stop="20190618023500 +0200" channel="srf1">
<programme start="20190618023500 +0200" stop="20190618033000 +0200" channel="srf1">
<programme start="20190618091000 +0200" stop="20190618093500 +0200" channel="srf1">
<programme start="20190618093500 +0200" stop="20190618095000 +0200" channel="srf1">
<programme start="20190618095000 +0200" stop="20190618100000 +0200" channel="srf1">
<programme start="20190618100000 +0200" stop="20190618103000 +0200" channel="srf1">
<programme start="20190618155000 +0200" stop="20190618173000 +0200" channel="srf1">
<programme start="20190618222500 +0200" stop="20190618234500 +0200" channel="srf1">
<programme start="20190618234500 +0200" stop="20190618235000 +0200" channel="srf1">
<programme start="20190618235000 +0200" stop="20190619012000 +0200" channel="srf1">
<programme start="20190619053000 +0200" stop="20190619063000 +0200" channel="srf1">
<programme start="20190619063000 +0200" stop="20190619073000 +0200" channel="srf1">
<programme start="20190619132000 +0200" stop="20190619141500 +0200" channel="srf1">
<programme start="20190619200500 +0200" stop="20190619205500 +0200" channel="srf1">
<programme start="20190619205500 +0200" stop="20190619215000 +0200" channel="srf1">
<programme start="20190620032500 +0200" stop="20190620033500 +0200" channel="srf1">
<programme start="20190620033500 +0200" stop="20190620034500 +0200" channel="srf1">
<programme start="20190620034500 +0200" stop="20190620051000 +0200" channel="srf1">
<programme start="20190620100000 +0200" stop="20190620103000 +0200" channel="srf1">
<programme start="20190620103000 +0200" stop="20190620110000 +0200" channel="srf1">
<programme start="20190620110000 +0200" stop="20190620113000 +0200" channel="srf1">
<programme start="20190620155500 +0200" stop="20190620173000 +0200" channel="srf1">
<programme start="20190620173000 +0200" stop="20190620174000 +0200" channel="srf1">
<programme start="20190620174000 +0200" stop="20190620180000 +0200" channel="srf1">
<programme start="20190620180000 +0200" stop="20190620181000 +0200" channel="srf1">
<programme start="20190620234000 +0200" stop="20190621011500 +0200" channel="srf1">
<programme start="20190621062500 +0200" stop="20190621073000 +0200" channel="srf1">
<programme start="20190621073000 +0200" stop="20190621090000 +0200" channel="srf1">

PR for mappings for ITV1/bluezoom

Hi,
I noticed that the teleboy ITV1 mapping between channels/epg is currently broken.
I can submit a PR for this if you're interested (lack permission currently).

Test cases needed to be run before commits to master

Ideas:

check validity of xml output
check that the output is above a threshold size

Am happy to contribute here.

Earlier start time

The start time should be selected so that the running programs are also included. One possibility would be to simply get the epg data from the started day completely.

Requirements in Readme

Send by Sascha Scandella by mail:

Please add the requirements and the how to contribute instructions (added by me) to the readme.