lynnesbian / ocrbot Goto Github PK

View Code? Open in Web Editor NEW

65.0 65.0 10.0 221 KB

An OCR (Optical Character Recognition) bot for Mastodon (and compatible) instances

Home Page: https://fedi.lynnesbian.space/@OCRbot

License: GNU Affero General Public License v3.0

Python 100.00%

mastodon ocr python python3 tesseract

ocrbot's People

Contributors

Stargazers

Watchers

Forkers

melissasage roxxers moneytech nottavi redsfyre dioraman oragegu prplecake sl1txdvd

ocrbot's Issues

Implement character limit

Right now, the character limit in config.json is ignored. It should be implemented, and output longer than the limit should be split over multiple posts.

Problem while running "python3 service.py"

Hi there Lynnesbian and other people willing to help or who have encountered the same problem as I.

While running the command "python3 service.py", I have the following warning.

This was run on an ubuntu 18.04 machine.

I thought it might be because of the ubuntu 18, so I tried again on debian 10.

And nope, still the same problem.

My first time setting the bot upwas successful though, everything worked like a charm. It was shortly after the v3.3.0 release I think, and it was on an ubuntu 20.04.

I wonder what changed that this time it doesn't work and what could have caused the problem.

Have a nice day people.

Clean up the output

OCRbot currently produces rather messy output, which is to be expected from an automated OCR program, but we can clean it up a little!

Don't post "Image 1:" if there's only one image
Don't post more than two line breaks at a time
Make sure that it never just posts a blank reply with nothing in it

Fix error messages

Right now, OCRbot doesn't actually post error messages. This needs to be fixed

Improve OCR accuracy

i found a cool project that can fix the perspective of input text. this means that if the whole image is tilted in the same way, we can fix it to be readable. https://mzucker.github.io/2016/10/11/unprojecting-text-with-ellipses.html
i found another cool project by the same person that can de-curve an image and make it nice and flat. https://mzucker.github.io/2016/08/15/page-dewarping.html
the default binarisation (converting a color image to black and white, as in, 100% black or 100% white, no grey) engine is kinda crappy, but we can use openCV to do a better job than the built-in one.
tesseract trips up when the text is riiight at the edge without a border. this is easily fixed by adding a 10px border around the input.

Images attached to posts that haven't federated yet aren't detected

As the title says

Reply to the original post

As someone who often calls the OCRbot on other people's posts when they forget alt text, I'd like to be able to do so without polluting my own timeline.

I'd like to delete my post after OCRbot replies to it, but I can't because that breaks threads. It would be great if the bot could reply to the original message (and mention me as before).

Invert white text on black backgrounds

OCRbot has a lot of trouble reading screenshots from "dark theme" UIs and other white on black text. This would fix that

Read DMs

Allow OCRbot to read images from DMs so people can DM it an image they want captioned!

Support for other languages

Right now, OCRbot only works with English. It'd be really great for it to support other languages too! This could be done either by

Specifying the language you want when mentioning OCRbot (easy)
Detecting the language of the text (hard)

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Timeouts

Hello,

Many thanks for your work on this useful project.
I'm getting timeouts when I try to run reply.py:

Apr 08 18:43:01 ocr-bot01 systemd[1]: Started OCRbot reply manager.
Apr 08 18:43:02 ocr-bot01 python3[367]: Logging in...
Apr 08 18:43:02 ocr-bot01 python3[367]: Using Tesseract (sh)
Apr 08 18:43:02 ocr-bot01 python3[367]: Available languages: eng, fra
Apr 08 18:43:02 ocr-bot01 python3[367]: Starting OCRbot.
Apr 08 18:48:02 ocr-bot01 python3[367]: Traceback (most recent call last):
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
Apr 08 18:48:02 ocr-bot01 python3[367]:     six.raise_from(e, None)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "<string>", line 3, in raise_from
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
Apr 08 18:48:02 ocr-bot01 python3[367]:     httplib_response = conn.getresponse()
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
Apr 08 18:48:02 ocr-bot01 python3[367]:     response.begin()
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3.6/http/client.py", line 297, in begin
Apr 08 18:48:02 ocr-bot01 python3[367]:     version, status, reason = self._read_status()
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
Apr 08 18:48:02 ocr-bot01 python3[367]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3.6/socket.py", line 586, in readinto
Apr 08 18:48:02 ocr-bot01 python3[367]:     return self._sock.recv_into(b)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
Apr 08 18:48:02 ocr-bot01 python3[367]:     return self.read(nbytes, buffer)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3.6/ssl.py", line 874, in read
Apr 08 18:48:02 ocr-bot01 python3[367]:     return self._sslobj.read(len, buffer)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3.6/ssl.py", line 631, in read
Apr 08 18:48:02 ocr-bot01 python3[367]:     v = self._sslobj.read(len, buffer)
Apr 08 18:48:02 ocr-bot01 python3[367]: socket.timeout: The read operation timed out
Apr 08 18:48:02 ocr-bot01 python3[367]: During handling of the above exception, another exception occurred:
Apr 08 18:48:02 ocr-bot01 python3[367]: Traceback (most recent call last):
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
Apr 08 18:48:02 ocr-bot01 python3[367]:     timeout=timeout
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
Apr 08 18:48:02 ocr-bot01 python3[367]:     _stacktrace=sys.exc_info()[2])
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 357, in increment
Apr 08 18:48:02 ocr-bot01 python3[367]:     raise six.reraise(type(error), error, _stacktrace)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
Apr 08 18:48:02 ocr-bot01 python3[367]:     raise value
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
Apr 08 18:48:02 ocr-bot01 python3[367]:     chunked=chunked)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 389, in _make_request
Apr 08 18:48:02 ocr-bot01 python3[367]:     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
Apr 08 18:48:02 ocr-bot01 python3[367]:     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
Apr 08 18:48:02 ocr-bot01 python3[367]: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='social.mental.af', port=443): Read timed out. (read timeout=300)
Apr 08 18:48:02 ocr-bot01 python3[367]: During handling of the above exception, another exception occurred:
Apr 08 18:48:02 ocr-bot01 python3[367]: Traceback (most recent call last):
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "reply.py", line 216, in <module>
Apr 08 18:48:02 ocr-bot01 python3[367]:     client.stream_user(rl) #go!
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "</usr/local/lib/python3.6/dist-packages/decorator.py:decorator-gen-83>", line 2, in stream_user
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/mastodon/Mastodon.py", line 77, in wrapper
Apr 08 18:48:02 ocr-bot01 python3[367]:     return function(self, *args, **kwargs)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/mastodon/Mastodon.py", line 1987, in stream_user
Apr 08 18:48:02 ocr-bot01 python3[367]:     return self.__stream('/api/v1/streaming/user', listener, run_async=run_async, timeout=timeout, reconnect_async=reconnect_async, reconnect_async_wait_sec=reconnect_async_wait_sec)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/mastodon/Mastodon.py", line 2324, in __stream
Apr 08 18:48:02 ocr-bot01 python3[367]:     connection = connect_func()
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/mastodon/Mastodon.py", line 2319, in connect_func
Apr 08 18:48:02 ocr-bot01 python3[367]:     timeout=(self.request_timeout, timeout))
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 546, in get
Apr 08 18:48:02 ocr-bot01 python3[367]:     return self.request('GET', url, **kwargs)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
Apr 08 18:48:02 ocr-bot01 python3[367]:     resp = self.send(prep, **send_kwargs)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
Apr 08 18:48:02 ocr-bot01 python3[367]:     r = adapter.send(request, **kwargs)
Apr 08 18:48:02 ocr-bot01 python3[367]:   File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 529, in send
Apr 08 18:48:02 ocr-bot01 python3[367]:     raise ReadTimeout(e, request=request)
Apr 08 18:48:02 ocr-bot01 python3[367]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='social.mental.af', port=443): Read timed out. (read timeout=300)
Apr 08 18:48:02 ocr-bot01 systemd[1]: ocr-bot.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 18:48:02 ocr-bot01 systemd[1]: ocr-bot.service: Failed with result 'exit-code'.

I don't understand why those timeouts appears, do you have any clue?

Switch to PyOCR

PyOCR is a lot nicer to use than pytesseract, and it seems to support a lot more functionality.

OCRbot shouldn't mention people

If you ask OCRbot to transcribe an image that contains, for example, a mstdn-ebooks bot handle, it will mention the bot, and then the bot will reply, and it will reply, and the bot will reply...

You can also use this to get around blocks.

This can be fixed by inserting a ZWS after any @s, making it impossible to create a string like @user.

Support for translations

OCRbot's error messages and content warnings are currently always served in English. As OCRbot supports more languages than just English, there should be translations of the messages available.

Fix people accidentally invoking OCRbot

One possible solution would be to ignore any mentions that don't come first - for example, respond to
@ OCRbot @ userA
but ignore
@ userA @ OCRbot

This solves the problem of people not realising why OCRbot ignored them when mentioned, but also means that more than one person can ask OCRbot to caption the same post.

AI image recognition/description support

Maybe you could use some artificial intelligence to detect non-text images and describe them.

Problems:

sometimes makes mistakes, but I guess you cannot prevent this
you likely need to use a third-party service, I guess(?)

add a parameter for the length of the tut, so that the text is split into parts

so if my instance support 500 char I would use the parameter 500 and I could then copy part by part to my tut.

alternatively you could have a parameter that would put a number at the line that the paramater value reaches the text. so if @ocrbot taglenght 200 the text would be like this

"transcribed text 200 char"
200 - "transcribed text more 200"
400 - "transcribed text more 200"

OCR bot starts clogging threads with his error message pics

For some reasons this happens quite often. For me, calling the bot from Friendica, it does not work any more. It only did one day, when I discovred it. A simple one line text error message with a link to further explanations would be less annoying and clogging. I'm thinking about to block this bot. On the other hand, I find it quite useful, if it works. So please look at some threads and solve the problems.

Excuse me for opening a new issue. I cannot judge, whether this issue is related to others here.

OCRbot broken after Mastodon 3.4.0 upgrade

I had an instance of OCR bot running and then upgraded my Mastodon instance from 3.3.0 to 3.4.0. OCRbot immediately stopped working. The Mastodon instance works fine, and other bots that do things like post RSS feeds are working fine. Here is the error message I get:

$ /home/ocrbot/ocrbot/OCRbot/service.py
Logging in...
Using Tesseract (sh)
Available languages: eng
Starting OCRbot.
Traceback (most recent call last):
File "/home/ocrbot/ocrbot/OCRbot/service.py", line 314, in
client.stream_user(rl) #go!
File "/home/ocrbot/.local/lib/python3.8/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "/home/ocrbot/.local/lib/python3.8/site-packages/mastodon/Mastodon.py", line 102, in wrapper
return function(self, *args, **kwargs)
File "/home/ocrbot/.local/lib/python3.8/site-packages/mastodon/Mastodon.py", line 3110, in stream_user
return self.__stream('/api/v1/streaming/user', listener, run_async=run_async, timeout=timeout, reconnect_async=reconnect_async, reconnect_async_wait_sec=reconnect_async_wait_sec)
File "/home/ocrbot/.local/lib/python3.8/site-packages/mastodon/Mastodon.py", line 3626, in __stream
connection = connect_func()
File "/home/ocrbot/.local/lib/python3.8/site-packages/mastodon/Mastodon.py", line 3553, in connect_func
raise MastodonNetworkError("Could not connect to streaming server: %s" % connection.reason)
mastodon.Mastodon.MastodonNetworkError: Could not connect to streaming server: Internal Server Error