multiprocessing Queue and Process calls fail on Atom based NAS's about plexconnect HOT 32 CLOSED

Is there any chance to check for "multiprocessing" support before running into the exception?
I wonder if something like
if "CPU and OS support multiprocessing":
...import multiprocessing
else:
...import threading
would be all we need?

from plexconnect.

My two cents: you should consider the simpler and more broadly supported queue/thread in all cases. It's not as if the DNS/HTTP server really require the power of true multiprocessing, and this would have the side benefit of simplifying process/thread management for the purposes of startup/shutdown.

from plexconnect.

Yes... you might be right. A "process" just sounds waaay cooler then a simple "thread", doesn't it? :-D

from plexconnect.

touché.

from plexconnect.

Could you do me a favour and list the functions that are available in your "multiprocessing" package?
Please start a new python session and do...
import multiprocessing
dir(multiprocessing)
I am interested in the output... :-D

from plexconnect.

On my DS713+...

DiskStation> ./python
Python 2.7.5 (default, May 30 2013, 20:38:38)
[GCC 4.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import multiprocessing
dir(multiprocessing)
['Array', 'AuthenticationError', 'BoundedSemaphore', 'BufferTooShort', 'Condition', 'Event', 'JoinableQueue', 'Lock', 'Manager', 'Pipe', 'Pool', 'Process', 'ProcessError', 'Queue', 'RLock', 'RawArray', 'RawValue', 'SUBDEBUG', 'SUBWARNING', 'Semaphore', 'TimeoutError', 'Value', 'all', 'author', 'builtins', 'doc', 'file', 'name', 'package', 'path', 'version', '_multiprocessing', 'active_children', 'allow_connection_pickling', 'cpu_count', 'current_process', 'freeze_support', 'get_logger', 'log_to_stderr', 'os', 'process', 'sys', 'util']

from plexconnect.

Hm... same list here on MacOS with multiprocessing working fine.
Just to make sure... with this NAS and the original PlexConnect.py, it would fail in that way?

[ from http://forums.plexapp.com/index.php/topic/70340-plexconnect-spk-for-synology-released/page-3?hl=atom#entry410345 ]

Traceback (most recent call last):
File "./PlexConnect.py", line 39, in
cmd_DNSServer = Queue()
File "/volume1/@appstore/python/lib/python2.7/multiprocessing/init.py", line 218, in Queue
return Queue(maxsize)
File "/volume1/@appstore/python/lib/python2.7/multiprocessing/queues.py", line 63, in init
self._rlock = Lock()
File "/volume1/@appstore/python/lib/python2.7/multiprocessing/synchronize.py", line 147, in init
SemLock.init(self, SEMAPHORE, 1, 1)
File "/volume1/@appstore/python/lib/python2.7/multiprocessing/synchronize.py", line 75, in init
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented

Or do you, jchaps, have newer OS without this issue?

from plexconnect.

Yes. It fails in that way. (I'm jchaps here, jchapman24 on Plex forums. You linked to my post.)

from plexconnect.

So strange... they ship a python lib that doesn't run on its own?
Anyway, the code doesn't fail when spawning processes but when installing the queue.
a - Do you have any information about the "multiprocessing" on your machine?
b - How about installing a "pipe" instead of the "queue"? -> http://stackoverflow.com/questions/8463008/python-multiprocessing-pipe-vs-queue Would that be supported?

from plexconnect.

PMS includes a newer python which should work.

from plexconnect.

Python 2.7.5 isn't that old, is it?

from plexconnect.

jchaps - I created a "pipe-only" version. Would you please grab the latest of branch "Pipe instead Queue" and try that?

from plexconnect.

I'll give it a shot as soon as I get home from the office.

On Tue, Jun 18, 2013 at 11:23 AM, iBaa [email protected] wrote:

jchaps - I created a "pipe-only" version. Would you please grab the latest
of branch "Pipe instead Queue" and try that?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/54#issuecomment-19618746
.

from plexconnect.

Working great on my Atom based DS412+

👍

from plexconnect.

This sounds good... I will wait for jchaps findings and then merge it to the main path.

from plexconnect.

I can confirm, it works!

It was able to navigate, play movies and tv video files (all in ATV2-appropriate .mp4). Music wouldn't load (no surprise).

Thanks for the update!

Also, let me know if there are specific findings you're interested in, and I'll post everything I can to help.

from plexconnect.

is there a reason to keep it as process and not as thread? I don't understand...

from plexconnect.

Don't know, is there?
I have no idea how multi-threading works - does it switch automaticly after time? Do the "threads" have to tell that they are done? Either DNSServer or WebServer listen to individual ports unless they read new data (okay, for shutdown purposes there is the "timeout" feature installed). Will threads be switched even during the waiting time?
With processes I just had a clearer idea as how things will/would/could work... especially if you have a multi-core-processor.
Do you have issues with using "processes" per se? The issue solved here was not about processes but NAS's missing support for semaphores.

from plexconnect.

I have a Synology Nas, and had the problem of the multiprocessing package.

I solved it by using threading (credits to AnthonyMann)

bwynants@449e05b

http://stackoverflow.com/questions/6683475/non-blocking-python-process-or-thread

from plexconnect.

bwynants what type of CPU is in your Synology, is it an Atom?

from plexconnect.

http://www.synology.com/products/spec.php?product_name=DS1512%2B&l#p_submenu

from plexconnect.

)) I have a Synology Nas, and had the problem of the multiprocessing package.
Right... but as I said, the issue was not the multiprocessing, but the sempahore used in the queue implementation. At least according to our findings above...

from plexconnect.

bwynants... would you please test the "Pipe instead Queue" branch, too. This should help your issue. I would really like to know :-D

from plexconnect.

Success on my MIPS-based Tomato based router which gave the same error. I'm sure this will fix my Synology NAS as well, just haven't tested it.

Linux router 2.6.22.19 # 52 Tue Apr 2 20:13:40 ICT 2013 mips GNU/Linux

The title could be changed to include many types of embedded devices is my guess. I'm aware of Atom NAS units as mentioned above and now MIPS Tomato routers. It's probably due to the limited embedded stack.

I used the changes from bwynants@449e05b to use threading and a few other small changes. Many of these should make their way back into the code probably. I can do pull requests later but most are trivial.

1. On this device Python is not in /usr/bin instead in /opt/bin. Instead of calling directly to that, the shebang should be /usr/bin/env python for all *.py scripts. That'll work everywhere correctly.
2. For Python 2.7 I had to fix an error (old syntax) in DNSServer.py: except socket.error as e: should be except socket.error, e:
3. Had to move the DNS server out of the way by patching DNSServer.py.
4. Changed DNSServer.py to feed ip_webserver instead of the detected IP address from getIP_self(). It wasn't doing a good job finding the right IP on a multihomed machine. This makes more sense though as the IP fed via DNS for the trailers.apple.com intercept should match the IP of the web server, no? If so, IP_self detection should be disabled when ip_webserver is defined to something other than 0.0.0.0. It's also be nice to prefer non-public IPs on IP detection :)
5. PMS detection didn't work, had to set it manually in Settings.cfg (odd, not sure why. maybe because there are many interfaces on this router?)

This plus two iptables to redirect traffic results in no modifications necessary on the ATV DNS settings and running on a box that already has port 53 & 80 in use. Woohoo!

For reference the iptables commands are below. 172.16.32.12 = ATV, 172.16.32.1 = router.

iptables -t nat -A PREROUTING -s 172.16.32.12 -d 172.16.32.1 -p udp --dport 53 -j DNAT --to-destination 172.16.32.1:8053
iptables -t nat -A PREROUTING -s 172.16.32.12 -d 172.16.32.1 -p tcp -m multiport --dports 80,443 -j DNAT --to-destination 172.16.32.1:8090

from plexconnect.

1. Really?
2. The ',' is the old syntax as far as I know. The 'as' is new... and less ambiguous.
3. your responsibility :-D
4. IP in DNS response to trailers... needs to point to PlexConnect machine, NOT the PMS one (in case they are different).
5. Do you see any way to fix this?

Overall I guess, I don't fully understand, how your comment fits into this issue. Does the latest code in branch "pipe instead queue" work? Did your really have issues with "multiprocessing" or more related to the semaphores used in Queue" as well?

from plexconnect.

1. It's a much more reliable way of finding it I believe independent of OS.
2. You are right, my symlink for python was wrong! I'm not a python guy, sorry. I saw a thread talking about this and read it as the other way around -- grr.
3. You got it!
4. Well, right. In my unique situation I'm redirecting traffic to my router and letting it direct it to PMS. This is pretty unique and unless I see a trend where this is used by others, I'll just maintain my own patch for it. Ignore this for now.
5. I'll work on this.

And finally, yes I missed being explicit that "pipe instead of queue" does not work on both Tomato and a Synology DS411+ (just tested). I get

Process Process-1:
Traceback (most recent call last):
  File "/opt/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/src/PlexConnect/DNSServer.py", line 158, in Run
    cmd = cmdQueue.get_nowait()
AttributeError: 'tuple' object has no attribute 'get_nowait'

What did work, however, is the branch at bwynants@449e05b which moves over to threading/queue/thread.

So I guess the point is that bwynants@449e05b works on this device and now also a Synology DS411+ (just tested).

In order to get it to work on Tomato (and other similar devices like dd-wrt, I'm guessing), there are a few small mods necessary namely items 1, 3, 5. Some may be necessary for other NAS units, too, like 1.

Item 2 is junk, and 4 is pretty fancy but probably won't be used by many unless they REALLY need to get off port 80.

from plexconnect.

Nope... you didn't use the latest sources. Line 158 in the pipe-branch reads "cmd = cmdQueue[1].recv()"...
It would be great if you could confirm the latest sources running - this would show, that even on your system multiprocessing is not the issue.

from plexconnect.

Updated and it does indeed work on both my Tomato router and the Synology DS411+.

from plexconnect.

Good thing. So I got three "OK" answers, zero "NOK". I will go and merge the changes to the base branch.

from plexconnect.

wojo - question to 3) You know, you can completely disable the PlexConnect DNSServer? If you are running a DNSServer on your own... let this one modifying the trailers app IP.

from plexconnect.

A bit off topic, but if I use dnsmasq's ability to rewrite to a different IP (with "address=/trailers.apple.com/172.16.32.1") then I won't be able to hit trailers.apple.com from any other device -- including PlexConnect itself if it ever has the ability to embed the original Trailers. This is because dnsmasq doesn't have conditional responses based on source IP for example.

I could configure PlexConnect to query a different upstream (Google DNS) so that my local DNS override doesn't take over. Or use iptables like I do to snarf off traffic and send it where I want (a non-port 53 DNS server which could be PlexConnect or another dnsmasq instance).

Back on topic though, I'm just happy we were able to confirm quite well that your branch works well on these embedded devices! Can't wait to test it on the mainline XML_templates branch.

from plexconnect.

"pipe" code merged into main branch. closed.

from plexconnect.