Using paramiko 1.15.2, I'm still having this issue:

Exception in thread Thread-2 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1642, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

from paramiko.

more information:

Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29fa9e6cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'

Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29f9db4cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'

Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29f9ca2cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'

Exception in thread Thread-8 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
 File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

Exception in thread Thread-7 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
 File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

from paramiko.

This usually happens when you don't close() your ssh connection(s) before the main thread exits.

from paramiko.

My app often crashes when it exits with a paramiko-related traceback. The crashes happen with Paramiko 1.7.7.1 and Python 2.7.2 (on Mac OS X 10.6.8). The crashes do not happen with Python 2.6.7, although they did happen with Python 2.6.4.

My app creates, uses, and closes an SFTPClient, which I've confirmed by turning on the paramiko.transport logger. The app does not use threads itself.

Here you can see the last two paramiko.transport log messages, followed by the traceback:

INFO [chan 1] sftp session closed.
DEBUG [chan 1] EOF sent (1)
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  File "/Users/murphyke/Documents/code/python/lib/python2.7/site-packages/paramiko/transport.py", line 1574, in run
Segmentation fault

from paramiko.

I'm running Ubuntu 11.04 and could reproduce the error with this test-script, which does close the connection:
https://bugs.launchpad.net/paramiko/+bug/786808/+attachment/2534903/+files/demo_simple.py
When run in a loop for ~100 times, the exception is raised from time to time. This happens as well as on Python 2.7.1 as on Python 2.6.6.

Any workarounds?

from paramiko.

Consolidating to this ticket from other downstream sources:

• ssh 28
• Fabric #513 (has a bunch more example tracebacks, all similar; plus an actual look at the root error which is pretty wacky -- import socket somehow resulting in a None value for the socket binding.)

I'll be investigating potential sources of this bug in core Python/stdlib, and failing that, looking at ways to reproduce and handle the issue more gracefully. (It's also possible that the traceback'd error is hiding actual, more solvable errors -- which might require asking for more user feedback.)

That work will initially be done in ssh but the module in question has not diverged much/at all, so I'll be sure to copy it into paramiko as soon as it's useful.

from paramiko.

Started digging a little; will update this comment as I go:

• Googling and already found an unrelated instance of the exact same problem albeit one with no solution.
• More instances, also under Python 2.6, also in Trac, can be found in Trac #3957.
  • One comment in that thread mentions they see "the traceback" frequently once switching to Python 2.6, so while they don't say WHICH traceback (there are multiple ones in that thread, only one of which is ours) it further implicates possibility of core Python bug & Python version playing a part.
  • Sadly since this only ever seems to come up in production systems, trying different Python versions is going to be hard/impossible to test.
• Django #9582 is the same issue, which points to Python #1731. Unfortunately, that says it was fixed in Python 2.5.2, and I'm 99% sure I've seen this in Python 2.6 and possibly 2.7 as well, on top of the previous bullet point mentioning 2.6.
  • The commit message in the core Python patch fixing #1731 mentions the core problem was a daemon thread trying to clean up when much of the runtime environment is already gone. Sounds like a decent explanation, so I wonder if there's a regression or if the patch simply didn't fix all possible occurrences of the problem.
• This archived paramiko ML discussion mentions it too and implies it's a bug in 2.5 (given the date and the fact that Python on OS X of the time was 2.5.1, guessing it's the same as above) fixed in 2.6.
• Manually searching the Python bug tracker for all (not just open) socket none error issues finds a lot of hits but none that appear related.
• https://bugs.launchpad.net/bzr/+bug/56316 is more of the same: yup, looks like the module unloaded by the time your code ran! Thanks, that much was obvious :( Starting to wonder if a dumb workaround might simply be to catch this actual AttributeError and attempt to re-import socket...
• http://stackoverflow.com/questions/1745232/solving-thread-cleanup-on-paramiko - conflicting messages here, yes the issue seems to be a thread not shutting down correctly, but unclear if it's possible to protect against that reliably.
• https://bugs.launchpad.net/paramiko/+bug/786808 - an old Paramiko bug report which includes a demo they claim can run in a loop & cause the issue.
  • EDIT: This is the same script as @ifischer pasted above. Good job @bitprophet!
  • This also reminds me that many of the hits so far mention SFTP -- be interesting to see if it is in fact always limited to situations where SFTP usage (in Fabric, e.g. put/get) is invoked.
• http://www.mail-archive.com/[email protected]/msg06998.html - search for 'error' and you'll find an interesting comment about how in general, interpreter shutdown sets all module globals to None and that reimporting is apparently a legit way to work around it.

from paramiko.

So, all of that taken under consideration, it sounds like:

• the cause is, as we sort of knew, threads terminating incorrectly / too late, and this setting socket to None;
• one fix might be to try and prevent threads from terminating poorly, though this is not well defined, and given how intermittent it is, it may not be preventable at all;
• another fix may be to catch this exception itself and try "re"-handling the error, either by reimporting socket (though I actually am not sure that would work -- it's chicken and egg, we can't tell we've been given a socket.error exception subclass without socket existing) or, while it's kind of ugly, add a new socket import inside the method in question.
  • I have to re-examine what the socket.error line is even doing, but it may make sense to simply treat an outer AttributeError the same as socket.error and take the same actions.

from paramiko.

Using the demo script, I can get this to occur under Python 2.6.7, but not Python 2.5.6. Also can't get it to happen via a fabfile doing roughly, but not exactly, the same thing. Latter not a huge deal, if we can fix this in the base case I'm sure it will help for when the issue crops up via Fabric.

Using Python 2.6, will start testing potential workarounds.

from paramiko.

I now seem to need to run the test script 1000x to get 1-2 errors, not 100x, but it does still seem to barf at least once every 1000x run. (My netbook's CPU & fan is kind of pissed at me right now, but whatever. It can deal.)

Interestingly, the test script's try/except does not function; the core threading/attribute related error prints itself and the print statement in the script never shows up. Not sure what that means for workaround attempts, but I'll find out.

from paramiko.

Quick notes from initial look at workarounds:

• transport.py explicitly tries to handle thread cleanup, which I hadn't noticed. In the test script this always runs every time for the one thread that gets created.
• The error handling within run feels like it could be made more thorough; I'm guessing the assumption is the handful of cascading except clauses handle every possible error (esp given it ends with except Exception) and so it doesn't need an overarching try/finally.
  • Given the above comment re: the test script, this may be technically correct: it may not be possible to capture the exception this ticket is about. If so, we're kind of hosed :(
• I tried adding a simple if socket is None: import socket to the top of run but that yields an UnboundLocalError. Not sure why; surely socket should exist at that point since it's a top level import?
  • If memory serves, re-importing an already-imported module should be a no-op, so perhaps a vanilla import socket inside run would be kosher. But years of "don't import things inside functions" makes me cringe at the thought & assume it'd still be biting somebody somewhere.

from paramiko.

• Changed to regular import socket at top of run; went a whole 1000-cycle run with no exceptions. Sadly the 2nd 1000-cycle run had one exception. Boo-urns.
• Minor note: moved to developing on a cloud instance, that happens to be running Python 2.6.6. Getting the exception significantly more frequently, as in 15 times per 1000x (in the 1st run anyway) instead of 1-2 times. (This is sans the import addition, so far.)
• Adding the import socket inside run on the cloud instance, I now only seem to get a different but very similar error, packet.py complaining about socket being None. This may have been what I was seeing up above when running on the Mac, I don't recall. Stands to reason an undead thread, with the original/first "imported module is None" error fixed, would then encounter it elsewhere.

from paramiko.

Fun times: literally everything top-level is in fact set to None during these interpreter explosions.

Once the first two socket.* related errors are "fixed" by local import socket calls, we start getting things like len and even sys.stdout (when trying to verify the len via a regular print statement!) being None and causing their own explosions.

Think we need a different solution here -- even if we tried doing something awful like if socket is None: sys.exit(0) or whatever, I doubt it'd work because sys.exit would be None too.

Googling for "most likely raised during interpreter shutdown" to see if there is any generic wisdom for this type of exception.

from paramiko.

Found Python #1856 which seems to imply it's a generic, known Python core bug that isn't fixed until Python 3.2. Ugh. And sadly their topic was how to fix it in the interpreter, not how to work around it in userland (with a few small exceptions.)

More results, most of them strongly related to each other:

• Python #4106
• Python #9207
• Mailing list post explaining the issue but with no obvious solution (and in fact it refers to us!)
• Python 9205 which has e.g. this patch that uses a bare try/except (normally bad) but reraises if sys.modules exists (clever). Still parsing the rest of this thread...
• This specific comment from 9205 lays out the clearest picture I've seen yet and encapsulates some of the solutions I've seen implied elsewhere.

from paramiko.

Tried the "hold references to things we use" approach, but it's really about the same as the "reimport stuff" approach: every time I patch up one object reference, others take its place. Practically any method within e.g. packet.Packetizer seems liable to run in this dead-interpreter state, and obviously that's a lot of potential references.

Next step: try the "check if sys.modules is None" test for squashing the exceptions.

from paramiko.

A preliminary test which wraps most of the body of run in the sys.modules-testing approach outlined in Python 9205 seems to yield good results, I ran one 1000-cycle test with zero errors. Will continue to up the # cycles until I'm positive it's helping. Will also try to counter-test to make sure it's not swallowing real exceptions, though the sys.modules bit should be near bulletproof.

EDIT: 2nd 1000-cycle ran 100% OK. Doing a 10,000-cycle test now, if that has no errors I'm gonna say this is fixed & hope the extra layer of try/except is sufficiently stable re: normal operations. EDIT 2: 10,000 cycles with no errors. Woo.

FWIW the test suite also still passes. Will try running some fabfile commands as well for a final sanity test.

from paramiko.

Commit over in ssh repo: bitprophet/ssh@3709d2e

About to port to Paramiko & push there. Then hopefully find some brave souls to test it out pre-release.

from paramiko.

There's the commit. What I did:

• Confirmed existence of bug in Paramiko pre-patch
• Confirmed test suite ran OK
• Patched
• Confirmed test suite still OK
• Confirmed 1000-cycle run with no exceptions
• Confirmed 10,000-cycle run with no exceptions.

from paramiko.

So, if anybody who was seeing this issue wants to nab latest master and confirm the fix, it'd be greatly appreciated!

from paramiko.

still has Exception output:

Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-23 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-11 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-38 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-34 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1602, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-13 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1602, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

from paramiko.

@jumping Which exact checkout SHA are you using in the above, and have you made 100% sure you're importing that checkout? (e.g. nuke virtualenv & reinstall, make sure no 'ssh' exists in system site-packages, that sort of thing.)

Can you clarify exactly how you are reproducing this & how frequently it's occurring for you?

Finally -- if this is easily reproducible & the above checks out, can you try adding a print statement to the sys.modules check we added, and see if that is somehow non-None when these exceptions occur?

tl;dr I cannot replicate this any longer with the fix in place, so the burden's back on you to help figure out what's still missing :)

from paramiko.

I get a similar problem. I am using Paramiko through Fabric through py.test. I get at the following exceptions:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/home/bernat/.virtualenvs/pytest/local/lib/python2.7/site-packages/paramiko/transp
ort.py", line 1641, in run
    self.sock.close()
  File "/home/bernat/.virtualenvs/pytest/local/lib/python2.7/site-packages/paramiko/proxy.
py", line 86, in close
    os.kill(self.process.pid, signal.SIGTERM)
AttributeError: 'NoneType' object has no attribute 'kill'

self.sys.modules is not empty. self.sys.modules["os"] even exists. To workaround the issue, I have modified proxy.py:

    def close(self):
        self.os.kill(self.process.pid, self.signal.SIGTERM)

And in __init__():

        self.os = os
        self.signal = signal

I didn't have any problem since. This may help find the root cause (sys.modules in one module does not reflect sys.modules in another?).

from paramiko.

I'm using Fabric in a background job and facing the same problem. Sometime it'll get the main thread hung, so I have to write another job to help killing it...

Pakcage version info:
Fabric==1.6.0
paramiko==1.10.1
pycrypto==2.6

Any follow up on this issue?

from paramiko.

Hi, any chance this bug fix can be moved to the release codebase so that its available publicly? I need a fix for this urgently. Or please suggest a workaround for this.

from paramiko.

On windows, there is a bug :

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 808, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\site-packages\paramiko-1.11.0-py2.7.egg\paramiko\transport.py", line 1641, in run
    self.sock.close()
  File "C:\Python27\lib\site-packages\paramiko-1.11.0-py2.7.egg\paramiko\proxy.py", line 86, in close
    os.kill(self.process.pid, signal.SIGTERM)
WindowsError: [Error 5] Access denied

To resolve it in proxy.py :

    def close(self):
         self.process.kill()
        #os.kill(self.process.pid, signal.SIGTERM)

from paramiko.

@agnivade If you mean the original bugfix it was published a year ago! If you mean @vincentbernat 's suggestion, I have to see if I can recreate the problem he is having to verify the fix.

Vincent - any help would be appreciated. All I'll be able to do is look up what I was doing a year ago & see if that test setup yields errors again (& if so, whether your change helps.) No guarantee these two issues are actually linked, though.

from paramiko.

No I meant the original bug fix. In my ubuntu, I don't see the paramiko version to be 1.11 when I download the latest debian. That's why I was asking this. Anyway, even with the fix from the latest branch, the problem persists. But currently, I can live with it because the program crashes only after the last item of the loop.

from paramiko.

@agnivade I can do nothing about your operating system's version :) but thanks for the update. Sounds like this (or related...there are a LOT of things that can cause this error message) issue is still popping up sometimes. This sort of bug is hard to eradicate :(

from paramiko.

I don't think I was clear. Its not about the version of my OS. Its the version of paramiko which is old, when I do an apt-get install. I am saying that you haven't pushed the latest version to the public ubuntu repository.

from paramiko.

@agnivade it is the job of the package maintainers of Debian and Ubuntu to update the package for their respective distributions based on the updated source code upstream (here).

Unless the maintainer or the upstream code is also one of the maintainer of the packages in the distributions, the only thing that can be done is to ask the package maintainer to update the package with the latest version of paramiko and include it in the distribution.

For Debian, you'll find the appropriate informations here : http://packages.qa.debian.org/p/paramiko.html

from paramiko.

@zertrin, I am the Debian package maintainer, and use I haven't gotten the upgrade from 1.10.1 to 1.12.0 done yet but it is on my TODO list.

from paramiko.

@jbouse :)

For the record, I wasn't complaining here, just explaining to @agnivade that it's not because upstream has been updated that all the packages in all the distros are updated right away. Maintaining a package isn't always trivial, takes time and above all isn't necessarily done by the upstream developers.

Thanks for the heads-up anyway!

from paramiko.

@zertrin there is also the fact that depending on which release of a given distro it may never be updated unless it's a security fix that is backported to the version in that release. ie- Debian stable will never version than 1.7.7.1 as that was what was in stable when it was released. Only Debian testing and unstable can be updated.

from paramiko.

Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner
  File "build/bdist.linux-i686/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

On ubuntu13.10 + paramiko-1.7.6

Fixed by update to paramiko-1.12.1

from paramiko.

I ran into the same issue, got solved through upgrading paramiko, thanks guys!

from paramiko.

I'm just looking to see if this issue is still current in latest versions as Debian BTS 698075 that references this issue hasn't seen any updates since Jan 2013 and I've since gotten 1.14.0 packaged and pushed out.

from paramiko.

I installed latest Paramiko 1.15.2 and using python 2.7
Got this error:
Captured stdout:
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
File "/opt/move/epg-tools/local/lib/python2.7/site-packages/paramiko/transport.py", line 1613, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

from paramiko.

I'm still suffering from this problem under Python 2.7.6, Fabric 1.8.2, Paramiko 1.10.1, and have developed the following workaround that seems to be working so far.

My program allocates an object in a main() function from a class that uses Fabric calls. I have Fabric close all ssh connections in this object's destructor because the object goes out of scope when main() returns.

import fabric.network
import fabric.state

class MyObject(...):
    ... methods that use fabric ...
    def __del__(self):
        fabric.state.output.status = False
        fabric.network.disconnect_all()

from paramiko.

Happens to me on Mac (10.11.5) dev when using execute @parallel tasks on multiple hosts. Task execute successfully but this exception happens on exit and is followed with Process finished with exit code 0 message which is somewhat confusing.

Fabric (1.11.1)
paramiko (1.15.2)
Exception in thread Thread-9 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner File "/usr/local/lib/python2.7/site-packages/paramiko/transport.py", line 1642, in run

from paramiko.

I'm running Python 2.7.14 and paramiko==2.4.1 and Fabric==1.14.0 and using fabric in an integration test to manage setting up external resources for the test. If I add the following, it seems to eliminate the exception. I can run the testing repeatedly without the tearDownClass() method, and I get the exception probably about 50% of the time, on exiting the test successfully. It is mainly annoying to see this exception print out as the last output from a successful test. :) With the tearDownClass below, I don't see the exception occurring:

@classmethod
def tearDownClass(cls):
    # This is part of trying to suppress a paramiko shutdown bug
    # https://github.com/paramiko/paramiko/issues/17
    logger.info('tearDownClass')
    fabric.state.output.status = False
    fabric.network.disconnect_all()

Kudos to @curtmcd for the suggestion of the two calls to fabric to disconnect_all()

from paramiko.

Hi! I'm helping bitprophet by going through the issues and PRs backlog, and for anything more than a few years old, we're just closing them out for now.

If this is still a live bug, please reopen the issue, and comment to let us know it's still of interest.

Thanks!

from paramiko.