Comments (4)
If you could provide a more complete example I would really appreciate it, someone else had mentioned some issues using the logging
module and I would like to see what's going on. I was actually able to get some logging working with the following:
from deco import *
import logging
logging.basicConfig()
logger = logging.getLogger()
@concurrent
def my_conc(i):
logger.warning(i)
@synchronized
def sync():
for i in range(10):
my_conc(i)
if __name__ == '__main__':
sync()
This actually does print stuff out, but I suspect your problem is more nuanced than this simple example. There are two things that I think could be going on with your program.
Logging Thread Safety vs. Process Safety
The logging module only claims that it is thread safe, not process safe. This could mean simply that some of your log lines might get interspersed, or something worse like the logging module's state becomes inconsistent.
Module Globals in Multiprocessing
When the multiprocessing module spawns new processes, module globals get recreated. The problem is that any changes that happen to the module globals in the parent process aren't necessarily reflected in the child process. At one point deco had a solution to this, but it turned out to contain some bugs and ultimately was discarded because of its complexity. I would be very interested in adding this back in if it fixes problems like these.
The easiest way to fix these is just use @concurrent.threaded
instead of @concurrent
so deco
will use threads instead of processes. The obvious downside is if your concurrent function is compute bound, you won't see any performance increase. If it's IO bound like waiting on network resources etc, this would be a good choice.
TL;DR:
I'd be interested to see any example code you could provide, it could prove very useful.
In the near term you might consider using @concurrent.threaded
instead of @concurrent
to try to fix the issue with logging
.
from deco.
Thanks for the example, it did work for me, so I did a little playing and found that it's most likely the "Module Globals in Multiprocessing" problem. I'm passing into a script with argsparse to check for flagged options.
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-v', '--verbose', dest='verbose', default=False, action='store_true')
parser.add_argument('--debug', dest='debug', default=False, action='store_true')
try:
options = parser.parse_args()
# logging setup
if options.debug:
logger.setLevel(logging.DEBUG)
elif options.verbose:
logger.setLevel(logging.INFO)
else:
logger.setLevel(logging.ERROR)
As you suggested using @concurrent.threaded
does work with the newly set logLevel, but the issue with GIL & threads... So it looks like it'll just need to pass the logLevel along with the function call.
from deco.
I just confirmed that passing the log level into the @concurrent
function does work. Just have to setLevel everytime it's called.
from deco.
Thanks a lot for the code sample. I think that's pretty reasonable motivation to try and re-implement the previous functionality that supported this. I'll add an issue for doing that.
from deco.
Related Issues (20)
- Can deco decorates nested functions? HOT 1
- conc_test.py was eror in my python 3 HOT 1
- Processor limit? HOT 3
- Can I specify the number of processes to use manually? HOT 1
- Decorators from deco cannot be used on class methods HOT 3
- KeyError with peterbe.com easy example on WinPython 3.6.3.0-64 HOT 4
- Bug with deco function call HOT 2
- wiki problems HOT 1
- Issues With Hanging Processes / Restart HOT 2
- Is nbody.py a Deco example? HOT 1
- Specify fixed number of CPUs? HOT 1
- User problem: where to look for when passing and receiving complex data structures? HOT 2
- Is it a good idea to pass read-only variables in a global like manner to the @concurrent function? HOT 3
- "posonlyargs" missing from arguments HOT 3
- Code slows down without sleep HOT 7
- get() without synchronized returns a tuple? HOT 2
- Multiple for loops HOT 4
- TypeError: cannot pickle '_nrt_python._MemInfo' object when passing a numba Dict to the @concurrent function HOT 2
- error_callback possible for deco?
- Setting and handling timeout HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deco.