Comments (4)
elbamos
commented on April 28, 2024
1
I've been experiencing an issue on OS X where I get a freeze up periodically when writing lots of histograms. I haven't traced it to a source yet which is why I didn't post an issue, but maybe it's related.
… On Jul 16, 2017, at 1:53 PM, Ethan Luo ***@***.***> wrote:
Hi,
thank you for providing this library. Some problems that I came across today.
If I try to make a histogram out of the weights of some module, it is possible to explode the number of bins and consume an extremely large amount of memory. The problem due to this line https://github.com/lanpa/tensorboard-pytorch/blob/master/tensorboard/summary.py#L118. The argument bins is never used, and as a results auto is used in np.histogram, which would produce bin of as large as 66618527586 (see attachment for an example of a bad weight matrix)
I am not sure what the preconditions for the HistogramProto are, but using the bins argument and passing it into make_histogram would solve the problem.
x.npy.zip
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
from tensorboardx.
lanpa
commented on April 28, 2024
The bins argument is overwritten intentionally:) Because I am wondering why tensorflow has a fixed binning scheme. I also tested your weight matrix under ipython with np.histogram(x, 'auto') and it also freezes my console. So I think there is a good reason why using fixed bins. BTW, By passing the bins parameter without overwriting it, you are instead using the tensorflow binning method. You may have a look at https://github.com/lanpa/tensorboard-pytorch/blob/master/tensorboard/writer.py#L212 . This will be fixed in next version. Thanks.
from tensorboardx.
ethanluoyc
commented on April 28, 2024
Yeah I was wondering why everything slows down suddenly and thought it might be a problem with pytorch not freeing the memory for some storage :) some digging led to this finding.
…________________________________
From: Tzu-Wei Huang <[email protected]>
Sent: Sunday, July 16, 2017 7:27:53 PM
To: lanpa/tensorboard-pytorch
Cc: Ethan Luo; Author
Subject: Re: [lanpa/tensorboard-pytorch] Histogram exploded the memory. (#1)
The bins argument is overwritten intentionally:) Because I am wondering why tensorflow has a fixed binning scheme. I also tested your weight matrix under ipython with np.histogram(x, 'auto') and it also freezes my console. So I think there is a good reason why using fixed bins. BTW, By passing the bins parameter without overwriting it, you are instead using the tensorflow binning method. You may have a look at https://github.com/lanpa/tensorboard-pytorch/blob/master/tensorboard/writer.py#L212 . This will be fixed in next version. Thanks.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#1 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFwsuC7k31zjqUZJna65aCZs9Jpx_p-lks5sOlYpgaJpZM4OZXUS>.
from tensorboardx.
lanpa
commented on April 28, 2024
https://github.com/numpy/numpy/search?q=histogram&type=Issues
I think the root cause is np.histogram. If problem continues, maybe we need to implement our own version.
from tensorboardx.
Related Issues (20)
- tensorboardX is incompatible with protobuf 4.21.0 HOT 16
- draw NaN with triangle
- Subsequent hparams missing HOT 4
- DeprecationWarning: crc32c.crc32 will be eventually removed, use crc32c.crc32c instead HOT 2
- AttributeError: 'NoneType' object has no attribute 'get_logdir' HOT 3
- Protobuf version HOT 4
- How can I connect previous log file into new one in tensorboard HOT 1
- How can i read tensorboard by python ? HOT 2
- Deprecation Warnings HOT 15
- OSS License compatibility question
- Remove upper pins on protobuf HOT 4
- tensorboardX is converting white/blank space and other non alphanumeric characters to underscore
- Update unittests for newer protobuf
- Possibility of v2.5.2 with upper pin on protobuf of <4? HOT 3
- The license_file parameter is deprecated HOT 3
- Loosening protobuf version limit breaking downstream packages HOT 7
- Is there an efficient way to log line plots? HOT 5
- EOFError multiprocessing HOT 1
- Set experiment name when using CometML HOT 1
- Support add_mesh for visdom writer?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorboardx.