lopuhin / kaggle-dstl Goto Github PK
View Code? Open in Web Editor NEWKaggle DSTL Satellite Imagery Feature Detection
Kaggle DSTL Satellite Imagery Feature Detection
first, i want very thanks the autor ,it`s a outstading project,howener ,there are some problem in it. it lead to the source can not work well.
i find two very serious problems.
A:the list is to explain the problem.
in train.py
` inputs.append(patch[:, m: -m, m: -m].astype(np.float32))
outputs.append(mask[:, m: -m, m: -m].astype(np.float32))
if self.hps.needs_dist:
dist_outputs.append(dist_mask[:, m: -m, m: -m].astype(np.float32))`
return (torch.from_numpy(np.array(inputs)),
torch.from_numpy(np.array(outputs)),
torch.from_numpy(np.array(dist_outputs)))
the return 3th args, (dist_outputs) is null . so it is not transform to torch.
the error message is: RuntimeError: the given numpy array has zero-sized dimensions. Zero-sized dimensions are not supported in PyTorch
in models.py
@property def needs_dist(self): return (self.dist_loss != 0 or self.dist_dice_loss != 0 or self.dist_jaccard_loss != 0)
you can find "title 1" `s if-else decide by this function
however , the source code default this 3 value is 0
in models.py
dist_loss = attr.ib(default=0.0) dist_dice_loss = attr.ib(default=0.0) dist_jaccard_loss = attr.ib(default=0.0)
the web course is : $ ./train.py checkpoint-floder --all --hps dice_loss=10,n_epochs=70
so come up the "title 1" problem.
B . the second problem
in train.py _train_on_feeds()
` def _train_on_feeds(self, gen_batch, n_batches: int, no_mp: bool):
losses = [[] for _ in range(self.hps.n_classes)] #self.hp.n_classes=10
jaccard_stats = self._jaccard_stats()
def log():
logger.info(
'Train loss: {loss:.3f}, Jaccard: {jaccard}, '
'speed: {speed:,} patches/s'.format(
loss=np.array(losses)[:, -log_step:].mean(),
speed=int(len(losses[0]) * self.hps.batch_size / (t1 - t00)),
jaccard=self._format_jaccard(jaccard_stats),
))
t0 = t00 = time.time()
log_step = 50
im_log_step = n_batches // log_step * log_step
map_ = (map if no_mp else
partial(utils.imap_fixed_output_buffer, threads=4))
for i, (x, y, dist_y) in enumerate(map_(gen_batch, range(n_batches))):
if losses[0] and i % log_step == 0:
for cls, ls in zip(self.hps.classes, losses):
self._log_value(
'loss/cls-{}'.format(cls), np.mean(ls[-log_step:]))
if self.hps.has_all_classes:
self._log_value(
'loss/cls-mean', np.mean([
l for ls in losses for l in ls[-log_step:]]))
pred_y = self.net(self._var(x)).data.cpu()
self._update_jaccard(jaccard_stats, y.numpy(), pred_y.numpy())
self._log_jaccard(jaccard_stats)
if i == im_log_step:
self._log_im(
x.numpy(), y.numpy(), dist_y.numpy(), pred_y.numpy())
step_losses = self.train_step(x, y, dist_y)
for ls, l in zip(losses, step_losses):
ls.append(l)
t1 = time.time()
dt = t1 - t0
if dt > 10:
log()
jaccard_stats = self._jaccard_stats()
t0 = t1
if losses:
log()`
the code : if losses[0] and i % log_step == 0: this if will never run.
I'm really really sorry. Because of my poor English skills, I didn't think "excuse me" meant contempt.I just wanted to ask one question.I am really sorry that I have brought you a bad mood。And I did have a problem when I run your source code that like "the surviving open question". Can you give a general analysis of the reasons? Thank you a thousand times, you're an angel.
How to run what in this repo on the data available at kaggle dstl competition?
Hey, first of, thanks for sharing, it is awesome!
I would just like to say that I tried to replicate your results and even though I had a lot of trouble with the attr
package (because you were using almost the dev version, right?) I got around it.
This however seems like a problem I cant fix on my own.
Upon this command:
./train.py checkpoint-folder --all --hps dice_loss=10,n_epochs=70
as given in the README, I get this error:
Traceback (most recent call last):
File "./train.py", line 704, in <module>
main()
File "./train.py", line 640, in main
all_im_ids = list(utils.get_wkt_data())
File "/root/sharedfolder/kaggle-dstl/utils.py", line 41, in get_wkt_data
with open('./train_wkt_v4.csv') as f:
FileNotFoundError: [Errno 2] No such file or directory: './train_wkt_v4.csv'
However it first gives me some long parameter filled dict, if that helps. Any chance you might share this file? Even if it is private?
Position:
train.py->main()
arg('--model-path',type=Path)
Question:
Could you tell me what to fill in here? ( train_geojson_v3 ?)
if is 'train_geojson_v3', it‘s error :No such file or directory: 'align_cache\6010_1_2_p.alignment'
@lopuhin Hi, I am new in this field and trying to extract building footprint from satellite images. please suggest how we can do this task?
What ‘s your prediction time for one image?
hi,
should it be
$ ./train.py checkpoint-folder --all --hps dist_loss=10,n_epochs=70
instead of
$ ./train.py checkpoint-folder --all --hps dice_loss=10,n_epochs=70
Thank you!
made a mistake when I ran the train.py.
RuntimeError: the given numpy array has zero-sized dimensions. Zero-sized dimensions are not supported in PyTorch
in train.py
arg('logdir', help='Path to log directory')
arg('--hps', help='Change hyperparameters in k1=v1,k2=v2 format')
arg('--all', action='store_true',
help='Train on all images without validation')
arg('--validation', choices=['random', 'stratified', 'square', 'custom'],
default='custom', help='validation strategy')
arg('--valid-only', action='store_true')
arg('--only',
help='Train on this image ids only (comma-separated) without validation')
arg('--clean', action='store_true', help='Clean logdir')
arg('--no-mp', action='store_true', help='Disable multiprocessing')
arg('--model-path', type=Path)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.