cybertronai / imagenet18 Goto Github PK

In train_imagenet_nv.py, 'phase' always equals None, since set_epoch(epoch) will delete cur_phase when it is not None.
line 143: dm.set_epoch(epoch)
line 318: self.phases.remove(phase)
line 155: phase = dm.get_phase(epoch)

Hi
I am new to AWS. Which resources increase request need to be filled for making this work?

Thanks for your good codes. If I want to use it on my own GPU machines. Can I do this? How should I config the IP and other configuration?

Thanks.

• More robust checks to detect if creation failed
• Document "delete_resources" more prominently for manual fixing of failed resource creation

Hello and thanks for your great work.
I tried to rerun your code on 2xP3.16xlarge. Everything seems fine up to creating machines. They run but ncluster cannot ssh to both machines:
1.imagenet: Exception connecting to "IP" via ssh (could be a timeout): [Errno None] Unable to connect to port 22 on "IP".

Is there any specific details that I needed to take care of?
I checked the created VMs and both allow SSH on port 22.
It is worth mentioning that I copied the key from the following path in the machine I am running your code /home/ubuntu/.ncluster/ncluster-ubuntu-831839378763-us-east-2.pem and could not ssh to both machines too. connection refused!

Running the main script command:

python train.py

I get:

Exception are  [AssertionError('No images match pytorch-efa01 in us-west-2, check name and image permissions')]
Traceback (most recent call last):
  File "train.py", line 453, in <module>
    main()
  File "train.py", line 305, in main
    job = ncluster.make_job(name=args.name,
  File "/usr/local/lib/python3.8/site-packages/ncluster/aws_backend.py", line 1294, in make_job
    raise exceptions[0]
  File "/usr/local/lib/python3.8/site-packages/ncluster/aws_backend.py", line 1268, in make_task_fn
    tasks[i] = make_task(f"{i}.{name}", run_name=run_name,
  File "/usr/local/lib/python3.8/site-packages/ncluster/aws_backend.py", line 1070, in make_task
    image = u.lookup_image(image_name)
  File "/usr/local/lib/python3.8/site-packages/ncluster/aws_util.py", line 397, in lookup_image
    assert len(images) > 0, f"No images match {str(wildcard)} in {get_region()}, check name and image permissions"
AssertionError: No images match pytorch-efa01 in us-west-2, check name and image permissions

> /usr/local/lib/python3.8/site-packages/ncluster/aws_util.py(397)lookup_image()
-> assert len(images) > 0, f"No images match {str(wildcard)} in {get_region()}, check name and image permissions"

I have set up the snapshots & created volumes from them using the scripts provided.

fix --spot flag