Integrate Set-of-Mark Visual Prompting for GPT-4V about self-operating-computer HOT 8 CLOSED

I am trying to implement SoM, since it seems to have the best accuracy.

from self-operating-computer.

@0xdevalias read up more on SoM. It looks like a very promising approach, thank you for opening this issue!

https://github.com/microsoft/SoM

from self-operating-computer.

I am implementing SoM now, and seems like the best way is to make another mode like som-mode and make a new prompt for the mode.

from self-operating-computer.

@Daisuke134 @0xdevalias Set-of-Mark prompting is now available. Swap in your best.pt from a YOLOv8 model and see how it performs!

from self-operating-computer.

@Daisuke134 interested to see what you find. I'm going to go learn more about SoM

from self-operating-computer.

read up more on SoM. It looks like a very promising approach, thank you for opening this issue!

@joshbickett No worries :)

from self-operating-computer.

See also:

• #31

from self-operating-computer.

I have been testing out SoM and seems pretty good. Here is the screenshot..
I will try adding this today, test it, and make PR.

from self-operating-computer.