Comments (7)
We have some similar objectives. For example, question answering, the answer will be followed by its bounding box. So, this is possible indeed as long as the format follows "[text sequence] "
from i-code.
Thank you for your answer @zinengtang!
So if I'm not mistaking, to do so we should first normalize the original bounding boxes in range [0, 1000] on the basis of width and height of the original image; then normalize them between [0, 1]; and then convert them into layout tokens by multiplying them for the layout vocabulary size (500). Am I getting it right?
Btw, I'm using the (not yet merged) code from the HuggingFace PR that is porting UDOP into Transformers. Works like a charm, but there might be some differences with your code.
from i-code.
@AleRosae can you share any snippets of your use of the PR? I got stuck on an early step.
Thanks in advance.
from i-code.
Hi @sromoam,
for inference you can use the standard generate() method:
model = UdopForConditionalGeneration.from_pretrained("udop_model")
outputs = model.generate(input_ids=input_ids,
bbox=bbox,
attention_mask=attention_mask,
pixel_values=pixel_values,
max_length=512,
use_cache=False,
num_beams=1,
return_dict_in_generate=True)
You can obtain input_ids, bboxes, attention_mask and pixel values using the UdopProcessor:
processor = UdopProcessor.from_pretrained("udop_model", apply_ocr=True)
encoding = processor(images=image, return_tensors="pt").to(device)
For finetuning, you can follow the Pix2Struct tutorial. Just be sure to also include words and bboxes in your dataloader, as Pix2Struct only takes images as input.
from i-code.
Hi @sromoam, for inference you can use the standard generate() method:
model = UdopForConditionalGeneration.from_pretrained("udop_model") outputs = model.generate(input_ids=input_ids, bbox=bbox, attention_mask=attention_mask, pixel_values=pixel_values, max_length=512, use_cache=False, num_beams=1, return_dict_in_generate=True)
You can obtain input_ids, bboxes, attention_mask and pixel values using the UdopProcessor:
processor = UdopProcessor.from_pretrained("udop_model", apply_ocr=True) encoding = processor(images=image, return_tensors="pt").to(device)
For finetuning, you can follow the Pix2Struct tutorial. Just be sure to also include words and bboxes in your dataloader, as Pix2Struct only takes images as input.
can you please provide which libraries did you import for UdopForConditionalGeneration as i am getting error like this
error : ImportError: cannot import name 'UdopForConditionalGeneration' from 'transformers'
from i-code.
Hi @sromoam, for inference you can use the standard generate() method:
model = UdopForConditionalGeneration.from_pretrained("udop_model") outputs = model.generate(input_ids=input_ids, bbox=bbox, attention_mask=attention_mask, pixel_values=pixel_values, max_length=512, use_cache=False, num_beams=1, return_dict_in_generate=True)
You can obtain input_ids, bboxes, attention_mask and pixel values using the UdopProcessor:
processor = UdopProcessor.from_pretrained("udop_model", apply_ocr=True) encoding = processor(images=image, return_tensors="pt").to(device)
For finetuning, you can follow the Pix2Struct tutorial. Just be sure to also include words and bboxes in your dataloader, as Pix2Struct only takes images as input.
can you please provide which libraries did you import for UdopForConditionalGeneration as i am getting error like this error : ImportError: cannot import name 'UdopForConditionalGeneration' from 'transformers'
solved the issue we need to have transformers version = "4.39.0.dev0"
which can be cloned from here https://github.com/huggingface/transformers/blob/main/src/transformers/__init__.py
the commit on Mar 18, 2024
from i-code.
@zinengtang I want to use the processor with my own OCR. What should be the format of the bouding boxes? 1. Normalized with heigh and width? 2. Normalized with height and width * 1000 3. Other option?
from i-code.
Related Issues (20)
- from core.common.utils import img_trans_torchvision, get_visual_bbox Module not found error
- layout token unkown HOT 1
- special vis token
- Image loading in dataloader code HOT 5
- Img2txt result is pretty bad on 16bit HOT 4
- Img2Img Broken? HOT 1
- Trainig pipeline of CoDi in i-Code-V3 HOT 6
- i-Code-V3: How could I implement training tasks in i-Code-V3?
- Can you provide classifier-free guidance probability? HOT 1
- Cuda out of memory HOT 1
- Finetuning on InfographicVQA HOT 12
- Release of the model UDOP-1024
- UDOP document editing/generation
- COCO Caption benchmark
- CoDi: Fine Tuning
- VAE in Training of each LDM in CoDi HOT 1
- Document Classification Mertic (UDOP) HOT 2
- Is the pretrained mae encoder weights available ? HOT 2
- Inference VRAM requirements? HOT 2
- Data Collator Incorrect When Using a Decoder Prefix
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from i-code.