Comments (5)
I'm not sure what you mean by "custom Huggingface Model". Would need to expand on that.
If you just want to use that model, you can run hgf"bert-base-german-cased"
from transformers.jl.
The "custom" was confusing, so i changed it.
We were able to run the german based bert models with hgf"some-german-bert-model".
Now we are trying to run dbmdz/german-gpt2 by hgf"dbmdz/german-gpt2".
but getting:
ERROR: KeyError: key :vocab not found
Stacktrace:
[1] getindex(h::Dict{String, Any}, key::Symbol)
@ Base ./dict.jl:481
[2] load_tokenizer(::Val{:gpt2}, model_name::String; force_fast_tkr::Bool, possible_files::Vector{String}, config::Transformers.HuggingFace.HGFGPT2Config, tkr_cfg::Nothing, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Transformers.HuggingFace ~/.julia/packages/Transformers/xjEIh/src/huggingface/implementation/gpt2/tokenizer.jl:50
[3] #load_tokenizer#103
@ ~/.julia/packages/Transformers/xjEIh/src/huggingface/tokenizer/tokenizer.jl:34 [inlined]
[4] load_tokenizer(model_name::String; possible_files::Nothing, config::Transformers.HuggingFace.HGFGPT2Config, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Transformers.HuggingFace ~/.julia/packages/Transformers/xjEIh/src/huggingface/tokenizer/tokenizer.jl:31
[5] load_hgf_pretrained(name::String)
@ Transformers.HuggingFace ~/.julia/packages/Transformers/xjEIh/src/huggingface/HuggingFace.jl:56
[6] top-level scope
@ REPL[6]:1
[7] top-level scope
@ ~/.julia/packages/CUDA/DfvRa/src/initialization.jl:52
Shouldn't the GPT2 Models be interchangeable in the GPT2 Text Generation Example?
from transformers.jl.
Shouldn't the GPT2 Models be interchangeable in the GPT2 Text Generation Example?
It should be. That's probably a bug. I'll fixed it this weekend.
Is this the only question of this issue?
from transformers.jl.
I found out that the main branch of dbmdz/german-gpt2 doesnt have a vocab.json.
An older version dbmdz/german-gpt2-faust has the vocab.json file and is also working, so it is not a problem with this lib.
Thank you for the fast response!
from transformers.jl.
It take longer than I thought, but with Transformers v0.1.22 it should be able to load tokenizer from dbmdz/german-gpt2
.
from transformers.jl.
Related Issues (20)
- Downloading BERT yields a 404 HOT 2
- Implementation of recommender system based on Transformers HOT 1
- Load local model HOT 1
- The text generation example outputs an error related to print_tree() HOT 1
- Masks are broken. HOT 5
- HuggingFace download broken on Julia 1.10 (by ValSplit.jl?) HOT 1
- Gpt2 tokenizer does not support different vocab_size HOT 1
- Adding support for checkpointing HOT 12
- update NNlib and Flux compat HOT 9
- State of quantization HOT 3
- Dolly example no longer works ... HOT 19
- OWL-ViT HOT 1
- AMDGPU support HOT 1
- DistilBertModel support HOT 1
- Attempting to download CLIP yields UnderVarError `unk_token` not defined
- Performance issue HOT 1
- [Question] Possible to retrieve layer-wise activations? HOT 4
- Adding phi model HOT 5
- Please support Lux.jl HOT 7
- Example Code always produces Max Length Sequences
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.jl.