The haha_2019 from bfarzin

haha_2019's People

Contributors

Stargazers

Watchers

haha_2019's Issues

CustTok returns backward tokens

Update to return backwards tokens with CustomTokenizer with bwd=True flag. Start with xxbos and then return the series of tokens in reverse order [::-1]. Proceed as usual with everything else. Model works "backwards" and should produce slightly different results. Ensemble with forward models and submit scores.

https://raw.githubusercontent.com/bfarzin/wiki103_from_scratch/master/Wiki103_from_scratch_fastai_v1_raw_articles-backward_tokens.ipynb

    "class Tokenizer():\n",
    "    \"Put together rules and a tokenizer function to tokenize text with multiprocessing.\"\n",
    "    def __init__(self, tok_func:Callable=SpacyTokenizer, lang:str='en', pre_rules:ListRules=None,\n",
    "                 post_rules:ListRules=None, special_cases:Collection[str]=None, n_cpus:int=None,bwd_toks=False):\n",
    "        self.tok_func,self.lang,self.special_cases = tok_func,lang,special_cases\n",
    "        self.pre_rules  = ifnone(pre_rules,  defaults.text_pre_rules )\n",
    "        self.post_rules = ifnone(post_rules, defaults.text_post_rules)\n",
    "        self.special_cases = special_cases if special_cases else defaults.text_spec_tok\n",
    "        self.n_cpus = ifnone(n_cpus, defaults.cpus)\n",
    "        self.bwd_toks = bwd_toks\n",
    "        \n",
    "...\n",
    "\n",
    "    def process_text(self, t:str, tok:BaseTokenizer) -> List[str]:\n",
    "        \"Process one text `t` with tokenizer `tok`.\"\n",
    "        for rule in self.pre_rules: t = rule(t)\n",
    "        toks = tok.tokenizer(t)\n",
    "        for rule in self.post_rules: toks = rule(toks)\n",
    "        return listify(toks[0])+toks[1:][::-1] if self.bwd_toks else toks  # <-- Just one line changed!"
   ]
  },

Recommend Projects