An interface to test our GEC environments.

• State transitions (state, action, reward, new state, done)
• Environment resetting
• Environment rendering
• Random action samples
• Update reward weights
• Update sampler weights
• Save interaction

• Implement the model (transformer encoder + final layer)
• Post-process encoding to get one value per input token
  • Apply reduce_mean and reduce_last

Text-based OpenAI Gym Environment

Data Format

A JSON file with source and list of target sentences.

• Script to convert M2 into JSON
• Data cleaning:
  • Normalize characters (both source and references)
  • Remove emojis M2 file does not have any emojis
  • Fix typos using Pyspellchecker (only source sentences. References are hopefully corrected!)
• Data filtering:
  • Number of tokens in the original sentences
  • References ending properly
  • Mean similarity between the original and reference sentences
• Generate metadata [Optional]
  • Number of sentences
  • Number of different references per sentence

Episode

• Reset
  • Select the state
    • Source and target sentences
• Take a step
  • Calculate the reward
    • Generate intermediate tokens
    • Compute GLEU scores with target sentences
    • Subtract \epsilon from the reward
  • Obtain the new state
  • Check episode termination
    • N-steps attempted
    • Token length condition
    • All keep actions condition
  • Return new state, reward and done

Reward function

$$r(s_t, a_t) = r_{gleu} + r_{delay} + r_{invalid-label}$$

• Create a simple Replay Buffer
• Implement the Q-loss function
• Implement target network weight updates

Create a weighted random sampler that generates biased random actions.

Features:

• Limit the number of non-keep labels in an action Done implicitly using the weights
• Generate sample weights based on label weights