xerions / ecto_export Goto Github PK

An export must be pre-/post-fixed with a header/footer.
preferably it should be a header but at the beginning of an export not all data may be available (at least not in an efficient and consistent way).

the following attributes should be included:

• schema version
• schema
• timestamp
• used filter (where: )
• exported record count
• username triggering the export
• machine: host/maschine_id, erlang nodename, release version
• jobid
• environment: database name and version

some attributes may require further discussion or may be dropped because they are not available in the context of ecto_export but must be handled by exd.

Quoting from the export docs:

  Start an export or import job.
  ## Params
  * `repo` - mandatory, the Ecto.Repo which contains the models to be exported
  * `models` - mandatory, list of modules which implement Ecto.Model
  * `options` - optional, Map whith the following options
    * `"filename"` - mandatory, string, the filename to export/import from
    * `"import"` - boolean, if true starts an import - default is false
    * `"formatter"` - atom, defines which module to use for string conversion - default is `Ecto.Export.Formatter.JSON`
  ## Result
  `{:ok, job_id}`

The mandatory option filename shows a painful limitation. ecto_export (as a low level library) must provide more flexible ways of accessing the data. Dumping data to stdout is the least ecto_export should support. That is todays "norm".
Compare to dump functionality of database systems, and even docker which allows piping whole images through a bare http-client based CLI. Compare to https://docs.docker.com/reference/api/docker_remote_api_v1.20/ REST endpoint GET /containers/(id)/export and still you can stream that on a UNIX system with the docker CLI.

persist the last used job id so they are not given out twice if the application running ecto_export restarts.

It shouldn't try to process all entries at once.