captaincodeman / datastore-mapper Goto Github PK

Whatever the job has implemented

I try to start and control a job from code, but cannot find the appropriate functions and interfaces. Could you please provide an example?

Generally, the public interfaces are hard to find. Maybe you could add the library to go-search.org and to godoc.org. Another possibility would be to move the implementation to a sub-package.

replace hard-coded values with real values

Allow a separate queue to be used for control tasks so that they can be given priority and the pool of real workers (doing the grunt tasks) controlled better.

Tidy up development examples to show how to consume the package and also what the outputs are.

If not scatter index exists for a query with a filter, use ordinal splitting

Nice Polymer web interface to present the REST API

Slice -> shard
Shard -> namespace
Namespace -> job

Both logic and implementation.

Why does the legacy implementation do so much extra work when it should have a lock?
Is it possible for a task to still be executing after another takes over the lease? etc...

Check on task start as part of lock: return error but with http OK status to abort any task retry
Would allow processing to halt and tasks to quickly drain

Existing rolled-up files can be skipped, single file rollups can be done as a move (or copy/delete) and rollup needs to handle the number of source files being over the cloud storage rollup limit (i.e. perform iterative rollup).

Add options to cleanup datastore mapper entities

Date range? ID? Auto cleanup job on completion (e.g. if cron tasks running frequently)

Make it easy to export data to cloud storage with the framework managing files etc...
Utilize cloud storage file chunks to rollup slices into shard, shard into namespace, namespace into job
Makes it easy to write - slices own their own file and can just overwrite in the event of a retry
Possible limitation is how many times files chunks can be combined (a bizarre limit of cloud storage, I can only imagine to protect against some runaway fork-bomb situation)

please add the license file.

BTW, thank you very much for this project.

Interaction with google.golang.org/appengine/datastore breaks with its newest version.
Error may reside within datastore-locker. But unlikely.

Running example1 causes no output or error message.

Logging the error output from Schedule in task.go (datastore-locker) will output:
'datastore: multiple Properties with Name "active", but Multiple is false'.
The property name changes each time to a property of common.

Not sure of the root cause (or change from google.golang.org/appengine/datastore) but it is possibly due to merging the entity and common properties. (job.go line 118)

In addition to updating the code to conform to the newest version of google.golang.org/appengine/datastore. I would recommend handling errors from datastore-locker/task.go (Schedule function specifically) within the startJobHandler function (job_spec.go, line 173).

I created a job that should transform entities from one kind to another by getting the values from the request:

`func (x *transform) Query(r *http.Request) (*mapper.Query, error) {
c := appengine.NewContext(r)

err := r.ParseForm()
if err != nil {
	return nil, err
}

from := r.Form.Get("from")
to := r.Form.Get("to")

q := mapper.NewQuery(from)
q = q.NamespaceEmpty()
return q, nil

}
`

How can I pass this value ("to") to
func (x *transform) Next(c context.Context, counters mapper.Counters, key *datastore.Key) error
?

Just to indicate progress for long jobs

Remove repetition
Allow credentials file / json to be passed in (write to non-app-owned buckets)

Allow a certain number of slice failures and either abort the job or continue with that slice missing

Allow querying and control via RESTful API