transloadit / go-sdk Goto Github PK
View Code? Open in Web Editor NEWTransloadit's official Go SDK, maintained by the community
Home Page: https://transloadit.com
License: MIT License
Transloadit's official Go SDK, maintained by the community
Home Page: https://transloadit.com
License: MIT License
So a log entry, the moment it discovers a new file. We can keep
2014/09/17 12:47:04 Successfully converted 'clip-2012-12-29 17;39;19.mov'.
as well.
For node modules, we have
release-major: build test
npm version major -m "Release %s"
git push
npm publish
I would like something similar for this Go project. So e.g. make release-minor
:
transloadify-<paltform>-<arch>-<version>
Remove the test template we create, so the test account does not end up with unnecessarily many templates.
To avoid
$ ./transloadify \
→ -key=xxxxxxxx \
→ -secret=xxxxxxxx \
→ -template=xxxxxxxx \
→ -input=./incoming \
→ -watch=true \
→ -output=./converted
2014/09/17 12:44:25 Converting all files in './incoming' using template 'xxxxxxxx' and putting the result into './converted'.
2014/09/17 12:44:25 Watching directory './incoming' for changes...
2014/09/17 12:47:04 Successfully converted 'clip-2012-12-29 17;39;19.mov'.
2014/09/17 12:48:05 Successfully converted 'clip-2012-12-29 17;39;19.mov'.
For Bonus points: You'll see in the node-sdk that the getBored function can fall back on an infra- file on S3 that contains all uploaders and picks a random one: https://github.com/transloadit/node-sdk/blob/master/src/TransloaditClient.coffee#L266. This is to shield the client from the unlikely event that the machine you got routed to (via dns, so it's sticky for a while), is unable to provide a bored instance due to high load or defects.
As discussed here: 2c5b62c#commitcomment-7580913
Scanning a local directory for new/incoming files, that will be automatically uploaded to Transloadit, converted by us using a template_id, and the results then saved to another local directory. The usecase here being people running their own ftp/upload system, and interested in converting every video to HLS for instance.
Currently they can be specified as cli args, but that is less than ideal as you can see the credentials in e.g. ps auxf
$ ./transloadify -h
Usage of ./transloadify:
-input=".": Input directory
-key="": Auth key
-output="": Output directory
-secret="": Auth secret
-template="": Template's id to create assemblies with
-watch=false: Watch input directory for changes
Discussed this with @tim-kos, and the Go SDK should add "queue": "batch", to every assembly.
That way the right queue can be selected for all steps. Wether coming from disk, or template_id.
It does not make sense if steps can have different queues. That will just make everything as slow as the weakest link
Polling assembly status so you can block locally, until we have finished encoding. Useful for piping/chaining CLI tools.
We have a blocking: true
option, so that we keep the connection with the client open until all encoding/processing is done. But it's undocumented and we'll probably phase it out. A better way would be for the client to periodically check the assembly status, and release control once it sees ASSEMBLY_COMPLETED, or an error.
This allows Tranloadit server processes to crash, or the network path to fail, making it less prone to breakage than keeping 1 connection open.
Hey!
WaitAssembly
fails for me with the following error: failed unmarshal http request: json: cannot unmarshal string into Go struct field AssemblyInfo.bytes_expected of type int
The corresponding assembly response is below. You can clearly see that bytes_expected
is null.
{
"ok": "ASSEMBLY_COMPLETED",
"http_code": 200,
"message": "The Assembly was successfully completed.",
"assembly_id": "10182eee988945da98a455cc1e3d1319",
"parent_id": null,
"account_id": "***REDACTED***",
"template_id": "229a0e60629241e3848af05038322ee6",
"instance": "rang.transloadit.com",
"assembly_url": "http://api2.rang.transloadit.com/assemblies/10182eee988945da98a455cc1e3d1319",
"assembly_ssl_url": "https://api2-rang.transloadit.com/assemblies/10182eee988945da98a455cc1e3d1319",
"uppyserver_url": "https://api2-rang.transloadit.com/companion/",
"companion_url": "https://api2-rang.transloadit.com/companion/",
"websocket_url": "https://api2-rang.transloadit.com/ws20034",
"tus_url": "https://api2-rang.transloadit.com/resumable/files/",
"bytes_received": 118604,
"bytes_expected": null,
"upload_duration": 0.886,
"client_agent": null,
"client_ip": null,
"client_referer": null,
"transloadit_client": "go-sdk:v1.1.1",
"start_date": "2019/11/15 13:51:53 GMT",
"upload_meta_data_extracted": true,
"warnings": [],
"is_infinite": false,
"has_dupe_jobs": false,
"execution_start": "2019/11/15 13:51:54 GMT",
"execution_duration": 1.727,
"queue_duration": 0.317,
"jobs_queue_duration": 0.01,
"notify_start": null,
"notify_url": null,
"notify_status": null,
"notify_response_code": null,
"notify_duration": null,
"last_job_completed": "2019/11/15 13:51:56 GMT",
"fields": {},
"running_jobs": [],
"bytes_usage": 763547,
"executing_jobs": [],
"started_jobs": [
":original:::original",
"original:::original",
"resize_full1080::original",
"resize_thumb360::original",
"export::original",
"thumb360::resize_thumb360",
"full1080::resize_full1080",
"export::thumb360",
"export::full1080"
],
"parent_assembly_status": null,
"params": "{\"auth\":{\"key\":\"****\",\"expires\":\"2019/11/15 14:51:53+00:00\"},\"template_id\":\"229a0e60629241e3848af05038322ee6\"}",
"template": "{\"steps\":{\":original\":{\"robot\":\"/upload/handle\"},\"original\":{\"use\":\":original\",\"robot\":\"/file/filter\",\"accepts\":[[\"${file.mime}\",\"regex\",\"image\"]],\"error_on_decline\":true},\"resize_full1080\":{\"use\":\"original\",\"robot\":\"/image/resize\",\"resize_strategy\":\"min_fit\",\"format\":\"jpg\",\"quality\":90,\"width\":1080,\"height\":1080,\"strip\":true,\"zoom\":true,\"background\":\"#ffffff\",\"rotation\":true,\"imagemagick_stack\":\"v2.0.7\"},\"resize_thumb360\":{\"use\":\"original\",\"robot\":\"/image/resize\",\"resize_strategy\":\"min_fit\",\"format\":\"jpg\",\"quality\":90,\"width\":360,\"height\":360,\"strip\":true,\"zoom\":true,\"background\":\"#ffffff\",\"rotation\":true,\"imagemagick_stack\":\"v2.0.7\"},\"full1080\":{\"robot\":\"/image/optimize\",\"use\":\"resize_full1080\",\"priority\":\"compression-ratio\"},\"thumb360\":{\"robot\":\"/image/optimize\",\"use\":\"resize_thumb360\",\"priority\":\"compression-ratio\"},\"export\":{\"use\":[\"original\",\"full1080\",\"thumb360\"],\"robot\":\"/s3/store\",\"credentials\":\"s3_user_content\",\"path\":\"${previous_step.name}/${file.original_id}_${previous_step.name}.${file.ext}\",\"acl\":\"private\"}}}",
"merged_params": "{\"steps\":{\":original\":{\"robot\":\"/upload/handle\"},\"original\":{\"use\":\":original\",\"robot\":\"/file/filter\",\"accepts\":[[\"${file.mime}\",\"regex\",\"image\"]],\"error_on_decline\":true},\"resize_full1080\":{\"use\":\"original\",\"robot\":\"/image/resize\",\"resize_strategy\":\"min_fit\",\"format\":\"jpg\",\"quality\":90,\"width\":1080,\"height\":1080,\"strip\":true,\"zoom\":true,\"background\":\"#ffffff\",\"rotation\":true,\"imagemagick_stack\":\"v2.0.7\"},\"resize_thumb360\":{\"use\":\"original\",\"robot\":\"/image/resize\",\"resize_strategy\":\"min_fit\",\"format\":\"jpg\",\"quality\":90,\"width\":360,\"height\":360,\"strip\":true,\"zoom\":true,\"background\":\"#ffffff\",\"rotation\":true,\"imagemagick_stack\":\"v2.0.7\"},\"full1080\":{\"robot\":\"/image/optimize\",\"use\":\"resize_full1080\",\"priority\":\"compression-ratio\"},\"thumb360\":{\"robot\":\"/image/optimize\",\"use\":\"resize_thumb360\",\"priority\":\"compression-ratio\"},\"export\":{\"use\":[\"original\",\"full1080\",\"thumb360\"],\"robot\":\"/s3/store\",\"credentials\":\"s3_user_content\",\"path\":\"${previous_step.name}/${file.original_id}_${previous_step.name}.${file.ext}\",\"acl\":\"private\"}},\"allow_steps_override\":true,\"auth\":{\"key\":\"****\",\"expires\":\"2019/11/15 14:51:53+00:00\"},\"template_id\":\"229a0e60629241e3848af05038322ee6\"}",
"uploads": [
{
"id": "dece9a004dda4a3ebbeb65eada3ddadf",
"name": "xxx.jpg",
"basename": "xxx",
"ext": "jpg",
"size": 117953,
"mime": "image/jpeg",
"type": "image",
"field": "asset",
"md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
"original_id": "dece9a004dda4a3ebbeb65eada3ddadf",
"original_basename": "xxx",
"original_name": "xxx.jpg",
"original_path": "/",
"original_md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
"from_batch_import": false,
"is_tus_file": false,
"tus_upload_url": null,
"url": "http://tmp-eu-west-1.transloadit.com.s3-eu-west-1.amazonaws.com/dece9a004dda4a3ebbeb65eada3ddadf.jpg",
"ssl_url": "https://s3-eu-west-1.amazonaws.com/tmp-eu-west-1.transloadit.com/dece9a004dda4a3ebbeb65eada3ddadf.jpg",
"meta": {
"width": 341,
"height": 341,
"date_recorded": null,
"date_file_created": null,
"date_file_modified": "2019/11/15 13:51:54 GMT",
"title": null,
"description": null,
"duration": null,
"location": null,
"aspect_ratio": "1.000",
"city": null,
"state": null,
"country": null,
"country_code": null,
"keywords": null,
"aperture": null,
"exposure_compensation": null,
"exposure_mode": null,
"exposure_time": null,
"flash": null,
"focal_length": null,
"f_number": null,
"iso": null,
"light_value": null,
"metering_mode": null,
"shutter_speed": null,
"white_balance": null,
"device_name": null,
"device_vendor": null,
"device_software": "Picasa",
"latitude": null,
"longitude": null,
"orientation": "Horizontal (normal)",
"has_clipping_path": false,
"creator": null,
"author": null,
"copyright": null,
"copyright_notice": null,
"frame_count": 1,
"colorspace": "RGB",
"has_transparency": false,
"dominant_colors": null,
"average_color": "#886a54",
"xp_title": null,
"xp_comment": null,
"xp_keywords": null,
"xp_subject": null
}
}
],
"results": {
"original": [
{
"id": "dece9a004dda4a3ebbeb65eada3ddadf",
"name": "xxx.jpg",
"basename": "xxx",
"ext": "jpg",
"size": 117953,
"mime": "image/jpeg",
"type": "image",
"field": "asset",
"md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
"original_id": "dece9a004dda4a3ebbeb65eada3ddadf",
"original_basename": "xxx",
"original_name": "xxx.jpg",
"original_path": "/",
"original_md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
"from_batch_import": false,
"is_tus_file": false,
"tus_upload_url": null,
"url": "***REDACTED***",
"ssl_url": "***REDACTED***",
"meta": {
"width": 341,
"height": 341,
"date_file_modified": "2019/11/15 13:51:54 GMT",
"aspect_ratio": "1.000",
"device_software": "Picasa",
"orientation": "Horizontal (normal)",
"has_clipping_path": false,
"frame_count": 1,
"colorspace": "RGB",
"has_transparency": false,
"average_color": "#886a54"
},
"cost": 23591,
"queue": "live",
"queueTime": 0,
"execTime": 0.58
}
],
"thumb360": [
{
"id": "17c477802c884ea39b768ce873443125",
"name": "xxx.jpg",
"basename": "xxx",
"ext": "jpg",
"size": 17366,
"mime": "image/jpeg",
"type": "image",
"field": "asset",
"md5hash": "04185b2e84f28e56c30318a6143dc2e7",
"original_id": "dece9a004dda4a3ebbeb65eada3ddadf",
"original_basename": "xxx",
"original_name": "xxx.jpg",
"original_path": "/",
"original_md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
"from_batch_import": false,
"is_tus_file": false,
"tus_upload_url": null,
"url": "http***REDACTED***",
"ssl_url": "***REDACTED***",
"meta": {
"width": 360,
"height": 360,
"date_file_modified": "2019/11/15 13:51:54 GMT",
"aspect_ratio": "1.000",
"has_clipping_path": false,
"frame_count": 1,
"colorspace": "sRGB",
"has_transparency": false,
"average_color": "#886a54"
},
"cost": 55424,
"queue": "live",
"queueTime": 0,
"execTime": 0.43
}
],
"full1080": [
{
"id": "165b7965da2c4751a38d5a1c38b21b82",
"name": "xxx.jpg",
"basename": "xxx",
"ext": "jpg",
"size": 77887,
"mime": "image/jpeg",
"type": "image",
"field": "asset",
"md5hash": "f754a69ad50523e674a484a8c0bdcfee",
"original_id": "dece9a004dda4a3ebbeb65eada3ddadf",
"original_basename": "xxx",
"original_name": "xxx.jpg",
"original_path": "/",
"original_md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
"from_batch_import": false,
"is_tus_file": false,
"tus_upload_url": null,
"url": "***REDACTED***",
"ssl_url": "***REDACTED***",
"meta": {
"width": 1080,
"height": 1080,
"date_file_modified": "2019/11/15 13:51:55 GMT",
"aspect_ratio": "1.000",
"has_clipping_path": false,
"frame_count": 1,
"colorspace": "sRGB",
"has_transparency": false,
"average_color": "#886954"
},
"cost": 250125,
"queue": "live",
"queueTime": 0.01,
"execTime": 0.55
}
]
},
"build_id": "300"
}
The code producing the error is really nothing special:
assembly := transloadit.NewAssembly()
assembly.TemplateID = "229a0e60629241e3848af05038322ee6"
data, err := ioutil.ReadAll(body)
if err != nil {
return nil, err
}
err = ioutil.WriteFile("/tmp/upload.jpg", data, 0644)
if err != nil {
panic(err)
}
log.Printf("assets: upload saved as /tmp/upload.jpg")
assembly.AddReader("asset", "xxx.jpg", ioutil.NopCloser(bytes.NewReader(data)))
log.Printf("assets: uploading...")
info, err := as.trans.StartAssembly(ctx, assembly)
if err != nil {
log.Printf("WARNING: assets: uploading failed: %+v", err)
return nil, err
}
log.Printf("assets: uploading complete")
spew.Dump(info)
log.Printf("assets: checking %v", info.AssemblySSLURL)
info, err = as.trans.WaitForAssembly(ctx, info)
if err != nil {
log.Printf("WARNING: assets: processing failed: %+v", err)
return nil, err
}
log.Printf("assets: processing complete")
spew.Dump(info)
and it fails at WaitForAssembly
call.
Note: I think I've redacted all private data. The image itself isn't private, just my userpic.
While discussing how to implement #5 to support the studio/producer
use-case it slowly became clear this should be a separate small cli utility, albeit supported by the lib for most part.
TRANSLOADIT_ACCOUNT=abc TRANSLOADIT_SECRET=def ./transloadify -template_id=ghi -dir_in=./incoming -dir_out=./var/www/carshow.tv/results
I would like it if
make build
I would love it if
I would lose my mind if
I think developers would love to point transloadify
to a ./some/path/in/git/encode-to-ipad.json
with instructions.
This way they can directly hack on templates (or just overriding parameters) without going back & forth to the crm.
Here's what I'm envisioning:
./transloadity -template=<id> # just a stored template
./transloadity -template=<id> -parameters=./params.json # a stored template, with tweaked parameters found in the json
./transloadity -parameters=./params.json # everything is coming out of the json, no stored template was used
Up for discussion
A readme upgrade somehow broke the tests. There must be a bug/race lurking somewhere. This should be addressed before we can confidently release this project
Felix' Feedback
As you mentioned, @Acconut, the godocs aren't fully finished yet. We would like that ofc before we release.
E.g.
kevin at kvz-mbp in ~
$ ./transloadify \
→ -template-file=./hls.json \
→ -input=~/Downloads \
→ -watch=true \
→ -preserve=true \
→ -output=~/Encoded
2014/09/25 12:53:35 Converting all files in '~/Downloads' and putting the result into '~/Encoded'.
2014/09/25 12:53:35 Using template file './hls.json' (read 16 steps).
2014/09/25 12:53:35 Watching directory '~/Downloads' for changes...
fatal error: all goroutines are asleep - deadlock!
goroutine 16 [chan send]:
github.com/transloadit/go-sdk.(*Watcher).processDir(0x2084f8640)
/home/gobuild/goproj/src/github.com/codeskyblue/gobuild/gopath/src/github.com/transloadit/go-sdk/watch.go:87 +0xdc
github.com/transloadit/go-sdk.(*Watcher).start(0x2084f8640)
/home/gobuild/goproj/src/github.com/codeskyblue/gobuild/gopath/src/github.com/transloadit/go-sdk/watch.go:60 +0x27
github.com/transloadit/go-sdk.(*Client).Watch(0x208554b40, 0x2084e4300, 0x0)
/home/gobuild/goproj/src/github.com/codeskyblue/gobuild/gopath/src/github.com/transloadit/go-sdk/watch.go:52 +0x13b
main.main()
/home/gobuild/goproj/src/github.com/codeskyblue/gobuild/gopath/src/github.com/transloadit/go-sdk/transloadify/transloadify.go:94 +0x75d
goroutine 19 [finalizer wait]:
runtime.park(0x15950, 0x476da8, 0x475889)
/home/gobuild/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x476da8, 0x475889)
/home/gobuild/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
/home/gobuild/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
/home/gobuild/go/src/pkg/runtime/proc.c:1445
I think the readme lacks quite a bit of documentation for all the available public methods. An example would be https://github.com/transloadit/node-sdk
It would be nice if the Readme also included where a template is used.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.