Giter Club home page Giter Club logo

upp-next-video-content-collection-mapper's Introduction

Next Video Content Collection Mapper

Circle CIGo Report Card Coverage Status

Introduction

Get the related content references from the Next video content ("related" field) by listening to kafka queue, creates a story package holding those references and puts a message with them on kafka queue for further processing and ingestion on Neo4j.

Installation

Download the source code, dependencies and test dependencies:

    go get -u github.com/kardianos/govendor
    go get -u github.com/Financial-Times/upp-next-video-content-collection-mapper
    cd $GOPATH/src/github.com/Financial-Times/upp-next-video-content-collection-mapper
    govendor sync
    go build .

Running locally

  1. Run the tests and install the binary:

     govendor sync
     govendor test -v -race
     go install
    
  2. Run the binary (using the help flag to see the available optional arguments):

     $GOPATH/bin/next-video-content-collection-mapper [--help]
    

Options:

    --app-system-code="upp-next-video-content-collection-mapper"    System Code of the application ($APP_SYSTEM_CODE)
    --app-name="Next Video Content Collection Mapper"               Application name ($APP_NAME)
    --service-name="next-video-content-collection-mapper"           Service name ($SERVICE_NAME)
    --port="8080"                                                   Port to listen on ($APP_PORT)
    --queue-addresses="http://%H:8080"                              Queue address ($Q_ADDR)
    --group="NextVideoContentCollectionMapper"                      Group used to read messages from queue ($Q_GROUP)
    --read-topic="NativeCmsPublicationEvents"                       Queue topic name from where to read the messages ($Q_READ_TOPIC)
    --read-queue="kafka"                                            The queue to read the messages from ($Q_READ_QUEUE)
    --write-topic="CmsPublicationEvents"                            Queue topic name where to write the messages ($Q_WRITE_TOPIC)
    --write-queue="kafka"                                           The queue to write the messages to ($Q_WRITE_QUEUE)

There are defaults values used for properties so when deployed locally it can be run the excutable only.

  1. Test:

curl http://localhost:8080/__next-video-content-collection-mapper/__health | jq

Build and deployment

Service endpoints

POST

/map

Example:

curl -X POST http://localhost:8080/map -H "Content-Type: application/json" -H "X-Request-Id: tid_12345" -H "X-Origin-System-Id: next-video-editor" -d @body.json

body.json:

{
	"_id": "58d8d6cc789d4c000f6b0169",
	"updatedAt": "2017-04-03T16:30:11.106Z",
	"createdAt": "2017-03-27T09:09:32.541Z",
	"mioId": 762380,
	"title": "Trump trade under scrutiny",
	"createdBy": "seb.morton-clark",
	"encoding": {
		"job": 358759376,
		"status": "COMPLETE",
		"outputs": [{
			"audioCodec": "mp3",
			"duration": 65904,
			"mediaType": "audio/mpeg",
			"url": "http://ftvideo.prod.zencoder.outputs.s3.amazonaws.com/e2290d14-7e80-4db8-a715-949da4de9a07/0x0.mp3"
		},
		{
			"audioCodec": "aac",
			"videoCodec": "h264",
			"duration": 65940,
			"mediaType": "video/mp4",
			"height": 360,
			"width": 640,
			"url": "http://ftvideo.prod.zencoder.outputs.s3.amazonaws.com/e2290d14-7e80-4db8-a715-949da4de9a07/640x360.mp4"
		},
		{
			"audioCodec": "aac",
			"videoCodec": "h264",
			"duration": 65940,
			"mediaType": "video/mp4",
			"height": 720,
			"width": 1280,
			"url": "http://ftvideo.prod.zencoder.outputs.s3.amazonaws.com/e2290d14-7e80-4db8-a715-949da4de9a07/1280x720.mp4"
		}]
	},
	"__v": 0,
	"byline": "Filmed by Niclola Stansfield. Produced by Seb Morton-Clark.",
	"description": "Global equities are on the defensive, led by weaker commodities and financials as investors scrutinise the viability of the Trump trade. The FT's Mike Mackenzie reports.",
	"image": "https://api.ft.com/content/ffc60243-2b77-439a-a6c9-0f3603ee5f83",
	"standfirst": "Mike Mackenzie provides analysis of the morning's market news",
	"updatedBy": "seb.morton-clark",
	"isPublished": false,
	"related": [{
		"uuid": "c4cde316-128c-11e7-80f4-13e067d5072c",
	}],
	"annotations": [{
		"id": "http://api.ft.com/things/059c58aa-53e6-306b-8642-e718c869ec09",
		"predicate": "http://www.ft.com/ontology/classification/isPrimarilyClassifiedBy"
	},
	{
		"id": "http://api.ft.com/things/d969d76e-f8f4-34ae-bc38-95cfd0884740",
		"predicate": "http://www.ft.com/ontology/classification/isClassifiedBy"
	},
	{
		"id": "http://api.ft.com/things/b43f1a91-b805-3453-8c36-1d164c047ca2",
		"predicate": "http://www.ft.com/ontology/annotation/mentions"
	},
	{
		"id": "http://api.ft.com/things/c2acc00f-5ad6-3f79-b6bb-d51dcd81508a",
		"predicate": "http://www.ft.com/ontology/annotation/about"
	},
	{
		"id": "http://api.ft.com/things/c2acc00f-5ad6-3f79-b6bb-d51dcd81508a",
		"predicate": "http://www.ft.com/ontology/annotation/majorMentions"
	}],
	"encodings": [{
		"mioId": 762380,
		"name": "Trump trade under scrutiny",
		"primary": true,
		"status": "COMPLETE",
		"job": 358759376,
		"outputs": [{
			"url": "http://ftvideo.prod.zencoder.outputs.s3.amazonaws.com/e2290d14-7e80-4db8-a715-949da4de9a07/0x0.mp3",
			"mediaType": "audio/mpeg",
			"duration": 65904,
			"audioCodec": "mp3"
		},
		{
			"url": "http://ftvideo.prod.zencoder.outputs.s3.amazonaws.com/e2290d14-7e80-4db8-a715-949da4de9a07/640x360.mp4",
			"width": 640,
			"height": 360,
			"mediaType": "video/mp4",
			"duration": 65940,
			"videoCodec": "h264",
			"audioCodec": "aac"
		},
		{
			"url": "http://ftvideo.prod.zencoder.outputs.s3.amazonaws.com/e2290d14-7e80-4db8-a715-949da4de9a07/1280x720.mp4",
			"width": 1280,
			"height": 720,
			"mediaType": "video/mp4",
			"duration": 65940,
			"videoCodec": "h264",
			"audioCodec": "aac"
		}]
	}],
	"canBeSyndicated": true,
	"transcription": {
		"status": "COMPLETE",
		"job": "1579674",
		"transcript": "<p>Here's what we're watching with trading underway in London. Global equities under pressure led by weaker commodities and financials as investors scrutinise the viability of the Trump trade. The dollar is weaker. Havens like yen, gold, and government bonds finding buyers. </p><p>As the dust settles over the failure to replace Obamacare, focus now on whether tax reform and other fiscal measures will eventuate. This is where the rubber meets the road for the Trump trade. High flying equity markets had been underpinned by the promise of big tax cuts and fiscal stimulus. And Wall Street is souring. </p><p>One big beneficiary of lower corporate taxes under Trump are small caps. They are now down 2 and 1/2% for the year. While the sector is still much higher since November, this is a key market barometer of prospects for the Trump trade. </p><p>Now while many still think some measure of tax reform or spending will eventuate, markets are very wary, namely of the risk that Congress and the Trump administration fail to reach agreement on legislation, that unlike health care reform, matters a great deal more to investors. </p><p>[MUSIC PLAYING] </p>",
		"captions": [{
			"format": "vtt",
			"url": "https://next-video-editor.ft.com/e2290d14-7e80-4db8-a715-949da4de9a07.vtt",
			"mediaType": "text/vtt"
		}]
	},
	"format": [],
	"type": "video",
	"id": "ad543253-ad5e-471c-a006-ebb395323028"
}

Response 200

Body:

{
	"payload": {
		"uuid": "151d4420-6ce6-3964-ad64-916561612973",
		"items": [{
			"uuid": "c4cde316-128c-11e7-80f4-13e067d5072c"
		}],
		"publishReference": "tid-12321123",
		"type": "story-package"
	},
	"contentUri": "http://next-video-content-collection-mapper.svc.ft.com/content-collection/story-package/151d4420-6ce6-3964-ad64-916561612973",
	"uuid": "151d4420-6ce6-3964-ad64-916561612973"
}

Response 400

If the mapping couldn't be performed because of invalid provided content.

Healthchecks

Admin endpoints are:

/__gtg

/__health

/__build-info

Following check is performed for health and gtg endpoints:

  • Checks that the connection to queue can be established.

Logging

  • The application uses logrus.
  • NOTE: /__build-info and /__gtg endpoints are not logged as they are called every second from varnish/vulcand and this information is not needed in logs/splunk.

upp-next-video-content-collection-mapper's People

Contributors

bernivarga avatar ioanaciont avatar iulianp26 avatar peteclark-ft avatar tabital avatar tamas-molnar avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.