cc-archive / cccatalog-api Goto Github PK

thanks!

Example: https://ccsearch.creativecommons.org/photos/15621775

Error:

Traceback (most recent call last):                                                                                    
  File "/usr/local/lib/python3.7/site-packages/rest_framework/fields.py", line 441, in get_attribute                  
    return get_attribute(instance, self.source_attrs)                                                                 
  File "/usr/local/lib/python3.7/site-packages/rest_framework/fields.py", line 98, in get_attribute                   
    instance = instance[attr]                                                                                         
KeyError: 'name' 

Tags for this image (note the empty tag):

15621775 | [{}, {"name": "business", "accuracy": 0.97734, "provider": "clarifai"}, {"name": "man", "accuracy": 0.93226, "provider": "clarifai"}, {"name": "paper", "accuracy": 0.89205, "provider": "clarifai"}, {
"name": "presentation", "accuracy": 0.89086, "provider": "clarifai"}, {"name": "people", "accuracy": 0.86843, "provider": "clarifai"}, {"name": "contemporary", "accuracy": 0.85604, "provider": "clarifai"}, {"nam
e": "identity", "accuracy": 0.84483, "provider": "clarifai"}, {"name": "facts", "accuracy": 0.81679, "provider": "clarifai"}, {"name": "blank", "accuracy": 0.81582, "provider": "clarifai"}, {"name": "intelligenc
e", "accuracy": 0.81183, "provider": "clarifai"}, {"name": "bill", "accuracy": 0.80434, "provider": "clarifai"}, {"name": "achievement", "accuracy": 0.7914, "provider": "clarifai"}, {"name": "horizontal", "accur
acy": 0.78989, "provider": "clarifai"}, {"name": "education", "accuracy": 0.77418, "provider": "clarifai"}, {"name": "template", "accuracy": 0.77406, "provider": "clarifai"}, {"name": "fine-looking", "accuracy":
 0.77116, "provider": "clarifai"}, {"name": "indoors", "accuracy": 0.76115, "provider": "clarifai"}, {"name": "company", "accuracy": 0.75583, "provider": "clarifai"}, {"name": "looking", "accuracy": 0.75398, "pr
ovider": "clarifai"}, {"name": "banner", "accuracy": 0.74502, "provider": "clarifai"}]

To Reproduce:

Visit
https://api.creativecommons.engineering/image/search?q=nature&provider=nypl
https://api.creativecommons.engineering/image/search?q=nature&provider=met
and verify images are returned.

Add the provider type as a parameter(&provider=nypl,met ).

https://api.creativecommons.engineering/image/search?q=nature&provider=nypl,met

Notice you get the following:

PUT /list/{id} to edit existing list
DELETE /list/{id}

Both will require an HTTP Basic token. This is intentionally simplistic from a security standpoint; we want users to be able to easily share and curate lists anonymously.

When you find an image (https://ccsearch.creativecommons.org/photos/16136571 for example) and what to explore further using the tags there should be a special category or filtering of all common names of CC licenses.

When you click on a tag for, example CC0, you expect to find more cc0 material. However that tag also appears on non cc0 licensed material. This is confusion to the end-user.

• Elasticsearch cluster
• EC2 instances

We will keep the database around just in case we need to import any other data to the new system.

Check Postgres for updates on tables every few seconds. Whenever there is sizable change to the table, use the bulk insert API to push the data to Elasticsearch.

This should improve our indexing speed considerably over the old Django signals based implementation.

Primary keys are an implementation detail that can change without notice (such as if we were to migrate to a new database). As such, we need to expose an alternative identifier besides the primary key.

This means we will have to set an alternative unique identifier on each image when it is loaded in the database. For entities that are created on the API server, such as lists, this can be done upon creation, but for entities created by the data pipeline (images), the unique identifier must be assigned when it is loaded into the database. See uuslug; we would have to perform an operation similar to this in the data pipeline.

@sclachar, any thoughts on this? Could we use the "identifier" column for this purpose (after adding a unique and not null constraint)?

This API will have clients which will need consumer keys and secrets (OAuth)...

Let's implement OAuth and see if we can use CCID for these purposes.

Return the top N (between 1 and 10000) tags and a count of their occurrences.

Query
curl -X GET api-dev.creativecommons.engineering/tag/n=10

Response

{
  "portrait" = 32843,
  "World War 1" = 22412,
  "vehicle" = 21032,
  "woman" = 19328,
  . . .
}

Cache Elasticsearch query server-side for at least 8 hours (our tag data is not changing frequently at this time so cache freshnesss is not a priority)

• Register a page view (cc-archive/cccatalog-frontend#9)
• Return everything we have on this particular image ID in Elasticsearch + the view count

Once the machine generated tags have been denormalized and stored in the 'image' table, I will get started on this.

To Reproduce:

Visit
https://api.creativecommons.engineering/image/search?q=nature
and select an image license from the returned images.

Add that license type as a parameter(&li=by).

https://api.creativecommons.engineering/image/search?q=nature&li=by

Notice you don't get any results.

If the user supplies the correct authorization header for a list, allow them to submit updates to the list.

Going to replace all links with HTTPS

We routinely crawl the web and consume APIs to find CC licensed works. As a search engine, we have to deal with link rot; images are often deleted or moved from their original location. During crawling, if we detect that an image has been deleted, we will mark it with the removed_from_source field in our database.

ingestion-server, our tool for building an Elasticsearch index, will need to be updated to discard images where removed_from_source=True.

Users need to be able to update and delete lists anonymously. To accommodate this, we can return an authorization token, which can then be stored on the user's machine (either in a cookie or localstorage).

It will be sent in plain text over SSL. An attacker with local access to the user's machine could steal this token and make updates to other lists, but this isn't really a threat to be concerned about since the stakes are really low.

We had to move our development database to US-East-1 (N Virginia) in order to make use of Amazon Glue and left everything else in US-West-1 (N California).

Cross-region communication causes data transfer fees and adds latency, so the dev cccatalog API server should be moved to US East 1 as well. This will entail setting up a new VPC.

There is ongoing discussion about the best ways to store and display image and tag metrics. This issue is meant to help flush out those ideas.

@aldenstpage
@pa-w

As discussed, this is a great project to start our contanerization.