fcrepo-exts / fcrepo-camel Goto Github PK

From @awoods: https://gist.github.com/awoods/643068077b4edae7708b

This is a really minor issue, but could someone remove the CodeCleanup branch? It is long out of date and really shouldn't be there any longer

eventType in the README is out of date, documenting that its values are in the http://fedora.info/definitions/v4/event namespace. Fcrepo5 uses the activitystream namespace now: https://github.com/fcrepo4/fcrepo4/blob/master/fcrepo-kernel-api/src/main/java/org/fcrepo/kernel/api/observer/EventType.java#L62

fcrepo-camel does not seem to be processing deletes properly. It appears to be trying to retrieve the F4 representation of the deleted item after it has been deleted, and then throwing an FcrepoOperationFailedException on the HTTP 410 response. To reproduce:

curl -X PUT -H "Content-Type: text/turtle" --data-binary "@object.rdf" "http://localhost:8080/fedora/rest/indexableObject"

object.rdf:

PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX indexing: <http://fedora.info/definitions/v4/indexing#>

<> indexing:hasIndexingTransformation "default"; rdf:type indexing:Indexable; dc:title "Test"

curl http://localhost:8983/solr/collection1/update?commit=true

curl -X DELETE http://localhost:8080/fedora/rest/indexableObject

Output from fcrepo-camel:

[- JmsConsumer[fedora_indexing]] DefaultErrorHandler ERROR Failed delivery for (MessageId: ID:springbok.local-55102-1429555066018-3:1:1:1:4 on ExchangeId: ID-springbok-mshome-net-54307-1429553996902-0-13). Exhausted after delivery attempt: 1 caught: org.fcrepo.camel.FcrepoOperationFailedException: HTTP operation failed invoking http://localhost:8080/fedora/rest/indexableObject with statusCode: 410 and message: Gone

Message History

RouteId ProcessorId Processor Elapsed (ms)
[fedora-to-solr ] [fedora-to-solr ] [activemq://fedora_indexing ] [ 15]
[fedora-to-solr ] [to1 ] [fcrepo:localhost:8080/fedora/rest ] [ 15]

Exchange

Exchange[
Id ID-springbok-mshome-net-54307-1429553996902-0-13
ExchangePattern InOnly
Headers {breadcrumbId=ID:springbok.local-55102-1429555066018-3:1:1:1:4, CamelRedelivered=false, CamelRedeliveryCounter=0, JMSCorrelationID=null, JMSDeliveryMode=2, JMSDestination=queue://fedora_indexing, JMSExpiration=0, JMSMessageID=ID:springbok.local-55102-1429555066018-3:1:1:1:4, JMSPriority=4, JMSRedelivered=false, JMSReplyTo=null, JMSTimestamp=1429555371813, JMSType=null, JMSXGroupID=null, JMSXUserID=null, org.fcrepo.jms.baseURL=http://localhost:8080/fedora/rest, org.fcrepo.jms.eventType=http://fedora.info/definitions/v4/repository#NODE_REMOVED, org.fcrepo.jms.identifier=/indexableObject, org.fcrepo.jms.properties=, org.fcrepo.jms.timestamp=1429555371809}
BodyType null
Body [Body is null]
]

Stacktrace

org.fcrepo.camel.FcrepoOperationFailedException: HTTP operation failed invoking http://localhost:8080/fedora/rest/indexableObject with statusCode: 410 and message: Gone
at org.fcrepo.camel.FcrepoClient.head(FcrepoClient.java:96)
at org.fcrepo.camel.FcrepoProducer.getMetadataUri(FcrepoProducer.java:149)
at org.fcrepo.camel.FcrepoProducer.doRequest(FcrepoProducer.java:134)
at org.fcrepo.camel.FcrepoProducer.process(FcrepoProducer.java:95)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:103)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:562)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:500)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:468)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1093)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:990)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[- JmsConsumer[fedora_indexing]] EndpointMessageListener WARN Execution of JMS message listener failed. Caused by: [org.apache.camel.RuntimeCamelException - org.fcrepo.camel.FcrepoOperationFailedException: HTTP operation failed invoking http://localhost:8080/fedora/rest/indexableObject with statusCode: 410 and message: Gone]
org.apache.camel.RuntimeCamelException: org.fcrepo.camel.FcrepoOperationFailedException: HTTP operation failed invoking http://localhost:8080/fedora/rest/indexableObject with statusCode: 410 and message: Gone
at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1364)
at org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback.done(EndpointMessageListener.java:186)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:107)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:562)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:500)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:468)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1093)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:990)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.fcrepo.camel.FcrepoOperationFailedException: HTTP operation failed invoking http://localhost:8080/fedora/rest/indexableObject with statusCode: 410 and message: Gone
at org.fcrepo.camel.FcrepoClient.head(FcrepoClient.java:96)
at org.fcrepo.camel.FcrepoProducer.getMetadataUri(FcrepoProducer.java:149)
at org.fcrepo.camel.FcrepoProducer.doRequest(FcrepoProducer.java:134)
at org.fcrepo.camel.FcrepoProducer.process(FcrepoProducer.java:95)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:103)
... 11 more

Hi fcrepo-camel team, i have following issue:
If try to move json-ld file bigger than 10-20 Mb, i get the following view:

INFO | Apache Camel 2.19.2 (CamelContext: camel-1) is shutting down
INFO | Starting to graceful shutdown 1 routes (timeout 120000 seconds)
INFO | Waiting as there are still 2 inflight and pending exchanges to complete, timeout in 120000 seconds. Inflights per route: [route1 = 2]
INFO | Waiting as there are still 2 inflight and pending exchanges to complete, timeout in 119999 seconds. Inflights per route: [route1 = 2] .....

from("file:data")
.setHeader(Exchange.HTTP_URI, constant(FedoraHeaderConstants.CONTENT_TYPE))
.setHeader(Exchange.HTTP_METHOD, constant("POST"))
.setHeader(Exchange.CONTENT_TYPE, constant("application/ld+json"))
.to("fcrepo:10.46.3.100:8080/fcrepo/rest/");

May be i need to add some other parameters?
Is it possible to send big files or i should use activemq?

Dear colleges i try to change .well-known/genid order, where the whole data saved, to self-defined order
structure.
How i can define it?
How is the best way to define hasParent or hasChild for Book with Pages and Chapters in Json-ld?

May be you know where to find it?

Best regards
Andrey

I try to index data in fedora 4 from json file, using following code:

                 `from("file:data2")
                .unmarshal(gsonDataFormat)
                .setBody().simple("${body.products}")
                .split().body()                                     
                .to("fcrepo:10.46.3.100:8080/fcrepo-webapp-4.7.4/rest");`

Simular code works if i try to index to Solr, but with Fedora 4.
Do you know should i add something else?