thymeflow / thymeflow-back Goto Github PK

View Code? Open in Web Editor NEW

7.0 8.0 1.0 973 KB

Thymeflow's back end implementation

License: GNU Affero General Public License v3.0

Scala 99.86% Java 0.13% Shell 0.02%

rdf-store personal-knowledge-base calendar email location contacts data-integration sbt back-end ontology

thymeflow-back's Introduction

thymeflow-back

thymeflow-back is the backend implementation of Thymeflow, a Web application that provides the following features:

Loads personal information into an RDF store. The following sources of personal information are currently supported:
- Email (IMAP/RFC2822)
- Calendar (CalDav/iCalendar)
- Contact (CardDav/vCard)
- Location data (Google Location History)
Provides easy to use configurators for Google/Microsoft accounts using OAuth.
Extracts Stays/Moves from Location data.
Automatically infers alignments between contact and mail data (persons), calendar and location data (event locations).
Provides a SPARQL endpoint for querying the RDF store, which features full-text search (Lucene SAIL).

For Knowledge representation, Thymeflow uses the http://schema.org ontology where possible, and its custom http://thymeflow.com/personal ontology otherwise.

thymeflow-front is the frontend implementation, which provides an interface for configuring Thymeflow, and data visualization.

thymeflow-back was principally built using the following technologies:

SBT For building the project.
RDF4J for the RDF Store.
Akka Actor System and Streams for the HTTP Server and Clients, and the scheduling of loading and enriching stages.

Configuration

Refer to core/src/main/resources/reference.conf for a list of configurable properties.
Google/Microsoft/Facebook OAuth client id/secret and Google geocoder API key should be configured.
Customize them by creating thymeflow/src/main/resources/application.conf and overriding the listed properties.

Usage

Install SBT 0.13 (http://www.scala-sbt.org/download.html).
Within the project's directory, run sbt in a console.
Within the SBT console, execute thymeflow/run [userGraphFile] where userGraphFile is an optional path to a file for storing the user local Graph.
By default, thymeflow-back will bind to localhost:8080. The SPARQL endpoint is located at /sparql.
Use thymeflow/front for the Web interface.

License/Copyright

Copyright is described in COPYRIGHT.txt.
License: AGPL v3, as described in LICENSE.txt.

thymeflow-back's People

Contributors

Stargazers

Watchers

Forkers

leiloong

thymeflow-back's Issues

Upgrade to Scala 2.12

It would improve a lot all the code that interacts with Java 8 libraries

Error when starting ES search node with a lot of initial data.

11:56:57.260 [thymeflow-akka.actor.default-dispatcher-9] INFO  c.t.t.s.e.FullTextSearchServer$ - [elastic-search] Started search node.
11:57:29.655 [thymeflow-akka.actor.default-dispatcher-9] ERROR com.thymeflow.actors.package$ - org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping [literal]) within 30s
    at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:278)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Facebook extractor breaks

Due to the regular updates in the Facebook API, the extraction of FB data breaks:

Invalid Scopes: 
publish_actions, user_about_me, user_education_history, 
user_games_activity, user_relationship_details, user_relationships, 
user_religion_politics, user_website, user_work_history, rsvp_event. 
This message is only shown to developers. Users of your app will ignore 
these permissions if present. Please read the documentation for valid permissions at:
https://developers.facebook.com/docs/facebook-login/permissions

I tried commenting out the invalid scopes in the code, however another error came up.

11:59:17.334 [thymeflow-akka.actor.default-dispatcher-4] INFO  com.thymeflow.api.MainApi$ - Facebook token received at time 20137 milliseconds
11:59:18.002 [scala-execution-context-global-49] WARN  com.thymeflow.api.MainApi$ - Facebook service: Received a non-refreshable token.
11:59:18.333 [scala-execution-context-global-49] ERROR c.t.s.f.FacebookSynchronizer$Publisher - FacebookPublisher: Error getting user data.
spray.json.DeserializationException: Object is missing required member 'updated_time'
        at spray.json.package$.deserializationError(package.scala:23)
        at spray.json.ProductFormats.fromField(ProductFormats.scala:60)
        at spray.json.ProductFormats.fromField$(ProductFormats.scala:50)
        at com.thymeflow.sync.facebook.FacebookSynchronizer$.fromField(FacebookSynchronizer.scala:31)
        at spray.json.ProductFormatsInstances$$anon$12.read(ProductFormatsInstances.scala:384)
        at spray.json.ProductFormatsInstances$$anon$12.read(ProductFormatsInstances.scala:356)
        at akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport.$anonfun$sprayJsonUnmarshaller$1(SprayJsonSupport.scala:30)
        at akka.http.scaladsl.util.FastFuture$.$anonfun$map$1(FastFuture.scala:23)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.map$extension(FastFuture.scala:23)
        at akka.http.scaladsl.unmarshalling.Unmarshaller.$anonfun$map$3(Unmarshaller.scala:26)
        at akka.http.scaladsl.unmarshalling.Unmarshaller.$anonfun$transform$3(Unmarshaller.scala:23)
        at akka.http.scaladsl.unmarshalling.Unmarshaller$$anon$1.apply(Unmarshaller.scala:58)
        at akka.http.scaladsl.unmarshalling.Unmarshal.to(Unmarshal.scala:25)
        at com.thymeflow.sync.facebook.FacebookSynchronizer$Publisher.$anonfun$queryBuilder$6(FacebookSynchronizer.scala:124)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:303)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.util.NoSuchElementException: key not found: updated_time
        at scala.collection.immutable.Map$Map4.apply(Map.scala:204)
        at spray.json.ProductFormats.fromField(ProductFormats.scala:57)
        ... 22 common frames omitted

HTTP Error 406 when requesting system-tasks from web browser

The value for access-control-allow-origin header returned for a request to http://localhost:8080/system-tasks is http://localhost:4200.
I therefore get a 406 HTTP error from a web browser.

Here are the headers received: