Adding helpers to the Publication about architecture HOT 8 CLOSED

IMO this goes beyond FXL, we also want to have helpers that can return the following info:

• are all resources in the readingOrder images/audio/video resources?
• are all resources in the readingOrder HTML or XHTML?
• can I extract images for all resources in readingOrder from a full FXL publication?
• is it a multi-lingual publication where all or most resources have an equivalent in another language and if so, which languages are available?
• which formats/bitrates/resolutions are available for an image-based/audio/video publication?

All these helpers could be part of the Publication object that's being proposed in #122.

Some of them simply require inspecting the manifest while others will require inspecting resources (which means fetching them first if its a Web Publication).

These helpers are also necessary to move away from the rather naive approach of relying on types (see #112).

from architecture.

You're right. The positions and the search feature could also fall into this case. Basically what is described as "Services" in the architecture schema.

can I extract images for all resources in readingOrder from a full FXL publication?

It could even go further: producing a new readingOrder pointing to each image.

This should definitely be extensible to meet custom needs.

Since these services might be CPU-consuming, they should be able to cache the produced values, which would be invalidated if the publication changes (e.g. DRM unlocked).

Here's a quick draft of a proposal.

PublicationService interface

Should be implemented by a Publication service/helper to represent its algorithm and hold its state (e.g. cache).

• onAttached(publication)
  • Called when the service is added to a publication, to get a reference to it.
  • The service can store the reference for further use, but as a weak link.
• invalidate()
  • Called when the publication (manifest, fetcher) changed, to invalidate any cache.
  • For example, called when the DRM is unlocked.

Then the rest of the API is free to be declared depending on the need of the service.

FetchedPublicationService interface

Specialization of the PublicationService for the streamer. Basically, it allows to expose a service as a JSON API in a Fetcher, e.g. through an HTTP server. This can be useful for a full JS app using the native streamer.

• val links: List<Link>
  • Routes to be added to the Fetcher to expose the JSON version of the API.
  • We could use a special scheme such as service:// for these links.
  • They would be added to the Manifest.links list and be discoverable using rel.
  • Templated URI could be used to send parameters to the service.
  • e.g. Link(href: "service://search{?query}", type: "application/json", rel: "search")
• fetch(action: String, parameters: Map<String, String>): HTTPResponse?
  • Called by the Fetcher when a href is routed towards this service.
  • The action (e.g. search) and query parameters are parsed for convenience.
  • It can return an HTTP response.

Publication

• services: Map<String, PublicationService>
  • Map where the services are stored.
• attach(key: String, service: PublicationService)
  • Adds or replaces the service at the given key.
  • service.onAttached(publication) will be called.

Depending on the language, we can declare type extensions to expose the services through a native-like API. Like the extensibility in the Publication models.

e.g.

extension Publication {

    func search(query: String) -> SearchResult? {
        let service = services["search"] as? SearchPublicationService
        return service?.search(query)
    }

}

Here's an example for the positions service of CBZ.

protocol PositionsService {
    let positions: [Locator]
}

extension Publication {

    var positions: [Locator] {
        (services["positions"] as? PositionsService)?.positions
    }

}

class CBZPositionsService: PositionsService, FetchedPublicationService {

    weak var publication: Publication?

    lazy var positions: [Locator] = 
        publication.manifest.readingOrder.map { Locator(...) }

    // Publication Service

    func onAttached(publication: Publication) {
        self.publication = publication
    }

    func invalidate() {
        self.positions = nil
    }

    // FetchedPublicationService

    val links = [
        Link(href="service://positions", rel="position", type="application/json")
    ]

    fun fetch(action: String, parameters: [String: String]) {
        if (action != "positions") return
        return HTTPResponse(200, positions.toJSON())
    }

}

publication.attach("positions", CBZPositionService())

And if a reading app wants to implement a DB cache for the positions, it can simply wrap and replace the PositionsService.

class CachedPositionsService: PositionsService, PublicationService {

    let service: PositionsService
    let repository: PositionsRepository

    // Very naive, it should cache the positions if it didn't have them.
    lazy var positions = repository.positions ?? service.positions

    ...
}

// Then we replace the service in the Publication
if let service = publication.services["positions"] as? PositionsService {
    publication.attach("positions", CachedPositionService(service, repository))
}

As you can see, it means that the reading app can replace a whole portion of algorithm and still be compatible with the rest of the ecosystem, if it implements the proper interfaces (e.g. PositionsService). For example, a reading app might want to implement a more efficient or precise search algorithm, or implement the search for a custom format without changing anything in the rest of the application.

from architecture.

Maybe there's a way to make the storage of the services type safe, and more convenient for the services provided by Readium.

enum PublicationServiceKey {
    case positions(PositionsService)
    case search(SearchService)
    case other(key: String, PublicationService)
}

publication.attach(.positions(CBZPositionService())

from architecture.

After discussing with @mickael-menu we'd like to make a distinction between:

• helpers which would be part of the shared models and would only access the manifest
• services which would be part of other components of our architecture and might require access to the resources of the publication

In Readium Web, such services would usually be available through HTTPS and the Publication Server.

Examples of helpers in our current codebase

• layoutOf which returns the EPUB layout of a resource based on both publication and resource-level properties
• expectedReadingProgression which calculates and returns a readingProgression when the value is set to auto

Examples of services in our current codebase

• positions which generates and returns a list of locators for the entire publication
• Synchronized Narration which converts SMIL resources to the W3C Synchronized Narration document

Future potential services

• Search
• Index/Glossary
• Thumbnail view (generate thumbnails for each resource/page in an EPUB FXL or DiViNa)

from architecture.

We can't add a lazy property in an extension with Swift and Kotlin, which might be a problem for helpers doing heavy processing.

Here's how we could implement a workaround by storing lazy values in a map.
An extension function would wrap its heavy processing with lazy { ... }

I used #function as the map key to automatically use the name of the extension function.

class Publication {
    
    private var values: [String: Any?] = [:]
    
    init() {}
    
    func lazy<T>(_ factory: () -> T) -> T {
        if let value = values[#function] as? T {
            return value
        } else {
            let value = factory()
            values[#function] = value
            return value
        }
    }
    
}

extension Publication {
    
    var isFXL: Bool {
        lazy {
            print("Calculating isFXL...")
            return true
        }
    }
    
}

let pub = Publication()
print(pub.isFXL)
print(pub.isFXL)

Output:

Calculating isFXL...
true
true

from architecture.

@mickael-menu That's a pretty cool workaround.

from architecture.

@jccr Thanks :)

from architecture.

An updated version is introduced in this proposal. I'm closing this issue.

from architecture.