featurebasedb / demo-taxi Goto Github PK

I'd rather put this on the pilosa repo, but I think not all the info here is appropriate for that.

@yuce's PR at #3 expands the "Intersect" tab to allow intersections of (unions within a frame). This makes the intersect queries more practically meaningful because they work on ranges of numerical values instead of single values. This is an improvement, but it's also a special-case step in the direction of a general query builder. Our next efforts in front-end work should at least be aware of that goal.

This issue is already ticketed at FeatureBaseDB/featurebase#240. That gives us a good starting point for building a general-purpose app to demonstrate the power of Pilosa. We will also want to expand on that. With a little planning, we can identify lower-level reusable components and build those as the need arises.

Ideally, we could use the same components for both the webUI and this demo app (or others - maybe a demo section on pilosa.com?). This is blocked by a few things:

• Framework: the webUI was built in startup mode, and with no dependencies. Eventually, this will likely change to use some sort of framework, build process, or other dependencies. We still need to evaluate options here. Ideally, any work we do prior to that should be adaptable to the outcome of that decision. Presumably we will prefer using the same system for both webUI and demo apps. However:
• The webUI framework decision is subject to licensing (and embedding) considerations, which won't necessarily apply to this app or other demo apps.
• The "TopN" tab presents histograms for each frame, which depends on understanding the schema of each frame. To work in the webUI, with an arbitrary index and frame, the app needs some way to retrieve more info about the schema from the server. This is not yet implemented, but there are several proposals about it: FeatureBaseDB/featurebase#316, FeatureBaseDB/featurebase#538, and an upcoming proposal from me about reorganizing and extending the content of the /schema and /status endpoints.

Short-term workaround until #4 is viable.

This is a short-term solution to the problem of having the demo app understand the meaning of the bitmapIDs in each frame. Long-term, I want Pilosa to provide some meaningful schema information that can be used by the client (#1, #4, input definition work, other proposals). Until that's available, the demo app should be able to automatically:

• populate the TopN dropdown with the frames present in the index
• populate the Intersect dropdowns with IDs and labels corresponding to what's actually in the index
• label the histogram x-axis properly

To this end, we can mirror the mapper definitions used in PDK, in the app, something like:

pickup_day: {0: Monday, 1: Tuesday, ...}
dist_miles: IntRange(0, 50)
precipitation_inches: FloatRange(0, 5, 0.02)  // (start, stop, step)

Then use TopN on each frame to discard IDs that have zero counts.

Things to check for appropriateness:

• current and historical code
• PR comments
• issues and their comments

I've looked over these things, but I'd appreciate another opinion, especially for any potential license/legal issues.

If there are any concerns, another alternative is to start a new repo from the current commit.

This depends on upcoming schema work, and will allow the app to work for arbitrary databases with minimal modification.