Pulsar is an un-structure focused intelligent data processing system, it extends SQL to handle the entire life cycle of data processing: collection, extraction, analysis, storage and BI, etc.

中文文档

• X-SQL: eXtended SQL to do all data jobs: collection, extraction, preparation, processing, storage, BI, etc
• Web spider: browser rendering, Ajax, scheduling, page scoring, monitoring, distributed, high performance, indexing by solr/elastic
• BI Integration: turn Web sites into tables and charts using just one simple SQL
• Big data: large scale, various storage: HBase/MongoDB

For more information check out platonic.fun

Extract data from a single page:

SELECT
    DOM_TEXT(DOM) AS TITLE,
    DOM_ABS_HREF(DOM) AS LINK
FROM
    LOAD_AND_SELECT('https://en.wikipedia.org/wiki/Topology', '.references a.external');

The SQL above downloads a Web page from wikipedia, find out the references section and extract all external reference links.

Extract data from a batch of pages, and turn them into a table:

SELECT
  DOM_BASE_URI(DOM) AS BaseUri,
  DOM_FIRST_TEXT(DOM, '.brand') AS Title,
  DOM_FIRST_TEXT(DOM, '.titlecon') AS Memo,
  DOM_FIRST_TEXT(DOM, '.pbox_price') AS Price,
  DOM_FIRST_TEXT(DOM, '#wrap_con') AS Parameters
FROM LOAD_OUT_PAGES_IGNORE_URL_QUERY('https://www.mia.com/formulas.html', '*:expr(width>=250 && width<=260 && height>=360 && height<=370 && sibling>30 ) a', 1, 20);

The SQL above visits an index page in mia.com, download detail pages and then extract data from them.

You can clone a copy of Pulsar code and run the SQLs yourself, or run them from our online demo.

Check sql-history.sql to see more example SQLs. All SQL functions can be found under ai.platon.pulsar.ql.h2.udfs.

Use the customized Metabase to write X-SQLs and turn Web sites into tables and charts immediately. Everyone in your company can ask questions and learn from WEB DATA now, for the first time.

bin/tools/install-depends.sh

You can skip this step, in such case, all data will lose after pulsar shutdown. Ubuntu/Debian:

sudo apt-get install mongodb

git clone https://github.com/platonai/pulsar.git
cd pulsar && mvn -Pthird -Pplugins

bin/pulsar

Web console http://localhost:8082 is already open in your browser now, enjoy playing with X-SQL.

Metabase is the easy, open source way for everyone in your company to ask questions and learn from data. With X-SQL support, everyone can organize knowledge not just from the company's internal data, but also from the WWW.

git clone https://github.com/platonai/metabase.git
cd metabase
bin/build && bin/start

Pulsar Enterprise Edition supports Auto Web Mining: unsupervised machine learning, no rules or training required, turn Web sites into tables automatically. Here are some examples: Auto Web Mining Examples