iboxdb / ftserver Goto Github PK

1. Install Java 8+

2. Install Maven 3+

3. Download this Project.

4. Run

$ cd FTServer
$ mvn package cargo:run

5. Open http://127.0.0.1:8088/

6. Press [Ctrl-C] to stop the container

Input a Full URL to index the Page, then search.

Move page forward by re-indexing the page.

[Word1 Word2 Word3] => text has Word1 and Word2 and Word3

["Word1 Word2 Word3"] => text has "Word1 Word2 Word3" as a whole

Search [https] or [http] => get almost all pages

Download Netbeans

iBoxDB

Semantic-UI

Jsoup

The results order based on the id() number in class PageText, descending order.

A Page has many PageTexts. if don't need multiple Texts, modify Html.getDefaultTexts(Page), returns only one PageText (the page description text only, Config.DescriptionOnly=true ).

the Page.GetRandomContent() method is used to keep the Search-Page-Content always changing, doesn't affect the real PageText order.

Use the ID number to control the order instead of loading all pages to memory.

search (... String keywords, long startId, long count)

startId => which ID(the id when you created PageText) to start, use (startId=Long.MaxValue) to read from the top, descending order

count => records to read, important parameter, the search speed depends on this parameter, not how big the data is.

set the startId as the last id from the results of search minus one

startId = search( "keywords", startId, count);
nextpage_startId = startId - 1 // this 'minus one' has done inside search()
...
//read next page
search("keywords", nextpage_startId, count)

mostly, the nextpage_startId is posted from client browser when user reached the end of webpage, and set the default nextpage_startId=Long.MaxValue, in javascript the big number have to write as String ("'" + nextpage_startId + "'")

Open

public Page Html.get(String url);

Set your private WebSite text

Page page = new Page();
page.url = url;
page.title = title;
page.text = replace(doc.body().text());
page... = ...
return page;

Setting JVM Memory from FTServer/.mvn/jvm.config , default is 4GB.

Setting Index Readonly Cache (Readonly_MaxDBCount) from FTServer/src/main/java/ftserver/Config.java .

Why does Tracker consume resources on my PC?

[user@localhost ~]$ tracker daemon -k

[user@localhost ~]$ rm -rf .cache/tracker/

[user@localhost ~]$ cat /proc/sys/fs/file-max
803882
[user@localhost ~]$ ulimit -a | grep files
open files                      (-n) 500000
[user@localhost ~]$  ulimit -Hn
500000
[user@localhost ~]$ ulimit -Sn
500000
[user@localhost ~]$ 


$ vi /etc/security/limits.conf
*         hard    nofile      500000
*         soft    nofile      500000
root      hard    nofile      500000
root      soft    nofile      500000

[user@localhost ~]$ sudo blockdev --report
//if Readahead(RA) bigger than hardware speed, can set it lower.
//it depends on hardware parameters.
[user@localhost ~]$ sudo blockdev --setra 128 /dev/sda
[user@localhost ~]$ sudo blockdev --setra 128 /dev/dm-0
[user@localhost ~]$ sudo blockdev --setra 128 /dev/dm-1
[user@localhost ~]$ lsblk -o NAME,RA

[user@localhost ~]$ free -m
[user@localhost ~]$ sudo sysctl vm.drop_caches=3

[user@localhost ~]$ firewall-cmd --add-port=8088/tcp --permanent

//Java 11 Version
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk

//Java 18 Version
export JAVA_HOME=/home/user/Downloads/jdk-18.0.1.1

//Java 21 Version
export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-21.0.2.0.13-1.el9.x86_64

$ alternatives --config java

C# ASP.NET Core Version

FTServer for Android with APK