how to use other databases to save the history data of PVs? about epicsarchiverap-env HOT 5 CLOSED

@RuizheIsaf This repository is only for the archiver appliance configuration support. Thus, please leave your questions to EPICS tech-talk. I remembered a long time ago; someone asked the same question, and we don't have any answers on them so far. I am happy to use MariaDB.

I think, you have to implement your own codes within the archiver appliance Java application. If you are interested in delving in, please check https://github.com/slacmshankar/epicsarchiverap/tree/master/src/main/org/epics/archiverappliance/config/persistence

Thanks,
@jeonghanlee

from epicsarchiverap-env.

thanks for your reply and help @jeonghanlee:
mariadb is stable relational database, but i am not sure whether searching data will be very slow when mariadb have saved too many data...because i have really too many PVs, and there are some pv to save waveform data...so i really worry about the performance of searching history data...
so now there are three ways to choose, one is to create a new tools to manage PVs and another way is to understand the code of EPICS AA, and change the part of persistence, at last i think EPICS AA has some configures or tricks to improve the performance of searching data...i think you are expert at this domain, could you give me some suggestions or some document source in order to understand clearly EPICS AA?
thank you for your help and I look forward to your reply

from epicsarchiverap-env.

@RuizheIsaf Please run your local AA, and see what MariaDB has. They only have the latest information about each signal, not the entire archiver data.

To retrieve all archived data from the AA, MariaDB doesn't do anything (Maybe something it does). All data are the local files that you can find in STS/MTS/LTS. Reading performance depends on your disk performance, network performance, and Tomcat application performance, not on MariaDB performance.

from epicsarchiverap-env.

@jeonghanlee
really thanks for your reply and help,
i know you have rich experience in deploying epics AA, could you share some exact performance of AA? Our devices make really too much data(40Gb/s) so I want to know how many servers can meet my needs.
And whether the data will loss when a PV's data is really big and change really fast, for example some device's frequence is 1Mhz...
i think that deploying EPICS AA in muti-server can solve these questions, but i am not sure, so i wanna learn some experience from you, really look forward to your reply!
Ruizhe

from epicsarchiverap-env.

@RuizheIsaf, no, I don't have enough experience to answer your questions. From naive experience, AA isn't your tool to handle 60GB/s data. You cannot dump this kind of data to AA. Or you may do. But you have to test it.

The next question I have is how do you handle this data? How long do you want to keep it? What kind of data do you want to put them in? How do you want to use it? What is the EPICS' role in this? EPICS cannot handle this kind of speed data at all.

Especially, disk space is another point.

1 mins : 3.6 TB
1 hour: 216 TB
1 day : 5,184 TB

Do you have enough storage for this space requirement? How do you transfer this kind of data between your device and storage?

I don't have answers to these questions, and this cannot be achieved with my bits of help. You need several good engineers to design the system which handles your system requirements.
HTH,
@jeonghanlee

from epicsarchiverap-env.