Giter Club home page Giter Club logo

sabi's Introduction

Welcome to Open Science at the SABI-Project - Seawater Aquarium Business Intelligence

This is a semi-scientific (or open-science) project that aims to gain insights from aquarium hobbyist for aquarium hobbyist according seawater ecosystems.

Vision I

In seawater forums, wikis, books we got advice on the regular values (max,min) of the important mineral levels and so on. Some things we fully understand, while on others we have just a lot of guesses about the impact, but often it stays a guess, as the complete system is very complex. I was wondering if we will be able to gain some more insights if we start to share our measurement data and placing some business intelligence style reporting on top of it. This should enable us to answer some questions like:

  • How often do all measure the CaCO3 level when not using the Balling method? Is there a probability of getting a cyano bacteria plague when measuring too less?
  • Is there a thing in common when Alveoproa dies (are there similar PO4 levels)?

There must be quite a lot of interesting questions, especially in the field of aquaristic forensics.

So this is the project to build a platform, which helps to answer them.

Vision 2.0 (in the next 10 years)

In the first stage of the project (see Vision I) we enabled the community to collect and share data. We then added BI concepts on top for being able to explore the data, mining for new insights driven by human curiosity.

The next step adds AI concepts to sabi. Imagine the following scenario: You have a cyano bacterium plague in your tank and you are planning to add a new fish or fiddling around with increasing the carbonate level. Sabis KI might advise you that your plan will probably prolong the plague.

For being able to do so, sabis AI will compare your tank parameters and recent history with the data of other users tanks who have done something similar to anticipate the outcome.

The AI challenge here is, that if the human provided data on a given problem context is bad for some reasons (e.g. missing of relevant parameters, inaccurate timelines and so on), then the AI starts to provide advises that will lead to false treatments. So if users starts to reports that advices were not successful the AI must revise the original training set and do some recalibration by its own.

Climate-Friendly (#greenwashing?)

I declare the Sabi Project climate-friendly because of:

  • You as a user, who is not demanding, that the Sabi Service is up and running 24/7 with 99% availability, accepting minor service outages, thus enabling the low-energy platform for the project.
  • The decision to use raspberry pis as operation platform, which have a very low energy profile (Have a look at my pis here: https://github.com/StefanSchubert/sabi/wiki/07.-Deployment-View ) The alternative would be running in a public cloud, which would allow 99% availability but surly a much bigger CO2 footprint (as well as bigger costs).
  • The private cloud at my homesite is powered by a green electricity tariff of my power supply provider. I in future I will generate my own electricity through solarcells on the roof.

Project Planning

Stage I

Being able to collect the basic values and to display them in a rather static reporting manner. (Reached)

Next: Transistion to Stage II. Precondition: at least 50 registered users.

Stage II

Offering some query mechanism to do some analysis. And maybe a set of some standard reports. If possible we might acquire support from one of the big BI vendors.

Stage III

Document insights gained through this project. If possible try to make forecasts (i.e. take care of that measurement level, if not raised it is likely that ... happen)

Possible NON-Scientific extensions

Solution for:

  • I have some siblings in my nano reef tank and need to do some gardening. But where to with the siblings? Where are all the other aquarists and is there someone nearby? They are organized in standard internet forums, but what if there are someone near but not located in the same forum I use (more or less frequently). If they all could be motivated using sabi it should be possible to introduce them to each other for nearby support purposes.

Release Planning

Just have a look at the Milestones from the Issue Board

Project history

Date News
31th Dez 2023 Introduced Support for fresh-water tanks.
05th Nov 2023 Service Release (Patchmanagement - e.g. Java 21)
09th Apr 2023 Hotfix Release (i18n issues)
18th Dez 2022 Feature Release (Reminder Service) / Technical Migration to Spring-Boot 3 / Marks Sabi-Version 1.2.0
7th Okt 2022 Released Plague-Center. All required base workflows have been implemented. This marks version 1.0.0 of sabi.
22nd Jul 2022 Article about SABI has been published in the journal "KORALLE", Issue Nr. 136. ๐Ÿ˜Š Available on start of August. Many Thanks goes to Daniel Knop!
26th Feb 2022 Official Softlaunch Day of the project. Starting with a twitter announcement. Going to spread the word in selected forums in the next days, hoping to get some Beta-Testers, Fellow-Coders and to collect valuable feedback.
27th Mai 2021 I managed to replace the self-signed TLS cert with a let's encrypt based one. This gets us rid of the browsers insecure warning.
16th Mai 2021 Sneak Preview available on https://sabi-project.net (Notice: Only available if you have an IPV6 Internet-Connection (your mobile with wlan switched off should do it, if your home has only the half internet available). As of the zero budget start we have a self signed TLS resulting in a browsers insecure warning. Still some bugs of course (see issue list), and features left till release 1.0 but it's already usable. In case you decide to create an account...it's already the production environment ;-) i.e. your data will be kept.

For Developers

Please have a look at:

sabi's People

Contributors

dependabot[bot] avatar kirillsinyuk avatar stefanschubert avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

sabi's Issues

Rest API for tanks

(pre condition - valid token in Request header)

@path("/tank/list")
@get
@produces("application/json")
JsonObject listTanks()
result: OK
tanks : [
{
"id":"1",
"name":"MyMiniReef"
},
{
"id":"2",
"name":"MyBigReef"
}
]

 result: ERROR
 message: Error description

@path("/tank/create")
@post
@consumes("application/json")
@produces("application/json")
JsonObject createTank(JsonObject psJson) # tank: name

  result: OK
  tankId: 2

  result: ERROR
  message: Error description

Obfuscate ResourceIDs

Providing internal objects database IDs as resource IDs are a potential security risk.
To minimize this risk all ResourceIDs that will be published to the clients needs to be obfuscated, such that a client won't be able to access a different object just by incrementing the ID. In addition any invalid ID provided by the client should be logged in a special fraud-detection log with the clients IP.

Improve register buttons state

The register process takes some secs because of the included email (which takes a bit on the pi because of the TLS handshake).

The user currently hardly see that the button has been pressed and tends to press it again.
This should be somehow improved.

Enable github action to run server modules MasterTestSuite

This seems to be a bit tricky, as because of relying on eclipselink we need to provide a link into mavens build repository, see servers pom.xml

                   <jvmArguments>
                        -javaagent:${settings.localRepository}/org/springframework/spring-instrument/${spring.instrument.version}/spring-instrument-${spring.instrument.version}.jar
                    </jvmArguments>

for the git runner we may use ~/.m2/ but then again it wouldn't match my local repository setup.
Hm....working with maven profile here?

Check and enhance unauth login case.

See testInvalidatedUserCanNotSignIn()

Needs to be investigated. currently results in an HttpRetryException, instead of retrning a
proper status-code. Needs to be adopted in such way, that the user get's a message about the incomplete registration process instead.

Reproduce with a registered user, who has not been validated yet.

Allow username as login

Currently the email address is being as login.
Shall we keep the email address as login and only identifier, or shall we use the username as possible login, too?

Implementing a native mobile client (iOS)

This is a rather an edge case, but for educational reasons I require to do a first iOS App to get a feeling about development costs here which might help me on business cases in my other life...

In addition the tasks for a major WebApp which will be most commonly used is currently taken by @Alexander-voss

Enabling Captcha for registration

To avoid DoS by spilling the database with new users the register process required a captcha token.
In addition we should limit the amount of new users per minute and provide an internal alert if the threshold is being exceeded.

Unauthorized Test with rest template shows strange behavior

Unexpected result by the test below is:

org.springframework.http.InvalidMediaTypeException: Invalid mime type "text;charset=ISO-8859-1": does not contain '/'

@Test
/**
 * Test to check that our WebSecurityConfig is effective.
 */
public void testUnauthorizedListUsersTankRequest() throws Exception {

    // Given User presentation by a faked auth token
    String authToken = "faked";

    // when this authorized user requests his aquarium list
    HttpHeaders headers = new HttpHeaders();
   // headers.setContentType(MediaType.APPLICATION_JSON);
    headers.add("Authorization", "Bearer " + authToken);

    HttpEntity<String> requestEntity = new HttpEntity<>(headers);
    ResponseEntity<String> responseEntity = restTemplate.exchange("/api/tank/list" , HttpMethod.GET, requestEntity, String.class);

    // then we should get a 403 as result.
    assertThat(responseEntity.getStatusCode(), equalTo(HttpStatus.FORBIDDEN));

}

Strange is that this testcase was derived by this one which is running with no complains:

@Test
public void testListUsersTank() throws Exception {
    // given some Testdata via mocking

    UserTo userTo = new UserTo();
    userTo.setEmail(MOCKED_USER);
    userTo.setId(1L);
    given(this.userDao.loadUserByEmail(MOCKED_USER)).willReturn(userTo);

    List<AquariumTo> testAquariums = new ArrayList<>(1);
    AquariumTo aquariumTo = getTestAquariumFor(userTo);
    testAquariums.add(aquariumTo);

    given(this.aquariumDao.findUsersTanks(userTo.getId())).willReturn(testAquariums);

    // and we need a valid authentication token for oure mocked user
    String authToken = TokenAuthenticationService.createAuthorizationTokenFor(MOCKED_USER);

    // when this authorized user requests his aquarium list
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_JSON);
    headers.add("Authorization", "Bearer " + authToken);

    HttpEntity<String> requestEntity = new HttpEntity<>(headers);

    // Notice the that the controller defines a list, the resttemplate will get it as array.
    ResponseEntity<String> responseEntity = restTemplate.exchange("/api/tank/list" , HttpMethod.GET, requestEntity, String.class);

    // then we should get a 202 as result.
    assertThat(responseEntity.getStatusCode(), equalTo(HttpStatus.ACCEPTED));

    // and our test aquarium
    AquariumTo[] myObjects = objectMapper.readValue(responseEntity.getBody(), AquariumTo[].class);
    assertThat(Arrays.asList(myObjects), hasItem(aquariumTo));

}

Average coral lifelines

e.g. Alveopora are told to hold often no longer than half a year. It would be interessting to see a difference in average water parameters between those with a short live and those who last longer...

LiveGo Preparations

Those tasks needs to be finished, before going live:

Configuration

  • Application Properties: change Development to Production
  • Add Nginx for Port 80 before the SpringBootApp
  • Use Let's encrypt
  • Automatically save production configs to local NAS

Establish Backup & Recovery Plan

  • Setting up Database replication
  • Recovery by Ansible deployment and documented Pi setup plus DB from replica

Operational

  • Wrote a small "Don't panic" operational guide.
  • Some Monitoring and Alerting?

Final QA

  • Write manual regression testplan for further releases
  • Conduct first regression QA according testplan

Adding Spanish resource bundles

I love Spain. Beneath my mother tongue and english for i18n capabilities, we shouldn't miss Spain here.
So the task is translating the existing resource bundle to provide a spanish version of sabi.

WEB GUI for BasicReport

BackenAPI should be already sufficient for it.

  • Just a nice chart page, showing users own measurements.
  • Capability to downloads own measures as CSV-File as Backup for the user or further processing

Remedy success

There are certain remedies against cyano bacteria. However they do not lead to success in all cases. It would be good to see if the remedy success rate depends on the type of cyano bacteria (thesis).

Example remedy:
Microbe special blend in combination with nite out II

Add open stats to a prometheus registry

Some kind of business monitoring. Would be nice for being able to track when user counts rises, to learn which promotions are successful and which are not.

(I already have prometheus running).

Wrong error code on register with duplicated username

If the username already exists, a 409 is expected as return code according to APIdoc.
Instead we get an HTTP 500.

sabi.log shows that the situation has been recognized:
java.sql.SQLException: Duplicate entry 'mezzoMix' for key 'UQ_USERNAME'

MOTD functionallity for sabi backend

For being able to notify the users on scheduled updates etc. we require to have a modt like info service on the backend.

rest api, locale based
messages are created manually in the database. They will have a valid date.

i18n of fish-catalogue description

Currently we have scientific name and a description as well as an uri to further reference, whereas the further reference may be a wiki which is again in a specific language.

For being able to really use the fish catalogue we nee to offer
i18n to (description, url)

Offering a measurement reminder via email

Is your feature request related to a problem? Please describe.
Often I'm just too busy and forgot to do measurements on a regular basis.

Describe the solution you'd like
It would be nice to have this possibility through sabi, e.g. located in the user profile.
Triggering a email reminder every week or x days.

Describe alternatives you've considered
This is just a calendar trigger. Users could choose their favorite calender solution.

Additional context
An email trigger could also be used to inform about maintenance windows, or even to place the Add of a project sponsor.

MVP: Rework JSF/Primefaces layout stylesheet to become more responsive.

Despite the fact of setting on a very mature frontend technology (jsf/primefaces) here, I'm trying to set on a mobile first approach. However some users my use their laptop.

Currently I'm focus on the functionality for the MVP.

For this task it would be just great, to have some help by a senior jsf/primefaces frontend engineer(in) to give it the right responsive shape.

The current stylesheet I have in place ist just a dummy. The primary goal is to have it look OK on a PCs browser, and the secondary it should be readable without the need to zoom in on a mobiles device screen.

I habe to admit, that this is no mobile first approach, you may persuade me to do it the other way round. But for now I'm playing with the idea to go native with the mobile option later on.

Improvement or registration API

Currently we have one UserTo for all: register and login - with optional values.
This is confusing for API developer. The task is to provide specialized TOs for the API usage. Internal it can still be mapped to a common user representation.

Process of building the fish-catalogue

There can't be an import through one of the wiki resources to avoid any copyright issues.
Also sabi aims not be be just another wiki. The fishes are far more better described in the many official wikis that are maintained by the different aquarists groups.

However, as the wikis are often "implement" a closed character, I will enable the users to describe the individual behavior of their fishes within sabi, which in turn will be subjected to the BI process in future. For being able to compare data about fishes we need a unique reference of them. This is the purpose of the fish catalogue.

How to build the catalogue? The idea is user generated content. Each tank owner knows his fishes, will be able to link them to the existing ones of the catalogue and make a proposal for new entries.
A new proposal can be instantly used by the user who makes it. However for public uses the proposal needs to be approved by a user with admin role (Responsibility of that user is: to avoid duplicates or any other legal issue).

A challenge will be to allow users to maintain the i18n aspects of the fish catalogue (description, url).

Handle javax.faces.application.ViewExpiredException

Describe the bug
Subsequent request after session timeout is not handled correctly.

To Reproduce
Steps to reproduce the behavior:

  1. Login
  2. Go to a view, wait 30+ min (or reduce the session value for testing)
  3. Navigate to another view
  4. See error, e.g. view not reacting

Expected behavior
Navigation to a view explaning the problem to the user and offer a link to the login page.

Implement a password strength check policy

See UserServiceImpl.java:

    // todo integrate pw-policy and throw an Password_Too_Weak

-> wiki docu and swagger API needs to be updates on that as well
-> clients needs to be adapted as well if there are any yet.

WEB GUI for UsersProfile

BackendAPI should be sufficient for it.

Just an overview of users profile, with the ability to change settings like language, country, password.
Not more for the beginning. Will be extended with future releases.

i18n of jsf GUI

  • Do some developer documentation on the used i18n design-concept on the wiki
  • Implement the concept (as example) for the login/registration dialog ( #12)

Rest API for login/register

@Path("/login")
@GET
@Produces("application/json")
 JsonObject login(String userName, String password)
     result: OK
     token: xyz4711

     result: ERROR
     message: Error description

@Path("/register")
@POST
@Produces("application/json")
 JsonObject login(String userName, String EmailAdress, String password, String CaptchaCode)

      result: OK
      token: xyz4711

      result: ERROR
      message: Error description

Secure Actuator endpoints

They shouldn't be open to the world, as they may reveal to much about the environment to a possible attacker.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.