Giter Club home page Giter Club logo

tscat's Introduction

sciqlop_logo


What Is SciQLop?

SciQLop (SCIentific Qt application for Learning from Observations of Plasmas) is a powerful and user-friendly tool designed for the visualization and analysis of in-situ space plasma data.

Using SciQLop will let you:

  • have a super easy access to tens of thousands of products from the top main data archives in the world,
  • explore multivariate time series effortlessly, with lightning-fast and transparent downloads as you scroll, zoom in, and zoom out,should
  • visualize custom products with simple python code executed on-the-fly,
  • easily label time intervals and make or edit catalog of events graphically and rapidely,
  • analyze your data in jupyter notebooks,
sciqlop_logo

Heliophysicists now benefit from decades of space exploration through many spacecraft missions. Exploring this massive amount of data to find events of interest, build catalogs, and conduct statistical multi-mission data analysis can be a daunting task if not having the right tool.

SciQLop aims at being this tool! A simple lightweight yet powerful graphical interface coupled to the limitless options brought by the Jupyter Notebook integration, that focuses on providing users with the easiest possible way to explore, label and analyze huge amounts of data. SciQLop is also the right tool for teaching space physics and in situ spacecraft data handling to students effortlessly.

Main Features

  • Interactive and responsive: SciQLop can handle millions of data points without compromising on interactivity. Users can scroll, zoom, move, and export plots with ease.

    SciQLop smooth navigation
  • User-friendly: Accessing data in SciQLop is as simple as a drag and drop from the tens of thousands of products readily available. Custom user products defined in Python behave exactly the same way and bring infinite possibilities.

    SciQLop drag and drop
  • Jupyter notebook integration: SciQLop can be used as a backend for Jupyter notebooks, allowing users to create and manipulate plots from within their notebooks, define new products and much more.

    SciQLop Jupyter integration
  • Catalogs: SciQLop provides a catalog system that allows users to easily label events in their data or visualize existing catalogs of events.

    SciQLop catalogs
  • Evolving and growing list of examples: SciQLop comes with a growing list of examples that demonstrate how to perform common tasks such as loading data, creating plots, and using the catalog system.

    SciQLop examples

Upcoming features

  • community-driven plugins repository: SciQLop will soon have a plugin system that will allow users to extend the software's capabilities by installing community-driven plugins.
  • catalogs coediting: SciQLop will allow users to coedit catalogs, making it easier to collaborate on event labeling and visualization, thereby also improving reproducibility of space physics studies.

How to install SciQLop

Mac Users

Since SciQLop 0.7.1 we produce a Mac App Bundle that you can download from the latest release page just pick the right architecture for your Mac (ARM64 for Apple M1/2/3 chips and x86_64 for intel ones).

Linux Users

If you are using a Linux distribution, you may not need to install anything, you can just download the AppImage from the latest release and run it (after making it executable).

From sources

Since SciQLop depends on specific versions of PySide6 you might prefer to use a dedicated virtualenv for SciQLop to avoid any conflict with any other Python package already installed in your system.

  • Using releases from PyPi
python -m pip install sciqlop
  • Using the latest code from GitHub
python -m pip install git+https://github.com/SciQLop/SciQLop

Once installed the sciqlop launcher should be in your PATH and you should be able to start SciQLop from your terminal.

sciqlop

or

python -m SciQLop.app

Python user API Examples:

SciQLop has a public API that allows users to create custom products and plots. Here are some examples:

  • Creating plot panels:
from SciQLop.user_api import TimeRange
from SciQLop.user_api.plot import create_plot_panel
from datetime import datetime

# all plots are stacked
p = create_plot_panel()
p.time_range = TimeRange(datetime(2015, 10, 22, 6, 4, 30), datetime(2015, 10, 22, 6, 6, 0))
p.plot("speasy/cda/MMS/MMS1/FGM/MMS1_FGM_BRST_L2/mms1_fgm_b_gsm_brst_l2")
p.plot("speasy/cda/MMS/MMS1/DIS/MMS1_FPI_BRST_L2_DIS_MOMS/mms1_dis_bulkv_gse_brst")
p.plot("speasy/cda/MMS/MMS1/DIS/MMS1_FPI_BRST_L2_DIS_MOMS/mms1_dis_energyspectr_omni_brst")

# tha_peif_sc_pot and tha_peif_en_eflux will share the same plot 
p2 = create_plot_panel()
p2.plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_en_eflux")
p2.plots[0].plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_sc_pot")
p2.plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_velocity_dsl")

NOTE: An easy way to get product paths, is to drag a product from Products Tree to any text zone or even your Python terminal.

  • Custom products AKA virtual products:
import numpy as np
from SciQLop.user_api.virtual_products import create_virtual_product, VirtualProductType
from SciQLop.user_api.plot import create_plot_panel
import speasy as spz


# define a custom product that calculates the magnitude of the magnetic field measured by ACE

def ace_b_magnitude(start: float, stop: float) -> spz.SpeasyVariable:
  b_gse = spz.get_data(spz.inventories.tree.amda.Parameters.ACE.MFI.ace_imf_all.imf, start, stop)
  return np.sqrt(b_gse["bx"] ** 2 + b_gse["by"] ** 2 + b_gse["bz"] ** 2)


ace_b_magnitude_virtual_prod = create_virtual_product(path='my_virtual_products/ace_b_magnitude',
                                                      product_type=VirtualProductType.Scalar, callback=ace_b_magnitude,
                                                      labels=["|b|"])

# plot the virtual product
p = create_plot_panel()
p.plot(ace_b_magnitude_virtual_prod)

More examples can be found in the examples folder, they are also available from the welcome screen.

How to contribute

Just fork the repository, make your changes and submit a pull request. We will be happy to review and merge your changes. Reports of bugs and feature requests are also welcome. Do not forget to star the project if you like it!

Credits

The development of SciQLop is supported by the CDPP.
We acknowledge support from the federation Plas@Par

Thanks

We would like to thank the developers of the following libraries that SciQLop depends on:

  • PySide6 for the GUI framework and Qt bindings.
  • QCustomPlot for providing the plotting library.
  • DiskCache for providing a simple and efficient caching system used in Speasy.
  • The Jupyter project for providing the Jupyter notebook integration.
  • Numpy for providing fast Python array library.

tscat's People

Contributors

jeandet avatar lgtm-migrator avatar pboettch avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

tscat's Issues

Feature: get tag list

We need a method to get a list of all existing tags (catalogues and/or events) from the library.

Config file to make Author field optional in API

If we add a config file, we could make Author field optional in create event/catalog functions. This makes sense since most use cases involves an user creating events or catalogs from his own machine.

Feature: history module

Description

To implement versioning of the whole database (to allow a diff-view or quick undo/redo-mechanism) a history (or snapshot) module (tscat.history) could be implemented which allows the user to create arbitrary but complete snapshots of the whole local database. This module should allow listing and navigating the snapshots. Maybe also allowing to show the difference between 2 snapshots.

Overring __getattr__() seems not be the best idea to control access to attributes

We use getattr() to check whether the object (Catalogue or Event) is still viable. Otherwise we raise when accessing an attribute.

If the access is OK the base-class's getattr() is called. Except that the default base class might not have a getattr()-method.

According to pythons docs, it is better to use __getattribute__() method for a better access control: https://docs.python.org/3/reference/datamodel.html#object.__getattribute__

Catalogue - import

  • JSON-format for a starter
  • event or catalogue with an identical UUID, but other fields and attributes are different:
    • overwrite=False -> Raise
    • overwrite=True -> OK

Should be able to create events with start_time=stop_time

Creating events with start_time=stop_time rises the following errror:

    167 elif key == 'stop' and hasattr(self, 'start'):
    168     if value <= self.start:
--> 169         raise ValueError("stop date has to be after start date")
    170 elif key in ['tags', 'products']:
    171     if any(type(v) != str for v in value):

ValueError: stop date has to be after start date

Filter events on catalogue presence

  • predicate to be added
    • is in Catalogue(A) or/and in Catalogue(B) (not for SmartCatalogue in the first version)
    • is in no Catalogue (orphan)
    • is in Trash
  • filtering on SmartCatalogues is not possible)

bug when creating events and catalog ?

SciQLop IPython Console Python 3.11.0 (main, Oct 24 2022, 00:00:00)
[GCC 12.2.1 20220819 (Red Hat 12.2.1-2)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.9.0 -- An enhanced Interactive Python. Type '?' for help.

import pandas as pd

df =
pd.read_csv('/home/michotte/Downloads/start_stop_multi_mp_xing.csv')

import tscat

with tscat.Session() as s:
   events = [s.create_event(start, stop, "bayane") for start, stop in
zip(df.start, df.stop)]

-----------------------------------------------------------------------
----
TypeError                                 Traceback (most recent call
last)
File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/base.py:1800, in
Connection._execute_context(self, dialect, constructor, statement,
parameters, execution_options, *args, **kw)
  1798         conn = self._revalidate_connection()
-> 1800     context = constructor(
  1801         dialect, self, conn, execution_options, *args, **kw
  1802     )
  1803 except (exc.PendingRollbackError, exc.ResourceClosedError):

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/default.py:1077, in
DefaultExecutionContext._init_compiled(cls, dialect, connection,
dbapi_connection, execution_options, compiled, parameters,
invoked_statement, extracted_parameters, cache_hit)
  1076 for compiled_params in self.compiled_parameters:
-> 1077     param = [
  1078         processors[key](compiled_params[key])
  1079         if key in processors
  1080         else compiled_params[key]
  1081         for key in positiontup
  1082     ]
  1083     parameters.append(dialect.execute_sequence_format(param))

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/default.py:1078, in <listcomp>(.0)
  1076 for compiled_params in self.compiled_parameters:
  1077     param = [
-> 1078         processors[key](compiled_params[key])
  1079         if key in processors
  1080         else compiled_params[key]
  1081         for key in positiontup
  1082     ]
  1083     parameters.append(dialect.execute_sequence_format(param))

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/dialects/sqlite/base.py:1004, in
DATETIME.bind_processor.<locals>.process(value)
  1003 else:
-> 1004     raise TypeError(
  1005         "SQLite DateTime type only accepts Python "
  1006         "datetime and date objects as input."
  1007     )

TypeError: SQLite DateTime type only accepts Python datetime and date
objects as input.

The above exception was the direct cause of the following exception:

StatementError                            Traceback (most recent call
last)
Cell In[4], line 1
----> 1 with tscat.Session() as s:
     2     events = [s.create_event(start, stop, "bayane") for start,
stop in zip(df.start, df.stop)]

File ~/sciqlop_env/lib64/python3.11/site-packages/tscat/__init__.py:55,
in Session.__exit__(self, exc_type, exc_value, exc_tb)
    54 def __exit__(self, exc_type, exc_value, exc_tb):
---> 55     backend().add_and_flush(self.entities)

File ~/sciqlop_env/lib64/python3.11/site-
packages/tscat/orm_sqlalchemy/__init__.py:274, in
Backend.add_and_flush(self, entity_list)
   272 def add_and_flush(self, entity_list: List[Union[orm.Event,
orm.Catalogue]]):
   273     self.session.add_all(entity_list)
--> 274     self.session.flush()

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/orm/session.py:3444, in Session.flush(self,
objects)
  3442 try:
  3443     self._flushing = True
-> 3444     self._flush(objects)
  3445 finally:
  3446     self._flushing = False

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/orm/session.py:3583, in Session._flush(self,
objects)
  3580     transaction.commit()
  3582 except:
-> 3583     with util.safe_reraise():
  3584         transaction.rollback(_capture_exception=True)

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/util/langhelpers.py:70, in
safe_reraise.__exit__(self, type_, value, traceback)
    68     self._exc_info = None  # remove potential circular
references
    69     if not self.warn_only:
---> 70         compat.raise_(
    71             exc_value,
    72             with_traceback=exc_tb,
    73         )
    74 else:
    75     if not compat.py3k and self._exc_info and
self._exc_info[1]:
    76         # emulate Py3K's behavior of telling us when an
exception
    77         # occurs in an exception handler.

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/util/compat.py:211, in raise_(***failed resolving
arguments***)
   208     exception.__cause__ = replace_context
   210 try:
--> 211     raise exception
   212 finally:
   213     # credit to
   214     #
https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/
   215     # as the __traceback__ object creates a cycle
   216     del exception, replace_context, from_, with_traceback

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/orm/session.py:3544, in Session._flush(self,
objects)
  3542 self._warn_on_events = True
  3543 try:
-> 3544     flush_context.execute()
  3545 finally:
  3546     self._warn_on_events = False

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/orm/unitofwork.py:456, in
UOWTransaction.execute(self)
   454 else:
   455     for rec in topological.sort(self.dependencies,
postsort_actions):
--> 456         rec.execute(self)

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/orm/unitofwork.py:630, in
SaveUpdateAll.execute(self, uow)
   628 @util.preload_module("sqlalchemy.orm.persistence")
   629 def execute(self, uow):
--> 630     util.preloaded.orm_persistence.save_obj(
   631         self.mapper,
   632         uow.states_for_mapper_hierarchy(self.mapper, False,
False),
   633         uow,
   634     )

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/orm/persistence.py:245, in save_obj(base_mapper,
states, uowtransaction, single)
   233     update = _collect_update_commands(
   234         uowtransaction, table, states_to_update
   235     )
   237     _emit_update_statements(
   238         base_mapper,
   239         uowtransaction,
  (...)
   242         update,
   243     )
--> 245     _emit_insert_statements(
   246         base_mapper,
   247         uowtransaction,
   248         mapper,
   249         table,
   250         insert,
   251     )
   253 _finalize_insert_update_commands(
   254     base_mapper,
   255     uowtransaction,
  (...)
   271     ),
   272 )

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/orm/persistence.py:1238, in
_emit_insert_statements(base_mapper, uowtransaction, mapper, table,
insert, bookkeeping)
  1232     result = connection._execute_20(
  1233         statement.values(value_params),
  1234         params,
  1235         execution_options=execution_options,
  1236     )
  1237 else:
-> 1238     result = connection._execute_20(
  1239         statement,
  1240         params,
  1241         execution_options=execution_options,
  1242     )
  1244 primary_key = result.inserted_primary_key
  1245 if primary_key is None:

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/base.py:1705, in
Connection._execute_20(self, statement, parameters, execution_options)
  1701     util.raise_(
  1702         exc.ObjectNotExecutableError(statement),
replace_context=err
  1703     )
  1704 else:
-> 1705     return meth(self, args_10style, kwargs_10style,
execution_options)

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/sql/elements.py:334, in
ClauseElement._execute_on_connection(self, connection, multiparams,
params, execution_options, _force)
   330 def _execute_on_connection(
   331     self, connection, multiparams, params, execution_options,
_force=False
   332 ):
   333     if _force or self.supports_execution:
--> 334         return connection._execute_clauseelement(
   335             self, multiparams, params, execution_options
   336         )
   337     else:
   338         raise exc.ObjectNotExecutableError(self)

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/base.py:1572, in
Connection._execute_clauseelement(self, elem, multiparams, params,
execution_options)
  1560 compiled_cache = execution_options.get(
  1561     "compiled_cache", self.engine._compiled_cache
  1562 )
  1564 compiled_sql, extracted_params, cache_hit =
elem._compile_w_cache(
  1565     dialect=dialect,
  1566     compiled_cache=compiled_cache,
  (...)
  1570     linting=self.dialect.compiler_linting |
compiler.WARN_LINTING,
  1571 )
-> 1572 ret = self._execute_context(
  1573     dialect,
  1574     dialect.execution_ctx_cls._init_compiled,
  1575     compiled_sql,
  1576     distilled_params,
  1577     execution_options,
  1578     compiled_sql,
  1579     distilled_params,
  1580     elem,
  1581     extracted_params,
  1582     cache_hit=cache_hit,
  1583 )
  1584 if has_events:
  1585     self.dispatch.after_execute(
  1586         self,
  1587         elem,
  (...)
  1591         ret,
  1592     )

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/base.py:1806, in
Connection._execute_context(self, dialect, constructor, statement,
parameters, execution_options, *args, **kw)
  1804     raise
  1805 except BaseException as e:
-> 1806     self._handle_dbapi_exception(
  1807         e, util.text_type(statement), parameters, None, None
  1808     )
  1810 if (
  1811     self._transaction
  1812     and not self._transaction.is_active
  (...)
  1816     )
  1817 ):
  1818     self._invalid_transaction()

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/base.py:2124, in
Connection._handle_dbapi_exception(self, e, statement, parameters,
cursor, context)
  2122     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  2123 elif should_wrap:
-> 2124     util.raise_(
  2125         sqlalchemy_exception, with_traceback=exc_info[2],
from_=e
  2126     )
  2127 else:
  2128     util.raise_(exc_info[1], with_traceback=exc_info[2])

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/util/compat.py:211, in raise_(***failed resolving
arguments***)
   208     exception.__cause__ = replace_context
   210 try:
--> 211     raise exception
   212 finally:
   213     # credit to
   214     #
https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/
   215     # as the __traceback__ object creates a cycle
   216     del exception, replace_context, from_, with_traceback

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/base.py:1800, in
Connection._execute_context(self, dialect, constructor, statement,
parameters, execution_options, *args, **kw)
  1797     if conn is None:
  1798         conn = self._revalidate_connection()
-> 1800     context = constructor(
  1801         dialect, self, conn, execution_options, *args, **kw
  1802     )
  1803 except (exc.PendingRollbackError, exc.ResourceClosedError):
  1804     raise

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/default.py:1077, in
DefaultExecutionContext._init_compiled(cls, dialect, connection,
dbapi_connection, execution_options, compiled, parameters,
invoked_statement, extracted_parameters, cache_hit)
  1075 if compiled.positional:
  1076     for compiled_params in self.compiled_parameters:
-> 1077         param = [
  1078             processors[key](compiled_params[key])
  1079             if key in processors
  1080             else compiled_params[key]
  1081             for key in positiontup
  1082         ]
  1083        
parameters.append(dialect.execute_sequence_format(param))
  1084 else:

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/engine/default.py:1078, in <listcomp>(.0)
  1075 if compiled.positional:
  1076     for compiled_params in self.compiled_parameters:
  1077         param = [
-> 1078             processors[key](compiled_params[key])
  1079             if key in processors
  1080             else compiled_params[key]
  1081             for key in positiontup
  1082         ]
  1083        
parameters.append(dialect.execute_sequence_format(param))
  1084 else:

File ~/sciqlop_env/lib64/python3.11/site-
packages/sqlalchemy/dialects/sqlite/base.py:1004, in
DATETIME.bind_processor.<locals>.process(value)
   994     return format_ % {
   995         "year": value.year,
   996         "month": value.month,
  (...)
  1001         "microsecond": 0,
  1002     }
  1003 else:
-> 1004     raise TypeError(
  1005         "SQLite DateTime type only accepts Python "
  1006         "datetime and date objects as input."
  1007     )

StatementError: (builtins.TypeError) SQLite DateTime type only accepts
Python datetime and date objects as input.
[SQL: INSERT INTO events (uuid, start, stop, author, tags, products,
removed, attributes) VALUES (?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: [{'stop': '2008-08-30 19:00:00', 'start': '2008-08-30
18:00:00', 'author': 'bayane', 'products': [], 'tags': [],
'attributes': {}, 'uuid': '60910139-8b03-4401-954f-69e8d1cf9ab4'}]]

Export: JSON

  • start with JSON-format
  • exportable is a Catalogue or DynamicCatalogue
  • Events are tagged with a private attribute _filtered used at import to see whether this event has been added manually to the catalogue or if it is only a result of its filter/predicate

Issue importing big catalog from VOTable

Timetable_archive_20231115_1045.tar.gz

Importing this catalog I get:

In [2]: tscat.import_votable("/home/jeandet/Downloads/Timetable_archive_20231115_1045/THEMIS_MPcrossings_2007-2016_V1.xml")
---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
File ~/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1910, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1909     if not evt_handled:
-> 1910         self.dialect.do_execute(
   1911             cursor, statement, parameters, context
   1912         )
   1914 if self._has_events or self.engine._has_events:

File ~/.local/lib/python3.10/site-packages/sqlalchemy/engine/default.py:736, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    735 def do_execute(self, cursor, statement, parameters, context=None):
--> 736     cursor.execute(statement, parameters)

OperationalError: too many SQL variables

The above exception was the direct cause of the following exception:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.