point72 / csp Goto Github PK

There are various formatting errors which occurred when we converted our docs to Markdown. For example, in https://github.com/Point72/csp/wiki/1.-Generic-Nodes-(csp.baselib) the csp.or_ node is formatted as csp.or\_ and in https://github.com/Point72/csp/wiki/2.-Statistics-Nodes-(csp.stats) the LaTeX formatting under the EMA section is terrible. I'm sure there are many similar issues in all the other documentation files as well.

It would be great if someone can go through all the docs and proof-read/fix these errors.

@ngoldbaum if you can create a branch that setups up a build env on windows, I can work on getting the windows build working

@robambalu and I were doing some back and forth on an older fedora-based linux with no ability to modify the system packages.

The install instructions should be straightforwardly runnable in such an environment.

Is your feature request related to a problem? Please describe.
At the moment, you can't define a csp node interactively in ipython (need to be part of an importable script)

Defining the following in IPython (inside of PyCharm)

@csp.node
def f(x: csp.ts[int]) -> csp.ts[int]:
    return x

Throws

OSError: could not get source code

Describe the solution you'd like
Ideally we should be able to define them in ipython, but at the very least, the error message should be a bit more helpful (i.e. explaining that defining nodes interactively is not supported yet)

Currently the scripts in the examples folder are not being run in the tests. We should define a test_examples.py that disovers example scripts, executes them to check they run without error, and if possible, verifies the output of the script.

As the release processes are updated, we need to document the new workflow to release on PyPI and conda-forge for current and future maintainers.

(Originally noted by @ptomecek, @ngoldbaum, and @trallard)

Items to be included

• Git for maintainers
  • Make origin your fork, set upstream to the real repo, add remotes for collaborators as needed
  • How to check out a PR locally
  • Dealing with old unmerged PRs (pushing fixups to other people's forks, when to do this and when not to)
  • Setting up commit signing and adding signing keys to github
• Github for maintainers
  • Triaging issues
  • Code review guidelines and checklist
  • Communication guidelines
• Doing a "normal" release
  • Choosing a new version number
  • Generating an author list from git
  • Writing release notes
• Tagging the release and pushing to github
  • Dealing with a release mistakes
  • Yanking for completely broken releases
  • Post releases for metadata fixes (e.g. a readme fix or to fix version constraints)
  • Permanent deletion for publishing private data, but this should be avoided
  • Doing a new release with a fix

Additional items that need documenting:

• versioning approaches (SemVer)

__________________________ TestNumpyAdapter.test_int ___________________________

self = <csp.tests.adapters.test_numpy.TestNumpyAdapter testMethod=test_int>

    def test_int(self):
        raw_vals = [7, -13, 21]
        res = csp.run(g, typ=int, values=np.array(raw_vals), dts=test_dts_ndarray, starttime=test_starttime)
        self.assertEqual(res["out"], list(zip(test_dts, raw_vals)))
    
        for dtype in ("b", "h", "i", "l", "object"):
            res = csp.run(
                g, typ=int, values=np.array(raw_vals, dtype=dtype), dts=test_dts_ndarray, starttime=test_starttime
            )
>           self.assertEqual(res["out"], list(zip(test_dts, raw_vals)))
E           AssertionError: Lists differ: [(dat[61 chars], 1, 2, 1, 0), 243), (datetime.datetime(2000, 1, 2, 2, 0), 21)] != [(dat[61 chars], 1, 2, 1, 0), -13), (datetime.datetime(2000, 1, 2, 2, 0), 21)]
E           
E           First differing element 1:
E           (datetime.datetime(2000, 1, 2, 1, 0), 243)
E           (datetime.datetime(2000, 1, 2, 1, 0), -13)
E           
E             [(datetime.datetime(2000, 1, 1, 1, 0), 7),
E           -  (datetime.datetime(2000, 1, 2, 1, 0), 243),
E           ?                                        ^^
E           
E           +  (datetime.datetime(2000, 1, 2, 1, 0), -13),
E           ?                                        ^^
E           
E              (datetime.datetime(2000, 1, 2, 2, 0), 21)]

csp/tests/adapters/test_numpy.py:34: AssertionError

Not sure exactly what change in numpy caused this. I'm a numpy maintainer and am happy to try to look more closely at this if you'd like me to.

A concise description and key features can be very helpful for new users (across several domains) looking into CSP.

We can discuss some ideas here, and update the wiki homepage and README with this information.

Some ideas for "key features" from discussions with @ptomecek:

• Infrastructure-agnostic
• Highly-customizable (with adapters)
• Interoperable with PyData ecosystem
• Support for historical data, which enables robust testing

We haven't yet decided whether we want to push for adapters to all be part of csp proper, or if we want/expect adapters to be maintained independently. Regardless, we currently have C++-based parquet and kafka adapters, python-based db, csv, perspective, websocket, symphony, and slack adapters, with plans for C++-based redis, andwebsocket adapters, so it seems likely that we will have a reliance on lots of python libraries.

We will want to establish a (better) pattern for optional dependency sets, at the very least moving optional dependencies to their own extras blocks in pyproject.toml, and having a more graceful handling at runtime when dependencies are missing.

Now that it is public, once we clear up the remaining externalization issues, we should do a public pypi release. Currently the 0.0.0 release is just a placeholder.

At the very least, we should tag a release so we don't block #10

Full build: https://github.com/Point72/csp/actions/runs/7821169540

e.g. python code in https://github.com/Point72/csp/wiki/0.-Introduction#graph-building-concepts

Is your feature request related to a problem? Please describe.

Docs navigation can be improved, especially keeping new users in mind. Currently, there is a topical-organization of docs, it'll be nicer to have a distinction between API references, concepts, tutorials, etc.

Describe the solution you'd like

Restructure the docs with some principles from Diátaxis - a framework for structuring technical documentation

Tentative/rough outline:

• Home
  • What is CSP?
  • Key features
  • Get Started: Link to tutorials, concepts, how-tos, and examples (i.e., provide a learning path)
  • Community: Developer guide, roadmap
• Get Started (Tutorials)
  • ref: #65
• Concepts
  • Most of https://github.com/Point72/csp/wiki/0.-Introduction (Specifics TBD) + csp.Struct (conceptually)
  • https://github.com/Point72/csp/wiki/3.-Statistics-Nodes-(csp.stats)#getting-started
  • Caching (TBD)
• How-to
  • Use Stats Nodes - ex: 20, 21
  • Use Adapters
  • Introduce cycles in the graph (csp.feedback) - ex.13
  • Create Dynamic graphs - ex.15
  • Write Adapters
  • Profile the code (TBD)
• References
  • API References
    • https://github.com/Point72/csp/wiki/1.-Generic-Nodes-(csp.baselib)
    • https://github.com/Point72/csp/wiki/2.-Math-Nodes-(csp.math)
    • https://github.com/Point72/csp/wiki/3.-Statistics-Nodes-(csp.stats)#api-reference-1
    • https://github.com/Point72/csp/wiki/4.-Random-Time-Series-Generation-(csp.random)
    • https://github.com/Point72/csp/wiki/6.-Dynamic-Graphs (csp.DynamicBasket, csp.Dynamic)
    • csp.struct
    • csp.profiler
  • Changelog
  • Glossary
• Examples
  • List all examples with descriptions and link to files
  • Add cross-links to other wiki pages/sections
• Developer guide
  • Contributing: Issues/PRs, docs, examples, and more!
  • Development setup
  • Release process, xref: #64
  • Roadmap

Also, note that we can create a custom Sidebar. So we can have atomic documentation pages (each page covers one main topic) and have as many pages as we need without worrying about overloading the default wiki TOC.

can use redis-plus-plus which seems full featured enough and easily integrates via vcpkg

we should implement ufunc dispatch so tools like sympy's lambdify can be used with csp.DelayedEdge. Details to follow, we are going to be missing a few functions. Note that it's not strictly necessary to do it this way (I believe you can just override the expected attributes, e.g. edge.log, but I think it's a bit cleaner)

• log
• exp
• sqrt
• erf
• sin
• cos
• tan
• arcsin
• arccos
• arctan

All should be implemented as C++ nodes

@AdamGlustein in case he knows of any fancy implementations, or if stdlib is insufficient

Here's what this looks like in practice. First, you can construct and perform operations symbolically using sympy. Here, we might calculate black scholes.

import sympy as sy
from sympy.stats import Normal as syNormal, cdf
from sympy.utilities.lambdify import lambdify

#spot, strike, vol, days till expiry, interest rate, call or put (1,-1)
spot, strike, vol, dte, rate, cp = sy.symbols('spot strike vol dte rate cp')

T = dte / 260.
N = syNormal('N', 0.0, 1.0)

d1 = (sy.ln(spot / strike) + (0.5 * vol ** 2) * T) / (vol * sy.sqrt(T))
d2 = d1 - vol * sy.sqrt(T)

expr = sy.exp(-rate * T) * (cp * spot * cdf(N)(cp * d1) - cp * strike  * cdf(N)(cp * d2))

Then we can extract the variable names and bind them to csp.DelayedEdge.

from csp import DelayedEdge, ts

syms = list(expr.free_symbols)
names = [s.name for s in syms]
edges = {n: DelayedEdge(ts[float], True) for n in names}

We can then use sympy to apply the symbolic expr on our delayed edges:

modules = ["scipy", "numpy"]
func = lambdify(syms, expr, modules=modules)
func(*edges.values())

Finally, you can bind these edges to e.g. input adapters. In addition to making this code work, we can add a few helper functions to make the wrapping easier.

This has added utility because right now, csp does not support dual numbers. So if I wanted e.g. calculate some greeks of the time value expression expr above, I can do:

delta = diff(expr, spot)
gamma = diff(delta, spot)
theta = diff(expr, dte)
vega = diff(expr, vol)

And I can use the same input adapters and delayed edges for all of these. Note that csp does not support graph collapsing right now, but we've started investigating it using exprtk. Being able to do things symbolically (which might result in inefficient computation) but then collapse into pure C++ or exprtk to regain efficiency would be a very cool and powerful combination.

Describe the bug
If a csp.Struct type is defined with a mutable default ( ie dict ), then all instances of that type will share the same default. Mutate it in one, will mutate for all. This is similar to the age-old python mutable default argument problem, but we have the ability to fix it by deepcopying defaults instead of shallow copying

To Reproduce

class S(csp.Struct):
    d : dict = {}

s1 = S()
s2 = S()

s2.d['a'] = 1
print(s1)
> S( d={'a': 1} )

The images linked here are not rendering properly: https://github.com/Point72/csp/wiki/2.-Statistics-Nodes-%28csp.stats%29/#exponential-moving-average

Create a new (crude) example of the pandas extension type functionality to illustrate basic usage.

Right now we rely on policy based data structures for CSP stats code. This tethers us to gcc which is itself a problem, but more critically this reliance on gcc breaks our ABI compatibility with pypi pyarrow on Mac. This compatibility was lucky at best already (we need to move to the C api), but I'd rather have more luck than less 😂.

The immediate repercussion of this reliance is that for the time being, the parquet adapter is disabled on Mac.

I tried installing the latest linux 3.8 wheel in my Windows WSL ubuntu env. It installed fine, but on import I'm hitting missing dependencies off the bat:

• ruamel.yaml:

>>> import csp
Fatal error on import of /project/_skbuild/linux-x86_64-3.8/cmake-build/cpp/csp/engine/csp_autogen/autogen_types.cpp: failed to import struct module csp.impl.types.autogen_types
Traceback (most recent call last):
  File "/home/rob/.local/lib/python3.8/site-packages/csp/impl/types/autogen_types.py", line 3, in <module>
    from csp.impl.struct import Struct
  File "/home/rob/.local/lib/python3.8/site-packages/csp/impl/struct.py", line 2, in <module>
    import ruamel.yaml
ModuleNotFoundError: No module named 'ruamel'
signal 6 (Aborted), address is 0x3e8000001e8 from
[bt]: (0) printBacktrace()
[bt]: (1) sigabrt_handler(int, siginfo_t*, void*)
[bt]: (2) /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fd56dfd6210]

-packaging:

>>> import csp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rob/.local/lib/python3.8/site-packages/csp/__init__.py", line 3, in <module>
    from csp.baselib import *
  File "/home/rob/.local/lib/python3.8/site-packages/csp/baselib.py", line 18, in <module>
    from csp.impl.wiring import DelayedEdge, Edge, OutputsContainer, graph, input_adapter_def, node
  File "/home/rob/.local/lib/python3.8/site-packages/csp/impl/wiring/__init__.py", line 3, in <module>
    from csp.impl.wiring.delayed_edge import DelayedEdge
  File "/home/rob/.local/lib/python3.8/site-packages/csp/impl/wiring/delayed_edge.py", line 8, in <module>
    from .graph import graph
  File "/home/rob/.local/lib/python3.8/site-packages/csp/impl/wiring/graph.py", line 15, in <module>
    from csp.impl.wiring.cache_support.dataset_partition_cached_data import DataSetCachedData, DatasetPartitionCachedData
  File "/home/rob/.local/lib/python3.8/site-packages/csp/impl/wiring/cache_support/dataset_partition_cached_data.py", line 12, in <module>
    from csp.adapters.output_adapters.parquet import resolve_array_shape_column_name
  File "/home/rob/.local/lib/python3.8/site-packages/csp/adapters/output_adapters/parquet.py", line 4, in <module>
    from packaging import version
ModuleNotFoundError: No module named 'packaging'

I noticed deprecated use of python setup.py sdist and python setup.py bdist_wheel in the Makefile.

Seems that we should be using python -m build instead according to https://packaging.python.org/en/latest/discussions/setup-py-deprecated/.

I'm trying to build csp in a VM with 4 GB of RAM and it keeps getting killed by the OOM killer. We should figure out how much RAM is actually needed on the build host and document that.

Is your feature request related to a problem? Please describe.
Due to the policy-based data structures (PBDS) library being gcc specific, I had to change the implementation of Quantile and Rank in cpp/cppnodes/statsimpl.cpp to not use an order statistics tree. Instead, they use a std::multiset to store sorted values, and iterate on that set in linear time to find each quantile/rank when triggered. This is much more inefficient than the O(log n) lookup in the PBDS tree.

Describe the solution you'd like
It would be helpful if someone implemented a balanced (red-black) order statistics tree (OST) natively in csp. The OST is a binary search tree with the additional invariant that each node contains the size of its own subtree (https://en.wikipedia.org/wiki/Order_statistic_tree). This allows for efficient (log n time) insert, find, delete, and index-based lookup.

Describe alternatives you've considered
There are actually 2 data structures that will achieve O(log n) insert, delete and index-based lookup

1. The order statistics tree mentioned above
2. An indexable skiplist. This is what pandas uses for their rolling quantiles. However, pandas has the advantage of static data, so they can size it up front knowing the maximum size of the data interval. In a time-windowed csp stats node, we do not know the maximum number of ticks which will fall in an interval, so sizing becomes a bit trickier (I think we'd have to resize/add links dynamically at powers of 2). Here is a link explaining the impl https://rhettinger.wordpress.com/2010/02/06/lost-knowledge/.

Additional context
This would be helpful for any non gcc users and would also make the code more readable since we're no longer using 2 different approaches for the same stats calculation.

Discuss and document some concrete user personas, which include knowledge/experience levels, core needs, common pain points, etc.). It can:

• help with defining the project's future scope and prioritizing issues
• help direct the narrative in documentation and examples (to match user needs)
• help new contributors understand and align on historical and future decisions

csp makes it easy to build things like chatbots, which react to incoming messages, do some work, and produce outputs, potentially keeping state in between. We have built chatbots with csp before, but it would be good to make adapters available. Since Slack is quite popular and has a well maintained python SDK, we should write a slack adapter.

Opening this ticket to track, but I'm already writing one which should hopefully be done shortly.

For perspective, we (but mostly @texodus) have setup really great benchmarking which allows us to track granular release-to-release performance metrics. Example:

PR: finos/perspective#2315

We have the profiler, but we should setup a benchmark suite and run it in CI (as perspective does).

Is your feature request related to a problem? Please describe.

Using GitHub wiki for docs doesn't allow us to have "reviews" in the documentation contribution workflow.

Describe the solution you'd like

Move the wiki pages into the repository as markdown files, and create an action that moves text over to the wiki. This means we'll treat the wiki as a read-only space.

This also allows us to have linting and testing to ensure consistency.

Describe alternatives you've considered

A docs website is the ideal solution, but is currently not do-able. However, having docs in the repo will allow easier transition to a docs generator if/when that happens.

Additional context

This was discussed with @ptomecek and @robambalu in a meeting today.

Next steps:

• Move the documentation as-is into the repo
• Give attribution fairly - add everyone who originally authored the docs as co-authors on the PR
• Add automation action

Right now csp bootstraps the majority of build dependencies using vcpkg. However, if we want to eventually add a build of csp to conda-forge we will need to define a conda recipe that gets its build dependencies from conda-forge instead.

From vcpkg.json we have:

{
    "name": "main",
    "version-string": "latest",
    "dependencies": [
      "abseil",
      "arrow",
      "brotli",
      "exprtk",
      "gtest",
      "librdkafka",
      "lz4",
      "parquet",
      "protobuf",
      "rapidjson",
      "thrift",
      "utf8proc"
    ],
    "overrides": [
      { "name": "arrow", "version": "7.0.0"}
    ],
    "builtin-baseline": "288e8bebf4ca67c1f8ebd49366b03650cfd9eb7d"
  }

This corresponds to the following conda-forge dependencies:

abseil-cpp
arrow-cpp 7.0.0
brotli
exprtk
gtest
librdkafka
lz4
libparquet
protobuf
rapidjson
thrift
utf8proc

The main problem I can see in this dependency list is the pin on arrow-cpp 7.0.0. It looks conda-forge hasn't updated this version in 1 year 8 months. Version 7.0.1 was updated a little more recently, a year ago.

Looking at older arrow-cpp packages on conda-forge, since they depend on Python, the pin would also restrict us from supporting newer python versions, since python 3.11 and 2.12 don't have arrow-cpp 7.0.0 packages available. It looks like arrow-cpp dropped its python dependency starting with arrow-cpp 10.

I don't have context for why the pin on arrow is set the way it is. Can it be loosened at all?

Describe the bug
When following the steps outlined in "Building with Conda on Linux", I observed the following build error. The build failed because the cmake version was less than 3.20.0; however, upon some debugging, it seems the wrong cmake version is being pulled in by skbuild.

(csp) $ which cmake
.../github/csp_venv/envs/csp/bin/cmake
(csp) $ cmake --version
cmake version 3.28.3

Then, upon attempting the build (make build), I get the following error:

CMake Error at CMakeLists.txt:6 (cmake_minimum_required):
  CMake 3.20.0 or higher is required.  You are running version 3.13.5

Printing out the cmake/version being used I get:

Path to cmake: /usr/bin/cmake3
Cmake version: 3.13.5

It seems that skbuild is using the wrong cmake, different from what is specified by my PATH.

To Reproduce

Expected behavior
make build should pull in the correct cmake binary and succeed

Desktop (please complete the following information):

• OS: CentOS Linux release 7.3.1611 (Core)

We have an adapter for https://symphony.com/ internally. We should open source this.

The documentation should make users aware of the following convenience methods on Edge: apply, pipe and run, with some simple examples of how they can be leveraged to simplify the syntax or work with csp more easily.

apply: https://github.com/Point72/csp/blob/main/csp/impl/wiring/edge.py#L136
pipe: https://github.com/Point72/csp/blob/main/csp/impl/wiring/edge.py#L157
run: https://github.com/Point72/csp/blob/main/csp/impl/wiring/edge.py#L173

The current set of examples can be improved. They need to be triaged into either experience levels (such as "beginner", "intermediate", "advanced") and/or by document type (such as How-tos, Tutorials, or API usage example).

The examples can also be refined to contain more comments and make context clearer, ideally connecting with the corresponding API documentation.

Lastly, it would be interesting to find new sample data sources to use in the examples.

csp nodes and graphs are type-annotated to return ts["T"], and are then connected to other nodes/graphs. Additionally, the return value of these ts["T"] which is an Edge, has a number of of convenience methods such as those in #40, a number of operators, and more to come including those in #60. Finally, csp.Struct is statically typed and should integrate properly in intellisense tools.

We should look to the tools themselves to see how to best integrate:
microsoft/vscode-python#2446
microsoft/python-language-server#260

Describe the bug
It seems like csp.Struct is missing a reference count on the py struct type of nested structs. If the type loses a refcount, it no longer remains valid

To Reproduce

import csp
from csp.impl.struct import defineStruct

class Outter(csp.Struct):
    s : csp.Struct

all = []
while True:
    sType = defineStruct( 'foo', { 'a' : dict })
    all.append(Outter(s = sType(a={ 'foo' : 'bar'})))
    print(all)

Is your feature request related to a problem? Please describe.
Logging around lack of memoization of nodes with unhashable type is too noisy (WARN)
https://github.com/Point72/csp/blob/main/csp/impl/mem_cache.py#L209

Describe the solution you'd like
Log at DEBUG level.

Describe alternatives you've considered
None

Additional context
It is alarming to users to see WARNING messages that can safely be ignored.

Describe the bug
Building csp on Python 3.12 fails due to accessing CPython internals that have been removed.

To Reproduce
Steps to reproduce the behavior:

1. Build csp with a Python 3.12 interpreter
2. Wait until it gets to building csp itself.
3. Fails with the following error:

/Users/goldbaum/Documents/csp/cpp/csp/python/PyNode.cpp: In member function 'void csp::python::PyNode::init(csp::python::PyObjectPtr, csp::python::PyObjectPtr)':
/Users/goldbaum/Documents/csp/cpp/csp/python/PyNode.cpp:64:55: error: 'struct PyGenObject' has no member named 'gi_code'
   64 |     PyCodeObject * code = ( PyCodeObject * ) pygen -> gi_code;
      |                                                       ^~~~~~~

See https://github.com/python/cpython/pull/100749/files for the relevant change in CPython

Expected behavior
Build without error

Desktop (please complete the following information):

• OS: [e.g. iOS] MacOS Sonoma
• csp main branch

Is your feature request related to a problem? Please describe.
The csp.typing Numpy1DArray and NumpyNDArray types are confusing for users and lead to a lot of unexpected typing issues. We should be consistent with Numpy and use the base np.ndarray type for all Numpy time-series objects. With the distinction, you get strange bugs like this:

        inp_arr = np.zeros(shape=(4,))
        @csp.graph
        def arr_1d(x: csp.ts[csp.typing.Numpy1DArray[float]]) -> csp.ts[np.ndarray]:
            return x
        res = csp.run(arr_1d(arr_1d(csp.const(inp_arr))), starttime=st, endtime=datetime(2020, 2, 7, 9, 1)) # should not raise

This raises an error:
csp.impl.types.instantiation_type_resolver.TSArgTypeMismatchError: In function accept_arr_1d: Expected ts[csp.typing.Numpy1DArray[float]] for argument 'x', got ts[csp.typing.NumpyNDArray[float]]

Describe the solution you'd like
For now, set the Numpy1DArray type to be equal to the NDArray type and make associated changes to the parquet reader/writer such that users need to opt-in to using shape information. By default we will treat the Numpy array as 1-dimensional.

Describe alternatives you've considered
Keeping the two types as separate will continue to lead to weird bugs as I showed above.

Additional context
This change has come up as a discussion topic before as the type distinction led to bugs in the past.

We have a requirement to build against csp libraries for extended C++ adapters using the newer string ABI.
As it stands now, the manylinux wheel is being built with the old CXX string ABI and therefore wont be compatible with existing builds.

Sample issue being hit, from autogenerated code:

In file included from /home/ra7293/user/hfalgo_csp_trading/_skbuild/linux-x86_64-3.11/cmake-build/cpp/csp-trading/adapters/onefeed/csp_autogen/onefeed_types.cpp:2:
/home/ra7293/user/hfalgo_csp_trading/_skbuild/linux-x86_64-3.11/cmake-build/cpp/csp-trading/adapters/onefeed/csp_autogen/onefeed_types.h: In member function 'const std::string& csp::autogen::Header::session_id() const':
/home/ra7293/user/hfalgo_csp_trading/_skbuild/linux-x86_64-3.11/cmake-build/cpp/csp-trading/adapters/onefeed/csp_autogen/onefeed_types.h:455:46: error: static assertion failed
  455 |         static_assert( sizeof( std::string ) == 8 );

code was generated using the csp wheel's libraries, which have std::string as 8 bytes ( old ABI )
but the compilation we're doing is the new ABI, and so this assertion fails

Describe the bug
In the introduction section of the documentation, under the heading Anatomy of a csp.node the code does not run. We get a python error instead.

To Reproduce
Go to https://github.com/Point72/csp/wiki/0.-Introduction, copy the example code, add from datetime import timedelta and run it:

from datetime import timedelta

@csp.node                                                                # 1
def demo_node(n: int, xs: ts[float], ys: ts[float]) -> ts[float]:        # 2
    with csp.alarms():                                                   # 3
        # Define an alarm time-series of type bool                       # 4
        alarm = csp.alarm(bool)                                          # 5 
                                                                         # 6
    with csp.state():                                                    # 7
        # Create a state variable bound to the node                      # 8
        s_sum = 0.0                                                      # 9
                                                                         # 10
    with csp.start():                                                    # 11
        # Code block that executes once on start of the engine           # 12
        # one can set timeseries properties here as well, such as        # 13
        # csp.set_buffering_policy(xs, tick_count=5)                     # 14
        # csp.set_buffering_policy(xs, tick_history=timedelta(minutes=1))# 15
        # csp.make_passive(xs)                                           # 16
        csp.schedule_alarm(alarm, timedelta(seconds=1), True)            # 17
                                                                         # 18
    with csp.stop():                                                     # 19
        # code block to execute when the engine is done                  # 20
                                                                         # 21
    if csp.ticked(xs, ys) and csp.valid(xs, ys):                         # 22
        s_sum += xs * ys                                                 # 23
                                                                         # 24
    if csp.ticked(alarm):                                                # 25
        csp.schedule_alarm(alarm, timedelta(seconds=1), True)            # 26
        return s_sum                                                     # 27

Expected behavior
Functional code. :) More seriously, adding pass on line 21 would solve the issue. At the same time one could add from datetime import timedelta to make the code more self-contained.

Error Message

Cell In[8], [line 24](vscode-notebook-cell:?execution_count=8&line=24)
    if csp.ticked(xs, ys) and csp.valid(xs, ys):                         # 22
    ^
IndentationError: expected an indented block after 'with' statement on line 21

Runtime Environment
0.1.0
3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:43:09) [GCC 12.3.0]
linux

We want to make sure releases get github releases along with the sdist and wheels. That will also allow us to upload wheels for more platforms if we hit pypi resource usage limits.

@timkpaine shared https://github.com/finos/perspective/blob/aaff331a2ede998aeae28ea61c806766e1b4c453/.github/workflows/build.yml#L1809 as an example to crib off of.

See below

In file included from /home/goldbaum/github/csp/_skbuild/linux-aarch64-3.11/cmake-build/cpp/csp/engine/csp_autogen/autogen_types.h:6,
                 from /home/goldbaum/github/csp/cpp/csp/engine/BasketInfo.cpp:2:
In member function 'void csp::Struct::decref()',
    inlined from 'void csp::TypedStructPtr<T>::decref() [with T = csp::autogen::DynamicBasketEvents]' at /home/goldbaum/github/csp/cpp/csp/engine/Struct.h:732:20,
    inlined from 'csp::TypedStructPtr<T>::~TypedStructPtr() [with T = csp::autogen::DynamicBasketEvents]' at /home/goldbaum/github/csp/cpp/csp/engine/Struct.h:432:19,
    inlined from 'void csp::DynamicOutputBasketInfo::addShapeChange(const csp::DialectGenericType&, bool)' at /home/goldbaum/github/csp/cpp/csp/engine/BasketInfo.cpp:167:5:
/home/goldbaum/github/csp/cpp/csp/engine/Struct.h:692:27: warning: pointer used after 'static void csp::Struct::operator delete(void*)' [-Wuse-after-free]
  692 |         if( --hidden() -> refcount == 0 )
      |               ~~~~~~~~~~~~^~~~~~~~
In member function 'void csp::Struct::decref()',
    inlined from 'void csp::TypedStructPtr<T>::decref() [with T = csp::Struct]' at /home/goldbaum/github/csp/cpp/csp/engine/Struct.h:732:20,
    inlined from 'csp::TypedStructPtr<T>::~TypedStructPtr() [with T = csp::Struct]' at /home/goldbaum/github/csp/cpp/csp/engine/Struct.h:432:19,
    inlined from 'void csp::DynamicOutputBasketInfo::addShapeChange(const csp::DialectGenericType&, bool)' at /home/goldbaum/github/csp/cpp/csp/engine/BasketInfo.cpp:164:45:
/home/goldbaum/github/csp/cpp/csp/engine/Struct.h:693:20: note: call to 'static void csp::Struct::operator delete(void*)' here
  693 |             delete this;
      |                    ^~~~

Describe the bug
Running e_11_websocket_output.py errors out with a type error.

To Reproduce

cd csp/examples
python e_11_websocket_output.py

Traceback (most recent call last):
  File "/Users/goldbaum/Documents/csp/csp/examples/e_11_websocket_output.py", line 58, in <module>
    csp.run(main, port, num_keys, starttime=datetime.utcnow(), endtime=timedelta(seconds=360), realtime=True)
  File "/Users/goldbaum/Documents/csp/csp/impl/wiring/runtime.py", line 183, in run
    with ExceptionContext():
  File "/Users/goldbaum/Documents/csp/csp/impl/error_handling.py", line 40, in __exit__
    raise exc_val.with_traceback(new_tb)
TypeError: baselibimpl.cpp:start:760:TypeError: MyData.fromts() field "angle" expected ts type DOUBLE but got INT64

Expected behavior
Script runs without error

Describe the bug

When there is a type mismatch in the return value of a graph, the message refers to argument 'None' instead of the return value, which is confusing to users, especially for more complicated examples where there are multiple inputs to the graph (especially when some arguments might default to "None").

csp.impl.types.instantiation_type_resolver.TSArgTypeMismatchError: Expected ts[str] for argument 'None', got ts[float]

To Reproduce

import csp
from datetime import datetime

@csp.graph
def graph(x: csp.ts[float]) -> csp.ts[str]:
    return x

csp.run(graph, csp.const(1.), starttime=datetime.utcnow())

Expected behavior

Change the error message to:

csp.impl.types.instantiation_type_resolver.TSArgTypeMismatchError: Expected ts[str] for return value, got ts[float]

It would also be nice if it could give you more information about which function it happened in.

Describe the bug
Running e_08_kafka.py errors out

To Reproduce
Steps to reproduce the behavior:

cd csp/examples
python e_08_kafka.py

Traceback (most recent call last):
  File "/Users/goldbaum/Documents/csp/csp/examples/e_08_kafka.py", line 6, in <module>
    from csp.adapters.kafka import (
ImportError: cannot import name 'ProtoMessageMapper' from 'csp.adapters.kafka' (/Users/goldbaum/Documents/csp/csp/adapters/kafka.py)

Expected behavior

No error

Is your feature request related to a problem? Please describe.
https://github.com/Point72/csp/wiki/0.-Introduction has great explanatory content. However, it can be too deep for first-time users who want to quickly learn the basics and try some examples.

Describe the solution you'd like
Have a more condensed “Getting started” tutorial with only the basic concepts and link to more detailed contents.

Additional context
Originally proposed by @melissawm, who also created the initial outline.

We should consider switching to the limited python api and build abi3-py38 wheels. This would considerably reduce our CI/CD footprint.

Is your feature request related to a problem? Please describe.
New users can benefit from knowing where the project is headed, what that can expect, and where they can potentially contribute.

Describe the solution you'd like
Discuss potential themes, focus areas, and priorities for the next 12-18, and create a roadmap document. We can then create specific issue tracking roadmap items.

Additional context
There is a preliminary list here: https://github.com/Point72/csp/wiki#roadmap

csp.abs is accidentally recursive, as it is defining the name abs but attempts to use the global python builtin abs in its python definition:

@node(cppimpl=_cspbaselibimpl.abs)
def abs(x: ts[float]) -> ts[float]:
    if csp.ticked(x):
        return abs(x)

decoupling from #22 so as to not mess with #26

• show full end-to-end setup
• document what oauth scopes are necessary for what functionality
• define minimum subset and make sure supported in code (e.g. things should turn off without erroring if missing scopes)
• make a minimal chatbot

This will be tricky given our use of threading, our various external adapters, reliance on filesystem, etc etc.

There are some references to non-existent or personal email addresses that need to be consolidated, will track them here:

• https://github.com/Point72/csp/blob/main/.github/CODE_OF_CONDUCT.md
• https://github.com/Point72/csp/blob/main/pyproject.toml#L13
• https://github.com/Point72/csp/blob/main/csp/examples/e_23_cpp_example/pyproject.toml#L7
• https://github.com/Point72/csp/blob/main/csp/examples/e_24_cpp_struct_example/pyproject.toml#L7