christopherliew / crypto-uncertainty-index Goto Github PK

• Develop aggregation logic
• Construct relevant pipelines

1. Explore Wiki Dumps and obtain initial raw data
2. Process data and construct Wiki Weasel initial dataset
3. [Good to Have] Explore Edit History to enrich Wiki Weasel tags

• Prepare Training Data Preprocessing Pipeline
• Prepare model tuning scripts with W&B

• Transformer
• Vinzsce's Model

• Hyperparam Tune with RayTune + W&B
• Select model and perform inference
• Build Hedge based Uncertainty Index

• Build pipelines to pull yfinance and other related ticker data
• Exploratory time series analysis with Lucey Index

Naming conventions and functional abstraction needs to be tidied up:

• Refactor dataclass to pydantic
• Refactor data engineering pipelines

• Add in Rich Tables for Pipelines
• Add in relevant Utils
• Abstract where possible

• Refactor CLI

• Add in help
• Fix Poetry integration

• Build baseline models and evaluate
• Rework Time Series analysis
• Rework / Tune Forecasting models
• Evaluate using Diebold Mariano
• Enhance BTC-USD data with more technical analysis indicators
• Build Dynamic Factor Model to synthesise TA indicators

Current indexing field is automatically created by ES upon bulk insert, to prevent duplication (i.e. idempotence) when re-running extraction pipelines, the _id field should use reddit's own comment / submission id which is always unique.

• Extract raw data from pkl files and Process
• Training Pipelines / Script for Topic Modelling
• Test out topic modelling algorithms
• Build up topics and define lexicon for uncertainty related topics and word distributions
• Construct new index
• Clean up noise in text
• Remodel with LDA with Gensim / Mallet for K = {2, 9} and Compute BIC/ Bayes Factor
• Run Top2Vec with Doc2Vec and tune for fewer clusters
• Test out Bigrams (Debugging in Progress / KIV)

• Update CLI docs + Add Examples
• Update Results
• Update set up

If time permits, explore using kedro to modularise code. See docs: https://kedro.readthedocs.io/en/stable/

EDA, Lexicon enhancement, etc. prior to building our text based keyword uncertainty index

• Explore raw data (Using notebooks & Kibana dashboards)
• Explore emoticons and crypto lexicons to create mappings for processing
• Reindex existing index to new ES index with custom text analyzer
• Build up preliminary text preprocessing and relevant pipelines using Elasticsearch / SpaCy / NLTK and others

• Code up extraction script and utilities
• Build & Test Data Extraction and Loading Pipelines
• Extract and load all crypto data
• Do up data dictionary

1. Set up local ES, Kib and Postgres
2. Set up Helpers and Schemas