gregtuc / stocksocket Goto Github PK

• Need a method that can kill all processes from the module.
• StockSocket.stop()

Hi Greg,

Great Work.

How can we add realtime volume updates to this script?

Thanks in advance.

Hey :)

i think directly using the finStreamer from yahoo is problematic because of copyright reasons. Its rather easy to reverse engineer the code though. After i used this library initially, i independently realized, that yahoo uses websockets and built my own (naive) implementation. I just need the current pricedata so my PricingData is rather simple, but it can easily be extended.

I think just reverse engineering the datastructure and building it yourself (like i did) is certainly the safer approach :)

const WebSocket = require("ws");
const { Type, Field } = require("protobufjs");

var PricingData = new Type("PricingData")
  .add(new Field("symbol", 1, "string"))
  .add(new Field("price", 2, "float"));

module.exports = class StockTicker {
  #client = null;
  #updateCallback = null;
  #symbols = [];

  constructor(symbols, updateCallback) {
    this.symbols = symbols;
    this.#updateCallback = updateCallback;
  }

  get symbols() {
    return this.#symbols;
  }

  set symbols(newSymbols) {
    if (Array.isArray(newSymbols)) {
      this.#symbols = newSymbols;
      if (this.#client) {
        this.#sendSymbolMessage();
      }
    } else {
      throw new TypeError("stocks needs to be an array of strings");
    }
  }

  #sendSymbolMessage() {
    console.log("Requesting symbols", this.symbols);
    this.#client.send(JSON.stringify({ subscribe: this.symbols }));
  }

  #onMessage(data) {
    const binary = Buffer.from(data, "base64");
    try {
      const msg = PricingData.decode(binary);
      const msgObj = PricingData.toObject(msg, { longs: Number });
      console.log(msgObj);

      if (this.#updateCallback) {
        this.#updateCallback(msgObj);
      }
    } catch (e) {
      console.log(e);
    }
  }

  activate() {
    if (this.#client) {
      console.warn("Already Active");
      return;
    }
    this.#client = new WebSocket("wss://streamer.finance.yahoo.com/");
    this.#client.once("open", () => {
      console.log("connected to yahoo finance socket");
      this.#sendSymbolMessage();
    });

    this.#client.on("message", this.#onMessage.bind(this));
  }
  deactivate() {
    if (!this.#client) {
      console.warn("Already Inactive");
      return;
    }
    this.#client.terminate();
    this.#client = null;
  }
};

• In addition to price data, price-change data should be scraped as well and added as a property to each data object returned to the callback function.

1. Get the selectors for the price-change data (on Yahoo, it appears directly next to the price).
2. Using a similar approach to as was used with the standard price data, create an array of selectors (as the selectors will break from premarket -> standard market -> postmarket, or for crypto) and then loop through them to find which is valid.
3. When the standard price changes, query for the price-change element and group it in with the response.

• Create a method in StockSocket.js called "addTicker" that accepts two parameters: a ticker (string), the callback (function).
• This method should add the ticker to globaltickers, and then pass the ticker and callback method to startDataFeed.
• Document in readme

• Warning of potential memory leak after 11 or more tickers are added.
• Likely source: The puppeteer processes are never actually closed, and the system interprets it as a memory leak.
• Likely solution: Add two methods: stopAll(), and stop() which kill puppeteer processes when called.

• Modify constant variable "potentialSelectors" on line 35 of StockSocket.js by adding the selector for pre-market data to the head (important) of the array list.
• Note for above: The selector should be trimmed to make it less sensitive to slight selector changes.

Module works fine for server-side implementation but has issues running client-side.

If not, is something like this planned?

• Automatically switch selectors based on the active trading period.

• Pre-Market selector: "#quote-header-info > div.Pos\(r\) > div.D\(ib\) > p > span"

• Post-Market selector: "#quote-header-info > div.Pos\(r\) > div > p > span"

• Standard-Market selector: #quote-header-info > div.Pos\(r\) > div > div > span"

• Crypto selector (24/7): "#quote-header-info > div.D\(ib\) > div > div > span"

• Puppeteer processes should be refactored to open pages for each ticker, rather than browsers. This will enormously improve speed and drastically reduce resource use.

Hi, first of all I must thank you for creating such wonderful yahoo live streamer ! I am Vishvesh Upadhyay from India. All the questions below are for NSE (National Stock Exchange) India.

I am very new to programming world ! and almost first time using Node.Js and JS program / module.

I am currently using yahooquery with sqlite3 on google colab I am facing quite an issue regarding timing of the price-volume data.

I need your guidance on how use your StockSocket. (because in python i just pip install -- and run code from Spyder and it gives me the result, I downloaded you repo and create package with npm init and npm install but dont know how to start the actual web streaming !)

1. Can you guide me how to setup it ? (I just need some direction for how to run it and get live streaming data then after I will use it with my python code)

2. I am running a python code which is fetching currently 40 tickers at once in almost 1 to 1.2 seconds. With your code is it possible to stream 40 tickers (or my target 160 tickers) simultaneously and reliably ?

3. If possible i need your views on if connection is dropped or disconnected then is there any mechanism to reconnect it automatically ?

4. Also i need your suggestion for my project. I am creating an stock alert system (then planning to upgrade it to full fledged algo system). My plan is to get data live data from StockSocket -> then feed it to Python - Pandas -> then to Sqlite3 Local database -> fetch multi timeframe OHLCV candle from DB using Pandas -> then running different analysis on them -> create alert system (max time for all tasks is up to 5-7 secs) and then evolve it to full fledged algo system.

Thanks and reply soon !

Regards,
Vishvesh Upadhyay.

Setup continuous integration, unit tests, and continually upgrade them

• Need to add two methods for removing one ticker or many tickers from being observed.

1. removeTicker(ticker)
2. removeTickers([tickers])

Hey, nice socket client! We use yahoo as a fallback data provider.
I lead engineering at Fey and we're looking for engineers (Backend | TS/Node). Wondering if you'd be interested to learn more?

• Setup testing that can check for the presence of selectors used to scrape data from Yahoo.
• Tests should check to verify access to standard stocks, crypto stocks, pre-market stock data, and post-market stock data.

if not, Are there any plans for making typing?

• Current Functionality: A for loop cycles through an array of potential JS path's to find the proper path for the inputted ticker.
• New Functionality: A new JS path was found that eliminates the differences between pages for crypto and standard stocks, and makes the selectors less sensitive to being broken.

• Selector to use for refactoring: document.getElementsByClassName('.Fz(36px))');

I want to pull the current data and not updates, I was wondering if this module supports this, or if I should use another module.
Thanks.

I am getting this error for each headless page it open.

Error: Evaluation failed: TypeError: Failed to execute 'observe' on 'MutationObserver': parameter 1 is not of type 'Node'.
at puppeteer_evaluation_script:26:16
at ExecutionContext._evaluateInternal (Q:\Node\Trader\node_modules\puppeteer\lib\cjs\puppeteer\common\ExecutionContext.js:218:19)
at processTicksAndRejections (node:internal/process/task_queues:94:5)
at async ExecutionContext.evaluate (Q:\Node\Trader\node_modules\puppeteer\lib\cjs\puppeteer\common\ExecutionContext.js:107:16)
at async startDataFeed (Q:\Node\Trader\node_modules\stocksocket\stocksocket.js:76:5)

And my project contains only the code example.

on starting it is just giving message "StockSocket has opened a WebSocket Connection with Yahoo." else the callback is not getting executed

• Current Functionality: Insert headless puppeteer instances and use mutation observers to detect changes in HTML text. This method is effective but needlessly intensive WRT resources and can error out if the DOM is ever changed.

• New Functionality: Intercept Yahoo Finance websocket traffic directly.

General Steps:

1. Convert the data (which is received as a base64 string) to a bytes array.
2. Import protobuf files from Yahoo and decode the bytes array into a JS object.

• Image of Websocket traffic:
  

• Image of New Data Response: