Giter Club home page Giter Club logo

stocksera's Introduction

Hello World ๐Ÿ‘‹

Page Count

I'm Guan Quan, currently studying in NUS Business Analytics. Feel free to message me for collaboration!

Languages:

python java javascript r ruby html

Data Science:

pytorch tensorflow pandas

Web Development:

django react rails vue express tailwind

Miscellaneous:

git docker aws

GitHub stats

stocksera's People

Contributors

guanquann avatar istrau2 avatar spartan737 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

stocksera's Issues

TypeError: unhashable type: 'slice'

python3 manage.py runserver 0.0.0.0:8000
Watching for file changes with StatReloader
Performing system checks...

SQL SERVER IS CONNECTED...
Version 7.2.0 of praw is outdated. Version 7.5.0 was released Sunday November 14, 2021.
SQL SERVER IS CONNECTED...
Not authorised to have access to admin functions
SQL SERVER IS CONNECTED...
System check identified no issues (0 silenced).
May 04, 2022 - 20:38:15
Django version 3.1.6, using settings 'stocksera.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
[04/May/2022 20:38:29] "GET /api/stocksera_trending HTTP/1.1" 301 0
[04/May/2022 20:38:29] "GET /api/stocksera_trending/ HTTP/1.1" 200 60
Internal Server Error: /
Traceback (most recent call last):
File "/home/settd/.local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/settd/.local/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/settd/stocks/Stocksera/app/views.py", line 34, in main
data = requests.get(f"{BASE_URL}/stocksera_trending", headers=HEADERS).json()[:10]
TypeError: unhashable type: 'slice'
[04/May/2022 20:38:29] "GET / HTTP/1.1" 500 61768

Sorry that may be a novice question, but any help would be much appreciated!

Get Earnings Calander failing on multiple versions

https://finance.yahoo.com/calendar/earnings?day=2021-09-26&size=100&offset=0
Traceback (most recent call last):
  File "tasks_to_run.py", line 126, in <module>
    get_earnings_calendar.insert_earnings_into_db(get_earnings_calendar.get_earnings(7, forward=True))
  File "/root/_production_website/AuroraLight/Stocksera/scheduled_tasks/get_earnings_calendar.py", line 70, in get_earnings
    name = tds[1].text
IndexError: list index out of range

This error I cannot get past, I have experienced it for multiple versions and have been able to conclude that the link is not broken. All I can gather is that possible the BS4 dom is broken in reference to scraping the data? I have not looked very deep into this one.

Historic Short Rate

Hi I'm trying to pull the historic borrow rates from the page. Is this possible with the API or is it ok to scrape this data off the site?

  • Thanks

Idea: can we integrate robinhood api into this.

  • Bot

algorithms

Just incase someone tries to use this, the algorithm each day will have a leverage ETF added for edging.

  • Buy hold
  • Buy and sell tomorrow if threshold is met
  • Just_sell tomorrow
  • Sell at overall account value

APIS know issues:

1. coingecko api needs to be replaced or add a try and continue in task script. (fixed -09-11-2021)
2. Yahoo API in the future maybe down. Need to redirect to another source in the future to prevent down time when the bot is requesting data.

https://github.com/guanquann/Stocksera/blob/32a9371324fb1268ca8018b3539581774a57a22d/fast_yahoo.py#L96C1-L97C1

Options expiration date list expands when selecting new date

Hi, I noticed the following issue on the Stock Options page: If you select an expiration date from the expiration date list, the list will expand with a duplicate of all the previous dates.

Steps to reproduce:

  1. Select a stock and go to Stock => Options
  2. Open the expiration date list, you can see that here are certain dates listed
  3. Select an expiration date from the list
  4. Open the expiration date list again and you will see that it's expanded with exactly the same dates (duplicate)

error database file

im getting error in:
-database , for some reason dont getting the sentiment column
-getiing error on database/indices/snp500_heatmap.csv no such file or directory
-EmptyDataError at /senate/ No columns to parse from file

Successfully created/updated database
https://www.reddit.com/r/wallstreetbets/comments/rd68ig/daily_discussion_thread_for_december_10_2021/
Empty DataFrame
Columns: [prvCls, volume, price, mkt_cap, mentions, sentiment, calls, puts]
Index: []
Traceback (most recent call last):
File "/GitHub/Stocksera/tasks_to_run.py", line 125, in
scrape_stocks_discussion_thread.wsb_change()
File "/GitHub/Stocksera/scheduled_tasks/reddit/stocks/scrape_discussion_thread.py", line 231, in wsb_change
"WHERE date_updated >= '{}' GROUP BY ticker ORDER BY SUM(mentions) DESC LIMIT 50".format(threshold_hour))
sqlite3.OperationalError: no such column: sentiment

tasks get_upcoming_events_date | TypeError: 'NoneType' object is not subscriptable

I am running like this
sudo docker run --rm stocksera_scheduled_tasks python3 scheduled_tasks/economy/get_upcoming_events_date.py
and get this error ? do you have also the same issue

Traceback (most recent call last):
  File "scheduled_tasks/economy/get_upcoming_events_date.py", line 99, in <module>
    main()
  File "scheduled_tasks/economy/get_upcoming_events_date.py", line 82, in main
    rrp_treasury_date = get_next_rrp_treasury_date(datetime.strptime(record_date[0], "%Y-%m-%d") + timedelta(days=1))
TypeError: 'NoneType' object is not subscriptable

get_latest_insider_trading.pyn error unable to decipher

Traceback (most recent call last):
  File "tasks_to_run.py", line 280, in <module>
    get_latest_insider_trading.main()
  File "/code/scheduled_tasks/discover/get_latest_insider_trading.py", line 89, in main
    latest_insider_trading()
  File "/code/scheduled_tasks/discover/get_latest_insider_trading.py", line 27, in latest_insider_trading
    insider_trader = finsider.get_insider()
  File "/usr/local/lib/python3.7/site-packages/finvizfinance/insider.py", line 85, in get_insider
    info_dict["SEC Form 4 Link"] = col.find("a").attrs["href"]
AttributeError: 'NoneType' object has no attribute 'attrs'

I cant figure this one out any help appreciated, i am on a older version 4 commits back mannually updating, so that is the updated code. Could the finvizfinance have any other associated files i need to change?

/short_interest/ source?

The Stocksera README hyperlink text says short_interest data is from shortinterest.com, however, the hyperlink target is highshortinterest.com.

Stocksera data has a short interest of >15% but highshortinterest.com supplies only >20%.
shortinterest.com appears to be domain parked.

What is the current data source?

twitter Too Many Requests

I am trying to run tasks_to_run.py but it never finishes. I get this error

scheduled_tasks_1  | Traceback (most recent call last):
scheduled_tasks_1  |   File "tasks_to_run.py", line 151, in <module>
scheduled_tasks_1  |     scrape_twitter_posts.main()
scheduled_tasks_1  |   File "/code/scheduled_tasks/twitter/scrape_trending_posts.py", line 17, in main
scheduled_tasks_1  |     json_response = connect_to_endpoint(url)
scheduled_tasks_1  |   File "/code/scheduled_tasks/twitter/twitter_connection.py", line 25, in connect_to_endpoint
scheduled_tasks_1  |     response.status_code, response.text
scheduled_tasks_1  | Exception: Request returned an errors: 429 {"title":"Too Many Requests","detail":"Too Many Requests","type":"about:blank","status":429}
stocksera_scheduled_tasks_1 exited with code 1

Any ideas how to make it to finish all the script.. thanks

Using Mysql on Docker with new changes

I want to transfer your old sqlite version to to the new mysql version and give the new code a try but cant figure out how to get docker to install mysql in the dockerfile or what steps to take.

Tickers and Repo page errors in django

Receiving many OperationalError at /ticker/ I have tried restarting creation and still cant clear up the errors on multiple pages

/tickers/

Environment:


Request Method: GET
Request URL: http://localhost:3131/ticker/

Django Version: 3.1.6
Python Version: 3.7.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'templatetags.filter',
 'ViewStocks']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/root/_startUp/Stocksera/ViewStocks/views.py", line 43, in stock_price
    information = check_market_hours(ticker_selected)
  File "/root/_startUp/Stocksera/helpers.py", line 84, in check_market_hours
    "DO UPDATE SET count=count+1", (ticker_selected, information["longName"]))

Exception Type: OperationalError at /ticker/
Exception Value: near "ON": syntax error

/reverse_repo/

Environment:


Request Method: GET
Request URL: http://localhost:3131/reverse_repo/

Django Version: 3.1.6
Python Version: 3.7.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'templatetags.filter',
 'ViewStocks']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/root/_startUp/Stocksera/ViewStocks/views.py", line 777, in reverse_repo
    with open(r"database/economic_date.json", "r+") as r:

Exception Type: FileNotFoundError at /reverse_repo/
Exception Value: [Errno 2] No such file or directory: 'database/economic_date.json'

Unexpected error on Short Vol

So i was running this yesteraday and got rid of all the errors except previously specified ones in issues.

now I am seeing this issue ? I am sorry haha

It is on the short vol request after grabbing the vol .txt's

dn.finra.org/equity/regsho/daily/CNMSshvol20230525.txt
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/multitasking/__init__.py", line 102, in _run_via_pool
    return callee(*args, **kwargs)
  File "/code/scheduled_tasks/reddit/stocks/fast_yahoo.py", line 184, in quick_stats_request_threaded
    shared.response_dict[request_idx] = quick_stats_request(request_symbol_list, field_list)
  File "/code/scheduled_tasks/reddit/stocks/fast_yahoo.py", line 194, in quick_stats_request
    'symbols': ','.join(request_symbol_list),
TypeError: sequence item 45: expected str instance, float found

Fast Yahoo seems to be breaking very often :/

Not able to figure this out it keeps breaking, let me know i just made sure all packages were up to date using the pip pur package

I am unable to find a personal fix this time i am supplying the errors I am seeing.

Traceback (most recent call last):
  File "scheduled_tasks/stocks/get_short_volume.py", line 103, in <module>
    main()
  File "scheduled_tasks/stocks/get_short_volume.py", line 99, in main
    get_daily_data_finra()
  File "scheduled_tasks/stocks/get_short_volume.py", line 89, in get_daily_data_finra
    stats_df = fast_yahoo.download_quick_stats(highest_shorted["Symbol"].to_list(), quick_stats)
  File "scheduled_tasks/stocks/../../scheduled_tasks/reddit/stocks/fast_yahoo.py", line 129, in download_quick_stats
    for retrieved_stats_dict in response_list:
TypeError: 'NoneType' object is not iterable

current fast_yahoo code:

import scheduled_tasks.reddit.stocks.shared as shared
import requests
import numbers
import multitasking as multitasking
import time
import math
import pandas as pd

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) '
                         'Chrome/50.0.2661.102 Safari/537.36'}


def download_advanced_stats(symbol_list, module_name_map, threads=True):
    """
    Downloads advanced yahoo stats for many tickers by doing one request per ticker.
    """
    num_requests = len(symbol_list)
    if threads:
        num_threads = min([num_requests, multitasking.cpu_count() * 2])
        multitasking.set_max_threads(num_threads)

    # get raw responses
    for request_idx, symbol in enumerate(symbol_list):
        if threads:
            get_ticker_stats_threaded(symbol, symbol, module_name_map)
        else:
            shared.response_dict[symbol] = get_ticker_stats(symbol, module_name_map)

    if threads:
        while len(shared.response_dict) < num_requests:
            time.sleep(0.01)

    # construct stats table from responses
    stats_table = []
    for symbol, retrieved_modules_dict in shared.response_dict.items():
        stats_list = [symbol]

        for module_name, stat_name_dict in module_name_map.items():
            retrieved_module_dict = None
            if retrieved_modules_dict is not None and module_name in retrieved_modules_dict:
                retrieved_module_dict = retrieved_modules_dict[module_name]

            if retrieved_module_dict is not None:
                for stat_name in stat_name_dict.keys():
                    stat_val = 'N/A'
                    if stat_name in retrieved_module_dict:
                        stat = retrieved_module_dict[stat_name]
                        if isinstance(stat, dict):
                            if stat:  # only if non-empty otherwise N/A
                                if stat_name == "shortPercentOfFloat":
                                    stat_val = stat['fmt']
                                else:
                                    stat_val = stat['raw']
                        elif isinstance(stat, str) or isinstance(stat, numbers.Number):
                            stat_val = stat
                        else:
                            raise TypeError('Expected dictionary, string or number.')
                    stats_list.append(stat_val)
            else:
                stats_list.extend(['N/A'] * len(stat_name_dict))

        stats_table.append(stats_list)

    # reset for future reuse
    shared.response_dict = {}

    columns = ['Symbol']
    for stat_name_dict in module_name_map.values():
        columns.extend(list(stat_name_dict.values()))

    financial_data_df = pd.DataFrame(stats_table, columns=columns)
    financial_data_df.set_index('Symbol', inplace=True)

    return financial_data_df


def download_advanced_stats_multi_thread(symbol_list, params, num_threads=100):
    df = pd.DataFrame()
    current_index = 0
    while current_index < len(symbol_list):
        quick_stats_df = download_advanced_stats(symbol_list[current_index:current_index + num_threads],
                                                 params, threads=True)
        df = pd.concat([df, quick_stats_df])
        current_index += num_threads
    return df


def download_quick_stats(symbol_list, quick_stats_dict, threads=True):
    """
    Downloads select ("quick") stats for many tickers using minimal number of http requests. Splits the ticker list
    into groups of 1000 and performs one request per group. eg if list has 2350 tickers, will split into 2 groups of
    1000 tickers and one group with the remaining 350 tickers, and will get quick stats with only 3 http requests. Only
    returns those tickers that are valid, thus can be used to validate tickers efficiently.
    """
    # through trial and errors, 1179 was the max without returning an errors, but that number feels too arbitrary
    max_params = 1000
    num_requests = math.ceil(len(symbol_list)/max_params)
    last_request_size = len(symbol_list) % max_params
    if last_request_size == 0:
        last_request_size = max_params

    if threads:
        num_threads = min([num_requests, multitasking.cpu_count() * 2])
        multitasking.set_max_threads(num_threads)

    # get raw responses
    for request_idx in range(num_requests):

        if request_idx == num_requests - 1:
            num_symbols = last_request_size
        else:
            num_symbols = max_params

        request_symbol_list = symbol_list[request_idx * max_params:request_idx * max_params + num_symbols]

        if threads:
            quick_stats_request_threaded(request_idx, request_symbol_list, list(quick_stats_dict.keys()))
        else:
            shared.response_dict[request_idx] = quick_stats_request(request_symbol_list, list(quick_stats_dict.keys()))

    if threads:
        while len(shared.response_dict) < num_requests:
            time.sleep(0.01)

    # construct stats table from responses
    stats_table = []
    for response_list in shared.response_dict.values():
        # each iteration is one symbol; (eg SIGL, AAPL)
        for retrieved_stats_dict in response_list:
            symbol = retrieved_stats_dict['symbol']
            stats_list = [symbol]
            if retrieved_stats_dict is not None:
                for quick_stat_name in quick_stats_dict.keys():
                    stat_val = 'N/A'
                    if quick_stat_name in retrieved_stats_dict:
                        stat = retrieved_stats_dict[quick_stat_name]
                        if isinstance(stat, dict):
                            if stat:  # only if non-empty otherwise N/A
                                if quick_stat_name == "floatShares":
                                    stat_val = stat['fmt']
                                else:
                                    stat_val = stat['raw']
                        elif isinstance(stat, str) or isinstance(stat, numbers.Number):
                            stat_val = stat
                        else:
                            raise TypeError('Expected dictionary, string or number.')
                    stats_list.append(stat_val)
            else:
                stats_list.extend(['N/A'] * len(quick_stats_dict.keys()))

            stats_table.append(stats_list)

    # reset for future reuse
    shared.response_dict = {}

    # construct dataframe
    columns = ['Symbol'] + list(quick_stats_dict.values())
    stats_df = pd.DataFrame(stats_table, columns=columns)
    stats_df.set_index('Symbol', inplace=True)

    return stats_df


@multitasking.task
def get_ticker_stats_threaded(request_idx, symbol, module_name_map):
    shared.response_dict[request_idx] = get_ticker_stats(symbol, module_name_map)


def get_ticker_stats(symbol, module_name_map):
    """
    Returns advanced stats for one ticker
    """

    url = 'https://query2.finance.yahoo.com/v10/finance/quoteSummary/' + symbol
    module_list = list(module_name_map.keys())
    params = {
        'modules': ','.join(module_list),
    }
    result = requests.get(url, params=params, headers=headers)
    if result.status_code != 200 and result.status_code != 404:
        result.raise_for_status()

    json_dict = result.json()
    if not "quoteSummary" in json_dict:
        return None
    if json_dict['quoteSummary']['result'] is None:
        return None
    module_dict = json_dict['quoteSummary']['result'][0]

    return module_dict


@multitasking.task
def quick_stats_request_threaded(request_idx, request_symbol_list, field_list):
    shared.response_dict[request_idx] = quick_stats_request(request_symbol_list, field_list)


def quick_stats_request(request_symbol_list, field_list):
    """
    Returns quick stats for up to 1000 tickers in one request. Only returns those tickers that are valid, thus can be
    used to validate tickers efficiently.
    """
    params = {
        'formatted': 'True',
        'symbols': ','.join(request_symbol_list),
        'fields': ','.join(field_list),
    }
    result = requests.get("https://query1.finance.yahoo.com/v6/finance/quote", params=params, headers=headers)
    if result.status_code != 200 and result.status_code != 404:
        result.raise_for_status()

    json_dict = result.json()
    if not "quoteResponse" in json_dict:
        return None
    data_list = json_dict['quoteResponse']['result']
    return data_list

sorry for all the issues just need some direction here i was able to fix prior

Crypto error @ Luna USD

I cant figure this error out.... its popping up no matter what I try and do so I may not understand the json or query.

16 :  USDC
17 :  USDT
18 :  AVAX
19 :  MATIC
20 :  SUSHI
21 :  ATOM
22 :  XTZ
23 :  LTC
24 :  BCH
25 :  LRC
26 :  AAVE
27 :  ALGO
28 :  BNB
29 :  SHIB
30 :  SCRT
31 :  ENS
32 :  RLY
33 :  CAKE
34 :  MANA
35 :  COTI
36 :  TRAC
37 :  FTM
38 :  LINK
39 :  VET
40 :  LUNA

Traceback (most recent call last):
  File "tasks_to_run.py", line 142, in <module>
    scrape_reddit_crypto.main()
  File "/code/scheduled_tasks/reddit/crypto/scrape_trending_posts.py", line 238, in main
    stats_table = get_financial_data(stats_table, crypto_id, symbol)
  File "/code/scheduled_tasks/reddit/crypto/scrape_trending_posts.py", line 154, in get_financial_data
    current_price = market_data["current_price"]["usd"]
KeyError: 'usd'

Incorrect data due to CUSIP change

On October 6th, XERS merged with SBBP and the combined company was relisted under a new CUSIP ID with a symbol of XERS. The SEC shows that the combined company had 234k FTDs on October 6th. In the data file provided from the SEC, the ticker symbol of the new company briefly becomes XERSZZZZ while the old has a ticker symbol of XERS for that day. From October 6th forward in the SEC data, XERS has a new CUSIP number. In my opinion you should try to separate the data based on CUSIP IDs as XERS before the merge is different from XERS after. I believe the FTDs listed under the new CUSIP are a combined total of FTDs from the old XERS and SBBP tickers with SBBP's FTDs converted to shares of the new XERS.

The data from the SEC contains the following:

20211006|98422E103|XERSZZZZ|243466|XERIS BIOPHARMA HLDGS INC COM |0.01
20211006|98422L107|XERS|1350|XERIS PHARMACEUTICALS INC|2.37
20211007|98422E103|XERS|9463|XERIS BIOPHARMA HLDGS INC COM |2.36

MYSQL interactive timeout

Hey I am using nohup to have the server run and read the output logs. but after a few hours I am getting this error.

4031 (HY000): The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.

I have found solutions to fix this but was wondering how you keep your connections active on the main site. What do you use to keep your docker server running?

Reverse Repo broken or Site maintance?

Ran the script and it failed which was intresting to me so I dug around the console and went to the supposed excel file link to get this.

https://websvcgatewayx2.frbny.org/autorates_tomo_external/services/v1_0/tomo/retrieveHistoricalExcel?f=01022014&t=09172021&ctt=true&&cta=true&ctm=true

Screen Shot 2021-09-17 at 8 19 00 PM

Nothing is working

Neither the site nor the Python app is working. At first, only the earnings calendar stopped functioning. So I built a scrape for it with Tradingview. Now nothing seems to be working. pythonanywhere.com had a major issue in early October which I believe is resolved.

This app may have to be rebooted on the server.
Alfred

How to run tasks on schedule

How do you recomend running tasks on a schedule as some of the scripts require you to call them from the root of the project directory.

Your insight here would be helpful as I can only think of hickory and cron jobs and those do not seem to work properly.

Options are working occasionally...

I have been trying to debug this one myself but im lost now I think it has to do with cache but not sure.

Apple's Options load 10% of the time and and will never load the default date url on APPL options for whatever reason. If I select an older date its random if it is working and future dates same thing for mmost tickers.

Now I noticed there is an option to claer oold option data in the tasks to run I havnt run that yet as I am unsure of the complete reason to do so.

I also notice at the bottom of some scripts like get earnings, there is a 'delete_old_earnings("2021-10-01")', Is there an area in update ticker_info that has something similar?

I am testing on ubuntu 18.04 and will be setting up windows machine shortly.

P.S. Now I have noticed a few things in the code the one that stands out the most is within the get_financial.py more than half the document is commented out. What was the reason you left this code here? Is it for error testing or fixing like in the helpers.py?

With get_stocks_summary.py

I face a big problem while running the task get_stocks_summary.py

I can get the original_df

Here're data


{'AOS': {'regularMarketOpen': '70.91', 'previousClose': '70.65', 'dayHigh': '71.62', 'fiftyTwoWeekHigh': '76.94', 'regularMarketDayLow': '70.44', 'fiftyTwoWeekLow': '55.41', 'regularMarketVolume': '724.29k', 'averageDailyVolume10Day': '1.43M', 'fiftyDayAverage': '68.61', 'twoHundredDayAverage': '68.79', 'trailingPE': '35.57', 'forwardPE': '18.38', 'marketCap': '10.65B', 'beta': '1.25', 'trailingAnnualDividendYield': '1.70%', 'trailingAnnualDividendRate': '1.20', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '122.83M', 'floatShares': '120.6M', 'shortRatio': '3.33', 'shortPercentOfFloat': '2.79%', 'trailingEps': '2.00', 'pegRatio': '2.31', 'enterpriseToRevenue': '2.74', 'netIncomeToCommon': '299.2M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Specialty Industrial Machinery', 'sector': 'Industrials', 'website': 'https://www.aosmith.com', 'longBusinessSummary': 'A. O. Smith Corporation manufactures and markets residential and commercial gas, heat pump and electric water heaters, boilers, tanks, and water treatment products in North America, China, Europe, and India. It operates through two segments, North America and Rest of World. The company offers water heaters for residences, restaurants, hotels, office buildings, laundries, car washes, and small businesses; commercial boilers for hospitals, schools, hotels, and other large commercial buildings, as well as residential boilers for homes, apartments, and condominiums; and water treatment products comprising point-of-entry water softeners, well water solutions, and whole-home water filtration products, on-the-go filtration bottles, point-of-use carbon, and reverse osmosis products for residences, restaurants, hotels, and offices. It also provides commercial water treatment and filtration product; expansion tanks, commercial solar water heating systems, swimming pool and spa heaters, and related products and parts; and electric wall-hung, gas tankless, combi-boiler, and heat pump and solar water heaters. The company offers its products primarily under the A. O. Smith, State, Lochinvar, and water softener brands. It distributes its products through independent wholesale plumbing distributors, as well as through retail channels consisting of hardware and home center chains, and manufacturer representative firms; and offers Aquasana branded products directly to consumers through e-commerce. A. O. Smith Corporation was founded in 1874 and is headquartered in Milwaukee, Wisconsin.', 'fullTimeEmployees': 12000, 'country': 'United States', 'longName': 'A. O. Smith Corporation', 'symbol': 'AOS', 'regularMarketPrice': '71.14', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.69%', 'regularMarketChange': '0.49', 'postMarketChangePercent': '0.37%', 'postMarketChange': '0.26', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'MMM': {'regularMarketOpen': '93.40', 'previousClose': '93.15', 'dayHigh': '93.40', 'fiftyTwoWeekHigh': '133.91', 'regularMarketDayLow': '91.27', 'fiftyTwoWeekLow': '85.35', 'regularMarketVolume': '3.46M', 'averageDailyVolume10Day': '3.15M', 'fiftyDayAverage': '95.02', 'twoHundredDayAverage': '102.14', 'trailingPE': 'N/A', 'forwardPE': '10.18', 'marketCap': '51.65B', 'beta': '0.98', 'trailingAnnualDividendYield': '6.43%', 'trailingAnnualDividendRate': '5.99', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '552.32M', 'floatShares': '551.81M', 'shortRatio': '2.75', 'shortPercentOfFloat': '2.06%', 'trailingEps': '-13.14', 'pegRatio': 'N/A', 'enterpriseToRevenue': '1.92', 'netIncomeToCommon': '-7.4B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Conglomerates', 'sector': 'Industrials', 'website': 'https://www.3m.com', 'longBusinessSummary': '3M Company provides diversified technology services in the United States and internationally. The company operates through four segments: Safety and Industrial; Transportation and Electronics; Health Care; and Consumer. The Safety and Industrial segment offers industrial abrasives and finishing for metalworking applications; autobody repair solutions; closure systems for personal hygiene products, masking, and packaging materials; electrical products and materials for construction and maintenance, power distribution, and electrical original equipment manufacturers; structural adhesives and tapes; respiratory, hearing, eye, and fall protection solutions; and natural and color-coated mineral granules for shingles. The Transportation and Electronics segment provides ceramic solutions; attachment tapes, films, sound, and temperature management for transportation vehicles; premium large format graphic films for advertising and fleet signage; light management films and electronics assembly solutions; packaging and interconnection solutions; and reflective signage for highway, and vehicle safety. The Healthcare segment offers health care procedure coding and reimbursement software; skin, wound care, and infection prevention products and solutions; dentistry and orthodontia solutions; and filtration and purification systems. The Consumer segment provides consumer bandages, braces, supports, and consumer respirators; cleaning products for the home; retail abrasives, paint accessories, car care DIY products, picture hanging, and consumer air quality solutions; and stationery products. It offers its products through e-commerce and traditional wholesalers, retailers, jobbers, distributors, and dealers. 3M Company was founded in 1902 and is headquartered in St. Paul, Minnesota.', 'fullTimeEmployees': 92000, 'country': 'United States', 'longName': '3M Company', 'symbol': 'MMM', 'regularMarketPrice': '92.32', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.89%', 'regularMarketChange': '-0.83', 'postMarketChangePercent': '-0.13%', 'postMarketChange': '-0.12', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'ABT': {'regularMarketOpen': '95.22', 'previousClose': '94.80', 'dayHigh': '95.73', 'fiftyTwoWeekHigh': '115.83', 'regularMarketDayLow': '93.79', 'fiftyTwoWeekLow': '89.67', 'regularMarketVolume': '4.4M', 'averageDailyVolume10Day': '6.35M', 'fiftyDayAverage': '97.37', 'twoHundredDayAverage': '103.78', 'trailingPE': '32.33', 'forwardPE': '21.68', 'marketCap': '164.93B', 'beta': '0.68', 'trailingAnnualDividendYield': '2.15%', 'trailingAnnualDividendRate': '2.04', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '1.74B', 'floatShares': '1.73B', 'shortRatio': '1.95', 'shortPercentOfFloat': '0.56%', 'trailingEps': '2.93', 'pegRatio': '-15.37', 'enterpriseToRevenue': '4.34', 'netIncomeToCommon': '5.14B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Medical Devices', 'sector': 'Healthcare', 'website': 'https://www.abbott.com', 'longBusinessSummary': "Abbott Laboratories, together with its subsidiaries, discovers, develops, manufactures, and sells health care products worldwide. It operates in four segments: Established Pharmaceutical Products, Diagnostic Products, Nutritional Products, and Medical Devices. The Established Pharmaceutical Products segment provides generic pharmaceuticals for the treatment of pancreatic exocrine insufficiency, irritable bowel syndrome or biliary spasm, intrahepatic cholestasis or depressive symptoms, gynecological disorder, hormone replacement therapy, dyslipidemia, hypertension, hypothyroidism, Mรฉniรจre's disease and vestibular vertigo, pain, fever, inflammation, and migraine, as well as provides anti-infective clarithromycin, influenza vaccine, and products to regulate physiological rhythm of the colon. The Diagnostic Products segment offers laboratory systems in the areas of immunoassay, clinical chemistry, hematology, and transfusion; molecular diagnostics systems that automate the extraction, purification, and preparation of DNA and RNA from patient samples, as well as detect and measure infectious agents; point of care systems; cartridges for testing blood; rapid diagnostics lateral flow testing products; molecular point-of-care testing for HIV, SARS-CoV-2, influenza A and B, RSV, and strep A; cardiometabolic test systems; drug and alcohol test, and remote patient monitoring and consumer self-test systems; and informatics and automation solutions for use in laboratories. The Nutritional Products segment provides pediatric and adult nutritional products. The Medical Devices segment offers rhythm management, electrophysiology, heart failure, vascular, and structural heart devices for the treatment of cardiovascular diseases; and diabetes care products, as well as neuromodulation devices for the management of chronic pain and movement disorders. Abbott Laboratories was founded in 1888 and is based in North Chicago, Illinois.", 'fullTimeEmployees': 115000, 'country': 'United States', 'longName': 'Abbott Laboratories', 'symbol': 'ABT', 'regularMarketPrice': '94.74', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.06%', 'regularMarketChange': '-0.06', 'postMarketChangePercent': '-0.85%', 'postMarketChange': '-0.81', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'ABBV': {'regularMarketOpen': '143.44', 'previousClose': '142.16', 'dayHigh': '143.88', 'fiftyTwoWeekHigh': '168.11', 'regularMarketDayLow': '141.92', 'fiftyTwoWeekLow': '130.96', 'regularMarketVolume': '5.29M', 'averageDailyVolume10Day': '6.32M', 'fiftyDayAverage': '148.01', 'twoHundredDayAverage': '147.82', 'trailingPE': '38.70', 'forwardPE': '13.59', 'marketCap': '250.76B', 'beta': '0.48', 'trailingAnnualDividendYield': '4.16%', 'trailingAnnualDividendRate': '5.92', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '1.77B', 'floatShares': '1.76B', 'shortRatio': '2.72', 'shortPercentOfFloat': '0.65%', 'trailingEps': '3.67', 'pegRatio': '-3.13', 'enterpriseToRevenue': '5.42', 'netIncomeToCommon': '6.47B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Drug Manufacturers - General', 'sector': 'Healthcare', 'website': 'https://www.abbvie.com', 'longBusinessSummary': "AbbVie Inc. discovers, develops, manufactures, and sells pharmaceuticals worldwide. The company offers Humira, a therapy administered as an injection for autoimmune, intestinal Behรงet's diseases, and pyoderma gangrenosum; Skyrizi to treat moderate to severe plaque psoriasis, psoriatic disease, and Crohn's disease; Rinvoq, a JAK inhibitor to treat rheumatoid and psoriatic arthritis, ankylosing spondylitis, atopic dermatitis, axial spondyloarthropathy, and ulcerative colitis; Imbruvica for the treatment of adult patients with blood cancers; and Venclexta/Venclyxto to treat hematological malignancies. It also provides facial injectables, plastics and regenerative medicine, body contouring, and skincare products; Vraylar for depressive disorder; Duopa and Duodopa to treat advanced Parkinson's disease; and Ubrelvy for the acute treatment of migraine with or without aura in adults; Qulipta for episodic migraine. In addition, the company offers Lumigan/Ganfort and Alphagan/Combigan for the reduction of elevated intraocular pressure(IOP) in patients with open angle glaucoma (OAG) or ocular hypertension; Restasis, a calcineurin inhibitor immunosuppressant indicated to increase tear production; and eye care products. Further, it provides Mavyret/Maviret to treat chronic hepatitis C virus (HCV) genotype 1-6 infection and HCV genotype 1 infection; Creon, a pancreatic enzyme therapy; Lupron to treat advanced prostate cancer, endometriosis and central precocious puberty, and patients with anemia caused by uterine fibroids; Linzess/Constella to treat irritable bowel syndrome with constipation and chronic idiopathic constipation; and Synthroid for hypothyroidism. It has collaborations with Calico Life Sciences LLC; REGENXBIO Inc.; I-Mab Biopharma; Genmab A/S; Janssen Biotech, Inc.; Genentech, Inc.; and California Institute for Biomedical Research (Calibr). The company was incorporated in 2012 and is headquartered in North Chicago, Illinois.", 'fullTimeEmployees': 50000, 'country': 'United States', 'longName': 'AbbVie Inc.', 'symbol': 'ABBV', 'regularMarketPrice': '142.03', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.09%', 'regularMarketChange': '-0.13', 'postMarketChangePercent': '0.25%', 'postMarketChange': '0.36', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'ACN': {'regularMarketOpen': '316.87', 'previousClose': '315.19', 'dayHigh': '319.46', 'fiftyTwoWeekHigh': '330.44', 'regularMarketDayLow': '315.38', 'fiftyTwoWeekLow': '242.80', 'regularMarketVolume': '1.81M', 'averageDailyVolume10Day': '1.97M', 'fiftyDayAverage': '310.57', 'twoHundredDayAverage': '295.59', 'trailingPE': '29.32', 'forwardPE': '23.91', 'marketCap': '199.47B', 'beta': '1.22', 'trailingAnnualDividendYield': '1.48%', 'trailingAnnualDividendRate': '4.65', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '627.61M', 'floatShares': '626.66M', 'shortRatio': '3.55', 'shortPercentOfFloat': '1.25%', 'trailingEps': '10.84', 'pegRatio': '2.77', 'enterpriseToRevenue': '3.03', 'netIncomeToCommon': '6.87B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Information Technology Services', 'sector': 'Technology', 'website': 'https://www.accenture.com', 'longBusinessSummary': 'Accenture plc, a professional services company, provides strategy and consulting, industry X, song, and technology and operation services worldwide. The company offers application services, including agile transformation, DevOps, application modernization, enterprise architecture, software and quality engineering, data management; intelligent automation comprising robotic process automation, natural language processing, and virtual agents; and application management services, as well as software engineering services; strategy and consulting services; data and analytics strategy, data discovery and augmentation, data management and beyond, data democratization, and industrialized solutions comprising turnkey analytics and artificial intelligence (AI) solutions; metaverse; and sustainability services. It also provides change management, HR transformation and delivery, organization strategy and design, talent strategy and development, and leadership and culture services; digital commerce; infrastructure services, including cloud infrastructure managed, cloud and data center, network, digital workplace, database platforms, service management, and cloud and infrastructure security services; data-enabled operating models; technology consulting and AI services; and technology consulting services. In addition, the company offers engineering and R&D digitization, smart connected products, product as-a-service enablement, capital projects, intelligent asset management, digital industrial workforce, and autonomous robotic systems; business process outsourcing; and services related to technology innovation. Further, it provides cloud, ecosystem, marketing, security, supply chain management, zero-based transformation, customer experience, finance consulting, mergers and acquisitions, and sustainability services. The company has a collaboration with Salesforce, Inc. to develop Salesforce Life Sciences Cloud. The company was founded in 1951 and is based in Dublin, Ireland.', 'fullTimeEmployees': 733000, 'country': 'Ireland', 'longName': 'Accenture plc', 'symbol': 'ACN', 'regularMarketPrice': '317.82', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.83%', 'regularMarketChange': '2.63', 'postMarketChangePercent': '-0.68%', 'postMarketChange': '-2.16', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'ADBE': {'regularMarketOpen': '589.00', 'previousClose': '585.20', 'dayHigh': '595.66', 'fiftyTwoWeekHigh': '595.66', 'regularMarketDayLow': '581.07', 'fiftyTwoWeekLow': '317.87', 'regularMarketVolume': '3.04M', 'averageDailyVolume10Day': '2.72M', 'fiftyDayAverage': '538.89', 'twoHundredDayAverage': '449.19', 'trailingPE': '50.76', 'forwardPE': '34.53', 'marketCap': '266.49B', 'beta': '1.32', 'trailingAnnualDividendYield': '0.00%', 'trailingAnnualDividendRate': '0.00', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '455.3M', 'floatShares': '454.31M', 'shortRatio': '1.59', 'shortPercentOfFloat': '1.08%', 'trailingEps': '11.53', 'pegRatio': '2.60', 'enterpriseToRevenue': '13.93', 'netIncomeToCommon': '5.12B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Software - Infrastructure', 'sector': 'Technology', 'website': 'https://www.adobe.com', 'longBusinessSummary': 'Adobe Inc., together with its subsidiaries, operates as a diversified software company worldwide. It operates through three segments: Digital Media, Digital Experience, and Publishing and Advertising. The Digital Media segment offers products, services, and solutions that enable individuals, teams, and enterprises to create, publish, and promote content; and Document Cloud, a unified cloud-based document services platform. Its flagship product is Creative Cloud, a subscription service that allows members to access its creative products. This segment serves content creators, students, workers, marketers, educators, enthusiasts, communicators, and consumers. The Digital Experience segment provides an integrated platform and set of applications and services that enable brands and businesses to create, manage, execute, measure, monetize, and optimize customer experiences from analytics to commerce. This segment serves marketers, advertisers, agencies, publishers, merchandisers, merchants, web analysts, data scientists, developers, and executives across the C-suite. The Publishing and Advertising segment offers products and services, such as e-learning solutions, technical document publishing, web conferencing, document and forms platform, web application development, and high-end printing, as well as Advertising Cloud offerings. The company offers its products and services directly to enterprise customers through its sales force and local field offices, as well as to end users through app stores and through its website at adobe.com. It also distributes products and services through a network of distributors, value-added resellers, systems integrators, software vendors and developers, retailers, and original equipment manufacturers. The company was formerly known as Adobe Systems Incorporated and changed its name to Adobe Inc. in October 2018. Adobe Inc. was founded in 1982 and is headquartered in San Jose, California.', 'fullTimeEmployees': 29239, 'country': 'United States', 'longName': 'Adobe Inc.', 'symbol': 'ADBE', 'regularMarketPrice': '585.31', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.02%', 'regularMarketChange': '0.11', 'postMarketChangePercent': '-0.27%', 'postMarketChange': '-1.60', 'preMarketChangePercent': '-0.26%', 'preMarketChange': '-1.51', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'ADM': {'regularMarketOpen': '72.06', 'previousClose': '72.11', 'dayHigh': '72.49', 'fiftyTwoWeekHigh': '98.12', 'regularMarketDayLow': '71.58', 'fiftyTwoWeekLow': '69.31', 'regularMarketVolume': '3.07M', 'averageDailyVolume10Day': '2.77M', 'fiftyDayAverage': '75.47', 'twoHundredDayAverage': '78.09', 'trailingPE': '10.07', 'forwardPE': '10.91', 'marketCap': '38.66B', 'beta': '0.84', 'trailingAnnualDividendYield': '2.43%', 'trailingAnnualDividendRate': '1.75', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '533.38M', 'floatShares': '529.77M', 'shortRatio': '1.83', 'shortPercentOfFloat': '1.01%', 'trailingEps': '7.15', 'pegRatio': '-17.10', 'enterpriseToRevenue': '0.48', 'netIncomeToCommon': '3.94B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Farm Products', 'sector': 'Consumer Defensive', 'website': 'https://www.adm.com', 'longBusinessSummary': "Archer-Daniels-Midland Company procures, transports, stores, processes, and merchandises agricultural commodities, products, and ingredients in the United States, Switzerland, the Cayman Islands, Brazil, Mexico, Canada, the United Kingdom, and internationally. The company operates in three segments: Ag Services and Oilseeds, Carbohydrate Solutions, and Nutrition. It originates, merchandises, stores, and transports agricultural raw materials, such as oilseeds and soft seeds. The company also engages in the agricultural commodity and feed product import, export, and distribution; and structured trade finance activities. In addition, it offers vegetable oils and protein meals; ingredients for the food, feed, energy, and industrial customers; crude vegetable oils, salad oils, margarine, shortening, and other food products; and partially refined oils to produce biodiesel and glycols for use in chemicals, paints, and other industrial products. Further, the company provides peanuts, peanut-derived ingredients, and cotton cellulose pulp; sweeteners, corn and wheat starches, syrup, glucose, wheat flour, and dextrose; alcohol and other food and animal feed ingredients; ethyl alcohol and ethanol; corn gluten feed and meal; distillers' grains; and citric acids. Additionally, the company provides natural flavors, flavor systems, natural colors, proteins, emulsifiers, soluble fiber, polyols, hydrocolloids, probiotics, prebiotics, enzymes, and botanical extracts; and other specialty food and feed ingredients; edible beans; formula feeds, and animal health and nutrition products; and contract and private label pet treats and foods. It also offers futures commission merchant; commodity brokerage services; cash margins and securities pledged to commodity exchange clearinghouses; and cash pledged as security under certain insurance arrangements. The company was founded in 1902 and is headquartered in Chicago, Illinois.", 'fullTimeEmployees': 40360, 'country': 'United States', 'longName': 'Archer-Daniels-Midland Company', 'symbol': 'ADM', 'regularMarketPrice': '72.03', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.11%', 'regularMarketChange': '-0.08', 'postMarketChangePercent': '-0.32%', 'postMarketChange': '-0.23', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'ADP': {'regularMarketOpen': '224.06', 'previousClose': '222.25', 'dayHigh': '226.47', 'fiftyTwoWeekHigh': '274.92', 'regularMarketDayLow': '223.28', 'fiftyTwoWeekLow': '201.46', 'regularMarketVolume': '1.31M', 'averageDailyVolume10Day': '2.48M', 'fiftyDayAverage': '240.61', 'twoHundredDayAverage': '228.59', 'trailingPE': '26.19', 'forwardPE': '22.41', 'marketCap': '92.74B', 'beta': '0.84', 'trailingAnnualDividendYield': '2.25%', 'trailingAnnualDividendRate': '5.00', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '411.3M', 'floatShares': '410.71M', 'shortRatio': '3.18', 'shortPercentOfFloat': '1.08%', 'trailingEps': '8.61', 'pegRatio': '1.99', 'enterpriseToRevenue': '5.19', 'netIncomeToCommon': '3.49B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Staffing & Employment Services', 'sector': 'Industrials', 'website': 'https://www.adp.com', 'longBusinessSummary': 'Automatic Data Processing, Inc. provides cloud-based human capital management solutions worldwide. It operates in two segments, Employer Services and Professional Employer Organization (PEO). The Employer Services segment offers strategic, cloud-based platforms, and human resources (HR) outsourcing solutions. Its offerings include payroll, benefits administration, talent management, HR management, workforce management, insurance, retirement, and compliance services, as well as integrated HCM solutions. The PEO Services segment provides HR outsourcing solutions to small and mid-sized businesses through a co-employment model. This segment offers benefits package, protection and compliance, talent engagement, expertise, comprehensive outsourcing, and recruitment process outsourcing services. The company was founded in 1949 and is headquartered in Roseland, New Jersey.', 'fullTimeEmployees': 63000, 'country': 'United States', 'longName': 'Automatic Data Processing, Inc.', 'symbol': 'ADP', 'regularMarketPrice': '225.48', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '1.45%', 'regularMarketChange': '3.23', 'postMarketChangePercent': '-0.13%', 'postMarketChange': '-0.30', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'AFL': {'regularMarketOpen': '80.96', 'previousClose': '80.98', 'dayHigh': '81.42', 'fiftyTwoWeekHigh': '82.72', 'regularMarketDayLow': '80.15', 'fiftyTwoWeekLow': '60.20', 'regularMarketVolume': '2.19M', 'averageDailyVolume10Day': '3.05M', 'fiftyDayAverage': '77.48', 'twoHundredDayAverage': '70.96', 'trailingPE': '11.06', 'forwardPE': '12.98', 'marketCap': '47.18B', 'beta': '0.89', 'trailingAnnualDividendYield': '2.05%', 'trailingAnnualDividendRate': '1.66', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '584.38M', 'floatShares': '579.22M', 'shortRatio': '4.94', 'shortPercentOfFloat': '2.02%', 'trailingEps': '7.26', 'pegRatio': '2.20', 'enterpriseToRevenue': '2.67', 'netIncomeToCommon': '4.37B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Insurance - Life', 'sector': 'Financial Services', 'website': 'https://www.aflac.com', 'longBusinessSummary': 'Aflac Incorporated, through its subsidiaries, provides supplemental health and life insurance products. It operates in two segments, Aflac Japan and Aflac U.S. The Aflac Japan segment offers cancer, medical, nursing care, work leave, GIFT, and whole and term life insurance products, as well as WAYS and child endowment plans under saving type insurance products in Japan. The Aflac U.S. segment provides cancer, accident, short-term disability, critical illness, hospital indemnity, dental, vision, long-term care and disability, and term and whole life insurance products in the United States. It sells its products through sales associates, brokers, independent corporate agencies, individual agencies, and affiliated corporate agencies. The company was founded in 1955 and is headquartered in Columbus, Georgia.', 'fullTimeEmployees': 12882, 'country': 'United States', 'longName': 'Aflac Incorporated', 'symbol': 'AFL', 'regularMarketPrice': '80.32', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.82%', 'regularMarketChange': '-0.66', 'postMarketChangePercent': '0.77%', 'postMarketChange': '0.62', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}, 

'AES': {'regularMarketOpen': '16.37', 'previousClose': '16.14', 'dayHigh': '16.40', 'fiftyTwoWeekHigh': '29.89', 'regularMarketDayLow': '16.08', 'fiftyTwoWeekLow': '11.43', 'regularMarketVolume': '6.16M', 'averageDailyVolume10Day': '10.66M', 'fiftyDayAverage': '15.42', 'twoHundredDayAverage': '20.71', 'trailingPE': 'N/A', 'forwardPE': '8.63', 'marketCap': '11B', 'beta': '1.05', 'trailingAnnualDividendYield': '4.11%', 'trailingAnnualDividendRate': '0.66', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '669.63M', 'floatShares': '666.64M', 'shortRatio': '1.83', 'shortPercentOfFloat': '4.26%', 'trailingEps': '-0.83', 'pegRatio': '1.13', 'enterpriseToRevenue': '3.19', 'netIncomeToCommon': '-560M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Utilities - Diversified', 'sector': 'Utilities', 'website': 'https://www.aes.com', 'longBusinessSummary': 'The AES Corporation operates as a diversified power generation and utility company. It owns and/or operates power plants to generate and sell power to customers, such as utilities, industrial users, and other intermediaries. The company also owns and/or operates utilities to generate or purchase, distribute, transmit, and sell electricity to end-user customers in the residential, commercial, industrial, and governmental sectors; and generates and sells electricity on the wholesale market. It uses a range of fuels and technologies to generate electricity, including coal, gas, hydro, wind, solar, and biomass; and renewables, such as energy storage and landfill gas. The company owns and/or operates a generation portfolio of approximately 32,326 megawatts. It has operations in the United States, Puerto Rico, El Salvador, Chile, Colombia, Argentina, Brazil, Mexico, Central America, the Caribbean, Europe, and Asia. The company was formerly known as Applied Energy Services, Inc. and changed its name to The AES Corporation in April 2000. The AES Corporation was incorporated in 1981 and is headquartered in Arlington, Virginia.', 'fullTimeEmployees': 9100, 'country': 'United States', 'longName': 'The AES Corporation', 'symbol': 'AES', 'regularMarketPrice': '16.22', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.50%', 'regularMarketChange': '0.08', 'postMarketChangePercent': '0.06%', 'postMarketChange': '0.01', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:14.940259'}}

'ALB': {'regularMarketOpen': '121.61', 'previousClose': '122.11', 'dayHigh': '123.40', 'fiftyTwoWeekHigh': '334.55', 'regularMarketDayLow': '117.86', 'fiftyTwoWeekLow': '116.35', 'regularMarketVolume': '2.38M', 'averageDailyVolume10Day': '4.01M', 'fiftyDayAverage': '162.50', 'twoHundredDayAverage': '205.79', 'trailingPE': '4.13', 'forwardPE': '5.56', 'marketCap': '13.97B', 'beta': '1.67', 'trailingAnnualDividendYield': '1.31%', 'trailingAnnualDividendRate': '1.60', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '117.35M', 'floatShares': '117.02M', 'shortRatio': '4.51', 'shortPercentOfFloat': '8.96%', 'trailingEps': '28.83', 'pegRatio': '0.42', 'enterpriseToRevenue': '1.66', 'netIncomeToCommon': '3.32B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Specialty Chemicals', 'sector': 'Basic Materials', 'website': 'https://www.albemarle.com', 'longBusinessSummary': 'Albemarle Corporation develops, manufactures, and markets engineered specialty chemicals worldwide. It operates through three segments: Lithium, Bromine, and Catalysts. The Lithium segment offers lithium compounds, including lithium carbonate, lithium hydroxide, lithium chloride, and lithium specialties and reagents, such as butyllithium and lithium aluminum hydride for use in lithium batteries for consumer electronics and electric vehicles, high performance greases, thermoplastic elastomers for car tires, rubber soles, plastic bottles, catalysts for chemical reactions, organic synthesis processes in the areas of steroid chemistry and vitamins, life sciences, pharmaceutical industry, and other markets. It also provides cesium products for the chemical and pharmaceutical industries; zirconium, barium, and titanium products for pyrotechnical applications that include airbag initiators; technical services for the handling and use of reactive lithium products; and lithium-containing by-products recycling services. The Bromine segment offers bromine and bromine-based products used in fire safety solutions; specialty chemicals, including elemental bromine, alkyl and inorganic bromides, brominated powdered activated carbon, and other bromine fine chemicals for use in chemical synthesis, oil and gas well drilling and completion fluids, mercury control, water purification, beef and poultry processing, and other industrial applications; and other specialty chemicals, such as tertiary amines for surfactants, biocides, disinfectants, and sanitizers. The Catalysts segment provides hydroprocessing, isomerization, and akylation catalysts; fluidized catalytic cracking catalysts and additives; and organometallics and curatives. The company serves the energy storage, petroleum refining, consumer electronics, construction, automotive, lubricants, pharmaceuticals, and crop protection markets. Albemarle Corporation was founded in 1887 and is headquartered in Charlotte, North Carolina.', 'fullTimeEmployees': 7400, 'country': 'United States', 'longName': 'Albemarle Corporation', 'symbol': 'ALB', 'regularMarketPrice': '119.01', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-2.54%', 'regularMarketChange': '-3.10', 'postMarketChangePercent': '0.32%', 'postMarketChange': '0.38', 'preMarketChangePercent': '-1.02%', 'preMarketChange': '-1.21', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:15.777495'}, 

'ALK': {'regularMarketOpen': '33.36', 'previousClose': '33.16', 'dayHigh': '34.18', 'fiftyTwoWeekHigh': '57.18', 'regularMarketDayLow': '33.35', 'fiftyTwoWeekLow': '30.75', 'regularMarketVolume': '2.55M', 'averageDailyVolume10Day': '2.31M', 'fiftyDayAverage': '36.10', 'twoHundredDayAverage': '44.08', 'trailingPE': '16.71', 'forwardPE': '6.40', 'marketCap': '4.34B', 'beta': '1.58', 'trailingAnnualDividendYield': '0.00%', 'trailingAnnualDividendRate': '0.00', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '128.05M', 'floatShares': '127.51M', 'shortRatio': '2.28', 'shortPercentOfFloat': '4.08%', 'trailingEps': '2.03', 'pegRatio': '0.50', 'enterpriseToRevenue': '0.58', 'netIncomeToCommon': '259M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Airlines', 'sector': 'Industrials', 'website': 'https://www.alaskaair.com', 'longBusinessSummary': 'Alaska Air Group, Inc., through its subsidiaries, operated airlines. The company operates through three segments: Mainline, Regional, and Horizon. The company offers scheduled air transportation services on Boeing and Airbus jet aircraft for passengers and cargo throughout the United States, and in parts of Mexico, Costa Rica, and Belize; and for passengers across a shorter distance network within the United States and Canada. Alaska Air Group, Inc. was founded in 1932 and is based in Seattle, Washington.', 'fullTimeEmployees': 23879, 'country': 'United States', 'longName': 'Alaska Air Group, Inc.', 'symbol': 'ALK', 'regularMarketPrice': '33.92', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '2.29%', 'regularMarketChange': '0.76', 'postMarketChangePercent': '0.00%', 'postMarketChange': '0.00', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:15.777495'}, 

'ARE': {'regularMarketOpen': '100.32', 'previousClose': '99.60', 'dayHigh': '100.99', 'fiftyTwoWeekHigh': '172.65', 'regularMarketDayLow': '98.75', 'fiftyTwoWeekLow': '90.73', 'regularMarketVolume': '791.75k', 'averageDailyVolume10Day': '1.34M', 'fiftyDayAverage': '103.17', 'twoHundredDayAverage': '121.56', 'trailingPE': '74.03', 'forwardPE': '20.48', 'marketCap': '17.65B', 'beta': '1.01', 'trailingAnnualDividendYield': '4.92%', 'trailingAnnualDividendRate': '4.90', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '173.78M', 'floatShares': '171.89M', 'shortRatio': '3.59', 'shortPercentOfFloat': '3.18%', 'trailingEps': '1.35', 'pegRatio': '-2.64', 'enterpriseToRevenue': '11.61', 'netIncomeToCommon': '236.16M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'REIT - Office', 'sector': 'Real Estate', 'website': 'https://www.are.com', 'longBusinessSummary': 'Alexandria Real Estate Equities, Inc. (NYSE: ARE), an S&P 500 company, is a best-in-class, mission-driven life science REIT making a positive and lasting impact on the world. As the pioneer of the life science real estate niche since our founding in 1994, Alexandria is the preeminent and longest-tenured owner, operator, and developer of collaborative life science, agtech, and advanced technology mega campuses in AAA innovation cluster locations, including Greater Boston, the San Francisco Bay Area, New York City, San Diego, Seattle, Maryland, and Research Triangle. The trusted partner to over 800 tenants, Alexandria has a total market capitalization of $28.3 billion and an asset base in North America of 75.1 million SF as of September 30, 2023, which includes 41.5 million RSF of operating properties and 5.6 million RSF of Class A/A+ properties undergoing construction, 8.9 million RSF of near-term and intermediate-term development and redevelopment projects, and 19.1 million SF of future development projects. Alexandria has a longstanding and proven track record of developing Class A/A+ properties clustered in life science, agtech, and advanced technology mega campuses that provide our innovative tenants with highly dynamic and collaborative environments that enhance their ability to successfully recruit and retain world-class talent and inspire productivity, efficiency, creativity, and success. Alexandria also provides strategic capital to transformative life science, agrifoodtech, climate innovation, and technology companies through our venture capital platform. We believe our unique business model and diligent underwriting ensure a high-quality and diverse tenant base that results in higher occupancy levels, longer lease terms, higher rental income, higher returns, and greater long-term asset value.', 'fullTimeEmployees': 593, 'country': 'United States', 'longName': 'Alexandria Real Estate Equities, Inc.', 'symbol': 'ARE', 'regularMarketPrice': '99.94', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.34%', 'regularMarketChange': '0.34', 'postMarketChangePercent': '-0.59%', 'postMarketChange': '-0.59', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:15.777495'}, 

'ALGN': {'regularMarketOpen': '197.67', 'previousClose': '197.52', 'dayHigh': '197.67', 'fiftyTwoWeekHigh': '413.20', 'regularMarketDayLow': '192.14', 'fiftyTwoWeekLow': '176.34', 'regularMarketVolume': '1.19M', 'averageDailyVolume10Day': '2.25M', 'fiftyDayAverage': '285.99', 'twoHundredDayAverage': '316.15', 'trailingPE': '41.02', 'forwardPE': '18.65', 'marketCap': '14.86B', 'beta': '1.58', 'trailingAnnualDividendYield': '0.00%', 'trailingAnnualDividendRate': '0.00', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '76.59M', 'floatShares': '72.01M', 'shortRatio': '3.63', 'shortPercentOfFloat': '3.54%', 'trailingEps': '4.72', 'pegRatio': '0.71', 'enterpriseToRevenue': '3.59', 'netIncomeToCommon': '362.81M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Medical Devices', 'sector': 'Healthcare', 'website': 'https://www.aligntech.com', 'longBusinessSummary': "Align Technology, Inc. designs, manufactures, and markets Invisalign clear aligners, and iTero intraoral scanners and services for orthodontists and general practitioner dentists in the United States, Switzerland, China, and internationally. It operates in two segments, Clear Aligner; and Imaging Systems and CAD/CAM Services (Systems and Services). The Clear Aligner segment offers comprehensive products, including Invisalign comprehensive package that addresses the orthodontic needs of teenage patients, such as mandibular advancement, compliance indicators, and compensation for tooth eruption; and Invisalign First Phase I and Invisalign First Comprehensive Phase 2 package for younger patients generally between the ages of seven and ten years, which is a mixture of primary/baby and permanent teeth. This segment's non-comprehensive products comprise Invisalign moderate, lite and express packages, and Invisalign go and Invisalign Go Plus; and non-case products that include retention products, Invisalign training, and adjusting tools used by dental professionals during the course of treatment. The Systems and Services segment offers iTero intraoral scanning system, a single hardware platform with software options for restorative or orthodontic procedures; restorative software for general practitioner dentists, prosthodontists, periodontists, and oral surgeons; and software for orthodontists for digital records storage, orthodontic diagnosis, and for the fabrication of printed models and retainers. This segment also provides Invisalign outcome simulator, a chair-side and cloud-based application for the iTero scanner; Invisalign progress assessment tool; and TimeLapse technology, which allows doctors or practitioners to compare a patient's historic 3D scans to the present-day scan, as well as subscription software, disposables, rentals, leases, and pay per scan services Align Technology, Inc. was incorporated in 1997 and is headquartered in Tempe, Arizona.", 'fullTimeEmployees': 22680, 'country': 'United States', 'longName': 'Align Technology, Inc.', 'symbol': 'ALGN', 'regularMarketPrice': '193.62', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-1.97%', 'regularMarketChange': '-3.90', 'postMarketChangePercent': '-0.01%', 'postMarketChange': '-0.01', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:15.777495'}, 

'ALLE': {'regularMarketOpen': '101.89', 'previousClose': '101.41', 'dayHigh': '102.74', 'fiftyTwoWeekHigh': '128.36', 'regularMarketDayLow': '101.52', 'fiftyTwoWeekLow': '95.94', 'regularMarketVolume': '804.14k', 'averageDailyVolume10Day': '967.46k', 'fiftyDayAverage': '103.89', 'twoHundredDayAverage': '109.95', 'trailingPE': '16.20', 'forwardPE': '14.38', 'marketCap': '8.99B', 'beta': '1.14', 'trailingAnnualDividendYield': '1.74%', 'trailingAnnualDividendRate': '1.76', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '87.79M', 'floatShares': '87.2M', 'shortRatio': '2.54', 'shortPercentOfFloat': '1.93%', 'trailingEps': '6.30', 'pegRatio': '1.38', 'enterpriseToRevenue': '2.97', 'netIncomeToCommon': '557.1M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Security & Protection Services', 'sector': 'Industrials', 'website': 'https://www.allegion.com', 'longBusinessSummary': 'Allegion plc manufactures and sells mechanical and electronic security products and solutions worldwide. The company offers door controls and systems and exit devices; locks, locksets, portable locks, and key systems and services; electronic security products and access control systems; time, attendance, and workforce productivity systems; doors, accessories, and other. It also provides services and software, which includes inspection, maintenance, and repair services for its automatic entrance solutions; and software as a service, including access control, IoT integration, and workforce management solutions, as well as aftermarket services, design and installation offerings, and locksmith services. The company sells its products and solutions to end-users in commercial, institutional, and residential facilities, including education, healthcare, government, hospitality, commercial office, and single and multi-family residential markets under the CISA, Interflex, LCN, Schlage, SimonsVoss, and Von Duprin brands. It sells its products and solutions through distribution and retail channels, such as specialty distribution, e-commerce, and wholesalers, as well as through various retail channels comprising do-it-yourself home improvement centers, on-line and e-commerce platforms, and small specialty showroom outlets. Allegion plc was incorporated in 2013 and is headquartered in Dublin, Ireland.', 'fullTimeEmployees': 12300, 'country': 'Ireland', 'longName': 'Allegion plc', 'symbol': 'ALLE', 'regularMarketPrice': '102.07', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.65%', 'regularMarketChange': '0.66', 'postMarketChangePercent': '-0.02%', 'postMarketChange': '-0.02', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:15.777495'}, 

'LNT': {'regularMarketOpen': '49.77', 'previousClose': '50.00', 'dayHigh': '49.78', 'fiftyTwoWeekHigh': '57.52', 'regularMarketDayLow': '48.64', 'fiftyTwoWeekLow': '45.15', 'regularMarketVolume': '2.1M', 'averageDailyVolume10Day': '1.8M', 'fiftyDayAverage': '49.78', 'twoHundredDayAverage': '52.22', 'trailingPE': '18.31', 'forwardPE': '16.16', 'marketCap': '12.77B', 'beta': '0.57', 'trailingAnnualDividendYield': '3.57%', 'trailingAnnualDividendRate': '1.78', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '255.18M', 'floatShares': '254.66M', 'shortRatio': '2.64', 'shortPercentOfFloat': '2.17%', 'trailingEps': '2.71', 'pegRatio': '2.48', 'enterpriseToRevenue': '5.28', 'netIncomeToCommon': '689M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Utilities - Regulated Electric', 'sector': 'Utilities', 'website': 'https://www.alliantenergy.com', 'longBusinessSummary': 'Alliant Energy Corporation operates as a utility holding company that provides regulated electricity and natural gas services. It operates through three segments: Utility Electric Operations, Utility Gas Operations, and Utility Other. The company, through its subsidiary, Interstate Power and Light Company (IPL), primarily generates and distributes electricity, and distributes and transports natural gas to retail customers in Iowa; sells electricity to wholesale customers in Minnesota, Illinois, and Iowa; and generates and distributes steam in Cedar Rapids, Iowa. Alliant Energy Corporation, through its other subsidiary, Wisconsin Power and Light Company (WPL), generates and distributes electricity, and distributes and transports natural gas to retail customers in Wisconsin; and sells electricity to wholesale customers in Wisconsin. It serves retail customers in the farming, agriculture, industrial manufacturing, chemical, and packaging and food industries. In addition, the company owns and operates a short-line rail freight service in Iowa; a barge, rail, and truck freight terminal on the Mississippi River; and a rail-served warehouse in Iowa, as well as offers freight brokerage services. Further, it holds interests in a natural gas-fired electric generating unit near Sheboygan Falls, Wisconsin; and a wind farm located in Oklahoma. Alliant Energy Corporation was incorporated in 1981 and is headquartered in Madison, Wisconsin.', 'fullTimeEmployees': 3129, 'country': 'United States', 'longName': 'Alliant Energy Corporation', 'symbol': 'LNT', 'regularMarketPrice': '49.61', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.78%', 'regularMarketChange': '-0.39', 'postMarketChangePercent': '0.00%', 'postMarketChange': '0.00', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:15.777495'}}

{'ALL': {'regularMarketOpen': '129.42', 'previousClose': '129.66', 'dayHigh': '129.78', 'fiftyTwoWeekHigh': '142.15', 'regularMarketDayLow': '127.93', 'fiftyTwoWeekLow': '100.57', 'regularMarketVolume': '735.79k', 'averageDailyVolume10Day': '2.18M', 'fiftyDayAverage': '116.63', 'twoHundredDayAverage': '115.70', 'trailingPE': 'N/A', 'forwardPE': '10.81', 'marketCap': '34.06B', 'beta': '0.52', 'trailingAnnualDividendYield': '2.71%', 'trailingAnnualDividendRate': '3.52', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '261.69M', 'floatShares': '260.07M', 'shortRatio': '2.36', 'shortPercentOfFloat': '1.50%', 'trailingEps': '-7.94', 'pegRatio': '-0.44', 'enterpriseToRevenue': '0.70', 'netIncomeToCommon': '-2.1B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Insurance - Property & Casualty', 'sector': 'Financial Services', 'website': 'https://www.allstate.com', 'longBusinessSummary': 'The Allstate Corporation, together with its subsidiaries, provides property and casualty, and other insurance products in the United States and Canada. The company operates through Allstate Protection; Protection Services; Allstate Health and Benefits; and Run-off Property-Liability segments. The Allstate Protection segment offers private passenger auto and homeowners insurance; other personal lines products; and commercial lines products under the Allstate and Encompass brand names. The Protection Services segment provides consumer product protection plans and related technical support for mobile phones, consumer electronics, furniture, and appliances; finance and insurance products, including vehicle service contracts, guaranteed asset protection waivers, road hazard tire and wheel, and paint and fabric protection; towing, jump-start, lockout, fuel delivery, and tire change services; device and mobile data collection services; data and analytic solutions using automotive telematics information; and identity protection services. This segment offers its products under various brands, including Allstate Protection Plans, Allstate Dealer Services, Allstate Roadside, Arity, Avail, and Allstate Identity Protection. The Allstate Health and Benefits segment provides life, accident, critical illness, short-term disability, and other health insurance products; stop-loss and fully insured group health products to employers; and short-term medical and medicare supplement insurance to individuals. The Run-off Property-Liability segment offers property and casualty insurance. It sells its products through agents, independent agents, call centers, retailers, voluntary benefits brokers, strategic partners, financial specialists, wholesale partners, and affinity groups, as well as through online and mobile applications. The company was founded in 1931 and is headquartered in Northbrook, Illinois.', 'fullTimeEmployees': 54000, 'country': 'United States', 'longName': 'The Allstate Corporation', 'symbol': 'ALL', 'regularMarketPrice': '129.28', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.29%', 'regularMarketChange': '-0.38', 'postMarketChangePercent': '0.29%', 'postMarketChange': '0.38', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:16.536731'}, 

'MO': {'regularMarketOpen': '40.49', 'previousClose': '40.51', 'dayHigh': '40.81', 'fiftyTwoWeekHigh': '48.11', 'regularMarketDayLow': '40.04', 'fiftyTwoWeekLow': '39.07', 'regularMarketVolume': '7.73M', 'averageDailyVolume10Day': '10.93M', 'fiftyDayAverage': '42.47', 'twoHundredDayAverage': '44.66', 'trailingPE': '8.20', 'forwardPE': '8.24', 'marketCap': '71.15B', 'beta': '0.69', 'trailingAnnualDividendYield': '9.38%', 'trailingAnnualDividendRate': '3.80', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '1.77B', 'floatShares': '1.77B', 'shortRatio': '1.85', 'shortPercentOfFloat': '0.88%', 'trailingEps': '4.90', 'pegRatio': '2.71', 'enterpriseToRevenue': '4.61', 'netIncomeToCommon': '8.74B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Tobacco', 'sector': 'Consumer Defensive', 'website': 'https://www.altria.com', 'longBusinessSummary': 'Altria Group, Inc., through its subsidiaries, manufactures and sells smokeable and oral tobacco products in the United States. The company provides cigarettes primarily under the Marlboro brand; cigars and pipe tobacco principally under the Black & Mild brand; moist smokeless tobacco products and snus products under the Copenhagen, Skoal, Red Seal, and Husky brands; and on! oral nicotine pouches. It sells its tobacco products primarily to wholesalers, including distributors; and large retail organizations, such as chain stores. The company was founded in 1822 and is headquartered in Richmond, Virginia.', 'fullTimeEmployees': 6300, 'country': 'United States', 'longName': 'Altria Group, Inc.', 'symbol': 'MO', 'regularMarketPrice': '40.20', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.77%', 'regularMarketChange': '-0.31', 'postMarketChangePercent': '0.17%', 'postMarketChange': '0.07', 'preMarketChangePercent': '0.20%', 'preMarketChange': '0.08', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:16.536731'}, 

'GOOGL': {'regularMarketOpen': '130.97', 'previousClose': '130.97', 'dayHigh': '132.21', 'fiftyTwoWeekHigh': '141.22', 'regularMarketDayLow': '130.78', 'fiftyTwoWeekLow': '84.86', 'regularMarketVolume': '25.66M', 'averageDailyVolume10Day': '37.36M', 'fiftyDayAverage': '133.78', 'twoHundredDayAverage': '117.78', 'trailingPE': '25.11', 'forwardPE': '19.62', 'marketCap': '1.65T', 'beta': '1.05', 'trailingAnnualDividendYield': '0.00%', 'trailingAnnualDividendRate': '0.00', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '5.92B', 'floatShares': '10.79B', 'shortRatio': '1.92', 'shortPercentOfFloat': '0.81%', 'trailingEps': '5.25', 'pegRatio': '1.26', 'enterpriseToRevenue': '5.25', 'netIncomeToCommon': '66.73B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Internet Content & Information', 'sector': 'Communication Services', 'website': 'https://www.abc.xyz', 'longBusinessSummary': 'Alphabet Inc. offers various products and platforms in the United States, Europe, the Middle East, Africa, the Asia-Pacific, Canada, and Latin America. It operates through Google Services, Google Cloud, and Other Bets segments. The Google Services segment provides products and services, including ads, Android, Chrome, hardware, Gmail, Google Drive, Google Maps, Google Photos, Google Play, Search, and YouTube. It is also involved in the sale of apps and in-app purchases and digital content in the Google Play store; and Fitbit wearable devices, Google Nest home products, Pixel phones, and other devices, as well as in the provision of YouTube non-advertising services. The Google Cloud segment offers infrastructure, cybersecurity, data, analytics, AI, and machine learning, and other services; Google Workspace that include cloud-based collaboration tools for enterprises, such as Gmail, Docs, Drive, Calendar, and Meet; and other services for enterprise customers. The Other Bets segment sells health technology and internet services. The company was founded in 1998 and is headquartered in Mountain View, California.', 'fullTimeEmployees': 182381, 'country': 'United States', 'longName': 'Alphabet Inc.', 'symbol': 'GOOGL', 'regularMarketPrice': '131.84', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.66%', 'regularMarketChange': '0.87', 'postMarketChangePercent': '-0.11%', 'postMarketChange': '-0.14', 'preMarketChangePercent': '-0.11%', 'preMarketChange': '-0.14', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:16.536731'}, 

'AMZN': {'regularMarketOpen': '142.97', 'previousClose': '142.71', 'dayHigh': '143.12', 'fiftyTwoWeekHigh': '145.86', 'regularMarketDayLow': '141.22', 'fiftyTwoWeekLow': '81.43', 'regularMarketVolume': '41.93M', 'averageDailyVolume10Day': '68.07M', 'fiftyDayAverage': '132.70', 'twoHundredDayAverage': '118.72', 'trailingPE': '72.86', 'forwardPE': '48.00', 'marketCap': '1.47T', 'beta': '1.17', 'trailingAnnualDividendYield': '0.00%', 'trailingAnnualDividendRate': '0.00', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '10.33B', 'floatShares': '9.06B', 'shortRatio': '1.45', 'shortPercentOfFloat': '1.02%', 'trailingEps': '1.95', 'pegRatio': '0.73', 'enterpriseToRevenue': '2.83', 'netIncomeToCommon': '20.08B', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Internet Retail', 'sector': 'Consumer Cyclical', 'website': 'https://www.aboutamazon.com', 'longBusinessSummary': "Amazon.com, Inc. engages in the retail sale of consumer products and subscriptions through online and physical stores in North America and internationally. It operates through three segments: North America, International, and Amazon Web Services (AWS). The company's products offered through its stores include merchandise and content purchased for resale; and products offered by third-party sellers. It also manufactures and sells electronic devices, including Kindle, Fire tablets, Fire TVs, Rings, Blink, eero, and Echo; and develops and produces media content. In addition, the company offers programs that enable sellers to sell their products in its stores; and programs that allow authors, musicians, filmmakers, Twitch streamers, skill and app developers, and others to publish and sell content. Further, it provides compute, storage, database, analytics, machine learning, and other services, as well as fulfillment, advertising, and digital content subscriptions. Additionally, the company offers Amazon Prime, a membership program. It serves consumers, sellers, developers, enterprises, content creators, and advertisers. The company was incorporated in 1994 and is headquartered in Seattle, Washington.", 'fullTimeEmployees': 1500000, 'country': 'United States', 'longName': 'Amazon.com, Inc.', 'symbol': 'AMZN', 'regularMarketPrice': '142.08', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '-0.44%', 'regularMarketChange': '-0.63', 'postMarketChangePercent': '-0.21%', 'postMarketChange': '-0.30', 'preMarketChangePercent': '-0.13%', 'preMarketChange': '-0.18', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:16.536731'}, 

'AMD': {'regularMarketOpen': '113.55', 'previousClose': '113.45', 'dayHigh': '115.32', 'fiftyTwoWeekHigh': '132.83', 'regularMarketDayLow': '112.88', 'fiftyTwoWeekLow': '60.05', 'regularMarketVolume': '41.4M', 'averageDailyVolume10Day': '65.45M', 'fiftyDayAverage': '103.80', 'twoHundredDayAverage': '100.92', 'trailingPE': '1,032.64', 'forwardPE': '28.90', 'marketCap': '183.5B', 'beta': '1.65', 'trailingAnnualDividendYield': '0.00%', 'trailingAnnualDividendRate': '0.00', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '1.62B', 'floatShares': '1.61B', 'shortRatio': '0.6', 'shortPercentOfFloat': '1.96%', 'trailingEps': '0.11', 'pegRatio': '4.63', 'enterpriseToRevenue': '8.17', 'netIncomeToCommon': '208M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Semiconductors', 'sector': 'Technology', 'website': 'https://www.amd.com', 'longBusinessSummary': 'Advanced Micro Devices, Inc. operates as a semiconductor company worldwide. It operates in four segments: Data Center, Client, Gaming, and Embedded segments. The company offers x86 microprocessors and graphics processing units (GPUs) as an accelerated processing unit, chipsets, data center, and professional GPUs; and embedded processors, and semi-custom system-on-chip (SoC) products, microprocessor and SoC development services and technology, data processing unites, field programmable gate arrays (FPGA), and adaptive SoC products. It also provides processors under the AMD Ryzen, AMD Ryzen PRO, Ryzen Threadripper, Ryzen Threadripper PRO, AMD Athlon, AMD Athlon PRO, and AMD PRO A-Series brand names; graphics under the AMD Radeon graphics and AMD Embedded Radeon graphics; and professional graphics under the AMD Radeon Pro graphics brand name. In addition, the company offers data center graphics under the Radeon Instinct and Radeon PRO V-series brands, as well as servers under the AMD Instinct accelerators brand; server microprocessors under the AMD EPYC brands; embedded processor solutions under the AMD Athlon, AMD Geode, AMD Ryzen, AMD EPYC, AMD R-Series, and G-Series brands; FPGA products under the Virtex-6, Virtex-7, Virtex UltraScale+, Kintex-7, Kintex UltraScale, Kintex UltraScale+, Artix-7, Artix UltraScale+, Spartan-6, and Spartan-7 brands; adaptive SOCs under the Zynq-7000, Zynq UltraScale+ MPSoC, Zynq UltraScale+ RFSoCs, Versal HBM, Versal Premium, Versal Prime, Versal AI Core, Versal AI Edge, Vitis, and Vivado brands; and compute and network acceleration board products under the Alveo brand. It serves original equipment and design manufacturers, public cloud service providers, system integrators, independent distributors, online and brick and mortar retailers, and add-in-board manufacturers through its direct sales force, independent distributors, and sales representatives. The company was incorporated in 1969 and is headquartered in Santa Clara, California.', 'fullTimeEmployees': 25000, 'country': 'United States', 'longName': 'Advanced Micro Devices, Inc.', 'symbol': 'AMD', 'regularMarketPrice': '113.59', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.12%', 'regularMarketChange': '0.14', 'postMarketChangePercent': '-0.51%', 'postMarketChange': '-0.58', 'preMarketChangePercent': '-0.27%', 'preMarketChange': '-0.31', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:16.536731'}, 

'AMCR': {'regularMarketOpen': '9.05', 'previousClose': '9.05', 'dayHigh': '9.09', 'fiftyTwoWeekHigh': '12.78', 'regularMarketDayLow': '9.03', 'fiftyTwoWeekLow': '8.45', 'regularMarketVolume': '5.3M', 'averageDailyVolume10Day': '10.82M', 'fiftyDayAverage': '9.06', 'twoHundredDayAverage': '10.16', 'trailingPE': '13.95', 'forwardPE': '12.26', 'marketCap': '13.12B', 'beta': '0.86', 'trailingAnnualDividendYield': '5.41%', 'trailingAnnualDividendRate': '0.49', 'totalAssets': 'N/A', 'navPrice': 'N/A', 'sharesOutstanding': '1.45B', 'floatShares': '1.44B', 'shortRatio': '3.84', 'shortPercentOfFloat': '1.83%', 'trailingEps': '0.65', 'pegRatio': '2.59', 'enterpriseToRevenue': '1.41', 'netIncomeToCommon': '962M', 'threeYearAverageReturn': 'N/A', 'fiveYearAverageReturn': 'N/A', 'industry': 'Packaging & Containers', 'sector': 'Consumer Cyclical', 'website': 'https://www.amcor.com', 'longBusinessSummary': 'Amcor plc develops, produces, and sells packaging products in Europe, North America, Latin America, Africa, and the Asia Pacific regions. The company operates through two segments, Flexibles and Rigid Packaging. The Flexibles segment provides flexible and film packaging products in the food and beverage, medical and pharmaceutical, fresh produce, snack food, personal care, and other industries. The Rigid Packaging segment offers rigid containers for various beverage and food products, including carbonated soft drinks, water, juices, sports drinks, milk-based beverages, spirits and beer, sauces, dressings, spreads, and personal care items; and plastic caps for various applications. The company sells its products through its direct sales force. Amcor plc was incorporated in 2018 and is headquartered in Zurich, Switzerland.', 'fullTimeEmployees': 41000, 'country': 'Switzerland', 'longName': 'Amcor plc', 'symbol': 'AMCR', 'regularMarketPrice': '9.07', 'quoteType': 'EQUITY', 'marketState': 'PRE', 'regularMarketChangePercent': '0.22%', 'regularMarketChange': '0.02', 'postMarketChangePercent': '0.33%', 'postMarketChange': '0.03', 'preMarketChangePercent': 'N/A', 'preMarketChange': 'N/A', 'holdings': 'N/A', 'sectorWeightings': 'N/A', 'next_update': '2023-11-09 09:22:16.536731'}}

But I can't continue to process the next step, it bring KeyError.


original_df["Current Price"] = pd.to_numeric(original_df["Current Price"], errors='coerce')
original_df["Prev Close"] = pd.to_numeric(original_df["Prev Close"], errors='coerce')
original_df["% Change"] = (original_df["Current Price"] - original_df["Prev Close"]) * 100 / \
                           original_df["Prev Close"]
original_df = original_df.reindex(symbol_list)
original_df.reset_index(inplace=True)

original_df = original_df[original_df["Market Cap"] != "N/A"]
original_df = original_df[["Symbol", "Market Cap", "% Change", "Sector", "Industry"]]

snp_out = pd.merge(snp, original_df, on="Symbol", how="left")
snp_out.to_csv(f"database/indices/snp500_heatmap.csv", index=False)

nasdaq_out = pd.merge(nasdaq, original_df, on="Symbol", how="left")
nasdaq_out.to_csv(f"database/indices/nasdaq100_heatmap.csv", index=False)

dia_out = pd.merge(dia, original_df, on="Symbol", how="left")
dia_out.to_csv(f"database/indices/dia_heatmap.csv", index=False)
 

about the py command

Hi,

I am using Anaconda environment, but it looks my system could not recognize the py command. If I use python setup/local.py, it gave me the error below:

Traceback (most recent call last):
File "/Users/vince/myspace/Stocksera/setup/local.py", line 4, in
subprocess.run("py scheduled_tasks/create_database.py")
File "/Users/vince/anaconda3/envs/st39/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/Users/vince/anaconda3/envs/st39/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Users/vince/anaconda3/envs/st39/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'py scheduled_tasks/create_database.py'

with helpers.py

There are one issue with the function get_ticker_list_stats(ticker_list), if the given ticker list is too long, you will have an error which told you to reduce the length, so I split it to 50 each, I think 100 is still ok. And remember with financialmodelingprep free tier, you only have limited number of request.

def get_ticker_list_stats(ticker_list):
    #print(ticker_list)
    chunks = split_list(ticker_list, 50)
    df = pd.DataFrame()
    for chunk in chunks:
       #AdDS4dPu2zBhQJV9pwTVEDMWFjFaNdxf
        try:
            r = requests.get(f"https://financialmodelingprep.com/api/v3/quote/{','.join(chunk)}?"
                         f"apikey={config_keys['FMP_KEY']}").json()
            #print(r)
            df = pd.concat([df,pd.DataFrame(r)])
        except:
            print()
    return df

Full list of data sources?

I've just found your repo today and was looking for a list of dates or list of data sources that Stocksera is currently setup to handle. If this has not been created, where could I start building that list and I will generate a PR.

What are some features or documentation that I could start contributing?

with fast_yahoo.py

I face an error while build and running this project with my old computer, luckily it still handle it well.
Here is my problem.

with latest fast_yahoo.py, you use v10 to get data, but we need to send crumb and cookie to make it work

here is my fix


import scheduled_tasks.reddit.stocks.shared as shared
import requests
import numbers
import multitasking as multitasking
import time
import json
import pandas as pd
from datetime import datetime, timedelta

apiBase = 'https://query2.finance.yahoo.com'
headers = { 
  "User-Agent": 
  "Mozilla/5.0 (Windows NT 6.1; Win64; x64)"
}

def getCredentials(cookieUrl='https://fc.yahoo.com', crumbUrl=apiBase+'/v1/test/getcrumb'):
  cookie = requests.get(cookieUrl).cookies
  crumb = requests.get(url=crumbUrl, cookies=cookie, headers=headers).text
  return {'cookie': cookie, 'crumb': crumb}
  
# Refer to https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working for more configs
# key is the website link, value is the part to extract
config = {'summaryDetail': ['regularMarketOpen', 'previousClose', 'dayHigh', 'fiftyTwoWeekHigh', 'regularMarketDayLow',
                            'fiftyTwoWeekLow', 'regularMarketVolume', 'averageDailyVolume10Day', 'fiftyDayAverage',
                            'twoHundredDayAverage', 'trailingPE', 'forwardPE', 'marketCap', 'beta',
                            'trailingAnnualDividendYield', 'trailingAnnualDividendRate', 'totalAssets', 'navPrice'],
          'defaultKeyStatistics': ['sharesOutstanding', 'floatShares', 'shortRatio', 'shortPercentOfFloat',
                                   'trailingEps', 'pegRatio', 'enterpriseToRevenue', 'netIncomeToCommon',
                                   'threeYearAverageReturn', 'fiveYearAverageReturn'],
          'summaryProfile': ['industry', 'sector', 'website', 'longBusinessSummary', 'fullTimeEmployees', 'country'],
          'price': ['longName', 'symbol', 'regularMarketPrice', 'quoteType', 'marketState',
                    'regularMarketChangePercent', 'regularMarketChange',
                    'postMarketChangePercent', 'postMarketChange', 'preMarketChangePercent', 'preMarketChange'],
          'topHoldings': ['holdings', 'sectorWeightings']}


def download_advanced_stats(symbol_list,  threads=True):
    """
    Downloads advanced yahoo stats for many tickers by doing one request per ticker.
    """
    num_requests = len(symbol_list)
    if threads:
        num_threads = min([num_requests, multitasking.cpu_count() * 2])
        multitasking.set_max_threads(num_threads)

    # get raw responses
    credentials = getCredentials()
    for request_idx, symbol in enumerate(symbol_list):
        if threads:
            get_ticker_stats_threaded(symbol, symbol, credentials)
        else:
            shared.response_dict[symbol] = get_ticker_stats(symbol, credentials)

    if threads:
        while len(shared.response_dict) < num_requests:
            time.sleep(0.01)

    # construct stats table from responses
    stats_table = []
    for symbol, retrieved_modules_dict in shared.response_dict.items():

        stats_list = [symbol]

        for module_name, stat_name_dict in config.items():
            retrieved_module_dict = None
            if retrieved_modules_dict is not None and module_name in retrieved_modules_dict:
                retrieved_module_dict = retrieved_modules_dict[module_name]

            if retrieved_module_dict is not None:
                for stat_name in stat_name_dict:
                    stat_val = 'N/A'
                    if stat_name in retrieved_module_dict:
                        stat = retrieved_module_dict[stat_name]
                        if isinstance(stat, dict):
                            if stat:  # only if non-empty otherwise N/A
                                stat_val = stat['fmt']
                        elif isinstance(stat, str) or isinstance(stat, numbers.Number) or isinstance(stat, list):
                            stat_val = stat
                    stats_list.append(stat_val)
            else:
                stats_list.extend(['N/A'] * len(stat_name_dict))

        stats_table.append(stats_list)

    # reset for future reuse
    shared.response_dict = {}

    columns = ['Symbol']
    for stat_name_dict in config.values():
        columns.extend(stat_name_dict)

    financial_data_df = pd.DataFrame(stats_table, columns=columns)
    financial_data_df["next_update"] = str(datetime.utcnow() + timedelta(seconds=600))
    financial_data_df.set_index('Symbol', inplace=True)
    financial_data_df = financial_data_df.to_json(orient="index").replace("\/", "/")
    financial_data_df = json.loads(financial_data_df)
    return financial_data_df


@multitasking.task
def get_ticker_stats_threaded(request_idx, symbol, credentials = getCredentials()):
    shared.response_dict[request_idx] = get_ticker_stats(symbol, credentials)

    

  
def get_ticker_stats(symbol, credentials=getCredentials()):
    """
    Returns advanced stats for one ticker
    """    
    url = f'https://query2.finance.yahoo.com/v10/finance/quoteSummary/{symbol}?' \
          f'modules=summaryDetail%2CdefaultKeyStatistics%2CsummaryProfile%2Cprice%2CtopHoldings&' \
          f'crumb={credentials["crumb"]}' \
          
    result = requests.get(url, cookies=credentials['cookie'], headers=headers)
    json_dict = result.json()
    if "quoteSummary" not in json_dict:
        return None
    if json_dict['quoteSummary']['result'] is None:
        return None
    module_dict = json_dict['quoteSummary']['result'][0]

    return module_dict

Docker image has Java error

scheduled_tasks_1  | Traceback (most recent call last):
scheduled_tasks_1  |   File "/usr/local/lib/python3.7/site-packages/tabula/io.py", line 85, in _run
scheduled_tasks_1  |     check=True,
scheduled_tasks_1  |   File "/usr/local/lib/python3.7/subprocess.py", line 488, in run
scheduled_tasks_1  |     with Popen(*popenargs, **kwargs) as process:
scheduled_tasks_1  |   File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
scheduled_tasks_1  |     restore_signals, start_new_session)
scheduled_tasks_1  |   File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
scheduled_tasks_1  |     raise child_exception_type(errno_num, err_msg, err_filename)
scheduled_tasks_1  | FileNotFoundError: [Errno 2] No such file or directory: 'java': 'java'
scheduled_tasks_1  | 
scheduled_tasks_1  | During handling of the above exception, another exception occurred:
scheduled_tasks_1  | 
scheduled_tasks_1  | Traceback (most recent call last):
scheduled_tasks_1  |   File "tasks_to_run.py", line 158, in <module>
scheduled_tasks_1  |     get_upcoming_events_date.main()
scheduled_tasks_1  |   File "/code/scheduled_tasks/economy/get_upcoming_events_date.py", line 69, in main
scheduled_tasks_1  |     retail_df = get_next_retail_sales_date()
scheduled_tasks_1  |   File "/code/scheduled_tasks/economy/get_upcoming_events_date.py", line 20, in get_next_retail_sales_date
scheduled_tasks_1  |     df = tabula.read_pdf(r"https://www.census.gov/retail/marts/www/martsdates.pdf", pages=1)[0]
scheduled_tasks_1  |   File "/usr/local/lib/python3.7/site-packages/tabula/io.py", line 322, in read_pdf
scheduled_tasks_1  |     output = _run(java_options, kwargs, path, encoding)
scheduled_tasks_1  |   File "/usr/local/lib/python3.7/site-packages/tabula/io.py", line 91, in _run
scheduled_tasks_1  |     raise JavaNotFoundError(JAVA_NOT_FOUND_ERROR)
scheduled_tasks_1  | tabula.errors.JavaNotFoundError: `java` command is not found from this Python process.Please ensure Java is installed and PATH is set for `java`

So I am trying to implement your Docker image on all of my OS, now that I have run through your commit 94 main script on Ubuntu 18.04 Server, Ubuntu 20.04 Desktop, Mac OS 11.14, and Windows 10.

I have been hitting this error on the final line after finishing the tasks file you compiled.

I was able to fix the initial nltk error without a bash connection into the image, I have only added the RUN after COPY . /code/

FROM python:3.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
RUN mkdir -p /usr/share/nltk_data \
    && cd /usr/share/nltk_data \
    && mkdir -p sentiment corpora \
    && curl https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/corpora/stopwords.zip > corpora/stopwords.zip \
    && curl https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/sentiment/vader_lexicon.zip > sentiment/vader_lexicon.zip

Do you have a different docker image with open-jdk you ran this one on ?

If not I will post my solution regardless.

IndexError with reddit crypto scrape

When trying to scrape crypto:

Traceback (most recent call last):
File "tasks_to_run.py", line 83, in
scrape_reddit_crypto.main()
File "/home/testadmin/Stocksera/scheduled_tasks/reddit/get_reddit_trending_crypto.py", line 317, in main

This line from get_reddit_trending_crypto.py gives an index error:

db.execute("SELECT * FROM cryptocurrency WHERE date_updated LIKE '%{}%'".format(dates[-1][0].split()[0], ))
IndexError: list index out of range

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.