roihala / stocker Goto Github PK

Add info to every alert message.
Tier, industry, and price

both can be found as keys in our collectors:
industry - profile, "primarySicCode" - drop the code number, we need the text
tier - securities, "tierDisplayName"
You can read them by using Collector.get_latest()

price - Use Site object to reach the url at:
https://backend.otcmarkets.com/otcapi/stock/trade/inside/{ticker}?symbol={ticker}
"previousClose"

• Make sure that the alert message remains user friendly

Investigate and fix

depends on #61

SECURITY DETAILS:
Outstanding Shares, Authorized Shares, Float, כל שינוי. מה היה לפי ומה עכשיו.

Currently alerts are treated separately, Unite them by ticker.
Notice that alerts from different collectors should be united also.

look at collect.py line 104

for consecutive sales on a given amount of time

/collector
put code
above security and symbol

It should connect with:
Bot - request the user for alert types,
diff_type - Support relevant types
collection - Support collections as types

think of how to support increase/decrease as well.

We want to alert when a company gets it's otciq certification
can be figured out by declaring "certification fields" and checking if they have changed for the first time

see #63 for more info

Let the user choose a collection to get history from in /history
#97 - blocker
#79

Consecutive sales periodically

Right now, if a value in an array is removed and the array is not empty, we will generate 2 diffs - one for a change and one for a remove.
Fix this.

e.g:
2020-11-04 15:36:01,641 INFO changes: [{'ticker': 'AGDY', 'date': '2020-11-04 13:36:00+00:00', 'changed_key': 'standardDirectorList.0.name', 'old': 'Erik Jablonowski', 'new': 'Gerald Agnew', 'diff_type': 'change'}, {'ticker': 'AGDY', 'date': '2020-11-04 13:36:00+00:00', 'changed_key': 'standardDirectorList', 'old': None, 'new': {'name': 'Erik Jablonowski', 'title': 'Director', 'boards': 'Independent Director'}, 'diff_type': 'add'}, {'ticker': 'AGDY', 'date': '2020-11-04 13:36:00+00:00', 'changed_key': 'officers.0.name', 'old': 'Joseph Amrams', 'new': 'Joseph Amram', 'diff_type': 'change'}]

old:
{"_id":{"$oid":"5fa2a8cf211617f2789cedbb"},"id":36334,"name":"Agri-Dynamics, Inc.","city":"Forest Hills","state":"NY","zip":"11375 ","countryId":"USA","country":"United States","address1":"107-23 71st Ave Suite# 193","website":"http://www.agri-dynamicsinc.com","phone":"515-329-0208","businessDesc":"AGRI-DYNAMICS, INC., (OTC MARKETS: AGDY) has been reinvented as a holding company like a SPAC that intends to acquire and joint venture with corporations that will help us increase revenue and generate valuable assets in various sectors. We specialize in the acquisition, investment and development of small or large businesses. The company's current research is in the energy, commodity, technology and agricultural sectors. We feel that today's market maintains a truly unprecedented opportunity for the acquisition in proven mining claims, high technology as well as valuable intellectual properties. Our aim is to grow our revenue, in part by assisting our subsidiaries to attain their revenue forecasts by making available additional financing, experienced personnel, distribution, marketing, and other services. AGRI-DYNAMICS, INC., is currently in search of valuable gold and silver mines that are close to production or will be in a reasonable amount of time. These assets can add immense value to the company quickly. AGRI-DYNAMICS, INC., (OTC Markets: AGDY) originally an agricultural and machine equipment Company, was founded in Iowa on December 8, 1961. For more information see the companies financial statements and related filings on OTCMarkets.com","stateOfIncorporation":"IA ","stateOfIncorporationName":"Iowa","countryOfIncorporation":"USA","countryOfIncorporationName":"United States","yearOfIncorporation":"1961","premierDirectorList":[{"name":"Joseph Amram","title":"Chairman","boards":""}],"standardDirectorList":[{"name":"Erik Jablonowski","title":"Director","boards":"Independent Director"}],"officers":[{"name":"Joseph Amrams","title":"CEO, President","boards":""}],"fiscalYearEnd":"6/30","filingCycle":"Q","edgarFilingStatus":"Alternative Reporting Standard","edgarFilingStatusId":"A","reportingStandard":"Dark: Alternative Reporting Standard","reportingStandardMin":"Dark or Defunct","isDark":true,"deregistered":true,"deregistrationDate":{"$numberLong":"1249876800000"},"is12g32b":false,"cik":"0000002817","isAlternativeReporting":true,"isBankThrift":false,"isNonBankRegulated":false,"isInternationalReporting":false,"isOtherReporting":false,"auditedStatusDisplay":"Not Available","auditStatus":"N","audited":false,"email":"[email protected]","numberOfEmployees":5,"numberOfEmployeesAsOf":{"$numberLong":"1544677200000"},"numberOfRecordShareholders":38,"numberOfRecordShareholdersDate":{"$numberLong":"1389070800000"},"primarySicCode":"1382 - Oil and gas exploration services","auditors":[],"investorRelationFirms":[],"legalCounsels":[],"investmentBanks":[],"corporateBrokers":[],"securities":[{"id":577,"tierId":22,"symbol":"AGDY","cusip":"008495202","compId":36334,"issueName":"AGRI-DYNAMICS INC","className":"Common Stock","primaryVenue":"OTC Link","categoryId":4,"categoryName":"No Information","isBB":false,"isPinkSheets":false,"isNoInfo":true,"hasLevel2":false,"isLevel2Entitled":false,"typeCode":"CS","typeName":"Common Stock","tierName":"Pink No Information","tierGroupId":"PS","tierCode":"PN","tierStartDate":{"$numberLong":"1404187200000"},"tierDisplayName":"Pink No Information","ratioAdr":1,"isAdr":false,"isGdr":false,"isTest":false,"isOtcQX":false,"isSponsored":false,"isPiggyBacked":true,"isCaveatEmptor":false,"canAccessBB":false,"noPar":false,"parValue":0.001,"outstandingShares":1900400000,"outstandingSharesAsOfDate":{"$numberLong":"1602216000000"},"authorizedShares":{"$numberLong":"52000000000"},"unlimitedAuthorizedShares":false,"authorizedSharesAsOfDate":{"$numberLong":"1602216000000"},"publicFloat":9423157,"publicFloatAsOfDate":{"$numberLong":"1374120000000"},"isExchangeQualified":false,"shortInterest":7190,"shortInterestChange":0,"shortInterestDate":{"$numberLong":"1602734400000"},"restrictedShares":1880000000,"restrictedSharesAsOfDate":{"$numberLong":"1602216000000"},"unrestrictedShares":20400000,"unrestrictedSharesAsOfDate":{"$numberLong":"1602216000000"},"sigFailDeliver":false,"numOfRecordShareholders":38,"numOfRecordShareholdersDate":{"$numberLong":"1389070800000"},"transferAgents":[{"id":2350,"type":"CS","name":"Action Stock Transfer Corporation","phone":"801-274-1088","website":"http://www.actionstocktransfer.com","email":"[email protected]","countryId":"USA","country":"United States","address1":"2469 E. Fort Union Blvd","address2":"Suite 214","city":"Salt Lake City","stateId":"UT","zip":"84121","roles":["Transfer Agent"],"isPublic":true,"hasLogo":false,"isGoodStanding":true,"isProhibited":false,"isQuestionable":false,"isAttorney":false,"isSponsor":false,"public":true}],"notes":["Capital Change=11-71 shares decreased by 1 for 5 reverse split."],"isUnsolicited":false,"showTrustedLogoForAuthorizedShares":true,"showTrustedLogoForOutstandingShares":true,"showTrustedLogoForRestrictedShares":true,"showTrustedLogoForUnrestrictedShares":true,"showTrustedLogoForDTCShares":false}],"otherSecurities":[],"estimatedMarketCap":12162560,"estimatedMarketCapAsOfDate":{"$numberLong":"1604379600000"},"blankCheck":false,"blindPool":false,"spac":false,"hasLatestFiling":true,"latestFilingType":"Interim Financial Report","latestFilingDate":{"$numberLong":"1388466000000"},"latestFilingUrl":"/company/financial-report/116399/content","tierGroup":"PS","tierCode":"PN","tierStartDate":{"$numberLong":"1404187200000"},"hasLogo":true,"companyLogoUrl":"/company/logo/AGDY","isCaveatEmptor":false,"otcAward":{"symbol":"AGDY","best50":false},"indexStatuses":[],"unableToContact":false,"isShell":false,"isBankrupt":false,"isProfileVerified":true,"profileVerifiedAsOfDate":"6/02/2020","venue":"OTC Link","isUnsolicited":false,"ticker":"AGDY","date":"2020-11-04 13:12:46+00:00"}

new:
{'id': 36334, 'name': 'Agri-Dynamics, Inc.', 'city': 'Forest Hills', 'state': 'NY', 'zip': '11375 ', 'countryId': 'USA', 'country': 'United States', 'address1': '107-23 71st Ave Suite# 193', 'website': 'http://www.agri-dynamicsinc.com', 'phone': '515-329-0208', 'businessDesc': "AGRI-DYNAMICS, INC., (OTC MARKETS: AGDY) has been reinvented as a holding company like a SPAC that intends to acquire and joint venture with corporations that will help us increase revenue and generate valuable assets in various sectors. We specialize in the acquisition, investment and development of small or large businesses. The company's current research is in the energy, commodity, technology and agricultural sectors. We feel that today's market maintains a truly unprecedented opportunity for the acquisition in proven mining claims, high technology as well as valuable intellectual properties. Our aim is to grow our revenue, in part by assisting our subsidiaries to attain their revenue forecasts by making available additional financing, experienced personnel, distribution, marketing, and other services. AGRI-DYNAMICS, INC., is currently in search of valuable gold and silver mines that are close to production or will be in a reasonable amount of time. These assets can add immense value to the company quickly. AGRI-DYNAMICS, INC., (OTC Markets: AGDY) originally an agricultural and machine equipment Company, was founded in Iowa on December 8, 1961. For more information see the companies financial statements and related filings on OTCMarkets.com", 'stateOfIncorporation': 'IA ', 'stateOfIncorporationName': 'Iowa', 'countryOfIncorporation': 'USA', 'countryOfIncorporationName': 'United States', 'yearOfIncorporation': '1961', 'premierDirectorList': [{'name': 'Joseph Amram', 'title': 'Chairman', 'boards': ''}], 'standardDirectorList': [{'name': 'Gerald Agnew', 'title': 'Director', 'boards': 'Independent Director'}, {'name': 'Erik Jablonowski', 'title': 'Director', 'boards': 'Independent Director'}], 'officers': [{'name': 'Joseph Amram', 'title': 'CEO, President', 'boards': ''}], 'fiscalYearEnd': '6/30', 'filingCycle': 'Q', 'edgarFilingStatus': 'Alternative Reporting Standard', 'edgarFilingStatusId': 'A', 'reportingStandard': 'Dark: Alternative Reporting Standard', 'reportingStandardMin': 'Dark or Defunct', 'isDark': True, 'deregistered': True, 'deregistrationDate': 1249876800000, 'is12g32b': False, 'cik': '0000002817', 'isAlternativeReporting': True, 'isBankThrift': False, 'isNonBankRegulated': False, 'isInternationalReporting': False, 'isOtherReporting': False, 'auditedStatusDisplay': 'Not Available', 'auditStatus': 'N', 'audited': False, 'email': '[email protected]', 'numberOfEmployees': 5, 'numberOfEmployeesAsOf': 1544677200000, 'numberOfRecordShareholders': 38, 'numberOfRecordShareholdersDate': 1389070800000, 'primarySicCode': '1382 - Oil and gas exploration services', 'auditors': [], 'investorRelationFirms': [], 'legalCounsels': [], 'investmentBanks': [], 'corporateBrokers': [], 'securities': [{'id': 577, 'tierId': 22, 'symbol': 'AGDY', 'cusip': '008495202', 'compId': 36334, 'issueName': 'AGRI-DYNAMICS INC', 'className': 'Common Stock', 'primaryVenue': 'OTC Link', 'categoryId': 4, 'categoryName': 'No Information', 'isBB': False, 'isPinkSheets': False, 'isNoInfo': True, 'hasLevel2': False, 'isLevel2Entitled': False, 'typeCode': 'CS', 'typeName': 'Common Stock', 'tierName': 'Pink No Information', 'tierGroupId': 'PS', 'tierCode': 'PN', 'tierStartDate': 1404187200000, 'tierDisplayName': 'Pink No Information', 'ratioAdr': 1.0, 'isAdr': False, 'isGdr': False, 'isTest': False, 'isOtcQX': False, 'isSponsored': False, 'isPiggyBacked': True, 'isCaveatEmptor': False, 'canAccessBB': False, 'noPar': False, 'parValue': 0.001, 'outstandingShares': 1900400000, 'outstandingSharesAsOfDate': 1602216000000, 'authorizedShares': 52000000000, 'unlimitedAuthorizedShares': False, 'authorizedSharesAsOfDate': 1602216000000, 'publicFloat': 9423157, 'publicFloatAsOfDate': 1374120000000, 'isExchangeQualified': False, 'shortInterest': 7190, 'shortInterestChange': 0.0, 'shortInterestDate': 1602734400000, 'restrictedShares': 1880000000, 'restrictedSharesAsOfDate': 1602216000000, 'unrestrictedShares': 20400000, 'unrestrictedSharesAsOfDate': 1602216000000, 'sigFailDeliver': False, 'numOfRecordShareholders': 38, 'numOfRecordShareholdersDate': 1389070800000, 'transferAgents': [{'id': 2350, 'type': 'CS', 'name': 'Action Stock Transfer Corporation', 'phone': '801-274-1088', 'website': 'http://www.actionstocktransfer.com', 'email': '[email protected]', 'countryId': 'USA', 'country': 'United States', 'address1': '2469 E. Fort Union Blvd', 'address2': 'Suite 214', 'city': 'Salt Lake City', 'stateId': 'UT', 'zip': '84121', 'roles': ['Transfer Agent'], 'isPublic': True, 'hasLogo': False, 'isGoodStanding': True, 'isProhibited': False, 'isQuestionable': False, 'isAttorney': False, 'isSponsor': False, 'public': True}], 'notes': ['Capital Change=11-71 shares decreased by 1 for 5 reverse split.'], 'isUnsolicited': False, 'showTrustedLogoForAuthorizedShares': True, 'showTrustedLogoForOutstandingShares': True, 'showTrustedLogoForRestrictedShares': True, 'showTrustedLogoForUnrestrictedShares': True, 'showTrustedLogoForDTCShares': False}], 'otherSecurities': [], 'estimatedMarketCap': 12162560.0, 'estimatedMarketCapAsOfDate': 1604379600000, 'blankCheck': False, 'blindPool': False, 'spac': False, 'hasLatestFiling': True, 'latestFilingType': 'Interim Financial Report', 'latestFilingDate': 1388466000000, 'latestFilingUrl': '/company/financial-report/116399/content', 'tierGroup': 'PS', 'tierCode': 'PN', 'tierStartDate': 1404187200000, 'hasLogo': True, 'companyLogoUrl': '/company/logo/AGDY', 'isCaveatEmptor': False, 'otcAward': {'symbol': 'AGDY', 'best50': False}, 'indexStatuses': [], 'unableToContact': False, 'isShell': False, 'isBankrupt': False, 'isProfileVerified': True, 'profileVerifiedAsOfDate': '6/02/2020', 'venue': 'OTC Link', 'isUnsolicited': False, 'ticker': 'AGDY', 'date': '2020-11-04 13:36:00+00:00'}

State the filing type in the alert message

Add unique emoji for each alert type

By default, registered user will have USER permissions.
Insert by hand different permissions for our team and Penny

alert when more than two shepherds entered a certain stock

• Filter duplications on write.
• Filter retro duplications

כתובות, דירקטורים, טלפונים אלו השינויים ההכי קריטיים

Company profile:

DESCRIPTION כל שינוי. לראות מה היה לפני ומה עכשיו.

COMPANY OFFICERS כל שינוי. לראות מה היה לפני ומה עכשיו.

COMPANY DIRECTORS כל שינוי. לראות מה היה לפני ומה עכשיו.

Adress EMAIL, phone number, website

in securities collection, keys:
outstandingShares
authorizedShares

When the diff is unusual, 10% difference (up or down) - alert the change

• Alert the diff followed by the precentage (ctrl+f otciq for an example)

Right now it's not checking which diffs were alerted.
Make it work perfectly

Those alerts should be united to one alert.

We don't know if it's necessary

Add as many tickers as possible to tickers.csv
A place to look at:
https://www.otcmarkets.com/research/stock-screener

Make sure that all of the logs go to the same log.

Here I'll write some stocks that came up with stocker and to collect a hit rate

See if there's any important fields in there:
https://developer.edgar-online.com/

• Note that the data is refreshed at 3am EST, Monday through Friday. There is no new data posted on the weekends or on national holidays.
  https://www.otcmarkets.com/market-activity/corporate-actions - go over every fucking option and scrape if needed
  https://sec.report/otc/financial-report
  https://backend.otcmarkets.com/otcapi/company/dns/tier/news?tierGroups=ALL&page=1&pageSize=50
  https://backend.otcmarkets.com/otcapi/company/financial-report/
  https://backend.otcmarkets.com/otcapi/company/sec-filings/
  https://backend.otcmarkets.com/otcapi/company/HBRM/financial-report?symbol=HBRM&page=1&pageSize=10&statusId=A&sortOn=releaseDate&sortDir=DESC

alert on "big" site updates for companies sites

Add them to the framework by using polymorphism

Filter out the following alerts:

• businessDesc encoding problems, as you can see in the comments
• Phone number format changes, such as:
  🚨 Detected change on ABCE:
  520-777-0511 ⏩⏩⏩ (520) 777-0511
  +1 (703) 273-0383 ⏩⏩⏩ (703) 273-0383
  8133210440 ⏩⏩⏩ +1 (813) 723-1206

Found this one
https://t.me/InviteMemberBot

manage and create monthly subscription

exemple

🚨 Detected change on RTON:
unrestrictedShares has changed:
2893145141 ⏩⏩⏩ 3044573712

to

🚨 Detected change on RTON:
unrestrictedShares has changed:
2,893,145,141 --------> 3,044,573,712

They should be listed in each server's crontab
Inherit from Runnable and follow the same API as collect.py, client.py etc...

If a company has accessed it's otciq account for the first time in a long while we want to be the first to know about that.

Look at a set of parameters that could only change with otciq access
alert with a unique diff_type and render it in the bot

🚨 Detected change on GRPS:
tierDisplayName has changed:
Pink Limited Information ⏩⏩⏩ Pink No Information

Add a sympathy finder for the bot with the /sympathy command.

Right now, we're sampling tickers according to the tickers listed in csv/tickers.csv
We want to dynamically create a list of tickers (based on tickers.csv)

The following command returns a json containing information about given stocks.
In this example I used SGIP, SOAN, EXMT tickers - and as you can see they are listed in the request.
Note that there is no key for un-tradeable tickers.

curl -X GET --header "Authorization: " "https://api.tdameritrade.com/v1/marketdata/quotes?apikey=7GF2GJN60ZAP2F9E4V7IXDMFZD5M6GQR&symbol=SGIP%2CSOAN%2CEXMT"

• Make sure that you don't exceed the maximum count of tickers in this API

Use this example to do the following:

1. Base on the biggest tickers.csv file in git history (the one with the most tickers)

2. Generate a list of tradeable tickers

3. Set a default price range of 0.0001 - 0.05
   (The response contains "lastPrice")

4. Do it in Runnable.extract_tickers

Make sure that no ticker raises this exception. Report problematic tickers

Company profile:

DESCRIPTION כל שינוי. לראות מה היה לפני ומה עכשיו.

COMPANY OFFICERS כל שינוי. לראות מה היה לפני ומה עכשיו.

COMPANY DIRECTORS כל שינוי. לראות מה היה לפני ומה עכשיו.

Adress EMAIL, phone number, website

In CollectorBase.get_sorted_history() we're trying to drop columns that contains the same value for each row but nested columns fuck us (columns of lists, dicts, etc...)

CollectorBase.__apply_filters is supposed to do this in line 91 but it doesn't work properly.

Use client.py in order to trigger CollectorBase.get_sorted_history() Create an instance of Profile() for convenience

It isn't necessary, add a @Property of name instead

This is not readable. Find a way to make it human friendly

Filter alerts that represent bad things by building an hierarchy.
e.g: pink current -> pink limited is bad and therefore should be removed.

Note that hierarchy already exist as a property.

add commas in the right locations for big numbers. e.g:
2000000 - 2,000,000

הנה משהו חיוני:
היסטוריית שינויים ב S/S
כדאי שיהיה אפשר לדעת בכמה זה גדל תוך כמה זמן (פשוט להוסיף תאריכי עדכון)

PennyWhale
כדאי שיהיה אפשר לדעת בכמה זה גדל תוך כמה זמן (פשוט להוסיף תאריכי עדכון)

Nati
מה הכוונה ?
כי השרת מבצע בדיקה יומית
אתה מתכוונת שאתה מבצע חיפוש לאחור ?

PennyWhale
HQGE

PennyWhale
🚨 Stocker alertsNov 20, 2020 2:45:34 AM
🚨 Detected change on HQGE:
unrestrictedShares has changed:
2621633000 ⏩⏩⏩ 3021633000

PennyWhale
למתי "2621633000 " היה נכון?
לאיזה תאריך ?

PennyWhale
למתי "2621633000 " היה נכון?

Nati
במקרה הזה התאריך היה 20 לנובמבר אבל לא היו לנו דגימות קודמות ככה שאין לי מושג מתי פעם לפני עודכן

PennyWhale
אבל אני אומר לעתיד
שהמידע הזה יהיה נגיש בתוכנה

Nati
אוקי סבבה יבדק

Find out costs and barriers

Register stocker and CI/CD as a service on the production server:
commands:
stocker
python3 ~/lib/stocker/alert.py --uri mongodb://admin:admin123@localhost:27017/stocker

CI/CD
webhook -hooks ~/webhooks/hooks.json -ip "51.91.11.169" -verbose

• Highlight diffs on long strings (like git does) and send as picture:
  https://code-maven.com/create-images-with-python-pil-pillow

Support a price range of alerts through the bot