stocker's Issues
Daily report
כתובות, דירקטורים, טלפונים אלו השינויים ההכי קריטיים
Company profile:
DESCRIPTION כל שינוי. לראות מה היה לפני ומה עכשיו.
COMPANY OFFICERS כל שינוי. לראות מה היה לפני ומה עכשיו.
COMPANY DIRECTORS כל שינוי. לראות מה היה לפני ומה עכשיו.
Adress EMAIL, phone number, website
LTNC is in tickers.csv but not alerted
Investigate and fix
Remove token and uri from Service.py
They should be listed in each server's crontab
Inherit from Runnable and follow the same API as collect.py, client.py etc...
Expand the processing power
Find out costs and barriers
Decide if /history is important
We don't know if it's necessary
Hierarchy filter
Filter alerts that represent bad things by building an hierarchy.
e.g: pink current -> pink limited is bad and therefore should be removed.
Note that hierarchy already exist as a property.
Synchronize securities diff alert
Right now it's not checking which diffs were alerted.
Make it work perfectly
Unite alerts by ticker
Currently alerts are treated separately, Unite them by ticker.
Notice that alerts from different collectors should be united also.
look at collect.py line 104
Remove 'name' from CollectorBase __init__
It isn't necessary, add a @Property of name instead
Add emojies to messages
Add unique emoji for each alert type
Get history by collection
Add collectors
See if there's any important fields in there:
https://developer.edgar-online.com/
- Note that the data is refreshed at 3am EST, Monday through Friday. There is no new data posted on the weekends or on national holidays.
https://www.otcmarkets.com/market-activity/corporate-actions - go over every fucking option and scrape if needed
https://sec.report/otc/financial-report
https://backend.otcmarkets.com/otcapi/company/dns/tier/news?tierGroups=ALL&page=1&pageSize=50
https://backend.otcmarkets.com/otcapi/company/financial-report/
https://backend.otcmarkets.com/otcapi/company/sec-filings/
https://backend.otcmarkets.com/otcapi/company/HBRM/financial-report?symbol=HBRM&page=1&pageSize=10&statusId=A&sortOn=releaseDate&sortDir=DESC
alert on "big" site updates for companies sites
price range of alerts
Support a price range of alerts through the bot
Handle nested columns in get_history
In CollectorBase.get_sorted_history() we're trying to drop columns that contains the same value for each row but nested columns fuck us (columns of lists, dicts, etc...)
CollectorBase.__apply_filters is supposed to do this in line 91 but it doesn't work properly.
Use client.py in order to trigger CollectorBase.get_sorted_history() Create an instance of Profile() for convenience
Add Payment to Telegram BOt
Found this one
https://t.me/InviteMemberBot
manage and create monthly subscription
Create a framework of alert types
It should connect with:
Bot - request the user for alert types,
diff_type - Support relevant types
collection - Support collections as types
think of how to support increase/decrease as well.
Fix "unsupported ticker"
Make sure that no ticker raises this exception. Report problematic tickers
Large bid on level 2
Filter duplications on mongo
- Filter duplications on write.
- Filter retro duplications
Abbriviate client.py as a telegram bot
Humanize numbers in diffs
add commas in the right locations for big numbers. e.g:
2000000 - 2,000,000
Nicer diff
get_sorted_history should filter keys by FILTER_KEYS
depends on #61
Alert Securities on a daily basis
Humanize diffs
- Highlight diffs on long strings (like git does) and send as picture:
https://code-maven.com/create-images-with-python-pil-pillow
/sympathy command
Add a sympathy finder for the bot with the /sympathy command.
Fix hierarchy
🚨 Detected change on GRPS:
tierDisplayName has changed:
Pink Limited Information ⏩⏩⏩ Pink No Information
Twitter sympathy alert
alert when more than two shepherds entered a certain stock
Fix logs
Make sure that all of the logs go to the same log.
isOtciq alert
We want to alert when a company gets it's otciq certification
can be figured out by declaring "certification fields" and checking if they have changed for the first time
see #63 for more info
Add OS/AS major change alert
in securities collection, keys:
outstandingShares
authorizedShares
When the diff is unusual, 10% difference (up or down) - alert the change
- Alert the diff followed by the precentage (ctrl+f otciq for an example)
Register as a service
Register stocker and CI/CD as a service on the production server:
commands:
stocker
python3 ~/lib/stocker/alert.py --uri mongodb://admin:admin123@localhost:27017/stocker
CI/CD
webhook -hooks ~/webhooks/hooks.json -ip "51.91.11.169" -verbose
Dynamically build list of tickers
Right now, we're sampling tickers according to the tickers listed in csv/tickers.csv
We want to dynamically create a list of tickers (based on tickers.csv)
The following command returns a json containing information about given stocks.
In this example I used SGIP, SOAN, EXMT tickers - and as you can see they are listed in the request.
Note that there is no key for un-tradeable tickers.
curl -X GET --header "Authorization: " "https://api.tdameritrade.com/v1/marketdata/quotes?apikey=7GF2GJN60ZAP2F9E4V7IXDMFZD5M6GQR&symbol=SGIP%2CSOAN%2CEXMT"
- Make sure that you don't exceed the maximum count of tickers in this API
Use this example to do the following:
-
Base on the biggest tickers.csv file in git history (the one with the most tickers)
-
Generate a list of tradeable tickers
-
Set a default price range of 0.0001 - 0.05
(The response contains "lastPrice") -
Do it in Runnable.extract_tickers
Add user permissions
By default, registered user will have USER permissions.
Insert by hand different permissions for our team and Penny
Array diffs
Right now, if a value in an array is removed and the array is not empty, we will generate 2 diffs - one for a change and one for a remove.
Fix this.
e.g:
2020-11-04 15:36:01,641 INFO changes: [{'ticker': 'AGDY', 'date': '2020-11-04 13:36:00+00:00', 'changed_key': 'standardDirectorList.0.name', 'old': 'Erik Jablonowski', 'new': 'Gerald Agnew', 'diff_type': 'change'}, {'ticker': 'AGDY', 'date': '2020-11-04 13:36:00+00:00', 'changed_key': 'standardDirectorList', 'old': None, 'new': {'name': 'Erik Jablonowski', 'title': 'Director', 'boards': 'Independent Director'}, 'diff_type': 'add'}, {'ticker': 'AGDY', 'date': '2020-11-04 13:36:00+00:00', 'changed_key': 'officers.0.name', 'old': 'Joseph Amrams', 'new': 'Joseph Amram', 'diff_type': 'change'}]
old:
{"_id":{"$oid":"5fa2a8cf211617f2789cedbb"},"id":36334,"name":"Agri-Dynamics, Inc.","city":"Forest Hills","state":"NY","zip":"11375 ","countryId":"USA","country":"United States","address1":"107-23 71st Ave Suite# 193","website":"http://www.agri-dynamicsinc.com","phone":"515-329-0208","businessDesc":"AGRI-DYNAMICS, INC., (OTC MARKETS: AGDY) has been reinvented as a holding company like a SPAC that intends to acquire and joint venture with corporations that will help us increase revenue and generate valuable assets in various sectors. We specialize in the acquisition, investment and development of small or large businesses. The company's current research is in the energy, commodity, technology and agricultural sectors. We feel that today's market maintains a truly unprecedented opportunity for the acquisition in proven mining claims, high technology as well as valuable intellectual properties. Our aim is to grow our revenue, in part by assisting our subsidiaries to attain their revenue forecasts by making available additional financing, experienced personnel, distribution, marketing, and other services. AGRI-DYNAMICS, INC., is currently in search of valuable gold and silver mines that are close to production or will be in a reasonable amount of time. These assets can add immense value to the company quickly. AGRI-DYNAMICS, INC., (OTC Markets: AGDY) originally an agricultural and machine equipment Company, was founded in Iowa on December 8, 1961. For more information see the companies financial statements and related filings on OTCMarkets.com","stateOfIncorporation":"IA ","stateOfIncorporationName":"Iowa","countryOfIncorporation":"USA","countryOfIncorporationName":"United States","yearOfIncorporation":"1961","premierDirectorList":[{"name":"Joseph Amram","title":"Chairman","boards":""}],"standardDirectorList":[{"name":"Erik Jablonowski","title":"Director","boards":"Independent Director"}],"officers":[{"name":"Joseph Amrams","title":"CEO, President","boards":""}],"fiscalYearEnd":"6/30","filingCycle":"Q","edgarFilingStatus":"Alternative Reporting Standard","edgarFilingStatusId":"A","reportingStandard":"Dark: Alternative Reporting Standard","reportingStandardMin":"Dark or Defunct","isDark":true,"deregistered":true,"deregistrationDate":{"$numberLong":"1249876800000"},"is12g32b":false,"cik":"0000002817","isAlternativeReporting":true,"isBankThrift":false,"isNonBankRegulated":false,"isInternationalReporting":false,"isOtherReporting":false,"auditedStatusDisplay":"Not Available","auditStatus":"N","audited":false,"email":"[email protected]","numberOfEmployees":5,"numberOfEmployeesAsOf":{"$numberLong":"1544677200000"},"numberOfRecordShareholders":38,"numberOfRecordShareholdersDate":{"$numberLong":"1389070800000"},"primarySicCode":"1382 - Oil and gas exploration services","auditors":[],"investorRelationFirms":[],"legalCounsels":[],"investmentBanks":[],"corporateBrokers":[],"securities":[{"id":577,"tierId":22,"symbol":"AGDY","cusip":"008495202","compId":36334,"issueName":"AGRI-DYNAMICS INC","className":"Common Stock","primaryVenue":"OTC Link","categoryId":4,"categoryName":"No Information","isBB":false,"isPinkSheets":false,"isNoInfo":true,"hasLevel2":false,"isLevel2Entitled":false,"typeCode":"CS","typeName":"Common Stock","tierName":"Pink No Information","tierGroupId":"PS","tierCode":"PN","tierStartDate":{"$numberLong":"1404187200000"},"tierDisplayName":"Pink No Information","ratioAdr":1,"isAdr":false,"isGdr":false,"isTest":false,"isOtcQX":false,"isSponsored":false,"isPiggyBacked":true,"isCaveatEmptor":false,"canAccessBB":false,"noPar":false,"parValue":0.001,"outstandingShares":1900400000,"outstandingSharesAsOfDate":{"$numberLong":"1602216000000"},"authorizedShares":{"$numberLong":"52000000000"},"unlimitedAuthorizedShares":false,"authorizedSharesAsOfDate":{"$numberLong":"1602216000000"},"publicFloat":9423157,"publicFloatAsOfDate":{"$numberLong":"1374120000000"},"isExchangeQualified":false,"shortInterest":7190,"shortInterestChange":0,"shortInterestDate":{"$numberLong":"1602734400000"},"restrictedShares":1880000000,"restrictedSharesAsOfDate":{"$numberLong":"1602216000000"},"unrestrictedShares":20400000,"unrestrictedSharesAsOfDate":{"$numberLong":"1602216000000"},"sigFailDeliver":false,"numOfRecordShareholders":38,"numOfRecordShareholdersDate":{"$numberLong":"1389070800000"},"transferAgents":[{"id":2350,"type":"CS","name":"Action Stock Transfer Corporation","phone":"801-274-1088","website":"http://www.actionstocktransfer.com","email":"[email protected]","countryId":"USA","country":"United States","address1":"2469 E. Fort Union Blvd","address2":"Suite 214","city":"Salt Lake City","stateId":"UT","zip":"84121","roles":["Transfer Agent"],"isPublic":true,"hasLogo":false,"isGoodStanding":true,"isProhibited":false,"isQuestionable":false,"isAttorney":false,"isSponsor":false,"public":true}],"notes":["Capital Change=11-71 shares decreased by 1 for 5 reverse split."],"isUnsolicited":false,"showTrustedLogoForAuthorizedShares":true,"showTrustedLogoForOutstandingShares":true,"showTrustedLogoForRestrictedShares":true,"showTrustedLogoForUnrestrictedShares":true,"showTrustedLogoForDTCShares":false}],"otherSecurities":[],"estimatedMarketCap":12162560,"estimatedMarketCapAsOfDate":{"$numberLong":"1604379600000"},"blankCheck":false,"blindPool":false,"spac":false,"hasLatestFiling":true,"latestFilingType":"Interim Financial Report","latestFilingDate":{"$numberLong":"1388466000000"},"latestFilingUrl":"/company/financial-report/116399/content","tierGroup":"PS","tierCode":"PN","tierStartDate":{"$numberLong":"1404187200000"},"hasLogo":true,"companyLogoUrl":"/company/logo/AGDY","isCaveatEmptor":false,"otcAward":{"symbol":"AGDY","best50":false},"indexStatuses":[],"unableToContact":false,"isShell":false,"isBankrupt":false,"isProfileVerified":true,"profileVerifiedAsOfDate":"6/02/2020","venue":"OTC Link","isUnsolicited":false,"ticker":"AGDY","date":"2020-11-04 13:12:46+00:00"}
new:
{'id': 36334, 'name': 'Agri-Dynamics, Inc.', 'city': 'Forest Hills', 'state': 'NY', 'zip': '11375 ', 'countryId': 'USA', 'country': 'United States', 'address1': '107-23 71st Ave Suite# 193', 'website': 'http://www.agri-dynamicsinc.com', 'phone': '515-329-0208', 'businessDesc': "AGRI-DYNAMICS, INC., (OTC MARKETS: AGDY) has been reinvented as a holding company like a SPAC that intends to acquire and joint venture with corporations that will help us increase revenue and generate valuable assets in various sectors. We specialize in the acquisition, investment and development of small or large businesses. The company's current research is in the energy, commodity, technology and agricultural sectors. We feel that today's market maintains a truly unprecedented opportunity for the acquisition in proven mining claims, high technology as well as valuable intellectual properties. Our aim is to grow our revenue, in part by assisting our subsidiaries to attain their revenue forecasts by making available additional financing, experienced personnel, distribution, marketing, and other services. AGRI-DYNAMICS, INC., is currently in search of valuable gold and silver mines that are close to production or will be in a reasonable amount of time. These assets can add immense value to the company quickly. AGRI-DYNAMICS, INC., (OTC Markets: AGDY) originally an agricultural and machine equipment Company, was founded in Iowa on December 8, 1961. For more information see the companies financial statements and related filings on OTCMarkets.com", 'stateOfIncorporation': 'IA ', 'stateOfIncorporationName': 'Iowa', 'countryOfIncorporation': 'USA', 'countryOfIncorporationName': 'United States', 'yearOfIncorporation': '1961', 'premierDirectorList': [{'name': 'Joseph Amram', 'title': 'Chairman', 'boards': ''}], 'standardDirectorList': [{'name': 'Gerald Agnew', 'title': 'Director', 'boards': 'Independent Director'}, {'name': 'Erik Jablonowski', 'title': 'Director', 'boards': 'Independent Director'}], 'officers': [{'name': 'Joseph Amram', 'title': 'CEO, President', 'boards': ''}], 'fiscalYearEnd': '6/30', 'filingCycle': 'Q', 'edgarFilingStatus': 'Alternative Reporting Standard', 'edgarFilingStatusId': 'A', 'reportingStandard': 'Dark: Alternative Reporting Standard', 'reportingStandardMin': 'Dark or Defunct', 'isDark': True, 'deregistered': True, 'deregistrationDate': 1249876800000, 'is12g32b': False, 'cik': '0000002817', 'isAlternativeReporting': True, 'isBankThrift': False, 'isNonBankRegulated': False, 'isInternationalReporting': False, 'isOtherReporting': False, 'auditedStatusDisplay': 'Not Available', 'auditStatus': 'N', 'audited': False, 'email': '[email protected]', 'numberOfEmployees': 5, 'numberOfEmployeesAsOf': 1544677200000, 'numberOfRecordShareholders': 38, 'numberOfRecordShareholdersDate': 1389070800000, 'primarySicCode': '1382 - Oil and gas exploration services', 'auditors': [], 'investorRelationFirms': [], 'legalCounsels': [], 'investmentBanks': [], 'corporateBrokers': [], 'securities': [{'id': 577, 'tierId': 22, 'symbol': 'AGDY', 'cusip': '008495202', 'compId': 36334, 'issueName': 'AGRI-DYNAMICS INC', 'className': 'Common Stock', 'primaryVenue': 'OTC Link', 'categoryId': 4, 'categoryName': 'No Information', 'isBB': False, 'isPinkSheets': False, 'isNoInfo': True, 'hasLevel2': False, 'isLevel2Entitled': False, 'typeCode': 'CS', 'typeName': 'Common Stock', 'tierName': 'Pink No Information', 'tierGroupId': 'PS', 'tierCode': 'PN', 'tierStartDate': 1404187200000, 'tierDisplayName': 'Pink No Information', 'ratioAdr': 1.0, 'isAdr': False, 'isGdr': False, 'isTest': False, 'isOtcQX': False, 'isSponsored': False, 'isPiggyBacked': True, 'isCaveatEmptor': False, 'canAccessBB': False, 'noPar': False, 'parValue': 0.001, 'outstandingShares': 1900400000, 'outstandingSharesAsOfDate': 1602216000000, 'authorizedShares': 52000000000, 'unlimitedAuthorizedShares': False, 'authorizedSharesAsOfDate': 1602216000000, 'publicFloat': 9423157, 'publicFloatAsOfDate': 1374120000000, 'isExchangeQualified': False, 'shortInterest': 7190, 'shortInterestChange': 0.0, 'shortInterestDate': 1602734400000, 'restrictedShares': 1880000000, 'restrictedSharesAsOfDate': 1602216000000, 'unrestrictedShares': 20400000, 'unrestrictedSharesAsOfDate': 1602216000000, 'sigFailDeliver': False, 'numOfRecordShareholders': 38, 'numOfRecordShareholdersDate': 1389070800000, 'transferAgents': [{'id': 2350, 'type': 'CS', 'name': 'Action Stock Transfer Corporation', 'phone': '801-274-1088', 'website': 'http://www.actionstocktransfer.com', 'email': '[email protected]', 'countryId': 'USA', 'country': 'United States', 'address1': '2469 E. Fort Union Blvd', 'address2': 'Suite 214', 'city': 'Salt Lake City', 'stateId': 'UT', 'zip': '84121', 'roles': ['Transfer Agent'], 'isPublic': True, 'hasLogo': False, 'isGoodStanding': True, 'isProhibited': False, 'isQuestionable': False, 'isAttorney': False, 'isSponsor': False, 'public': True}], 'notes': ['Capital Change=11-71 shares decreased by 1 for 5 reverse split.'], 'isUnsolicited': False, 'showTrustedLogoForAuthorizedShares': True, 'showTrustedLogoForOutstandingShares': True, 'showTrustedLogoForRestrictedShares': True, 'showTrustedLogoForUnrestrictedShares': True, 'showTrustedLogoForDTCShares': False}], 'otherSecurities': [], 'estimatedMarketCap': 12162560.0, 'estimatedMarketCapAsOfDate': 1604379600000, 'blankCheck': False, 'blindPool': False, 'spac': False, 'hasLatestFiling': True, 'latestFilingType': 'Interim Financial Report', 'latestFilingDate': 1388466000000, 'latestFilingUrl': '/company/financial-report/116399/content', 'tierGroup': 'PS', 'tierCode': 'PN', 'tierStartDate': 1404187200000, 'hasLogo': True, 'companyLogoUrl': '/company/logo/AGDY', 'isCaveatEmptor': False, 'otcAward': {'symbol': 'AGDY', 'best50': False}, 'indexStatuses': [], 'unableToContact': False, 'isShell': False, 'isBankrupt': False, 'isProfileVerified': True, 'profileVerifiedAsOfDate': '6/02/2020', 'venue': 'OTC Link', 'isUnsolicited': False, 'ticker': 'AGDY', 'date': '2020-11-04 13:36:00+00:00'}
Add ticker info to alert message
Add info to every alert message.
Tier, industry, and price
both can be found as keys in our collectors:
industry - profile, "primarySicCode" - drop the code number, we need the text
tier - securities, "tierDisplayName"
You can read them by using Collector.get_latest()
price - Use Site object to reach the url at:
https://backend.otcmarkets.com/otcapi/stock/trade/inside/{ticker}?symbol={ticker}
"previousClose"
- Make sure that the alert message remains user friendly
Unite array alerts
Filter unnecessary alerts
Filter out the following alerts:
- businessDesc encoding problems, as you can see in the comments
- Phone number format changes, such as:
🚨 Detected change on ABCE:
520-777-0511 ⏩⏩⏩ (520) 777-0511
+1 (703) 273-0383 ⏩⏩⏩ (703) 273-0383
8133210440 ⏩⏩⏩ +1 (813) 723-1206
Add security details data
SECURITY DETAILS:
Outstanding Shares, Authorized Shares, Float, כל שינוי. מה היה לפי ומה עכשיו.
Add company profile data
Company profile:
DESCRIPTION כל שינוי. לראות מה היה לפני ומה עכשיו.
COMPANY OFFICERS כל שינוי. לראות מה היה לפני ומה עכשיו.
COMPANY DIRECTORS כל שינוי. לראות מה היה לפני ומה עכשיו.
Adress EMAIL, phone number, website
Add FILTER_KEYS and DROP_KEYS to SiteCollector
Add them to the framework by using polymorphism
Dilution alert
for consecutive sales on a given amount of time
Dilution alert
Classify filings by keywords
State the filing type in the alert message
add comma in telegram large numbers massage
exemple
🚨 Detected change on RTON:
unrestrictedShares has changed:
2893145141 ⏩⏩⏩ 3044573712
to
🚨 Detected change on RTON:
unrestrictedShares has changed:
2,893,145,141 --------> 3,044,573,712
Add lest price to telegram
/collector
put code
above security and symbol
isFirstAccess alert
If a company has accessed it's otciq account for the first time in a long while we want to be the first to know about that.
Look at a set of parameters that could only change with otciq access
alert with a unique diff_type and render it in the bot
More tickers!
Add as many tickers as possible to tickers.csv
A place to look at:
https://www.otcmarkets.com/research/stock-screener
Documentation
Here I'll write some stocks that came up with stocker and to collect a hit rate
Summarize ss dates from past to present time
הנה משהו חיוני:
היסטוריית שינויים ב S/S
כדאי שיהיה אפשר לדעת בכמה זה גדל תוך כמה זמן (פשוט להוסיף תאריכי עדכון)
PennyWhale
כדאי שיהיה אפשר לדעת בכמה זה גדל תוך כמה זמן (פשוט להוסיף תאריכי עדכון)
Nati
מה הכוונה ?
כי השרת מבצע בדיקה יומית
אתה מתכוונת שאתה מבצע חיפוש לאחור ?
PennyWhale
HQGE
PennyWhale
🚨 Stocker alertsNov 20, 2020 2:45:34 AM
🚨 Detected change on HQGE:
unrestrictedShares has changed:
2621633000 ⏩⏩⏩ 3021633000
PennyWhale
למתי "2621633000 " היה נכון?
לאיזה תאריך ?
PennyWhale
למתי "2621633000 " היה נכון?
Nati
במקרה הזה התאריך היה 20 לנובמבר אבל לא היו לנו דגימות קודמות ככה שאין לי מושג מתי פעם לפני עודכן
PennyWhale
אבל אני אומר לעתיד
שהמידע הזה יהיה נגיש בתוכנה
Nati
אוקי סבבה יבדק
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.