bennythadikaran / eod2 Goto Github PK

Error log
CRITICAL: Unhandled exception
Traceback (most recent call last):
File "D:\python\eod2-main\src\init.py", line 58, in
if not defs.dates.nextDate():
^^^^^^^^^^^^^^^^^^^^^
File "D:\python\eod2-main\src\defs\defs.py", line 95, in nextDate
t_str = local_time.strftime("%-I:%M%p") # 7:00PM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Invalid format string
CRITICAL: Unhandled exception
Traceback (most recent call last):
File "D:\python\eod2-main\src\init.py", line 58, in
if not defs.dates.nextDate():
^^^^^^^^^^^^^^^^^^^^^
File "D:\python\eod2-main\src\defs\defs.py", line 95, in nextDate
t_str = local_time.strftime("%-I:%M%p") # 7:00PM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When no user defined watchlists or preset exists, running py plot.py --ls throws the below error.

`AttributeError: 'Config' object has no attribute 'PRESET'

Issue will be fixed in the upcoming update.

Hello Benny,

It just a suggestion, you are filtering the EQ, BE and BZ. It does not have SME based stocks.
Can you include the same in the code.

Thanks
Saurabh

Always reproducible.
There is no issue with the flag set to False.
If I rerun the program, it works fine and will fail for the next date.

Following is the log.

Downloading Files
Starting Data Sync
Converting to AmiBroker format
EOD sync complete.

Nifty PE at 22.81

Index sync complete.
Makings adjustments for splits and bonus
Error while making adjustments. TypeError('cannot do positional indexing on DatetimeIndex with these indexers [[False False False ... False False True]] of type ndarray')
All adjustments have been discarded.
Rolling back changes from 2024-03-22: src\eod2_data\daily

ETFs are getting popular and NSE is pushing more of them into the market.

EOD2 currently does not sync ETF stocks. Sync data for existing and future ETF stocks.

H:\python\eod2\src>py init.py
CRITICAL: Unhandled exception
Traceback (most recent call last):
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 1042, in validate_conn
conn.connect()
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
self.do_handshake()
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 486, in send
resp = conn.urlopen(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\packages\six.py", line 769, in reraise
raise value.with_traceback(tb)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 1042, in validate_conn
conn.connect()
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
self.do_handshake()
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "H:\python\eod2\src\init.py", line 34, in
special_sessions = defs.downloadSpecialSessions()
File "H:\python\eod2\src\defs\defs.py", line 157, in downloadSpecialSessions
res = requests.get(f"{base_url}/main/special_sessions.txt")
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

Hello,

Makings adjustments for splits and bonus is not working.
Example 360one.csv. The stock was adjusted on 2nd march 2023 but the same was not synced by the program.

Downloading Files
Starting Data Sync
EOD sync complete
Delivery Data updated
Delivery sync complete
Index sync complete.
Makings adjustments for splits and bonus
Cleaning up files
0 files deleted
02 Mar 2023: Done

Thanks
Saurabh

User @ramkdlr raised a suggestion to merge the OHLCV and delivery files into a single file.

See discussion here

The new columns will be like
Open, High, Low, Close, Volume, TOTAL_TRADES, QTY_PER_TRADE, DLV_QTY

Changes will be reflected in plot.py and dget.py (No changes in their interface)

Some of recent splits announced are not getting adjusted into EOD2 data. This was due to the regex not detecting the split. This has been fixed. However i need to check for any splits that were missed and action them. I will post an annoucement once the data has been corrected.

Hi Benny,

I could see BhavCopy available at NSE site, but script seems to be having some fetching issues, any idea why so?

INFO: Downloading Files
INFO: Sat, 06 Jul 2024: Market Closed

INFO: Downloading Files
WARNING: NSE file is unavailable or not yet updated.

Thanks,
Rohit

I have recently created NseIndiaApi and released a pip package NSE pypi

Use this package and replace all NSE related functionality in EOD2 like downloading reports, corporate actions etc.

This way, i can make changes or bug fixes in NseIndiaApi and propagate the changes to other repos with minimal work or duplication.

Hi Benny,

Firstly, this is amazing !, I have never seen something prepared so neatly. Kudos !

I tried this few days back and it ran well, however today I tried updating the data and encountered below error, couldn't figure out what's missing as internet connection is running well. I can download other stuff manually. Can you please suggest.

D:\eod2-main\src>init.py
Downloading Files
Download error. Try again later: ChunkedEncodingError(ProtocolError('Connection broken: IncompleteRead(271 bytes read, 138 more expected)', IncompleteRead(271 bytes read, 138 more expected)))

I tried many programs in python to convert the downloaded CSV data to metastock or text file. But it's not working. The data is not working with metastock. If you kindly help with that I'll be grateful.

In rare cases, NSE may delay updating its delivery reports by a day or more. EOD2 will wait for all reports to update before syncing data. See issue #59 for details.

This is inconvenient as latest EOD data is not available because of a single report.

To remedy this issue and prevent future disruptions, here is the final protocol EOD2 will follow going forward:

1. If delivery report if unavailable, continue with updating all other data. Empty string will be used in place of Delivery data. (This is interpreted as NAN values in Pandas/Numpy).
2. Add the date to a pending list.
3. On every sync, loop over this list and attempt to download and update the delivery report for that date.
4. If update is successful, remove the date from the pending list.
5. If delivery report is not updated after 5 days, abort any future attempts. (Thanks to user @rohit1409 for his inputs 👍 )

This feature has been implemented and will be updated on the repo soon. 😺

If src/data folder does not exist, drawing lines on chart and quitting or switching charts will raise a FileNotFoundError

Chart lines are saved on quitting or switching charts. In this case if data folder does not exists, operation fails.

Error has been fixed, Update to be released shortly.

Hello,

There seems to be another issue related to holiday.json.

{
"tradingDate": "28-Jun-2023",
"weekDay": "Wednesday",
"description": "Bakri Id",
"Sr_no": 10
},

This is the holiday.json which is incorrect as the holiday was changed yesterday by NSE.

I have fixed it by removing the file and it downloaded the same but try to see if you can get the latest file everyday.

Thanks
Saurabh

Hi Benny, not sure what's not right.. files are available at NSE page for eod however code throwing back this error.. please have a look whenever convenient. Tx

INFO: Updating NSE EQUITY actions
INFO: Updating NSE SME actions
INFO: Downloading Files
WARNING: NSE file is unavailable or not yet updated.

Hello Benny,

Today session 20 Jan 2024 is not getting downloaded and saying All upto date.
Can you pls check the same.

Thanks
Saurabh

C:\Users\Admin\Documents\Stocks\Code\PythonRS\eod2\src>pip install nse
Collecting nse
Downloading nse-0.2.6.post2-py3-none-any.whl.metadata (2.6 kB)
Requirement already satisfied: mthrottle>=0.0.1 in c:\python311\lib\site-packages (from nse) (0.0.1)
Requirement already satisfied: requests>=2.31.0 in c:\python311\lib\site-packages (from nse) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\python311\lib\site-packages (from requests>=2.31.0->nse) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in c:\python311\lib\site-packages (from requests>=2.31.0->nse) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\python311\lib\site-packages (from requests>=2.31.0->nse) (2.0.3)
Requirement already satisfied: certifi>=2017.4.17 in c:\python311\lib\site-packages (from requests>=2.31.0->nse) (2023.5.7)
Downloading nse-0.2.6.post2-py3-none-any.whl (21 kB)
Installing collected packages: nse
Successfully installed nse-0.2.6.post2

C:\Users\Admin\Documents\Stocks\Code\PythonRS\eod2\src>python init.py
NSE Holiday list updated
Downloading Files
RuntimeError('NSE file is unavailable or not yet updated.')

Kansai Nerolac under symbol code KANSAINER was not adjusted on 4th July. I could not figure the reason for this.

• All stocks which announced bonus on the 4th July were adjusted.
• All stocks which recently announced bonus have been adjusted.
• Post 4th July no bonus or splits were announced.

I ran the adjustment in isolation and found no issues with the code.

The CSV file has been adjusted and added to src/eod2_data/daily. Repo will be updated shortly

A user contacted me on Friday with an error while running EOD2 sync. The error seem to point to duplicate entries in one of the csv files.

I couldn't get the source of the error, but i decided to check my own data for any such issues.

• No duplicate entries found in daily or delivery folders.
• There was however an extra column DELIV_PER in some delivery files. Source of the error was in src/defs/defs.py in the header_text variable. This defines the column headers for delivery files whenever a new file is created.
  • Code corrections were made and 182 files with this extra column was cleaned up.
  • 5 files with just column headers and no data were deleted.

I have written a script diagnostics.py which will look for common errors. I intend to run this weekly before updating data on the repo.

Users can use diagnostics.py to run checks on their own data and report issues.

Nse Indices report use the - character when no data is available. It messes up the data types in pandas.

I have edited defs.py:updateIndexEOD to replace it with zeros. src/eod2_data/daily/India Vix.csv has been manually edited to add zeros.

No other files have this issue so far. Update to follow shortly.

Hello Benny,

Today while syncing error is coming.
Error:

saurabhgarg@MacBook-Pro src % python3 --version
Python 3.9.6
saurabhgarg@MacBook-Pro src % python3 init.py
sh: color: command not found
Downloading Files
Starting Data Sync
EOD sync complete
Index sync complete.
Makings adjustments for splits and bonus
Error while making adjustments. KeyError(datetime.datetime(2024, 1, 25, 0, 0))
All adjustments have been discarded.
Rolling back changes from 2024-01-25: /Users/saurabhgarg/shivshakti/eod2/src/eod2_data/daily
Rollback successful
saurabhgarg@MacBook-Pro src %

Can you pls check your repo.

Thanks
Saurabh

Hello,

The program was running file till now but today (06/03/2023) the issue is coming in cleanup of the files.

in
Downloading Files
Starting Data Sync
EOD sync complete
Delivery Data updated
Delivery sync complete
Index sync complete.
Makings adjustments for splits and bonus
Cleaning up files
Traceback (most recent call last):
File "/Users/saurabhgarg/shivshakti/eod2/init.py", line 51, in
defs.cleanup([bhavFile])
File "/Users/saurabhgarg/shivshakti/eod2/defs.py", line 553, in cleanup
lastUpdated = datetime.strptime(getLastDate(entry.path), fmt)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '' does not match format '%Y-%m-%d'

Since the lastupdate.txt file get updated in the end after cleanup so running the prog will make it double entries in daily and delivery folder.

Thanks
Saurabh

@BennyThadikaran a feature request
can there be a generic mechanism for saving data that I can extend to write to other datalakes?

Originally posted by @Prady04 in #54 (comment)

plot --watch hold-w --tf weekly --sma 30 --m-rs --preset-save hold
Preset saved as 'hold'
Traceback (most recent call last):
  File "/home/benny/Documents/python/eod2/src/plot.py", line 176, in <module>
    plotter.plot(symList[plotter.idx])
  File "/home/benny/Documents/python/eod2/src/defs/Plotter.py", line 218, in plot
    df = self._prepData(sym)
  File "/home/benny/Documents/python/eod2/src/defs/Plotter.py", line 742, in _prepData
    df = getDataFrame(
  File "/home/benny/Documents/python/eod2/src/defs/utils.py", line 176, in getDataFrame
    df = csv_loader(fpath, candle_count, end_date=toDate)
  File "/home/benny/Documents/python/eod2/src/defs/utils.py", line 128, in csv_loader
    current_dt = get_date(start, chunk)
  File "/home/benny/Documents/python/eod2/src/defs/utils.py", line 46, in get_date
    date = datetime.strptime(date_str, date_fmt)
  File "/usr/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/lib/python3.10/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data 'Date' does not match format '%Y-%m-%d'

The error occurs when loading a file using the utils.py:csv_loader function, as a chunk is read from the start of the file.

Under some circumstances the final chunk may read something like this b'Date,Open,High,Low,Close,Volume,TOT'

Trying to parse the Date string as a Datetime value results in a ValueError. Issue was fixed by catching the ValueError, appending the chunk and returning the data as a DataFrame.

Updates to be posted soon.

defs.toAmiBrokerFormat uses TOTTRDVAL instead of TOTTRDQTY for volume.

Issue has been fixed. Repo will be updated shortly.

Thanks Benny. Will go through the same as it seems i need to update python on my system to 3.10 and if do upgrade all my automation will break with brokers so creating virtual env to make eod2 module work.

Thanks

Originally posted by @cotton1234 in #78 (comment)

Hello Benny,

I was looking at the eod daily folder and could find only some indices and not all the indices.

where as in the file ind_close_all_01082023.csv there are lot of indices.

Can i define somewhere what all indices i need to sync.

Also i am planning to use the same logic for downloading stock data from BSE as some stocks are only listed on it.

Thanks
Saurabh