Getting the following error but it doesn't reproduce reliably which is odd. <p dir

pandas.errors.ParserError: Error tokenizing data. about pybaseball HOT 6 CLOSED

jldbc commented on July 23, 2024

pandas.errors.ParserError: Error tokenizing data.

from pybaseball.

Comments (6)

commented on July 23, 2024

Interesting. I haven't encountered such an error after running multiple queries, though I did reduce the time span because it would take awhile. Could you post an image of what you run? Here's what I see on one such attempt:

from pybaseball.

jldbc commented on July 23, 2024

I ran into this testing on a multiyear timespan statcast('2008-03-20','2011-11-11'), but haven't seen it since. I'll see if it reproduces over the weekend.

From this it looks like it's caused by one or several rows having unexpected formatting (maybe a missing or extra delimiter), but I've since added some extra error handling that may have fixed this.

from pybaseball.

jldbc commented on July 23, 2024

Closing this as I haven't run into the error or heard anybody else mention it in a while

from pybaseball.

jnederlo commented on July 23, 2024

I'm getting a similar error but only sometimes.

pandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 35, saw 4

Happens when I'm running statcast_single_game() and sometimes when running statcast() although the latter is less often. If I try re-running it, sometimes it works and sometimes it doesn't. I see the answer here which suggests adding error_bad_lines=False to pd.read_csv().

I also see that this is referenced in /pybaseball/statcast.py on line 35 (and 43) but is commented out:

data = pd.read_csv(io.StringIO(s.decode('utf-8')))#, error_bad_lines=False) # skips 'bad lines' breaking scrapes. still testing this.

Is there any harm with me including error_bad_lines=False, and why does it only error about half the time??

I'm a little worried that if I include the above 'fix' then I might occasionally be missing data that is sometimes there, and sometimes not.

from pybaseball.

jnederlo commented on July 23, 2024

Update to my above question. When I run it with error_bad_lines=False I got this error as a result (when I try running it again it works... seems to fail roughly 30% of the time):

Traceback (most recent call last):
  File "run.py", line 94, in <module>
    method=method)
  File "/home/jnederlo/mlb/pipeline/pipeline.py", line 21, in __init__
    self.choose_process()
  File "/home/jnederlo/mlb/pipeline/pipeline.py", line 31, in choose_process
    self.scrape(day)
  File "/home/jnederlo/mlb/pipeline/pipeline.py", line 51, in scrape
    parser = Parser({'templateName': 'statcast_game'}, day, self.db, self.statcast_game_ids)
  File "/home/jnederlo/mlb/parser/parser.py", line 40, in __init__
    self.sources[self.data['templateName']]()
  File "/home/jnederlo/mlb/parser/parser.py", line 353, in statcast_game
    df = statcast_single_game(mlb_game_pk)
  File "/home/jnederlo/.local/share/virtualenvs/mlb-58UnosLS/lib/python3.6/site-packages/pybaseball/statcast.py", line 215, in statcast_single_game
    data = postprocessing(data, team)
  File "/home/jnederlo/.local/share/virtualenvs/mlb-58UnosLS/lib/python3.6/site-packages/pybaseball/statcast.py", line 155, in postprocessing
    data[numeric_cols] = data[numeric_cols].astype(float)
  File "/home/jnederlo/.local/share/virtualenvs/mlb-58UnosLS/lib/python3.6/site-packages/pandas/core/frame.py", line 2133, in __getitem__
    return self._getitem_array(key)
  File "/home/jnederlo/.local/share/virtualenvs/mlb-58UnosLS/lib/python3.6/site-packages/pandas/core/frame.py", line 2177, in _getitem_array
    indexer = self.loc._convert_to_indexer(key, axis=1)
  File "/home/jnederlo/.local/share/virtualenvs/mlb-58UnosLS/lib/python3.6/site-packages/pandas/core/indexing.py", line 1269, in _convert_to_indexer
    .format(mask=objarr[mask]))
KeyError: "['release_speed' 'release_pos_x' 'release_pos_z' 'batter' 'pitcher' 'zone'\n 'hit_location' 'balls' 'strikes' 'game_year' 'pfx_x' 'pfx_z' 'plate_x'\n 'plate_z' 'on_3b' 'on_2b' 'on_1b' 'outs_when_up' 'inning' 'hc_x' 'hc_y'\n 'fielder_2' 'vx0' 'vy0' 'vz0' 'ax' 'ay' 'az' 'sz_top' 'sz_bot'\n 'hit_distance_sc' 'launch_speed' 'launch_angle' 'effective_speed'\n 'release_spin_rate' 'release_extension' 'game_pk' 'pitcher.1'\n 'fielder_2.1' 'fielder_3' 'fielder_4' 'fielder_5' 'fielder_6' 'fielder_7'\n 'fielder_8' 'fielder_9' 'release_pos_y' 'estimated_ba_using_speedangle'\n 'estimated_woba_using_speedangle' 'woba_value' 'woba_denom' 'babip_value'\n 'iso_value' 'launch_speed_angle' 'at_bat_number' 'pitch_number'\n 'home_score' 'away_score' 'bat_score' 'fld_score' 'post_away_score'\n 'post_home_score' 'post_bat_score' 'post_fld_score'] not in index"```

from pybaseball.

lyledanley commented on July 23, 2024

I'm also getting this error from the statcast_batter() function, again not reliably. Running a for loop with about rows. First time it was raised at row 592. Second time it was raised at 925. I'm thinking I'll just define a custom function with exception handling and go back to the rows that get skipped. Just wanted you to know it's still happening.

from pybaseball.

pandas.errors.ParserError: Error tokenizing data. about pybaseball HOT 6 CLOSED

Comments (6)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent