rlwastro / mastcasjobs Goto Github PK
View Code? Open in Web Editor NEWA simple interface to the MAST CasJobs server (home of GALEX, Kepler, the Hubble Source Catalog, PanSTARRS, etc.)
License: MIT License
A simple interface to the MAST CasJobs server (home of GALEX, Kepler, the Hubble Source Catalog, PanSTARRS, etc.)
License: MIT License
Hi Rick
My mastcasjobs quick queue script has been running happily for about a week.
Yesterday evening it started giving errors, as did my long queue casjobs scripts,
More info since you added the error retrieval.
the mastjobs quick is giving
ExecuteQuickJob failed with status: 500
System.Data.SqlClient.SqlException: Could not find prepared statement with handle -1.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
and similar for the long queue casjobs
SubmitJob failed with status: 500
System.Data.SqlClient.SqlException: Could not find prepared statement with handle -1.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
As before I'm not seeing these queries reach my history via the web interface.
I've emailed archive support but wondering
a) if mastcasjobs queries work for you
b) any ideas
Thanks!
Mike
Rick,
I'm experiencing the same issue MikeeRead was having a month ago:
Exception: ExecuteQuickJob failed with status: 500
System.Data.SqlClient.SqlException: Could not find prepared statement with handle -1.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
Nothing is showing up in my history.
Cheers,
Michael Stroh
Hi
I'd been using mastcastjobs to successfully run quick queries for several days eg
jobs = mastcasjobs.MastCasJobs(username=un, password=pwd, context="PanSTARRS_DR2")
checkResults = jobs.quick(checkQuery, task_name="check")
jobs.drop_table_if_exists(myDBTable)
since yesterday I've been getting errors. ExecuteQuickJob failed with status: 500
I'm also getting these errors going back to previous successful queries. I can run the same queries interactively via the quick web form.
I've contacted archive support but am wondering if anything has changed with the API's interface. Our mastcasjobs was installed 9 days ago and seems to point to the https URL
base_url="https://mastweb.stsci.edu/ps1casjobs/services/jobs.asmx",
the jobs submitted via mastcasjobs do not seem to reach my "history". Before they were (even the ones that failed I think).
Anyone else see the same issue?
Thanks
Mike
I'm walking through your readme to test a simple query to CasJobs via the python CasJobs (by dfm).
When doing that, I get this error:
TypeError: __init__() got an unexpected keyword argument 'request_type'
It seems that the python CasJobs library defines a class but does not take "request_type" as an argument.
I have a script that uses the list_tables
method.
When executing I get
File "[...]/python3.9/site-packages/mastcasjobs.py", line 118, in list_tables
return [l[1:-1]for l in res.split('\n')[1:-1]]
AttributeError: 'Table' object has no attribute 'split'
list_tables()
assumes the output of quick
to be a string but it was changed to be a Table
by default. That is what's causing the problem I presume.
Hello Rick, I'm getting an ExpatError
running the sample code in the repo.
import mastcasjobs
import getpass
import os
if not os.environ.get('CASJOBS_USERID'):
os.environ['CASJOBS_USERID'] = input('Enter Casjobs UserID:')
if not os.environ.get('CASJOBS_PW'):
os.environ['CASJOBS_PW'] = getpass.getpass('Enter Casjobs password:')
query = """select o.objID, o.raMean, o.decMean,
o.nDetections, o.ng, o.nr, o.ni, o.nz, o.ny,
m.gMeanPSFMag, m.rMeanPSFMag, m.iMeanPSFMag, m.zMeanPSFMag, m.yMeanPSFMag
from fGetNearbyObjEq(187.706,12.391,50.0/60.0) nb
inner join ObjectThin o on o.objid=nb.objid and o.nDetections>1
inner join MeanObject m on o.objid=m.objid and o.uniquePspsOBid=m.uniquePspsOBid
"""
jobs = mastcasjobs.MastCasJobs(context="PanSTARRS_DR2")
results = jobs.quick(query, task_name="python cone search")
print(results)
---------------------------------------------------------------------------
ExpatError Traceback (most recent call last)
Input In [15], in <cell line: 11>()
1 query = """select o.objID, o.raMean, o.decMean,
2 o.nDetections, o.ng, o.nr, o.ni, o.nz, o.ny,
3 m.gMeanPSFMag, m.rMeanPSFMag, m.iMeanPSFMag, m.zMeanPSFMag, m.yMeanPSFMag
(...)
6 inner join MeanObject m on o.objid=m.objid and o.uniquePspsOBid=m.uniquePspsOBid
7 """
9 jobs = mastcasjobs.MastCasJobs(context="PanSTARRS_DR2")
---> 11 results = jobs.quick(query, task_name="python cone search")
12 print(results)
File ~\anaconda3\envs\stsci\lib\site-packages\mastcasjobs\__init__.py:152, in MastCasJobs.quick(self, q, context, task_name, system, astropy)
131 def quick(self, q, context=None, task_name="quickie", system=False, astropy=True):
132 """
133 Run a quick job. Like CasJobs method but adds astropy option.
134
(...)
150
151 """
--> 152 results = super(MastCasJobs,self).quick(q, context=context, task_name=task_name, system=system)
153 if astropy:
154 return MastCasJobs.convert_quick_table(results)
File ~\anaconda3\envs\stsci\lib\site-packages\casjobs.py:176, in CasJobs.quick(self, q, context, task_name, system)
173 params = {"qry": q, "context": context, "taskname": task_name,
174 "isSystem": system}
175 r = self._send_request("ExecuteQuickJob", params=params)
--> 176 return self._parse_single(r.text, "string")
File ~\anaconda3\envs\stsci\lib\site-packages\casjobs.py:148, in CasJobs._parse_single(self, text, tagname)
134 def _parse_single(self, text, tagname):
135 """
136 A hack to get the content of the XML responses from the CAS server.
137
(...)
146
147 """
--> 148 return minidom.parseString(text)\
149 .getElementsByTagName(tagname)[0].firstChild.data
File ~\anaconda3\envs\stsci\lib\xml\dom\minidom.py:1998, in parseString(string, parser)
1996 if parser is None:
1997 from xml.dom import expatbuilder
-> 1998 return expatbuilder.parseString(string)
1999 else:
2000 from xml.dom import pulldom
File ~\anaconda3\envs\stsci\lib\xml\dom\expatbuilder.py:925, in parseString(string, namespaces)
923 else:
924 builder = ExpatBuilder()
--> 925 return builder.parseString(string)
File ~\anaconda3\envs\stsci\lib\xml\dom\expatbuilder.py:223, in ExpatBuilder.parseString(self, string)
221 parser = self.getParser()
222 try:
--> 223 parser.Parse(string, True)
224 self._setup_subset(string)
225 except ParseEscape:
ExpatError: not well-formed (invalid token): line 14, column 132
I also get the same error running the first query in Hubble Source Catalog SWEEPS Proper Motion Notebook. In this case, I tried the slow, and the fast version.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [11], in <cell line: 4>()
5 print("Retrieving table MyDB.{} (if it exists)".format(DBtable))
----> 6 tab = jobs.fast_table(DBtable, verbose=True)
7 except ValueError:
File ~\anaconda3\envs\stsci\lib\site-packages\mastcasjobs\__init__.py:208, in MastCasJobs.fast_table(self, table, verbose)
206 except Exception as e:
207 # raise ValueError("table MyDB.{} not found".format(table)) from None
--> 208 raise_from(ValueError("table MyDB.{} not found".format(table)), None)
209 # get table from the quick_casjobs.cgi service
File <string>:3, in raise_from(value, from_value)
ValueError: table MyDB.SWEEPS not found
During handling of the above exception, another exception occurred:
ExpatError Traceback (most recent call last)
Input In [11], in <cell line: 4>()
8 print("Table MyDB.{} not found, running query to create it".format(DBtable))
10 # drop table if it already exists
---> 11 jobs.drop_table_if_exists(DBtable)
13 #get main information
14 query = """
15 select a.ObjID, RA=a.raMean, Dec=a.decMean, RAerr=a.raMeanErr, Decerr=a.decMeanErr,
16 c.NumFilters, c.NumVisits,
(...)
30 join AstromSumPropMagAper2Cat c on a.ObjID=c.ObjID
31 """.format(DBtable)
File ~\anaconda3\envs\stsci\lib\site-packages\mastcasjobs\__init__.py:129, in MastCasJobs.drop_table_if_exists(self, table)
120 def drop_table_if_exists(self, table):
121 """
122 Drop table from MyDB without an error if it does not exist
123
(...)
127
128 """
--> 129 results = self.quick("DROP TABLE IF EXISTS {}".format(table),context="MYDB")
File ~\anaconda3\envs\stsci\lib\site-packages\mastcasjobs\__init__.py:152, in MastCasJobs.quick(self, q, context, task_name, system, astropy)
131 def quick(self, q, context=None, task_name="quickie", system=False, astropy=True):
132 """
133 Run a quick job. Like CasJobs method but adds astropy option.
134
(...)
150
151 """
--> 152 results = super(MastCasJobs,self).quick(q, context=context, task_name=task_name, system=system)
153 if astropy:
154 return MastCasJobs.convert_quick_table(results)
File ~\anaconda3\envs\stsci\lib\site-packages\casjobs.py:176, in CasJobs.quick(self, q, context, task_name, system)
173 params = {"qry": q, "context": context, "taskname": task_name,
174 "isSystem": system}
175 r = self._send_request("ExecuteQuickJob", params=params)
--> 176 return self._parse_single(r.text, "string")
File ~\anaconda3\envs\stsci\lib\site-packages\casjobs.py:148, in CasJobs._parse_single(self, text, tagname)
134 def _parse_single(self, text, tagname):
135 """
136 A hack to get the content of the XML responses from the CAS server.
137
(...)
146
147 """
--> 148 return minidom.parseString(text)\
149 .getElementsByTagName(tagname)[0].firstChild.data
File ~\anaconda3\envs\stsci\lib\xml\dom\minidom.py:1998, in parseString(string, parser)
1996 if parser is None:
1997 from xml.dom import expatbuilder
-> 1998 return expatbuilder.parseString(string)
1999 else:
2000 from xml.dom import pulldom
File ~\anaconda3\envs\stsci\lib\xml\dom\expatbuilder.py:925, in parseString(string, namespaces)
923 else:
924 builder = ExpatBuilder()
--> 925 return builder.parseString(string)
File ~\anaconda3\envs\stsci\lib\xml\dom\expatbuilder.py:223, in ExpatBuilder.parseString(self, string)
221 parser = self.getParser()
222 try:
--> 223 parser.Parse(string, True)
224 self._setup_subset(string)
225 except ParseEscape:
ExpatError: not well-formed (invalid token): line 14, column 132
conda
environment running Jupyter Labname: stsci
channels:
- defaults
- conda-forge
dependencies:
- appdirs=1.4.4=pyhd3eb1b0_0
- asdf=2.11.1=pyhd8ed1ab_0
- asdf-standard=1.0.2=pyhd8ed1ab_0
- asdf-transform-schemas=0.2.2=pyhd8ed1ab_0
- astropy=5.0.4=py310h9128911_0
- astroquery=0.4.6=pyhd8ed1ab_0
- asttokens=2.0.5=pyhd3eb1b0_0
- atomicwrites=1.4.0=py_0
- attrs=21.4.0=pyhd3eb1b0_0
- backcall=0.2.0=pyhd3eb1b0_0
- beautifulsoup4=4.11.1=py310haa95532_0
- black=19.10b0=py_0
- blas=1.0=mkl
- blosc=1.21.0=h19a0ad4_0
- bottleneck=1.3.4=py310h9128911_0
- brotli=1.0.9=ha925a31_2
- brotlipy=0.7.0=py310h2bbff1b_1002
- bzip2=1.0.8=he774522_0
- ca-certificates=2022.4.26=haa95532_0
- certifi=2020.6.20=pyhd3eb1b0_3
- cffi=1.15.0=py310h2bbff1b_1
- cfitsio=3.470=he774522_6
- charls=2.2.0=h6c2663c_0
- charset-normalizer=2.0.4=pyhd3eb1b0_0
- click=8.0.4=py310haa95532_0
- cloudpickle=2.0.0=pyhd3eb1b0_0
- colorama=0.4.4=pyhd3eb1b0_0
- coverage=6.3.2=py310h2bbff1b_0
- cryptography=37.0.1=py310h21b164f_0
- cycler=0.11.0=pyhd3eb1b0_0
- cytoolz=0.11.0=py310h2bbff1b_0
- dask-core=2022.2.1=pyhd3eb1b0_0
- debugpy=1.5.1=py310hd77b12b_0
- decorator=5.1.1=pyhd3eb1b0_0
- eigen=3.3.7=h59b6b97_1
- entrypoints=0.4=py310haa95532_0
- et_xmlfile=1.1.0=py310haa95532_0
- executing=0.8.3=pyhd3eb1b0_0
- fonttools=4.25.0=pyhd3eb1b0_0
- freetype=2.10.4=hd328e21_0
- fsspec=2022.2.0=pyhd3eb1b0_0
- giflib=5.2.1=h62dcd97_0
- glib=2.69.1=h5dc1a3c_1
- gst-plugins-base=1.18.5=h9e645db_0
- gstreamer=1.18.5=hd78058f_0
- gwcs=0.17.1=pyhd8ed1ab_0
- hdf5=1.10.6=h7ebc959_0
- html5lib=1.1=pyhd3eb1b0_0
- hypothesis=6.29.3=pyhd3eb1b0_0
- icc_rt=2019.0.0=h0cc432a_1
- icu=58.2=ha925a31_3
- idna=3.3=pyhd3eb1b0_0
- imagecodecs=2021.8.26=py310h6cc35ac_0
- imageio=2.9.0=pyhd3eb1b0_0
- importlib-metadata=4.11.3=py310haa95532_0
- importlib-resources=5.7.1=pyhd8ed1ab_0
- importlib_metadata=4.11.3=hd3eb1b0_0
- importlib_resources=5.7.1=pyhd8ed1ab_0
- iniconfig=1.1.1=pyhd3eb1b0_0
- intel-openmp=2021.4.0=haa95532_3556
- ipykernel=6.9.1=py310haa95532_0
- ipython=8.3.0=py310haa95532_0
- jedi=0.18.1=py310haa95532_1
- jmespath=0.10.0=pyhd3eb1b0_0
- joblib=1.1.0=pyhd3eb1b0_0
- jpeg=9e=h2bbff1b_0
- jsonschema=4.4.0=py310haa95532_0
- jupyter_client=7.2.2=py310haa95532_0
- jupyter_core=4.10.0=py310haa95532_0
- keyring=23.4.0=py310haa95532_0
- kiwisolver=1.3.1=py310hd77b12b_0
- lcms2=2.12=h83e58a3_0
- lerc=3.0=hd77b12b_0
- libaec=1.0.4=h33f27b4_1
- libdeflate=1.8=h2bbff1b_5
- libffi=3.4.2=h604cdb4_1
- libiconv=1.16=h2bbff1b_2
- libogg=1.3.5=h2bbff1b_1
- libpng=1.6.37=h2a8f88b_0
- libprotobuf=3.5.1=he0781b1_0
- libtiff=4.2.0=hd0e1b90_0
- libvorbis=1.3.7=he774522_0
- libwebp=1.2.2=h2bbff1b_0
- libwebp-base=1.2.2=h2bbff1b_0
- libxml2=2.9.12=h0ad7f3c_1
- libxslt=1.1.34=he774522_0
- libzopfli=1.0.3=ha925a31_0
- locket=0.2.1=py310haa95532_2
- lxml=4.8.0=py310h1985fb9_0
- lz4-c=1.9.3=h2bbff1b_1
- matplotlib=3.5.1=py310haa95532_1
- matplotlib-base=3.5.1=py310hd77b12b_1
- matplotlib-inline=0.1.2=pyhd3eb1b0_2
- mkl=2021.4.0=haa95532_640
- mkl-service=2.4.0=py310h2bbff1b_0
- mkl_fft=1.3.1=py310ha0764ea_0
- mkl_random=1.2.2=py310h4ed8f06_0
- mplcursors=0.5.1=pyhd8ed1ab_0
- mplfinance=0.12.9b0=pyhd8ed1ab_0
- munkres=1.1.4=py_0
- mypy_extensions=0.4.3=py310haa95532_1
- natsort=7.1.1=pyhd3eb1b0_0
- nest-asyncio=1.5.5=py310haa95532_0
- networkx=2.7.1=pyhd3eb1b0_0
- nltk=3.7=pyhd3eb1b0_0
- numexpr=2.8.1=py310hb57aa6b_1
- numpy=1.21.5=py310h6d2d95c_2
- numpy-base=1.21.5=py310h206c741_2
- opencv=4.5.4=py310hc26a207_3
- openjpeg=2.4.0=h4fc8c34_0
- openpyxl=3.0.9=pyhd3eb1b0_0
- openssl=1.1.1o=h2bbff1b_0
- packaging=21.3=pyhd3eb1b0_0
- pandas=1.4.2=py310hd77b12b_0
- pandas-datareader=0.10.0=pyhd3eb1b0_0
- parso=0.8.3=pyhd3eb1b0_0
- partd=1.2.0=pyhd3eb1b0_1
- pathspec=0.7.0=py_0
- patsy=0.5.2=py310haa95532_1
- pcre=8.45=hd77b12b_0
- photutils=1.4.0=py310h2873277_0
- pickleshare=0.7.5=pyhd3eb1b0_1003
- pillow=9.0.1=py310hdc2b20a_0
- pip=21.2.4=py310haa95532_0
- pluggy=1.0.0=py310haa95532_1
- prompt-toolkit=3.0.20=pyhd3eb1b0_0
- psutil=5.8.0=py310h2bbff1b_1
- pure_eval=0.2.2=pyhd3eb1b0_0
- py=1.11.0=pyhd3eb1b0_0
- pycparser=2.21=pyhd3eb1b0_0
- pyerfa=2.0.0=py310h2bbff1b_0
- pygments=2.11.2=pyhd3eb1b0_0
- pyopenssl=22.0.0=pyhd3eb1b0_0
- pyparsing=3.0.4=pyhd3eb1b0_0
- pyqt=5.9.2=py310hd77b12b_6
- pyrsistent=0.18.0=py310h2bbff1b_0
- pysocks=1.7.1=py310haa95532_0
- pytest=7.1.1=py310haa95532_0
- pytest-arraydiff=0.3=py310h9909e9c_0
- pytest-astropy=0.9.0=pyhd3eb1b0_0
- pytest-astropy-header=0.1.2=pyhd3eb1b0_0
- pytest-cov=3.0.0=pyhd3eb1b0_0
- pytest-doctestplus=0.11.1=pyhd3eb1b0_0
- pytest-filter-subpackage=0.1.1=pyhd3eb1b0_0
- pytest-mock=3.6.1=pyhd3eb1b0_0
- pytest-openfiles=0.5.0=pyhd3eb1b0_0
- pytest-remotedata=0.3.2=py310haa95532_0
- python=3.10.4=hbb2ffb3_0
- python-dateutil=2.8.2=pyhd3eb1b0_0
- python_abi=3.10=2_cp310
- pytz=2021.3=pyhd3eb1b0_0
- pyvo=1.3=pyhd8ed1ab_0
- pywavelets=1.3.0=py310h2bbff1b_0
- pywin32=302=py310h2bbff1b_2
- pywin32-ctypes=0.2.0=py310haa95532_1000
- pyyaml=6.0=py310h2bbff1b_0
- pyzmq=22.3.0=py310hd77b12b_2
- qt=5.9.7=vc14h73c81de_0
- regex=2022.3.15=py310h2bbff1b_0
- requests=2.27.1=pyhd3eb1b0_0
- scikit-image=0.19.2=py310h4ed8f06_0
- scikit-learn=1.0.2=py310h4ed8f06_0
- scipy=1.7.3=py310h6d2d95c_0
- seaborn=0.11.2=pyhd3eb1b0_0
- semantic_version=2.8.5=pyhd3eb1b0_0
- setuptools=61.2.0=py310haa95532_0
- sip=4.19.13=py310hd77b12b_0
- six=1.16.0=pyhd3eb1b0_1
- snappy=1.1.9=h6c2663c_0
- sortedcontainers=2.4.0=pyhd3eb1b0_0
- soupsieve=2.3.1=pyhd3eb1b0_0
- sqlite=3.38.3=h2bbff1b_0
- stack_data=0.2.0=pyhd3eb1b0_0
- statsmodels=0.13.2=py310h2bbff1b_0
- threadpoolctl=2.2.0=pyh0d69192_0
- tifffile=2021.7.2=pyhd3eb1b0_2
- tk=8.6.11=h2bbff1b_1
- toml=0.10.2=pyhd3eb1b0_0
- tomli=1.2.2=pyhd3eb1b0_0
- toolz=0.11.2=pyhd3eb1b0_0
- tornado=6.1=py310h2bbff1b_0
- tqdm=4.64.0=py310haa95532_0
- traitlets=5.1.1=pyhd3eb1b0_0
- typed-ast=1.4.3=py310h2bbff1b_1
- typing_extensions=4.1.1=pyh06a4308_0
- tzdata=2022a=hda174b7_0
- urllib3=1.26.9=py310haa95532_0
- vc=14.2=h21ff451_1
- vs2015_runtime=14.27.29016=h5e58377_2
- wcwidth=0.2.5=pyhd3eb1b0_0
- webencodings=0.5.1=py310haa95532_1
- wheel=0.37.1=pyhd3eb1b0_0
- win_inet_pton=1.1.0=py310haa95532_0
- wincertstore=0.2=py310haa95532_2
- xz=5.2.5=h8cc25b3_1
- yaml=0.2.5=he774522_0
- zfp=0.5.5=hd77b12b_6
- zipp=3.8.0=py310haa95532_0
- zlib=1.2.12=h8cc25b3_2
- zstd=1.4.9=h19a0ad4_0
- pip:
- casjobs==0.0.1
- cython==0.29.28
- fastkde==1.0.19
- mastcasjobs==0.0.4
- multitasking==0.0.10
- yfinance==0.1.67
prefix: C:\Users\trenton\anaconda3\envs\stsci
Best Regards,
Trenton
Hi,
Our research group wrote a code with mascasjobs 0.1 and it brokes when we run it with the latest version (i got a lot of errors from astropy).
I've tested it and the code only works when i revert the commits of mastcasjobs til 3c27f7f.
When running a simple code using mastcasjobs.py, the code breaks with the following error. I have not yet been able to get a fix for this.
~/anaconda3/lib/python3.7/site-packages/mastcasjobs.py in init(self, username, password, userid, request_type, context, base_url, wsid_url, fast_url)
97
98 super(MastCasJobs,self).init(userid=userid, password=password,
---> 99 base_url=base_url, request_type=request_type, context=context)
100
101 def list_tables(self, context="MYDB"):
TypeError: init() got an unexpected keyword argument 'request_type'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.