equinor / tagreader-python Goto Github PK
View Code? Open in Web Editor NEWA Python package for reading trend data from the OSIsoft PI and Aspen InfoPlus.21 historians
Home Page: https://equinor.github.io/tagreader-python/
License: MIT License
A Python package for reading trend data from the OSIsoft PI and Aspen InfoPlus.21 historians
Home Page: https://equinor.github.io/tagreader-python/
License: MIT License
Don't just pick the first one, but warn user to specify if more than one default map is found for a tag.
Convert to Poetry in order to align with other projects.
When there are spaces in the tagname the search function does not work correctly with imstype='aspenone'.
It returns [] even when the tag exists.
fromtime = "2023-01-31 12:01"
totime = "2023-02-10 12:00"
df = c.read(tags, fromtime, totime, 60)
tagreader 3.0.1 with pandas 1.5.0 reads the above correctly. After an update to tagreader 3.0.2 and pandas 2.0.1 it does not. totime is then interpreted as October 2, not February 10.
utils.py issues the following warning:
UserWarning: Parsing dates in %Y-%m-%d %H:%M format when dayfirst=True was specified. Pass dayfirst=False
or specify a format to silence this warning.
date_stamp = pd.to_datetime(date_stamp, dayfirst=True)
Ref mail from Juan today: Reading the test tag from AspenOne resulted in value alternating between 8 and nothing with completely wrong tagname. Perhaps an issue with AspenOne reporting PI data on a different format? Not sure this needs looking into since PI data should primarily be handled by PI Web API.
Make sure tags are not repeated in query and/or report non-unique tags. Something like tags = list(dict.fromkeys(tags))
Current pipeline has 2 issues:
Issue added on GitLab by @knudsvik 2018-06-12:
Based on experience, 100.000 rows got a time-out.
Hi,
I would like to use tagreader on the UNIX. However tagreader (2.3.0) accesses the windows registry. This happens in find_registry_key* and (I think) are used to . Are theses keys also stored on the UNIX machines so it would be possible to extend tagreader use to UNIX?
All the best, Stefan
Hi - seems that the tag search function returns a list with entities (tag + description). I would like to like to grab the tags only in an easy way - and then perhaps get units and get descriptions etc. later. Would it be possible not to combine tag and description in one list entity, or is it an easy way for users to extract the tag from the return of the search function?
I want to see name and type of data server that client is connected to when calling print(client). Will make it fast and easy to pass connection information to loggers...
When trying to pull "old" data with tagreader, there is an issue with using AspenOne. When connecting to the current server, in this case MEL-IMS, everything works fine, but there are no tags found when connecting to a different server (MEL-Y08Y14-IMS).
A work-around is to use the OBDC drivers, but I have some issues with that after upgrading to 3.9.
See attached images for examples.
Search for tag works fine with imstype='pi, with imstype='piwebapi' it crashes.
Ref conversation on Teams.
All NaN-values that are not properly returned as NaN should be. So far: "1.#QNAN" found for Aspen Web API.
Other possibilites (from https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html):
‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, ‘1.#IND’, ‘1.#QNAN’, ‘’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘n/a’, ‘nan’, ‘null’
import getpass
from requests_ntlm import HttpNtlmAuth
user = "mydomain\" + getpass.getuser()
pwd = getpass.getpass()
auth = HttpNtlmAuth(user, pwd)
c = tagreader.IMSClient(datasource="myplant", url="api.mycompany.com/aspenone", imstype="aspenone", auth=auth, verifySSL=False)
c.connect()
d.disconnect() <----- Add methode for disconnecting not leaving a lot of open connections
import sys
sys.path.insert(0, "../")
import tagreader
import getpass
import tables
from tagreader.utils import add_statoil_root_certificate
from tagreader import list_sources
add_statoil_root_certificate()
#print(list_sources("aspen"))
c = tagreader.IMSClient("TRB", "aspenone")
c.connect()
#print(c.search(tag="tagname*", desc="**"))
tags = ['some-tag']
start_time = '01-JAN-2020 08:00:00'
end_time = '10-JAN-2020 08:00:00'
ts = 60
tagToPrint = c.read(tags, start_time, end_time, ts)
print(tagToPrint);
Traceback (most recent call last):
File "C:\Users*\Desktop\tagreader-python.py", line 5, in
import tables
File "C:\Users*\AppData\Roaming\Python\Python38\site-packages\tables_init_.py", line 82, in
raise ImportError(
ImportError: Could not load any of ['hdf5.dll', 'hdf5dll.dll', 'pytables_hdf5.dll'], please ensure that it is installed in the package folder.
Fails because folder extratests\tests is not included in public part of repo.
Any other to consider?
Make it possible to verify that provided is available for connection prior to / independent of actually attempting query.
Our use case can then fail nicely before reading a lot of config files to get the tags.
Collecting tables
Downloading tables-3.6.1.tar.gz (4.6 MB)
|¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 4.6 MB 6.4 MB/s
ERROR: Command errored out with exit status 1:
command: 'C:\Users*\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\\AppData\Local\Temp\pip-install-s96626t1\tables\setup.py'"'"'; file='"'"'C:\Users\\AppData\Local\Temp\pip-install-s96626t1\tables\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users*\AppData\Local\Temp\pip-pip-egg-info-4d2o5bx_'
cwd: C:\Users*\AppData\Local\Temp\pip-install-s96626t1\tables
Complete output (17 lines):
Traceback (most recent call last):
File "", line 1, in
File "C:\Users*\AppData\Local\Temp\pip-install-s96626t1\tables\setup.py", line 634, in
libdir = compiler.has_function(package.target_function,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\distutils\ccompiler.py", line 792, in has_function
objects = self.compile([fname], include_dirs=include_dirs)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\distutils_msvccompiler.py", line 323, in compile
self.initialize()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\distutils_msvccompiler.py", line 220, in initialize
vc_env = _get_vc_env(plat_spec)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\site-packages\setuptools\msvc.py", line 314, in msvc14_get_vc_env
return _msvc14_get_vc_env(plat_spec)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\site-packages\setuptools\msvc.py", line 268, in _msvc14_get_vc_env
raise distutils.errors.DistutilsPlatformError(
distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
* Using Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)]
* USE_PKGCONFIG: False
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
In order to avoid dependencies and NIH, we want to use diskcache as the caching backend instead of HDF5. This is a zero-dependency solution that uses SQLight as an ACID backend. This is much safer and easier, plus it has a lot of nice features such as expiry, caching strategies, statistics, etc.
Added on GitLab by @smolvik1 2019-06-18
When reading missing values followed by reading valid values, the following error occurs during cache store:
ValueError: invalid combination of [values_axes] on appending data [name->values_block_0,cname->values_block_0,dtype->float64,kind->float,shape->(1, 721)] vs current table [name->values_block_0,cname->values_block_0,dtype->bytes24,kind->string,shape->None]
When reading valid values followed by reading missing values, the following error occurs during cache store:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
When reading missing values twice in a row, the original data are None while the data read from cache are NaN.
It seems as if at least the ODBC based handlers return None
instead of NaN
, and this is handled in an unwanted way by the cache. Probably better to convert None
to np.nan
before storing to cache and returning results.
Hi,
I'm trying to get the most recent version of this package because I'm having the JSON parsing issue. However when I try to update directly from github, I get:
Failed to clone 'extratests' a second time, aborting
Which makes sense, since that folder is a dead link
The .read function (tags, start_time, end_time, time_interval etc.) seems to return data as a list, one element per tag, with all information. When I want to write the data to a .csv file I would like to convert it to a format that is easier to work with - e.g. a "time" column with the time data and one column "tag-name" with the tag values. Any tips?
Aspen IP.21 ODBC returns a single status value which is
PI ODBC and Web API can return three booleans that may be relevant:
PI ODBC can also return Status, which is documented here. It is somewhat more detailed, but probably unnecessarily so.
Assuming AspenOne can provide the same as IP.21 ODBC,, an initial suggestion is to combine the three booleans from PI into a numerical value corresponding to Aspen IP.21 ODBC nomenclature. Assuming that Good and Questionable are mutually exclusive: Status = Questionable + 2*(1-Good) + 4 * Substituted
Calling get_units() for non-existing tag results in a crash. Provide user with informative feedback.
Probably similar issue for other calls that assume existing tags.
In the case when a tag (e.g. "tag_id") does not exist, tagreader returns a dataframe with a column named "tag-id" and numpy nan values. When get_status=True and a tag does not exist tagreader returns the "tag_id" column with nan values, but not "tag_id::status" with nan values.
This could of course be handled by the code calling tagreader, but I think it makes sense for tagreader to return "tag_id::status" with nan values since this column is expected when get_status=True is used and a missing tag is passed to tagreader.
Hi,
It seems like tagreader cannot find anything from SNA, but it is working completely fine for Snorre B. Yes, I have access to the data.
Using the following code to search for 11#35-DENS
causes Requests.JSONDecodeError.
import os
import datetime
import tagreader
def connect_to_aspen():
print("Venter på svar frå aspen")
API = "aspenone"
sources = tagreader.list_sources(API)
if "MO-IP21AB" not in sources:
print("Kunne ikkje koble til MO-IP21AB")
os._exit(0)
else:
source = "MO-IP21AB"
client = tagreader.IMSClient(source, API)
client.connect()
return client
if __name__ == "__main__":
pp = "11#35-DENS"
end = datetime.date.today()
start = end + datetime.timedelta(days=-30)
client = connect_to_aspen()
m = client.search(pp)
print(m)
Backtrace:
Traceback (most recent call last):
File "C:\Users\hsysl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\requests\models.py", line 972, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "f:\tettleiksmeister\main.py", line 24, in <module>
m = client.search(pp)
File "C:\Users\hsysl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tagreader\clients.py", line 273, in search
return self.handler.search(tag, desc)
File "C:\Users\hsysl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tagreader\web_handlers.py", line 349, in search
description = self._get_tag_description(tagname)
File "C:\Users\hsysl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tagreader\web_handlers.py", line 403, in _get_tag_description
j = res.json()
File "C:\Users\hsysl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\requests\models.py", line 976, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The code is currently a bit messy and error-prone in order to accommodate measurements as both Value and Value.Value since the latter is in use by digitalsets and by summary data. However, error codes are also reported as Value.Value, resulting in erroneous interpretation or error codes as actual data. Probably requires a rewrite of read_tag() result handling.
Ref mail from Juan today.
Interpolated results contain end-points, so when specifying e.g. start_time 12:00:00 and end_time 13:00:00 with sampling interval 60s, the results are of length 61. This is fine and expected. However, aggregated result lengths are sometimes 60, other times 61, depending on handler. Behavior must be the same, regardless of handler.
PI ODBC: 61
IP.21 ODBC: 60
PI WEB API: 60
AspenOne: 61
The half-interval offset added in get_next_timeslice is probably relevant.
Create new IMSClient method and corresponding handler methods which let users perform raw queries against SQL database or Web API.
Downloading tables-3.8.0.tar.gz (8.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 8.8 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
ld: library not found for -lhdf5
clang: error: linker command failed with exit code 1 (use -v to see invocation)
cpuinfo failed, assuming no CPU features: 'flags'
* Using Python 3.11.2 (main, Mar 8 2023, 10:10:03) [Clang 14.0.0 (clang-1400.0.29.202)]
* Found cython 0.29.35
* USE_PKGCONFIG: True
.. ERROR:: Could not find a local HDF5 installation.
You may need to explicitly state where your local HDF5 headers and
library can be found by setting the HDF5_DIR
environment
variable or by using the --hdf5
command-line option.
[end of output]
If querying for raw data (read_type=ReaderType.RAW
) but also specifying the frequency argument a warning could be thrown to inform the user that the frequency is not used.
When reading tags that are not of numeric type, the tag is converted to a NaN value. This is due to line 489 of webhandlers.py and can be resolved by just removing the line. The line attempts to convert the tag to a numeric value, but it really should first check that the tag is numeric in the first place or allow the user to specify which tags are numeric.
# Ensure non-numericals like "1.#QNAN" are returned as NaN
df["Value"] = pd.to_numeric(df.Value, errors="coerce")
https://github.com/equinor/tagreader-python/blob/master/tagreader/web_handlers.py
when tag is not existing, get_units fails and returns:
Error using web_handlers>_get_tag_unit (line 372)
Python Error: KeyError
while get_descriptions returns empty string
Ref mail from Juan earlier today:
ValueError: invalid combination of [values_axes] on appending data [name->values_block_0,cname->values_block_0,dtype->float64,kind->float,shape->(1, 10000)] vs current table [name->values_block_0,cname->values_block_0,dtype->bytes24,kind->string,shape->None]
This happens for at least some tags (but not all) using specific query parameters. Other tags using the same parameters work fine. Changing the interval from e.g. 60 seconds to 600 seconds for a tag that fails makes the query work. Seems rather idiosyncratic. Disabling the cache makes the query work.
Issue added on GitLab by @knudsvik 2018-06-12:
Include pull of metadata for collected tags. Especially tag description and units could be useful, but maybe range also?
if made available these data can be used for column headings, label descriptions etc.
Error in read_tag() during attempted timestamp conversion due to missing key "Timestamp" in empty df from json.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.