Comments (16)
MarkRivers
commented on June 8, 2024
I am not sure. One thing I see is interesting. The interval between errors is almost always an integer number of 30 seconds, and is much more commonly an integer multiple of 60 seconds, i.e. exactly the same seconds and ms:
rad1-log.txt
2019/09/16 20:20:14.783 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/16 20:23:14.783 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/16 20:24:14.783 drvAmptek::sendCommand error calling CH_.SendCommand(0)
rad2-log.txt Note that there was a period of almost an hour with no errors.
2019/09/16 20:16:42.746 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/16 20:17:12.745 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/16 20:17:42.746 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/16 21:15:11.746 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/16 21:16:11.746 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/16 21:17:11.746 drvAmptek::sendCommand error calling CH_.SendCommand(0)
However starting about 17:20:30 both IOCs began getting errors exactly once a minute, and the errors were withing 26 ms of each other.
This suggests it could be due to a common cause like network traffic interference?
radlog1
2019/09/17 17:20:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:20:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:22:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:22:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:26:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:26:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:27:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:27:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:28:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:28:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:29:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:29:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:30:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:30:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:31:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:31:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:32:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:32:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:33:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:33:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:34:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:34:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:35:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:35:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:36:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:36:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:37:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:37:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:38:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:38:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/17 17:39:30.368 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:39:30.368 LAB-090Row:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
radlog2
2019/09/17 17:20:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:22:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:26:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:28:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:29:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:30:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:31:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:32:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:33:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:34:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:35:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:36:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:37:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:38:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:39:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:40:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:41:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:42:29.345 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:43:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:45:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:46:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:47:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:49:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:50:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:51:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:52:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:53:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:54:29.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:54:58.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:55:28.345 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:55:58.345 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:56:28.345 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:56:58.345 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:57:28.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:57:58.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:58:28.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 17:58:58.346 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 19:02:29.446 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 19:29:58.446 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 19:30:58.446 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 19:31:58.446 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 19:33:28.446 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/17 19:34:28.446 drvAmptek::sendCommand error calling CH_.SendCommand(0)
I have forgotten what periodic polling the driver does when it is not acquiring. What is the rate and what is it reading?
from mca.
MarkRivers
commented on June 8, 2024
I am assuming that these errors are when it is reading the status via ReadStatus record. What scan rate are you using for ReadStatus.SCAN?
from mca.
hinxx
commented on June 8, 2024
Thank you for the debugging notes!
The rate of ReadStatus.SCAN is set to 1 second.
I have forgotten what periodic polling the driver does when it is not acquiring. What is the rate and what is it reading?
The acquisition has been started and is running for days now. I'm sure what you mean with the question above.
This suggests it could be due to a common cause like network traffic interference?
Could be. It is a lab network and I'm not sure what routing equipment is in place.
I think I'll have both devices connected to a dedicated machine, without using the network they're on right now to see if that helps.
from mca.
MarkRivers
commented on June 8, 2024
I have forgotten what periodic polling the driver does when it is not acquiring. What is the rate and what is it reading?
The acquisition has been started and is running for days now. I'm sure what you mean with the question above.
I asked that question before I looked at the code. When ReadStatus processes it reads the high voltage, temperature and a few other things. Since you have ReadStatus.SCAN=1 second it is doing that at 1 Hz, i.e. 60 times per minute. What seems really strange is that the errors almost always happen exactly every 30'th or 60'th time.
I have pushed a new version that adds a ASYN_TRACEIO_DRIVER debugging in sendCommand() if it succeeds. If you enable ASYN_TRACEIO_DRIVER for a while you will send not only the error messages but also the success messages. It will be interesting to see if indeed it is succeeding at 1 Hz and failing once a minute.
from mca.
hinxx
commented on June 8, 2024
I have the debug output with the above change. This is with no errors, as far as I can tell. It seem quite chatty.
I can observe CH_.SendCommand(3) about 200 ms into each second, and then CH_.SendCommand(0) until that second is done. The CH_.SendCommand(0) commands come in a burst with 100 ms in between the calls.
2019/09/23 10:37:33.273 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 10:37:33.323 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.323 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.374 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.374 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.449 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.450 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.549 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.549 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.649 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.650 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.749 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.749 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.849 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.850 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.949 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.999 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.999 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:34.262 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 10:37:34.313 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.313 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:34.364 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.364 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:34.449 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.450 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:34.549 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.550 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:34.649 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.650 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:34.749 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.750 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:34.849 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.850 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:34.949 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.999 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:34.999 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:35.273 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 10:37:35.323 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.323 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:35.374 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.374 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:35.449 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.450 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:35.549 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.550 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:35.649 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.650 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:35.749 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.750 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:35.849 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.850 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:35.949 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.999 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:35.999 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:36.262 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 10:37:36.313 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.313 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:36.364 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.364 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:36.449 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.450 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:36.549 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.550 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:36.649 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.650 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:36.749 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.749 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:36.849 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.849 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:36.949 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.999 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:36.999 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:37.263 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 10:37:37.313 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.313 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:37.364 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.364 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:37.449 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.450 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:37.549 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.550 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:37.649 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.650 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:37.749 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.750 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:37.849 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.850 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:37.949 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.999 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:37.999 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:38.263 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 10:37:38.313 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.313 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:38.364 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.364 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:38.449 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.450 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:38.549 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.550 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:38.649 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.650 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:38.749 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.750 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:38.849 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.850 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:38.949 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.999 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:38.999 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:39.273 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 10:37:39.323 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:39.323 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:39.374 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:39.374 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:39.449 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:39.450 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:39.549 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:39.549 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:39.649 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:39.650 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:39.749 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:39.750 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:39.849 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:39.850 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:39.949 drvAmptek::sendCommand called CH_.SendCommand(0) OK
from mca.
hinxx
commented on June 8, 2024
I disabled the trace output for a while. Then as is started having trouble, I re-enabled the trace output, again.
2019/09/23 10:44:59.559 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:44:59.660 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:44:59.759 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:44:59.860 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:44:59.959 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:44:59.959 LAB:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/23 10:45:00.019 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.080 drvAmptek::sendCommand error calling CH_.SendCommand(3)
2019/09/23 10:45:00.140 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.201 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.261 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.360 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.460 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.560 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.659 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.760 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.760 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:00.859 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.860 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:00.959 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:00.959 LAB:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/23 10:45:01.019 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.019 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.080 drvAmptek::sendCommand error calling CH_.SendCommand(3)
2019/09/23 10:45:01.140 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.140 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.201 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.201 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.261 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.261 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.360 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.360 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.460 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.460 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.560 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.560 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.660 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.660 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.759 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.760 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.860 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.860 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:01.959 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:01.959 LAB:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/23 10:45:02.019 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.019 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.080 drvAmptek::sendCommand error calling CH_.SendCommand(3)
2019/09/23 10:45:02.140 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.140 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.201 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.201 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.261 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.261 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.359 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.360 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.460 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.460 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.560 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.560 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.660 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.660 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.759 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.760 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.860 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.860 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:02.959 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:02.959 LAB:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/23 10:45:03.019 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.019 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.080 drvAmptek::sendCommand error calling CH_.SendCommand(3)
2019/09/23 10:45:03.140 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.140 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.201 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.201 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.261 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.261 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.360 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.360 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.460 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.460 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.560 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.560 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.660 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.660 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.760 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.760 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.860 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.860 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:03.959 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:03.959 LAB:RAD1:ReadStatus devAsynInt32::processCallbackOutput process error
2019/09/23 10:45:04.019 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 10:45:04.020 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:45:04.080 drvAmptek::sendCommand error calling CH_.SendCommand(3)
2019/09/23 10:45:04.140 drvAmptek::sendCommand error calling CH_.SendCommand(0)
from mca.
hinxx
commented on June 8, 2024
BTW, this is still on the lab LAN, not point-to-point. The EPICS IOC is not able to talk to the device if I use run DHCP server (on the machine as the IOC) and hook up the detector to it through a network switch; isolated from the lab LAN. What is strange is that the detector asks for IP, gets it and I can even ping it. The IOC, as it starts, finds no detectors. Strange.
OTOH, I left the other unit over the weekend connected using USB link and there there were zero errors in the IOC console. No warning messages or anything else.
from mca.
hinxx
commented on June 8, 2024
If I change these settings I can see different rate of the CH_.SendCommand(0) messages, which probably makes sense:
from mca.
hinxx
commented on June 8, 2024
Here is a long session logged with screen command. The asyn driver trace was enabled to see all commands.
We can observe situations like this:
2019/09/23 11:52:54.617 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 11:52:54.617 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 11:52:54.716 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 11:52:54.777 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 11:52:54.777 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 11:52:55.090 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 11:52:55.141 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 11:52:55.141 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
One command fails while many others prior to failed one and after work fine.
Then, couple of hours later, the device stops responding to any requests from the IOC, and does not recover:
2019/09/23 14:26:26.617 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 14:26:26.617 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 14:26:26.716 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 14:26:26.777 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 14:26:26.777 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 14:26:26.837 drvAmptek::sendCommand error calling CH_.SendCommand(3)
2019/09/23 14:26:26.898 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 14:26:26.898 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 14:26:26.958 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 14:26:26.958 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 14:26:27.027 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 14:26:27.027 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 14:26:27.127 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 14:26:27.127 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 14:26:27.227 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 14:26:27.227 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 14:26:27.327 drvAmptek::sendCommand error calling CH_.SendCommand(0)
2019/09/23 14:26:27.327 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
The detector did not respond to the pings anymore.
I had to power cycle the detector to get it to respond to the IOC commands again.
For the record, since this happened during working hours, and my laptop with the amptek IOC also being connected to the same LAN as the detector, I have not noticed any network problems myself.
from mca.
hinxx
commented on June 8, 2024
Another, shorter, session logged with screen. This happened while I was writing previous comment to this issue. The device all of a sudden stopped responding to IOC commands.
Restating the IOC does not get device to respond:
# Use this line for Ethernet
drvAmptekConfigure("MCA", 0, "172.30.150.74")
Searching for Amptek modules on network:
Found 0 Amptek units
Searching for Amptek modules on network:
Found 0 Amptek units
Searching for Amptek modules on network:
Found 0 Amptek units
2019/09/26 08:47:25.804 drvAmptek::connectDevice ERROR: Network DPP device 172.30.150.74 not found, total devices found=0
2019/09/26 08:47:25.804 drvAmptek::connectDevice error calling SendCommand for XMTPT_SEND_STATUS
2019/09/26 08:47:25.804 drvAmptek::drvAmptek cannot connect to device on interface type=0 addressInfo=172.30.150.74, status=3
from mca.
MarkRivers
commented on June 8, 2024
Is it possible that this coincides with a network security scan being done? It could be that your laptop handles the security scan fine, but the Amptek does not. Is it possible to connect the Amptek directly to a second NIC and see if the problem goes away?
This seems like something you can raise with Amptek technical support.
from mca.
MarkRivers
commented on June 8, 2024
I have the debug output with the above change. This is with no errors, as far as I can tell. It seem quite chatty.
I can observe CH_.SendCommand(3) about 200 ms into each second, and then CH_.SendCommand(0) until that second is done. The CH_.SendCommand(0) commands come in a burst with 100 ms in between the calls.
2019/09/23 10:37:33.273 drvAmptek::sendCommand called CH_.SendCommand(3) OK
2019/09/23 10:37:33.323 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.323 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.374 drvAmptek::sendCommand called CH_.SendCommand(0) OK
2019/09/23 10:37:33.374 asynPortDriver:readFloat64: function=7, name=MCA_DWELL_TIME, value=1.000000
2019/09/23 10:37:33.449 drvAmptek::sendCommand called CH_.SendCommand(0) OK
When you were seeing this was the detector acquiring or idle?
from mca.
hinxx
commented on June 8, 2024
When you were seeing this was the detector acquiring or idle?
It was acquiring.
Is it possible to connect the Amptek directly to a second NIC and see if the problem goes away?
Yes, that is on a todo list to try single out the issue, as soon as I find time.
Since all is fine over USB, it is either the DP5G Ethernet controller or our network..
from mca.
hinxx
commented on June 8, 2024
BTW, this is still on the lab LAN, not point-to-point. The EPICS IOC is not able to talk to the device if I use run DHCP server (on the machine as the IOC) and hook up the detector to it through a network switch; isolated from the lab LAN. What is strange is that the detector asks for IP, gets it and I can even ping it. The IOC, as it starts, finds no detectors. Strange.
I think tracked the issue to this one.
It seems that when CDppSocket::BroadCastSendTo() uses INADDR_BROADCAST to find devices over UDP broadcasts on port 3040 it does not send them out on the 'right' network interface nor all interfaces, but only on one (i'm on multihomed PC wifi+wired).
Googled answers suggest that it is up to the routing table to decide where a SINGLE sendto() will end up transmitting the data (https://stackoverflow.com/a/683774). As soon I used 192.168.1.255 (instead of INADDR_BROADCAST) the device was found and is chatting with the IOC as we speak on the point-to-point link (well with a small Ethernet switch in between, but no other devices).
I'm going to try to improve the discovery part of the code to better handle situations like this - will open another github ticket for this. Would you be prepared to accept a pull request for improvement like this?
from mca.
MarkRivers
commented on June 8, 2024
Sure, I'll be happy to accept a pull request.
from mca.
hinxx
commented on June 8, 2024
I've been testing the IOC and the device on an Ethernet peer-to-peer link, PC <--> device.
Over the test period of three days I did not encounter any issues with the communication.
I think it is safe to say that what I'm seeing is not an IOC or device problems, and that this issue can be closed.
from mca.
Related Issues (12)
- Test issue
- DP5G HV value taken from status packet HOT 1
- Amptek USB driver fails to connect HOT 2
- Can't control multiple Amptek via USB on single IOC HOT 2
- Another test issue
- make error reported：Can't find 'scalerSupport.dbd' HOT 2
- make error:'getNumParams' was not declared in this scope HOT 2
- iocAmptek HOT 11
- mca fails to build for win32-x86-static against base-3.15.4 HOT 8
- $(P)PresetReal defined twice in SIS38XX.template HOT 1
- mcaCanberra, mcaAIM, and nmcDemo are built unconditionally on OS X HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mca.