progress / iceberg Goto Github PK

Updating iceberg to latest March release and running oemanager status is not working, it is returning:

Buildfile: /usr3/wrk/traininga/utils/oemanager.xml

status:
     [echo] Code Release: 2023-04-23T17:24:41.290
     [echo]
     [echo] DLC Home: /usr1/dlc
     [echo] OpenEdge Version: 12.2.11
   [PCTRun] Starting output to file: status_traininga_2023-04-23T17_33_39.797-07_00.txt ...
   [PCTRun]
   [PCTRun] Utility Runtime: 12.2.11.0.1427
   [PCTRun] Report Executed: 2023-04-23T17:33:39.798-07:00
   [PCTRun]  PASOE Instance: http://localhost:17160
   [PCTRun] Invalid handle.  Not initialized or points to a deleted object. (3135)
   [PCTRun]
   [PCTRun] Manager Properties
   [PCTRun]
   [PCTRun] No MSAgents running
   [PCTRun]
   [PCTRun] Session Manager Metrics (Not Enabled)
   [PCTRun]
   [PCTRun] Client HTTP Sessions: 0
   [PCTRun]

BUILD SUCCESSFUL

Reverting back to an older build of iceberg (version before utils.zip deployment to PAS instance), I don't run into the same issue.

Not sure why this is happening, appreciate any help. would like to keep my local iceberg instance in sync with latest from this repo.

We run progression test on our PAS. This generates over a thousand Web Service Requests. Takes several hours to finalize.
All requests are getting through with a few exceptions. So now we would like to trace on memory leaks using the iceberg tool.
When I activate the metrics on the PAS each request on the Web Service Requests is crashing on a potential leak.

2022-04-12T10:38:34.509+0200 010664 004528 1 AS-7 CcsLevelServices:w:00001ae4 DEBUG WebservicesHandler - ctor
2022-04-12T10:38:34.512+0200 010664 004528 1 AS-7 CcsLevelServices:w:00001ae4 DEBUG WebservicesHandler.HandleRequest - start
2022-04-12T10:38:37.881+0200 010664 004528 1 AS-7 CcsLevelServices:w:00001ae4 MSAS Thread (7) crash in Session AS-7. Potential leak 19MB.
2022-04-12T10:38:37.881+0200 010664 004528 4 AS-7 CcsLevelServices:w:00001ae4 MSAS sending session info to broker : num_user_sessions= 0
2022-04-12T10:38:37.881+0200 010664 004528 4 AS-7 CcsLevelServices:w:00001ae4 MSAS sending session info to broker : sock= 1132
2022-04-12T10:38:37.881+0200 010664 004528 4 AS-7 CcsLevelServices:w:00001ae4 MSAS sending session info to broker : disown sock= 1132
2022-04-12T10:38:37.881+0200 010664 004528 4 AS-7 CcsLevelServices:w:00001ae4 MSAS sending session info to broker : own sock= 1132
2022-04-12T10:38:37.881+0200 010664 004528 4 AS-7 CcsLevelServices:w:00001ae4 MSAS session info message sent ... status= 0
2022-04-12T10:38:37.881+0200 010664 004528 3 AS-7 CcsLevelServices:w:00001ae4 MSAS csMtSendABLSessionInfo : session info message sent to broker: num_sessions= 0
2022-04-12T10:38:37.881+0200 010664 004528 3 AS-7 CcsLevelServices:w:00001ae4 MSAS Adding 'Close all procedure libraries' work to Resource Mgr Queue.
2022-04-12T10:38:37.882+0200 010664 004528 2 AS-7 CcsLevelServices:w:00001ae4 CONN Disconnecting from database dwh, user number 5. (9545)

Big question here is, Why does it crash on a potential memory leak?

Hello,

I am using iceberg to manage our PAS instance, when using the proant status command, the various memory metrics reported do not match what the OS level process is consuming.

In the example below, I have an OEPAS APSV ABL session running that is consuming large amounts of memory, but none of the metrics inside the tool add up to the total memory being consumed by the process.

Buildfile: /usr/local/iceberg/build.xml

status:
     [echo] Code Release: 2022-07-07T11:56:15.191
     [echo] DLC Home: /usr3/dlc
     [echo] OpenEdge Version: 12.2.10
   [PCTRun] Starting output to file: status_training_2022-10-27T09_51_03.891-07_00.txt ...
   [PCTRun]
   [PCTRun] Utility Runtime: 12.2.10.0.1386
   [PCTRun] Report Executed: 2022-10-27T09:51:03.894-07:00
   [PCTRun]  PASOE Instance: http://localhost:17128
   [PCTRun]
   [PCTRun] ABL Application Information [training - v12.2.10 ( 2022-06-10 )]
   [PCTRun]     WebApp: ROOT
   [PCTRun]       APSV: ENABLED
   [PCTRun]       SOAP: DISABLED
   [PCTRun]       REST: DISABLED
   [PCTRun]        WEB: DISABLED
   [PCTRun]
   [PCTRun] Manager Properties
   [PCTRun]             Maximum Agents:           1
   [PCTRun]             Minimum Agents:           1
   [PCTRun]             Initial Agents:           1
   [PCTRun]     Max. Connections/Agent:         300
   [PCTRun]     Max. ABLSessions/Agent:         300
   [PCTRun]         Idle Conn. Timeout:     300,000 ms (00:05:00.000)
   [PCTRun]       Idle Session Timeout:   1,800,000 ms (00:30:00.000)
   [PCTRun]         Idle Agent Timeout:   1,800,000 ms (00:30:00.000)
   [PCTRun]      Idle Resource Timeout:           0 ms (00:00:00.000)
   [PCTRun]         Conn. Wait Timeout:       3,000 ms (00:00:03.000)
   [PCTRun]       Request Wait Timeout:      15,000 ms (00:00:15.000)
   [PCTRun]     Initial Sessions/Agent:          60
   [PCTRun]     Min. Avail. Sess/Agent:           1
   [PCTRun]
   [PCTRun] > Agent PID 14484034: AVAILABLE
   [PCTRun]     Est. Agent Lifetime: 01:09:08.549
   [PCTRun]     DynMax ABL Sessions:          300
   [PCTRun]      Total ABL Sessions:           33
   [PCTRun]      Avail ABL Sessions:           32
   [PCTRun]        Open Connections:            2
   [PCTRun]         Overhead Memory: 866,162 KB
   [PCTRun]
   [PCTRun]     SESSION ID      STATE           STARTED                         LIFETIME        SESS. MEMORY    BOUND/ACTIVE CLIENT SESSION
   [PCTRun]                4    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       7,449 KB      -
   [PCTRun]                7    ACTIVE          10/27/2022 08:41:55.463+07:00   01:09:08.549       5,434 KB      -
   [PCTRun]                8    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]                9    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               10    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               11    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               12    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               13    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               14    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               15    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               16    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               17    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               18    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               19    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               20    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               22    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               23    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               26    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               29    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               31    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               32    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               34    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               35    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               38    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               42    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               45    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               49    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               50    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               53    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               56    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               62    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               63    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]               65    IDLE            10/27/2022 08:41:55.463+07:00   01:09:08.549       3,521 KB      -
   [PCTRun]       Active Agent-Sessions: 1 of 33 (3% Busy)
   [PCTRun]     Utilized Agent-Sessions: 2 of 33 (>3,522 KB)
   [PCTRun]        Approx. Agent Memory: 988,192 KB
   [PCTRun]
   [PCTRun] Session Manager Metrics (Count-Based)
   [PCTRun]            # Requests to Session:          201
   [PCTRun]           # Agent Responses Read:          473 (0 Errors)
   [PCTRun]         # Agent Requests Written:          404 (0 Errors)
   [PCTRun]     Concurrent Connected Clients:            1 (Max: 6)
   [PCTRun]       # Reserve ABLSession Waits:            0
   [PCTRun]     # Reserve ABLSession Timeout:            0
   [PCTRun]
   [PCTRun] Client HTTP Sessions: 1
   [PCTRun]     STATE     SESS STATE  BOUND     LAST ACCESS / STARTED           ELAPSED TIME  SESSION MODEL    ADAPTER   SESSION ID                                                                                                                                                                                              REQUEST ID
   [PCTRun]
   [PCTRun]     RUNNING   RESERVED    NO        2022-10-27T09:50:59.569-0700    00:00:04.498  SESSION_FREE     APSV      7F9C029DBBBA14BFD1AC1E82A2B0499FD2753E3EB092.training                                                                                                                                                 ROOT:a:0000020e
   [PCTRun]     |- ClientConn: 172.16.104.254   2022-10-27T09:50:59.569-0700    00:00:04.498  Proc: mytest123.p
   [PCTRun]     |-- AgentConn: PID 14484034  RESERVED  Agent: xxx-xxxxx/172.16.4.16:17129  Local: /172.16.4.16:37852
   [PCTRun]

BUILD SUCCESSFUL
Total time: 0 seconds

Looking at the OS level topas utility, it reports 26GB for the _mproapsv process.

Name           PID  CPU%  PgSp Owner   
_mproaps   14484034  2.0 26.3G prosvr  
java       16843502  1.8 39.9M root    
java        5505790  0.1  130M prosvr  
perl5.28   22086076  0.1 5.43M lpar2rrd
reaffin      459030  0.0  640K root    
java       21234068  0.0  124M prosvra 
bash       17105498  0.0 1.05M root    
topas      15270370  0.0 9.75M mirde   
ipsec_lo   22020564  0.0 1020K root    
_mproaps   22348164  0.0 23.2M prosvr  
_mprosrv   23527926  0.0 2.41M prodb
java       19268342  0.0  148M prosvr
_mprosrv   23462254  0.0 2.39M prodb
vtiol        655644  0.0 6.31M root
java       20251242  0.0  148M prosvra
nvme_kpr    5570882  0.0 1.25M root
swapper         520  0.0  448K root
_mprshut   20840770  0.0 2.07M prodb
Exiting from topas1  0.0  872K root

Is there a way to report on what ABL agent session is using up the memory? In this case, my test case program (mytest123.p) is written to allocate a bunch of memory on purpose for testing, but that is not being reported inside the tool.

Is there a way for the tool to report on what agent session is holding onto this memory? In this case, it should be session id 7.

metrics:
[echo] DLC Home: C:\ccs\oe122
[echo] OpenEdge Version: 12.2.14
[delete] Deleting: C:\Application\myIP.properties
[PCTRun] This version of PROGRESS does not allow compiles. (471)
[PCTRun] R-code file not located for "C:\Users\PETERW1\AppData\Local\Temp\pctinit119065757.p". (473)
[PCTRun] ** Unable to run startup procedure C:\Users\PETERW1\AppData\Local\Temp\pctinit119065757.p. (492)

BUILD FAILED
C:\Application\build.xml:802: PCTRun returned: 2

The compiled runtimes are in the delivery package. So, it could run only using those.
Does the PCTRun forces compile? This sould run the metrics.r

    <PCTRun
           dlcHome="${dlcHome}"
           graphicalMode="false"
           procedure="${deploy.dir}/build/metrics${proc.ext}">
        <propath>
            <pathelement path="${basedir}"/>
            <pathelement path="${dlcHome}/tty/netlib/OpenEdge.Net.pl"/>
        </propath>
        <Parameter name="CatalinaBase" value="${pas.path}"/>
        <Parameter name="ABLApp" value="${ablapp}"/>
        <Parameter name="Type" value="${type}"/>
        <Parameter name="Host" value="${localIP}"/>
        <Parameter name="State" value="${state}"/>
        <Parameter name="Opts" value="${opts}"/>
        <Parameter name="Scheme" value="${scheme}"/>
        <Parameter name="Monitor" value="${monitor}"/>
        <Parameter name="Port" value="${port}"/>
    </PCTRun>

Search "proc.ext" (2 hits in 1 file of 1 searched)
C:\Application\build.xml (2 hits)
Line 138:
Line 802: procedure="${deploy.dir}/build/metrics${proc.ext}">

OE 11.7.15. Instance stafrted and running under that version. The instance is newly-created with almost no modifications ( a couple of webhandlers added).
Iceberg running from source; or at least as-is from a git clone on the repo.

The proant status command is run from an 11.7 proenv shell.

In addition to the error 3135 in the output, none of the ABL sessions are shown in the output.

c:\Work\git\iceberg\PAS\Management\OEManager\ABL>proant status
Buildfile: c:\Work\git\iceberg\PAS\Management\OEManager\ABL\build.xml
 
status:
     [echo] Code Release: 2022-11-08T14:50:26.421
     [echo] DLC Home: C:\Progress\OPENED~4
     [echo] OpenEdge Version: 11.7.15
   [PCTRun] Starting output to file: status_dvfs2_2022-12-11T20_16_14.112-05_00.txt ...
   [PCTRun]
   [PCTRun] Utility Runtime: 11.7.15.0.2183
   [PCTRun] Report Executed: 2022-12-11T20:16:14.113-05:00
   [PCTRun]  PASOE Instance: http://localhost:8810/
   [PCTRun]
   [PCTRun] ABL Application Information [dvfs2 - v11.7.15 ( 2022-05-31 )]
   [PCTRun]     WebApp: ROOT
   [PCTRun]       APSV: ENABLED
   [PCTRun]       SOAP: ENABLED
   [PCTRun]       REST: ENABLED
   [PCTRun]        WEB: ENABLED
   [PCTRun]
   [PCTRun] Manager Properties
   [PCTRun]             Maximum Agents:          10
   [PCTRun]             Minimum Agents:           0
   [PCTRun]             Initial Agents:           1
   [PCTRun]     Max. Connections/Agent:          16
   [PCTRun]     Max. ABLSessions/Agent:         200
   [PCTRun]         Idle Conn. Timeout:     300,000 ms (00:05:00.000)
   [PCTRun]       Idle Session Timeout:     300,000 ms (00:05:00.000)
   [PCTRun]         Idle Agent Timeout:     300,000 ms (00:05:00.000)
   [PCTRun]      Idle Resource Timeout:           0 ms (00:00:00.000)
   [PCTRun]         Conn. Wait Timeout:       3,000 ms (00:00:03.000)
   [PCTRun]       Request Wait Timeout:      15,000 ms (00:00:15.000)
   [PCTRun]     Initial Sessions/Agent:           2
   [PCTRun] Invalid handle.  Not initialized or points to a deleted object. (3135)
   [PCTRun]
   [PCTRun] > Agent PID 70128: AVAILABLE
   [PCTRun]     Est. Agent Lifetime: 00:12:08.576
   [PCTRun]
   [PCTRun]     SESSION ID      STATE           STARTED                         LIFETIME        SESS. MEMORY    BOUND/ACTIVE CLIENT session
   [PCTRun]       active Agent-Sessions: 0 of 0 (0% Busy)
   [PCTRun]     Utilized Agent-Sessions: 0 of 0 (>801 KB)
   [PCTRun]        Approx. Agent Memory: 0 KB
   [PCTRun]
   [PCTRun] Session Manager Metrics (Count-Based)
   [PCTRun]
   [PCTRun] Client HTTP Sessions: 5
   [PCTRun]     STATE     SESS STATE  BOUND     LAST ACCESS / STARTED           ELAPSED time  session MODEL    ADAPTER   session ID                                                     REQUEST ID
   [PCTRun]
   [PCTRun]     RUNNING   RESERVED    NO        2022-12-11T20:12:06.732-0500    00:04:07.490  SESSION_FREE     WEB       UkA_pelZQS6L61_DxpBbYA                                       ROOT:w:0000001d
   [PCTRun]     |- ClientConn: [127.0.0.1](http://127.0.0.1/)        2022-12-11T20:12:06.727-0500    00:04:07.495  Proc: Progress.Web.InternalWebHandler:HandleRe
   [PCTRun]     |-- AgentConn: PID 70128  RESERVED  Agent: /0.0.0.0:62002  Local: /192.168.42.123:59475

Hi,

Using iceberg to monitor PAS instance, the Utilized Agent-Sessions reports a non-zero value when there is nothing running, and I am trying to understand what to make of this.

Is this the same as SessionManager's maxConcurrentClients metric? I noticed in the bracket it says (>3,414 KB).

What does this mean?

[echo] Code Release: 2023-04-23T17:24:41.290 [echo] [echo] NOTICE: Utilities are being executed outside of a PAS instance! [echo] DLC Home: /usr3/dlc [echo] OpenEdge Version: 12.2.11 [PCTRun] Starting output to file: status_training_2023-05-17T07_57_42.763-07_00.txt ... [PCTRun] [PCTRun] Utility Runtime: 12.2.11.0.1427 [PCTRun] Report Executed: 2023-05-17T07:57:42.763-07:00 [PCTRun] PASOE Instance: http://localhost:17128 [PCTRun] [PCTRun] ABL Application Information [training - v12.2.11 ( 2022-11-04 )] [PCTRun] WebApp: ROOT [PCTRun] APSV: ENABLED [PCTRun] SOAP: DISABLED [PCTRun] REST: DISABLED [PCTRun] WEB: DISABLED [PCTRun] [PCTRun] Manager Properties [PCTRun] Maximum Agents: 1 [PCTRun] Minimum Agents: 1 [PCTRun] Initial Agents: 1 [PCTRun] Max. Connections/Agent: 300 [PCTRun] Max. ABLSessions/Agent: 300 [PCTRun] Idle Conn. Timeout: 300,000 ms (00:05:00.000) [PCTRun] Idle Session Timeout: 1,800,000 ms (00:30:00.000) [PCTRun] Idle Agent Timeout: 1,800,000 ms (00:30:00.000) [PCTRun] Idle Resource Timeout: 300,000 ms (00:05:00.000) [PCTRun] Conn. Wait Timeout: 3,000 ms (00:00:03.000) [PCTRun] Request Wait Timeout: 15,000 ms (00:00:15.000) [PCTRun] Initial Sessions/Agent: 60 [PCTRun] Min. Avail. Sess/Agent: 1 [PCTRun] [PCTRun] > Agent PID 12845746: AVAILABLE [PCTRun] Est. Agent Lifetime: 22:39:12.937 [PCTRun] DynMax ABL Sessions: 300 [PCTRun] Total ABL Sessions: 60 [PCTRun] Avail ABL Sessions: 60 [PCTRun] Open Connections: 0 [PCTRun] Overhead Memory: 171,222 KB [PCTRun] [PCTRun] SESSION ID STATE STARTED LIFETIME SESS. MEMORY BOUND/ACTIVE CLIENT SESSION [PCTRun] 4 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 12,616 KB - [PCTRun] 7 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 8 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 9 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 10 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 11 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 12 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 7,163 KB - [PCTRun] 13 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 14 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 15 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 16 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 13,034 KB - [PCTRun] 17 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 18 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 19 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 20 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 21 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 22 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 23 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 24 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 25 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 26 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 27 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 28 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 29 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 30 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 31 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 32 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 33 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 34 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 35 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 36 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 37 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 38 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 39 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 40 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 41 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 42 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 43 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 44 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 45 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 46 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 47 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 48 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 49 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 50 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 51 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 52 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 53 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 54 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 55 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 56 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 57 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 58 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 59 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 60 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 61 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 62 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 63 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 64 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] 65 IDLE 05/16/2023 09:18:29.937+07:00 22:39:12.937 3,413 KB - [PCTRun] Active Agent-Sessions: 0 of 60 (0% Busy) [PCTRun] Utilized Agent-Sessions: 3 of 60 (>3,414 KB) [PCTRun] Approx. Agent Memory: 398,605 KB [PCTRun] [PCTRun] Session Manager Metrics (Count-Based) [PCTRun] [PCTRun] Client HTTP Sessions: 0 [PCTRun]

On Windows, when Progress is installed using PAS for OE PROD Bundle, oemanager.bat does not work.

proenv>oemanager.bat status
C:\OpenEdge\WRK\training\utils\oemanager.xml:621: Execute failed: java.io.IOException: Cannot run program "C:\Progress\OpenEdge\bin_progres" (in directory "C:\OpenEdge\WRK\training\utils"): CreateProcess error=2, The system cannot find the file specified

Using OE 12.8.1 on Windows 2016.

We install PASOE application server via the OE PROD Bundle, but it looks like iceberg has a dependency on _progress binary, and that does not ship with this license SKU.

Is this expected and what is the minimum licensing on Windows required to run Iceberg?

This is more of an enhancement suggestion than an issue/bug.

Using the APSV adapter (state-free) in PASOE, if we are dealing with stuck ABL session and need to terminate them (trimsingle), we need to pass in the AgentSessionId.

Looking at the status output, it would be very useful to have the AgentSessionId somewhere with the Client HTTP Sessions output, that area has all the information (proc and IP) one would need to quickly identify the bad program but only from Session Manager perspective. See Screenshot.

I know we can use the "stacks" option, and that prints both the AgentSessionId and BrokerSessionId, it would be great if we could somehow link the AgentSessionId and BrokerSessionId on the "status" output.

Not sure if this request would work well across all the different adapter types (we only use APSV), feel free to close out if does not fit within the scope of how the status output was designed to be used as.

These are very useful tools to monitor PAS, fills the gap since we can no longer use asbman to pull status, stacks and kill procs.

It would be useful (as a feature enhancement) to add the ability to terminate specific sessions/agents instead of all and/or idle, that way we can use this tool to terminate stuck sessions/threads.

Looking at deploying the "monitoring" side of iceberg in production. The collector and monitoring application is deployed on the same machine (OE 12.2.12, AIX).

I can successfully deploy the Monitoring/Collector PASOE instance/database, and the Monitoring/Application code without issues, enabling metrics (pulse). Using default ports, I can get to the web UI and see monitoring data.

The moment I go to deploy Monitoring/Application to a 2nd PASOE instance (server has multiple PASOE instances), I get errors and there are no metrics being reported in the UI for the 2nd instance.

[2023-08-19T10:39:27.235-07:00] B.Intake | INFO - Health data received from production @ http://127.0.0.1:15605 [2023-08-19T10:39:27.236-07:00] B.Intake | ERROR - Error Parsing HealthCheck Dataset for production @ http://127.0.0.1:15605: Error parsing JSON: expected brace, but found bracket. (15358) [2023-08-19T10:39:27.238-07:00] B.Intake | ERROR - Error Processing Health Data: Unable to read HealthCheck JSON into dataset for further processing.

The monitored PASOE instances are configured with identical settings other than the name/ports.

If I wipe everything and start over, this time deploying Monitoring/Application to the instance that was reporting errors processing health data FIRST - it works without any issues. Adding the 2nd instance (that previously worked when deployed to first), starts to experience the same errors.

Seems like the moment we go to add a 2nd instance to be monitored, the 2nd instance reports errors and no metrics are generated.

When enabling TRACE logging, and reviewing the monitor/temp folder, there is a bunch of .json files that get generated that have references to the FIRST instance that monitoring is deployed on, but no reference to the SECOND instance that has errors.

Any idea what we could do to debug this?

I suppose one could create multiple collector/monitor instances to potentially work around this, but it seems like the solution was built to support pulling in metrics from multiple PASOE instances.

Hello,

Iceberg supports dumping stacks for all MSagents as per docs...

oemanager stacks - [RO] Obtain stack information for all MSAgents for an ABL App

It would be useful in some cases to get stacks for a single session, being able to pass in something like -Dsessid=[SESSION_ID] and get stacks for a single session instead of all.

The API to do this is exposed via oemanager.war, specifically: /applications/{appName}/agents/{agentID}/sessions/{sessionID}/stacks

Would it be possible to add this feature?

When using the "locks" option, the database output is trimmed and when there are multiple databases at play, its not known what database is being referenced.

locks:
     [echo] Code Release: 2023-08-07T13:35:46.788
     [echo]
     [echo] DLC Home: /usr3/dlc
     [echo] OpenEdge Version: 12.2.12
   [PCTRun]
   [PCTRun] Scanning for Table Locks from connected PASN clients...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]     Getting lock stats for xxxxx...
   [PCTRun]
   [PCTRun] Usr#        User            Domain          Tenant          Database                Table           Flags                   PID     SessionID
   [PCTRun] 4         [email protected]                                  /usr3/xxxxdbs/                                               20775274         69
   [PCTRun] 5         [email protected]                                  /usr3/xxxxdbs/                                               20775274         70
   [PCTRun] 6                                                           /usr3/xxxxdbs/                                               15466794         45
   [PCTRun] 7         [email protected]                                    /usr3/xxxxdbs/                                               15466794         43
   [PCTRun] 9         [email protected]                                  /usr3/xxxxdbs/                                               15466794         44
   [PCTRun] 20                                                          /usr3/xxxxdbs/                                               20775274         71
   [PCTRun] 3         [email protected]                                  /usr3/xxxxdbs/                                               20775274         69
   [PCTRun] 8         [email protected]                                  /usr3/xxxxdbs/                                               15466794         44
   [PCTRun] 19                                                          /usr3/xxxxdbs/                                               20775274         71
   [PCTRun]
   [PCTRun] traininga MSAgent PID 15466794:
   [PCTRun]
   [PCTRun]     Call Stack for Session ID #43:
   [PCTRun]             xxxxx.p

BUILD SUCCESSFUL
Total time: 3 seconds

Looks like it's trying to output the full path, and even though the full path to the database is short, it only prints partial output of the path and not the database name itself.

I suggest dropping the path, and only including the db name.

Looks like there is already a function to figure out the database logical name at the start of the script (Getting lock stats for ....), one could use that instead of the full path.

I have implemented the monitor and application on OpenEdge 12.2.3.
When activate the metrics on the application I encountered the following behavior.
The metrics was set on the available agent. The agent shutdown and a new agent was started.
The new agent was not available during the activate procedure. Basically the monitor on this application was only pushing his short live time info to the monitor.

Solution is an upgrade to 12.2.7 or higher.

Hello,

I have iceberg monitoring deployed, and noticed that when its running for longer than a few days it has issues rendering the agent activity tab. See this screenshot. The tab is always in spinning circle mode, I waited up to 20 minutes before aborting the call.

I tried different browser and clearing caches. The overview, session activity, and health trend tabs do not seem to be affected.

While it is spinning, on the backend I can see that the _mproapsv process is very busy.

root 21430980 16187812 261 Aug 31 - 219:06 /usr1/dlc/bin/_mproapsv -logginglevel 2 -logfile /usr1/wrk/monitor/logs/monitor.agent.{yyyy-mm-dd}.log -uburl AppServerDC://0.0.0.0:58731/ -logname monitor -logentrytypes ASPlumbing,DB.Connects -ubpropfile /usr1/wrk/monitor/conf/openedge.properties -ASID 1 -ipver IPv4 -sectok XXXXXXXXXXXXXXXXXXXXXX -T /usr1/wrk/monitor/temp -pf /usr1/wrk/monitor/openedge/startup.pf

What seems to work is if I use the admin function to expunge old metrics, I am able to pull up the agent activity tab.

Seems like it has issues rendering if we've been collecting metrics for more than a few days. My question is, does the tool support long-term metric collection?