IPMI consoles disconnect issue with AMI BMC

[whowutwut]

I'm not sure if this is caused by the instability of the AMI BMC with regards to sol activate or if we have multiple sessions established for this node.

But I saw this behavior today:

[root@stratton01 c910env]# rcons fs3
[Enter `^Ec?' for help]
goconserver(2018-03-09T11:03:37-05:00): Hello 10.6.29.1:48462, welcome to the session of fs3
Acquiring startup lock...done
[SOL Session operational.  Use ~? for help]
Info: SOL payload already de-activated
SOL session closed by BMC
Error in SOL session
Could not receive message, error: EOF.
[Enter `^Ec.' to exit]
Session is teminated unexpectedly, retrying....
[Enter `^Ec?' for help]
goconserver(2018-03-09T11:03:52-05:00): Hello 10.6.29.1:48538, welcome to the session of fs3
Acquiring startup lock...done
[SOL Session operational.  Use ~? for help]

Red Hat Enterprise Linux Server 7.4 (Maipo)
Kernel 3.10.0-693.el7.ppc64le on an ppc64le

fs3 login: rooSOL session closed by BMC
Error in SOL session
Could not receive message, error: EOF.
[Enter `^Ec.' to exit]
Session is teminated unexpectedly, retrying....
[Enter `^Ec?' for help]
goconserver(2018-03-09T11:04:06-05:00): Hello 10.6.29.1:48568, welcome to the session of fs3
Acquiring startup lock...done
[SOL Session operational.  Use ~? for help]
Info: SOL payload already de-activated
SOL session closed by BMC
Error in SOL session
Could not receive message, error: EOF.
[Enter `^Ec.' to exit]
Session is teminated unexpectedly, retrying....
[Enter `^Ec?' for help]
goconserver(2018-03-09T11:04:20-05:00): Hello 10.6.29.1:48594, welcome to the session of fs3
Acquiring startup lock...done

[SOL Session operational.  Use ~? for help]

Red Hat Enterprise Linux Server 7.4 (Maipo)
Kernel 3.10.0-693.el7.ppc64le on an ppc64le

fs3 login: Error sending SOL data: FAIL
                                       SOL session closed by BMC
Error in SOL session
Could not receive message, error: EOF.
[Enter `^Ec.' to exit]
Session is teminated unexpectedly, retrying....
Disconnected

This is on Firestone EUH boxes.

[whowutwut]

Could be multiple sessions:

[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol activate
Info: SOL payload already active on another session
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol deactivate
[root@stratton01 c910env]#

Even after deactivate:

[root@stratton01 c910env]# rcons fs3
[Enter `^Ec?' for help]
goconserver(2018-03-09T11:06:50-05:00): Hello 10.6.29.1:48932, welcome to the session of fs3
SOL session closed by BMC
Error in SOL session
Could not receive message, error: EOF.
[Enter `^Ec.' to exit]
Session is teminated unexpectedly, retrying....
Disconnected

But it still looks like it's active...

[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol activate
Info: SOL payload already active on another session
[root@stratton01 c910env]#

So not sure how to find the culprit

[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol activate
Info: SOL payload already active on another session
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol deactivate
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol deactivate
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol deactivate
Info: SOL payload already de-activated
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol activate
Info: SOL payload already active on another session
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol deactivate
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol deactivate
Info: SOL payload already de-activated
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol deactivate
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol deactivate
Info: SOL payload already de-activated
[root@stratton01 c910env]# ipmitool-xcat -I lanplus -U ADMIN -P admin -H 50.3.27.1 sol activate
[SOL Session operational.  Use ~? for help]



SOL session closed by BMC
Error in SOL session
[root@stratton01 c910env]#
[root@stratton01 c910env]#
[root@stratton01 c910env]#

[whowutwut]

@hu-weihua Is this node being used for FVT? I saw reference to it here: c910f03c05k25

[chenglch]

@whowutwut , the console for fs3 is back, I think there is another service trying to connect to the console at that time.

[root@stratton01 ~]# rcons fs3
[Enter `^Ec?' for help]
goconserver(2018-03-16T01:36:06-04:00): Hello 10.6.29.1:44056, welcome to the session of fs3
[Disconnected]
[root@stratton01 ~]# rpower fs3 status
fs3: off

[whowutwut]

@chenglch yep, thanks... I think there are multiple servers connecting to the console, so this causes problems with the AMI bmc...

[hu-weihua]

@whowutwut sorry for reply later, yes, fs3 is being used by daily regression system for p8 physical machine test.