Unable to Generate report using CNES Plugin in SonarQube-7.9 LTS Application.

[alok347]

Issue:

Unable to Generate report using CNES Plugin in SonarQube-7.9 LTS Application.

Expected behaviour

The CNES plugin("cnesreport":"3.2.2 [SonarQube CNES Report]) is installed in the SonarQube7.9.4 Enterprise Edition version in my environment. We have currently 4k+ users and more than 2k projects on the server. The Cnes plugin is throwing a error "{"errors":[{"msg":"An error has occurred. Please contact your administrator"}]}".

Note : the sonarqube application is running behind a reverse proxy front ended by apache 2.4 version. The JVM memory for all the java processes are 8GB. I did not see any resource(memory, CPU etc) crunch while generating the report.

Actual behavior

I was able to reproduce the issue by selecting few projects separately through the webUI. The application processed the request later resulting the application unresponsive. Please refer to the below screenshot.

Also refer to the below server logs

'''2020.11.02 14:10:18 ERROR web[AXUoGUG3DesDejYxAPfD][o.s.s.w.WebServiceEngine] Fail to process request http://sonarserver.com/sonar/api/cnesreport/report?key=example.com&branch=example-tests+&token=xxxxxxxxxxxxxxxxxxxxxxxx=Generate
fr.cnes.sonar.report.exceptions.SonarQubeException: Impossible to reach SonarQube instance.
at fr.cnes.sonar.report.providers.RequestManager.get(RequestManager.java:149)
at fr.cnes.sonar.report.providers.AbstractDataProvider.stringRequest(AbstractDataProvider.java:346)
at fr.cnes.sonar.report.providers.QualityProfileProvider.getQualityProfiles(QualityProfileProvider.java:76)
at fr.cnes.sonar.report.factory.ReportModelFactory.create(ReportModelFactory.java:119)
at fr.cnes.sonar.report.ReportCommandLine.execute(ReportCommandLine.java:135)
at fr.cnes.sonar.plugin.ws.ExportTask.handle(ExportTask.java:81)
at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:110)
at org.sonar.server.ws.WebServiceFilter.doFilter(WebServiceFilter.java:88)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:126)
at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:95)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:87)
at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:71)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:76)
at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.requestid.RequestIdFilter.doFilter(RequestIdFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Fail to request https://sonarserver.com/sonar/api/qualityprofiles/export?language=c&name=TEST
at org.sonarqube.ws.client.HttpConnector.doCall(HttpConnector.java:176)
at org.sonarqube.ws.client.HttpConnector.get(HttpConnector.java:112)
at org.sonarqube.ws.client.HttpConnector.call(HttpConnector.java:99)
at fr.cnes.sonar.report.providers.RequestManager.get(RequestManager.java:147)
... 51 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:609)
at okhttp3.internal.platform.Platform.connectSocket(Platform.java:124)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:221)
at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:196)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:145)
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:192)
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at org.sonarqube.ws.client.HttpConnector.doCall(HttpConnector.java:173)
... 54 common frames omitted
'''

Also we have noticed that there is a increase of CLOSE-WAIT sockets during the activity.

Kindly Look into the issue and assist me on the same. This is a high priority issue as it is affecting the production environment.

Steps to reproduce behaviour

1. goto more.
2. select CNES.
3. select a project from the list of projects(from the drop down menu).
4. Generate the report.
   Throws the exception.

Detection version
CNES Plugin version: "cnesreport":"3.2.2 [SonarQube CNES Report]

[Sancretor]

Hi @alok347
My first guess would be an issue regarding your proxy configuration.
Are you sure that this exact URL you quote (http://sonarserver.com/sonar/) is accessible from within your SonarQube server ?
The CNES Report tool needs to access the SonarQube APIs to gather data, so this URL must be accessible.

[alok347]

Hi @Sancretor ,

The Production application is accessible through web browser. I am able to call web API's from linux console and web browser both. https://sonarserver.com/sonar. Please refer tothe below API output:

{"Health":"GREEN","Health Causes":[],"System":{"Server ID":"ABCDEF789WXYZ","Version":"7.9.3.33349","External User Authentication":"LDAP","High Availability":false,"Official Distribution":true,"Force authentication":true,"Home Dir":"/d/y/sonarqube-7.9.3","Data Dir":"/d/y/sonarqube-7.9.3/data","Temp Dir":"/d/y/sonarqube-7.9.3/temp","Processors":8},"Database":{"Database":"PostgreSQL","Database Version":"9.6.17","Username":"dbuser","URL":"jdbc:postgresql://server/dbname","Driver":"PostgreSQL JDBC Driver","Driver Version":"42.2.5"},

We see many background projects getting analysed. I have also successfully generated reports for the same projects. There is 25% Success rate i would say while generating the report. I want to understand why is that we see such strange behaviour while using the plugin. There are multiple incidents reported by many users while using CNES plugin for generating the reoprt.
Any Idea why do we see CLOSE_WAIT sockets piling up during the cnes report generation process?

[Sancretor]

HI @alok347

Weird behaviour ... we never saw that on our own platform, with an NGINX reverse proxy, and we use it for years.
I don't remember receiving any other issue on our GitHub related to this kind of behaviour.
CLOSE_WAIT sockets piling up and timeout connections are generally symptoms of an issue with a webserver not correctly handling sockets... maybe your Apache or SonarQube directly.
Many things could cause this, but this seems to be related to your own environment.

I don't know how the plugin could be the cause of this problem, so I don't know how we can help you solve this.
This should be investigated by your sysadmins I guess...

[alok347]

Hi @Sancretor

Okay, but why the Cnes plugin is throwing an error "{"errors":[{"msg":"An error has occurred. Please contact your administrator"}]}" when we try to generate a report for any project?

[Sancretor]

The plugin is throwing an error because one of its API calls returns a timeout.
This is probably a consequence of the CLOSE_WAIT sockets issue, as they usually reveal issues in webserver ability to manage sockets, and end up in multiple timeouts for those sockets.

Caused by: java.lang.IllegalStateException: Fail to request https://sonarserver.com/sonar/api/qualityprofiles/export?language=c&name=TEST
...
Caused by: java.net.SocketTimeoutException: connect timed out

[mchezhian]

Is "sonar.ws.timeout" property configurable as command line argument in CNES JAR (Standalone)? What's the default timeout for the backend API calls?

[Sancretor]

Hi

@alok347
We released the 4.0.0 version some time ago, with SQ 8.9 compatibility.
Did you try to use it ?
If so, did it solve the issue ?
We completely change the way the plugin works so these errors should not happen anymore.

@mchezhian
This parameter is a SonarQube parameter and not related to our tool, so I can't answer correctly.
I'm afraid you have to dig inside the official SQ documentation.

[Sancretor]

Without any news, I close this issue.
If anything similar happens again, we'll re-open it.

[alok347]

@Sancretor Sorry for the delayed response. I have installed and tested on SonarQube-8.9.6 version as well. For few projects i get the same error. I'll need to dig a bit deeper on this to get the stacktrace right.
Do you have any suggestion as what can be the reason of the plugin timeout?

[Sancretor]

Hi @alok347
Timeouts could be caused by a lot of things.
If your SonarQube server is under heavy load, if your proxy is under heavy load, if your proxy cannot route correctly requests to the SonarQube server, the SQ project is huge with so much data that the API requests timed out (we never saw that though), and so on...
SonarQube have a parameter to enter "debug" or "info" logging mode; but you also have to look into your proxy who might be the cause, depending on the routing rules you've configured.

[alok347]

Hi Team,

We had few users reporting the same issue again while generating the reports on sonarqube using cnes plugin.
While we are trying to debug the root cause, we thought reproducing this issue on our QA application as well. We did see the access.logs had logstack about the cnes plugin's backend task(various GET API's related to project) but it suddenly stopped.

Also, we restarted the QA as we wanted to test something and since morning today i am getting the below error while generating the report:

2022.05.06 16:25:50 ERROR web[AYCYj9MQABaM8yWdAAwX][o.s.s.w.WebServiceEngine] Fail to process request http://sonarqube.com/sonar/api/cnesreport/report?key=abcd&branch=development&author=Singh+Alok&token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&generation=Generate
fr.cnes.sonar.report.exceptions.SonarQubeException: Impossible to reach SonarQube instance.
at fr.cnes.sonar.report.ReportCommandLine.execute(ReportCommandLine.java:124)
at fr.cnes.sonar.plugin.ws.ExportTask.handle(ExportTask.java:86)
at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:110)
at org.sonar.server.ws.WebServiceFilter.doFilter(WebServiceFilter.java:88)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:126)
at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:95)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:87)
at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:71)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:76)
at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.requestid.RequestIdFilter.doFilter(RequestIdFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
2022.05.06 16:26:06 INFO web[AYCYj9MQABaM8yWdAAws][o.s.u.c.UpdateCenter] The plugin 'abap' version : 3.8.0.2034 has not been found on the update center.

Note: sonarqube is reachable on URL https://sonarqube.com/sonar but the cnes plugin is trying to reach on http://sonarqube.com/sonar which is not accessible on port 80 hence the error we see.

Why is the plugin trying to reach the sonarqube on http?

Let us know your thoughts .

Regards,
Alok Singh

[Sancretor]

Hi @alok347
Thanks for the feedback !
Weird error, as the plugin should use the exact same address you use to contact it...
I'll try to reproduce this error too, and keep you in touch.

[alok347]

@Sancretor ,

Just wanted to check if we made any progress in reproducing the issue.
We are blocked with few reports generation on the application.

Regards,
Alok Singh

[alok347]

Hi @Sancretor ,

Please refer to the details of the SonarQube environment and plugin version:

cnes 3.3.1 version.
SonarQube 7.9

I looked at the source code(RequestManager.java) which triggered the above message:

// Execute the request. final HttpConnector httpConnector = builder.build(); WsResponse response; try { response = httpConnector.call(new GetRequest(path)); } catch (Exception e) { throw new SonarQubeException("Impossible to reach SonarQube instance.", e); }

This is where port 80 is used instead of port 443.

Regards,
Alok Singh

[Sancretor]

Hi @alok347

I just did a test again but I'm still not able to reproduce your error, with an HTTPS front proxy, either in standalone or plugin mode, it works here.

But I see you're still using an old release of the plugin, which does not work the same way as the last ones, 4.1.1 for instance. Did you try with this version ? In plugin mode, it directly uses SonarQube internal classes so no https/http calls anymore. The standalone mode still works with http API calls though.

Otherwise, I must admit I'm running out of ideas...