Unexpected Error when accessing webhook search results

[david-mcfall-fxs]

Expected Behavior

Webhook search results should be displayed

Current Behavior

When we click any of the links that use webhooks/search.html, TeamCity displays an unexpected error and a stack trace.

Steps to Reproduce (for bugs)

1. In TeamCity, navigate to Administration/WebHooks
2. Where it says: "There are ## WebHooks configured in this TeamCity Server.", click the "## WebHooks" link.

Your Environment

• tcWebHooks Version: 1.2.4
• TeamCity Version: 2023.05.4
• TeamCity server Operating System: Windows Server 2019
• Are you using a WebHook Template?: Yes
• Dev reported the issue Oct 30th 2023

[netwolfuk]

I'm not able to reproduce this on 1.2.5 and there is no code in the diff from 1.2.4 to 1.2.5 that would affect this.
Can I get more detail on the stack trace? There will be a stack trace on the user's page, and also in the teamcity-server.log

[david-mcfall-fxs]

Trace below:

Trace: java.lang.NullPointerException
at webhook.teamcity.extension.bean.WebhookConfigAndBuildTypeListHolder.(WebhookConfigAndBuildTypeListHolder.java:90)
at webhook.teamcity.extension.bean.ProjectWebHooksBean.addWebHookConfigHolder(ProjectWebHooksBean.java:168)
at webhook.teamcity.extension.bean.ProjectWebHooksBean.buildWithoutNew(ProjectWebHooksBean.java:144)
at webhook.teamcity.extension.WebHookSearchController.doHandle(WebHookSearchController.java:78)
at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:114)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177)
at jetbrains.buildServer.controllers.BaseController.handleRequest(BaseController.java:93)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:51)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at jetbrains.buildServer.maintenance.WebDispatcherServlet.doService(WebDispatcherServlet.java:38)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:8)
at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:43)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at jetbrains.buildServer.https.HttpsRedirectFilter.doFilter(HttpsRedirectFilter.java:31)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at jetbrains.buildServer.web.jsp.JspPrecompilerFilter.doFilter(JspPrecompilerFilter.java:181)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:8)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at jetbrains.buildServer.web.UserIdProviderFilter.doFilter(UserIdProviderFilter.java:7)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at jetbrains.buildServer.web.BannedIPsFilter.doFilter(BannedIPsFilter.java:3)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at jetbrains.buildServer.web.NodeInfoHeaderFilter.doFilter(NodeInfoHeaderFilter.java:7)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108)
at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:2)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:10)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at jetbrains.buildServer.diagnostic.web.HttpRequestsDurationMetricsReporter.doFilter(HttpRequestsDurationMetricsReporter.java:4)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at jetbrains.buildServer.web.HttpSecurityHeadersFilter.doFilter(HttpSecurityHeadersFilter.java:58)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at jetbrains.buildServer.controllers.filters.DisableSessionCookieTokenAuthFilter.doFilter(DisableSessionCookieTokenAuthFilter.java:11)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at jetbrains.buildServer.controllers.filters.ProxyDetailsFilter.doFilter(ProxyDetailsFilter.java:11)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at jetbrains.buildServer.controllers.filters.ClearSecurityContextFilter.doFilter(ClearSecurityContextFilter.java:15)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74)
at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:74)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:29)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)

[david-mcfall-fxs]

Dev has a hunch that it's related to a webhook that was deleted via kotlin script. Perhaps there's a broken relationship causing the crash.

[netwolfuk]

Yes, I am starting to think the same thing. There is a cache that backs the search page, and it has obviously not updated from the edit. I am not sure if I am reacting to the kotlin edit broadcast event.

It might be possible to just edit a webhook in the same project using the webhook UI. That should refresh the cache, but I'll need to check the code. It's worth a try for now.

[netwolfuk]

I found a project on my local teamcity that has VCS config setup. Although, mine is XML, not Kotlin.
I removed a webhook and can still see it in the list. However, trying to edit it or delete it fails.

I then edited another webhook in the UI for the same project, but the cache is only updated for that webhook, not refreshed for all the webhooks in that project.

I can't however reproduce the exact same issue you are having with the search page.

Actions for me:

1. Create a kotlin config and do more testing.
2. Provide a way to refresh the cache. This might be easiest done by refreshing the cache for all webhooks for a project on every edit.
3. Figure out how to receive events when config is updated via config as code (kotlin or XML), and refresh the cache then.

If you have some more detail about what the devs did to get the stack trace that would be good.

[david-mcfall-fxs]

Some actions recalled from dev: "I had multiple tabs open of the project, and I deleted the webhook from this sub-project and another sub-project, I moved a template from the sub-project to root, I added webhooks on the root project, then I went to that sub-project to move the template it was using to root."

[netwolfuk]

Thanks @david-mcfall-fxs for the updated information.

I've converted my Versioned Settings to kotlin. With further testing like the above I still can't reproduce the issue sorry.

For you, if you retart TeamCity, this will fix the issue. The cache is only held in memory, and restarting will call the initialise function to repopulate the cache.

I have raised a support request with Jetbrains to ask how to listen for events when the configuration is edited via VCS.

I have also coded up a fix that rebuilds the cache for all webhooks on a project when a webhook on that project is edited. I will test that and then do a release (assuming I don't get reply from Jetbrains) ASAP. However, this week is very busy already.

Your best next step is to restart TeamCity. This will at least force the cache to rebuild, and should fix the error you're seeing.

[david-mcfall-fxs]

No worries, TC service restarts are not terribly disruptive for us. I restarted the service a few minutes ago and we have the webhook searches working again. Thank you for all your efforts on this.