Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to clone huge repository with large files #1525

Open
StenaviN opened this issue Feb 5, 2021 · 10 comments
Open

Unable to clone huge repository with large files #1525

StenaviN opened this issue Feb 5, 2021 · 10 comments
Assignees
Labels
bug Something isn't working pinned Pinnend issues are not touched by the stale bot

Comments

@StenaviN
Copy link

StenaviN commented Feb 5, 2021

Issue description

We have pretty huge repository with large files. And now we are unable to clone the entire repo. Mercurial client hangs.

Describe how to reproduce the bug

Mercurial client hangs if we clone repo via SCM Manager: hg clone https://scm-server/scm/repo/hg/my-repo

Everything works as expected if we serve repo using hg serve and clone it using hg cone http://scm-server:8000 my-repo

Describe your SCM-Manager setup

SCM Manager version 2.13.0 on Windows
Mercurial version 5.6.1

No special setup except for HTTPS.

Which operating system, browser and versions do you use?

Operating system doesn't matter: same behavior on Windows and Linux.

Log of the bug

Full error log and config are attached.

scm-server.err.log
server-config.xml.txt

When Mercurial client hangs, SCM Manager error log contains one of the following records about exception:

javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape (%) pattern
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: 
javax.servlet.ServletException: java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape (%) pattern
	at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: 
java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape (%) pattern
	at java.base/java.net.URLDecoder.decode(URLDecoder.java:225)
	at java.base/java.net.URLDecoder.decode(URLDecoder.java:142)
	at sonia.scm.util.HttpUtil.decode(HttpUtil.java:373)
	at sonia.scm.web.WireProtocol.decodeValue(WireProtocol.java:182)
	at sonia.scm.web.WireProtocol.getHeaderDecoded(WireProtocol.java:178)
	at sonia.scm.web.WireProtocol.parseHgArgHeader(WireProtocol.java:145)
	at sonia.scm.web.WireProtocol.parseHgArgHeaders(WireProtocol.java:139)
	at sonia.scm.web.WireProtocol.commandsOf(WireProtocol.java:100)
	at sonia.scm.web.WireProtocol.isWriteRequest(WireProtocol.java:81)
	at sonia.scm.web.HgPermissionFilter.isDefaultWriteRequest(HgPermissionFilter.java:94)
	at sonia.scm.web.HgPermissionFilter.isWriteRequest(HgPermissionFilter.java:81)
	at sonia.scm.web.filter.PermissionFilter.service(PermissionFilter.java:108)
	at sonia.scm.web.HgPermissionFilter.service(HgPermissionFilter.java:64)
	at sonia.scm.repository.spi.InitializingHttpScmProtocolWrapper$ProtocolWrapper.serve(InitializingHttpScmProtocolWrapper.java:148)
	at sonia.scm.repository.spi.HttpScmProtocol.serve(HttpScmProtocol.java:58)
	at sonia.scm.web.protocol.HttpProtocolServlet.service(HttpProtocolServlet.java:96)
	at sonia.scm.web.protocol.HttpProtocolServlet.service(HttpProtocolServlet.java:81)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at sonia.scm.legacy.RepositoryLegacyProtocolRedirectFilter$Worker.noRedirect(RepositoryLegacyProtocolRedirectFilter.java:127)
	at sonia.scm.legacy.RepositoryLegacyProtocolRedirectFilter$Worker.checkPathElements(RepositoryLegacyProtocolRedirectFilter.java:100)
	at sonia.scm.legacy.RepositoryLegacyProtocolRedirectFilter$Worker.doFilter(RepositoryLegacyProtocolRedirectFilter.java:89)
	at sonia.scm.legacy.RepositoryLegacyProtocolRedirectFilter.doFilter(RepositoryLegacyProtocolRedirectFilter.java:71)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.filter.MDCFilter.doFilter(MDCFilter.java:103)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at com.cloudogu.scm.cas.browser.ForceCasLoginFilter.doFilter(ForceCasLoginFilter.java:74)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.web.filter.AuthenticationFilter.processChain(AuthenticationFilter.java:264)
	at sonia.scm.web.filter.AuthenticationFilter.handleAuthentication(AuthenticationFilter.java:223)
	at sonia.scm.web.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:107)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.web.filter.AuthenticationFilter.processChain(AuthenticationFilter.java:264)
	at sonia.scm.web.filter.AuthenticationFilter.handleAuthentication(AuthenticationFilter.java:223)
	at sonia.scm.web.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:107)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.security.TokenExpiredFilter.doFilter(TokenExpiredFilter.java:66)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.filter.BaseUrlFilter.doFilter(BaseUrlFilter.java:109)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:44)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:834)

or

javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IllegalStateException: STREAM
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: 
javax.servlet.ServletException: java.lang.IllegalStateException: STREAM
	at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: 
java.lang.IllegalStateException: STREAM
	at org.eclipse.jetty.server.Response.getWriter(Response.java:800)
	at sonia.scm.web.HgCGIExceptionHandler.sendError(HgCGIExceptionHandler.java:204)
	at sonia.scm.web.HgCGIExceptionHandler.handleException(HgCGIExceptionHandler.java:119)
	at sonia.scm.web.cgi.DefaultCGIExecutor.execute(DefaultCGIExecutor.java:173)
	at sonia.scm.web.HgCGIServlet.process(HgCGIServlet.java:149)
	at sonia.scm.web.HgCGIServlet.handleRequest(HgCGIServlet.java:119)
	at sonia.scm.web.HgCGIServlet.service(HgCGIServlet.java:103)
	at sonia.scm.repository.spi.ScmProviderHttpServletDecorator.service(ScmProviderHttpServletDecorator.java:45)
	at sonia.scm.web.filter.PermissionFilter.service(PermissionFilter.java:116)
	at sonia.scm.web.HgPermissionFilter.service(HgPermissionFilter.java:64)
	at sonia.scm.repository.spi.InitializingHttpScmProtocolWrapper$ProtocolWrapper.serve(InitializingHttpScmProtocolWrapper.java:148)
	at sonia.scm.repository.spi.HttpScmProtocol.serve(HttpScmProtocol.java:58)
	at sonia.scm.web.protocol.HttpProtocolServlet.service(HttpProtocolServlet.java:99)
	at sonia.scm.web.protocol.HttpProtocolServlet.service(HttpProtocolServlet.java:84)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at sonia.scm.legacy.RepositoryLegacyProtocolRedirectFilter$Worker.noRedirect(RepositoryLegacyProtocolRedirectFilter.java:127)
	at sonia.scm.legacy.RepositoryLegacyProtocolRedirectFilter$Worker.checkPathElements(RepositoryLegacyProtocolRedirectFilter.java:100)
	at sonia.scm.legacy.RepositoryLegacyProtocolRedirectFilter$Worker.doFilter(RepositoryLegacyProtocolRedirectFilter.java:89)
	at sonia.scm.legacy.RepositoryLegacyProtocolRedirectFilter.doFilter(RepositoryLegacyProtocolRedirectFilter.java:71)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at com.cloudogu.scm.cas.browser.ForceCasLoginFilter.doFilter(ForceCasLoginFilter.java:74)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.filter.MDCFilter.doFilter(MDCFilter.java:103)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.web.filter.AuthenticationFilter.processChain(AuthenticationFilter.java:266)
	at sonia.scm.web.filter.AuthenticationFilter.handleAuthentication(AuthenticationFilter.java:225)
	at sonia.scm.web.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:109)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.web.filter.AuthenticationFilter.processChain(AuthenticationFilter.java:266)
	at sonia.scm.web.filter.AuthenticationFilter.handleAuthentication(AuthenticationFilter.java:225)
	at sonia.scm.web.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:109)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.security.TokenExpiredFilter.doFilter(TokenExpiredFilter.java:66)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at sonia.scm.filter.BaseUrlFilter.doFilter(BaseUrlFilter.java:109)
	at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:93)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:44)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:834)
@sdorra sdorra self-assigned this Feb 5, 2021
@sdorra sdorra added the bug Something isn't working label Feb 5, 2021
@sdorra
Copy link
Member

sdorra commented Feb 5, 2021

It looks like there is some mercurial command on the wire which we don't understand. Do you use any mercurial extensions on the client or on the server, which are modifying the clone behaviour?

@StenaviN
Copy link
Author

StenaviN commented Feb 5, 2021

Here are server-side extensions:

[extensions]
purge=
color= 
progress=
largefiles=
clonebundles=

Here are client-side extensions:

[extensions]
color =
purge =
progress =
extdiff =
mq =
rebase =
mercurial_keyring =
largefiles =

The client hangs on this stage:

PS C:\Users\NameSurname> hg clone https://scm-server/scm/repo/hg/my-repo
destination directory: my-repo
applying clone bundle from https://scm-server/my-bundle.zstd-v2.hg
adding changesets
adding manifests
adding file changes
added 39278 changesets with 341456 changes to 153351 files (+32 heads)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 91 changesets with 336 changes to 207 files (+3 heads)
new changesets a818d847d9ed:9389908f3297
39278 local changesets published
updating to branch default
getting changed largefiles

And here is the same clone when repo is served via hg serve:

PS C:\Users\NameSurname> hg clone http://scm-server my-repo
destination directory: my-repo
applying clone bundle from https://scm-server/my-bundle.zstd-v2.hg
adding changesets
adding manifests
adding file changes
added 39278 changesets with 341456 changes to 153351 files (+32 heads)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 91 changesets with 336 changes to 207 files (+3 heads)
new changesets a818d847d9ed:9389908f3297
39278 local changesets published
updating to branch default
getting changed largefiles
2166 largefiles updated, 0 removed
32717 files updated, 0 files merged, 0 files removed, 0 files unresolved

Client hung without clonebundles servier-side extension as well.

@StenaviN
Copy link
Author

StenaviN commented Feb 5, 2021

A few other notes:

  • SCM Manager handles multiple (but much smaller) repositories without an issue
  • Here is a subset of stdout which hg serve prints when I do clone and seems it is the stage of getting changed largefiles

hg-serve.log

The very last part of the long request looks suspiciously: statlfile+sha%3D97aeb0c54a122661145d15987c864d108c33aa5e%3 which doesn't look like a valid URL-encoded string at the end: %3. However I don't know either Mercurial printed full request to stdout or truncated it.

@StenaviN
Copy link
Author

StenaviN commented Feb 7, 2021

Well, if we look at the log attached, all x-hgarg requests are truncated which makes me think that it is Mercurial logging just intentionally truncate them before printing to stdout.

BTW, the default config has the following:

    <!-- increase header size for mercurial -->
    <Set name="requestHeaderSize">16384</Set>
    <Set name="responseHeaderSize">16384</Set>

What is the reason, if hg serve perfectly works with default maxhttpheaderlen which is 1024 bytes?
Does SCM Manager somehow modify headers from the client?
And what is going on under the hood in the chain: Mercurial client --> Jetty --> SCM Manager --> hg?

@sdorra
Copy link
Member

sdorra commented Feb 8, 2021

Ok, it looks like there is a problem with the largefile extension (the statlfile is a command of the largefile extension). We will try to reproduce it.

What is the reason, if hg serve perfectly works with default maxhttpheaderlen which is 1024 bytes?

The increase of the header size is for older mercurial versions, which are using http headers to transfer information about the head of the repository and if you have many heads then you need a large buffer for the headers.

Does SCM Manager somehow modify headers from the client?

No.

And what is going on under the hood in the chain: Mercurial client --> Jetty --> SCM Manager --> hg?

Most of what the SCM-Manager does during a clone is handling authorization, the rest is pipeing the client request from the web to a forked mercurial process. It gets more interesting if we talk about push. But what exactly do you want to know?

@StenaviN
Copy link
Author

StenaviN commented Feb 8, 2021

But what exactly do you want to know?

Just trying to help to identify the root cause of the issue. Just curious, if vanilla hg serve works well without any modification in config file, why clone stops working through SCM Manager if SCM Manager just pipeing all the requests down to a forked Mercurial process?

I managed to make it to work doing the following:

server-config.xml:

    <Set name="requestHeaderSize">2097152</Set>
    <Set name="responseHeaderSize">2097152</Set>

Mercurial.ini:

[server]
maxhttpheaderlen=2097152

I've also noticed a separate option for Mercurial Configuration in SCM Manager: Enable HttpPostArgs Protocol which is turned OFF. If we use the latest version of Mercurial, should we turn it ON? It is called experimental but at the same time said it is supported since Mercurial 3.8.

@sdorra
Copy link
Member

sdorra commented Feb 8, 2021

I've done some more research and the newest versions of mercurial still use http headers for the http command protocol. It looks like the largefiles plugin uses the same mechanism for its files. This is probably the reason, why you need such a large header size for jetty.

The HttpPostArgs Protocol is included since mercurial 3.8 but it seems to be still considered as experimental and sadly the configuration option is currently broken, see #1532.

@StenaviN
Copy link
Author

StenaviN commented Feb 8, 2021

This is probably the reason, why you need such a large header size for jetty.

But that doesn't explain why everything works with vanilla hg serve with default header size 1024 bytes.

@sdorra
Copy link
Member

sdorra commented Feb 8, 2021

I'm not sure if hg serve has the same header limitations as jetty or if hg serve behaves different as hgweb.wsgi (which is used by SCM-Manager) in terms of http headers. I will try in the next few days to capture the network traffic with ngrep between hg serve and SCM-Manager and make a comparison.

eheimbuch pushed a commit that referenced this issue Feb 9, 2021
Fixes a regression which was introduced with #1416. In #1416 we have reimplemented the way configuration is passed to the mercurial cgi handler. Before #1416 we used environment variables which are picked up by hgweb.py, after #1416 we pass mercurial configurations as command line parameters directly in the HgCGIServlet. But sadly the configuration option for httppostargs uses still an environment variable, which is not picked up by anyone.

See #1525
@stale
Copy link

stale bot commented Mar 12, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Issue is stale and will be closed if no further activity occurs label Mar 12, 2021
@sdorra sdorra added the pinned Pinnend issues are not touched by the stale bot label Mar 13, 2021
@stale stale bot removed the stale Issue is stale and will be closed if no further activity occurs label Mar 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pinned Pinnend issues are not touched by the stale bot
Projects
None yet
Development

No branches or pull requests

2 participants