Hanging Threads!

#27195

We are having an issue since upgrading to the latest version of PD4ML.

Every so often (1-2 times a day), our application ends up hanging and a thread dump at the time reveals there are hundreds of hung requests waiting for the PDF generation. I think that Apache literally runs out of request threads.

An example of the stack on each of these threads is below.

Any idea why this would be happening, or how to get around the problem?

“http-nio-8080-exec-410” – Thread t@47951
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
– locked <479d2106> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
– locked <5a70bc4c> (a sun.security.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
– locked <3ed4cb14> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
– locked <61b70923> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
– locked <61b70923> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:3053)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getHeaderFieldKey(HttpsURLConnectionImpl.java:316)
at org.zefer.cache.CookieManager.storeCookies(Unknown Source)
at org.zefer.cache.b.getResourceAsBytes(Unknown Source)
at org.zefer.html.doc.PD4MLHtmlParser.parse(Unknown Source)
at org.zefer.html.doc.PD4MLHtmlParser.buildDocument(Unknown Source)
at org.zefer.pd4ml.PD4ML.o00000(Unknown Source)
at org.zefer.pd4ml.PD4ML.render(Unknown Source)
at org.zefer.pd4ml.PD4ML.render(Unknown Source)
at com.touch.clarity.go.app.pdf.HtmlToPdfConverter.convert(HtmlToPdfConverter.java:65)

#30158

Hmm… First, PD4ML does not start new threads – so the hundreds of hanging thread should match a number of conversion requests.

As I see from the stack trace, it hangs on a network operation – it is difficult to say why it cannot read data from a socket. Did you check logs on the remote side, where it tries to read the data by HTTPS from?

#30159

The request is firing back into the same application (to get the HTML version of the page I’m converting into a PDF).

Interestingly there is no sign of any other requests in the thread dump apart from the initial user requests for PDFs. So it looks like the request from the PDF conversion isn’t making it back to the application.

This either means that the request isn’t getting out (or back in) from the PD4ML component, or perhaps the system is somehow locked up with all the initial user requests that the resultant HTML requests can’t get in. This may also explain the fact that this issue is intermittent.

Can you suggest anything?

[PD4ML]

#30160

It seems it hangs by an attempt to read cookie data from HTTPS response header. Not that easy to debug…

As a general recommendation limit the timeout for the network operations:

HashMap map = new HashMap();
map.put(PD4Constants.PD4ML_SOCKET_TIMEOUT , "60000"); // in ms
pd4ml.setDynamicParams(map);

That should help to avoid the running out of threads. But it does not address the problem reason.

The best workaround would be to avoid HTTPS protocol by the local data exchange at all. A reference like <a src=”/images/logo.png”> is more preferable than <a src=”https://myserver/images/logo.png&#8221;>

Please take into account, the link like <a src=”https://myserver/images/logo.png&#8221;> can be built implicitly from <a src=”/images/logo.png”> and <base href=”https://myserver/&#8221;>. In the case an overriding of the base with PD4ML API call parameters should help.

If the HTTPS is a must, you have to investigate the network issue reason… We need some more input

• This reply was modified 5 years, 4 months ago by PD4ML.
• This reply was modified 5 years, 4 months ago by PD4ML.
• This reply was modified 5 years, 4 months ago by PD4ML.
• This reply was modified 5 years, 4 months ago by PD4ML.

#30161

I have changed the code to encode the HTML directly within the initial request, thus the second request should not be required. I’ll let you know if this solves the problem.

Thanks for the help

#30162

i use the pd4ml with version 1.0, there is no method to set TIME_OUT parameter, any suggestions to reslove it ?

#30163

Please use the most recent PD4ML v3.10.9. There is no practical benefits to use that old version as v1.0

[Author]

Posts