Possible Issue with logging ResponseSize caused by insufficient storage type about thredds HOT 4 OPEN

Great question! I believe this particular call is only used in a few places where the size will be much smaller than 2 GB (such as serving a catalog response in html or xml). There are other places in the code, like here that explicitly handle larger transfers, such as HTTPServer requests. In those cases, I can confirm the correct bytes transferred size will show up in threddsServlet.log. That said, because this isn't handled uniformly in the codebase, I would not be too surprised if what you describe above happens for some protocols/cases. Do you have a case where you see this behavior happening?

from thredds.

I have found it by parsing logs from the past few years looking for primarily transactions that match /thredds/fileServer/.....nc.

That line you linked will set the 'Content-Length' header, but it leaves the httpResponseBodyLength member in TdsServletResponseWrapper untouched. So, any time contentLength > Integer.MAX_VALUE the log will report -1 for the size, as I understand it.

The member/class that is used to report to "Request Completed" entries in the threddsServlet.log.
https://github.com/Unidata/thredds/blob/v5.0.0-beta5/tds/src/main/java/thredds/servlet/filter/TdsServletResponseWrapper.java#L23

Where the httpResponseBodyLength member is used.

from thredds.

Ah, thank you! I think I see now. We normally parse the Apache logs to get stats on how much data our TDS instances serve, but we should try to accurately capture that in the threddsServlet logs as well. I'll be out of the office for the next few weeks, but I can take a look again once I get back (unless @ethanrd wants to give this a look).

from thredds.

That is what we have arrived at as well, using the Apache logs.

from thredds.