mysterious E275 error message repeatedly showing up in KG2 logfiles of services under heavy load

[saramsey]

We frequently see error messages like this in the /tmp/RTX_OpenAPI_kg2.elog and /tmp/RTX_OpenAPI_kg2beta.elog files:

*******ERROR: self.response has no job_id attr! E2752023-09-22T09:15:00.336429 ERROR: (3095) [OverLimit] Query could not be run due to exceeded limits

I think it is coming from this line of code:

RTX/code/ARAX/ARAXQuery/ARAX_query.py

Line 268 in 8c435f0

eprint("*******ERROR: self.response has no job_id attr! E275")

I think this condition is actually expected to occur whenever we respond to a client request with a HTTP status code of 429, since in that case we are not processing the query and thus there is no job_id. I think. If that is true, then could we please update the error message so it doesn't look like something anomalous that we need to debug? Maybe change to an INFO message or something.

[saramsey]

Perhaps I'm wrong and this is truly an unexpected (or not easily explainable) condition, in which case, we should dig deeper. :-)

[sundareswarpullela]

I'll check @saramsey

[edeutsch]

Yes, the E275 is my fault. I did not understand why this condition was happening and thus put in this lame message.
I was thinking it was not because of OverLimit, because I think this is the relevant code for over limits:

RTX/code/ARAX/ARAXQuery/ARAX_query.py

Lines 329 to 333 in 8c435f0

	job_id = query_tracker.create_tracker_entry(attributes)
	if job_id == -999:
	response.error(f"Query could not be run due to exceeded limits", error_code="OverLimit", http_status=429)
	return response

and I don't think it should be hitting the E275 code during over limits, but I'm not certain.

If would be good if we can understand what is happening and handle it more gracefully. I did not get to it.

[saramsey]

Thank you @edeutsch for giving the historical perspective here.