Certain domains query cause Error 500 #60

mnmnc · 2018-06-29T07:58:10Z

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	RedHat
OS version (client)	Seven,
Package Type	Docker

Problem Description

Error 500 when asked for a domain doesok.top, baserpolaser.tk, letsgotohome.tk

Steps to Reproduce

Query hipposcore for any of doesok.top, baserpolaser.tk, letsgotohome.tk
Get error 500

curl -i -H "Content-Type: application/json" -X POST -d '{"ferasoplertyh.tk" : {"type" : "domain"} }' "https://domain.some/hippocampe/api/v1.0/hipposcore"
HTTP/1.0 500 INTERNAL SERVER ERROR
Content-Type: text/html
Content-Length: 291
Server: Werkzeug/0.14.1 Python/2.7.14
Date: Fri, 29 Jun 2018 07:44:28 GMT
Set-Cookie: XXXXXXX; path=/; HttpOnly; Secure
Connection: keep-alive

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

Logs:

2018-06-29 08:00:43,423 :: services.more :: INFO :: {u'ferasoplertyh.tk': {u'type': u'domain'}}
2018-06-29 08:00:43,540 :: services.hipposcore :: INFO :: hipposcore.calcHipposcore launched
2018-06-29 08:00:43,545 :: services.hipposcore :: ERROR :: hipposcore.calcHipposcore failed, no idea where it came from...
Traceback (most recent call last):
  File "/opt/Hippocampe/core/services/hipposcore.py", line 76, in calcHipposcore
    P = P + (n3 * n2 * n1)
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'
2018-06-29 08:01:53,198 :: services.modules.common.ES :: INFO :: ES.checkES launched
Traceback (most recent call last):
    resMsearch = searchIntel.littleMsearch(source.coreIntelligence, source.typeNameESIntel, parsedPage)
    res = es.msearch(body = req)
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 1183, in msearch
    headers={'content-type': 'application/x-ndjson'})
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 314, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 175, in perform_request
    raise ConnectionError('N/A', str(e), e)
ConnectionError: ConnectionError(('Connection aborted.', error(104, 'Connection reset by peer'))) caused by: ProtocolError(('Connection aborted.', error(104, 'Connection reset by peer')))

The text was updated successfully, but these errors were encountered:

obikao · 2018-10-12T02:05:35Z

Yeah I had that issue too. This code base is a bit atrocious:
Here is a super hacky solution:

add this after line 66, before hipposcoreDict = calcHipposcore(new_response)

logger.info("initiating fix")

try:
	new_response = {}
	for ip, data in response.iteritems():
		temp_list = []
		for item in data:
			if item["idSource"] == '':
				item["idSource"] = fixThisBrokenFunction(item["source"])
			temp_list.append(item)
		new_response[ip] = temp_list
except Exception as e:
	logger.error('failed to fix their broken stuff', exc_info = True)
	logger.error(response)
	report = dict()
	report['error'] = str(e)

and then right before if name == "main":

def fixThisBrokenBullshit(source):
	es = getES()

	data = {
		'query' : {
			'bool' : {
				'must' : [
					{'match' : {'source' : source}}
					]
				}
			}
		}

	res = es.search(body=data)

	for i in res['hits']['hits']:
		if i["_source"]["idSource"] != "":
			return i["_source"]["idSource"]
			break
		else:
			continue

obikao · 2018-10-12T02:05:48Z

This is in more.py btw

mnmnc changed the title ~~Certain domains qeury cause Error 500~~ Certain domains query cause Error 500 Jun 29, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Certain domains query cause Error 500 #60

Certain domains query cause Error 500 #60

mnmnc commented Jun 29, 2018 •

edited

obikao commented Oct 12, 2018 •

edited

obikao commented Oct 12, 2018

Certain domains query cause Error 500 #60

Certain domains query cause Error 500 #60

Comments

mnmnc commented Jun 29, 2018 • edited

Request Type

Work Environment

Problem Description

Steps to Reproduce

obikao commented Oct 12, 2018 • edited

obikao commented Oct 12, 2018

mnmnc commented Jun 29, 2018 •

edited

obikao commented Oct 12, 2018 •

edited