-
Notifications
You must be signed in to change notification settings - Fork 631
MHN Troubleshooting Guide
Brady Sullivan edited this page Oct 24, 2017
·
14 revisions
This is a Work in Progress troubleshooting guide to MHN, compiled from responses to questions sent to the MHN Google Group (modern-honey-network@googlegroups.com).
Password reset through the web app is not working and/or retrieving http://YOUR-SITE/static/mhn.rules causes a 404.
I believe this is because your celery worker is not working. Can you try this?
sudo supervisorctl status
Is the status of your "mhn-celery-worker" FATAL or something other than RUNNING? If so, then try this:
sudo supervisorctl restart
If the mhn-celery-worker process still does not start cleanly. Then try this:
cd /var/log/mhn/
sudo chown www-data mhn.log
sudo supervisorctl start mhn-celery-worker
If this fails (i.e. "sudo supervisorctl status" shows the process is not RUNNING after about 10 sec), then review these logs?
/var/log/mhn/worker.err
/var/log/mhn/worker.log
/var/log/mhn/mhn.log
This should steer you in the direction of the problem.
Can you describe the error you're encountering when logging in? Password failure? Connection failure?
If password failure, can you try this (you should see similar output to mine below, if not, it indicates a problem).
The select statement is useful to verify that the user you expect to exist does in fact exist.
# sudo su -
# cd $MHN_HOME/server/
# sqlite3 mhn.db
SQLite version 3.7.9 2011-11-01 00:52:41
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from user;
1|USERNAME@SITE.com|sklfdjhkasdlfhklsadhfklasdhfkldsahklsd|1|
sqlite> .quit
# cd $MHN_HOME/
# . env/bin/activate
# cd server
# python manual_password_reset.py
Walk through the password reset and make sure you see user found, updating password at the end.
- I don't see events flowing back to my MHN server even though I am scanning/wget'ing my sensor
Can you run these commands from your honeypot system and provide the output?
Before running these commands start a script to wget against your honeypot.
$ sudo netstat -luntp
$ sudo supervisorctl status
$ sudo iptables -L
$ sudo tcpdump -nnNN tcp port 10000
I am expecting output like this:
$ sudo netstat -luntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 561/sshd
tcp6 0 0 :::21 :::* LISTEN 3763/dionaea
tcp6 0 0 :::22 :::* LISTEN 561/sshd
tcp6 0 0 :::1433 :::* LISTEN 3763/dionaea
tcp6 0 0 :::443 :::* LISTEN 3763/dionaea
tcp6 0 0 :::445 :::* LISTEN 3763/dionaea
tcp6 0 0 :::5060 :::* LISTEN 3763/dionaea
tcp6 0 0 :::5061 :::* LISTEN 3763/dionaea
tcp6 0 0 :::135 :::* LISTEN 3763/dionaea
tcp6 0 0 :::3306 :::* LISTEN 3763/dionaea
tcp6 0 0 :::42 :::* LISTEN 3763/dionaea
tcp6 0 0 :::80 :::* LISTEN 3763/dionaea
udp 0 0 0.0.0.0:68 0.0.0.0:* 464/dhclient3
udp 0 0 0.0.0.0:40077 0.0.0.0:* 3763/dionaea
udp6 0 0 :::5060 :::* 3763/dionaea
udp6 0 0 :::69 :::* 3763/dionaea
$ sudo supervisorctl status
dionaea RUNNING pid 3763, uptime 3 days, 22:31:19
$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
$ sudo tcpdump -nnNN tcp port 10000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:46:14.009646 IP 1.2.3.4.42873 > 5.6.6.8.10000: Flags [P.], seq 1180349317:1180349611, ack 2474834734, win 913, options [nop,nop,TS val 85084174 ecr 169636000], length 294
16:46:14.012967 IP 5.6.6.8.10000 > 1.2.3.4.42873: Flags [.], ack 294, win 162, options [nop,nop,TS val 169661444 ecr 85084174], length 0
Can you log into your dionaea box and look at its config (/etc/dionaea/dionaea.conf)? Scroll to the hpfeeds section.
Your should see something like this:
hpfeeds = {
hp1 = {
server = "1.2.3.4"
port = "10000"
ident = "XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX"
secret = "XXXXXXXXXXXXXXXX"
// dynip_resolve: enable to lookup the sensor ip through a webservice
dynip_resolve = "http://queryip.net/ip/"
}
Is your "server" variable correct? It should be the IP of your MHN server. If so, can you try this (from your dionaea box)?
nc -vvv <YOUR-SERVER-IP> 10000
Do you see something like this? Or does it just hang?
$ nc -vvv <YOUR-SERVER-IP> 10000
Connection to <YOUR-SERVER-IP> 10000 port [tcp/webmin] succeeded!
@hp2?Z
If it hangs either your hpfeeds service isn't running on your server or there may be a firewall blocking it or there may be another network connectivity issue.
- I don't see events flowing back to my MHN server even though I am scanning/wget'ing my sensor
On your MHN server box, as root run these commands. You should get similar out to mine. If your output differs, this could be the problem.
# netstat -luntp | grep 10000
tcp 0 0 0.0.0.0:10000 0.0.0.0:*
LISTEN 980/python
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# cd /var/log/mhn/
# tail -f hpfeeds-broker.err
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 49187)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 39176)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 42873)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 42873)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 46710)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
# mongo hpfeeds
MongoDB shell version: 2.6.2
connecting to: hpfeeds
> db.auth_key.find()
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "mnemosyne",
"subscribe" : [ "conpot.events", "thug.events", "beeswarm.hive",
"dionaea.capture", "dionaea.connections", "thug.files",
"beeswarn.feeder", "cuckoo.analysis", "kippo.sessions",
"glastopf.events", "glastopf.files", "mwbinary.dionaea.sensorunique",
"snort.alerts" ], "secret" : "XXXXXXXXXXXXXX", "publish" : [ ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "honeymap",
"subscribe" : [ "geoloc.events" ], "secret" : "XXXXXXXXXXXXXX",
"publish" : [ ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "geoloc",
"subscribe" : [ "dionaea.connections", "dionaea.capture",
"glastopf.events", "beeswarm.hive", "kippo.sessions", "conpot.events",
"snort.alerts" ], "secret" : "XXXXXXXXXXXXXX", "publish" : [
"geoloc.events" ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "subscribe" : [ ], "secret"
: "XXXXXXXXXXXXXX", "identifier" :
"XXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX", "publish" : [
"mwbinary.dionaea.sensorunique", "dionaea.capture",
"dionaea.capture.anon", "dionaea.caputres", "dionaea.connections" ] }
Type "it" for more
>
The last entry is the entry for one of our honeypots. The identifier
and secret need to match what is in your honeypot's
/etc/dionaea/dionaea.conf config under the hpfeeds section. If these
don't match your honeypot will not be able to talk with the server.
Because your honeypot is on a private network, you will almost certainly not see any events show up on the Honeymap since it only renders events that can be IP Geo located and private IPs do not get geolocated by Maxmind.
Can you confirm that you see no events on http://YOUR-SERVER/ui/attacks/ and no counts on the base page? If so, can you run this command? You should have output similar to mine:
# mongo mnemosyne
MongoDB shell version: 2.6.2
connecting to: mnemosyne
> db.getCollectionNames()
[
"daily_stats",
"dork",
"file",
"hpfeed",
"session",
"system.indexes",
"url"
]
> db.session.find()
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipCall", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-05-20T20:12:14.028Z"), "source_ip" :
"192.198.107.186", "source_port" : 5070, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T12:25:29.493Z"), "source_ip" :
"209.126.99.96", "source_port" : 5086, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T12:25:32.782Z"), "source_ip" : "107.22.35.136",
"source_port" : 55261, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:48.040Z"), "source_ip" : "107.22.35.136",
"source_port" : 52915, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:47.137Z"), "source_ip" : "178.252.183.100",
"source_port" : 3955, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "httpd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:51.478Z"), "source_ip" : "115.79.59.61",
"source_port" : 3453, "destination_port" : 80, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "httpd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:50.476Z"), "source_ip" : "115.79.59.61",
"source_port" : 3454, "destination_port" : 80, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:53.105Z"), "source_ip" : "107.22.35.136",
"source_port" : 52917, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:52.274Z"), "source_ip" : "93.174.93.218",
"source_port" : 52569, "destination_port" : 21320, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:59.602Z"), "source_ip" : "178.252.183.100",
"source_port" : 4053, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:59.507Z"), "source_ip" : "181.193.66.61",
"source_port" : 43670, "destination_port" : 23, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T13:05:58.327Z"), "source_ip" :
"209.126.99.96", "source_port" : 5068, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:58.169Z"), "source_ip" : "107.22.35.136",
"source_port" : 52919, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:04.196Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:03.233Z"), "source_ip" : "107.22.35.136",
"source_port" : 52921, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:03.061Z"), "source_ip" : "178.252.183.100",
"source_port" : 4077, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T13:06:01.699Z"), "source_ip" :
"176.58.113.94", "source_port" : 6072, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:05.730Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:04.967Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:08.296Z"), "source_ip" : "107.22.35.136",
"source_port" : 52923, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
The issue is, out of the box, mnemosyne filters out attacks coming from rfc1918 addresses so, you need to configure it not to do so. Here are the steps:
As root, run these commands.
cd /opt/mnemosyne/
git fetch origin
git stash
git merge origin/master
git stash pop
Now, edit mnemosyne.cfg and append this to the end of the file.
[normalizer]
ignore_rfc1918 = False
Lastly, run this command:
supervisorctl restart mnemosyne
Then run.
supervisorctl status
You should see something like this:
geoloc RUNNING pid 16719, uptime 1:25:31
honeymap RUNNING pid 17223, uptime 0:30:29
hpfeeds-broker RUNNING pid 980, uptime 10 days, 5:19:51
mhn-celery-beat RUNNING pid 961, uptime 10 days, 5:19:51
mhn-celery-worker RUNNING pid 14878, uptime 3:44:43
mhn-uwsgi RUNNING pid 9580, uptime 9 days, 3:46:32
mnemosyne RUNNING pid 17749, uptime 0:05:26
Try this:
cd /opt/mhn/server
sudo sqlite3 mhn.db
sqlite> select * from user;
Find the ID associated with the fat fingered email address. It will probably be 1.
sqlite> update user set email='your.real.email@gmail.com' where id=1;
sqlite> .quit
Now run the password reset script
sudo su -
cd /opt/mhn
source env/bin/activate
python server/manual_password_reset.py
edit this file: /opt/mhn/server/config.py
Change this variable to the correct URL:
HONEYMAP_URL = 'http://10.0.0.1:3000'
Then restart mhn-uwsgi:
sudo supervisorctl restart mhn-uwsgi
- Ubuntu 16.04 It's likely mongodb isn't running after reboot.
sudo systemctl start mongodb
sudo supervisorctl restart all