PFSense Configuration (discussion continued from upstream)

[msmollin]

Continuing the discussion from upstream issue: trailofbits#292

I have successfully managed to finally connect to my algo server.

Following the instructions here: https://gist.github.com/davidemyers/c74ddbb5d37e4c8288cd60cfb0f1b445
Will get you 99% of the way. However, as noted in the above thread, PFSense 2.4 beta does not support ECDSA private keys. However, PFSense ships with StrongSwan 5.5 which does support them.

After some digging, I came across https://forum.pfsense.org/index.php?topic=98257.0 which gave me the missing clue: the WebGUI is just incorrectly marking the key RSA in /var/etc/ipsec/ipsec.secrets. The solution here is to go all the way through @davidemyers directions and then, once completed, open the file editor found in PFSense (Diagnostics -> Edit File) and edit the secrets file and change RSA to ECDSA.

Of course this is extremely fragile. As soon as a new cert/private key combo is added, this will get over-written. The real solution here would be to attempt to get a patch merged upstream into PFSense to fix whatever code is driving the WebGUI to incorrectly decipher and parse this code.

Until I (or someone else) gets around to fixing that though, this solution resolves my issue.

Any thoughts from upstream about this?

[davidemyers]

Thanks for creating this fork. I didn't realize commenting on Gists didn't have the same functionality.

What's odd is that I can connect to Algo from pfSense without having to modify ipsec.secrets, even though it lists the key as RSA. I suspected I was lucky to be getting away with this, but I wonder why it works for me and not you.

Maybe I'll create a quick-and-dirty patch for use with the pfSense System_Patches package to force the key type to ECDSA.

[msmollin]

That would be neat. My PHP chops are rusty but I could also a take look into it if you don't have time. I realized that this is actually more brittle than I even originally found. You can edit the file and then bring the tunnel up once and when you disconnect it resets the key designation in the secrets file and it all goes to heck.

Yeah I don't get how you're so lucky to not have to muck with this haha :) I finally found the real offending log entry complaining about attempting to load the private key for the cert and failing, and no amount of re-creating, renaming, or modification would fix it.

[davidemyers]

Here's a very quick-and-dirty patch that simply forces ECDSA to be written to ipsec.secrets for all certs. To install it, install the System_Patches package and use it to apply this patch with Path Strip Count = 0 and Base Directory = /.

+++ /etc/inc/vpn.inc	2017-07-01 08:55:57.615420000 -0400
@@ -678,7 +678,7 @@
 				@chmod($ph1certfile, 0600);
 
 				/* XXX" Traffic selectors? */
-				$pskconf .= " : RSA {$ph1keyfile}\n";
+				$pskconf .= " : ECDSA {$ph1keyfile}\n";
 			} else {
 				list ($myid_type, $myid_data) = ipsec_find_id($ph1ent, 'local');
 				list ($peerid_type, $peerid_data) = ipsec_find_id($ph1ent, 'peer', $rgmap);

[davidemyers]

One big drawback to this approach is that during a software update, the system reboots and IPsec restarts before the patch gets re-applied.

[msmollin]

Well clearly the solution is to never update the software. Juuuust kidding!

Hmmm that's not a big dealbreaker for me right now as I have this tunnel disabled by default and I only enable it when testing it currently. But ideally it would survive a reboot.

I'll see if I can take this and setup a function in the file to regex through the $ph1keyfile incoming file and determine if it should be a ECDSA key or RSA. Shouldn't be too bad since the first line of the EC private keys is different from the RSA keys. I imagine that would be something we could then submit to PFSense for inclusion in 2.4.

[davidemyers]

The patch survives reboots, but after an update you have to disable and re-enable IPsec or else ipsec.secrets still contains RSA. Which still works for me for some reason.

I think a proper patch for 2.4 would be more involved, starting with adding a UI option for Mutual ECDSA when setting up the Phase 1.

[msmollin]

Oh I see so its only on update + reboot? A regular reboot there's no problem?

[davidemyers]

Right. The System_Patches package reverts all of its patches before a software update. You can optionally specify that a patch gets automatically re-applied, but this happens after IPsec has been started.

[iamvishnurajan]

With the recent release of pfSense 2.4, I gave this a whirl and was successful in establishing the VPN to my Digital Ocean server. I can't say enough to thank you for the work and effort to document the instructions and patches here well.

One item I find however that I haven't been able to solve, despite various tinkering, is the download speed is ~1/3 of my regular connection. It is specific to the connection via the pfSense box. In other words, regular connection (no VPN) is about 200 download / 35 upload. If I connect my MacBook to Algo with the mobileconfig profile, I get pretty close to that (say 180 / 35). When I connect through pfSense though, I pretty consistently get ~60 / 35.

My pfSense box is fairly powerful as it is an old desktop PC that I re-purposed. Hardware encryption is turned on. I checked when running the speedtest, CPU usage is in single digit range.

I wanted to know if anyone else has experienced this or if any ideas as to what might be causing that much of a download speed drop when connecting to Algo through pfSense?

[davidemyers]

Unfortunately for me my connection is so slow that I can't do any meaningful speed tests, but please let us know if you figure out what's going on.

[msmollin]

I also ran into this @iamvishnurajan on the $5/mo Droplets. Best as I can tell, it's because the smallest Droplets don't have AES-NI in any form available, so all VPN encrypt/decrypt is handled by the single core CPU on those Droplets, and around 60 mbps is where the CPU hits 100% utilization (from my informal testing).

I haven't tried one of the faster droplets yet, but it might be worth trying. I personally only have a 75/75 connection at home so it's "fast enough" for my uses.

[iamvishnurajan]

Tried a bit more testing on this tonight. Still no change, but I think it is something about pfSense.

Here's what I tried tonight:
-Briefly resized to a $20 droplet and tried that. Still same, max around 60 Mbps download. So went back to the $5 droplet.
-Changed the Phase 2 setup so I was only tunneling the machine I was testing on and not the rest of my LAN. Confirmed my other PC saw WAN IP, MacBook saw the Algo IP. Still same, around 60 download.
-Disabled the IPsec connection on pfSense and connected to the same droplet from MacOS .mobileconfig profile. I then get in the 150+ download range.
-In all three cases, upload speeds do not seem to be significantly affected (or at least not that I can tell, but my upload is only 35 Mbps).

So - I think I've at least narrowed the problem down to something about pfSense that it is maxing out around 60 Mbps download.

I'll keep poking on this when I can and see if I can find anything. Not that 60 Mbps isn't good enough for most of my usage, but now I'm curious to figure out where it is getting limited.

[davidemyers]

Maybe the issue is fragmentation. Try lowering the value for MSS_IPV4 in router-updown.sh.

Before making the change you should disconnect the IPsec connection from pfSense.

[iamvishnurajan]

Just a quick update - played with a few MSS values, lowering down to 1200. No obvious benefit from that. Going to try and do more testing this weekend to see if I can isolate the problem further.

[iamvishnurajan]

Just FYI - no major breakthroughs on my end. Did more troubleshooting to try and isolate. Other than it seeming like something about how pfSense is interacting or has implemented StrongSwan/IPSEC that is limiting the connection to ~60 Mbps, I haven't found any knobs.