Oxidized v.0.30.1 backup FortiGate config issue - only backup 70% devices #3124
Comments
|
looks like the job only run 76 out 111 devices, then stopped and restart load previous devise again. Not sure why? D, [2024-04-15T11:41:25.481373 #4626] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 70 of 111 |
|
Is this a regression from 0.29.1 or did you directly try 0.30.1? |
|
This is directly using oxidized 0.30.1 with fresh new installation. not upgrade. /var/lib/gems/3.0.0/gems/oxidized-0.30.1/lib/oxidized/source/csv.rb:16:in `setup' |
|
Please check the full error, try using journalctl -u oxidized.service then scroll (g) to the last lines. In the initial post you only see half of the error message. |
|
Thanks, this provided a lot of information, can see some errors. uploaded last block message, others are repeating: Apr 15 21:16:28 ubuntu2204 oxidized[5956]: #<Oxidized::Job:0x000055e8c9237690 /var/lib/gems/3.0.0/gems/oxidized-0.30.1/lib/oxidized/job.rb:8 ru> |
|
buffer.rb#L342 error message should be |
|
I am getting lost, is that net-ssh-7.2.3 problem? How to fix this not supported key issue? |
|
I'd say search through the logs, see if you can find exactly what key type does it not support. I also had issues with net-ssh and fortigates. I did try to change the kex on the fortigates but it would still pick the one that it doesn't work, but you can experiment. |
|
FortiGate support ED25519 key, also net-ssh 5 and later support ed25519 Why buffer.rb script only support RSA, DSA, and ECDSA keys? Can ED25519 key be added in for support? |
ed25519 host keys work just fine - https://github.com/net-ssh/net-ssh/blob/master/lib/net/ssh/buffer.rb#L336
They are supported right now. The problem is fortinet is advertising |
|
What needs to happen herei is that fortinet needs to be fixed, it needs to start advertiising its RSA key types correctly as In the meanwhile, #3123 has workaround where you can change 1 line in We could also submit fix to |
|
To recreate the problem without using oxidized, try the following: Create a test.rb with the contents: replace @host with the device IP and @user with 'some username' Save, then do ruby test.rb It should give you the same error without the trouble of searching through logs. |
|
when /^ssh-rsa$/, /^rsa-sha2-(256|512)$/ does not work for me. because some of the FortiGate finished the config backup process. here is one of the working Net:SSH debug info. irb(main):009:0> Net::SSH.start("10.8.254.254", "usernamexx", :password => "passwordxx", verbose: Logger::DEBUG)
D, [2024-04-17T17:33:52.635300 #21409] DEBUG -- socket[334]: received packet nr 5 type 51 len 32 |
|
Unfortunately in the provided output, the line Apr 17 21:00:42 ubuntu2204 oxidized[21859]: /var/lib/gems/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/buffer.rb:342:in read_keyblob': unsupported key type >Is truncated, and missing the actual type it is seeing which it perceives as unsupported, so hard to say anything. |
|
So, how to get the detail of this "unsupported key type" ? |
|
Also from that Net:SSH debug info, we can see FortiGate is using SSH-2.0-AqTN with host_key: ssh-ed25519 net.ssh.transport.algorithms[320]: negotiated: |
the ed25519 is supported, problem is the malformet type advertised for rsa-sha2-512, which causes net-ssh to raise. Unfortunately I don't have time to walk you through all this. |
|
finally fixed the problem by myself. Down graded to net-ssh-7.1.0 :-) |
|
I am going to load another 500 FortiGate devices. 💯 |
|
after I loaded another 200 devices, net-ssh-7.1.0 start having the same problem too. Apr 18 10:56:24 ubuntu2204 oxidized[23691]: /var/lib/gems/3.0.0/gems/net-ssh-7.1.0/lib/net/ssh/buffer.rb:342:in `read_keyblob': unsupported key> |
|
If you can arrange a 'broken' test host for me to connect to, I can look into this. I'd use 91.198.120.1 as source address. |
|
I reiterate the suggestion to run the test code i showed above as to separate this issue from oxidized because it is a net-ssh problem |
|
Thanks for you try to help me. I got it working again after using when /^ssh-rsa$/, /^rsa-sha2-(256|512)$/ option. I will keep monitoring it for a few days. next task to setup e-mail alert if there is a config change. |
Do bear in mind, we cannot submit this change to upstream, as it is non-sensical, problem is far-end, and this is crude workaround to interoperate with broken far end. We could submit to upstream change where unparseable host keys will not cause raise, but I cannot produce that fix, unless I have access to broken device to test against.
|
|
I am not sure how you can repeat this problem in Lab. |
|
Are you sure you've correctly identified the problem, I suspect your problem is, some of your FortiGates are olrder ones, which advertise Therefore, reprodo would be easy, just test against broken FortiGate. There is 0 reason, why the proposed workaround would do anything to allivate any undeterministic race condition problem. |
|
can we discuss it privately? avoid to leaking my firewall information to public. what's your e-mail address? |
Yes that's fine, saku@ytti.fi |
Hi
I installed Oxidized in my Ubuntu 22.04 server. system is up running without any error. first I loaded 6 Foritgate devices, all backup process fine. oxidized-web also showing green for each devices.
Then I loaded 110 Fortigate devices, oxidized-web start showing blue, last update-never and last changed - unknown. Then I enabled SSH debug. If the Fortigate devices had backup successful, I can see their SSH log message. If the Fortigate failed config backup. I did not see their SSH log message. Also oxidized-web start crash after a few minutes.
--------when oxidized.service crashed, i see this error----------
Not sure what's the problem, Any idea?
Regards
Kevin
(Updated by @robertcheramy for readability)
The text was updated successfully, but these errors were encountered: