Rewrite fortios model

[robertcheramy]

• I have read and understood docs/Issues.md.

Is your feature request related to a problem? Please describe.
I added diagnose hardware deviceinfo psu and get system interface transceiver to the FortiOS model in a monkey patch.
The handling of cmd :all is not working properly, and sending all the commands in post do makes the model difficult to read and maintain.

Describe the solution you'd like
I plan to:

• rewrite the fortios model to make it simpler.
• produce YAML simulation files
• remove | grep . from the commands as we can handle the pager
• probably add an if statement on cmd that is evaluated at runtime, something like cmd 'show full-configuration', if: -> { vars(:fullconfig) }

Other ideas are welcome and can be added to this issue.

I have no fortigates with vdoms, so I won't be enable to test everything. As this change could break the FortiOS model for some users, I plan to announce it in the Release notes of 0.35, and make the change in 0.36, so that interested users can test the model (and provide YAML simulation files) before the 0.36 release. => Setting the Milestone to 0.35 for now, in order to remember it when releasing 0.35.

Describe alternatives you've considered
Adding more commands into to model without rewriting it. I think cleaning the model is a better solution and worth the effort.

[Serazio]

Hi,

we're backing up a lot of fortigates and facing the problem with the changing hashes for every backup. As we need password hashes inside our backups, we implemented a temporary external solution for backing up FortiGates. With the hashes oxidized and git as backend it's generating a ton of unnecessary commits.

If its somehow possible to read existing backups, it would be great to implement a workflow like the following:

• Retrieve full backup
• Remove all hashes from backup
• If existing: Compare "hashless" backup with previous "hashless" backup
• If "hashless" backups differ from each other
• • Save "hashless" backup
• • Save original backup
• else
• • do nothing

In our current solution we are using the following regex:

  replace: "\\1 <line removed>"
- regexp: "(set .+ ENC) .+"
  replace: "\\1 <configuration removed>"
- regexp: "(set .*secret) .+"
  replace: "\\1 <configuration removed>"
- regexp: "(set (?:passwd|password|key|group-password|auth-password-l1|auth-password-l2|rsso|history0|history1)) .+"
  replace: "\\1 <configuration removed>"
- regexp: "(set private-key )[\\s\\S]*?-+END (ENCRYPTED|RSA|OPENSSH) PRIVATE KEY-+\\n?\"$"
  replace: "\\1 <configuration removed>"

There is one problem with this solution: It would not detect if somebody changed a by FortiGate hashed value, like passwords or certificates, and will not create a new backup then.

[robertcheramy]

If its somehow possible to read existing backups, it would be great to implement a workflow like the following:

• Retrieve full backup
• Remove all hashes from backup
• If existing: Compare "hashless" backup with previous "hashless" backup
• If "hashless" backups differ from each other
• • Save "hashless" backup
• • Save original backup
• else
• • do nothing

I am considering to code something like this (not basing on hashes, based on regexp). It has to be activated by a vars as it won't backup changed passwords anymore.

[robertcheramy]

The rewrite of the fortios midel can be seen in branch fix/3680-rewrite-fortios.

I've tested it against different Fortigates models (including one with vdom), but I don't have other variants (FortiADC, FortiMail, FortiManager...) to test.

The output is changed a bit (see fortios*output.txt in https://github.com/ytti/oxidized/tree/fix/3680-rewrite-fortios/spec/model/data). I've tried to keep only relevant data. If something is missing or something can be further stripped, add a comment here, so that I can update the model.

Note: the model will only run in the branch fix/3680-rewrite-fortios or in current master, as it uses features that will be introduced by 0.36.0. I can provide a docker image if it helps.

@electrocret - you improved support for FortiADC in PR #3428. I don't have a FortiADC to test, maybe you can test the model, or even provide a YAML simulation file?

@raunz - you introduced different possibilities to show the configuration in PR #2373 . Are there still models out there which don't support | grep .?

@systeembeheerder, @Punicaa - you made changes to the fortios model in the last years - could you check if the new model works for you?

Thank you everyone for your time & help!

[electrocret]

@robertcheramy , I only have a handful of FortiADC's, and they're in a rather inaccessible to oxidized part of my network (Oxidized has to bounce through proxies to get to them for backup). I believe they're also running fairly outdated code. The platform in general is on the way out at my company and gets little attention (We only have them as part of an acquisition).

Long story short - I feel it'd be more effort than it's worth to proactively build the test data for oxidized to test against than it would be to push your rewrite down to my oxidized and fix it after the fact, if anything breaks.

[raunz]

FortiManager and FortiAnalyzer VM's only accept command: "show". They don't know anything about "show full-configuration", "|" or "grep"
sample VM config-versions:
FMG-VM64-HV-7.4-FW-build2744-250926
FAZVM64-HV-7.6-FW-build3579-250825

In my accessible Fortinet hardware devices (approx 50 different hardware/software combinations), all executed command "show full-configuration | grep ." successfully. It seems, that only "show full-configuration" without grep is no longer relevant.

@raunz - you introduced different possibilities to show the configuration in PR #2373 . Are there still models out there which don't support | grep .?

I tried to test the branched model, but it uses new .keep_lines feature which isn't available in current latest gem, 0.35.