Feat/significant changes

CHANGELOG.md

@@ -8,14 +8,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Added
 - String refinements: introduce `keep_lines` and `reject_lines` methods (@robertcheramy)
+- Support for storing configurations only on significant changes (@robertcheramy)
 
 ### Changed
 - Refactored models: Use `keep_lines` and `reject_lines` in aosw, arubainstant, asa, efos, firelinuxos, fsos, ironware, mlnxos and perle to (@robertcheramy)
-
+- Modified models to support store mode on significant changes: ios, fortios, perle (@robertcheramy)
 
 ### Fixed
 - apc_aos: set comment to "; " to match comments in config.ini (@robertcheramy)
 - h3c: fix overly permissive prompt regexp causing false matches. Fixes #3673 (@robertcheramy)
+- extra/device2yaml.rb: fix \r being removed at end of line (@robertcheramy)
+- perle: remove trailing \r (the device sends \r\r\n) (@robertcheramy)
 
 
 ## [0.35.0 - 2025-12-04]

docs/Configuration.md

@@ -548,3 +548,32 @@ end
 ```
 
 Remove a previous metadata by setting it to `nil`.
+
+## Store configuration only on significant changes
+Some devices produce configuration changes even though nothing relevant
+changed. For example, Cisco IOS produces a `Last configuration change at` as
+soon as you exit config mode, and FortiOS encrypts its passwords with a
+different salt on every run.
+
+By setting the [variable](#options-credentials-vars-etc-precedence)
+`output_store_mode` to `on_significant`, you can tell Oxidized only to
+store the configuration when significant changes occurred. The default is to
+always store the configuration.
+```yaml
+vars:
+  output_store_mode: on_significant
+```
+
+For this to work, the model must implement `cmd :significant_changes`:
+```ruby
+  cmd :significant_changes do |cfg|
+    cfg.reject_lines [
+      'Last configuration change at',
+      'NVRAM config last updated at'
+    ]
+  end
+```
+
+Note that store on significant change only applies to the main configuration,
+and will not affect
+[output types](Creating-Models.md#advanced-feature-output-type)

docs/Creating-Models.md

@@ -245,7 +245,7 @@ A good (and optional) practice for submissions is to provide a
 further developments could break it, and facilitates debugging issues without
 having access to a physical network device for the model.
 
-## Advanced features
+## Advanced feature: output type
 
 The loosely-coupled architecture of Oxidized allows for easy extensibility in more advanced use cases as well.
 

docs/ModelUnitTests.md

@@ -4,6 +4,7 @@ effort to use. There are three different default unit tests for models:
 - [Device Simulation](ModelUnitTests.md#device-simulation)
 - [Device Prompt](ModelUnitTests.md#device-prompt)
 - [Secrets](ModelUnitTests.md#secrets)
+- [Significant Changes](ModelUnitTests.md#significant-changes)
 
 You only need to provide test files under [/spec/model/data](/spec/model/data),
 and the tests will be run automatically with `rake test`. See
@@ -187,6 +188,28 @@ pass:
   - 'hash-mgmt-user rocks password hash <secret removed> usertype read-only'
 ```
 
+## Significant Changes
+You can test if the model correctly detects significant changes from a YAML
+simulation file (`#simulation.yaml`) when run with variable
+`output_store_mode` set to `on_significant`.
+
+The output is checked against a file with the same 
+prefix as the yaml simulation file, but with the suffix
+`#significant_changes.yaml`.
+
+The `#significant_changes.yaml` file contains two sections with a list of
+strings or regular expressions to test:
+- pass: the test passes only if the output contains these strings (significant changes).
+- fail: the test fails if the output contain these strings (non-significant changes).
+
+```yaml
+pass:
+ - "! Processor ID: FCL2XXXXXXX"
+fail:
+ - "! Last configuration change at 13:57:08 CET Wed Mar 13 2024"
+ - "! NVRAM config last updated at 15:26:39 CET Wed Mar 13 2024 by oxidized"
+```
+
 ## Custom tests
 When you write custom tests for your models, please do not use the filenames
 mentioned above, as it will interfere with the standard tests. If you need to

docs/Ruby-API.md

@@ -98,6 +98,7 @@ gather its configuration. It can be called with:
 * A string and a block
 * `:all` and a block
 * `:secret` and a block
+* `:significant_changes` and a block
 
 The block takes a single parameter `cfg` containing the output of the command
 being processed.
@@ -118,6 +119,11 @@ given block before emitting it to hide secrets if secret hiding is enabled. The
 block should replace any secrets with `'<hidden>'` and return the resulting
 string.
 
+Calling `cmd` with `:significant_changes` and a block will pass the final
+configuration to the given block. The resulting string should contain
+significant changes only and will be used to
+[decide if the configuration should be stored](Configuration.md#store-configuration-only-on-significant-changes).
+
 Execution order is `:all`, `:secret`, and lastly the command specific block, if
 given.
 

extra/device2yaml.rb

@@ -91,9 +91,8 @@ def yaml_output(prepend = '')
 
   prepend = @sequence_prepend_output + prepend
 
-  # as we want to prepend 'prepend' to each line, we need each_line and chomp
-  # chomp removes the trainling \n
-  @ssh_output.each_line(chomp: true) do |line|
+  # each_line(chomp: true) would remove \r\n, so we prefer split
+  @ssh_output.split("\n", -1).each do |line|
     # encode line and remove the first and the trailing double quote
     line = line.dump[1..-2]
     if firstline

lib/oxidized/model/fortios.rb

@@ -108,12 +108,18 @@ class FortiOS < Oxidized::Model
     cfg.join
   end
 
+  cmd :significant_changes do |cfg|
+    cfg.reject_lines [
+      /^ +set \S+ ENC \S+$/
+    ]
+  end
+
   cfg :telnet do
     username /^[lL]ogin:/
     password /^Password:/
   end
 
   cfg :telnet, :ssh do
-    pre_logout "exit\n"
+    pre_logout "exit"
   end
 end

lib/oxidized/model/ios.rb

@@ -54,6 +54,13 @@ class IOS < Oxidized::Model
     cfg
   end
 
+  cmd :significant_changes do |cfg|
+    cfg.reject_lines [
+      /^! (Last|No) configuration change (at|since)/,
+      '! NVRAM config last updated at'
+    ]
+  end
+
   cmd 'show version' do |cfg|
     comments = []
     comments << cfg.lines.first
@@ -126,12 +133,17 @@ class IOS < Oxidized::Model
     cmd_line = 'show running-config'
     cmd_line += ' view full' if vars(:ios_rbac)
     cmd cmd_line do |cfg|
-      cfg = cfg.each_line.to_a[3..-1]
-      cfg = cfg.reject { |line| line.match /^ntp clock-period / }.join
-      cfg = cfg.each_line.reject do |line|
-        line.match /^! (Last|No) configuration change (at|since).*/ unless line =~ /\d+\sby\s\S+$/
-      end.join
-      cfg.gsub! /^Current configuration : [^\n]*\n/, ''
+      cfg = cfg.cut_head(3)
+      cfg = cfg.reject_lines [
+        /^ntp clock-period /,
+        /^Current configuration : \S+/
+      ]
+      unless vars("output_store_mode") == "on_significant"
+        cfg = cfg.reject_lines [
+          # Only store the line "configuration change" when a user is specified
+          /^! (Last|No) configuration change (at|since)(?!.*\d+ by \S+$)/
+        ]
+      end
       cfg.gsub! /^ tunnel mpls traffic-eng bandwidth[^\n]*\n*(
                     (?: [^\n]*\n*)*
                     tunnel mpls traffic-eng auto-bw)/mx, '\1'

lib/oxidized/model/model.rb

@@ -306,6 +306,13 @@ def screenscrape
       @input.class.to_s.match(/Telnet/) || vars(:ssh_no_exec)
     end
 
+    def significant_changes(config)
+      self.class.cmds[:significant_changes].each do |block|
+        config = instance_exec config, &block
+      end
+      config
+    end
+
     private
 
     def process_cmd_output(output, name)

lib/oxidized/model/perle.rb

@@ -5,7 +5,8 @@ class Perle < Oxidized::Model
   comment '! '
 
   cmd :all do |cfg|
-    cfg.cut_both
+    cfg = cfg.cut_both
+    cfg.delete "\r"
   end
 
   cmd 'show version verbose' do |cfg|
@@ -27,6 +28,12 @@ class Perle < Oxidized::Model
 
   cmd 'show running-config'
 
+  cmd :significant_changes do |cfg|
+    cfg.reject_lines [
+      /^tacacs-server key 7 \$0\$\S+==$/
+    ]
+  end
+
   cfg :ssh do
     post_login 'terminal length 0'
     pre_logout 'exit'

lib/oxidized/output/file.rb

@@ -21,8 +21,14 @@ def setup
 
       # node: node name (String)
       # outputs: Oxidized::Models::Outputs
-      # opts: hash of node vars
+      # opts: dict of optional parameters:
+      #  - group: node group
+      #  - significant_changes:
+      #      nil / not set / true  -> store as usual
+      #      false                 -> do not store
       def store(node, outputs, opt = {})
+        return false if opt[:significant_changes] == false
+
         file = ::File.expand_path @cfg.directory
         file = ::File.join ::File.dirname(file), opt[:group] if opt[:group]
         FileUtils.mkdir_p file

lib/oxidized/output/git.rb

@@ -35,6 +35,16 @@ def setup
         end
       end
 
+      # file: node name (String)
+      # outputs: Oxidized::Models::Outputs
+      # opts: dict of optional parameters:
+      #  - msg: commit message
+      #  - email: committer email
+      #  - user: committer name
+      #  - group: node group
+      #  - significant_changes:
+      #      nil / not set / true: store as usual
+      #      false: skip general config, only store configs where type != nil
       def store(file, outputs, opt = {})
         @msg   = opt[:msg]
         @user  = opt[:user]  || @cfg.user
@@ -58,7 +68,7 @@ def store(file, outputs, opt = {})
           update type_repo, file, type_cfg
         end
 
-        update repo, file, outputs.to_cfg
+        update repo, file, outputs.to_cfg unless opt[:significant_changes] == false
       end
 
       # Returns the configuration of group/node_name

lib/oxidized/output/gitcrypt.rb

@@ -86,7 +86,7 @@ def store(file, outputs, opt = {})
           update type_repo, file, type_cfg
         end
 
-        update repo, file, outputs.to_cfg
+        update repo, file, outputs.to_cfg unless opt[:significant_changes] == false
       end
 
       def fetch(node, group)

lib/oxidized/worker.rb

@@ -62,6 +62,27 @@ def reload
       @nodes.load
     end
 
+    def self.significant_changes?(job, output)
+      node = job.node
+      model = node.model
+      return true unless model.vars(:output_store_mode) == "on_significant"
+
+      unless output.respond_to?(:fetch)
+        logger.error("Detection of significant changes needs an output " \
+                     "capable of fetching the last configuration")
+        return true
+      end
+
+      old = model.significant_changes output.fetch(node, node.group)
+      new = model.significant_changes job.config.to_cfg
+      if old == new
+        logger.debug "No significant change on node #{node.name}"
+        false
+      else
+        true
+      end
+    end
+
     private
 
     def process_success(node, job)
@@ -72,8 +93,12 @@ def process_success(node, job)
       msg += " from #{node.from}" if node.from
       msg += " with message '#{node.msg}'" if node.msg
       output = node.output.new
-      if output.store node.name, job.config,
-                      msg: msg, email: node.email, user: node.user, group: node.group
+
+      significant_changes = Worker.significant_changes?(job, output)
+      if output.store(node.name, job.config,
+                      msg: msg, email: node.email, user: node.user,
+                      group: node.group,
+                      significant_changes: significant_changes)
         node.modified
         logger.info "Configuration updated for #{node.group}/#{node.name}"
         Oxidized.hooks.handle :post_store, node:      node,

spec/model/atoms.rb

@@ -192,4 +192,16 @@ def initialize(model, desc, type = 'secret')
       @skip = true if @output_test.skip?
     end
   end
+
+  class TestSignificantChange < TestPassFail
+    GLOB = '*#significant_changes.yaml'.freeze
+    attr_reader :output_test
+
+    def initialize(model, desc, type = 'significant_changes')
+      super
+
+      @output_test = TestOutput.new(@model, @desc, 'output')
+      @skip = true if @output_test.skip?
+    end
+  end
 end