Apc_aos blocks the whole oxidized instance

[robertcheramy]

• I have read and understood docs/Issues.md.

Describe the bug
I use oxidized to retrieve the configuration of our Schneider/APC UPS systems with scp using the apc_aos model.
After some days, oxidized does not fetch any configurations anymore.
Removing the apc_aos devices solves the problem.

To Reproduce
Note: I'm working on this. Changing the logging facility of oxidized was the first step, so I can dump a backtrace of each thread and change the logging level on the fly (=> debug).

Steps to reproduce the behavior:

1. Fetch the configs of apc_aos devices
2. Wait some days
3. Oxidized does not retrieve any configuration anymore

This is reproducible on our production server.

Expected behavior
Using the aos_apc model does not block oxidized from retrieving configurations.

Configuration

Not relevant

Logs

No relevant logs for the moment

Running environment (please complete the following information):

• OS: official container version 0.33.0 [currently testing with 0.34.1]
• oxidized version: 0.33.0 [currently testing with 0.34.1]
• oxidized-web version: 0.17.0
• Manufacturer model an software version: APC UPS Management Card AP9640
• oxidized model name: apc_aos

Additional context

[robertcheramy]

Work related to this bug goes into Branch fix/3590-apc_aos

[ytti]

ìnput/ssh inherits from input not from cli so disconnect is never called?

Ah no, it includes CLI and gets get from there, so disconnect should be called. But do check if amount of file descriptors open keeps increasing after every run, eventually making it impossible to open new descriptors. And if the threads never die., eventually starving system from starting new threads.

[robertcheramy]

Net::SCP#download! gets stuck an never returns, so the threads are not freed:

2025-07-25 06:42:05.092233 W [24:Oxidized::Job 'APC124.xx.xx event_loop.rb:115] Thread Dump -- Backtrace:
/var/lib/gems/3.2.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:115:in `select'
/var/lib/gems/3.2.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:115:in `ev_select_and_postprocess'
/var/lib/gems/3.2.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:30:in `process'
/var/lib/gems/3.2.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:226:in `process'
/var/lib/gems/3.2.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in `block in loop'
/var/lib/gems/3.2.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in `loop'
/var/lib/gems/3.2.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in `loop'
/var/lib/gems/3.2.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/channel.rb:272:in `wait'
/var/lib/gems/3.2.0/gems/net-scp-4.1.0/lib/net/scp.rb:321:in `download!'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/input/scp.rb:33:in `cmd'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/model/model.rb:123:in `cmd'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/model/model.rb:173:in `block in get'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/model/model.rb:172:in `each'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/model/model.rb:172:in `get'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/input/cli.rb:14:in `get'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/node.rb:75:in `run_input'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/node.rb:50:in `block in run'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/node.rb:44:in `each'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/node.rb:44:in `run'
/var/lib/gems/3.2.0/gems/oxidized-0.34.1/lib/oxidized/job.rb:13:in `block in initialize'

[ytti]

Maybe we should enforce Timeout here:
https://github.com/ytti/oxidized/blob/master/lib/oxidized/job.rb#L13

Obviously we need to fix the input/scp for this as well, but might be good to have backstop in the thread too, so any future bug with same symptom breaks the node being collected, not oxidized.

Possibly something like this:

diff --git a/lib/oxidized/job.rb b/lib/oxidized/job.rb
index 6e217eaf..8645ce89 100644
--- a/lib/oxidized/job.rb
+++ b/lib/oxidized/job.rb
@@ -10,11 +10,15 @@ module Oxidized
       self.name = "Oxidized::Job '#{@node.name}'"
       super do
         logger.debug "Starting fetching process for #{@node.name} at #{Time.now.utc}"
-        @status, @config = @node.run
+        Timeout.timeout(Oxidized.config.timeout * (@node.model.cmds.size+1)) do
+          @status, @config = @node.run
+        end
         @end             = Time.now.utc
         @time            = @end - @start
         logger.debug "Config fetched for #{@node.name} at #{@end}"
       end
+    rescue Timeout::Error
+      logger.debug "#{node.name} thread timed out"
+      @status = :timeout
+      @end = Time.now.utc ## should these be in ensure, to be more dry?
+      @time = @end - @start
     end
   end
 end

Didn't test at all, even for syntax.

[ytti]

SCP likely needs line like this somewhere:
https://github.com/ytti/oxidized/blob/master/lib/oxidized/input/ssh.rb#L81

[robertcheramy]

Yes, and also ssh options. I'm working on it.

[robertcheramy]

The actual fix solves the timeout at connect, but not while Net::SCP's download! [frequency: one random APC of about 300] and I'm not sure (yet) that I can fix this without touching the code of Net::SCP.

I would implement a configurable time limit (not timeout) per job (I think a default of 300 seconds should be OK) based on the proposal of @ytti above.

[robertcheramy]

I'm testing the fix on live systems, will PR & merge in master if OK.