bug: An alert was sent out when a required command failed repeatedly

[capnspacehook]

Describe the bug

I created a config for a service that recently had a new version released. I install the service on the server via the package manager, so I use a latest_version.require.command to run a script that verifies that the new release has a corresponding release via the package manager. The script is written so if the package manager has the new version it returns 0, and 1 otherwise.

I noticed that whenever a required command fails the command is run dozens of times in the span of a few seconds over and over. Ideally this command would be run only on the interval of the configured service (in my case I'm using the default of 10m).

Despite the required command failing repeatedly, I got a notification for the new release.

To Reproduce
Steps to reproduce the behavior:

1. Create a service with a latest_version.require.command that always returns non-zero
2. Ensure the service has a newer release than the deployed version
3. Watch the logs as the required command is run over and over

Expected behavior

I expected to see the required command run once a configured update interval, and not to get a notification until the required command was successful.

Desktop (please complete the following information):

• OS: Linux
• Browser: chrome
• Version: 143

Additional context

Redacted example of a service that triggered this for me:

service:
  my/service:                                                                                                                            
    latest_version:                                                                                                                                   
      type: github                                                         
      url: my/service                                                                                                               
      require:                                                             
        command: ["sh", "/bin/check-pkg.sh", "my_service", "{{ version }}"]                          
    deployed_version:                                                      
      type: url                                                                                                                                       
      url: https://service.example.com/api/version     
      regex: "([0-9.]+)"                                                                                                                              
    notify:                                                                
      ntfy:                                                                                                                                           
        params:                                                            
          click: "https://github.com/my/service/releases/{{ version }}"                               
      web_url: "https://github.com/my/service/releases/{{ version }}"

[JosephKav]

Hi,
I think I'd need to know a bit more about what your script runs and what alerts you're getting. You're using a github Lookup, so the require.command will run once for each release tag until it finds one that passes

Edit: looks like require.command doesn't have the version you're checking for, it just uses the previously verified latest_version. I'll get a fix out for this shortly! Thanks for reporting

[capnspacehook]

Thanks for the quick fix @JosephKav! Does that PR also fix the issue where a failed latest version command is repeated in what seems a hot loop?

It might be nice to standardize exit codes for commands Argus runs, ie 0 is true, 1 is false (but still successful exec) and anything else is a failure that should be logged differently, ie wrong number of args

[JosephKav]

Argus/service/latest_version/types/github/query.go

Lines 407 to 410 in 1e25ecf

	// If the Command didn't return successfully.
	if err := l.Require.ExecCommand(version, logFrom); err != nil {
	return "", "", err //nolint:wrapcheck
	}

Argus/service/latest_version/types/github/query.go

Lines 449 to 456 in 1e25ecf

	var firstErr error
	for _, release := range filteredReleases {
	if v, rd, err := l.releaseMeetsRequirements(release, logFrom); err == nil {
	return v, rd, nil
	} else if firstErr == nil {
	firstErr = err
	}
	}

To me it doesn't look like it will repeat a command that fails. Prior to #796, every command would have used the current latest_version.

I think what was happening was:

• you have a command that checks apt for a release being available ({{ version }}).
• latest_version query finds a new release
• your latest_version.require.command runs your command to check that release is available in apt (but it checks with the previous version, not this new candidate)
• because of this, the command will pass and this newer version will be treated as available.
• now all future queries will run that command with this version that isn't available, so now the commands will all fail (they all used the same value for {{ version }})

[capnspacehook]

That sounds like a reasonable explanation, I'll let you know if it doesn't fix the issue for me after the new version is released. Thanks again!

[JosephKav]

Hopefully it does, thank you.
Please do if it doesn't fix it