More on log rotation

[TheMint]

Hi,

Following my previous request for CSV log rotations with header, I was wondering if it is possible to save my rotated logs in a different directory path than the path where the latest log file is written. This makes collecting logs easier later on as I don't have to explicitly avoid interacting with the file currently being written.

I was also wondering if it was possible to add a custom string to the filename of rotated files. From what I've read, RotatingFileSinkConfig::RotationNamingScheme currently supports Index, Time, and DateAndTime, but it would be nice to have a custom tag like "rotated" specifically on log files that have been rotated.

Thanks!

[odygrd]

Hey,

Unfortunately, implementing this functionality in the current RotatingFileSink is quite complex due to its design. For example, during each rotation, the sink actively renames files, with .1. being the most recent rotated file. This means that:

• file.log is always the active log.
• On rotation, file.log becomes file.1.log, file.1.log becomes file.2.log, and so on.

This renaming scheme ensures a consistent order but makes collecting rotated logs separately more challenging, as filenames are altered and context can vary. Additionally, existing features like cleaning old files and recovering indexes (in append mode) further increase the complexity of adding support for saving rotated logs in a different directory or customizing the filename format.

If your use case is simple, you can create a custom rotating sink. Below is an example of a custom implementation that rotates logs into a separate directory without renaming them:

#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/FileSink.h"

#include <utility>

class CustomSink final : public quill::FileSink
{
public:
  CustomSink(quill::fs::path const& filename, size_t max_size, quill::FileSinkConfig const& config)
    : quill::FileSink(filename, config), _max_size(max_size)
  {

  }

  ~CustomSink() override = default;

  QUILL_ATTRIBUTE_HOT void write_log(quill::MacroMetadata const* log_metadata, uint64_t log_timestamp,
                                     std::string_view thread_id, std::string_view thread_name,
                                     std::string const& process_id, std::string_view logger_name,
                                     quill::LogLevel log_level, std::string_view log_level_description,
                                     std::string_view log_level_short_code,
                                     std::vector<std::pair<std::string, std::string>> const* named_args,
                                     std::string_view log_message, std::string_view log_statement) override
  {
    // rotate
    if (_file_size + log_statement.size() > _max_size)
    {
      // We need to flush and also fsync before actually getting the size of the file
      quill::FileSink::flush_sink();
      quill::FileSink::fsync_file(true);

      if (get_file_size(this->_filename) <= 0)
      {
        // Also check the file size is > 0  to better deal with full disk
        return;
      }

      this->close_file();

      // rename the file and move it to a new dir
      std::filesystem::path rotated_path = this->_filename.parent_path() / "rotated";
      std::filesystem::create_directory(rotated_path);

      std::filesystem::path rotated_file = rotated_path / (this->_filename.filename().string() + "." + std::to_string(++_idx));
      std::filesystem::rename(this->_filename, rotated_file);

      this->open_file(this->_filename, "w");
      _file_size = 0;
    }

    // write to file
    quill::FileSink::write_log(log_metadata, log_timestamp, thread_id, thread_name, process_id,
                         logger_name, log_level, log_level_description, log_level_short_code,
                         named_args, log_message, log_statement);

    _file_size += log_statement.size();
  }

  /***/
  QUILL_NODISCARD static size_t get_file_size(quill::fs::path const& filename)
  {
    return static_cast<size_t>(quill::fs::file_size(filename));
  }

private:
  size_t _file_size{0};
  size_t _idx{0};
  size_t _max_size{0};
};

int main()
{
  // Start the backend thread
  quill::BackendOptions backend_options;
  quill::Backend::start(backend_options);

  // Frontend
  size_t const max_size = 512;

  auto rotating_file_sink = quill::Frontend::create_or_get_sink<CustomSink>(
    "rotating_file.log",
    max_size,
    []()
    {
      // See RotatingFileSinkConfig for more options
      quill::FileSinkConfig cfg;
      cfg.set_open_mode('w');
      return cfg;
    }());

  quill::Logger* logger = quill::Frontend::create_or_get_logger(
    "root", std::move(rotating_file_sink),
    quill::PatternFormatterOptions{"%(time) [%(thread_id)] %(short_source_location:<28) "
                                   "LOG_%(log_level:<9) %(logger:<12) %(message)",
                                   "%H:%M:%S.%Qns", quill::Timezone::GmtTime});

  for (int i = 0; i < 20; ++i)
  {
    LOG_INFO(logger, "Hello from rotating logger, index is {}", i);
  }
}

The above example rotates logs based on file size, moves them to a rotated/ subdirectory, and avoids renaming older files. You could extend this to add custom strings to the filenames as well.

Alternatively, you can use RotationNamingScheme::DateAndTime for rotated filenames, and pair it with a script to filter or organize logs after they are written.

#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/RotatingFileSink.h"

#include <utility>

int main()
{
  // Start the backend thread
  quill::BackendOptions backend_options;
  quill::Backend::start(backend_options);

  // Frontend
  auto rotating_file_sink = quill::Frontend::create_or_get_sink<quill::RotatingFileSink>(
    "rotating_file.log",
    []()
    {
      // See RotatingFileSinkConfig for more options
      quill::RotatingFileSinkConfig cfg;
      cfg.set_open_mode('w');
      cfg.set_filename_append_option(quill::FilenameAppendOption::None);
      cfg.set_rotation_naming_scheme(quill::RotatingFileSinkConfig::RotationNamingScheme::DateAndTime);
      cfg.set_rotation_time_daily("18:30");
      cfg.set_rotation_max_file_size(1024); // small value to demonstrate the example
      return cfg;
    }());

  quill::Logger* logger = quill::Frontend::create_or_get_logger(
    "root", std::move(rotating_file_sink),
    quill::PatternFormatterOptions{"%(time) [%(thread_id)] %(short_source_location:<28) "
                                   "LOG_%(log_level:<9) %(logger:<12) %(message)",
                                   "%H:%M:%S.%Qns", quill::Timezone::GmtTime});

  for (int i = 0; i < 8; ++i)
  {
    LOG_INFO(logger, "Hello from rotating logger, index is {}", i);
  }

  std::this_thread::sleep_for(std::chrono::seconds {5});

  for (int i = 0; i < 8; ++i)
  {
    LOG_INFO(logger, "Hello from rotating logger, index is {}", i);
  }

  std::this_thread::sleep_for(std::chrono::seconds{8});

  for (int i = 0; i < 4; ++i)
  {
    LOG_INFO(logger, "Hello from rotating logger, index is {}", i);
  }
}

When using the DateAndTime rotation naming scheme, you will see files with a timestamp in their names. For example:

• rotating_file.20250311_142537.log
• rotating_file.20250311_142542.log
• rotating_file.log

Here, rotating_file.log is the current log being actively written, while rotating_file.20250311_142537.log and rotating_file.20250311_142542.log are rotated logs with timestamps. This can make collecting and organizing logs easier.