🔎 Search Terms
file, disk usage
The problem
Environment: Ubuntu 18.04 LTS
On running the winston file transport, we found that with the tailable option is set to true, on rotation of logs, files are renamed up to max files, but old file descriptors remain open by the node process.
The result is that if you calculate file space usage using a tool like "du" the deleted files no longer show up, but the disk space is still "used" and will be reported as used by tool like "df". If the node application using the winston file transport in this way is left running long enough, eventually all file descriptors on the partition will be in use by the node process, which will result in the failure to create any new files due to no file descriptor availability.
What version of Winston presents the issue?
v3.2.1 and v3.6.0
What version of Node are you using?
v14.19.0
If this worked in a previous version of Winston, which was it?
unknown (but works in Node v12 and Node v10)
Minimum Working Example
This example will set up the file rotation such that you will quickly see the issue. You can play with the MAX_SIZE, MAX_FILES, and TIMEOUT_MS if the issue is happening too fast.
'use strict';
const MAX_SIZE = 2000;
const MAX_FILES = 10;
const TIMEOUT_MS = 1000;
const winston = require('winston');
const {format} = require('logform');
const path = require('path');
const {combine, timestamp, json} = format;
// Set up file transport - I assume logs folder exists
const logsDir = path.resolve(process.cwd(), './logs');
const filePath = path.join(logsDir, 'example.log');
const fileTransport = {
tailable: true,
maxsize: MAX_SIZE,
maxFiles: MAX_FILES,
filename: filePath,
handleExceptions: true,
level: 'debug',
};
fileTransport.format = combine(
timestamp({format: 'YYYY-MM-DD HH:mm:ss.SSS'}),
json(),
);
// Create logger with file transport
const logger = winston.createLogger({
transports: [new winston.transports.File(fileTransport)],
exitOnError: false,
});
logger.log('info', 'Created file logger');
// Add periodic logs so that files are rotated
let i = 0;
setTimeout(timeoutLogger, TIMEOUT_MS);
function timeoutLogger() {
i++;
logger.log('info', 'another log message ' + i);
logger.log('error', 'error message ' + i);
logger.log('http', 'http log message ' + i);
logger.log('verbose', 'verbose log message ' + i);
logger.log('debug', 'debug log message ' + i);
setTimeout(timeoutLogger, TIMEOUT_MS);
}While running this example, use a terminal to check the file descriptor status using:
lsof | grep example | grep deletedps aux | grep node (to find the node PID) and then ls -al /proc/${PID}/fd
You'll see "deleted" files associated with open file descriptors.
Additional information
As I mentioned above, this problem is occurring in Node 14 and we did NOT see it in Node 12 or Node 10. A check through the Node 14 documentation showed a change to the fd garbage collection that may be what is triggering this issue now:
nodejs/node#28396
I poked around the winston file transport code, and the problem seems to be tied to calling the _rotateFile() function multiple times (multiple logs triggering the logic that the max file size has been reached and the file needs to be rotated). That function then calls the _incFile() function multiple times. The asynchronous handling of the multiple calls seems to cause the file descriptor cruft.
Just to gain some proof, I modified the line lib/winston/transports/file.js:172 as follows, and that does fix the problem.
// Original line:
this._endStream(() => this._rotateFile());
// My change:
this._endStream(() => {
if (this._rotate === true) {
this._rotate = false;
this._rotateFile();
}
});I don't suppose that is a good and robust fix, but maybe it points to a potential solution. I'm happy to try out potential solutions if the experts have thoughts on that. Please let me know if you need any other information or if there is anything I can do to help resolve this.
🔎 Search Terms
file, disk usage
The problem
Environment: Ubuntu 18.04 LTS
On running the winston file transport, we found that with the tailable option is set to true, on rotation of logs, files are renamed up to max files, but old file descriptors remain open by the node process.
The result is that if you calculate file space usage using a tool like "du" the deleted files no longer show up, but the disk space is still "used" and will be reported as used by tool like "df". If the node application using the winston file transport in this way is left running long enough, eventually all file descriptors on the partition will be in use by the node process, which will result in the failure to create any new files due to no file descriptor availability.
What version of Winston presents the issue?
v3.2.1 and v3.6.0
What version of Node are you using?
v14.19.0
If this worked in a previous version of Winston, which was it?
unknown (but works in Node v12 and Node v10)
Minimum Working Example
This example will set up the file rotation such that you will quickly see the issue. You can play with the MAX_SIZE, MAX_FILES, and TIMEOUT_MS if the issue is happening too fast.
While running this example, use a terminal to check the file descriptor status using:
lsof | grep example | grep deletedps aux | grep node(to find the node PID) and thenls -al /proc/${PID}/fdYou'll see "deleted" files associated with open file descriptors.
Additional information
As I mentioned above, this problem is occurring in Node 14 and we did NOT see it in Node 12 or Node 10. A check through the Node 14 documentation showed a change to the fd garbage collection that may be what is triggering this issue now:
nodejs/node#28396
I poked around the winston file transport code, and the problem seems to be tied to calling the
_rotateFile()function multiple times (multiple logs triggering the logic that the max file size has been reached and the file needs to be rotated). That function then calls the_incFile()function multiple times. The asynchronous handling of the multiple calls seems to cause the file descriptor cruft.Just to gain some proof, I modified the line lib/winston/transports/file.js:172 as follows, and that does fix the problem.
I don't suppose that is a good and robust fix, but maybe it points to a potential solution. I'm happy to try out potential solutions if the experts have thoughts on that. Please let me know if you need any other information or if there is anything I can do to help resolve this.