Hi, I am training my model using the AdobeDataAffineHR class in data_loader.py as a pipeline. But there is a stable increment of buff/cache memory and the process would be killed in some epochs, and those memory won't be released after exiting the program. And I comment all codes except the data fetching and the memory still increases. I also try to set num_workers of dataloader to be 0 and the same phenomenon occurs too.
Does anyone know how to solve this bug?
Hi, I am training my model using the AdobeDataAffineHR class in data_loader.py as a pipeline. But there is a stable increment of buff/cache memory and the process would be killed in some epochs, and those memory won't be released after exiting the program. And I comment all codes except the data fetching and the memory still increases. I also try to set num_workers of dataloader to be 0 and the same phenomenon occurs too.
Does anyone know how to solve this bug?