support dynamic data list for the dataset APIs (2/July)

[wyli]

Is your feature request related to a problem? Please describe.
At the workflow level, would be great to have the capability of changing the input training dataset on the fly, without restarting the running engine. This would require some extra works for the cache-based datasets.

cc @SachidanandAlle @diazandr3s

[SachidanandAlle]

one simple option could be.. take filename as input.. instead of list[dict].. and at every epoch load dataset dict from file..

[Nic-Ma]

Hi @SachidanandAlle ,

I think we are still not very clear about your request, is there any example code?

Thanks.

[wyli]

I think Sachi's workaround is to have a file such as 'training.json' to list the names of training images. we keep the 'training.json' unchanged, and change the content of 'training.json' during training.

[Nic-Ma]

I will try to enhance the CacheDataset to support the "rerun" of caching progress if datalist updated during training.

Thanks.

[SachidanandAlle]

How about persistent Dataset.. ?

[Nic-Ma]

Hi @SachidanandAlle ,

Sure, let me try to investigate both.

Thanks.

[Nic-Ma]

Hi @SachidanandAlle ,

I submitted PR: #2436 for this ticket, could you please help review it?

Thanks in advance.

[SachidanandAlle]

This is little difficult to use.. I have to call update explicity. Basically I have to save the dataset object..
I thought we can call use filename and whenver iterator starts, we simple load list from the file...

[Nic-Ma]

Hi @SachidanandAlle ,

Thanks for your feedback.
To support this feature in caching datasets, I think at least we need a way to notify the dataset to reload data and compute cache content, right? I think you will reload datalist for example every 100 epochs? If you need to reload datalist for every epoch, you don't need to cache at all.
Or maybe I misunderstood your question?

Thanks.

[diazandr3s]

Thanks for you help with this @Nic-Ma
Rather than reloading datalist every certain number of epochs, it should be loaded after new label is detected.

[Nic-Ma]

Hi @diazandr3s ,

Thanks for providing the feedback.
How did you detect a new label came? I think it should be controlled in your app program, then you can call the new set_data() API of datasets when you detected new label?

Thanks.