Lock-step batch processing & writing #5
Description
This would be an experiment to see if it speeds up indexing overall.
Right now, many instances of multifeed-index run concurrently, writing their own batches to whatever storage they use, plus writing index state changes (fetchState,storeState). Having 10 indexes over 10000 entries /w batchsize=100 means 100 * 2 * 10 = 2000 writes.
What if multifeed-index managed a set of views? If it used shared storage for storing state, and, when possible, did mapping in lock-step, it could reduce the # of writes considerably, more like 100 writes instead, from the above example. Fewer reads too, since a batch of 100 entries would be sent to all views at the same time.
These gains are only possible when all views are indexed up to the same point. Otherwise, they'd need to have separate fetch/process/write loops. It might still cost less to detect this, wipe all views to get them back to the same state, and then re-index them all together.