Relying on numpy.random makes results hard to reproduce

[larsmans]

... and it also confuses users of LDA.

In scikit-learn, we've solved this by requiring a random_state argument to anything that wants random numbers; this may either be an np.random.RandomState object, or the seed for one. You might want to borrow this habit.

[adammenges]

👍

[piskvorky]

Here's a link to a pull request that did the same for GloVe; looks simple enough:
maciejkula/glove-python#32

[cvint13]

When I run the model I still get different results every time despite using a fixed seed:

lda = LdaMulticore(corpus, num_topics=100, workers=4,id2word=dictionary,
                   random_state=np.random.RandomState(seed=10101010),
                  alpha=np.array([.0075] * 100))

[gojomo]

@cvint13 Anything multithreaded will also be subject to scheduling jitter from the OS – operations won't happen in the same order, and thus results will vary. For full reproduceability, you'd have to move to a (much-slower) single-threaded calculation (workers=1).

[cvint13]

Ahhh ok, that sucks.

[piskvorky]

For completeness -- LdaMulticore uses processes, not threads.

@menshikh-iv is it expected that @cvint13 would get different results despite a fixed seed? Are the individual jobs processed in arbitrary (non-deterministic) order by multiple workers?

[menshikh-iv]

@piskvorky yes (for multicore implementation only).