The Analysis stops working after some time

[caiusb]

At first I was thinking it was a memory problem (see issue #19). But I worked on that, and it still didn't help. So I think that the problem is more severe and somewhere else.

The whole thing crashes silently apparently at some commits. Unfortunately, most of the non trivial repositories experience this bug. I will work on trying to solve it soon.

[caiusb]

I narrowed down the problem to the fact that doing a diff on a very large file (over 10,000 lines of code) takes a long time and a lot of memory. And after some time, the whole thing dies very silently. The JVM does not throw out a log, no exception is thrown, nothing. It just dies and leave no clue as to why.

Also, since the whole takes over 30 mins, it's tricky to get to the bottom of it. My current idea is to skip very large files, so that we can still have some partial results. However, that will probably invalidate our results, since we bias them to change only in files up X lines of code. That might not look good in the paper. The actual change in the very huge file is very small (about 10 or so lines), so looking manually in these huge files might also be an option, if I can't get the automatic analysis to work.

Any other ideas @altern and @cdmihai?

[altern]

Let's just skip large files. I suggest to exclude files from analysis if they are larger than ~2000 LOC. Or at least turn off syntax analysis for the large files

[caiusb]

The problem is that even though the files are large, the actual change is small (as it is in this case). So it's a valid change that was surely done by a human, and that we exclude for the very reason that our tool can't handle them...

It's fine to exclude, but we might get into trouble when we want to defend ourselves in the paper as to why did we choose to do this (other than obvious reason).

[caiusb]

So I fixed one of the problems. For every new file that changed, I create a new file in the eclipse workspace. This way, I avoid everything from filling up with muck. I also close, the editor and delete the file in the end just to make sure I don't fill my hard drive and memory with useless files and stuff.

However, it continues to fail silently. Now it does it on the 5th commit. To make matters worse, it works perfectly fine on my laptop but fails miserably on the server. I think that this is a configuration issue so I will try to find what exactly is it. If everything else fails, I might need to run the analysis on my laptop...