Index doesn't work as expected with partial NULL keys

[PhantomYdn]

ArcadeDB Version: 10.2021

JDK Version: 11

OS: win

The following testcase is failing if executed first time and pretty OK if executed second and etc.

@Test
	public void testDocumentAfterCreation() {
		DocumentType typeRoot = database.getSchema().getOrCreateDocumentType("TestRoot");
		typeRoot.getOrCreateProperty("name", String.class);
		typeRoot.getOrCreateTypeIndex(INDEX_TYPE.FULL_TEXT, true, "name");
		database.command("sql", "delete from TestRoot");
		
		DocumentType typeChild = database.getSchema().getOrCreateDocumentType("TestChild");
		typeChild.setParentTypes(Arrays.asList(typeRoot));
		MutableDocument doc =  database.newDocument("TestChild");
		doc.set("name", "Document Name");
		assertEquals("Document Name", doc.get("name"));
		doc.save();
		assertEquals("Document Name", doc.get("name"));
		try(ResultSet rs = database.query("sql", "select from TestChild where name = :name", CommonUtils.toMap("arg0", "Test2", "name", "Document Name"))) {
			assertTrue(rs.hasNext());
			Document docRetrieved = rs.next().getElement().orElse(null);
			assertEquals("Document Name", docRetrieved.get("name"));
			assertFalse(rs.hasNext());
		}
	}

Type of the index and uniqueness doesn't matter here.

Moreover, slight modification of indexes leads to the testcase which always fail: first, second and etc.

	@Test
	public void testDocumentAfterCreation2() {
		DocumentType typeRoot = database.getSchema().getOrCreateDocumentType("TestRoot2");
		typeRoot.getOrCreateProperty("name", String.class);
		typeRoot.getOrCreateProperty("parent", Type.LINK);
		typeRoot.getOrCreateTypeIndex(INDEX_TYPE.LSM_TREE, true, "name", "parent");
		database.command("sql", "delete from TestRoot2");
		
		DocumentType typeChild = database.getSchema().getOrCreateDocumentType("TestChild2");
		typeChild.setParentTypes(Arrays.asList(typeRoot));
		MutableDocument doc =  database.newDocument("TestChild2");
		doc.set("name", "Document Name");
		assertEquals("Document Name", doc.get("name"));
		doc.save();
		assertEquals("Document Name", doc.get("name"));
		try(ResultSet rs = database.query("sql", "select from TestChild2 where name = :name", CommonUtils.toMap("arg0", "Test2", "name", "Document Name"))) {
			assertTrue(rs.hasNext());
			Document docRetrieved = rs.next().getElement().orElse(null);
			assertEquals("Document Name", docRetrieved.get("name"));
			assertFalse(rs.hasNext());
		}
	}

Both test cases become successful if delete line with the creation of an index.

[lvca]

@PhantomYdn thanks for the report with attached the test case, I'm going to test it today.

[lvca]

It looks like the problem is with the doc.save(): it doesn't put the keys in the index because it's not considering indexes of sub-types. So when the query uses the index, it can't find the document because it was never indexed in the first place. Working on it.

[lvca]

Digging a little bit more the issue is about how indexes on the base type are used. In ArcadeDB an index is on a bucket. If you're using a Type composed by, let's say, 4 buckets, you have 4 indexes managed transparently by the TypeIndex attached to the type.

This allows nice things (in comparison to OrientDB), like:

• real parallel operation by squeezing multi-core CPUs with one thread = one bucket+index and zero conflict/lock/mutex
• drop a bucket means dropping the associated indexes without a full scan and re-indexing

So one way could be to auto-create indexes, one per bucket, as soon as a type add another as super-type, but what to do in case the super-type relationship is removed? The indexes should stay just undetached from the super-type.

Any thoughts on this are welcome.

[lvca]

I can already see me writing tons of new test cases coming to cover all these cases that were actually missing :-( Any help is welcome :-)

[PhantomYdn]

On top of that: what will you do with unique indexes on super type? Uniqueness should be guaranteed across all child, subchild and etc... Regarding help: do you expect to extend some existing testcase class or new one will be OK?

[lvca]

I think a new test class is more appropriate, I've just created com.arcadedb.index.LSMTreeIndexPolymorphicTest with your test case there (now commented). I'm going to fix this issue and add test cases, feel free to contribute on that test class if you can. Thanks.

[lvca]

@PhantomYdn fixed in #153, will merge if no regressions are found. I've elaborated your test case adding add + remove of the parent and testing everything.