In previous versions of neo4j, people have used an "lookup node" pattern to access certain subgraphs or subsets of their data. The "lookup node" pattern basically creates a single node to act as the entry of an index, and relationships from that "lookup node" to whatever the contents of the index would be.
This can also be used for other use cases, such as creating partitions.
More broadly, these kinds of modeling tricks turn out to be very useful to capture a variety of different kinds of domain semantics, including all sorts of set membership expression:
- Expressing that all linked nodes share some common property, without expressing the property individually in every node (i.e. companies whose home country = 'USA')
- Expressing that all linked nodes are part of the same semantic class ("Person")
- Expressing more complex pre-computed graphs of data, akin to "views" in an RDBMS.
Now that the 2.0 series of neo4j supports labels, there's an alternate approach that's possible. Instead of creating an index node, you could attach a label to all of the nodes that would otherwise participate in the "grouping" indicated by that "lookup node" and the relationships leading off of it.
Create sample graphs of 100,000 nodes, and evaluate the relative performance characteristics of using a "lookup node" approach versus a "labeling" approach. Prior to neo4j 2, the lookup node seemed the way to go. Now that labels are possible, we want to evaluate which approach is better, and discuss the modeling consequences.
Using labels is often about twice as fast as using a "lookup node", even after accounting for the amount of time spent locating the lookup node. Data supporting this conclusion can be found below.
I found this a bit counter-intuitive, as normally we think of neo4j as being more performant on queries that traverse relationships than on "bulk node lookup" type queries. The lookup node pattern exploits relationship traversal, where purely label-based queries are simple queries over nodes (albeit facilitated by the label index).
Nodes can take on as many labels as we like. So if the only purpose of a "lookup node" is to indicate set membership, (to include index, partition, or semantic class membership), it would seem that labels are the way to go irrespective of what kind of set membership we're trying to model.
On the other hand, what labels don't give you is the ability to express properties on the labels, or dynamic querying of the labels themselves. So if the set of things that we were expressing itself needed metadata, from a modeling perspective we'd be better off making that a node, and being able to attach properties and further relationships to that node. We can certainly invent a new label for every possible set grouping, but it could become problematic later. Consider a set grouping that grouped 100,000 nodes into a "partition". If you wanted to state about that partition that it should be deleted after 30 days, then it would seem to be better to make the partition a node itself and annotate it with properties, rather than naming a label something like "PartitionApril1DeleteAfter30Days". :)
#!/usr/bin/perl -w
use strict;
my $count = shift(@ARGV);
if(!$count || $count <= 0) { $count = 1_000; }
my $d = `date`; chomp($d);
print "CREATE (n:PARTITION { name: '$d' });\n";
for(my $x=0; $x<$count; $x++) {
print "MATCH (p:PARTITION { name: '$d' }) CREATE p-[r:HAS]->(n:TESTIDX { prop: 'Counter $x' });\n";
} # End forThis script is run and piped through neo4j-shell.
#!/usr/bin/perl -w
use strict;
my $count = shift(@ARGV);
if(!$count || $count <= 0) { $count = 1_000; }
my $pNo = int(rand()*1000);
for(my $x=0; $x<$count; $x++) {
print "CREATE (n:TEST { prop: 'Counter $x' });\n";
} # End forWe'll now look at some sample queries run on this data to see which perform better and how.
neo4j-sh (?)$ match (p:PARTITION)-[r:HAS]->(n:TESTIDX) return count(distinct(n));
+--------------------+
| count(distinct(n)) |
+--------------------+
| 100000 |
+--------------------+
neo4j-sh (?)$ match (n:TEST) return count(distinct(n));
+--------------------+
| count(distinct(n)) |
+--------------------+
| 100000 |
+--------------------+
$ perl -e 'for(my $x=0; $x<20; $x++) { print "MATCH (p:PARTITION) return p;\n"; }' | neo4j-shell | grep ms
2 ms
4 ms
3 ms
4 ms
6 ms
3 ms
4 ms
3 ms
3 ms
5 ms
7 ms
5 ms
3 ms
3 ms
4 ms
9 ms
4 ms
3 ms
2 ms
2 ms
$ perl -e 'for(my $x=0; $x<20; $x++) { print "match (p:PARTITION)-[r:HAS]->(n:TESTIDX) return n order by n.prop limit 10;\n"; }' | neo4j-shell | grep ms
403 ms
386 ms
405 ms
404 ms
433 ms
411 ms
403 ms
385 ms
383 ms
386 ms
392 ms
389 ms
402 ms
403 ms
380 ms
403 ms
399 ms
391 ms
394 ms
402 ms
$ match (p:TEST) return p order by p.prop limit 10;
x@abhoth:~/neo4j-test$ perl -e 'for(my $x=0; $x<20; $x++) { print "match (p:TEST) return p order by p.prop limit 10;\n"; }' | neo4j-shell | grep ms
193 ms
197 ms
201 ms
219 ms
209 ms
204 ms
209 ms
207 ms
213 ms
199 ms
201 ms
206 ms
206 ms
199 ms
197 ms
196 ms
199 ms
200 ms
213 ms
199 ms
Executing this query 100 times yields the following times:
$ perl -e 'for(my $x=0; $x<100; $x++) { print "match (p:PARTITION)-[r:HAS]->(n:TESTIDX) return count(n);\n"; }' | neo4j-shell | grep ms
277 ms
281 ms
279 ms
273 ms
274 ms
296 ms
277 ms
291 ms
295 ms
284 ms
285 ms
285 ms
279 ms
411 ms
295 ms
335 ms
319 ms
339 ms
309 ms
285 ms
289 ms
295 ms
282 ms
288 ms
276 ms
281 ms
280 ms
274 ms
293 ms
287 ms
312 ms
282 ms
305 ms
281 ms
285 ms
295 ms
286 ms
295 ms
299 ms
289 ms
296 ms
294 ms
298 ms
315 ms
310 ms
307 ms
309 ms
302 ms
304 ms
294 ms
296 ms
281 ms
287 ms
295 ms
303 ms
316 ms
322 ms
288 ms
313 ms
313 ms
320 ms
317 ms
302 ms
307 ms
302 ms
303 ms
297 ms
314 ms
292 ms
288 ms
308 ms
296 ms
291 ms
275 ms
271 ms
291 ms
296 ms
316 ms
313 ms
329 ms
312 ms
303 ms
302 ms
293 ms
284 ms
295 ms
279 ms
276 ms
281 ms
326 ms
393 ms
293 ms
274 ms
287 ms
292 ms
275 ms
285 ms
281 ms
283 ms
277 ms
Now on the index nodes.
$ perl -e 'for(my $x=0; $x<100; $x++) { print "match (p:TEST) return count(p);\n"; }' | neo4j-shell | grep ms
121 ms
119 ms
120 ms
122 ms
124 ms
123 ms
120 ms
121 ms
128 ms
121 ms
123 ms
120 ms
123 ms
116 ms
121 ms
131 ms
126 ms
132 ms
125 ms
121 ms
121 ms
121 ms
120 ms
121 ms
131 ms
122 ms
124 ms
120 ms
118 ms
122 ms
120 ms
115 ms
120 ms
115 ms
121 ms
118 ms
117 ms
120 ms
119 ms
126 ms
123 ms
122 ms
120 ms
122 ms
116 ms
117 ms
120 ms
117 ms
119 ms
117 ms
116 ms
115 ms
119 ms
118 ms
117 ms
115 ms
118 ms
117 ms
116 ms
116 ms
121 ms
122 ms
114 ms
119 ms
119 ms
122 ms
118 ms
118 ms
117 ms
119 ms
123 ms
122 ms
115 ms
119 ms
124 ms
118 ms
123 ms
119 ms
124 ms
124 ms
120 ms
115 ms
117 ms
118 ms
128 ms
121 ms
123 ms
120 ms
119 ms
120 ms
120 ms
128 ms
118 ms
114 ms
116 ms
117 ms
118 ms
120 ms
121 ms
120 ms
Nice. Thanks for the great write-up.