Package org.apache.lucene.search.grouping

package org.apache.lucene.search.grouping
Grouping.

This module enables search result grouping with Lucene, where hits with the same value in the specified single-valued group field are grouped together. For example, if you group by the author field, then all documents with the same value in the author field fall into a single group.

Grouping requires a number of inputs:

  • groupSelector: this defines how groups are created from values per-document. The grouping module ships with selectors for grouping by term, and by long and double ranges.
  • groupSort: how the groups are sorted. For sorting purposes, each group is "represented" by the highest-sorted document according to the groupSort within it. For example, if you specify "price" (ascending) then the first group is the one with the lowest price book within it. Or if you specify relevance group sort, then the first group is the one containing the highest scoring book.
  • topNGroups: how many top groups to keep. For example, 10 means the top 10 groups are computed.
  • groupOffset: which "slice" of top groups you want to retrieve. For example, 3 means you'll get 7 groups back (assuming topNGroups is 10). This is useful for paging, where you might show 5 groups per page.
  • withinGroupSort: how the documents within each group are sorted. This can be different from the group sort.
  • maxDocsPerGroup: how many top documents within each group to keep.
  • withinGroupOffset: which "slice" of top documents you want to retrieve from each group.

The implementation is two-pass: the first pass (FirstPassGroupingCollectorManager) gathers the top groups, and the second pass (TopGroupsCollectorManager) gathers documents within those groups. Both passes use CollectorManager to support concurrent (parallel) collection across segments. If the search is costly to run you may want to use the CachingCollectorManager class, which wraps the first-pass manager and caches all collected documents (and optionally scores) per segment slice so they can be replayed into the second-pass manager without re-running the query. This way you only run the query once, but you pay a RAM cost to (briefly) hold all hits. Results are returned as a TopGroups instance.

Groups are defined by GroupSelector implementations:

Known limitations:

  • Sharding is not directly supported, though is not too difficult, if you can merge the top groups and top documents per group yourself.

Typical usage for the generic two-pass grouping search looks like this using the grouping convenience utility (optionally using caching for the second pass search): 

   GroupingSearch groupingSearch = new GroupingSearch("author");
   groupingSearch.setGroupSort(groupSort);

   if (useCache) {
     // Sets cache in MB
     groupingSearch.setCachingInMB(4.0, true);
   }

   if (requiredTotalGroupCount) {
     groupingSearch.setAllGroups(true);
   }

   TermQuery query = new TermQuery(new Term("content", searchTerm));
   TopGroups<BytesRef> result = groupingSearch.search(indexSearcher, query, groupOffset, groupLimit);

   // Render groupsResult...
   if (requiredTotalGroupCount) {
     int totalGroupCount = result.totalGroupCount;
   }

To use the single-pass BlockGroupingCollectorManager, first, at indexing time, you must ensure all docs in each group are added as a block, and you have some way to find the last document of each group. One simple way to do this is to add a marker binary field: 

   // Create Documents from your source:
   List<Document> oneGroup = ...;

   Field groupEndField = new StringField("groupEnd", "x", Field.Store.NO);
   oneGroup.get(oneGroup.size()-1).add(groupEndField);

   // You can also use writer.updateDocuments(); just be sure you
   // replace an entire previous doc block with this new one.  For
   // example, each group could have a "groupID" field, with the same
   // value for all docs in this group:
   writer.addDocuments(oneGroup);
Then, at search time: 
   Query groupEndDocs = new TermQuery(new Term("groupEnd", "x"));
   Weight lastDocPerGroup = indexSearcher.createWeight(
       indexSearcher.rewrite(groupEndDocs), ScoreMode.COMPLETE_NO_SCORES, 1);
   BlockGroupingCollectorManager<BytesRef> manager = new BlockGroupingCollectorManager<>(
       groupSort,
       groupOffset,
       topNGroups,
       needsScores,
       lastDocPerGroup,
       withinGroupSort,
       docOffset,
       docsPerGroup);
   TopGroups groupsResult = indexSearcher.search(new TermQuery(new Term("content", searchTerm)), manager);

   // Render groupsResult...
Or alternatively use the GroupingSearch convenience utility: 
   // Per search:
   GroupingSearch groupingSearch = new GroupingSearch(groupEndDocs);
   groupingSearch.setGroupSort(groupSort);
   groupingSearch.setIncludeMaxScore(needsScores);
   TermQuery query = new TermQuery(new Term("content", searchTerm));
   TopGroups groupsResult = groupingSearch.search(indexSearcher, query, groupOffset, groupLimit);

   // Render groupsResult...
Note that the groupValue of each GroupDocs will be null, so if you need to present this value you'll have to separately retrieve it (for example using stored fields, FieldCache, etc.).

Another collector manager is the AllGroupHeadsCollectorManager that can be used to retrieve all most relevant documents per group. Also known as group heads. This can be useful in situations when one wants to compute group based facets / statistics on the complete query result. The manager supports concurrent collection and can also be used with the GroupingSearch convenience utility, but when one only wants to compute the most relevant documents per group it is better to just use the manager directly as done here below. 


   AllGroupHeadsCollectorManager<BytesRef> manager =
       new AllGroupHeadsCollectorManager<>(
           () -> new TermGroupSelector(groupField), sortWithinGroup);
   AllGroupHeadsCollectorManager.GroupHeadsResult result =
       indexSearcher.search(new TermQuery(new Term("content", searchTerm)), manager);
   // Return all group heads as int array
   int[] groupHeadsArray = result.retrieveGroupHeads();
   // Return all group heads as FixedBitSet.
   int maxDoc = indexSearcher.getIndexReader().maxDoc();
   Bits groupHeadsBitSet = result.retrieveGroupHeads(maxDoc);

Grouped faceting is handled by TermGroupFacetCollectorManager. For each facet value it counts how many distinct groups have at least one matching document with that value. The manager supports concurrent collection; groups that span multiple search slices are counted once. 


   TermGroupFacetCollectorManager manager =
       new TermGroupFacetCollectorManager(
           "author", "category", false, null, offset + limit, minCount, orderByCount);
   GroupedFacetResult facetResult =
       indexSearcher.search(new TermQuery(new Term("content", searchTerm)), manager);
   int totalCount = facetResult.getTotalCount();
   int missingCount = facetResult.getTotalMissingCount();
   List<GroupedFacetResult.FacetEntry> entries = facetResult.getFacetEntries(offset, limit);