Add index for documents scraped with keyset pagination in tender repository
- Introduced a new index `documents_scraped_created_at_id_idx` in the tender repository to support keyset pagination for documents marked as scraped, sorted by creation date. - This enhancement improves the efficiency of querying scraped documents, facilitating better performance in data retrieval and management. This update significantly optimizes the tender repository's handling of scraped documents, enhancing overall data access capabilities.
This commit is contained in:
@@ -172,6 +172,13 @@ func NewRepository(mongoManager *orm.ConnectionManager, logger logger.Logger) Te
|
||||
{Key: "processing_metadata.documents_scraped", Value: 1},
|
||||
{Key: "processing_metadata.documents_scraped_at", Value: -1},
|
||||
}),
|
||||
// Panel list: documents_scraped=true sorted by created_at desc (keyset pagination).
|
||||
*orm.NewIndex("documents_scraped_created_at_id_idx", bson.D{
|
||||
{Key: "created_at", Value: -1},
|
||||
{Key: "_id", Value: -1},
|
||||
}).WithPartialFilterExpression(bson.M{
|
||||
"processing_metadata.documents_scraped": true,
|
||||
}),
|
||||
*orm.NewIndex("scraped_documents_scraped_at_idx", bson.D{
|
||||
{Key: "scraped_documents.scraped_at", Value: 1},
|
||||
}),
|
||||
|
||||
Reference in New Issue
Block a user