The sorting exceeds the memory limit during MongoDB query

org.springframework.data.mongodb.UncategorizedMongoDbException: Query failed with error code 17144 and error message 'Executor error: Overflow sort stage buffered data usage of 33554898 bytes exceeds internal limit of 33554432 bytes' on server 127.0.0.1:27017; nested exception is com.mongodb.MongoQueryException: Query failed with error code 17144 and error message 'Executor error: Overflow sort stage buffered data usage of 33554898 bytes exceeds internal limit of 33554432 bytes' on server 127.0.0.1:27017
    at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:107)
    at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2114)
    at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1957)
    at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1763)
    at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1746)
    at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:624)
    at com.controller.TestController.getList(TestController.java:1216)
    at com..controller.TestController$$FastClassBySpringCGLIB$$d9a15731.invoke(<generated>)

Error reason: when the sorting field does not use the index, if it exceeds 32M memory, it will be Abort, and the statement will directly return an error

Sort operation used more than the maximum 33554432 bytes of RAM., 33554432 bytes is exactly 32Mb, while mongodb's sort operation takes the data to the memory for sorting. In order to save memory, the default limit for sort operation is 32Mb. When the amount of data is getting larger and larger, it throws an exception when it exceeds 32Mb!

Solution: the specific field to be referenced can be defined according to the requirements, where 1 indicates ascending order and - 1 indicates descending order.

MongoDB sorting method: MongoDB can use index scanning to sort, so the result will not include SORT stage. Otherwise, if MongoDB cannot use the index to sort, the query plan will include SORT stage.

The efficiency of using index scanning is much higher than that of directly sorting the result set in memory. Therefore, MongoDB limits the use of sorting memory in order to make the execution of query statements more efficient, so it stipulates that only 32M can be used.

Pay attention to keep the ascending and descending order of the combined sort in the query and the direction in the combined index the same or opposite

Index operation statement:

db.getCollection('document').getIndexes() view the current index

db.getCollection('document').createIndex({"age": 1}) creates a single index

db.getCollection('document').createIndex({"age": 1,"name":1},{"name":"user"}) contains multiple key s under an index name

db.getCollection('document').dropIndex("age") deletes the index

Tags: mongo

Posted by lucidpc on Mon, 23 May 2022 00:13:13 +0300