Skip to content

Instantly share code, notes, and snippets.

@courville
Last active December 29, 2021 18:22
Show Gist options
  • Save courville/1ff449bbc6b9afc42a9d43eba63cae2b to your computer and use it in GitHub Desktop.
Save courville/1ff449bbc6b9afc42a9d43eba63cae2b to your computer and use it in GitHub Desktop.
avoid SQLiteBlobTooBigException: Row too big to fit into CursorWindow
// run small queries
static final int WINDOW_SIZE = 1000;
Cursor cursor = getFileListCursor(PARAM_ALL, null);
final int numberOfRows = cursor.getCount();
cursor.close();
int index = 0;
int window = WINDOW_SIZE;
int count = 0;
do {
if (index + window > numberOfRows)
window = numberOfRows - index;
cursor = getFileListCursor(PARAM_ALL, BaseColumns._ID + " ASC LIMIT " + index + "," + window);
if (DBG) Log.d(TAG, "startScraping: new batch fetching cursor from index=" + index + ", window=" + window + " -> index+window=" + (index + window) + "<=" + numberOfRows);
if (DBG) Log.d(TAG, "startScraping: new batch cursor has size " + cursor.getCount());
while (cursor.moveToNext()) {
count++;
if (DBG) Log.d(TAG, "startScraping: processing cursor number=" + count + "/" + numberOfRows);
}
index += window;
cursor.close();
} while (index < numberOfRows);
@courville
Copy link
Author

Another way avoiding number of queries if in processing db is altered:

                        Cursor cursor = getFileListCursor(PARAM_ALL, null);
                        numberOfRows = cursor.getCount();
                        int numberOfRowsRemaining = numberOfRows;
                        cursor.close();
                        do {
                            if (window > numberOfRowsRemaining)
                                window = numberOfRowsRemaining;
                            cursor = getFileListCursor(PARAM_ALL, BaseColumns._ID + " ASC LIMIT " + window);
                            if (DBG) Log.d(TAG, "startScraping: new batch fetching window=" + window + " entries <=" + numberOfRowsRemaining);
                            if (DBG) Log.d(TAG, "startScraping: new batch cursor has size " + cursor.getCount());
                            while (cursor.moveToNext()) {
                                count++;
                                if (DBG) Log.d(TAG, "startScraping: processing cursor number=" + count + "/" + numberOfRows);
                            }
                            cursor.close();
                            numberOfRowsRemaining -= window;
                        } while (numberOfRowsRemaining > 0);

@courville
Copy link
Author

Note that using API30+, LIMIT is not supported anymore, it has to be done through a bundle like:

if (Build.VERSION.SDK_INT > Build.VERSION_CODES.Q) { // API>30 requires bundle to LIMIT
                final Bundle bundle = new Bundle();
                bundle.putString(ContentResolver.QUERY_ARG_SQL_SELECTION, WHERE_UNSCANNED);
                bundle.putStringArray(ContentResolver.QUERY_ARG_SQL_SELECTION_ARGS, null);
                bundle.putStringArray(ContentResolver.QUERY_ARG_SORT_COLUMNS, new String[]{BaseColumns._ID});
                bundle.putInt(ContentResolver.QUERY_ARG_SORT_DIRECTION, ContentResolver.QUERY_SORT_DIRECTION_ASCENDING);
                bundle.putInt(ContentResolver.QUERY_ARG_LIMIT, window);
                bundle.putInt(ContentResolver.QUERY_ARG_OFFSET, 0);
                c = cr.query(VideoStoreInternal.FILES, ID_DATA_PROJ, bundle, null);
            } else {
                c = cr.query(VideoStoreInternal.FILES, ID_DATA_PROJ,
                        WHERE_UNSCANNED, null, BaseColumns._ID + " ASC LIMIT " + window);
            }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment