Fetch WP "posts" in chunks.

functions.php

<?php
/**
 * Fetch posts in chunks.
 *
 * This function fetches posts in chunks of a specified size, which can be useful for large datasets.
 *
 * @param string $post_type      The type of post to fetch. Default is 'post'.
 * @param int    $posts_per_page The number of posts to fetch per page. Default is 100.
 * @param array  $custom_args    Custom arguments to pass to WP_Query. Default is an empty array.
 *                                Note: 'post_type', 'posts_per_page', 'paged', and 'no_found_rows'
 *                                are reserved and will be overridden.
 * @param int    $max_pages      Safety ceiling for maximum number of pages to fetch. Default is 50.
 *
 * @return array An array of all fetched WP_Post objects.
 */
function tprt_fetch_posts_in_chunks( $post_type = 'post', $posts_per_page = 100, $custom_args = [], $max_pages = 50 ) {
  $paged     = 1;
  $all_posts = [];

  // Keep fetching posts until there are no more posts or the safety ceiling is reached.
  while ( $paged <= $max_pages ):
    // Merge custom arguments with required arguments.
    // Required arguments are merged last to prevent custom_args from overriding chunking logic.
    $args = array_merge( $custom_args, [
      'post_type'      => $post_type,
      'posts_per_page' => $posts_per_page,
      'paged'          => $paged,
      'no_found_rows'  => true,
    ] );

    $query = new WP_Query( $args );

    // If there are no posts, break the loop.
    if ( ! $query->have_posts() ) break;

    // Collect post objects directly without setting up global post state.
    $all_posts = array_merge( $all_posts, $query->posts );

    $paged++;
  endwhile;

  return $all_posts;
}

It is considered to be bad practice to use the following parameter:

'posts_per_page' => -1

This should more efficiently get all the "posts".

One should consider storing the results in a transient as well.

Adding an example for folks who would like to see how to use this function.

$custom_args = array(
    'orderby' => 'title',
    'order'   => 'ASC',
);
$all_posts = tprt_fetch_posts_in_chunks('post', 50, $custom_args);

$all_posts is an array of post objects