Exporting every image from a Drupal site as zip files with the Batch API

To let editors download every image on a Drupal 10 or 11 site, run the export as a Batch API operation that adds files to a ZipArchive in chunks bounded by bytes, split the result into archives of a couple of gigabytes, and serve them from the private file system. The part that catches people out is that ZipArchive::addFile() does almost nothing: the source files are only read when you call close(), so that is where your request time and memory go.

Why a plain "zip everything" request fails

A site with years of uploads can hold tens of thousands of images and many gigabytes. Building one zip inside a single request hits PHP's execution time limit, the web server's timeout, or the disk. A Drupal batch is the obvious tool: it splits work across many short requests and shows a progress bar. But a batch only helps if each request really is short, and "100 files per request" is a poor proxy for that when one file may be 40 KB and the next 40 MB.

Why ZipArchive::close() is where the work happens

With ZipArchive, addFile() only records that a path should be included. Nothing is read until close(), which rewrites the whole archive. Two consequences follow:

• Timing addFile() calls tells you nothing about how long the request will take.
• If a source file disappears between addFile() and close(), close() fails and you lose the whole request's work.

So I would bound each request by the total bytes queued, and keep the file count only as a secondary cap. I would also check that each queued source still exists just before closing.

How to choose what goes into each request and each archive

There are two limits, and they do different jobs. The per-request limit keeps one HTTP request short. The per-archive limit keeps each download practical for a person to fetch and unpack. A limit of around 200 MB per request and 2 GB per archive worked as a starting point, and both are easy to tune.

The batch sandbox holds the cursor. Paging by the last processed file ID is safer than paging by offset, because files can be added or removed while the batch runs.

<?php

use Drupal\Core\Database\Database;

const REQUEST_BYTES = 200 * 1024 * 1024;
const ARCHIVE_BYTES = 2 * 1024 * 1024 * 1024;
const REQUEST_FILES = 100;

function export_images_step(string $run_id, array &$context): void {
  $sandbox = &$context['sandbox'];
  if (!isset($sandbox['last_fid'])) {
    $sandbox += ['last_fid' => 0, 'part' => 1, 'part_bytes' => 0, 'done' => 0];
    $sandbox['total'] = (int) public_images()->countQuery()->execute()->fetchField();
  }

  $rows = public_images()
    ->fields('f', ['fid', 'uri', 'filesize'])
    ->condition('f.fid', $sandbox['last_fid'], '>')
    ->orderBy('f.fid')
    ->range(0, REQUEST_FILES)
    ->execute()
    ->fetchAll();

  $zip = NULL;
  $request_bytes = 0;
  foreach ($rows as $row) {
    $size = (int) $row->filesize;
    if ($zip && $request_bytes + $size > REQUEST_BYTES) {
      break;
    }
    if ($sandbox['part_bytes'] && $sandbox['part_bytes'] + $size > ARCHIVE_BYTES) {
      if ($zip) {
        break;
      }
      $sandbox['part']++;
      $sandbox['part_bytes'] = 0;
    }
    $zip ??= open_archive($run_id, $sandbox['part']);

    $sandbox['last_fid'] = $row->fid;
    $sandbox['done']++;
    $entry = substr($row->uri, strlen('public://'));
    if ($zip->locateName($entry) !== FALSE) {
      continue;
    }
    $zip->addFile(\Drupal::service('file_system')->realpath($row->uri), $entry);
    $zip->setCompressionName($entry, \ZipArchive::CM_STORE);
    $request_bytes += $size;
    $sandbox['part_bytes'] += $size;
  }

  $zip?->close();
  $context['finished'] = $rows ? min(1, $sandbox['done'] / max(1, $sandbox['total'])) : 1;
}

Two details in that loop are worth explaining. The check that breaks out of the loop when the archive is full only does so once something is already queued; otherwise the loop rolls over to a new part and carries on. And a single file larger than the request limit still gets processed on its own, because the first file in a request is never refused.

Should the images be compressed?

No. JPEG, PNG and WebP are already compressed, so deflating them spends CPU for a saving of a percent or two. setCompressionName($entry, \ZipArchive::CM_STORE) stores them as they are and shortens every request. If your media includes SVG or other text-based files, you may want to deflate only those.

What breaks on real sites

• Duplicate rows. Several file_managed rows can point at the same URI. Without the locateName() check you get duplicate entries, or an error, in the archive. The same applies to the size estimate shown to the user: group by URI.
• Missing files. Database rows outlive files on disk, especially on sites copied between environments. Record the ones you skipped, tell the user how many, and log the list rather than failing the export.
• Half-built archives. Write to a temporary name and rename to the final name only in the batch's finished callback. Otherwise someone can download a partial zip while the batch is still running.
• Disk space. Check disk_free_space() against the total image size before starting, and show a form error if there is not enough room.

How to serve and clean up the archives

Keep the archives in the private file system, not the public one. Serve them through a controller that returns a BinaryFileResponse with a content disposition of attachment, behind a restricted permission. Constrain the route's filename parameter with a regular expression so a request can never name anything other than a generated archive.

requirements:
  _permission: 'administer media'
  archive: '^[a-z0-9-]+\.zip$'

Archives this size should not live forever, so implement hook_cron() to delete anything older than a day, including abandoned temporary files from batches that died halfway. If the site uses Ultimate Cron, give the job its own schedule entry so the cleanup is visible and can be run on demand.

Key takeaways

• Bound each Batch API request by the bytes queued, because ZipArchive reads the source files on close() and not on addFile().
• Page by the last processed file ID rather than by offset, so files uploaded or deleted mid-run do not cause skips or repeats.
• Store already-compressed images instead of deflating them, and split the output into archives of a size a person can realistically download.
• Build under a temporary name, rename on completion, and treat duplicate or missing files as expected data problems rather than fatal errors.
• Serve archives from the private file system behind a permission and a strict filename pattern, and delete them with cron.