Increase screenshot upload timeout

deliver_screenshot_ingestion.rb

require "deliver"

# Stops deliver from duplicating screenshots on App Store Connect.
#
# deliver uploads every screenshot with `wait_for_processing: false`, then
# UploadScreenshots#wait_for_complete polls until no screenshot is in the
# `UPLOAD_COMPLETE` state. The problem: a screenshot App Store Connect is still
# ingesting is briefly absent from the listing entirely — it is neither
# `UPLOAD_COMPLETE` nor `COMPLETE`, it simply isn't returned yet. So the poll
# sees zero `UPLOAD_COMPLETE` and returns, App Store Connect having had only a
# few seconds to ingest. The verifier (verify_local_screenshots_are_uploaded)
# then can't find those screenshots' checksums, declares them missing, and
# re-uploads them — but the pre-retry `delete! unless complete?` can't remove an
# asset that's mid-ingestion, so the original finishes alongside the re-upload
# and the screenshot ends up duplicated.
#
# We hit this on every App Store submission (26.25.1, 26.25.2), always the first
# one or two screenshots — the ones App Store Connect happened to still be
# ingesting when the ~6-12s poll returned. Upstream: fastlane#30094.
#
# The fix holds wait_for_complete open until App Store Connect actually reports
# every local screenshot present, closing the race that let the verifier declare
# mid-ingestion screenshots missing. The extra wait is bounded by a short, fixed
# grace window (NOT screenshot_processing_timeout, which defaults to 3600s): a
# genuinely failed upload — one that never produces an `UPLOAD_COMPLETE` asset —
# must fall through to deliver's normal retry promptly, not stall the publish
# for an hour and get the CI step (capped well under that) killed before the
# retry can run.
module DeliverScreenshotIngestionStabilizer
  # Upper bound on the extra time we'll wait for App Store Connect to surface a
  # just-uploaded screenshot before handing back to deliver's verifier. Sized
  # for ingestion lag (seconds to low minutes — the race showed at 6-12s), and
  # applied per upload attempt, so deliver's recursive retries stay well inside
  # the publish step's CI budget.
  INGESTION_GRACE_SECONDS = 120

  def upload_screenshots(localizations, screenshots_per_language, timeout_seconds, tries: 5)
    # wait_for_complete's signature can't carry the local screenshot list, so
    # hand it over through an ivar for the ingestion cross-check there.
    @screenshots_pending_ingestion = screenshots_per_language
    super
  end

  def wait_for_complete(iterator, timeout_seconds)
    # Consume the stashed list, clearing it so a future call that reaches here
    # without a preceding upload_screenshots can't read a stale set.
    pending = @screenshots_pending_ingestion
    @screenshots_pending_ingestion = nil

    states = nil
    last_error = nil
    waited = 0

    loop do
      begin
        # Call super every iteration, not once: it runs deliver's own
        # UPLOAD_COMPLETE processing wait (bounded by its timeout) and returns
        # fresh states. So a screenshot that only surfaces mid-wait — as
        # UPLOAD_COMPLETE or FAILED — is waited on and reflected in the states we
        # ultimately hand back to deliver's retry, never a stale snapshot from
        # before the asset existed.
        states = super(iterator, timeout_seconds)

        return states if pending.nil? || pending.empty?
        # super returned with something still UPLOAD_COMPLETE — it can only do
        # that by hitting its own processing timeout, so the asset has already had
        # the full wait. Hand back to deliver's retry now (as stock would) rather
        # than re-entering super and stacking another timeout per loop.
        return states if states.fetch("UPLOAD_COMPLETE", 0) > 0
        # Every local screenshot is visible on App Store Connect — ingestion has
        # settled, so hand back. We wait for ALL locals even when one is already
        # FAILED: a still-ingesting sibling must reach a terminal state before the
        # retry runs, or the retry's verifier counts it missing and re-uploads it
        # (duplicating it) mid-ingestion. The returned states still carry any
        # FAILED count, so the retry re-uploads only the genuinely failed one.
        #
        # verify reads its own App Store Connect listing, distinct from the one
        # `states` came from. Re-read state once it passes so the states we return
        # reflect any asset that surfaced between the two reads (its
        # UPLOAD_COMPLETE/FAILED) rather than the snapshot from before it existed —
        # and super waits out a just-surfaced UPLOAD_COMPLETE in the process.
        return super(iterator, timeout_seconds) if verify_local_screenshots_are_uploaded(iterator, pending)
      rescue => error
        # A transient App Store Connect hiccup mid-wait must not abort the
        # publish: the lane's outer rescue would re-run the whole upload and
        # re-introduce the duplicates this patch exists to prevent. We poll App
        # Store Connect far more than stock deliver's single verify, so swallow
        # the blip and keep polling within the window.
        last_error = error
        # Fully qualified: this module is prepended at top-level scope, where the
        # bare `UI` alias isn't resolvable and raised "uninitialized constant
        # DeliverScreenshotIngestionStabilizer::UI", failing every App Store publish
        # that reached screenshot processing (run 28074342909).
        FastlaneCore::UI.error("Ignoring transient error while confirming screenshot ingestion: #{error.message}")
      end

      # Bound only the ingestion-surfacing wait — count the sleeps below, NOT
      # super's processing wait (deliver's own, already capped by its timeout). A
      # truly failed or never-uploaded screenshot never surfaces, so we give up
      # here after a bounded wait rather than the multi-hour stall that reusing
      # screenshot_processing_timeout (3600s) as the deadline would cause.
      if waited >= INGESTION_GRACE_SECONDS
        # Never got a clean states read (super kept erroring) — surface the error
        # as stock fastlane would rather than returning nil.
        raise last_error if states.nil? && last_error

        return states
      end

      FastlaneCore::UI.message("App Store Connect is still ingesting screenshots — waiting before verifying to avoid duplicate uploads (fastlane#30094)…")
      sleep(5)
      waited += 5
    end
  end
end

Deliver::UploadScreenshots.prepend(DeliverScreenshotIngestionStabilizer)