Package · sdk-s5w

Upload pipeline

What happens between calling upload() and getting a File: which path is taken, when the hash is computed, how a break in the middle of a chunk is survived, and why one situation ends in an honest “don’t know”. Everything below follows the code of Api\Files and Api\Uploads.

Choosing the path

text

files()->upload($path) / uploadContent($bytes)
      │
      ├─ size ≤ chunkThreshold (16 MiB) ──► POST /v1/files, multipart, one request
      │                                      transferTimeout, no retries
      │
      └─ size > chunkThreshold ───────────► uploads()->sendSource()  (see below)

Why the threshold is 16 MiB, not 32. The server accepts request bodies up to 32 MiB, and files almost that large could go in one request (Files::SINGLE_REQUEST_LIMIT — 32 MiB minus 64 KiB for the multipart framing). But a multipart request that breaks at megabyte 29 loses everything, while chunks continue from the break. On files over 16 MiB three requests instead of one go unnoticed, the resilience does not. The S5W panel does the same. The threshold is set in S5wConfig::$chunkThreshold, but no higher than SINGLE_REQUEST_LIMIT.

Why a single request is not retried. If the request reached the server and the file was stored, and only the answer broke, a retry creates a second file — name (1).ext. A broken connection cannot tell “didn’t arrive” from “arrived, answer lost”, so the SDK does not take the risk: retries from the configuration does not apply to this request.

The byte source

Inside, a file on disk and a string in memory are one object, Support\Source. A file is read one chunk at a time: fseek() to the offset and fread() of the needed length. No more than one chunk (8 MiB) is held in memory at once, however large the file.

The file’s size is fixed before the start (filesize() after clearstatcache()). If the file shrinks during the upload and a read returns an empty string too early, the SDK cancels the session and throws S5wException with the file ended at … — it changed during the upload.

Hash and deduplication

text

deduplicate && no imageOptions ?
  yes → sha256(file) ── passed to start()
  no  → no hash

POST /v1/uploads {name, size, folder, sha256?, imageOptions?}
  │
  ├─ these bytes are already in the bucket ──► the server creates the file at once
  │                                             answer: file != null, id = "", chunkSize = 0
  │                                             SDK: onProgress(size, size) → return file
  │
  └─ no ──► session {id, offset: 0, chunkSize: 8 MiB}
            SDK: onStart(session) → chunk loop

The hash is computed by hash_file() as a stream, without loading the file into memory. It is needed twice: in start() — so the server can find existing content, and on complete() — the server compares it with what it received and refuses the file on a mismatch (ValidationException).

start() runs without retries, even with retries > 1 in the configuration: with a hash this request can create the file itself, and a retry after a lost answer would create a second one.

Deduplication and image options

With imageOptions the hash is neither computed nor sent. Two reasons.

The hash says nothing. With processing, the bucket gets the result, not the original. The original’s hash never matches it, so deduplication cannot work and the time spent hashing is wasted.

Older servers got it wrong. Before S5W 1.1.1, when the original’s hash matched an unprocessed file already in the bucket, the server created the new file on that original and silently skipped the options: instead of photo.webp at 300×200 you got photo (1).png at its original size, with processed: null. 1.1.1 fixed this — with processing options the server takes the ordinary path. The SDK does not send a hash with options in any case, so it works correctly with older servers too.

The chunk loop

text

reported = offset; onProgress(offset, size); failures = 0

while offset < size:
  chunk = source.read(offset, min(chunkSize, size - offset))
  chunk empty?             → abort, S5wException (the file changed)

  PATCH /v1/uploads/{id}   Upload-Offset: offset, body — the chunk's bytes
      200                  → a new session
      network / 5xx / 409  → recover()            (see below)
      any other refusal    → abort, throw          (the data is wrong, a retry won't help)

  offset grew?             → reported = offset; failures = 0; onProgress(offset, size)

finish()

Chunks are read from the server’s offset, not from a counter in the SDK. So any answer — an ordinary one, one after recovery, one after resume() — automatically puts the read where the server expects the next byte.

The Upload-Offset header turns possible file corruption into an ordinary error: if the server already has this chunk (its answer was lost), it answers 409 instead of gluing it in a second time.

What counts as progress. onProgress is called when the offset grew — whether we learned it from the chunk’s answer or from status() after recovery. Only then is the failure counter reset: three failures in a row stop the upload, failures with progress in between do not.

Recovery

When sending a chunk broke (TransportException), or the server answered 5xx or 409, recover() runs:

text

failures += 1
failures ≥ ATTEMPTS (3)?  → error to the log, UploadInterruptedException(id, offset, size, cause)

pause 1000 × 2^(failures-1) ms        1 s, then 2 s
warning to the log

GET /v1/uploads/{id}
  answered       → debug to the log "server holds X", return the server's session
  did not answer → return the previous session — the next attempt will tell

The key is to ask the server, not to just repeat the chunk. A broken connection does not say whether the chunk arrived: it may have landed whole while only the answer got lost. Resending the same chunk would get 409; sending the next one — 409 too. status() gives the exact answer, and the loop continues from the right place.

Verified by simulation: with a “lost answer” on the first chunk, status() returned offset 8 388 608 — the chunk was on the server, and the SDK went straight on to the second one without resending anything.

The pauses are short and growing: a short glitch passes within a second, and with an unreachable server the upload stops after about three seconds instead of hanging. The session is not cancelled — it can be continued.

Why cancel on other refusals: 400, 413, 404, 422 are about the data itself — a chunk of the wrong size, past the declared size, a missing session. Resending the same bytes would get the same answer, so the SDK releases the session (abort(), errors of the cancel ignored) and throws the original exception.

Assembling the file

text

for attempt 1..ATTEMPTS:
  POST /v1/uploads/{id}/complete      transferTimeout
      201                  → debug "completed", return File
      4xx refusal          → abort, throw          (hash mismatch etc.)
      network / 5xx        → GET /v1/uploads/{id}
                                404           → UploadUnconfirmedException
                                session alive → pause, another attempt
                                no answer     → pause, another attempt
  attempts ran out        → UploadInterruptedException

Lost answer to `complete`

The only place where the SDK cannot learn the truth.

Assembly is the last request. If its answer is lost, the SDK asks the server about the session:

the session is alive — assembly did not happen and can be repeated. The common case: the request never arrived;
the session is gone (404) — the server closed it. Usually that means the file was assembled, but on an assembly error the server removes the session too. Which one it was can no longer be told from the session, and the created file’s id was only in the lost answer.

In the second case the SDK throws UploadUnconfirmedException with the session id and the name. Looking the file up itself — by name in the folder — the SDK would not: the folder may hold another file with the same name uploaded by someone else, and returning it would be worse than saying “don’t know” honestly. Verified by simulation: in this scenario the file did turn out to be in the folder.

Resuming: `resume()`

text

resume(id, file)
  GET /v1/uploads/{id}              404 → NotFoundException (no session — start over)
  file size ≠ session size          → S5wException (not the same file)
  debug "resumed at offset"
  chunk loop from the server's offset   (the same retries and recovery)
  finish()

If the server already has every byte (if it was assembly that broke, say), the loop sends no chunk and goes straight to complete().

resume() checks only the size, not the content: a substituted file of the same size would be appended. The hash guards against that: if it was sent at the start (the default, except for uploads with imageOptions), the server refuses the file at assembly.

Events in the log

Moment	Level	Message
deduplication	debug	`upload deduplicated: {name} ({size} bytes) → file {file}, nothing sent`
session opened	debug	`upload {id} started: {name}, {size} bytes in chunks of {chunk}`
chunk retry	warning	`upload {id}: chunk at {offset} failed (attempt {n}/3), retry in {ms} ms: {error}`
recovered	debug	`upload {id}: the server holds {offset} of {size} bytes, continuing`
attempts ran out	error	`upload {id} interrupted at {offset} of {size} after 3 attempts: {error}`
data refused	error	`upload {id} refused at {offset}: {error}`
resumed	debug	`upload {id} resumed at {offset} of {size} bytes`
assembly retry	warning	`upload {id}: completing failed (attempt {n}/3), retry in {ms} ms: {error}`
assembly unconfirmed	error	`upload {id} completed but not confirmed: {error}`
done	debug	`upload {id} completed: {name} → file {file}`

Plus a line for every HTTP request — see HTTP and logging.

What the pipeline does not do

Parallel chunks. The server accepts chunks strictly in order, so parallelism would gain nothing.
A configurable number of attempts. Uploads::ATTEMPTS = 3 is a constant.
Resuming data from memory. resume() works only with a file: after a process restart the bytes in memory are gone.

Upload pipeline

Choosing the path

The byte source

Hash and deduplication

Deduplication and image options

The chunk loop

Recovery

Assembling the file

Lost answer to complete

Resuming: resume()

Events in the log

What the pipeline does not do

Lost answer to `complete`

Resuming: `resume()`