Node.js Worker Example: Object Storage Handoff and Signed Access

Short answer: have a Node.js worker build a large ZIP export outside the request path, hand its bytes to a bounded multipart upload, and issue a signed download link only after the object is verified and the export is marked ready. That rule is less about an archive library than about defining who owns each failure. The request handler owns authorization and job creation. The worker owns compression and an upload attempt. The control plane owns the durable export state. Object storage owns object durability and later lifecycle cleanup. If one process owns all four, the first retry turns a normal operational event into an ambiguous customer result. What did a stopped export teach the control plane? Consider a bounded recovery exercise: an archive worker is deliberately stopped after it has sent a multipart chunk but before it records the receipt. The next worker must not assume the previous process's memory still describes reality. It needs an export ID that is stable across attempts, an attempt ID for the current lease, and durable checkpoints that can be reconciled against the storage session before more bytes are sent. This is the invariant: an export is a business object; an upload is an attempt to materialize it. Those identifiers should never be interchangeable. Without that separation, a retry can make two publication rows for one request, complete an object whose parts the database cannot explain, or produce a signed link before the final object exists. The archive may even open correctly, which is why this class of failure is unpleasant: the evidence arrives in support tickets or retention cleanup rather than in the worker log. A 5-minute completion objective does not help if the result is the wrong export. The preventative path is a small state machine. A job moves from queued to building, uploading, verifying, and ready; a lease expiry returns work to reconciliation rather than blindly rebuilding it. ready is the only state allowed to request signed access. The exact labels are unimportant, but the transition ownership is not. Keep the object key deterministic from the export ID, make publication idempotent, and treat an existing ready record as the answer to duplicate delivery. I would also keep the URL out of logs. A signed download link is a bearer credential, not harmless diagnostic text. How should a Node.js worker multipart-upload a ZIP export and issue a signed download link? Node.js streams are a good fit for the producer side because backpressure can keep compression from racing ahead of the network sink. The worker should read from its source, write archive bytes, assemble one part at a time, and cap in-flight uploads. The important capacity-planning calculation is local: part buffer size multiplied by upload concurrency, plus compressor, runtime, and source-reader memory, must fit inside every worker's memory budget. Don't pick those values from a happy-path export. Exercise incompressible input and a slow sink before setting the concurrency cap. The following Go example expresses the control contract rather than a provider SDK. It is intentionally portable: a Node.js worker can implement the same interfaces with stream events and promises, while the storage adapter supplies its own multipart and signing details. Notice that Read determines the real final-part length; padding the last part to a configured size would change the archive bytes. package export import ( "bytes" "context" "fmt" "io" ) type PartReceipt struct { Number int Token string } type MultipartStore interface { Begin(context.Context, string) (string, error) UploadPart(context.Context, string, int, io.Reader, int64) (PartReceipt, error) Complete(context.Context, string, []PartReceipt) error Verify(context.Context, string) error SignDownload(context.Context, string) (string, error) } type Exports interface { SaveSession(context.Context, string, string) error SavePart(context.Context, string, PartReceipt) error MarkReady(context.Context, string, string) (bool, error) } func PublishArchive(ctx context.Context, exportID string, zipStream io.Reader, partSize int64, store MultipartStore, exports Exports) (string, error) { objectKey := fmt.Sprintf("exports/%s/archive.zip", exportID) sessionID, err := store.Begin(ctx, objectKey) if err != nil { return "", fmt.Errorf("begin upload: %w", err) } if err := exports.SaveSession(ctx, exportID, sessionID); err != nil { return "", fmt.Errorf("checkpoint session: %w", err) } parts := make([]PartReceipt, 0) buf := make([]byte, partSize) for number := 1; ; number++ { n, readErr := io.ReadFull(zipStream, buf) if readErr == io.EOF && n == 0 { break } if readErr != nil && readErr != io.ErrUnexpectedEOF { return "", fmt.Errorf("read archive: %w", readErr) } receipt, err := store.UploadPart(ctx, sessionID, number, bytes.NewReader(buf[:n]), int64(n)) if err != nil { return "", fmt.Errorf("upload part %d: %w", number, err) } if err := exports.SavePart(ctx, exportID, receipt); err != nil { return "", fmt.Errorf("checkpoint part %d: %w", number, err) } parts = append(parts, receipt) if readErr == io.ErrUnexpectedEOF { break } } if err := store.Complete(ctx, sessionID, parts); err != nil { return "", fmt.Errorf("complete upload: %w", err) } if err := store.Verify(ctx, objectKey); err != nil { return "", fmt.Errorf("verify object: %w", err) } if _, err := exports.MarkReady(ctx, exportID, objectKey); err != nil { return "", fmt.Errorf("mark ready: %w", err) } return store.SignDownload(ctx, objectKey) } The sample keeps the focus on ordering: checkpoint accepted receipts, complete once, verify, publish readiness, then sign. With parallel part uploads, retain an ordered receipt collector and a strict semaphore; otherwise the neat streaming story turns into an unbounded memory policy under network latency. The link has a different lifetime from the export. A customer may request a fresh signed URL several times while the immutable object remains inside its retention window. That separation makes revocation, authorization checks, and support behavior legible. It also means a link generator must never be the mechanism that decides an export is complete. Buy the storage substrate; keep the export control plane The storage substrate and the export workflow have different operational shapes. Durability, multipart semantics, object metadata, range reads, and lifecycle rules are commodity storage concerns. Authorization, deduplication, customer-visible status, worker leases, and SLO reporting describe the application. An adapter at that boundary gives a team one place to test its assumptions instead of spreading provider behavior through every job handler. Choice On-call load Lock-in Useful when The catch Managed object storage behind an internal adapter Lower for physical durability; moderate for export workflow Limited to the adapter if tested Most teams that need durable exports Signing, lifecycle behavior, and multipart recovery still need integration tests Direct storage calls throughout workers Low at first High in application code A small service with one stable backend A migration changes business logic, operational tooling, and tests Self-hosted object storage High Lower API dependence, higher operational commitment An isolated environment with storage operators Capacity, repair, upgrades, and availability remain the team's pager Local worker disk Low for tiny internal jobs High coupling to worker placement Short-lived, size-capped reports Downloads disappear when work is rescheduled The catch is that an adapter is not evidence of portability. Run the same conformance suite against another implementation before relying on it: interrupted parts, duplicate completion, stale leases, authorization before signing, range retrieval, expiry, and retention cleanup. If the second adapter cannot pass, the interface only hides lock-in. Self-hosting is not suitable when the export team cannot also carry storage capacity and repair responsibility. Conversely, multipart machinery is excessive for a small, short-lived internal report with a hard size cap and no durable download requirement. Your mileage may vary with data residency rules and existing operational staff; those constraints should drive the buy-versus-build decision more than a feature checklist. Operate the whole path, not just the upload The user-facing SLO should measure accepted exports reaching ready within a size-aware objective. Worker CPU is useful capacity telemetry, but it is not a success metric. Pair it with queue delay, active leases, upload duration, verification duration, bytes emitted, terminal state counts, and the age of the oldest nonterminal export. A low-CPU worker pool beside a growing queue points toward dispatch or lease handling; saturated workers with stable queue delay point toward planned capacity work. Test at state boundaries. Stop a worker before and after every durable checkpoint. Deliver the same job twice. Request a signed download link before readiness. Feed a poorly compressible source through a deliberately slow sink, download the finished archive, and verify its manifest and content digest. These tests are not glamorous, but they cover the places where a big ZIP export, a multipart upload, and a background worker disagree about time. Make the recovery test concrete enough to falsify the design. Start one export, record the generated export ID and object key, and halt the worker after a part acknowledgement while withholding the checkpoint write; then let a replacement worker acquire the lease and force it to list or otherwise reconcile the accepted parts before it resumes. Repeat the exercise after the final multipart completion but before the ready transition, because that is where an implementation often rebuilds a finished archive or makes a second publication record. The expected result is boring: one immutable object identity, one customer-visible export, a traceable attempt history, and no signed access until verification and publication have both happened. Capture those assertions in the integration suite rather than in a runbook alone. A runbook helps the person on call, but a repeatable test prevents a later optimization to worker concurrency, queue delivery, or database transactions from quietly changing what retry means. Boring is good. Lifecycle policy supplies the final cleanup clock. Completed objects should follow the product retention policy, while abandoned multipart sessions need a separate conservative cleanup rule so they do not accumulate indefinitely. Object lifecycle management can enforce retention cleanup, but it cannot explain export status to a user or repair a broken application transition. Keep both layers explicit. References https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html https://cloud.google.com/storage/docs