Generate an accurate S3 ETAG in Node.js for any file (including multipart)
MIT License
Generate an accurate S3 ETAG in Node.js for any file (including multipart).
Please Note, this only works for unencrypted buckets.
npm install s3-etag
import { generateETag } from 's3-etag';
// Simple MD5 hash of contents for non-multipart files
const etag = generateETag(absoluteFilePath);
// MD5 hash of combined contents & part number (see below) for multipart files
const partSizeInBytes = 10 * 1024 * 1024; // 10mb
const etag = generateETag(absoluteFilePath, partSizeInBytes);
This is a Node.js implementation of this algorithm.
At a high level:
partSizeInBytes
is specified, return MD5 hash of file contentspartSizeInBytes
is specified:
partSizeInBytes
to the file sizeIf the partSizeInBytes is unknown, you can find it by using AWS CLI:
head-object
to retrieve the object's metadata using the following command, where raw-files
is the bucket name, and IMG2345.CR2
is the keyaws s3api head-object --bucket raw-files --key IMG2345.CR2
{
"AcceptRanges": "bytes",
"ContentType": "text/html",
"LastModified": "2024-05-22T08:45:10+00:00"
"ContentLength": 28333464,
"ETag": "\"85ae33db28930d3afe594da14cd190bb-2\"",
"VersionId": "28P4UkX5sCO.8vbyMojvecHndkHDwf",
"ContentType": "binary/octet-stream",
"ServerSideEncryption": "AES256",
"Metadata": {}
}
-n
at the end of the eTag, where n
is a number >= 2, representing the number of parts/chunks. For single part objects, the eTag will simply be the MD5 of the object.aws s3api head-object --bucket raw-files --key IMG2345.CR2 --part-number 1
to get the metadata of the first part/chunk. This will return
{
"AcceptRanges": "bytes",
"ContentType": "text/html",
"LastModified": "2024-05-22T08:45:10+00:00"
"ContentLength": 17179870,
"ETag": "\"85ae33db28930d3afe594da14cd190bb-2\"",
"VersionId": "28P4UkX5sCO.8vbyMojvecHndkHDwf",
"ContentType": "binary/octet-stream",
"ServerSideEncryption": "AES256",
"Metadata": {},
"PartsCount": 2
}
ContentLength
value as the partSizeInBytes
See LICENSE.md.