Data Integrity Checking

• Data integrity checking overview
• Data integrity checking for uploads
• Data integrity checking for downloads
• Checksum algorithms
• More details about data integrity checking
• How to avoid the 'missing hash' issue
• How to fix the hashes for already uploaded files

Data Integrity Checking Overview

With S3 Browser, you can reliably upload and download your files to and from Amazon S3. S3 Browser supports data integrity features to ensure that data is not corrupted while traversing the network.

Data Integrity Checking for Uploads

To enable data integrity checking for uploads:

1. Click Tools, then Options.

Click Tools, Options

2. In the Options dialog, switch to the Data Integrity tab:

Click Tools, Options, Data Integrity

3. Check the Check data integrity during file upload option.

4. Click Save Changes.

When this option is enabled, S3 Browser calculates a checksum for each file you upload using the selected algorithm and sends it to Amazon S3. Amazon S3 then calculates the checksum on the server side and compares it with the one provided by S3 Browser. If the checksums do not match, an error is returned and the file will not be stored.

S3 Browser also stores the calculated checksum as a custom metadata header (x-amz-meta-checksum-<algorithm>) for data integrity verification during download.

This option must also be enabled if you want to verify data integrity when downloading files (see below).

Data Integrity Checking for Downloads

To enable data integrity checking for downloads:

1. Click Tools, then Options.

Click Tools, Options

2. In the Options dialog, switch to the Data Integrity tab:

Click Tools, Options, Data Integrity

3. Check the Check data integrity during file download option.

4. Click Save Changes.

When this option is enabled, S3 Browser calculates a checksum for each downloaded file and compares it with the value returned by the server (Amazon S3). If the checksums do not match, an error is returned and the file will not be saved locally.

You can choose between two data integrity check modes:

• Flexible - Only files with a valid checksum are verified. Files with missing or invalid checksums are treated as valid.
• Strict - All files are checked. Files with missing or invalid checksums are treated as corrupted.

Checksum algorithms

S3 Browser supports the following checksum algorithms for data integrity checking:

• CRC-32
• CRC-32C
• CRC-64/NVME
• SHA-1
• SHA-256

You can select the desired algorithm in the Data Integrity tab under Tools, Options.

Note that many S3-compatible storage providers may only support SHA-256. Using other algorithms such as CRC-32, CRC-32C, or SHA-1 may not be supported or may cause upload/download failures if the storage does not recognize them.

More Details About Data Integrity Checking

S3 Browser uses the following approach to retrieve the checksum for a remote file:

1. It first attempts to read the checksum from the custom metadata header x-amz-meta-checksum-<algorithm>.

2. If no checksum is found in the metadata, S3 Browser checks the AWS x-amz-checksum-<algorithm> header.

3. If neither of the above headers contains a valid checksum, it falls back to the ETag header. The ETag may contain an MD5 hash for files uploaded via single-part upload, provided the file is not encrypted with server-side encryption using Amazon S3-managed keys (SSE-S3) or customer-provided keys (SSE-C).

If a valid checksum is found in any of the above headers, S3 Browser compares it with the locally calculated checksum to verify the integrity of the downloaded file.

However, there may be cases where a valid checksum is not available, and a data integrity check cannot be performed.

For example, Amazon S3 does not store checksums for files uploaded without the x-amz-checksum-<algorithm> header. Also, the ETag does not contain a valid MD5 checksum for files uploaded via the Multipart Upload feature or for files encrypted with SSE-S3 or SSE-C.

Amazon S3-compatible storage solutions may use different algorithms or formats for storing checksums, or they may not provide checksums at all.

S3 Browser offers two modes for handling files with missing or unknown checksums:

Flexible - Only files with a valid checksum are verified. Files with missing or invalid checksums are treated as valid.

Strict - All files are checked. Files with missing or invalid checksums are treated as corrupted.

How to Avoid the 'Missing Hash' Issue

One way to avoid the missing hash issue is to enable data integrity checking when uploading files. When this option is enabled, S3 Browser calculates a checksum and stores it in the file's metadata. (Amazon S3 allows storing custom metadata for each file.)

Later, when you download the file, S3 Browser can read the checksum from the metadata and compare it with the newly calculated value to verify file integrity.

Alternatively, you can disable multipart uploads in Tools, Options, General. However, this may reduce upload speed. Additionally, files larger than 5 GB require multipart upload, so this option only applies to smaller files.

How to Fix the Hashes for Already Uploaded Files

The method below allows you to fix file hashes for files up to 5 GB in size. There is no way to update hashes for larger files. To resolve hash issues for files over 5 GB, you must re-upload them with data integrity checking enabled, or switch to Flexible mode for data integrity checking during downloads.

To fix the hashes for files up to 5 GB in size:

1. Click Tools, Options and open the General tab.
2. Disable the Enable multipart uploads option.
3. Select one or more files in the file browser.
4. Open the HTTP Headers tab and click Apply.

S3 Browser will update the ETag header with the correct hash using a simple COPY request.