Comments (5)
This is not a bug.
The md5 it is only checked by the server when it is sent with the entire blob or it is set via uploadProperties / uploadBlockList where it is not validated, but will be returned on subsequent downloads. Any subsequent upload will overwrite the Content Md5 of the pre-existing blob.
For example if you upload a blob in a single put request (i.e. <64MB) with a content md5 header it is validated prior to storing the blob, however if you upload the blob, then do a uploadProperties with a md5 this value is simply stored as metadata. This is due to scale reasons, i.e. if you blob was 250 gb the service would have to retrieve all of the data in order to validate the md5 which is not feasible.
The Java library provided has a few conveniences to help for your scenario. Specifically BlobRequestOptions provides a setting UseTransactionalMd5 (default to false) and StoreBlobContentMd5(default to false) which will calculate and store the values you are seeking automatically. As explained in the blog the Blobs content md5 and the md5 of the data in a single request ( referred to as transactional md5) are different.
See http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/18/windows-azure-blob-md5-overview.aspx for more. Specifically the REST section.
from azure-sdk-for-java.
Thanks for the request options. Now I can work around to upload the full blob and get md5 checksum from blob properties after the upload to check it.
However, the following Java code should fail when upload but it went through no problem. I tried to set the md5 just before upload, and the md5 is from a different file than the file being uploaded. The upload process didn't complain.
package movideo.azuretest;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URISyntaxException;
import java.security.InvalidKeyException;
import org.apache.commons.codec.digest.DigestUtils;
import com.microsoft.windowsazure.services.blob.client.BlobContainerPermissions;
import com.microsoft.windowsazure.services.blob.client.BlobContainerPublicAccessType;
import com.microsoft.windowsazure.services.blob.client.CloudBlob;
import com.microsoft.windowsazure.services.blob.client.CloudBlobClient;
import com.microsoft.windowsazure.services.blob.client.CloudBlobContainer;
import com.microsoft.windowsazure.services.blob.client.CloudBlockBlob;
import com.microsoft.windowsazure.services.blob.client.ListBlobItem;
import com.microsoft.windowsazure.services.core.storage.CloudStorageAccount;
import com.microsoft.windowsazure.services.core.storage.StorageException;
import com.microsoft.windowsazure.services.core.storage.utils.Base64;
public class Test1
{
public static final String storageConnectionString = "<<Clould connection string>>";
/**
* @param args
* @throws URISyntaxException
* @throws InvalidKeyException
* @throws StorageException
* @throws IOException
* @throws FileNotFoundException
*/
public static void main(String[] args) throws InvalidKeyException, URISyntaxException, StorageException, FileNotFoundException, IOException
{
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference("mycontainer1");
container.createIfNotExist();
/*
* Test point 1:
* I have tried to make the public access CONTAINER, BLOB and PRIVATE.
*/
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
containerPermissions.setPublicAccess(BlobContainerPublicAccessType.BLOB);
container.uploadPermissions(containerPermissions);
CloudBlockBlob blob = container.getBlockBlobReference("playlist/7/100x20.jpg");
/*
* Test point 2:
* Here are two files with only one byte difference (simulate a network transmit error).
* Tried upload srcFile's md5 with modSrcFile, which should fail, but not.
*/
File srcFile = new File("/home/mwan/workspace/files/azure/launch.jpg");
File modSrcFile = new File("/home/mwan/workspace/files/azure/launch-mod.jpg");
FileInputStream inputStream = new FileInputStream(srcFile);
/*
* Test point 3:
* Tried to set the md5 and not set the md5.
* According to {@link http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/18/windows-azure-blob-md5-overview.aspx}
* set the md5 should make the server to check it.
*/
blob.getProperties().setContentMD5(Base64.encode(DigestUtils.md5(inputStream)));
/*
* Test point 4:
* Tried to upload the correct file and the incorrect file.
*/
// blob.upload(inputStream, srcFile.length());
blob.upload(new FileInputStream(modSrcFile), srcFile.length());
/*
* Test point 5:
* Get the md5 back from server to verify the file is uploaded successfully, but always getting null.
*/
for(ListBlobItem blobItem : container.listBlobs("playlist/7/"))
{
CloudBlob ablob = new CloudBlockBlob(blobItem.getUri(), blobClient);
System.out.println(ablob.getName() + " md5 => " + ablob.getProperties().getContentMD5());
}
}
}
from azure-sdk-for-java.
There are two ways to upload a block blob, either in a single put ( up to 64 mb) or as individual blocks with a trailing put block list. The single put will validate the md5 as all data is at the server when the header is retrieved, the put block list will not validate it as all the data is not part of the request itself, instead it will simply store it on the bog. We expose the latter via the BlobOutputStream which is accessible via openOutputStream, or used internally for uploads that meet certain criteria. If the blob is over the SinglePutThreshold (configurable on the client) or if the InputStream is non markable the upload methods will revert to the OutputStream, In your case I would recommend using the BufferedInputStream to work around this.
We will do better in documenting this more fully in the future, however the BlobRequestOptions properties are designed to offer the most seamless way of working with md5.
joe
from azure-sdk-for-java.
I came across the same problem. The single block blob put you suggested doesn't validate MD5 when I use the Java SDK. I set the MD5 with blob.getProperties().setContentMD5(). I have already set setStoreBlobContentMD5 and setUseTransactionalContentMD5 of the BlobRequestOptions object to true and use blob.upload(inpustream, length, null, blobRequestOptions, null) to supply the options.
I believe this is a bug. It's either the MD5 validation is ignored at the server side or the MD5 is not sent to the server when the blob upload request is composed.
I suggest you try it out the code and you'll know it isn't working. Otherwise, please include the code you've tested is working.
from azure-sdk-for-java.
See also https://stackoverflow.com/a/69319211/32453
from azure-sdk-for-java.
Related Issues (20)
- openai: Unrecognized request argument supplied:stream_options HOT 2
- [resource manager] Refactor resourcemanager tests to work with latest TestProxy HOT 3
- Live Test federated identity support
- [Monitor Ingestion] Refactor Monitor Ingestion tests to work with latest TestProxy
- [Test Proxy Attestation] Refactor and re-record test HOT 1
- Investigate canSendKnownLengthNoBufferAsyncFluxBinaryData time out issue
- [Test proxy] Refactor and re-record quantum jobs tests
- [Test Proxy] Refactor and re-record needed for storage tests
- [FEATURE REQ] Add support for managed identities in azure-storage-file-share HOT 3
- [BUG] Multiple createPathFile internal operations causing uploadFIle api to convert finite size file to zero bytes file in ADLS gen2 HOT 9
- [OpenAI][Assistants] Add custom code for `uploadFile` for server requirement for `filename`
- Legacy flag for the Java SDK
- sdk/cosmos/azure-cosmos-spark_3-5_2-12 - "_corrupt_record" column isn't getting populated when reading from Cosmos DB HOT 5
- [BUG] Different embedding 3 vectors in Azure vs. OpenAI HOT 3
- User can't manage the lifecycle of THREAD_POOL in Azure Table Storage Java SDK HOT 2
- [BUG] Azure SQL ActiveDirectoryManagedIdentity authentication with sleuth HOT 3
- Is it missing partial update function for blob HOT 7
- [Text Translation] Use the latest test proxy
- [BUG] Unable to list Virtual Networks in an Subscription HOT 4
- [FEATURE REQ] [OpenAI Client] Feature parity with official Python client (and also fully support vLLM) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-sdk-for-java.