Azure Storage Blobs client library: How to configure block size separately from the buffer size? - Stack Overflow

admin管理员组

文章数量:1400455

I'm using Azure Storage Blobs client library for .NET to write to an append blob.

I would like to configure a small AppendBlobOpenWriteOptions.BufferSize so that, in the event of an application crash, as little data as possible is lost.

However, when I set the AppendBlobOpenWriteOptions.BufferSize to a small value, such as 1 KB, the block size also becomes 1 KB.

It seems that the AppendBlobOpenWriteOptions.BufferSize is responsible for both the buffer size and the block size.

Is there a way to configure the block size separately from the buffer size?

I want to keep the default block size of 4 MB because, if the block size is too small, I will hit the block limit for my blob (a maximum of 50,000 blocks per blob).

I'm using Azure Storage Blobs client library for .NET to write to an append blob.

I would like to configure a small AppendBlobOpenWriteOptions.BufferSize so that, in the event of an application crash, as little data as possible is lost.

However, when I set the AppendBlobOpenWriteOptions.BufferSize to a small value, such as 1 KB, the block size also becomes 1 KB.

It seems that the AppendBlobOpenWriteOptions.BufferSize is responsible for both the buffer size and the block size.

Is there a way to configure the block size separately from the buffer size?

I want to keep the default block size of 4 MB because, if the block size is too small, I will hit the block limit for my blob (a maximum of 50,000 blocks per blob).

• azure
• azure-blob-storage

Share Improve this question Follow edited Mar 25 at 11:53 Venkat V 7,94322 gold badges44 silver badges1515 bronze badges Recognized by Microsoft Azure Collective asked Mar 25 at 11:32 yaskovdevyaskovdev 1,34422 gold badges1414 silver badges2626 bronze badges 3

• Set AppendBlobOpenWriteOptions.BufferSize to a small value and use Stream.FlushAsync() periodically to commit data without reducing the block size @yaskovdev – Vinay B Commented Mar 25 at 12:01
• @VinayB, thank you for the suggestion. I tried using Stream.FlushAsync, but unfortunately, flushing creates a new block as well. You can see a demo of this behavior in this example. – yaskovdev Commented Mar 26 at 9:17
• Have you tried the solution shared by HarshithaBathini and still facing the issue with the same code?@yaskovdev – Vinay B Commented Mar 26 at 10:05

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Configuring block size separately from the buffer size using C#

As you mentioned theAppendBlobOpenWriteOptions.BufferSize sets both the buffer size and the block size when using AppendBlobClient.OpenWriteAsync().

There is no direct way to configure them separately using OpenWriteAsync().

To keep a small buffer size to reduce data loss but want to keep the default block size of 4 MB, use the AppendBlockAsync() instead of OpenWriteAsync().

You should manually set the buffer size by using this command

 AppendBlobClient.AppendBlockAsync()

I tested in my environment, I manually controlled the block size and buffer size by writing 4 MB blocks using AppendBlockAsync().

using System;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Specialized;
class Program
{
    static async Task Main(string[] args)
    {
        string connectionString = "<Your Connection String>";
        string containerName = "<Container Name>";
        string blobName = "<Blob Name>";
        var blobServiceClient = new BlobServiceClient(connectionString);
        var blobContainerClient = blobServiceClient.GetBlobContainerClient(containerName);
        var appendBlobClient = blobContainerClient.GetAppendBlobClient(blobName);
        if (!await appendBlobClient.ExistsAsync())
        {
            await appendBlobClient.CreateAsync();
        }

        int blockSize = 4 * 1024 * 1024; 
        int bufferSize = 64 * 1024; 
        byte[] buffer = new byte[blockSize];
        string content = "Hello, this is a test write to an append blob.\n";
        byte[] data = Encoding.UTF8.GetBytes(content);
        long totalBytesWritten = 0;
        int bytesFilled = 0;
        var tasks = new List<Task>();
        while (totalBytesWritten < 12582912) 
        {
            int bytesToCopy = Math.Min(data.Length, blockSize - bytesFilled);
            Array.Copy(data, 0, buffer, bytesFilled, bytesToCopy);
            bytesFilled += bytesToCopy;
            if (bytesFilled == blockSize)
            {
                byte[] writeData = new byte[blockSize];
                Array.Copy(buffer, writeData, blockSize);
                tasks.Add(AppendBlockAsync(appendBlobClient, writeData));
                totalBytesWritten += blockSize;
                bytesFilled = 0; 
            }
        }
        if (bytesFilled > 0)
        {
            byte[] finalBlock = new byte[bytesFilled];
            Array.Copy(buffer, finalBlock, bytesFilled);
            tasks.Add(AppendBlockAsync(appendBlobClient, finalBlock));
            totalBytesWritten += bytesFilled;
        }
        await Task.WhenAll(tasks); 
        Console.WriteLine($"Done. Total Bytes Written: {totalBytesWritten}");
    }
       static async Task AppendBlockAsync(AppendBlobClient appendBlobClient, byte[] data)
    {
        using var ms = new MemoryStream(data);
        await appendBlobClient.AppendBlockAsync(ms);
        Console.WriteLine($"Appended {data.Length} bytes.");
    }
}

Output: each append operation writing 4 MB blocks, I had added 12 MB, splitted into 3 separate blocks of each 4 MB. This will not reach the limit of 50,000 blocks too quickly.

AppendBlobClient.AppendBlockAsync Method

本文标签：