amazon s3 - A way to implement files upload session - Stack Overflow

admin管理员组

文章数量:1122846

I've been exploring the topic of upload sessions, and I still have a lot of questions.

I've observed how some websites implement upload sessions, such as Mangadex, in the following way:

1. Send a request to the server to retrieve the old upload session ID. If it exists, they delete the old session.
2. Send another request to get a new upload session ID.
3. Batch upload user files to S3 (3 files per request). The response contains some information like a generated file ID (Uuid v4 in this case), file hash, original file name, etc.
4. Send a commit request to the server to confirm the upload session is complete. The server processes the session (resize, scramble, etc.).

My questions:

1. Can I handle the file upload process from the client to S3 without an intermediary server that generates a file ID, or must I pass the file through an intermediary to generate the file ID?

2. If I must pass to the server to generate file IDs, how can I scale when it can handle up to 1000 images per minute? My VPS has 4GB of RAM.

The process above is only my assumption. If it is wrong, please provide me with the correct way to implement a file upload session.

I've been exploring the topic of upload sessions, and I still have a lot of questions.

I've observed how some websites implement upload sessions, such as Mangadex, in the following way:

1. Send a request to the server to retrieve the old upload session ID. If it exists, they delete the old session.
2. Send another request to get a new upload session ID.
3. Batch upload user files to S3 (3 files per request). The response contains some information like a generated file ID (Uuid v4 in this case), file hash, original file name, etc.
4. Send a commit request to the server to confirm the upload session is complete. The server processes the session (resize, scramble, etc.).

My questions:

1. Can I handle the file upload process from the client to S3 without an intermediary server that generates a file ID, or must I pass the file through an intermediary to generate the file ID?

2. If I must pass to the server to generate file IDs, how can I scale when it can handle up to 1000 images per minute? My VPS has 4GB of RAM.

The process above is only my assumption. If it is wrong, please provide me with the correct way to implement a file upload session.

• file
• amazon-s3
• multipartfile

Share Improve this question Follow edited yesterday James Z 12.3k1010 gold badges2727 silver badges4747 bronze badges asked yesterday DogguDoggu 9811 gold badge22 silver badges1010 bronze badges 3

• aws amplify may be worth having a look at to avoid any server side processing. can’t remember what data is available after upload — docs.amplify.aws/javascript/build-a-backend/storage/… – dijksterhuis Commented yesterday
• you could generate a custom UUID client side and then set it as a key-value in the metadata of the object during the upload to S3 — docs.amplify.aws/javascript/build-a-backend/storage/… you can list s3 objects to get the UUIDs later on (should be part of the returned data from what i remember) docs.amplify.aws/javascript/build-a-backend/storage/list-files/… – dijksterhuis Commented yesterday
• I will try that. Thank you for the link – Doggu Commented yesterday

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

You could use presignedURL to do that:

1. Send a request from client side to server side to generate presignedURL (server side will authenticate and process where the file will be placed,... )
2. After that, server will send back to client side a presignedURL with a timeout setting. Client side will use that URL to make a request to upload file directly into S3

You maybe ask about the file information (size, metadata,... ) You could get that information and send to server in the first request. It means you will send the file information in the request which send back the presignedURL. At that process, you could verify something such as:

• Is client side domain in your system?
• File type in the acceptance list?
• ... etc

Note:

• Maybe we would need to verify some properties in both client side and server
• If you uploaded many times with the same presignedURL. It would replace the old one
• Maybe you need another service to check the file content (in case end-user uploads sensitive file or something like that)

I hope my idea could help you

本文标签： amazon s3A way to implement files upload sessionStack Overflow