javascript - AWS Lambda read-only file system error failed to create directory with Docker image - Stack Overflow

admin管理员组

文章数量:1300048

Problem

Docker image piles successfully, however fails when ran from Lambda because of its read only file system.

Summary

Luminati-proxy has a docker integration for their proxy manager. I copied over their docker file and appended it to my own docker file for pushing out a script to AWS Lambda. The building of the docker image was successful, but when pushed off to Lambda, it failed because of a read only file system error:

Failed to create directory /home/sbx_user1051/proxy_manager: [code=EROFS] Error: EROFS: read-only file system, mkdir '/home/sbx_user1051'
2022-02-28 19:37:22.049 FILE (8): Failed to create directory /home/sbx_user1051/proxy_manager: [code=EROFS] Error: EROFS: read-only file system, mkdir '/home/sbx_user1051' 

Analysis

Upon examining the trackback, the error is focused on the proxy_manager installation and fails with directory changes (mkdir, mk_work_dir ...). These changes are made within the .js files of the GitHub which is pulled from the docker file as the proxy_manager installation. Obviously the only mutable directory on Lambda is the /tmp directory, but is there a workaround for getting this set up without resorting to putting everything under the /tmp directory as it wipes itself upon runtime? Reinstalling a proxy_manager each run is not at all ideal...

Answer?

Could this be as simple as setting environment stipulations such as:

ENV PATH=...
ENV LD_LIBRARY_PATH=...

If so, I how should they be configured? I am adding the docker file below for quick reference:

FROM node:14.18.1
RUN wget -q -O - .pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] / stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
      --no-install-remends \
    && rm -rf /var/lib/apt/lists/*

USER root
RUN npm config set user root
RUN npm install -g [email protected]
RUN npm install -g @luminati-io/luminati-proxy
ENV DOCKER 1
CMD ["luminati", "--help"]

I appreciate the insight!

Problem

Docker image piles successfully, however fails when ran from Lambda because of its read only file system.

Summary

Luminati-proxy has a docker integration for their proxy manager. I copied over their docker file and appended it to my own docker file for pushing out a script to AWS Lambda. The building of the docker image was successful, but when pushed off to Lambda, it failed because of a read only file system error:

Failed to create directory /home/sbx_user1051/proxy_manager: [code=EROFS] Error: EROFS: read-only file system, mkdir '/home/sbx_user1051'
2022-02-28 19:37:22.049 FILE (8): Failed to create directory /home/sbx_user1051/proxy_manager: [code=EROFS] Error: EROFS: read-only file system, mkdir '/home/sbx_user1051' 

Analysis

Upon examining the trackback, the error is focused on the proxy_manager installation and fails with directory changes (mkdir, mk_work_dir ...). These changes are made within the .js files of the GitHub which is pulled from the docker file as the proxy_manager installation. Obviously the only mutable directory on Lambda is the /tmp directory, but is there a workaround for getting this set up without resorting to putting everything under the /tmp directory as it wipes itself upon runtime? Reinstalling a proxy_manager each run is not at all ideal...

Answer?

Could this be as simple as setting environment stipulations such as:

ENV PATH=...
ENV LD_LIBRARY_PATH=...

If so, I how should they be configured? I am adding the docker file below for quick reference:

FROM node:14.18.1
RUN wget -q -O - https://dl-ssl.google./linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google./linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
      --no-install-remends \
    && rm -rf /var/lib/apt/lists/*

USER root
RUN npm config set user root
RUN npm install -g [email protected]
RUN npm install -g @luminati-io/luminati-proxy
ENV DOCKER 1
CMD ["luminati", "--help"]

I appreciate the insight!

• javascript
• node.js
• amazon-web-services
• docker
• github

Share Improve this question Follow asked Mar 6, 2022 at 3:58 Luke HamiltonLuke Hamilton 73777 silver badges2020 bronze badges 5

• This question should be reduced purely to getting proxy_manager to run under different directories - there is no inherent issue with AWS here and you're more likely to get answers in that case as this is 2 Qs related in 1. – Ermiya Eskandary Commented Mar 6, 2022 at 8:26
• @ErmiyaEskandary I do realize that, I am trying to be as general as possible so that the question can get answered as well as aid others who have the same problem. I could have worded it better and expressed how docker image works locally but not on AWS because of AWS directory restrictions. That is good insight though, I appreciate it – Luke Hamilton Commented Mar 7, 2022 at 2:30
• @LukeHamilton I don't get it RUN mand is done during build, so how is npm install for luminati-proxy be the root cause? The other npm install would also be modifying other directories. Are you saying the error is actually from CMD, just that it is using the files pulled during install? – Winson Tanputraman Commented Mar 11, 2022 at 2:52
• Also wouldn't the proxy try to write logs, which will again fail due to the read-only restriction? To me sounds like you have to configure the working dir in tmp anyway. If the mkdir is a one time thing, I think you can initiate this by executing the mand as a RUN luminati --help once? – Winson Tanputraman Commented Mar 11, 2022 at 3:08
• @RegisterSole I am confused by it as well. I have offboarded other Docker images to Lambda with no trouble which means it has to be the proxy portion of the docker file. It's hard to pinpoint exactly where the error es from, the only thing I know is that there is a problem with the write/read file permissions which explains the error in the summary portion of my post. – Luke Hamilton Commented Mar 11, 2022 at 15:15

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 5 +25

TL;DR:

• You should instead leverage an S3 bucket to store, read and modify any file. All lambdas and microservices. In general, should always be treated as stateless
• All Luminati-proxy functionality es prebuilt within amazon lambdas and API Gateway
• Lambda functions are not meant to run long-running processes as they are limited to 15 minutes maximum so the design of the container that you are trying to run in lambdas has to have AWS serverless architecture considerations in its design

Explanation:

According to the documentation of AWS Lambda functions:

The container image must be able to run on a read-only file system. Your function code can access a writable /tmp directory with 512 MB of storage.

Since containers based on Linux based images are already supposed to have a folder called /tmp you should pretty much be able to access that folder any time from your code to read( remember, read-only FS)

If you are looking to store content amazon's solution for that is for you to have any content created and manage over an S3 bucket, buckets are as easy to use as if you read a file locally but will remain accessible after the lambda instance finishes the workload

Please refer to Read file from aws s3 bucket using node fs and Upload a file to Amazon S3 with NodeJS for more details on how to use an S3 bucket. There are plenty of ways to achieve it regardless of the language been used.

This is all based on a best practice promoted by AWS over their platform. Where lambdas remain stateless

AWS Lambda provides /tmp folder for users to write files on lambda, as I don't know about your question context but hope this help. You can write files to AWS Lambda at /tmp folder eg. I want to create a file demo.txt at runtime/programmatically using AWS lambda, then i can write the file to /tmp/demo.txt

本文标签： javascriptAWS Lambda readonly file system error failed to create directory with Docker imageStack Overflow