javascript - AWS Lambda tmp folder is not shared between executions - Stack Overflow

admin管理员组

文章数量:1290927

I'm trying to temporary cache some data for AWS lambda function executions to omit HTTP request that this lambda does every time right now and optimise its' speed.

In "viewer request" I have the following logic (example is pseudo-code) -

exports.handler = async (event) => {
   // check /tmp folder for data
   const cachedData = await getDataFromCache();

   if (cachedData) {
     return cachedData;
   }

   const data = await getDataFromApi();
   
   // save data for 2 mins in /tmp for re-use in next executions
   await saveDataInCache(data);

   return data;
}

As you can see, I'm trying to save the data in /tmp folder for a few mins in order to re-use it in next executions and reduce the number of API requests.

During "viewer request" lambda execution, when I cache data inside /tmp folder, I immediately see that data is available in the logs -

async function saveDataInCache(data = {}) {
  // save data in tmp folder
  try {
    await fs.outputJson('/tmp/cache.json', {
      data,
      updatedAt: Date.now()
    });
  } catch (err) {
    console.log(`page data save cache error ${err}`);
  }

  // check that data is immediately available
  let fileData = {};
  try {
    fileData = await fs.readJson('/tmp/cache.json');
  } catch (err) {
    console.log(`page data read cache error ${err}`);
  }

  console.log(`check immediate value ${JSON.stringify(fileData)}`);
}

However every time when "viewer request" tries to read the data before saving, it always returns error - ENOENT: no such file or directory, open '/tmp/cache.json' -

async function getDataFromCache() {
  let fileData = {};

  try {
    fileData = await fs.readJson('/tmp/cache.json');
  } catch (err) {
    console.log(`page data read cache error ${err}`);
  }

  if (isEmpty(fileData)) {
    return undefined;
  }

  const { data = {}, updatedAt = 0 } = fileData;
  const maxCacheTime = updatedAt + DATA_TTL;
  const currentTime = Date.now();
  const isCacheExpired = currentTime > maxCacheTime;

  if (!isCacheExpired) {
    return data;
  }

  return undefined;
}

Does it mean that /tmp folder content is not shared between executions? Thanks!

I'm trying to temporary cache some data for AWS lambda function executions to omit HTTP request that this lambda does every time right now and optimise its' speed.

In "viewer request" I have the following logic (example is pseudo-code) -

exports.handler = async (event) => {
   // check /tmp folder for data
   const cachedData = await getDataFromCache();

   if (cachedData) {
     return cachedData;
   }

   const data = await getDataFromApi();
   
   // save data for 2 mins in /tmp for re-use in next executions
   await saveDataInCache(data);

   return data;
}

As you can see, I'm trying to save the data in /tmp folder for a few mins in order to re-use it in next executions and reduce the number of API requests.

During "viewer request" lambda execution, when I cache data inside /tmp folder, I immediately see that data is available in the logs -

async function saveDataInCache(data = {}) {
  // save data in tmp folder
  try {
    await fs.outputJson('/tmp/cache.json', {
      data,
      updatedAt: Date.now()
    });
  } catch (err) {
    console.log(`page data save cache error ${err}`);
  }

  // check that data is immediately available
  let fileData = {};
  try {
    fileData = await fs.readJson('/tmp/cache.json');
  } catch (err) {
    console.log(`page data read cache error ${err}`);
  }

  console.log(`check immediate value ${JSON.stringify(fileData)}`);
}

However every time when "viewer request" tries to read the data before saving, it always returns error - ENOENT: no such file or directory, open '/tmp/cache.json' -

async function getDataFromCache() {
  let fileData = {};

  try {
    fileData = await fs.readJson('/tmp/cache.json');
  } catch (err) {
    console.log(`page data read cache error ${err}`);
  }

  if (isEmpty(fileData)) {
    return undefined;
  }

  const { data = {}, updatedAt = 0 } = fileData;
  const maxCacheTime = updatedAt + DATA_TTL;
  const currentTime = Date.now();
  const isCacheExpired = currentTime > maxCacheTime;

  if (!isCacheExpired) {
    return data;
  }

  return undefined;
}

Does it mean that /tmp folder content is not shared between executions? Thanks!

• javascript
• node.js
• amazon-web-services
• caching
• aws-lambda

Share Improve this question Follow asked Apr 29, 2021 at 8:03 KosmetikaKosmetika 21.3k3838 gold badges112112 silver badges177177 bronze badges 3

• How much time passed between the two writing and reading requests? Can you see in the AWS logs that you actually hit the same execution environment a second time instead of a new one? – luk2302 Commented Apr 29, 2021 at 8:06
• > How much time passed between the two writing and reading requests? - I do requests immediately one after another, maybe 1 min passes in between – Kosmetika Commented Apr 29, 2021 at 8:08
• > Can you see in the AWS logs that you actually hit the same execution environment a second time instead of a new one? - I think you're right, after doing more calls I'm starting to see the data retrieved from the tmp folder. So it was due to different execution environments – Kosmetika Commented Apr 29, 2021 at 8:30

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 8

The /tmp directory is private to each Lambda instance/execution context, it's not shared among all Lambda instances. That means if your request is handled by a different execution context, the data won't be present there.

Unfortunately you can't really control how these execution contexts are created and deleted, that's something Lambda handles for you in the background.

Short answer is as Maurice said, /tmp is unique to each container, so if you're lucky you'll get the same container back, but it's not guaranteed. After the container is disposed of all files are gone.

Use EFS, S3, or Lambda layers if you need real persistent storage between Lambda's.

That being said what's the use case?

Cache.json updated during each execution

Then use EFS/s3(bit slower than EFS) OR a NoSQL database if you're already using one, another collection isn't going to change the world and will be super fast.

Static file cache.json

Either bundle this up with your lambda code or put it in a layer!

本文标签： javascriptAWS Lambda tmp folder is not shared between executionsStack Overflow