javascript - Semaphore equivalent in Node js , variable getting modified in concurrent request? - Stack Overflow

admin管理员组

文章数量:1247246

I am facing this issue for the past 1 week and I am just confused about this. Keeping it short and simple to explain the problem.

We have an in memory Model which stores values like budget etc.Now when a call is made to the API it has a spent associated with it.

We then check the in memory model and add the spent to the existing spend and then check to the budget and if it exceeds we donot accept any more clicks of that model. for each call we also udpate the db but that is a async operation.

A short example

api.get('/clk/:spent/:id', function(req, res) {
   checkbudget(spent, id);
}

checkbudget(spent, id){
  var obj =    in memory model[id]
  obj.spent+= spent;
  obj.spent > obj.budjet // if greater.
    obj.status = 11 // 11 is the stopped status
    update db and rebuild model. 
}

This used to work fine but now with concurrent requests we are getting false spends out spends increase more than budget and it stops after some time. We simulated the call with j meter and found this.

As far as we could find node is async so by the time the status is updated to 11 many threads have already updated the spent for the campaign.

How to have a semaphore kind of logic for Node.js so that the variable budget is in sync with the model

update

 db.addSpend(campaignId, spent, function(err, data) {
        campaign.spent += spent;
        var totalSpent = (+camp.spent) + (+camp.cpb);
        if (totalSpent  > camp.budget) {
            logger.info('Stopping it..');
            camp.status = 11; // in-memory stop
            var History = [];
            History.push(some data);
            db.stopCamp(campId, function(err, data) {
                if (err) {
                    logger.error('Error while stopping );
                }
                model.campMAP = buildCatMap(model);
                model.campKeyMap = buildKeyMap(model);
                db.campEventHistory(cpcHistory, false, function(err) {
                    if (err) {
                        logger.error(Error);
                    }
                })
            });
        }
    });

GIST of the code can anyone help now please

I am facing this issue for the past 1 week and I am just confused about this. Keeping it short and simple to explain the problem.

We have an in memory Model which stores values like budget etc.Now when a call is made to the API it has a spent associated with it.

We then check the in memory model and add the spent to the existing spend and then check to the budget and if it exceeds we donot accept any more clicks of that model. for each call we also udpate the db but that is a async operation.

A short example

api.get('/clk/:spent/:id', function(req, res) {
   checkbudget(spent, id);
}

checkbudget(spent, id){
  var obj =    in memory model[id]
  obj.spent+= spent;
  obj.spent > obj.budjet // if greater.
    obj.status = 11 // 11 is the stopped status
    update db and rebuild model. 
}

This used to work fine but now with concurrent requests we are getting false spends out spends increase more than budget and it stops after some time. We simulated the call with j meter and found this.

As far as we could find node is async so by the time the status is updated to 11 many threads have already updated the spent for the campaign.

How to have a semaphore kind of logic for Node.js so that the variable budget is in sync with the model

update

 db.addSpend(campaignId, spent, function(err, data) {
        campaign.spent += spent;
        var totalSpent = (+camp.spent) + (+camp.cpb);
        if (totalSpent  > camp.budget) {
            logger.info('Stopping it..');
            camp.status = 11; // in-memory stop
            var History = [];
            History.push(some data);
            db.stopCamp(campId, function(err, data) {
                if (err) {
                    logger.error('Error while stopping );
                }
                model.campMAP = buildCatMap(model);
                model.campKeyMap = buildKeyMap(model);
                db.campEventHistory(cpcHistory, false, function(err) {
                    if (err) {
                        logger.error(Error);
                    }
                })
            });
        }
    });

GIST of the code can anyone help now please

• javascript
• node.js
• concurrency

Share Improve this question Follow edited Aug 19, 2017 at 7:40 INFOSYS asked Aug 18, 2017 at 11:24 INFOSYSINFOSYS 1,58599 gold badges2929 silver badges5050 bronze badges 17

• We then check the in memory model and add the spent to the existing spend does this update the in memory model? – Jaromanda X Commented Aug 18, 2017 at 11:32
• update db and rebuild model. so, you only rebuild model once status == 11? is the rebuild synchronous? or is it done once the DB is updated, asynchronously no doubt - really some REAL code would be far easier to answer – Jaromanda X Commented Aug 18, 2017 at 11:34
• Yes this updated the model @JaromandaX – INFOSYS Commented Aug 18, 2017 at 11:35
• Yes only when it is 11 we update the db but we update the in memory model before updating the db, yes db call is async . Sorry i cannot share production code – INFOSYS Commented Aug 18, 2017 at 11:36
• 1 Concurrency, as in multi-threading or multi-process? You are aware that this would not solve your problem, but plicate it even more, right? – mingos Commented Aug 18, 2017 at 12:55

 |  Show 12 more ments

2 Answers 2

Sorted by: Reset to default 8

Q: Is there semaphore or equivalent in NodeJs?

A: No.

Q: Then how do NodeJs users deal with race condition?

A: In theory you shouldn't have to as there is no thread in javascript.

Before going deeper into my proposed solution I think it is important for you to know how NodeJs works.

For NodeJs it is driven by an event based architecture. This means that in the Node process there is an event queue that contains all the "to-do" events.

When an event gets pop from the queue, node will execute all of the required code until it is finished. Any async calls that were made during the run were spawned as other events and they are queued up in the event queue until a response is heard back and it is time to run them again.

Q: So what can I do to ensure that only 1 request can perform updates to the database at a time?

A: I believe there are many ways you can achieve this but one of the easier way out is to use the set_timeout API.

Example:

api.get('/clk/:spent/:id', function(req, res) {
   var data = { 
       id: id
       spending: spent
   }
   canProceed(data, /*functions to exec after canProceed=*/ checkbudget);
}

var canProceed = function(data, next) {
    var model = in memory model[id];

    if (model.is_updating) {
        set_timeout(isUpdating(data, next), /*try again in=*/1000/*milliseconds*/);
    }
    else {
        // lock is released. Proceed.
        next(data.spending, data.id)
    }
}


checkbudget(spent, id){
  var obj =    in memory model[id]

  obj.is_updating = true; // Lock this model

  obj.spent+= spent;
  obj.spent > obj.budjet // if greater.
    obj.status = 11 // 11 is the stopped status
    update db and rebuild model. 
    obj.is_updating = false; // Unlock the model
}

Note: What I got here is pseudo code as well so you'll may have to tweak it a bit.

The idea here is to have a flag in your model to indicate whether a HTTP request can proceed to do the critical code path. In this case your checkbudget function and beyond.

When a request es in it checks the is_updating flag to see if it can proceed. If it is true then it schedules an event, to be fired in a second later, this "setTimeout" basically bees an event and gets placed into node's event queue for later processing

When this event gets fired later, the checks again. This occurs until the is_update flag bees false then the request goes on to do its stuff and is_update is set to false again when all the critical code is done.

Not the most efficient way but it gets the job done, you can always revisit the solution when performance bees a problem.

The existing answer is terrible.

First of all, yes, there is a semaphore in Nodejs: https://www.npmjs./package/semaphore.

Second of all, their proposed solution is a spin lock (https://en.wikipedia/wiki/Spinlock) which is just about the worst solution possible for critical sections. Please just use a semaphore and not these weird nonstandard terrible coding practices. Just because nodejs runs on one process does NOT mean that there aren't critical sections and race conditions...

Edit: I did make a mutex object the other day. If you need some inspiration:

import chalk from "chalk";

type PromiseKey = {
  [key: string]: Promise<any> | undefined;
}
type ResolverKey = {
  [key: string]: ((value: unknown) => void) | undefined;
}

export const SEMAPHORE_DEBUG = false;

export class StringSemaphore {
  private kvLock: PromiseKey;
  private resolvers: ResolverKey;
  constructor() {
    this.kvLock = {};
    this.resolvers = {};
  }
  async isLocked(key: string): Promise<boolean> {
    if (!!this.kvLock[key]) {
      if (SEMAPHORE_DEBUG) {
        console.log(chalk.bgMagenta('[Semaphore]'), key, 'blocked.')
      }
      await this.kvLock[key];
      delete this.kvLock[key];
      return true;
    }
    return false;
  }
  lock(key: string) {
    if (SEMAPHORE_DEBUG) {
      console.log(chalk.bgMagenta('[Semaphore]'), key, 'locked.')
    }
    const lockPromise = new Promise((resolve) => {
      this.resolvers[key] = resolve;
    });
    this.kvLock[key] = lockPromise;
  }
  release(key: string, value: unknown) {
    if (SEMAPHORE_DEBUG) {
      console.log(chalk.bgMagenta('[Semaphore]'), key, 'released.')
    }
    if (this.resolvers[key]) {
      this.resolvers[key](value);
      delete this.resolvers[key];
    }
  }
}

Usage:

const mySemaphore = new StringSemaphore()
function myJob(jobID: string) {
  if (mySemaphore.isLocked(jobID)) {
    // will block until job of same ID is pleted
    // now you can perform the database result lookup
    return;
  }
  // the job was not locked so now you can enter the critical section
  mySemaphore.lock(jobID);
  // do critical section
  // save result to db or something
  mySemaphore.release(jobID);
}

You need to make a few changes to convert it into a true semaphore such as by making the lock and release methods increment and decrement, but you should get the idea of what to do.

本文标签：