admin管理员组

文章数量:1400188

I have a Laravel 5.3 installation running as a pure API application and need to connect from several different applications.

Everything is working fine (after all it's Laravel we're talking about :P), except I can't figure out one thing:

I have a MQTT server that is listening for messages from several devices (doesn't matter what). These messages contain information about a Job Class and method that need to be called on the backend.

I can't call the API directly, the devices simply don't support this (they do, but it's a lot more effort than using MQTT to transmit data). My thought was to push a new job onto the queue defining which Laravel Job Class to call (and which method). The problem is the JSON serialization...

The MQTT server is running on NodeJS, my queues are running on Redis. I remember a Tweet from Taylor where he mentioned that theoretically it could be possible to serialize the JSON needed and push to the queue from outside Laravel, and have the job processed by Laravel.

Anyone any idea how to approach this? Is there a documentation available about the structure of the JSON?

I should also mention that this solution NodeJS push queue, consumed by Laravel worker did not work for me. Same result as above, the job is placed on the queue but discarded without being processed or any errors thrown.

The sample data structure for a queued Event in Redis looks like this:

"{\"job\":\"Illuminate\\\\Broadcasting\\\\BroadcastEvent\",\"data\":{\"event\":\"O:28:\\\"App\\\\Events\\\\NotificationEvent\\\":5:{s:7:\\\"\\u0000*\\u0000name\\\";s:12:\\\"notification\\\";s:4:\\\"data\\\";a:4:{s:4:\\\"testkey\\\";s:14:\\\"testval\\\";s:9:\\\"timestamp\\\";s:19:\\\"2017-02-24 11:07:48\\\";s:5:\\\"event\\\";s:12:\\\"notification\\\";s:5:\\\"class\\\";s:28:\\\"App\\\\Events\\\\NotificationEvent\\\";}s:10:\\\"\\u0000*\\u0000channel\\\";N;s:7:\\\"\\u0000*\\u0000user\\\";O:45:\\\"Illuminate\\\\Contracts\\\\Database\\\\ModelIdentifier\\\":2:{s:5:\\\"class\\\";s:8:\\\"App\\\\User\\\";s:2:\\\"id\\\";i:2;}s:6:\\\"socket\\\";N;}\"},\"id\":\"XuUKRTf8CTSdzaVgp2gRcvmxQqLcpBUG\",\"attempts\":1}"

Based on that structure, I would think the object (that needs to be serialized) should look similar to this:

{
"job":"EventClass@method", //<-- Just a name
"data":{
    "event":"EventClass", //<-- Just a name
    "name":"EventName", //<-- Just a name
    "data":{
    "key":"value"
    "event":"EventName" //<-- Same as data.name
    "class":"EventClass@method" //<-- This is actually being called
    }
}

There is additional information included in what Laravel actually puts on the queue (like a timestamp, user model identifier, etc), but I don't think that's necessary to trigger the job.

The data needs to be serialized in JS to achieve a similar output than with php serialize() (or better, to get a string that can be unserialized by php's unserialize().

I achieved to do so with the php-serialization NPM module (Thanks to Simon Svensson), but the job still isn't being consumed by Laravel (discarded but not executed)

Thanks in advance for any help :)

EDIT SOLUTION

Thanks to Simon's answer, here's the solution on how to serialize job data in Javascript and push onto a Laravel queue (and have Laravel process the whole thing automatically).

Note that this is an example for using queues with Redis. When using Beanstalkd or Database based queues, this might look different (or not).

This is the code I successfully used:

var serialize,Class,job,jobUser,jobData,serialized,result;

serialize = require('php-serialization').serialize;
Class = require('php-serialization').Class;

job = new Class("App\\Events\\NotificationEvent");

job.__addAttr__("name","string","notification","string","protected");

jobData = new Class();
jobData.__addAttr__("testkey","string","testval","string");
jobData.__addAttr__("timestamp","string","2017-02-24 11:07:48","string");
jobData.__addAttr__("event","string","notification","string");
jobData.__addAttr__("class","string","App\\Events\\NotificationEvent","string");
job.__addAttr__("data","string",jobData,"array","public");

job.__addAttr__("channel","string",null,"null","protected");

jobUser = new Class("Illuminate\\Contracts\\Database\\ModelIdentifier")
jobUser.__addAttr__("class","string","App\\User","string","public");
jobUser.__addAttr__("id","string",2,"integer","public");
job.__addAttr__("user","string",jobUser,"object","protected");

job.__addAttr__("socket","string",null,"null","public");

serialized = serialize(job,"object");

result = {
    job:"Illuminate\\Broadcasting\\BroadcastEvent",
    data:{
        event:serialized
    },
    id:"XuUKRTf8CTSdzaVgp2gRcvmxQqLcpBUG",
    attempts:1
};

queue.rpush('queues:default',JSON.stringify(result));

I haven't figured out yet what the ID is exactly for, I successfully pushed jobs onto the queue with always the same Id. I guess if you're pushing jobs at a fast pace and they are stored at the same time it could be a problem. Since it's a string you could substitute it with any random ID you like (the random ID generated by Laravel is 32 characters, I think it's a good idea to keep this length).

Attempts should be set to 1 when initially pushing the job. If Laravel can't process the job, it will push it back onto the queue and increase the attempts value.

I have a Laravel 5.3 installation running as a pure API application and need to connect from several different applications.

Everything is working fine (after all it's Laravel we're talking about :P), except I can't figure out one thing:

I have a MQTT server that is listening for messages from several devices (doesn't matter what). These messages contain information about a Job Class and method that need to be called on the backend.

I can't call the API directly, the devices simply don't support this (they do, but it's a lot more effort than using MQTT to transmit data). My thought was to push a new job onto the queue defining which Laravel Job Class to call (and which method). The problem is the JSON serialization...

The MQTT server is running on NodeJS, my queues are running on Redis. I remember a Tweet from Taylor where he mentioned that theoretically it could be possible to serialize the JSON needed and push to the queue from outside Laravel, and have the job processed by Laravel.

Anyone any idea how to approach this? Is there a documentation available about the structure of the JSON?

I should also mention that this solution NodeJS push queue, consumed by Laravel worker did not work for me. Same result as above, the job is placed on the queue but discarded without being processed or any errors thrown.

The sample data structure for a queued Event in Redis looks like this:

"{\"job\":\"Illuminate\\\\Broadcasting\\\\BroadcastEvent\",\"data\":{\"event\":\"O:28:\\\"App\\\\Events\\\\NotificationEvent\\\":5:{s:7:\\\"\\u0000*\\u0000name\\\";s:12:\\\"notification\\\";s:4:\\\"data\\\";a:4:{s:4:\\\"testkey\\\";s:14:\\\"testval\\\";s:9:\\\"timestamp\\\";s:19:\\\"2017-02-24 11:07:48\\\";s:5:\\\"event\\\";s:12:\\\"notification\\\";s:5:\\\"class\\\";s:28:\\\"App\\\\Events\\\\NotificationEvent\\\";}s:10:\\\"\\u0000*\\u0000channel\\\";N;s:7:\\\"\\u0000*\\u0000user\\\";O:45:\\\"Illuminate\\\\Contracts\\\\Database\\\\ModelIdentifier\\\":2:{s:5:\\\"class\\\";s:8:\\\"App\\\\User\\\";s:2:\\\"id\\\";i:2;}s:6:\\\"socket\\\";N;}\"},\"id\":\"XuUKRTf8CTSdzaVgp2gRcvmxQqLcpBUG\",\"attempts\":1}"

Based on that structure, I would think the object (that needs to be serialized) should look similar to this:

{
"job":"EventClass@method", //<-- Just a name
"data":{
    "event":"EventClass", //<-- Just a name
    "name":"EventName", //<-- Just a name
    "data":{
    "key":"value"
    "event":"EventName" //<-- Same as data.name
    "class":"EventClass@method" //<-- This is actually being called
    }
}

There is additional information included in what Laravel actually puts on the queue (like a timestamp, user model identifier, etc), but I don't think that's necessary to trigger the job.

The data needs to be serialized in JS to achieve a similar output than with php serialize() (or better, to get a string that can be unserialized by php's unserialize().

I achieved to do so with the php-serialization NPM module (Thanks to Simon Svensson), but the job still isn't being consumed by Laravel (discarded but not executed)

Thanks in advance for any help :)

EDIT SOLUTION

Thanks to Simon's answer, here's the solution on how to serialize job data in Javascript and push onto a Laravel queue (and have Laravel process the whole thing automatically).

Note that this is an example for using queues with Redis. When using Beanstalkd or Database based queues, this might look different (or not).

This is the code I successfully used:

var serialize,Class,job,jobUser,jobData,serialized,result;

serialize = require('php-serialization').serialize;
Class = require('php-serialization').Class;

job = new Class("App\\Events\\NotificationEvent");

job.__addAttr__("name","string","notification","string","protected");

jobData = new Class();
jobData.__addAttr__("testkey","string","testval","string");
jobData.__addAttr__("timestamp","string","2017-02-24 11:07:48","string");
jobData.__addAttr__("event","string","notification","string");
jobData.__addAttr__("class","string","App\\Events\\NotificationEvent","string");
job.__addAttr__("data","string",jobData,"array","public");

job.__addAttr__("channel","string",null,"null","protected");

jobUser = new Class("Illuminate\\Contracts\\Database\\ModelIdentifier")
jobUser.__addAttr__("class","string","App\\User","string","public");
jobUser.__addAttr__("id","string",2,"integer","public");
job.__addAttr__("user","string",jobUser,"object","protected");

job.__addAttr__("socket","string",null,"null","public");

serialized = serialize(job,"object");

result = {
    job:"Illuminate\\Broadcasting\\BroadcastEvent",
    data:{
        event:serialized
    },
    id:"XuUKRTf8CTSdzaVgp2gRcvmxQqLcpBUG",
    attempts:1
};

queue.rpush('queues:default',JSON.stringify(result));

I haven't figured out yet what the ID is exactly for, I successfully pushed jobs onto the queue with always the same Id. I guess if you're pushing jobs at a fast pace and they are stored at the same time it could be a problem. Since it's a string you could substitute it with any random ID you like (the random ID generated by Laravel is 32 characters, I think it's a good idea to keep this length).

Attempts should be set to 1 when initially pushing the job. If Laravel can't process the job, it will push it back onto the queue and increase the attempts value.

Share Improve this question edited May 23, 2017 at 10:29 CommunityBot 11 silver badge asked Feb 26, 2017 at 6:26 Jan SchuermannJan Schuermann 4191 gold badge6 silver badges18 bronze badges 3
  • 1 If my latest update doesn't solve your issues and you still need help debugging the queue (which probably involves adding debugging statements into the vendor/laravel/framework folder), try asking either on IRC ( laravel.io/chat ) or Slack ( larachat.co ). – sisve Commented Feb 26, 2017 at 8:55
  • The downvote is for what? Resolution of a problem that has no answer anywhere else? – Jan Schuermann Commented Feb 27, 2017 at 10:00
  • I upvoted your question Jan - I'm very glad I'm not the only one trying to do this :) – alexkb Commented Feb 7, 2019 at 2:04
Add a ment  | 

4 Answers 4

Reset to default 3

First, note that this is the format of the jobs in the database-based queue in Laravel 5.3. Newer versions of Laravel contains changes.

The payload column should contain a json object of the following format. The job (...\\CallQueuedHandler@call) can be hard-coded in this scenario. I believe the mandName key is for display-purposes only. The mand key, however, is the harder part, it should be a valid object that unserialize() supports. It looks like there are packages available on npm for this purpose, a quick search turned up php-serialization.

{
    "job": "Illuminate\\Queue\\CallQueuedHandler@call",
    "data": {
        "mandName": "App\\Jobs\\MyJobClass",
        "mand": "O:19:\"App\\Jobs\\MyJobClass\"... /* stuff */"
    }
}

The json payload you've provided results in the following object. Both the job and the data keys are important.

{
  "job": "Illuminate\\Broadcasting\\BroadcastEvent",
  "data": {
    "event": "O:28:\"App\\Events\\NotificationEvent\":5:{s:7:\"\u0000*\u0000name\";s:12:\"notification\";s:4:\"data\";a:4:{s:4:\"testkey\";s:14:\"testval\";s:9:\"timestamp\";s:19:\"2017-02-24 11:07:48\";s:5:\"event\";s:12:\"notification\";s:5:\"class\";s:28:\"App\\Events\\NotificationEvent\";}s:10:\"\u0000*\u0000channel\";N;s:7:\"\u0000*\u0000user\";O:45:\"Illuminate\\Contracts\\Database\\ModelIdentifier\":2:{s:5:\"class\";s:8:\"App\\User\";s:2:\"id\";i:2;}s:6:\"socket\";N;}"
  },
  "id": "XuUKRTf8CTSdzaVgp2gRcvmxQqLcpBUG",
  "attempts": 1
}

The problematic part, I presume, is the serialized object. Reformatted in a way that is easier to read (but totally breaks it) ...

O:28:"App\Events\NotificationEvent":5:{
    // protected $name = 'notification'
    s:7:" * name";s:12:"notification";

    // public $data = array(...)
    s:4:"data";a:4:{
        // 'testkey => 'testval'
        s:4:"testkey";s:14:"testval";

        // 'timestamp' => '2017-02-24 11:07:48';
        s:9:"timestamp";s:19:"2017-02-24 11:07:48";

        // 'event' => 'notification';
        s:5:"event";s:12:"notification";

        // 'class' => App\Events\NotificationEvent::class;
        s:5:"class";s:28:"App\Events\NotificationEvent";
    }

    // protected $channel = null;
    s:10:"\0*\0channel";N;

    // protected $user = (instance of ModelIdentifier)
    s:7:"\0*\0user";O:45:"Illuminate\Contracts\Database\ModelIdentifier":2:{
        // public $class = App\User::class;
        s:5:"class";s:8:"App\User";

        // public $id = 2;
        s:2:"id";i:2;
    }

    // public $socket = null;
    s:6:"socket";N;
}

This format exposes the fact that your job uses the SerializesModels trait that replaces references to models as a simple entry containing class+identifier, and will restore them during __wakeup.

I believe your issue is with the mental parsing of the json and the serialize format; your guessed structure is ... wrong.

The next steps would not be to guess anything. 1. Duplicate this exact test notification you already have the payload for. Just copy-paste it. (You may need to change the id, I guess it is used for deduplication.) 2. Build the event data using php-serialization and aim to build something identical to the original event payload. No changes at all. 3. If it works this far, feel free to change the serialized event data to see what happens.

For people that need to go with a "classic" job, here is the code, based on Jan Schuermanns original question and solution:

const php = require('php-serialization');

var job_class = 'App\\Jobs\\TestJob';
var data = {}

job = new php.Class(job_class);

job.__addAttr__("data","string", JSON.stringify(data),"string","private");

result = {
    id: 1,
    type: 'job',
    displayName: job_class,
    job: "Illuminate\\Queue\\CallQueuedHandler@call",
    maxTries: null,
    delay: null,
    timeout: null,
    timeoutAt: null,
    data: {
        mandName: job_class,
        mand: php.serialize(job, "object")
    }
};

sample job class:

<?php

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;

class TestJob implements ShouldQueue
{
    private $data;

    public function __construct($data)
    {
        $this->data = $data;
    }

    public function handle() {
        var_dump(json_decode($this->data));
        return;
    }
}

leads to:

[2019-08-17 ...] Processing: App\Jobs\TestJob
object(stdClass)#1870 (0) {
}
[2019-08-17 ...] Processed:  App\Jobs\TestJob

EDIT: If you have to deal with umlauts or other special characters, you might need to use latin1 with iconv-lite:

iconv.decode(Buffer.from(JSON.stringify(data)), 'latin1')

in

job.__addAttr__("data","string", JSON.stringify(data),"string","private");

...
queueChannel.sendToQueue(queuePublish, Buffer.from(JSON.stringify(result), 'latin1'));

This solution is fairly dirty and might not work in the future, but...

You can replicate "the old way" of sending jobs to queue:

\Queue::push('\Namespace\Of\Class@handlerMethod', ['foo' => 'bar']);

and in the handler:

public function handlerMethods($job, $data)
{
    $job->delete;
    // do whatever with data
}

From Node.js you would do something like:

SQS.sendMessage({
    MessageBody: JSON.stringify({
        job: '\Namespace\Of\Class@handlerMethod',
        data: {
            foo: 'bar',
        }
    }),
    QueueUrl: yourQueueUrl
 }, err => { /* handle */ });

Drawbacks

  • you have to manually delete the job

  • I'm not sure dead letter queue would work with this approach (it might)

  • cannot serialize models this way - if you need to get a model out of data at the handler, need to pass primary key and retrieve it that way

  • haven't tried this with an actual job and my handler class was fairly "utilitarian", but chances are, it might not work with actual laravel jobs

  • this approach es from Laravel 4, so this might be already deprecated and even if not, might be removed / changed in the future

Do this at your own risk, but here's proof it's possible.

simply expose an API where the job processor is written. If we are talking about the node js and laravel. An API needs to be exposed which will accept the HTTP request and place the job and return to node js server.

Later on, if you have to execute the job with a different server which has different language installed the solution given by others will not work and will be restricted to a specific framework.

本文标签: javascriptPush to Laravel queue from outside Laravel (NodeJS)Stack Overflow