Connecting Laravel Queues with Python for Task Processing

Websolutionstuff | Nov-12-2024 | Categories : Laravel Python

In this guide, I’ll show you how to connect Laravel queues with Python to handle resource-heavy tasks such as data analysis or image processing. This setup leverages Laravel’s queue system to trigger Python scripts, enabling you to offload complex tasks from your main application efficiently.

We'll walk through setting up a queue in Laravel, writing a job to call a Python script, and executing tasks asynchronously.

Connecting Laravel Queues with Python for Task Processing

Step 1: Set Up the Laravel Queue System

Ensure that your Laravel app is configured to use a queue driver, like Redis or database, which will manage your tasks.

In env file

QUEUE_CONNECTION=database

Then run the migration to create the queue table.

php artisan queue:table
php artisan migrate

Step 2: Create a Laravel Job

Create a Laravel job to process tasks in the queue. This job will call a Python script for tasks such as image processing or data analysis.

php artisan make:job ProcessPythonTask

In the ProcessPythonTask job, configure the job to call a Python script. Open app/Jobs/ProcessPythonTask.php and update it as follows.

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Symfony\Component\Process\Process;
use Symfony\Component\Process\Exception\ProcessFailedException;

class ProcessPythonTask implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $data;

    public function __construct($data)
    {
        $this->data = $data;
    }

    public function handle()
    {
        $process = new Process(['python3', base_path('scripts/process_data.py'), json_encode($this->data)]);
        $process->run();

        if (!$process->isSuccessful()) {
            throw new ProcessFailedException($process);
        }

        \Log::info('Python script output: ' . $process->getOutput());
    }
}

In this example, the job sends data to the Python script process_data.py located in the scripts directory.

Step 3: Create the Python Script

In the root directory of your Laravel app, create a scripts folder. Inside it, create process_data.py, where you can add code for tasks like data analysis.

# scripts/process_data.py

import sys
import json

def main(data):
    # Simulate a data processing task
    result = sum(data)  # Example: Sum up numbers in data
    print(f"Processed Result: {result}")

if __name__ == "__main__":
    # Read data from Laravel
    data = json.loads(sys.argv[1])
    main(data)

This script receives data from Laravel and processes it. The example here sums up numbers but can be replaced with any processing logic, such as analyzing data or processing images.

Step 4: Dispatch the Job in Laravel

In your Laravel controller, dispatch the ProcessPythonTask job with relevant data.

use App\Jobs\ProcessPythonTask;

public function processData()
{
    $data = [1, 2, 3, 4, 5]; // Example data to be processed by Python
    ProcessPythonTask::dispatch($data);

    return response()->json(['message' => 'Task is being processed.']);
}

This will push the job to the queue, where it will be picked up by the queue worker and processed by the Python script.

Step 5: Run the Queue Worker

Start the Laravel queue worker to process jobs in the queue.