/
Task Queuing System [5.3.0-B1]

Task Queuing System [5.3.0-B1]

Owned by Alex
Last updated: Apr 01, 2017

Tasks in general can be of 2 types:

  • interactive - a progress bar is displayed showing how much % of task is done
  • non-interactive - nothing is shown to user
  • user triggered - user presses a button (or similar control) to trigger a task
  • system triggered - tasks are created on their own

Here is the matrix of what's currently supported by In-Portal:

 InteractiveNon-Interactive
User Triggered
  • all import scripts (e.g. product import in catalog)
  • rebuilding of category permissions cache
  • template parser template recompilation
  • e-mail queue
System Triggered
  • not supported
  • all scheduled tasks

Thoughts:

  • interactive tasks are blocking UI and therefore user can't do something else in Admin Console while they are running
  • user don't create about real-time status update of interactive tasks in most (but not all) cases, but just wants to know when they're completed
  • scheduled tasks (system triggered non-interactive tasks):
    • provide limited insight about task execution status
    • only retain last task execution status
  • e-mail queue:
    • successfully executed tasks are removed from queue (the "E-mail Logs" section sort of compensates for that currently)
    • no error information is stored within task

Solution

Part 1 (db tables & units) - 4.5h (sum)

  1. create "ITaskHandler" interface with "handleTaskRun(kDBItem $task_run)" method - 0.5h
  2. create "TaskQueue" database table (unit name "task-queue") with following columns: - 1h
    • Id
    • TaskHandlerClass - FQCN of PHP class, which is responsible for processing this task queue record (must implement "ITaskHandler" interface)
    • TaskData - JSON-encoded data, that is needed for task execution (e.g. e-mail recipient, IDs of records to be processed)
    • ScheduledOn - when task needs to be processed (set at queuing time)
    • QueuedOn - when task was queued
    • QueuedById - who queued the task
    • LastStatus - status from last attempt of this queue record processing (same options as for "TaskRuns.Status" field)
    • MaxRetries - if task fails specified number of times (5 by default), then don't retry it
    • TaskRunsFailed - number failed task runs (number is reset, when task execution was successful)
  3. create "TaskRuns" database table (unit name: "task-run") with following columns: - 1h
    • Id
    • TaskQueueId - ID of record from "TaskQueue" table, that is responsible for creation this run
    • StartedOn - when task run was started; NULL set to moment, when status changes from "scheduled" to "processing"
    • PercentsCompleted - "0" by default, but will be updated as task is being processed
    • FinishedOn - when task was finished executing (regardless of status)
    • Results - JSON-encoded results in any format, that be later displayed in human-readable form
    • Status - same statuses as for "TaskQueue.LastStatus" column
      • "scheduled" - initially, when record is created;
      • "running" - when somebody is processing the record;
      • "success" - when execution finished without errors;
      • "error" - when known error happened during processing;
      • "timeout" - when associated task runner died unexpectedly
    • StandardOutput - what was written to "stdout" stream during this task run
    • ErrorOutput - what was written to "stderr" stream during this task run
    • ErrorCode - non-empty when known error happened
    • ErrorMessage - non-empty when known error happened
    • TaskRunnerId - NULL by default; ID of task runner that processing/processed given task run
  4. create "TaskRunners" database table (unit name: "task-runner") with following columns: - 1h
    1. Id
    2. ProcessId - the PID of process, that started/created task runner
    3. StartedOn - when process was started
    4. FinishedOn - when process was finished; NULL initially
    5. Status - status of task runner:
      1. "running" - default; means task runner is running
      2. "success" - set, when task runner decides to kill itself
      3. "timeout" - set by overseer when task runner in "running" status and associated process isn't running
  5. in "task-run:OnAfterItemUpdate" event will: - 0.5h
    1. load "task-queue" object associated with updated task run
    2. get all "task-run" records for that "task-queue" (via sql); then sort them from recent to old (via php)
    3. set following fields on "task-queue" object:
      1. "LastStatus" to "Status" of most recent "task-run"
      2. "TaskRunsFailed" to count of "task-run" records in "error" and "timeout" statuses (if last run is failed)
      3. "TaskRunsFailed" to "0" (if last run was successful)
  6. in "task-runner::OnBeforeItemCreate" event set "ProcessId" to PID of current process - 0.1h
  7. in "task-runner:OnAfterItemUpdate" event, when "Status" changes from "running" to "timeout" set all "task-run" status, that are processed by this task runner from "running" to "timeout" as well - 0.3h
  8. the "task-run" would be sub-item of "task-queue" unit - 0.1h

Part 2 (adding tasks & runs) - 5h (sum)

  1. create "TaskQueue" class (not item of "task-queue" unit) - 0.1h
  2. add protected "TaskQueue::createTaskHandler($class_name)" method, that will: - 0.5h
    1. create instance of given class or throw an exception when failed
    2. if created object doesn't implement "ITaskHandler" interface, then throw an exception
    3. return the object
  3. add public "TaskQueue::addTask($task_handler_class, array $task_data, $scheduled_on, $max_retries = null)" method, that will: - 0.4h
    1. call "TaskQueue::createTaskHandler" method to verify, that class given in "$task_handler_class" parameter is valid
    2. consider "$max_retries" as "5" when not given
    3. create new db record using provided data using "task-queue" object
  4. add public "TaskQueue::refreshTaskRunnersStatus()" method that will: - 0.5h
    1. get all "task-runner" in "running" status
    2. if associated process isn't running anymore, then set "task-runner" status from "running" to "timeout" (the "task-runner::OnAfterItemUpdate" would update connected task runs)
  5. add protected "TaskQueue::createTaskRun(kDBItem $task_queue)" method, that will: - 0.5h
    1. create new task run (and return it's ID) for given queue record, when all of following rules aren't violated:
      1. only 1 (or less) running "task-run" can exist for one "task-queue" record
      2. "TaskRunsFailed" must be smaller, then "MaxRetries" on associated "task-queue" record
    2. return "null" otherwise
  6. add protected "TaskQueue::createMissingTaskRuns()" method, that will: - 0.5h
    1. get all records from "TaskQueue" table, for which task runs can be created:
      1. ScheduledOn < NOW()
      2. LastStatus is not "running"
      3. TaskRunsFailed must be smaller, then MaxRetries
    2. call the "TaskQueue::createTaskRun" method on each of them (method can return "NULL" in some cases, but that's ok)
  7. add protected "TaskQueue::getTaskRunnerCount()" method, that will return number of "task-runner" in "running" status - 0.3h
  8. add protected "TaskQueue::getMissingTaskRunnerCount()" method, that will: - 0.2h
    1. get value of  "TaskRunnerLimit" system setting
    2. call "TaskQueue::getTaskRunnerCount" method
    3. return difference or 0, when difference is negative
  9. add public "TaskQueue::runStandalone()", that will: - 0.5h
    1. call "TaskQueue::getTaskRunner" method
    2. if object is returned call "->process()" method on it
  10. add public "TaskQueue::createMissingTaskRunners()" method, that will: - 0.5h
    1. if in CLI:
      1. call "TaskQueue::getMissingTaskRunnerCount" method
      2. if it returned "0" do nothing
      3. execute command (see last plan) X number of times in background processes (X - number returned above)
    2. if not in CLI:
      1. call "TaskQueue::runStandalone" method
  11. add public "TaskQueue::processQueue()" method, that will: - 0.4h
    1. call "TaskQueue::refreshTaskRunnersStatus" method
    2. call "TaskQueue::createMissingTaskRuns" method
  12. create "task-queue:OnProcess" event, that: - 0.1h
    1. would be called as Scheduled Task on a regular basis (e.g. each 5 minutes)
    2. would call "TaskQueue::processQueue" method
  13. create "task-queue:OnCreateTaskRunners" event, that: - 0.3h
    1. would be called as Scheduled Task on a regular basis (e.g. each 5 minutes) - can be disabled if needed
    2. will call "TaskQueue::createMissingTaskRunners" method
  14. create "task-queue:OnDebug" event, that will call "TaskQueue::runStandalone" method - 0.2h

Part 3 (running runs) - 4.5h (sum)

  1. add "declare(ticks = 1);" on top of "/tools/run_event.php" file - 0.1h
  2. create new "TaskRunnerLimit" system setting set to "8" by default - 0.2h
  3. create "TaskRunner" class with: - 0.2h
    1. add "TaskRunner::taskRunnerId" property
    2. add "TaskRunner::lastSignal" property
  4. add protected "TaskRunner::signalHandler" method, that will store received signal in the "TaskRunner::lastSignal" property - 0.5h
  5. add "TaskRunner::__construct($task_runner_id)" method, that will: - 0.5h
    1. store given "$task_runner_id" into "TaskRunner:taskRunnerId" property
    2. if executed from CLI (PHP_SAPI constant check), then use "pcntl_signal" function to register "TaskRunner::signalHandler" method as signal listeners for following signals:
      1. SIGINT
      2. SIGTERM
      3. SIGKILL
      4. SIGHUP
  6. add public "TaskQueue::getTaskRunner()" method, that will: - 0.5h
    1. call "TaskQueue::getMissingTaskRunnerCount" method
    2. if method returned "0", then return "null"
    3. create new "task-runner" object
    4. return instance of "TaskRunner" class initialized with ID of just created task runner
  7. add protected "TaskRunner::getNextTaskRunId()" method, that will: - 0.5h
    1. acquire WRITE lock "TaskRuns" database table (solves racing condition in parallel environment)
    2. pick 1st available "task-run" in "scheduled" status (FIFO logic)
    3. release above acquired lock
    4. return found task run id or "null" when nothing was found
  8. add protected "TaskRunner::processTaskRun($task_run_id)", that will: - 1h
    1. load "task-run" by given ID from the database or throw an exception if wasn't found
    2. if given "task-run" isn't in "scheduled" status, then throw an exception
    3. set following fields and save changes to db immediately:
      1. "TaskRunnerId"  to value of "TaskRunner::taskRunnerId" property
      2. "Status" to "processing"
    4. create task handler by calling "TaskQueue::createTaskHandler" method
    5. enable redirection of "stdout" and "stderr" into temp files
    6. call the "handleTaskRun" (wrapped within try/catch block) on that object providing task run object (was loaded above) as an argument
    7. store contents of above temp files into "StandardOutput" and "ErrorOutput" fields of "task-run" object
    8. the above method can update given object fields at will and save to db (e.g. "PercentsCompleted" and "Results")
    9. when exception was caught, then:
      1. set "Status" to "error"
      2. set "ErrorCode" to exception code
      3. set "ErrorMessage" to exception message
    10. when no exception was caught, then:
      1. set "Status" to "success"
      2. set "ErrorCode" and "ErrorMessage" to empty value
    11. set "FinishedOn" to time, when task was finished (with error or not)
    1. save changes to db
  9. add public "TaskRunner::process()" method, that will consist of while loop, where each iteration will: - 0.5h
    1. call "TaskRunner::getNextTaskRunId" method
    2. call "TaskRunner::processTaskRun" with ID found above (if ID was found)
    3. in either of following cases set "FinishedOn" to NOW() on associated "task-runner" record and exit
      1. "TaskRunner::lastSignal" is set
      2. overall memory consumption is more than 100MB
      3. it's not CLI mode
    4. sleep for X of seconds
  10. add "task-runner:OnProcess" event, that will: - 0.5h
    1. call the "TaskQueue::getTaskRunner" method
    2. if an object is returned, then call "->process()" method on it

Part 4 (rotation + UI) - 3h (sum)

  1. in the "Configuration > Website > Scheduled Tasks" section: - 1.5h
    1. show "Scheduled Tasks" tab (would represent existing grid)
    2. add "Task Queue" tab with list of records from "task-queue" unit sorted by "ScheduledOn DESC"
    3. the "task-queue" record editing window will consist of 2 tabs: 
      1. General - all "task-queue" fields
      2. Task Runs - grid of associated task runs; rows in "error" or "timeout" status would have red background
    4. add "Task Runners" tab with list of records from "task-runner" unit sorted by "StartedOn DESC"; rows in "timeout" status would have red background
  2. create "TaskQueueRotationInterval" setting (same configuration as for e-mail logs) - 0.5h
  3. create "task-queue:OnRotate" event (scheduled task), that would delete old (same concert as for e-mail logs) successful "task-queue" records along with their "task-run" records and long ago finished "task-runner" records - 1h

Part 5 (usage)

  1. configure either of following, but not as scheduled task, because it will block all other scheduled tasks:

    • command: /usr/bin/env php /path/to/in-portal/tools/run_event.php task-runner:OnProcess password_here

    • setup "upstart" or "supervisord" or any other tool to ensure presence of X processes powered by above command

    • add X records to "crontab" file powered by above command

  2. task can be created through calling "TaskQueue::addTask" method by whoever needs it, e.g.:

    • user presses a button
    • scheduled task decides to offload some work
    • etc.

Quote: 17h*1.4=24h

Related Discussions

Related Tasks