Versions Compared

Version Old Version 4 New Version Current
Changes made by

Alex

Alex

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Here is the matrix of what's currently supported by In-Portal:

 InteractiveNon-Interactive
User Triggered
  • all import scripts (e.g. product import in catalog)
  • rebuilding of category permissions cache
  • template parser template recompilation
  • e-mail queue
System Triggered
  • not supported
  • all scheduled tasks

Thoughts:

  • interactive tasks are blocking UI and therefore user can't do something else in Admin Console while they are running
  • user don't create about real-time status update of interactive tasks in most (but not all) cases, but just wants to know when they're completed
  • scheduled tasks (system triggered non-interactive tasks):
    • provide limited insight about task execution status
    • only retain last task execution status
  • e-mail queue:
    • successfully executed tasks are removed from queue (the "E-mail Logs" section sort of compensates for that currently)
    • no error information is stored within task

Solution

Part 1 (db tables & units) - 4.5h (sum)

  1. create "TaskQueueRunnerITaskHandler" interface with "processTaskQueueRunhandleTaskRun(kDBItem $task_queue_run)" method - 0.5h
  2. create "TaskQueue" database table (unit name "task-queue") with following columns: - 1h
    • Id
    • QueuedOn - when task was queued
    • QueuedById - who queued the task
    • ScheduledOn - when task needs to be processed (set at queuing time)
    • TaskClass - TaskHandlerClass - FQCN of PHP class, which is responsible for processing this task queue record (must implement "TaskQueueRunnerITaskHandler" interface)
    • TaskData - JSON-encoded data, that is needed for task execution (e.g. e-mail recipient, IDs of records to be processed)
    • ScheduledOn - when task needs to be processed (set at queuing time)
    • QueuedOn - when task was queued
    • QueuedById - who queued the task
    • LastStatus - status from last attempt of this queue record processing - {scheduled (default), processing, success, failed, timeout}; when queue processed in parallel, then last ended task status(same options as for "TaskRuns.Status" field)
    • MaxRetries - if task fails specified number of times (5 by default), then don't retry it
    • FailedRetries TaskRunsFailed - number failed retry count task runs (number is reset, when task execution was successful)
  3. create "TaskQueueRunsTaskRuns" database table with table (unit name: "task-run") with following columns: - 1h
    • Id
    • TaskQueueId - associated task queue recordID of record from "TaskQueue" table, that is responsible for creation this run
    • StartedOn - when task run was started executing; NULL set to moment, when status changes from "scheduled" to "processing"
    • PercentsCompleted - "0" by default, but will be updated as task is being processed
    • FinishedOn - when task was finished executing (regardless of status)
    • Results - JSON-encoded results in any format, that be later displayed in human-readable form
    • Status - same statuses as for "TaskQueue.LastStatus" column
      • "scheduled" - initially, when record is created;
      • "processingrunning" - when somebody is processing the record;
      • "success" - when execution finished without errors;
      • "error" - when known error happened during processing;
      • "timeout" - when status wasn't updated within 1 day (configurable per-queue record or system-wide, but can't be empty)associated task runner died unexpectedly
    • StandardOutput - what was written to "stdout" stream during this task run
    • ErrorOutput - what was written to "stderr" stream during this task run
    • ErrorCode - non-empty when known error happened
    • ErrorMessage - non-empty when known error happened
    • TaskRunnerId - NULL by default; ID of task runner that processing/processed given task run
  4. create "TaskRunners" database table (unit name: "task-runner") with following columns: - 1h
    1. Id
    2. ProcessId - the PID of
    the
    1. process, that
    is handling (or was handling in past) this task (process that starts running this task run will put own PID in here)
  5. create units (the "task-queue" and "task-queue-run"), that corresponding to above described database tables
  6. in "TaskQueueRunEventHandler::OnAfterItemUpdate" aggregate totals from all runs from associated task queue record and update it (task queue record)
    1. started/created task runner
    2. StartedOn - when process was started
    3. FinishedOn - when process was finished; NULL initially
    4. Status - status of task runner:
      1. "running" - default; means task runner is running
      2. "success" - set, when task runner decides to kill itself
      3. "timeout" - set by overseer when task runner in "running" status and associated process isn't running
  7. in "task-run:OnAfterItemUpdate" event will: - 0.5h
    1. load "task-queue" object associated with updated task run
    2. get all "task-run" records for that "task-queue" (via sql); then sort them from recent to old (via php)
    3. set following fields on "task-queue" object:
      1. "LastStatus" to "Status" of most recent "task-run"
      2. "TaskRunsFailed" to count of "task-run" records in "error" and "timeout" statuses (if last run is failed)
      3. "TaskRunsFailed" to "0" (if last run was successful)
  8. in "task-runner::OnBeforeItemCreate" event set "ProcessId" to PID of current process - 0.1h
  9. in "task-runner:OnAfterItemUpdate" event, when "Status" changes from "running" to "timeout" set all "task-run" status, that are processed by this task runner from "running" to "timeout" as well - 0.3h
  10. the "task-run" would be sub-item of "task-queue" unit - 0.1h

Part 2 (adding tasks & runs) - 5h (sum)

  1. create new "ParallelTaskQueueRunCount" system setting set to "8" by default
  2. create "TaskQueueHelper" class
  3. add public "TaskQueueHelper::queueTask($task_class, "TaskQueue" class (not item of "task-queue" unit) - 0.1h
  4. add protected "TaskQueue::createTaskHandler($class_name)" method, that will: - 0.5h
    1. create instance of given class or throw an exception when failed
    2. if created object doesn't implement "ITaskHandler" interface, then throw an exception
    3. return the object
  5. add public "TaskQueue::addTask($task_handler_class, array $task_data, $scheduled_on, $max_retries = null)" method, that will:create task queue record with given settings - 0.4h
    1. call "TaskQueue::createTaskHandler" method to verify, that class given in "$task_handler_class" parameter is valid
    2. consider "$max_retries" as "5" when not given
    3. throw an exception, when specified "$task_class" doesn't exist
    add protected "TaskQueueHelper::synchronizeTaskRunStatus
    1. create new db record using provided data using "task-queue" object
  6. add public "TaskQueue::refreshTaskRunnersStatus()" method that will: - 0.5h
    1. get all task runs, that are running currently
    2. get status of their PIDs
    3. for all task runs which PIDs are dead set their status to "timeout"
    add protected "TaskQueueHelper
    1. "task-runner" in "running" status
    2. if associated process isn't running anymore, then set "task-runner" status from "running" to "timeout" (the "task-runner::OnAfterItemUpdate" would update connected task runs)
  7. add protected "TaskQueue::createTaskRun(kDBItem $task_queue)" method, that will: - 0.5h
    1. will create new task run (and return it's ID) for given queue record, when all of following rules aren't violated:
      1. only 1 active (status = processing) task run can exist at same time (for a given queue record)sequential failed task run count (both "error" and "timeout" statuses are considered as failed) can't be more than max allowed retry countor less) running "task-run" can exist for one "task-queue" record
      2. "TaskRunsFailed" must be smaller, then "MaxRetries" on associated "task-queue" record
    2. return "null" otherwise
  8. add protected "TaskQueueHelperTaskQueue::createMissingTaskRuns()" method, that will: - 0.5h
    1. get value of "ParallelTaskQueueRunCount" system setting and compare it to number of currently running task runsif there are smaller number of tasks running, then possible:
      get all task queue recordsall records from "TaskQueue" table, for which task runs can be created (:
      1. ScheduledOn < NOW() +
      2. LastStatus is not running right now + failed retry count < max retry count)"running"
      3. TaskRunsFailed must be smaller, then MaxRetries
    2. call the "TaskQueueHelperTaskQueue::createTaskRun" method on each of them (method can return "NULL" in some cases, but that's ok)
  9. add protected "TaskQueue::getTaskRunnerCount()" method, that will return number of "task-runner" in "running" status - 0.3h
  10. add protected "TaskQueue::getMissingTaskRunnerCount()" method, that will: - 0.2h
    1. get value of  "TaskRunnerLimit" system setting
    2. call "TaskQueue::getTaskRunnerCount" method
    3. return difference or 0, when difference is negative
  11. add public "TaskQueue::runStandalone()", that will: - 0.5h
    1. call "TaskQueue::getTaskRunner" method
    2. if object is returned call "->process()" method on it
  12. add public "TaskQueueHelperTaskQueue::createTaskRunscreateMissingTaskRunners()" method, that will: - 0.5h
    1. if in CLI:
      1. call "
      TaskQueueHelper
      1. TaskQueue::
      synchronizeTaskRunStatus
      1. getMissingTaskRunnerCount" method
      2. if it returned "0" do nothing
      3. execute command (see last plan) X number of times in background processes (X - number returned above)
    2. if not in CLI:
      1. call "
      TaskQueueHelper
      1. TaskQueue::
      createMissingTaskRuns
      1. runStandalone" method
  13. add "TaskQueueEventHandler::OnCreateTaskRunspublic "TaskQueue::processQueue()" method, that will: - 0.4h
    1. call "TaskQueue::refreshTaskRunnersStatus" method
    2. call "TaskQueue::createMissingTaskRuns" method
  14. create "task-queue:OnProcess" event, that: - 0.1h
    1. would be called as Scheduled Task on a regular basis (e.g. each 5 minutes)
    2. would call "TaskQueue::processQueue" method
  15. create "task-queue:OnCreateTaskRunners" event, that: - 0.3h
    1. would be called as Scheduled Task on a regular basis (e.g. each 5 minutes) - can be disabled if needed
    2. will call "TaskQueue::createMissingTaskRunners" method
  16. create "task-queue:OnDebug" event, that will call "TaskQueue::runStandalone" method - 0.2h

Part 3 (running runs) - 4.5h (sum)

  1. add "declare(ticks = 1);" on top of "/tools/run_event.php" file - 0.1h
  2. create new "TaskRunnerLimit" system setting set to "8" by default - 0.2h
  3. create "TaskRunner" class with: - 0.2h
    1. add "TaskRunner::taskRunnerId" property
    2. add "TaskRunner::lastSignal" property
  4. add protected "TaskRunner::signalHandler" method, that will store received signal in the "TaskRunner::lastSignal" property - 0.5h
  5. add "TaskRunner::__construct($task_runner_id)" method, that will: - 0.5h
    1. store given "$task_runner_id" into "TaskRunner:taskRunnerId" property
    2. if executed from CLI (PHP_SAPI constant check), then use "pcntl_signal" function to register "TaskRunner::signalHandler" method as signal listeners for following signals:
      1. SIGINT
      2. SIGTERM
      3. SIGKILL
      4. SIGHUP
  6. add public "TaskQueueHelper"TaskQueue::getTaskRunner()" method, that will: - 0.5h
    1. call "TaskQueue::getMissingTaskRunnerCount" method
    2. if method returned "0", then return "null"
    3. create new "task-runner" object
    4. return instance of "TaskRunner" class initialized with ID of just created task runner
  7. add protected "TaskRunner::getNextTaskRunId()" method, that will: - 0.5h
    1. acquire WRITE lock "TaskRuns" database table (solves racing condition in parallel environment)
    2. pick 1st available "task-run" in "scheduled" status (FIFO logic)
    3. release above acquired lock
    4. return found task run id or "null" when nothing was found
  8. add protected "TaskRunner::processTaskRun($task_run_id)", that will: - 1h
    1. load "task-run" by given ID from the database (if failed or throw an exception if wasn't found
    2. if given "task-run" isn't in "scheduled" status, then throw an exception)
    3. set following fields and save changes to db immediately:
      1. "ProcessIdTaskRunnerId"  to current process idvalue of "TaskRunner::taskRunnerId" property
      2. "Status" to "processing"
    4. create instance of class from "TaskClass" field of associated task queue recordtask handler by calling "TaskQueue::createTaskHandler" method
    5. enable redirection of "stdout" and "stderr" into temp files
    6. call the "processTaskQueueRunhandleTaskRun" (wrapped within try/catch block) on that object providing task run object (was loaded above) as an argument
    7. store contents of above temp files into "StandardOutput" and "ErrorOutput" fields of "task-run" object
    8. the above method can update given object fields at will and save to db (e.g. "PercentsCompleted" and "Results")
    9. when exception was caught, then:
      1. set "Status" to "error"
      2. set "ErrorCode" to exception code
      3. set "ErrorMessage" to exception message
    10. when no exception was caught, then:
      1. set "Status" to "success"
      2. set "ErrorCode" and "ErrorMessage" to empty value
    11. set "FinishedOn" to time, when task was finished (with error or not)
    1. save changes to db
  9. add public "TaskQueueEventHandlerTaskRunner::OnProcessTaskRunprocess()" eventmethod, that will consist of while loop, where each iteration will:
    1. when executed from web context throw an exception
    2. acquire WRITE lock "TaskQueueRuns" database table (prevent 2 events executed at same time using same task run)
    3. pick 1st available run (FIFO logic)
    4. release above acquired lock
    5. if none found, then exit
    6. call "TaskQueueHelper::processTaskRun" method with found task run id

Part 4 (rotation)

  1.  - 0.5h
    1. call "TaskRunner::getNextTaskRunId" method
    2. call "TaskRunner::processTaskRun" with ID found above (if ID was found)
    3. in either of following cases set "FinishedOn" to NOW() on associated "task-runner" record and exit
      1. "TaskRunner::lastSignal" is set
      2. overall memory consumption is more than 100MB
      3. it's not CLI mode
    4. sleep for X of seconds
  2. add "task-runner:OnProcess" event, that will: - 0.5h
    1. call the "TaskQueue::getTaskRunner" method
    2. if an object is returned, then call "->process()" method on it

Part 4 (rotation + UI) - 3h (sum)

  1. in the "Configuration > Website > Scheduled Tasks" section: - 1.5h
    1. show "Scheduled Tasks" tab (would represent existing grid)
    2. add "Task Queue" tab with list of records from "task-queue" unit sorted by "ScheduledOn DESC"
    3. the "task-queue" record editing window will consist of 2 tabs: 
      1. General - all "task-queue" fields
      2. Task Runs - grid of associated task runs; rows in "error" or "timeout" status would have red background
    4. add "Task Runners" tab with list of records from "task-runner" unit sorted by "StartedOn DESC"; rows in "timeout" status would have red background
  2. create "TaskQueueRotationInterval" setting (same configuration as for e-mail logs) - 0.5h
  3. create "TaskQueueEventHandler:task-queue:OnRotate" event (scheduled task), that would delete old (same concert as for e-mail logs) successful "task-queue" records along with their runs"task-run" records and long ago finished "task-runner" records - 1h

Part 5 (usage)

  1. configure either of following, but not as scheduled task, because it will block all other scheduled tasks:

    • command: /usr/bin/env php /path/to/in-portal/tools/run_event.php task-queuerunner:OnProcessTaskRun OnProcess password_here

    • setup "upstart" or "supervisord" or any other tool to ensure presence of X processes powered by above command

    • add X records to "crontab" file powered by above command

  2. there won't be any built-in UI for this functionality, because it's too general to be usable by user, but specialized sections (e.g. "E-mail Queue") can read data from these tables to keep user informed

  3. task can be created through calling "

    TaskQueueHelper

    TaskQueue::

    queueTask

    addTask" method by whoever needs it, e.g.:

    • user presses a button
    • scheduled task decides to offload some work
    • etc.

Quote: 17h*1.4=24h

Related Discussions

...