Versions Compared

Version Old Version 11 New Version Current
Changes made by

Alex

Alex

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Here is the matrix of what's currently supported by In-Portal:

 InteractiveNon-Interactive
User Triggered
  • all import scripts (e.g. product import in catalog)
  • rebuilding of category permissions cache
  • template parser template recompilation
  • e-mail queue
System Triggered
  • not supported
  • all scheduled tasks

Thoughts:

  • interactive tasks are blocking UI and therefore user can't do something else in Admin Console while they are running
  • user don't create about real-time status update of interactive tasks in most (but not all) cases, but just wants to know when they're completed
  • scheduled tasks (system triggered non-interactive tasks):
    • provide limited insight about task execution status
    • only retain last task execution status
  • e-mail queue:
    • successfully executed tasks are removed from queue (the "E-mail Logs" section sort of compensates for that currently)
    • no error information is stored within task

...

  1. create "ITaskHandler" interface with "handleTaskRun(kDBItem $task_run)" method - 0.5h
  2. create "TaskQueue" database table (unit name "task-queue") with following columns: - 1h
    • Id
    • TaskHandlerClass - FQCN of PHP class, which is responsible for processing this task queue record (must implement "ITaskHandler" interface)
    • TaskData - JSON-encoded data, that is needed for task execution (e.g. e-mail recipient, IDs of records to be processed)
    • ScheduledOn - when task needs to be processed (set at queuing time)
    • QueuedOn - when task was queued
    • QueuedById - who queued the task
    • LastStatus - status from last attempt of this queue record processing (same options as for "TaskRuns.Status" field)
    • MaxRetries - if task fails specified number of times (5 by default), then don't retry it
    • TaskRunsFailed - number failed task runs (number is reset, when task execution was successful)
  3. create "TaskRuns" database table (unit name: "task-run") with following columns: - 1h
    • Id
    • TaskQueueId - ID of record from "TaskQueue" table, that is responsible for creation this run
    • StartedOn - when task run was started; NULL set to moment, when status changes from "scheduled" to "processing"
    • PercentsCompleted - "0" by default, but will be updated as task is being processed
    • FinishedOn - when task was finished executing (regardless of status)
    • Results - JSON-encoded results in any format, that be later displayed in human-readable form
    • Status - same statuses as for "TaskQueue.LastStatus" column
      • "scheduled" - initially, when record is created;
      • "running" - when somebody is processing the record;
      • "success" - when execution finished without errors;
      • "error" - when known error happened during processing;
      • "timeout" - when associated task runner died unexpectedly
    • StandardOutput - what was written to "stdout" stream during this task run
    • ErrorOutput - what was written to "stderr" stream during this task run
    • ErrorCode - non-empty when known error happened
    • ErrorMessage - non-empty when known error happened
    • TaskRunnerId - NULL by default; ID of task runner that processing/processed given task run
  4. create "TaskRunners" database table (unit name: "task-runner") with following columns: - 1h
    1. Id
    2. ProcessId - the PID of process, that started/created task runner
    3. StartedOn - when process was started
    4. FinishedOn - when process was finished; NULL initially
    5. Status - status of task runner:
      1. "running" - default; means task runner is running
      2. "success" - set, when task runner decides to kill itself
      3. "timeout" - set by overseer when task runner in "running" status and associated process isn't running
  5. in "task-run:OnAfterItemUpdate" event will: - 0.5h
    1. load "task-queue" object associated with updated task run
    2. get all "task-run" records for that "task-queue" (via sql); then sort them from recent to old (via php)
    3. set following fields on "task-queue" object:
      1. "LastStatus" to "Status" of most recent "task-run"
      2. "TaskRunsFailed" to count of "task-run" records in "error" and "timeout" statuses (if last run is failed)
      3. "TaskRunsFailed" to "0" (if last run was successful)
  6. in "task-runner::OnBeforeItemCreate" event set "ProcessId" to PID of current process - 0.1h
  7. in "task-runner:OnAfterItemUpdate" event, when "Status" changes from "running" to "timeout" set all "task-run" status, that are processed by this task runner from "running" to "timeout" as well - 0.3h
  8. the "task-run" would be sub-item of "task-queue" unit - 0.1h

Part 2 (adding tasks & runs) -

...

5h (sum)

  1. create "TaskQueue" class (not item of "task-queue" unit) - 0.1h
  2. add protected "TaskQueue::createTaskHandler($class_name)" method, that will: - 0.5h
    1. create instance of given class or throw an exception when failed
    2. if created object doesn't implement "ITaskHandler" interface, then throw an exception
    3. return the object
  3. add public "TaskQueue::addTask($task_handler_class, array $task_data, $scheduled_on, $max_retries = null)" method, that will: - 0.4h
    1. call "TaskQueue::createTaskHandler" method to verify, that class given in "$task_handler_class" parameter is valid
    2. consider "$max_retries" as "5" when not given
    3. create new db record using provided data using "task-queue" object
  4. add public "TaskQueue::refreshTaskRunnersStatus()" method that will: - 0.5h
    1. get all "task-runner" in "running" status
    2. if associated process isn't running anymore, then set "task-runner" status from "running" to "timeout" (the "task-runner::OnAfterItemUpdate" would update connected task runs)
  5. add protected "TaskQueue::createTaskRun(kDBItem $task_queue)" method, that will: - 0.5h
    1. will create new task run (and return it's ID) for given queue record, when all of following rules aren't violated:
      1. only 1 (or less) running "task-run" can exist for one "task-queue" record
      2. "TaskRunsFailed" must be smaller, then "MaxRetries" on associated "task-queue" record
    2. return "null" otherwise
  6. add protected "TaskQueue::createMissingTaskRuns()" method, that will: - 0.5h
    1. get all records from "TaskQueue" table, for which task runs can be created:
      1. ScheduledOn < NOW()
      2. LastStatus is not "running"
      3. TaskRunsFailed must be smaller, then MaxRetries
    2. call the "TaskQueue::createTaskRun" method on each of them (method can return "NULL" in some cases, but that's ok)
  7. add protected "TaskQueue::getTaskRunnerCount()" method, that will return number of "task-runner" in "running" status - 0.3h
  8. add protected "TaskQueue::getMissingTaskRunnerCount()" method, that will: - 0.2h
    1. get value of  "TaskRunnerLimit" system setting
    2. call "TaskQueue::getTaskRunnerCount" method
    3. return difference or 0, when difference is negative
    add protected
  9. add public "TaskQueue::runStandalone()", that will: - 0.5h
    1. call "TaskQueue::getTaskRunner" method
    2. if object is returned call "->process()" method on it
  10. add public "TaskQueue::createMissingTaskRunners()" method, that will: - 1h0.5h
    1. if in CLI:
      1. call "TaskQueue::getMissingTaskRunnerCount" method
      2. if it returned "0" do nothing
      3. execute command (see last plan) X number of times in background processes (X - number returned above)
    2. if not in CLI:
      1. call "TaskQueue::getTaskRunner" methodif object is returned call "->process()runStandalone" method on it
  11. add public "TaskQueue::processQueue()" method, that will: - 0.4h
    1. call "TaskQueue::refreshTaskRunnersStatus" method
    2. call "TaskQueue::createMissingTaskRunnerscreateMissingTaskRuns" method (might not be the best idea if user wants to manage task runners externally)
  12. create "task-queue:OnProcess" event, that: - 0.1h
    1. would be called as Scheduled Task on a regular basis (e.g. each 5 minutes)
    2. would call "TaskQueue::createMissingTaskRunsprocessQueue" method
  13. create "task-queue:OnProcessOnCreateTaskRunners" event, that:  - 0.1h3h
    1. would be called as Scheduled Task on a regular basis (e.g. each each 5 minutes) would call - can be disabled if needed
    2. will call "TaskQueue::createMissingTaskRunners" method
  14. create "task-queue:OnDebug" event, that will call "TaskQueue::processQueuerunStandalone" method - 0.2h

Part 3 (running runs) - 4.5h (sum)

...

  1. configure either of following, but not as scheduled task, because it will block all other scheduled tasks:

    • command: /usr/bin/env php /path/to/in-portal/tools/run_event.php task-runner:OnProcess password_here

    • setup "upstart" or "supervisord" or any other tool to ensure presence of X processes powered by above command

    • add X records to "crontab" file powered by above command

  2. task can be created through calling "TaskQueue::addTask" method by whoever needs it, e.g.:

    • user presses a button
    • scheduled task decides to offload some work
    • etc.

Quote: 16.5h17h*1.4=23h24h

Related Discussions

...