Task Queuing System [5.3.0-B1]

Tasks in general can be of 2 types:

• interactive - a progress bar is displayed showing how much % of task is done
• non-interactive - nothing is shown to user
• user triggered - user presses a button (or similar control) to trigger a task
• system triggered - tasks are created on their own

Here is the matrix of what's currently supported by In-Portal:

Thoughts:

• interactive tasks are blocking UI and therefore user can't do something else in Admin Console while they are running
• user don't create about real-time status update of interactive tasks in most (but not all) cases, but just wants to know when they're completed
• scheduled tasks (system triggered non-interactive tasks):
  • provide limited insight about task execution status
  • only retain last task execution status
• e-mail queue:
  • successfully executed tasks are removed from queue (the "E-mail Logs" section sort of compensates for that currently)
  • no error information is stored within task

Solution

To solve this I'm proposing to:

1. create "TaskQueue" database table with following columns:
   • Id
   • QueuedOn - when task was queued
   • QueuedById - who queued the task
   • ScheduledOn - when task needs to be processed (set at queuing time)
   • TaskClass - which PHP class is responsible for processing this task queue record
   • TaskData - data, that is needed for task execution (e.g. e-mail recipient, IDs of records to be processed)
   • LastStatus - status from last attempt of this queue record processing - {scheduled (default), processing, success, failed}
   • MaxRetries - if task fails specified number of times (5 by default), then don't retry it
   • FailedRetries - failed retry count (number is reset, when task execution was successful)
2. create "TaskQueueDetails" database table with following columns:
   • Id
   • TaskQueueId - associated task queue record
   • StartedOn - when task was started executing
   • PercentsCompleted - 0 by default, but will be updated as task is being processed
   • FinishedOn - when task was finished executing
   • Status - {processing (default), success, error, timeout} - create record right before execution start; error - when known error happened during processing; timeout - when status wasn't created within 1 day
   • ErrorCode - non-empty when known error happened
   • ErrorMessage - non-empty when known error happened
3. probably there won't be any UI for this functionality, because it's too general to be usable by user, but specialized sections (e.g. "E-mail Queue") can read data from these tables to keep user updated
4. task can be created by whoever needs it, e.g. user presses a button, scheduled task decides to offload some work, etc.
5. a scheduled task needs to be created to process records from "TaskQueue" table:
   1. pick records with "Status = scheduled" and "ScheduledOn < NOW()"
   2. create class instance (from "TaskClass" field) responsible for task processing
   3. if class isn't found, then:
      1. create record in "TaskQueueDetails" table with "error" status and error message set to "Task class not found"
      2. set "FailedRetries" to "MaxRetries"
   4. if class found:
      1. create record in "TaskQueueDetails" table with "processing" status
      2. let class to process task
      3. update record in "TaskQueueDetails" table by setting "success" status (if no errors happened), "error" status (and ErrorCode/ErrorMessage fields) when error happened
6. the "task-queue-detail" unit would automatically update associated "task-queue" record to indicate last status and such fields
7. delete data about successfully executed tasks after 1 month (configurable as log rotation) since task was executed

Related Discussions

• [scheduled tasks] Scheduled task handling architectural changes [5.3.0-B1]

Related Tasks