From Submission to Queue

The following is an explanation of how tasks go from a user submission to queued on a grid.

Dataset Submission

The submit process takes a dataset config as input, usually through the website. It then creates the following entries:

• dataset
• config
• task_rel - one for each task

Note that if the IceProd instance you submit through does not know about a dataset dependency, submission will fail. Try submitting through the master instance in that case.

Buffering

Jobs and tasks get buffered, or allocated, when there is empty space on a queue that they can run at. Whole jobs are buffered at once, so more than the requested number of tasks may be buffered.

The following entries are made:

• search
• task
• job

Algorithm

• Get the possible datasets we can buffer from
  • Get number of jobs submitted for each dataset
  • Subtract off already buffered jobs
• Get task_rel for each dataset
• For each dataset, and potential job to buffer:
  • Generate task dependencies for each task
    • Get a task_rel for the current task
    • If missing a task_rel, load it
    • Figure out which task is referenced for each task_rel dependency
  • If a dependency is not met, do not buffer
  • Else, buffer the job and associated tasks

Queueing

Once a task is in waiting or idle, it can be queued (pending dependencies and requirements). See Lifecycles for the actual happenings from this point on.