NAME rq v2.1.0 SYNOPSIS rq (queue | export RQ_Q=q) mode [mode_args]* [options]* DESCRIPTION ruby queue (rq) is a tool used to create instant linux clusters by managing sqlite databases as nfs mounted priority work queues. multiple instances of rq running from multiples hosts can work from these queues to distribute processing load to n nodes - bringing many dozens of otherwise powerful cpus to their knees with a single blow. clearly this software should be kept out of the hands of free radicals, seti enthusiasts, and j. safran. the central concept of rq is that n nodes work in isolation to pull jobs from an central nfs mounted priority work queue in a synchronized fashion. the nodes have absolutely no knowledge of each other and all communication if done via the queue meaning that, so long as the queue is available via nfs and a single node is running jobs from it, the system will continue to process jobs. there is no centralized process whatsoever - all nodes work to take jobs from the queue and run them as fast as possible. this creates a system which load balances automatically and is robust in face of node failures. the first argument to any rq command is the name of the queue. this name may be omitted if, and only if, the environment variable RQ_Q has been set to contain the absolute path of target queue. rq operates in one of the modes create, submit, list, status, delete, update, query, execute, configure, snapshot, lock, backup, rotate, feed, or help. depending on the mode of operation and the options used the meaning of 'mode_args' may change. MODES the following mode abbreviations exist c => create s => submit l => list ls => list t => status d => delete rm => delete u => update q => query e => execute C => configure S => snapshot L => lock b => backup r => rotate f => feed h => help create, c : create a queue. the queue must be located on an nfs mounted file system visible from all nodes intended to run jobs from it. examples : 0) to create a queue ~ > rq /path/to/nfs/mounted/q create or simply ~ > rq /path/to/nfs/mounted/q c submit, s : submit jobs to a queue to be proccesed by a feeding node. any 'mode_args' are taken as the command to run. note that 'mode_args' are subject to shell expansion - if you don't understand what this means do not use this feature and pass jobs on stdin. when running in submit mode a file may by specified as a list of commands to run using the '--infile, -i' option. this file is taken to be a newline separated list of commands to submit, blank lines and comments (#) are allowed. if submitting a large number of jobs the input file method is MUCH, more efficient. if no commands are specified on the command line rq automatically reads them from STDIN. yaml formatted files are also allowed as input (http://www.yaml.org/) - note that the output of nearly all rq commands is valid yaml and may, therefore, be piped as input into the submit command. when submitting the '--priority, -p' option can be used here to determine the priority of jobs. priorities may be any whole number - zero is the default. note that submission of a high priority job will NOT supplant currently running low priority jobs, but higher priority jobs WILL always migrate above lower priority jobs in the queue in order that they be run as soon as possible. constant submission of high priority jobs may create a starvation situation whereby low priority jobs are never allowed to run. avoiding this situation is the responsibility of the user. the only guaruntee rq makes regarding job execution is that jobs are executed in an 'oldest highest priority' order and that running jobs are never supplanted. examples : 0) submit the job ls to run on some feeding host ~ > rq q s ls 1) submit the job ls to run on some feeding host, at priority 9 ~ > rq -p9 q s ls 2) submit 42000 jobs (quietly) from a command file, marking them as restartable should the node they are running on reboot. ~ > wc -l cmdfile 42000 ~ > rq q s --quiet --restartable < cmdfile 3) submit 42 priority 9 jobs from a command file. ~ > wc -l cmdfile 42 ~ > rq -p9 q s < cmdfile 4) submit 42 priority 9 jobs from a command file, marking them as 'important' using the '--tag, -t' option. ~ > wc -l cmdfile 42 ~ > rq -p9 -timportant q s < cmdfile 5) re-submit all the 'important' jobs (see 'query' section below) ~ > rq q query tag=important | rq q s 6) re-submit all jobs which are already finished (see 'list' section below) ~ > rq q l f | rq q s list, l, ls : list mode lists jobs of a certain state or job id. state may be one of pending, running, finished, dead, or all. any 'mode_args' that are numbers are taken to be job id's to list. states may be abbreviated to uniqueness, therefore the following shortcuts apply : p => pending r => running f => finished d => dead a => all examples : 0) show everything in q ~ > rq q list all or ~ > rq q l all or ~ > export RQ_Q=q ~ > rq l 1) show q's pending jobs ~ > rq q list pending 2) show q's running jobs ~ > rq q list running 3) show q's finished jobs ~ > rq q list finshed 4) show job id 42 ~ > rq q l 42 status, t : status mode shows the global state the queue. there are no 'mode_args'. the meaning of each state is as follows: pending => no feeder has yet taken this job running => a feeder has taken this job finished => a feeder has finished this job dead => rq died while running a job, has restarted, and moved this job to the dead state note that rq cannot move jobs into the dead state unless it has been restarted. this is because no node has any knowledge of other nodes and cannot possibly know if a job was started on a node that died, or is simply taking a very long time. only the node that dies, upon restart, can determine that is has jobs that 'were started before it started' and move these jobs into the dead state. normally only a machine crash would cause a job to be placed into the dead state. dead jobs are never automatically restarted, this is the responsibility of an operator. examples : 0) show q's status ~ > rq q t delete, d : delete combinations of pending, finished, dead, or jobs specified by jid. the delete mode is capable of parsing the output of list and query modes, making it possible to create custom filters to delete jobs meeting very specific conditions. 'mode_args' are the same as for list. note that it is NOT possible to delete a running job. rq has a decentralized architechture which