NAME rq v3.0.0 SYNOPSIS rq (queue | export RQ_Q=q) mode [mode_args]* [options]* URIS http://codeforpeople.com/lib/ruby/rq/ http://raa.ruby-lang.org/project/rq/ http://www.linuxjournal.com/article/7922 DESCRIPTION ruby queue (rq) is a zero-admin zero-configuration tool used to create instant unix clusters. rq requires only a central nfs filesystem in order to manage a simple sqlite database as a distributed priority work queue. this simple design allows researchers with minimal unix experience to install and configure, in only a few minutes and without root privileges, a robust unix cluster capable of distributing processes to many nodes - bringing dozens of powerful cpus to their knees with a single blow. clearly this software should be kept out of the hands of free radicals, seti enthusiasts, and one mr. j safran. the central concept of rq is that n nodes work in isolation to pull jobs from an centrally mounted nfs priority work queue in a synchronized fashion. the nodes have absolutely no knowledge of each other and all communication is done via the queue meaning that, so long as the queue is available via nfs and a single node is running jobs from it, the system will continue to process jobs. there is no centralized process whatsoever - all nodes work to take jobs from the queue and run them as fast as possible. this creates a system which load balances automatically and is robust in face of node failures. although the rq system is simple in it's design it features powerful functionality such as priority management, predicate and sql query , compact streaming command-line processing, programmable api, hot-backup, and input/capture of the stdin/stdout/stderr io streams of remote jobs. to date rq has had no reported runtime failures and is in operation at dozens of research centers around the world. INVOCATION the first argument to any rq command is the always the name of the queue while the second is the mode of operation. the queue name may be omitted if, and only if, the environment variable RQ_Q has been set to contain the absolute path of target queue. for instance, the command ~ > rq queue list is equivalent to ~ > export RQ_Q=queue ~ > rq list this facility can be used to create aliases for several queues, for example, a .bashrc containing alias MYQ="RQ_Q=/path/to/myq rq" alias MYQ2="RQ_Q=/path/to/myq2 rq" would allow syntax like MYQ2 submit < joblist MODES rq operates in modes create, submit, resubmit, list, status, delete, update, query, execute, configure, snapshot, lock, backup, rotate, feed, recover, ioview, cron, help, and a few others. the meaning of 'mode_args' will naturally change depending on the mode of operation. the following mode abbreviations exist, note that not all modes have abbreviations c => create s => submit r => resubmit l => list ls => list t => status d => delete rm => delete u => update q => query e => execute C => configure S => snapshot L => lock b => backup R => rotate f => feed io => ioview 0 => stdin 1 => stdout 2 => stderr h => help create, c : creates a queue. the queue must be located on an nfs mounted file system visible from all nodes intended to run jobs from it. nfs locking must be functional on this file system. examples : 0) to create a queue ~ > rq /path/to/nfs/mounted/q create or, using the abbreviation ~ > rq /path/to/nfs/mounted/q c submit, s : submit jobs to a queue to be proccesed by some feeding node. any 'mode_args' are taken as the command to run. note that 'mode_args' are subject to shell expansion - if you don't understand what this means do not use this feature and pass jobs on stdin. when running in submit mode a file may by specified as a list of commands to run using the '--infile, -i' option. this file is taken to be a newline separated list of commands to submit, blank lines and comments (#) are allowed. if submitting a large number of jobs the input file method is MUCH, more efficient. if no commands are specified on the command line rq automatically reads them from stdin. yaml formatted files are also allowed as input (http://www.yaml.org/) - note that the output of nearly all rq commands is valid yaml and may, therefore, be piped as input into the submit command. the leading '---' of yaml file may not be omitted. when submitting the '--priority, -p' option can be used here to determine the priority of jobs. priorities may be any whole number including negative ones - zero is the default. note that submission of a high priority job will NOT supplant a currently running low priority job, but higher priority jobs WILL always migrate above lower priority jobs in the queue in order that they be run as soon as possible. constant submission of high priority jobs may create a starvation situation whereby low priority jobs are never allowed to run. avoiding this situation is the responsibility of the user. the only guaruntee rq makes regarding job execution is that jobs are executed in an 'oldest-highest-priority' order and that running jobs are never supplanted. jobs submitted with the '--stage' option will not be eligible to be run by any node and will remain in a 'holding' state until updated (see update mode) into the 'pending' mode, this option allows jobs to entered, or 'staged', in the queue and then made candidates for running at a later date. rq allows the stdin of commands to be provided and also captures the stdout and stderr of any job run (of course standard shell redirects may be used as well) and all three will be stored in a directory relative the the queue itself. the stdin/stdout/stderr files are stored by job id and there location (though relative to the queue) is shown in the output of 'list' (see docs for list). examples : 0) submit the job ls to run on some feeding host ~ > rq q s ls 1) submit the job ls to run on some feeding host, at priority 9 ~ > rq -p9 q s ls 2) submit a list of jobs from file. note the '-' used to specify reading jobs from stdin ~ > cat joblist job1.sh job2.sh job2.sh ~ > rq q submit --infile=joblist 3) submit a joblist on stdin ~ > cat joblist | rq q submit - or ~ > rq q submit - rq q submit cat --stdin=cat.in 5) submit cat as a job, providing the stdin for the cat job on stdin ~ > cat cat.in | rq q submit cat --stdin=- or ~ > rq q submit cat --stdin=- wc -l cmdfile 42 ~ > rq -p9 -timportant q s < cmdfile 6) re-submit all the 'important' jobs (see 'query' section below) ~ > rq q query tag=important | rq q s - 8) re-submit all jobs which are already finished (see 'list' section below) ~ > rq q l f | rq q s 9) stage the job wont_run_yet to the queue in a 'holding' state. no feeder will run this job until it's state is upgraded to 'pending' ~ > rq q s --stage wont_run_yet resubmit, r : resubmit jobs back to a queue to be proccesed by a feeding node. resubmit is essentially equivalent to submitting a job that is already in the queue as a new job and then deleting the original job except that using resubmit is atomic and, therefore, safer and more efficient. resubmission respects any previous stdin provided for job input. read docs for delete and submit for more info. examples : 0) resubmit job 42 to the queue ~> rq q resubmit 42 1) resubmit all failed jobs ~> rq q query exit_status!=0 | rq q resubmit - 2) resubmit job 4242 with different stdin ~ rq q resubmit 4242 --stdin=new_stdin.in list, l, ls : list mode lists jobs of a certain state or job id. state may be one of pending, holding, running, finished, dead, or all. any 'mode_args' that are numbers are taken to be job id's to list. states may be abbreviated to uniqueness, therefore the following shortcuts apply : p => pending h => holding r => running f => finished d => dead a => all examples : 0) show everything in q ~ > rq q list all or ~ > rq q l all or ~ > export RQ_Q=q ~ > rq l 1) show q's pending jobs ~ > rq q list pending 2) show q's running jobs ~ > rq q list running 3) show q's finished jobs ~ > rq q list finished 4) show job id 42 ~ > rq q l 42 5) show q's holding jobs ~ > rq q list holding status, t : status mode shows the global state the queue and statistics on it's the cluster's performance. there are no 'mode_args'. the meaning of each state is as follows: pending => no feeder has yet taken this job holding => a hold has been placed on this job, thus no feeder will start it running => a feeder has taken this job finished => a feeder has finished this job dead => rq died while running a job, has restarted, and moved this job to the dead state note that rq cannot move jobs into the dead state unless it has been restarted. this is because no node has any knowledge of other nodes and cannot possibly know if a job was started on a node that subsequently died, or that it is simply taking a very long time to complete. only the node that dies, upon restart, can determine that it owns jobs that 'were started before it started running jobs', an impossibility, and move these jobs into the dead state. normally only a machine crash would cause a job to be placed into the dead state. dead jobs are automatically restarted if, and only if, the job was submitted with the '--restartable' flag. status breaks down a variety of canned statistics about a nodes' performance based solely on the jobs currently in the queue. only one option affects the ouput: '--exit'. this option is used to specify additionaly exit code mappings on which to report. normally rq will report any job with an exit code of 0 as being 'successes' and any job with an exit code that is not 0, or a status of 'dead', as being 'failures'. if the '--exit' switch is used then additional mappings can be specified, note that the the semantics for 'successes' and 'failures' does not change - this keyword specifies extra mappings. examples : 0) show q's status ~ > rq q t 2) show q's status, consider any exit code of 42 will be listed as 'ok' ~ > rq q t --exit ok=42 3) show q's status, consider any exit code of 42 or 43 will be listed as 'ok' and 127 will be listed as 'command_not_found'. notice the quoting required. ~ > rq q t --exit 'ok=42,43 command_not_found=127' delete, d : delete combinations of pending, holding, finished, dead, or jobs specified by jid. the delete mode is capable of parsing the output of list and query modes, making it possible to create custom filters to delete jobs meeting very specific conditions. 'mode_args' are the same as for list. note that it is NOT possible to delete a running job. rq has a decentralized architechture which means that compute nodes are completely independant of one another; an extension is that there is no way to communicate the deletion of a running job from the queue the the node actually running that job. it is not an error to force a job to die prematurely using a facility such as an ssh command spawned on the remote host to kill it. once a job has been noted to have finished, whatever the exit status, it can be deleted from the queue. examples : 0) delete all