By default, make install installs all files in /usr/local/bin, /usr/local/lib, /usr/local/sbin,/usr/local/include, and /usr/local/man . You can specify an installation prefix other than/usr/local using --prefix as an argument to ./configure, for example:
./configure --prefix=$HOME
Verify you have environment variables configured so your system can find the shared libraries and binary files for TORQUE.
To set the library path, add the directory where the TORQUE libraries will be installed. For example, if your TORQUE libraries are installed in /opt/torque/lib, execute the following:
> set LD_LIBRARY_PATH=$(LD_LIBRARY_PATH):/opt/torque/lib > ldconfig
Cluster Resources recommends that the TORQUE administrator be root. |
1.2.1 Initialize/Configure TORQUE on the Server (pbs_server)
$TORQUEHOME/server_priv/ contains configuration and other information needed for pbs_server. One of the files in this directory is serverdb. serverdb contains configuration parameters forpbs_server and its queues. In order for pbs_server to run, serverdb has to be initialized.
serverdb can be initialized in two ways:
- pbs_server -t create
- Execute ./torque.setup from the build directory.
Restart pbs_server after initializing serverdb.
> qterm > pbs_server
1.2.1.1 pbs_server -t create
The '-t create' option tells pbs_server to create the serverdb file and initialize it with a minimum configuration to run pbs_server. To see the configuration, use qmgr:
> pbs_server -t create > qmgr -c 'p s' # # Set server attributes. # set server acl_hosts = kmn set server log_events = 511 set server mail_from = adm set server scheduler_iteration = 600 set server node_check_rate = 150 set server tcp_timeout = 6
A single queue named 'batch' and a few needed server attribues are created.
1.2.1.2 ./torque.setup
The torque.setup script uses pbs_server -t create to initialize serverdb, and then adds a user as a manager and operator of TORQUE and other commonly used attributes. The syntax is:
- ./torque.setup <username>
> ./torque.setup ken > qmgr -c 'p s' # # Create queues and set their attributes. # # # Create and define queue batch # create queue batch set queue batch queue_type = Execution set queue batch resources_default.nodes = 1 set queue batch resources_default.walltime = 01:00:00 set queue batch enabled = True set queue batch started = True # # Set server attributes. # set server scheduling = True set server acl_hosts = kmn set server managers = ken@kmn set server operators = ken@kmn set server default_queue = batch set server log_events = 511 set server mail_from = adm set server scheduler_iteration = 600 set server node_check_rate = 150 set server tcp_timeout = 6 set server mom_job_sync = True set server keep_completed = 300
1.2.2 Specify Compute Nodes
The environment variable $TORQUEHOME is where configuration files are stored. For TORQUE 2.1 and later, $TORQUEHOME is /var/spool/torque/. For earlier versions, $TORQUEHOME is/usr/spool/PBS/.
The pbs_server needs to know which systems on the network are its compute nodes. Each node must be specified on a line in the server's nodes file. This file is located at$TORQUEHOME/server_priv/nodes. In most cases, it is sufficient to specify just the names of the nodes on individual lines; however, various properties can be applied to each node.
Syntax of nodes file:node-name[:ts] [np=] [gpus=] [properties]
The [:ts] option marks the node as timeshared. Timeshared nodes are listed by the server in the node status report, but the server does not allocate jobs to them.
The [np=] option specifies the number of virtual processors for a given node. The value can be less than, equal to, or greater than the number of physical processors on any given node.
The [gpus=] option specifies the number of GPUs for a given node. The value can be less than, equal to, or greater than the number of physical GPUs on any given node.
The node processor count can be automatically detected by the TORQUE server ifauto_node_np is set to TRUE. This can be set using the command qmgr -c "set server auto_node_np = True". Setting auto_node_np to TRUE overwrites the value of np set in$TORQUEHOME/server_priv/nodes.
The [properties] option allows you to specify arbitrary strings to identify the node. Property strings are alphanumeric characters only and must begin with an alphabetic character.
Comment lines are allowed in the nodes file if the first non-white space character is the pound sign (#).
The example below shows a possible node file listing.
$TORQUEHOME/server_priv/nodes :# Nodes 001 and 003-005 are cluster nodes # node001 np=2 cluster01 rackNumber22 # # node002 will be replaced soon node002:ts waitingToBeReplaced # node002 will be replaced soon # node003 np=4 cluster01 rackNumber24 node004 cluster01 rackNumber25 node005 np=2 cluster01 rackNumber26 RAM16GB node006 node007 np=2 node008:ts np=4 ...
1.2.3 Configure TORQUE on the Compute Nodes
If using TORQUE self extracting packages with default compute node configuration, no additional steps are required and you can skip this section.
If installing manually, or advanced compute node configuration is needed, edit the$TORQUEHOME/mom_priv/config file on each node. The recommended settings are below.
$TORQUEHOME/mom_priv/config :$pbsserver headnode # note: hostname running pbs_server $logevent 255 # bitmap of which events to log
This file is identical for all compute nodes and can be created on the head node and distributed in parallel to all systems.
1.2.4 Finalize Configurations
After serverdb and the server_priv/nodes file are configured, and MOM has a minimal configuration, restart the pbs_server on the server node and the pbs_mom on the compute nodes.
Compute Nodes:> pbs_mom
Server Node:
> qterm -t quick > pbs_server
After waiting several seconds, the pbsnodes -a command should list all nodes in state free.