<div dir="ltr"><div><div>Hello<br><br></div>We are having a small 60 node cluster with 12 cores each.<br><br></div><div>We are using Torque 2.5.12 and Maui 3.3<br></div><div><br>When i submit jobs on torque using multi-core  using #PBS directive , the jobs get queued in the queue. I can run the job with qsub in Interactive mode.<br>
<br>#PBS -l &quot;nodes=1:ppn=4&quot;<br><br>The error i get in  maui log file as follows :<br><br><br>10/24 20:16:10 ALERT:    job 427 cannot run in any partition<br>10/24 20:16:10 ALERT:    cannot create new reservation for job 427 (shape[1] 4)<br>
10/24 20:16:10 ALERT:    cannot create new reservation for job 427<br>10/24 20:16:10 ALERT:    job &#39;427&#39; cannot run (deferring job for 3600 seconds)<br><br>-----------------------------------------------------------------------------------------------------<br>
The  showref  shows the following<br> <br>Reservations<br><br>ReservationID       Type S       Start         End    Duration    N/P    StartTime<br><br><br>0 reservations located<br><br>---------------------------------------------------------------------------------------------------------------------<br>
<br>Showbf gives the following error:<br><br>Node is blocked  by reservation NONE in INFINITY<br><br>----------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
<br>checkjob  gives the following<br><br>checking job 427<br><br>State: Idle  EState: Deferred<br>Creds:  user:guestuser2  group:guestuser2  class:p60  qos:DEFAULT<br>WallTime: 00:00:00 of 1:00:00<br>SubmitTime: Thu Oct 24 11:11:50<br>
  (Time Queued  Total: 10:03:14  Eligible: 00:00:00)<br><br>Total Tasks: 4<br><br>Req[0]  TaskCount: 4  Partition: ALL<br>Network: [NONE]  Memory &gt;= 0  Disk &gt;= 0  Swap &gt;= 0<br>Opsys: [NONE]  Arch: [NONE]  Features: [NONE]<br>
<br><br>IWD: [NONE]  Executable:  [NONE]<br>Bypass: 0  StartCount: 0<br>PartitionMask: [ALL]<br>Flags:       RESTARTABLE<br><br>job is deferred.  Reason:  NoResources  (cannot create reservation for job &#39;427&#39; (intital reservation attempt)<br>
)<br>Holds:    Defer  (hold reason:  NoResources)<br>PE:  4.00  StartPriority:  58<br>cannot select job 427 for partition DEFAULT (job hold active)<br><br>-----------------------------------------------------------------------------------------------------------------------------------------------<br>
<br>My queue configuration is as follows<br><br> queue_type = Execution<br>        max_queuable = 120<br>        max_user_queuable = 120<br>        total_jobs = 3<br>        state_count = Transit:0 Queued:-2 Held:0 Waiting:0 Running:5 Exiting:0<br>
        max_running = 20<br>        resources_max.ncpus = 64<br>        resources_max.walltime = 48:00:00<br>        resources_default.neednodes = 1<br>        resources_default.nodect = 1<br>        resources_default.nodes = 1<br>
        resources_default.walltime = 01:00:00<br>        mtime = Thu Oct 24 15:28:41 2013<br>        resources_assigned.nodect = 0<br>        max_user_run = 10<br>        enabled = True<br>        started = True<br>----------------------------------------------------------------------------------------------------------------------------------<br>
<br>My Pbs server configuration is as follows<br><br> server_state = Idle<br>        total_jobs = 3<br>        state_count = Transit:0 Queued:-129 Held:127 Waiting:0 Running:5 Exiting:0<br>        acl_hosts = krplpadul001<br>
        default_queue = p60<br>        log_events = 511<br>        mail_from = adm<br>        resources_assigned.nodect = 0<br>        scheduler_iteration = 600<br>        node_check_rate = 150<br>        tcp_timeout = 6<br>
        pbs_version = 2.5.12<br>        auto_node_np = False<br>        next_job_number = 428<br>        net_counter = 12 4 2<br>        record_job_info = True<br>        record_job_script = True<br>        job_log_keep_days = 7<br>
<br><br>-------------------------------------------------------------------------------------------------------------------------------------------------------<br>Maui schedctl shows the following<br><br><br>REJECTNEGPRIOJOBS[0]              FALSE<br>
ENABLENEGJOBPRIORITY[0]           FALSE<br>ENABLEMULTINODEJOBS[0]            TRUE<br>ENABLEMULTIREQJOBS[0]             FALSE<br>BFPRIORITYPOLICY[0]               [NONE]<br>JOBPRIOACCRUALPOLICY            QUEUEPOLICY<br>NODELOADPOLICY                  ADJUSTSTATE<br>
USEMACHINESPEED                 FALSE<br>USESYSTEMQUEUETIME              TRUE<br>USELOCALMACHINEPRIORITY         FALSE<br>NODEUNTRACKEDLOADFACTOR         1.2<br>JOBNODEMATCHPOLICY[0]<br><br>JOBMAXSTARTTIME[0]                  INFINITY<br>
<br>METAMAXTASKS[0]                   0<br>NODESETPOLICY[0]                  [NONE]<br>NODESETATTRIBUTE[0]               [NONE]<br>NODESETLIST[0]<br>NODESETDELAY[0]                   00:00:00<br>NODESETPRIORITYTYPE[0]            MINLOSS<br>
NODESETTOLERANCE[0]                 0.00<br><br>BACKFILLPOLICY[0]                 FIRSTFIT<br>BACKFILLDEPTH[0]                  0<br>BACKFILLPROCFACTOR[0]             0<br>BACKFILLMAXSCHEDULES[0]           10000<br>BACKFILLMETRIC[0]                 PROCS<br>
<br>BFCHUNKDURATION[0]                00:00:00<br>BFCHUNKSIZE[0]                    0<br>PREEMPTPOLICY[0]                  REQUEUE<br>MINADMINSTIME[0]                  00:00:00<br>RESOURCELIMITPOLICY[0]<br>NODEAVAILABILITYPOLICY[0]         COMBINED:[DEFAULT]<br>
NODEALLOCATIONPOLICY[0]           MINRESOURCE<br>TASKDISTRIBUTIONPOLICY[0]         DEFAULT<br>RESERVATIONPOLICY[0]              CURRENTHIGHEST<br>RESERVATIONRETRYTIME[0]           00:00:00<br>RESERVATIONTHRESHOLDTYPE[0]       NONE<br>
RESERVATIONTHRESHOLDVALUE[0]      0<br><br>FSPOLICY                        [NONE]<br>FSPOLICY                        [NONE]<br>FSINTERVAL                      12:00:00<br>FSDEPTH                         8<br>FSDECAY                         1.00<br>
<br><br><br># Priority Weights<br><br>SERVICEWEIGHT[0]                  1<br>TARGETWEIGHT[0]                   1<br>CREDWEIGHT[0]                     1<br>ATTRWEIGHT[0]                     1<br>FSWEIGHT[0]                       1<br>
RESWEIGHT[0]                      1<br>USAGEWEIGHT[0]                    1<br>QUEUETIMEWEIGHT[0]                1<br>XFACTORWEIGHT[0]                  0<br>SPVIOLATIONWEIGHT[0]              0<br>BYPASSWEIGHT[0]                   0<br>
TARGETQUEUETIMEWEIGHT[0]          0<br>TARGETXFACTORWEIGHT[0]            0<br>USERWEIGHT[0]                     0<br>GROUPWEIGHT[0]                    0<br>ACCOUNTWEIGHT[0]                  0<br>QOSWEIGHT[0]                      0<br>
CLASSWEIGHT[0]                    0<br>FSUSERWEIGHT[0]                   0<br>FSGROUPWEIGHT[0]                  0<br>FSACCOUNTWEIGHT[0]                0<br>FSQOSWEIGHT[0]                    0<br>FSCLASSWEIGHT[0]                  0<br>
ATTRATTRWEIGHT[0]                 0<br>ATTRSTATEWEIGHT[0]                0<br>NODEWEIGHT[0]                     0<br>PROCWEIGHT[0]                     0<br>MEMWEIGHT[0]                      0<br>SWAPWEIGHT[0]                     0<br>
DISKWEIGHT[0]                     0<br>PSWEIGHT[0]                       0<br>PEWEIGHT[0]                       0<br>WALLTIMEWEIGHT[0]                 0<br>UPROCWEIGHT[0]                    0<br>UJOBWEIGHT[0]                     0<br>
CONSUMEDWEIGHT[0]                 0<br>USAGEEXECUTIONTIMEWEIGHT[0]       0<br>REMAININGWEIGHT[0]                0<br>PERCENTWEIGHT[0]                  0<br>XFMINWCLIMIT[0]                   00:02:00<br><br><br># partition DEFAULT policies<br>
<br>REJECTNEGPRIOJOBS[1]              FALSE<br>ENABLENEGJOBPRIORITY[1]           FALSE<br>ENABLEMULTINODEJOBS[1]            TRUE<br>ENABLEMULTIREQJOBS[1]             FALSE<br>BFPRIORITYPOLICY[1]               [NONE]<br>JOBPRIOACCRUALPOLICY            QUEUEPOLICY<br>
NODELOADPOLICY                  ADJUSTSTATE<br>JOBNODEMATCHPOLICY[1]<br><br>JOBMAXSTARTTIME[1]                  INFINITY<br><br>METAMAXTASKS[1]                   0<br>NODESETPOLICY[1]                  [NONE]<br>NODESETATTRIBUTE[1]               [NONE]<br>
NODESETLIST[1]<br>NODESETDELAY[1]                   00:00:00<br>NODESETPRIORITYTYPE[1]            MINLOSS<br>NODESETTOLERANCE[1]                 0.00<br><br># Priority Weights<br><br>XFMINWCLIMIT[1]                   00:00:00<br>
<br>RMAUTHTYPE[0]                     CHECKSUM<br><br>CLASSCFG[[NONE]]  DEFAULT.FEATURES=[NONE]<br>CLASSCFG[[ALL]]  DEFAULT.FEATURES=[NONE]<br>CLASSCFG[p60]  DEFAULT.FEATURES=[NONE]<br>CLASSCFG[long]  DEFAULT.FEATURES=[NONE]<br>
QOSPRIORITY[0]                    0<br>QOSQTWEIGHT[0]                    0<br>QOSXFWEIGHT[0]                    0<br>QOSTARGETXF[0]                      0.00<br>QOSTARGETQT[0]                    00:00:00<br>QOSFLAGS[0]<br>
QOSPRIORITY[1]                    0<br>QOSQTWEIGHT[1]                    0<br>QOSXFWEIGHT[1]                    0<br>QOSTARGETXF[1]                      0.00<br>QOSTARGETQT[1]                    00:00:00<br>QOSFLAGS[1]<br>
# SERVER MODULES:  MX<br>SERVERMODE                      NORMAL<br>SERVERNAME<br>SERVERHOST                      krplpadul001<br>SERVERPORT                      42559<br>LOGFILE                         maui.log<br>LOGFILEMAXSIZE                  10000000<br>
LOGFILEROLLDEPTH                1<br>LOGLEVEL                        3<br>LOGFACILITY                     fALL<br>SERVERHOMEDIR                   /usr/local/maui/<br>TOOLSDIR                        /usr/local/maui/tools/<br>
LOGDIR                          /usr/local/maui/log/<br>STATDIR                         /usr/local/maui/stats/<br>LOCKFILE                        /usr/local/maui/maui.pid<br>SERVERCONFIGFILE                /usr/local/maui/maui.cfg<br>
CHECKPOINTFILE                  /usr/local/maui/<a href="http://maui.ck">maui.ck</a><br>CHECKPOINTINTERVAL              00:05:00<br>CHECKPOINTEXPIRATIONTIME        3:11:20:00<br>TRAPJOB<br>TRAPNODE<br>TRAPFUNCTION<br>RESDEPTH                        24<br>
<br>RMPOLLINTERVAL                  00:00:30<br>NODEACCESSPOLICY                SHARED<br>ALLOCLOCALITYPOLICY             [NONE]<br>SIMTIMEPOLICY                   [NONE]<br>ADMIN1                          root<br>ADMINHOSTS                      ALL<br>
NODEPOLLFREQUENCY               0<br>DISPLAYFLAGS<br>DEFAULTDOMAIN<br>DEFAULTCLASSLIST                [DEFAULT:1]<br>FEATURENODETYPEHEADER<br>FEATUREPROCSPEEDHEADER<br>FEATUREPARTITIONHEADER<br>DEFERTIME                       1:00:00<br>
DEFERCOUNT                      24<br>DEFERSTARTCOUNT                 1<br>JOBPURGETIME                    0<br>NODEPURGETIME                   2140000000<br>APIFAILURETHRESHHOLD            6<br>NODESYNCTIME                    600<br>
JOBSYNCTIME                     600<br>JOBMAXOVERRUN                   00:10:00<br>NODEMAXLOAD                     0.0<br><br>PLOTMINTIME                     120<br>PLOTMAXTIME                     245760<br>PLOTTIMESCALE                   11<br>
PLOTMINPROC                     1<br>PLOTMAXPROC                     512<br>PLOTPROCSCALE                   9<br>SCHEDCFG[]                        MODE=NORMAL SERVER=krplpadul001:42559<br># RM MODULES: PBS SSS WIKI NATIVE<br>
RMCFG[KRPLPADUL001] AUTHTYPE=CHECKSUM EPORT=15004 TIMEOUT=00:00:09 TYPE=PBS<br>SIMWORKLOADTRACEFILE            workload<br>SIMRESOURCETRACEFILE            resource<br>SIMAUTOSHUTDOWN                 OFF<br>SIMSTARTTIME                    0<br>
SIMSCALEJOBRUNTIME              FALSE<br>SIMFLAGS<br>SIMJOBSUBMISSIONPOLICY          CONSTANTJOBDEPTH<br>SIMINITIALQUEUEDEPTH            16<br>SIMWCACCURACY                   0.00<br>SIMWCACCURACYCHANGE             0.00<br>
SIMNODECOUNT                    0<br>SIMNODECONFIGURATION            NORMAL<br>SIMWCSCALINGPERCENT             100<br>SIMCOMRATE                      0.10<br>SIMCOMTYPE                      ROUNDROBIN<br>COMINTRAFRAMECOST               0.30<br>
COMINTERFRAMECOST               0.30<br>SIMSTOPITERATION                -1<br>SIMEXITITERATION                -1<br><br><br><br></div><div>What am i missing here that maui can not schedule jobs?<br><br>Sincerely,<br></div>
<div>Amjad<br></div></div>