<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:537356822;
mso-list-type:hybrid;
mso-list-template-ids:-1223119274 -1206773510 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:22.5pt;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Anthony,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> Assuming that you are running the perl script on the head node of the cluster,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>and assuming you have a newer torque (version 2.5.4 is sufficient) installed,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>how about the following: If t1 contains the pbs job:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>perl -e '$A=`qsub -I -x -lnodes=1:ppn=1 t1; echo $? `; print $A'<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> Those are backticks before qsub and after echo $? <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> The –I –x combination causes the qsub to print output to the screen, and to hold onto the <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>job while it is run, which is what I think that you want.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I am print the value of $A, but you could use it in your perl program.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>E.g.for me:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>t1 contains:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>date<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>sleep 3<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>echo 'test'<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>sleep 3<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>date<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Issuing the command <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>time perl -e '$A=`qsub -I -x -lnodes=1:ppn=1 t1; echo $? `; print $A'<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>yields: <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>qsub: waiting for job 150965.hpc5 to start<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>qsub: job 150965.hpc5 ready<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Wed Aug 17 15:50:20 CDT 2011<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>test<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Wed Aug 17 15:50:26 CDT 2011<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>qsub: job 150965.hpc5 completed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>0.027u 0.042s 0:06.26 0.9% 0+0k 0+16io 0pf+0w<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoListParagraph style='margin-left:22.5pt;text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Jim<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:16.0pt;font-family:Consolas;color:#1F497D'>James Coyle, PhD<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:16.0pt;font-family:Consolas;color:#1F497D'> High Performance Computing Group <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:16.0pt;font-family:Consolas;color:#1F497D'> Iowa State Univ. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:16.0pt;font-family:Consolas;color:#1F497D'>web: <a href="http://www.public.iastate.edu/~jjc"><span style='color:blue'>http://jjc.public.iastate.edu/</span></a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> torqueusers-bounces@supercluster.org [mailto:torqueusers-bounces@supercluster.org] <b>On Behalf Of </b>William Nelson<br><b>Sent:</b> Wednesday, August 17, 2011 10:36 AM<br><b>To:</b> Torque Users Mailing List<br><b>Cc:</b> torquedev@supercluster.org<br><b>Subject:</b> Re: [torqueusers] Interacting with Torque/Maui from Perl scripts<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi Anthony,<o:p></o:p></p><div><p class=MsoNormal>I don't have a specific answer to your question but I can point you to some example code. InterProScan (<a href="http://www.ebi.ac.uk/Tools/pfa/iprscan/">http://www.ebi.ac.uk/Tools/pfa/iprscan/</a>) is a PERL project that integrates with PBS, LSF, or SGE. We use it with Torque by making small modifications to the PBS conf files. It runs several analysis tool in parallel on the cluster and then merges the results. <o:p></o:p></p></div><div><p class=MsoNormal>-Bill<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Wed, Aug 17, 2011 at 1:33 AM, Anthony Ferrari <<a href="mailto:ferraria@gmail.com">ferraria@gmail.com</a>> wrote:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hi all,<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Basically, I would like to launch a job on a cluster with specific resources and dependencies, wait for the job to complete or fail, check that all has gone OK, then do something with the generated files ...<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>To launch jobs, I am using a customized version o<span style='font-family:"Arial","sans-serif"'>f<span style='background:white'> PBS-Client-0.09 package written by Ka-Wai Mak. </span></span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif";background:white'>I also saw that packages named PBS-0.03 by Todd Merritt and perl-PBS by Garrick Staples exist but they do not seem maintained or seem outdated (please correct me if I am wrong).</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>What do you personally use ?</span><o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>My problem here is more after the job submission, to keep 'control' of the job.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Because with the PBS-Client module, once the job is submitted, Torque does what it has to do and that's it. It does not give any information in return to my Perl script.</span><o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>I was thinking about several solutions to achieve this but I do not find them very smart ... like :</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - using qstat | grep '^myjobid' in a while loop (with open3 perl function)</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - waiting for the output & error files to be written where I expect them (hum...)</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - use an epilogue script in the job to do something useful...</span><o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>But there may exist <span style='background:white'>brighter </span>answers to this question. Perhaps with some modules I don't know about.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Do you have any advice ?</span><o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>And actually this is to know when the job has finished but how do you check that the job completed and did not crash ?</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Do you always parse the created logs ? </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Is there a way to propagate the exit code of the program launched with qsub to the perl script running on the master node ?</span><o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Thank you very much.</span><o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Cheers,</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Anthony</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='color:#888888'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='color:#888888'><o:p> </o:p></span></p></div><p class=MsoNormal style='margin-bottom:12.0pt'><br>_______________________________________________<br>torqueusers mailing list<br><a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br><a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><o:p></o:p></p></div><p class=MsoNormal><br><br clear=all><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>-- <span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#444444'><o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#444444'>Bill Nelson<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#444444'>Bioinformatics Developer<o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#444444'>TAIR<br>Carnegie Institution of Washington<br>Dept. of Plant Biology<br>260, Panama<br>Stanford, CA 94305<br>USA<br><br>Cell: (650) 720-5708</span><o:p></o:p></p><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#444444'>Phone: </span><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#444444'>Phone: (650) 739-4343</span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#444444'><br>Fax: (650) 325-6857</span><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p></div></div></div></body></html>