Difference between revisions of "MSblender TACC"

From Marcotte Lab
Jump to: navigation, search
(Run MSGF+)
 
(32 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
== Before you start ==
 
== Before you start ==
 
* To use this setting, your TACC account needs to be allocated to our lab project('A-cm10'). If you don't have an account, create it at https://portal.tacc.utexas.edu/. Then, ask Edward to assign your account as a member of lab project.
 
* To use this setting, your TACC account needs to be allocated to our lab project('A-cm10'). If you don't have an account, create it at https://portal.tacc.utexas.edu/. Then, ask Edward to assign your account as a member of lab project.
* Generally, 'longhorn' has shorter queue than 'lonestar'. So if you don't need large memory(> 4GB), use 'longhorn'.  
+
* This document is for [https://portal.tacc.utexas.edu/user-guides/stampede 'stampede'].  
* Always work at $SCRATCH directory, not at /corral or your $HOME.  
+
* Currently in most cases I use three search engines: comet, X!Tandem, and MS-GF+.
* All source codes are available at /corral/utexas/A-cm10/src.MS/.
+
* You don't need to run 'MSblender' modeling on TACC, because it does not take that long. I normally run all searches at TACC, then transfer the output to my local machine to run MSblender. So it only covers 'search' part. For running MSblender, please see [[MSblender]] page.
* All packages are installed at /corral/utexas/A-cm10/src.MS/local/.
+
* To run InsPecT, you need to set LD_LIBRARY_PATH for expat library. Type below command before running InsPecT (or put it on '$HOME/.profile_user'  
+
<pre> $ export LD_LIBRARY_PATH="/corral/utexas/A-cm10/src.MS/local/lib/" </pre>
+
* Default python interpreter is 2.4 at TACC. You need to load 2.7.1 as below.
+
<pre> $ module load python</pre>
+
  
== Currently installed packages ==
+
== Install MSblender (and comet, MSGF+, X!Tandem) ==  
These packages are installed at lonestar.
+
<pre>$ cd ~
* TPP-4.5.1 + X!Tandem (2010.10.01.1)
+
* /corral/utexas/A-cm10/src.MS/local/tppbin/xinteract (integrated wrapper for *Prophet)
+
* /corral/utexas/A-cm10/src.MS/local/tppbin/tandem (X!Tandem with k-score support)
+
* Crux 1.37
+
** /corral/utexas/A-cm10/src.MS/local/bin/crux
+
* Tide 1.0
+
** /corral/utexas/A-cm10/src.MS/local/bin/tide-index
+
** /corral/utexas/A-cm10/src.MS/local/bin/tide-msconvert
+
** /corral/utexas/A-cm10/src.MS/local/bin/tide-search
+
* MSGFDB (20120106)
+
** /corral/utexas/A-cm10/src.MS/MSGFDB/current/MSGFDB.jar
+
* InsPecT (20120109)
+
** /corral/utexas/A-cm10/src.MS/local/bin/inspect
+
 
+
== Install MS-toolbox & MSblender ==
+
* I recommend to install MS-toolbox at your home individually, because everyone may want to use different search parameters.
+
<pre>$ module load git
+
$ cd ~
+
 
$ mkdir git
 
$ mkdir git
 
$ cd git
 
$ cd git
$ git clone git@github.com:marcottelab/MS-toolbox.git
+
$ git clone https://github.com/marcottelab/MSblender.git</pre>
$ git clone git@github.com:marcottelab/MSblender.git</pre>
+
 
* You don't need to compile MSblender codes under 'src' directory. Executable file is already available at /corral/utexas/A-cm10/src.MS/local/bin/msblender.
+
== Prepare a working space ==
 +
<pre>$ module load python
 +
$ cd $SCRATCH
 +
$ mkdir myProject
 +
$ cd myProject
 +
$ mkdir mzXML
 +
$ mkdir DB
 +
$ mkdir comet
 +
$ mkdir MSGF+
 +
$ mkdir tandemK</pre>
 +
 
 +
== Prepare database ==
 +
* You can run this process on any computer. If it takes longer than a minute, it would be better to process it on other than TACC login node (your account may be locked).
 +
 
 +
<pre>$ python $HOME/git/MSblender/pre/fasta-reverse.py my_seq.fa
 +
$ cat my_seq.fa.* > my_seq.combined.fa</pre>
 +
 +
=== DB setup for X!tandem ===
 +
<pre> $ $HOME/git/MSblender/extern/fasta_pro.exe my_seq.combined.fa</pre>
 +
 
 +
You may see the message like below:
 +
 
 +
<pre>$ ~/git/MSblender/extern/fasta_pro.exe my_seq.combined.fa
 +
fasta_pro file conversion utility, v. 2006.09.15
 +
input path = my_seq.combined.fa
 +
output path = my_seq.combined.fa.pro
 +
db type = plain</pre>
 +
 
 +
=== DB setup for comet ===
 +
You don't need to do anything for this.
 +
 
 +
=== DB setup for MSGF+ ===
 +
It uses significant amount of computing resources (i.e. memory), so it may not be suitable to run on login node.
 +
 
 +
<pre>$ module load jdk64
 +
$ java -Xmx4000M -cp /home1/00992/linusben/git/MSblender/extern/MSGFPlus.jar edu.ucsd.msjava.msdbsearch.BuildSA -d XenopusHybrid_xlJGIv16_xtJGIv83.combined.fa -tda 0</pre>
 +
 
 +
== Prepare mzXML files ==
 +
 
 +
Copy your mzXML files on this diretory ($SCRATCH/myProject/mzXML).
 +
 
 +
== Run comet ==
 +
<pre>$ cd $SCRATCH/myProject/comet
 +
$ ~/git/MSblender/extern/comet.linux.exe -p</pre>
 +
 
 +
Edit 'comet.params.new' file. Typically, you need to change the following lines.
 +
<pre>num_threads = 16
 +
 
 +
peptide_mass_tolerance = 20.0
 +
peptide_mass_units = 2
 +
 
 +
search_enzyme_number = 2  ## See the end of param file for the type of enzymes
 +
 
 +
output_txtfile = 1
 +
output_pepxmlfile = 0</pre>
 +
 
 +
Then, create the launcher script (called 'stampede-comet.sh') as below.
 +
 
 +
<pre>#!/bin/bash
 +
#SBATCH -n 16
 +
#SBATCH -p normal
 +
#SBATCH -t 24:00:00
 +
 
 +
#SBATCH -o cmt.o%j
 +
COMET="$HOME/git/MSblender/extern/comet.linux.exe"
 +
 
 +
DB="../DB/my_seq.combined.fa"
 +
DBNAME=$(basename $DB)
 +
DBNAME=${DBNAME/.fa/}
 +
 
 +
PARAM="./comet.params.new"
 +
 
 +
#SBATCH -J "cmt"
 +
for MZXML in $(ls ../mzXML/*mzXML)
 +
do
 +
  OUT=$(basename $MZXML)
 +
  OUT=${OUT/.mzXML/}"."$DBNAME".comet"
 +
  time $COMET -P$PARAM -D$DB -N$OUT $MZXML
 +
done
 +
</pre>
 +
 
 +
Then, submit the job by typing 'sbatch stampede-comet.sh'
 +
 
 +
== Run MSGF+ ==
 +
 
 +
Create 'stampede-MSGF+.sh' file as below.
 +
<pre>#!/bin/bash
 +
#SBATCH -n 16
 +
#SBATCH -p normal
 +
#SBATCH -t 24:00:00
 +
 
 +
#SBATCH -o mg+.o%j
 +
set -x
 +
 
 +
module load jdk64
 +
 
 +
MSGFplus_JAR="$HOME/git/MSblender/extern/MSGFPlus.jar"
 +
 
 +
DB="../DB/my_seq.combined.fa"
 +
 
 +
DBNAME=$(basename $DB)
 +
DBNAME=${DBNAME/.fa/}
 +
 
 +
#SBATCH -J "mg+"
 +
for MZXML in $(ls ../mzXML/*mzXML)
 +
do
 +
  OUT=$(basename $MZXML)
 +
  OUT=${OUT/.mzXML/}"."$DBNAME".MSGF+.mzid"
 +
  TBL=${OUT/.mzid/.tsv}
 +
  time java -Xmx20000M -jar $MSGFplus_JAR -d $DB -s $MZXML -o $OUT -t 20ppm -tda 0 -ntt 2 -e 1 -inst 3
 +
  time java -Xmx20000M -cp $MSGFplus_JAR edu.ucsd.msjava.ui.MzIDToTsv -i $OUT -o $TBL -showQValue 1 -showDecoy 1 -unroll 0
 +
done</pre>
 +
 
 +
Then, submit the job by typing 'sbatch stampede-MSGF+.sh'
 +
 
 +
== Run X!Tandem ==
 +
 
 +
<pre>$ cd $SCRATCH/myProject/tandemK
 +
$ ~/git/MSblender/search/prepare-tandemK-high.py ../mzXML/ ../DB/my_seq.combined.fa.pro</pre>
 +
 
 +
First argument of prepare-tendemK-high.py is a directory for mzXML, and second one is for .pro database generated by fasta_pro.exe as above.
 +
 
 +
You will see *.xml files matched to your mzXML files (X!Tandem input), 'tandem-taxonomy.xml' file (another X!Tandem input), and run-tandemK.sh (a script to run X!tandem).
 +
 
 +
Make the following launcher ('stampede-tandemK.sh'), and submit it as 'sbatch stampede-tandemK.sh'.
 +
 
 +
<pre>#!/bin/bash
 +
#SBATCH -n 16
 +
#SBATCH -p normal
 +
#SBATCH -t 24:00:00
 +
 
 +
#SBATCH -o tK.o%j
 +
#SBATCH -J "tK"
 +
set -x
 +
 
 +
bash ./run-tandemK.sh</pre>
  
== Let's start ==
+
If you have many mzXML files, you can run it parallel by splitting run-tandemK.sh with 'split -l' command, and run individual splitted script at 'stampede-tandemK.sh'.
<pre>$ cd $SCRATCH
+
$ mkdir my-project
+
$ cd my-project
+
$ ~/git/MS-toolbox/bin/mstb-setup.py
+

Latest revision as of 15:39, 29 June 2015

Contents

Before you start

  • To use this setting, your TACC account needs to be allocated to our lab project('A-cm10'). If you don't have an account, create it at https://portal.tacc.utexas.edu/. Then, ask Edward to assign your account as a member of lab project.
  • This document is for 'stampede'.
  • Currently in most cases I use three search engines: comet, X!Tandem, and MS-GF+.
  • You don't need to run 'MSblender' modeling on TACC, because it does not take that long. I normally run all searches at TACC, then transfer the output to my local machine to run MSblender. So it only covers 'search' part. For running MSblender, please see MSblender page.

Install MSblender (and comet, MSGF+, X!Tandem)

$ cd ~
$ mkdir git
$ cd git
$ git clone https://github.com/marcottelab/MSblender.git

Prepare a working space

$ module load python
$ cd $SCRATCH
$ mkdir myProject
$ cd myProject
$ mkdir mzXML
$ mkdir DB
$ mkdir comet
$ mkdir MSGF+
$ mkdir tandemK

Prepare database

  • You can run this process on any computer. If it takes longer than a minute, it would be better to process it on other than TACC login node (your account may be locked).
$ python $HOME/git/MSblender/pre/fasta-reverse.py my_seq.fa
$ cat my_seq.fa.* > my_seq.combined.fa

DB setup for X!tandem

 $ $HOME/git/MSblender/extern/fasta_pro.exe my_seq.combined.fa

You may see the message like below:

$ ~/git/MSblender/extern/fasta_pro.exe my_seq.combined.fa 
fasta_pro file conversion utility, v. 2006.09.15
 input path = my_seq.combined.fa
output path = my_seq.combined.fa.pro
db type = plain

DB setup for comet

You don't need to do anything for this.

DB setup for MSGF+

It uses significant amount of computing resources (i.e. memory), so it may not be suitable to run on login node.

$ module load jdk64
$ java -Xmx4000M -cp /home1/00992/linusben/git/MSblender/extern/MSGFPlus.jar edu.ucsd.msjava.msdbsearch.BuildSA -d XenopusHybrid_xlJGIv16_xtJGIv83.combined.fa -tda 0

Prepare mzXML files

Copy your mzXML files on this diretory ($SCRATCH/myProject/mzXML).

Run comet

$ cd $SCRATCH/myProject/comet
$ ~/git/MSblender/extern/comet.linux.exe -p

Edit 'comet.params.new' file. Typically, you need to change the following lines.

num_threads = 16

peptide_mass_tolerance = 20.0
peptide_mass_units = 2

search_enzyme_number = 2   ## See the end of param file for the type of enzymes

output_txtfile = 1
output_pepxmlfile = 0

Then, create the launcher script (called 'stampede-comet.sh') as below.

#!/bin/bash
#SBATCH -n 16
#SBATCH -p normal
#SBATCH -t 24:00:00

#SBATCH -o cmt.o%j
COMET="$HOME/git/MSblender/extern/comet.linux.exe"

DB="../DB/my_seq.combined.fa"
DBNAME=$(basename $DB)
DBNAME=${DBNAME/.fa/}

PARAM="./comet.params.new"

#SBATCH -J "cmt"
for MZXML in $(ls ../mzXML/*mzXML)
do
  OUT=$(basename $MZXML)
  OUT=${OUT/.mzXML/}"."$DBNAME".comet"
  time $COMET -P$PARAM -D$DB -N$OUT $MZXML
done

Then, submit the job by typing 'sbatch stampede-comet.sh'

Run MSGF+

Create 'stampede-MSGF+.sh' file as below.

#!/bin/bash
#SBATCH -n 16
#SBATCH -p normal
#SBATCH -t 24:00:00

#SBATCH -o mg+.o%j
set -x

module load jdk64

MSGFplus_JAR="$HOME/git/MSblender/extern/MSGFPlus.jar"

DB="../DB/my_seq.combined.fa"

DBNAME=$(basename $DB)
DBNAME=${DBNAME/.fa/}

#SBATCH -J "mg+"
for MZXML in $(ls ../mzXML/*mzXML)
do
  OUT=$(basename $MZXML)
  OUT=${OUT/.mzXML/}"."$DBNAME".MSGF+.mzid"
  TBL=${OUT/.mzid/.tsv}
  time java -Xmx20000M -jar $MSGFplus_JAR -d $DB -s $MZXML -o $OUT -t 20ppm -tda 0 -ntt 2 -e 1 -inst 3
  time java -Xmx20000M -cp $MSGFplus_JAR edu.ucsd.msjava.ui.MzIDToTsv -i $OUT -o $TBL -showQValue 1 -showDecoy 1 -unroll 0
done

Then, submit the job by typing 'sbatch stampede-MSGF+.sh'

Run X!Tandem

$ cd $SCRATCH/myProject/tandemK
$ ~/git/MSblender/search/prepare-tandemK-high.py ../mzXML/ ../DB/my_seq.combined.fa.pro

First argument of prepare-tendemK-high.py is a directory for mzXML, and second one is for .pro database generated by fasta_pro.exe as above.

You will see *.xml files matched to your mzXML files (X!Tandem input), 'tandem-taxonomy.xml' file (another X!Tandem input), and run-tandemK.sh (a script to run X!tandem).

Make the following launcher ('stampede-tandemK.sh'), and submit it as 'sbatch stampede-tandemK.sh'.

#!/bin/bash
#SBATCH -n 16
#SBATCH -p normal
#SBATCH -t 24:00:00

#SBATCH -o tK.o%j
#SBATCH -J "tK"
set -x

bash ./run-tandemK.sh

If you have many mzXML files, you can run it parallel by splitting run-tandemK.sh with 'split -l' command, and run individual splitted script at 'stampede-tandemK.sh'.