Difference between revisions of "JBrowse Tutorial 2013"
RobertBuels (Talk | contribs) (→JBrowse Introduction) |
m (Adding has topic) |
||
(16 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
− | This [[JBrowse]] tutorial was presented by [[User:RobertBuels|Robert Buels]] at the [[2013 GMOD Summer School]] in July 2013. | + | This [[Has topic::JBrowse]] tutorial was presented by [[User:RobertBuels|Robert Buels]] at the [[2013 GMOD Summer School]] in July 2013 using JBrowse 1.9.8. |
+ | |||
+ | This tutorial uses the AWS AMI '''ami-d2c8b1bb''' named '''2013 GMOD start day 3''' in the US-East region. | ||
== Prerequisites == | == Prerequisites == | ||
Line 19: | Line 21: | ||
This is how they were installed: <b>(don't do this yourself)</b> | This is how they were installed: <b>(don't do this yourself)</b> | ||
<pre class="dont"> | <pre class="dont"> | ||
− | + | sudo apt-get install libpng12-0 libpng12-dev build-essential libncurses5-dev | |
</pre> | </pre> | ||
</div> | </div> | ||
Line 41: | Line 43: | ||
* prepare a directory for JBrowse | * prepare a directory for JBrowse | ||
− | + | <span class="enter">cd /var/www</span> | |
− | + | <span class="enter">sudo mkdir jbrowse_demo</span> | |
− | + | <span class="enter">sudo chown ubuntu.ubuntu jbrowse_demo</span> | |
− | + | <span class="enter">cd jbrowse_demo</span> | |
* download the demo bundle from jbrowse.org and unzip it | * download the demo bundle from jbrowse.org and unzip it | ||
− | + | <span class="enter">wget http://jbrowse.org/info/GMOD_Jul_2013/GMOD_Summer_School_2013_JBrowse.zip</span> | |
− | + | <span class="enter">unzip GMOD_Summer_School_2013_JBrowse.zip</span> | |
− | + | <span class="enter">cd GMOD_Summer_School_2013_JBrowse/</span> | |
− | + | <span class="enter">unzip JBrowse-1.9.8.zip</span> | |
− | + | <span class="enter">mv JBrowse-1.9.8 jbrowse</span> | |
* run <code>setup.sh</code> to configure this copy of JBrowse | * run <code>setup.sh</code> to configure this copy of JBrowse | ||
− | + | <span class="enter">cd jbrowse</span> | |
− | + | <span class="enter">./setup.sh</span> | |
=== Starting Point === | === Starting Point === | ||
− | Visit in web browser: <nowiki>http://</nowiki>{{Template:AWSurl}}/jbrowse_demo/jbrowse/ | + | Visit in web browser: |
+ | |||
+ | <nowiki>http://</nowiki>{{Template:AWSurl}}/jbrowse_demo/GMOD_Summer_School_2013_JBrowse/jbrowse/index.html | ||
You should see a "Congratulations" page. | You should see a "Congratulations" page. | ||
Line 68: | Line 72: | ||
There are four basic steps to setting up an instance of JBrowse: | There are four basic steps to setting up an instance of JBrowse: | ||
− | # | + | # Load and format reference sequences |
# Format data for tracks | # Format data for tracks | ||
# Configure direct-access tracks | # Configure direct-access tracks | ||
Line 79: | Line 83: | ||
<pre> | <pre> | ||
− | + | dropdb chado | |
− | + | createdb chado | |
− | + | bzip2 -cd ~/Documents/Software/schema/chado/complete_db.bz2 | psql chado | |
</pre> | </pre> | ||
--> | --> | ||
Line 108: | Line 112: | ||
Run this from within the <tt>jbrowse</tt> directory (you could run it elsewhere, but you'd have to explicitly specify the location of the data directory on the command line). | Run this from within the <tt>jbrowse</tt> directory (you could run it elsewhere, but you'd have to explicitly specify the location of the data directory on the command line). | ||
− | + | <span class="enter">cd /var/www/jbrowse_demo/GMOD_Summer_School_2013_JBrowse/jbrowse</span> | |
− | + | <span class="enter">bin/prepare-refseqs.pl --gff ../scf1117875582023.gff</span> | |
Refresh it in your web browser, you should new see the JBrowse UI and a sequence track, which will show you the DNA base pairs if you zoom in far enough. | Refresh it in your web browser, you should new see the JBrowse UI and a sequence track, which will show you the DNA base pairs if you zoom in far enough. | ||
Line 156: | Line 160: | ||
Run the <tt>bin/biodb-to-json.pl</tt> script with this config file to format this track, and the others in the file: | Run the <tt>bin/biodb-to-json.pl</tt> script with this config file to format this track, and the others in the file: | ||
− | + | <span class="enter">bin/biodb-to-json.pl --conf ../pythium-1.conf</span> | |
Refresh JBrowse in your web browser. You should now see a bunch of annotation tracks. | Refresh JBrowse in your web browser. You should now see a bunch of annotation tracks. | ||
Line 166: | Line 170: | ||
The <tt>bin/generate-names.pl</tt> script collects those lists of names from all the tracks and combines them into one big tree that the client uses to search. | The <tt>bin/generate-names.pl</tt> script collects those lists of names from all the tracks and combines them into one big tree that the client uses to search. | ||
− | + | <span class="enter">bin/generate-names.pl -v</span> | |
Visit in web browser, try typing a feature name, such as '''maker-scf1117875582023-snap-gene-0.26-mRNA-1'''. Notice that JBrowse tries to auto-complete what you type. | Visit in web browser, try typing a feature name, such as '''maker-scf1117875582023-snap-gene-0.26-mRNA-1'''. Notice that JBrowse tries to auto-complete what you type. | ||
Line 176: | Line 180: | ||
* We'll add a RepeatMasker track: | * We'll add a RepeatMasker track: | ||
− | + | <span class="enter">bin/flatfile-to-json.pl --trackLabel repeatmasker \ | |
− | --type match:repeatmasker | + | --type match:repeatmasker --key RepeatMasker \ |
--className generic_parent \ | --className generic_parent \ | ||
--subfeatureClasses '{"match_part" : "feature"}' --gff ../repeats.gff</span> | --subfeatureClasses '{"match_part" : "feature"}' --gff ../repeats.gff</span> | ||
Line 183: | Line 187: | ||
* And then a RepeatRunner track: | * And then a RepeatRunner track: | ||
− | + | <span class="enter">bin/flatfile-to-json.pl --trackLabel repeatrunner \ | |
− | --type protein_match:repeatrunner | + | --type protein_match:repeatrunner \ |
--key RepeatRunner --className generic_parent \ | --key RepeatRunner --className generic_parent \ | ||
--subfeatureClasses '{"match_part" : "feature"}' --gff ../repeats.gff</span> | --subfeatureClasses '{"match_part" : "feature"}' --gff ../repeats.gff</span> | ||
Line 206: | Line 210: | ||
This can be added by either editing the <tt>data/trackList.json</tt> file with a text editor, or by running something like this at the command line to inject the track configuration: | This can be added by either editing the <tt>data/trackList.json</tt> file with a text editor, or by running something like this at the command line to inject the track configuration: | ||
− | + | <span class="enter">echo '{ | |
"label" : "bam_alignments", | "label" : "bam_alignments", | ||
"key" : "BAM alignments", | "key" : "BAM alignments", | ||
Line 234: | Line 238: | ||
<syntaxhighlight lang="javascript"> | <syntaxhighlight lang="javascript"> | ||
{ | { | ||
− | "label" : " | + | "label" : "bigwig_bam_coverage", |
− | "key" : "BAM coverage", | + | "key" : "BigWig - BAM coverage", |
"storeClass" : "BigWig", | "storeClass" : "BigWig", | ||
"urlTemplate" : "../../simulated-sorted.bam.coverage.bw", | "urlTemplate" : "../../simulated-sorted.bam.coverage.bw", | ||
Line 245: | Line 249: | ||
This can be added by either editing the <tt>data/trackList.json</tt> file with a text editor, or by running something like this at the command line to inject the track configuration: | This can be added by either editing the <tt>data/trackList.json</tt> file with a text editor, or by running something like this at the command line to inject the track configuration: | ||
− | + | <span class="enter">echo ' { | |
"label" : "bam_coverage", | "label" : "bam_coverage", | ||
"key" : "BAM coverage", | "key" : "BAM coverage", | ||
Line 272: | Line 276: | ||
This can be added by either editing the <tt>data/trackList.json</tt> file with a text editor, or by running something like this at the command line to inject the track configuration: | This can be added by either editing the <tt>data/trackList.json</tt> file with a text editor, or by running something like this at the command line to inject the track configuration: | ||
− | + | <span class="enter">echo ' { | |
"label" : "bam_variation", | "label" : "bam_variation", | ||
"key" : "VCF simulated variation", | "key" : "VCF simulated variation", | ||
Line 290: | Line 294: | ||
The demo bundle contains an example <tt>trackMetadata.csv</tt> file, which can be copied into the <tt>data</tt> directory for use with this configuration. | The demo bundle contains an example <tt>trackMetadata.csv</tt> file, which can be copied into the <tt>data</tt> directory for use with this configuration. | ||
− | + | <span class="enter">cp trackMetadata.csv jbrowse/data</span> | |
Then a simple faceted track selection configuration might look like: | Then a simple faceted track selection configuration might look like: | ||
Line 331: | Line 335: | ||
== Common Problems == | == Common Problems == | ||
− | * JSON syntax errors | + | * JSON syntax errors in configuration files (2.x series will stop this madness!) |
− | + | ||
== Future JBrowse Plans == | == Future JBrowse Plans == | ||
− | See the [[Media: | + | See the [[Media:JBrowse_gmod_summerschool_jul2013.pdf|accompanying slides (PDF)]] |
− | + | ||
== Other links == | == Other links == | ||
Line 343: | Line 345: | ||
* Config file ref: http://jbrowse.org/code/jbrowse-master/docs/config.html | * Config file ref: http://jbrowse.org/code/jbrowse-master/docs/config.html | ||
* DIV test: http://jbrowse.org/test/boatdiv/boat.html | * DIV test: http://jbrowse.org/test/boatdiv/boat.html | ||
+ | |||
+ | [[Category:Tutorials]] | ||
+ | [[Category:JBrowse]] |
Latest revision as of 20:48, 27 August 2013
This JBrowse tutorial was presented by Robert Buels at the 2013 GMOD Summer School in July 2013 using JBrowse 1.9.8.
This tutorial uses the AWS AMI ami-d2c8b1bb named 2013 GMOD start day 3 in the US-East region.
Contents
- 1 Prerequisites
- 2 JBrowse Introduction
- 3 Setting up JBrowse
- 4 JBrowse Features
- 5 Upgrading an Existing JBrowse
- 6 Common Problems
- 7 Future JBrowse Plans
- 8 Other links
Prerequisites
These have already been set up on the VM image.
Optional, for generating images from Wiggle files:
- libpng12-0
- libpng12-dev
- a C++ compiler
Optional, for BAM files (setup.sh tries to install these for you in the JBrowse directory):
- samtools, and its dependency libncurses5-dev
- perl module: Bio::DB::SAM
Other prerequisites are installed by JBrowse automatically.
This is how they were installed: (don't do this yourself)
sudo apt-get install libpng12-0 libpng12-dev build-essential libncurses5-dev
Make sure you can copy/paste from the wiki.
It's also very useful to know how to tab-complete in the shell.
JBrowse Introduction
How and why JBrowse is different from most other web-based genome browsers, including GBrowse.
More detail: paper
Setting up JBrowse
Getting JBrowse
- prepare a directory for JBrowse
cd /var/www sudo mkdir jbrowse_demo sudo chown ubuntu.ubuntu jbrowse_demo cd jbrowse_demo
- download the demo bundle from jbrowse.org and unzip it
wget http://jbrowse.org/info/GMOD_Jul_2013/GMOD_Summer_School_2013_JBrowse.zip unzip GMOD_Summer_School_2013_JBrowse.zip cd GMOD_Summer_School_2013_JBrowse/ unzip JBrowse-1.9.8.zip mv JBrowse-1.9.8 jbrowse
- run
setup.sh
to configure this copy of JBrowse
cd jbrowse ./setup.sh
Starting Point
Visit in web browser:
http://ec2-##-##-##-##.compute-1.amazonaws.com/jbrowse_demo/GMOD_Summer_School_2013_JBrowse/jbrowse/index.html
You should see a "Congratulations" page.
Basic Steps
There are four basic steps to setting up an instance of JBrowse:
- Load and format reference sequences
- Format data for tracks
- Configure direct-access tracks
- Index feature names
Features from a directory of files
Here, we'll use the Bio::DB::SeqFeature::Store adaptor in "memory" mode to read a directory of files. There are adaptors available for use with many other databases, such as Chado and Bio::DB::GFF.
Config file: pythium-1.conf
{ "description": "GMOD Summer School 2013 P. ultima Example", "db_adaptor": "Bio::DB::SeqFeature::Store", "db_args" : { "-adaptor" : "memory", "-dir" : ".." }, ...
Specify reference sequences
The first script to run is bin/prepare-refseqs.pl; that script is the way you tell JBrowse about what your reference sequences are. Running bin/prepare-refseqs.pl also sets up the "DNA" track.
Run this from within the jbrowse directory (you could run it elsewhere, but you'd have to explicitly specify the location of the data directory on the command line).
cd /var/www/jbrowse_demo/GMOD_Summer_School_2013_JBrowse/jbrowse bin/prepare-refseqs.pl --gff ../scf1117875582023.gff
Refresh it in your web browser, you should new see the JBrowse UI and a sequence track, which will show you the DNA base pairs if you zoom in far enough.
Load Feature Data
Next, we'll use biodb-to-json.pl to get feature data out of the database and turn it into JSON data that the web browser can use.
In this case, we have specified all of our track configurations in pythium-1.conf
.
... "TRACK DEFAULTS": { "class": "feature" }, "tracks": [ { "track": "Genes", "key": "Genes", "feature": ["mRNA"], "autocomplete": "all", "class": "transcript", "subfeature_classes" : { "CDS" : "transcript-CDS", "UTR" : "transcript-UTR" } }, ... ]
track specifies the track identifier (a unique name for the track, for the software to use). This should be just letters and numbers and - and _ characters; using other characters makes things less convenient.
key specifies a human-friendly name for the track, which can use any characters you want.
feature gives a list of feature types to include in the track.
autocomplete including this setting makes the features in the track searchable.
urltemplate specifies a URL pattern that you can use to link genomic features to specific web pages.
class specifies the CSS class that describes how the feature should look.
For this particular track, I've specified the transcript feature class.
Run the bin/biodb-to-json.pl script with this config file to format this track, and the others in the file:
bin/biodb-to-json.pl --conf ../pythium-1.conf
Refresh JBrowse in your web browser. You should now see a bunch of annotation tracks.
Index feature names
When you generate JSON for a track, if you specify "autocomplete" then a listing of all of the feature names from that track (along with feature locations) will also be generated and used to provide feature searching and autocompletion.
The bin/generate-names.pl script collects those lists of names from all the tracks and combines them into one big tree that the client uses to search.
bin/generate-names.pl -v
Visit in web browser, try typing a feature name, such as maker-scf1117875582023-snap-gene-0.26-mRNA-1. Notice that JBrowse tries to auto-complete what you type.
Features from GFF3 or BED files
We're going to add a couple more tracks that come from a flat file, repeats.gff. To get feature data from flat files into JBrowse, we use flatfile-to-json.pl.
- We'll add a RepeatMasker track:
bin/flatfile-to-json.pl --trackLabel repeatmasker \
--type match:repeatmasker --key RepeatMasker \
--className generic_parent \
--subfeatureClasses '{"match_part" : "feature"}' --gff ../repeats.gff
- And then a RepeatRunner track:
bin/flatfile-to-json.pl --trackLabel repeatrunner \
--type protein_match:repeatrunner \
--key RepeatRunner --className generic_parent \
--subfeatureClasses '{"match_part" : "feature"}' --gff ../repeats.gff
Visit in web browser; you should see the two new RepeatMasker and RepeatRunner tracks.
BAM alignments
JBrowse can display alignments directly from a BAM file on your web server. Simply place the BAM file in a directory accessible to your web server, and add a snippet of configuration to JBrowse to add the track, similar to:
{ "label" : "bam_alignments", "key" : "BAM alignments", "storeClass" : "JBrowse/Store/SeqFeature/BAM", "urlTemplate" : "../../simulated-sorted.bam", "type" : "Alignments2" }
This can be added by either editing the data/trackList.json file with a text editor, or by running something like this at the command line to inject the track configuration:
echo '{
"label" : "bam_alignments",
"key" : "BAM alignments",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../../simulated-sorted.bam",
"type" : "Alignments2"
}' | bin/add-track-json.pl data/trackList.json
BAM coverage
{ "label" : "bam_coverage", "key" : "BAM Coverage", "storeClass" : "JBrowse/Store/SeqFeature/BAM", "urlTemplate" : "../../simulated-sorted.bam", "type" : "SNPCoverage" }
Quantitative data
BigWig
JBrowse can display quantitative data directly from a BigWig file on your web server. Simply place the BigWig file in a directory accessible to your web server, and add a snippet of configuration to JBrowse to add the track, similar to:
{ "label" : "bigwig_bam_coverage", "key" : "BigWig - BAM coverage", "storeClass" : "BigWig", "urlTemplate" : "../../simulated-sorted.bam.coverage.bw", "type" : "JBrowse/View/Track/Wiggle/XYPlot", "variance_band" : true }
This can be added by either editing the data/trackList.json file with a text editor, or by running something like this at the command line to inject the track configuration:
echo ' {
"label" : "bam_coverage",
"key" : "BAM coverage",
"storeClass" : "BigWig",
"urlTemplate" : "../../simulated-sorted.bam.coverage.bw",
"type" : "JBrowse/View/Track/Wiggle/XYPlot",
"variance_band" : true
} ' | bin/add-track-json.pl data/trackList.json
Variation Data
VCF tracks
JBrowse can also display VCF variation data directly from a VCF file on your web server that has been compressed with Heng Li's bgzip and tabix. Simply place the .vcf.gz and .vcf.gz.tbi files in a directory accessible to your web server, and add a snippet of configuration to JBrowse to add the track, similar to:
{ "label" : "bam_variation", "key" : "VCF simulated variation", "storeClass" : "JBrowse/Store/SeqFeature/VCFTabix", "urlTemplate" : "../../simulated-sorted.vcf.gz", "type" : "HTMLVariants" }
This can be added by either editing the data/trackList.json file with a text editor, or by running something like this at the command line to inject the track configuration:
echo ' {
"label" : "bam_variation",
"key" : "VCF simulated variation",
"storeClass" : "JBrowse/Store/SeqFeature/VCFTabix",
"urlTemplate" : "../../simulated-sorted.vcf.gz",
"type" : "HTMLVariants"
} ' | bin/add-track-json.pl data/trackList.json
Faceted Track Selection
JBrowse has a very powerful faceted track selector that can be used to search for tracks using metadata associated with them.
The track metadata is kept in a CSV-format file, with any number of columns, and with a "label" column whose contents must correspond to the track labels in the JBrowse configuration.
The demo bundle contains an example trackMetadata.csv file, which can be copied into the data directory for use with this configuration.
cp trackMetadata.csv jbrowse/data
Then a simple faceted track selection configuration might look like:
trackSelector: { type: 'Faceted', }, trackMetadata: { sources: [ { type: 'csv', url: 'data/trackMetadata.csv' } ] }
The jbrowse_conf.json file in the jbrowse directory already conveniently contains this stanza, commented out. Uncomment it, refresh your browser, and you should now see the faceted track selector activated.
JBrowse Features
Highlighting interesting things
To highlight a region, you can either right-click on a feature and select 'highlight this', or you can set the highlight explicitly to a certain genomic region by clicking "View -> Set highlight" in the menu bar.
Beginning in JBrowse 1.10.0 you can also highlight a region with the mouse by clicking the highlighter tool (next to the Go button) and clicking and dragging to highlight a region.
Opening local files
JBrowse can display GFF3, BAM, BigWig, and VCF+Tabix files directly from your local machine without the need to transfer any data to the server. Just use the "File -> Open" tool from the menu bar to add tracks using local files.
Combination tracks (coming in 1.10.0)
Starting in version 1.10.0, users can define tracks that are combinations of the data in other tracks. The operations used to combine these tracks can be set operations (union, intersection, subtraction), arithmetic operations for quantitative tracks (addition, subtraction, multiplication, division), and/or masking operations to just highlight or mask some regions based on data in another track.
To add a combination track, select "File->Add combination track" from the menu bar, and drag existing tracks into the new combination track to start combining them.
Upgrading an Existing JBrowse
If the old JBrowse is 1.3.0 or later, simply move the data directory from the old JBrowse directory into the new JBrowse directory.
Common Problems
- JSON syntax errors in configuration files (2.x series will stop this madness!)
Future JBrowse Plans
See the accompanying slides (PDF)
Other links
- Config file ref: http://jbrowse.org/code/jbrowse-master/docs/config.html
- DIV test: http://jbrowse.org/test/boatdiv/boat.html