.gitmodules

-[submodule "testiphy"]
-	path = testiphy
-	url = https://gitlab.com/testiphy/testiphy.git
-	ignore = untracked

.travis.yml

@@ -53,15 +53,25 @@ matrix:
 
 before_install:
   # We need to re-export CC and CXX here, because travis exports CXX=g++ or clang++ AFTER we set CXX.
+  - PATH=$HOME/local/bin:$PATH
+  - echo $PATH
   - if [ -n "${C_COMPILER}" ]; then export CC="${C_COMPILER}"; fi
   - if [ -n "${CXX_COMPILER}" ]; then export CXX="${CXX_COMPILER}"; fi
   - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew update && brew install meson; fi
   - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip && unzip -q ninja-linux.zip -d ~/bin && pyenv local 3.6 && pip3 install meson; fi
 
 script:
-  - meson build --prefix=$HOME/bali-phy
-  - ninja -C build test
+  - meson build --prefix=$HOME/local
   - ninja -C build install
+# Run internal test suite
+  - cd tests
+  - ./run-tests.py bali-phy
+# Run testiphy test suite
+  - cd
+  - git clone https://gitlab.com/testiphy/testiphy.git
+  - cd $HOME/testiphy
+  - which bali-phy || true
+  - ./testiphy bali-phy
 
 env:
   global:

NEWS

-* 3.2.0
+UNRELEASED
+ - Fixes
+ - 
+
+* 3.3 (Aug 6, 2018)
+  - Fixes
+    - make bp-analyze able to parse output files from 3.0-beta.
+    - rename Goldman & Yang codon model to gy94 from incorrect yn94.
+    - fix run file for f81 model.
+    - fix incorrect LG model.
+    - normalize WAG and LG standard frequencies.
+  - Codon models
+    - gy94 and mg94 now have no submodels and are rate matrices.
+    - f1x4, f3x4, and f61 now compute named codon frequencies.
+    - add gy94_ext and mg94_ext for using any nucleotide rate matrix.
+    - add mg94k for mg94_ext[hky85]
+  - Triplet models
+    - fix up x3, x3_xym, x3x3
+    - add +dNdS function so we can do e.g. hky85+x3+dNdS
+  - Functions in models
+    - Add new syntax function[x,...] for specifying models.
+    - new functions map, zip, zipWith.
+    - m1a,m2a,..,m8a_test,branch-site, etc. now take functions as arguments.
+    - let-bind names for lambda-dependent arguments, but inside the lambda.
+  - Model framework
+    - +fe as synonym for +f[Frequencies.uniform]
+    - +f and +gwF now take exchange models like wag as a submodel.
+    - logging now returns an object that can change shape dynamically.
+    - function calls can now look like f[g[x],y] instead of just f[x,y]
+    - arguments now referenced with @arg to fix function[x,add[x,x]]
+    - fixes to unification and constraints.
+    - suppress gamma:shift and exponential:shift
+  - Haskell
+    - modules can now re-export imported modules.
+    - refactor module SModel into SModel.Codons, SModel.ReversibleMarkov, etc.
+    - remove hky85', etc now that models can call hky85 directly.
+    - add lazy IO
+    - make sampling in the Random monad lazy
+    - correctly set rates in sample'
+    - do normal 0 1 => do sample $ normal 0 1
+  - Docs
+    - Reorganize docs on substitution models.
+    - Move docs on partitions to new section.
+  - Etc
+    - Allow specifying alphabet for --print.
+    - Allow running gy94 from --print.
+  - Testing
+    - Find out why different package disagree on wag+f[wag_freq]
+
+* 3.2 (Jun 25, 2018)
   - Fixes
     - Increase test timeout for internal testsuite and testiphy.
     - HTML report: don't hide header behind top-bar in Chrome.
   - alignment-smc improvements.
-* 3.1.5
+  - change alignment-diff back to red
+  - Add NEWS file.
+
+* 3.1.5 (Jun 13, 2018)
   - Fixes
     - Make all programs use shipped libstdc++
     - Make MDS plots handle bp-analyze --subsample
     - Increase test timeouts
     - Correctly write initial alignment for fixed alignment partitions.
     - Don't write "file:" for MDS URLs in HTML report
+  - Help
     - Print citations with pmid and pmcid in help.
-  - Add Frequencies.uniform[] function.
     - Print help for 0-argument functions like 'dna'
+    - Improve help for fMutSel and fMutSel0
+  - Add Frequencies.uniform[] function.
   - Make SEV handle site-compression.
   - HTML report: print version number + lots of cosmetic improvements.
   - Add new tool tree-tool (and map page, etc.)
   - Change alignments-diff highlight color back to red.
   - Clean up DP matrix code.
-* 3.1.4
+
+* 3.1.4 (Jun 9, 2018)
   - Fix prior on alpha in Rates.gamma
-* 3.1.3
+
+* 3.1.3 (Jun 9, 2018)
   - Fix mean_length prior in RS05 model.
   - Add some more color-schemes for drawing alignments-diff output.
   - Add ferns exon/intro data set.
-* 3.1.2
+
+* 3.1.2 (May 6, 2018)
   - Fix a testsuite bug
-* 3.1.1
+  - Haskell
+    - parse exports list
+
+* 3.1.1 (May 5, 2018
+  - Fixes
+    - Don't require testiphy
   - Print priors on models in their own section.
+
+* 3.1 (May 2, 2018)
+  - Fixes
+    - Build alignment-thin and alignments-diff
+  - Show model and priors readably
+  - bali-phy accepted into Debian
+  - Change 'logp' => 'posterior'
+
+* 3.0.2 (Mar 10, 2018)
+  - Docs
+    - man pages for bali-phy and tools
+    - man pages online
+
+* 3.0.1 (Mar 6, 2018)
+  - Fixes
+    - Fix crash in alignment-thin
+
+* 3.0 (Feb 12, 2018)
+  - Models
+    - Mixture models now work.
+    - MultiRate[model,dist,n_bins] also works
+    - let[var=E1,E2] now works
+    - Rename RA[a] to RevCTMC[a]
+    - Stop representing integer, double, and bool as strings.
+  - Report
+    - subsample trees when running trees-distances for MDS plot
+  - Docs
+    - Add help for more functions.
+  - MCMC
+    - Implement SPR+A
+    - Destroy SPR_by_NNI with fire.
+
+* 3.0-beta6 (Jan 5, 2018)
+  - Fixes
+    - Make sorting of DP:rates and M3:omegas work again.
+    - sample branch lengths from prior -- fixes very long initial tree.
+  - specify branch lengths as a List[Double]
+  - Models
+    - Make lists into a collection of Cons[ ] functions.h
+  + Speedup on 25-muscle because we aren't proposing the branch lengths badly
+  - Eliminate compiler warnings.
+  * Print expressions
+  - Allow non-modifiable branch lengths if we aren't doing MCMC.
+  - Improve pre-burnin
+  - read parameter values as json
+  - write out run parameters as json.
+  - Improve help, add help for more functions.
+  * stop finding branch lengths by parameter name
+  * stop finding scales by parameter name
+  * print parameter values that are data structures
+  * meson build infrastructure
+  * Read json using nlohmann::json
+  * Switch to new ptree structure.
+
+3.0-beta5 (Dec 6, 2018)
+  - Fixes
+    - Don't crash on --smodel=GTR+x3
+  - Short parameter names
+  - Add --link option.
+  - Complain if linking non-existant attributes.
+  - Logging
+    * Logging of [(String,a)]
+    * Rationalize logging of frequencies and exchangabilities
+    * DirichletOn prior for frequencies and exchangabilities.
+  * Loading functions from files.
+  - Allow expressing the branch length prior in terms of the tree.
+  - Automatically convert integers to double when needed, and hide the conversions.
+  - Help
+    - Make simple|advanced|expert|(developer) options.
+    - Help for all commands.
\ No newline at end of file

README.md

@@ -7,7 +7,7 @@ Install
 
 Please visit the [releases page](http://www.bali-phy.org/download.php) to download official binaries.
 
-You can also install via homebrew on a Mac.
+You can also install via homebrew on a Mac, and using `apt-get` on recent version of Debian or Ubuntu.
 
 Documentation
 ------------
@@ -30,25 +30,35 @@ You will need a C++ compiler that understands C++14.
 You will also need to install
  * cairo graphics library (optional, but required to build the `draw-tree` program)
 
-If you build with meson and ninja, you need
+You will also need
  * python3
  * ninja
+ * meson >= 0.45
 
-You need meson version >= 0.45 to build bali-phy.
-
-Build with meson
-----------------
+Install Prerequisites
+---------------------
 ```bash
 sudo apt-get install g++ libcairo2-dev ninja-build python3
+```
+You also need to install meson.  First try:
+```bash
+sudo apt-get install meson
+```
 
-git clone https://github.com/bredelings/BAli-Phy.git
-cd BAli-Phy/
-git submodule update --init
-
-# We need a recent version of meson
+If the version of meson is not at least 0.45, then you need to install
+meson through pip:
+```bash
 python3 -m venv meson
 source meson/bin/activate
 pip3 install meson
+```
+
+Build BAli-Phy
+--------------
+```
+git clone https://github.com/bredelings/BAli-Phy.git
+cd BAli-Phy/
+git submodule update --init   # This is optional, it allows running the testiphy testsuite
 
 meson build --prefix=$HOME/Applications/bali-phy
 ninja -C install
@@ -58,7 +68,7 @@ ninja -C test
 Adding bali-phy to your `$PATH`
 ------------------------------
 
-In order to run the installed software, you should [add bali-phy to your $PATH](http://bali-phy.org/README.xhtml#installation).
+In order to run the installed software, you should [add bali-phy to your $PATH](http://bali-phy.org/README.xhtml#path).
 
 Installed locations
 ------------------

debian/changelog

+bali-phy (3.3+dfsg-1) UNRELEASED; urgency=medium
+
+  * New upstream version
+
+ -- Benjamin Redelings <bredelings@telomere.biology.duke.edu>  Mon, 06 Aug 2018 13:12:11 -0700
+
 bali-phy (3.2+dfsg-1) unstable; urgency=medium
 
   * New upstream version

doc/GNUmakefile

-all: README.html README.xhtml README.pdf Tutorial.html Tutorial.xhtml Tutorial.pdf Tutorial2.html Tutorial2.xhtml Tutorial2.pdf Tutorial3.html Tutorial3.xhtml Tutorial3.pdf Tutorial4.html Tutorial4.xhtml Tutorial4.pdf
+all: README.html README.xhtml README.pdf Tutorial.html Tutorial.xhtml Tutorial.pdf Tutorial2.html Tutorial2.xhtml Tutorial2.pdf Tutorial3.html Tutorial3.xhtml Tutorial3.pdf Tutorial4.html Tutorial4.xhtml Tutorial4.pdf developer.html
 
 clean:
 	-@rm -f *.html *.xhtml REAMDE.xml *.fo *.pdf *~
@@ -13,6 +13,9 @@ clean:
 %.html: %.xml docbook-html.xsl
 	xsltproc --nonet --xinclude docbook-html.xsl $*.xml > $@
 
+%.html: %.md
+	pandoc -s $*.md --css doc.css --toc -N > $@
+
 %.xhtml: %.xml docbook-xhtml.xsl
 	xsltproc --nonet --xinclude docbook-xhtml.xsl $*.xml > $@
 

doc/Tutorial.html

@@ -152,8 +152,8 @@ Now lets use the analysis script to run all the summaries and make a report:
 <p>This is described in more detail in section 4.3 of the <a class="ulink" href="http://www.bali-phy.org/README.html" target="_top">manual</a>.</p>
 <p>
 Download HIV env and pol alignments with the same sequence names:
-</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>wget http://nucleus.biology.duke.edu/~bredelings/alignment_files/env-common.fasta</code></strong>
-<code class="prompt">%</code> <strong class="userinput"><code>wget http://nucleus.biology.duke.edu/~bredelings/alignment_files/pol-common.fasta</code></strong>
+</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>wget http:</code></strong>//nucleus.biology.duke.edu/~bredelings/alignment_files/env-common.fasta
+<code class="prompt">%</code> <strong class="userinput"><code>wget http:</code></strong>//nucleus.biology.duke.edu/~bredelings/alignment_files/pol-common.fasta
 </pre><p>Try to analyze these genes jointly under a codon model:
 </p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy --test --alphabet Codons env-common.fasta pol-common.fasta --smodel M0</code></strong>
 </pre><p>Take a look at the lengths of these files, to see whether the sequencealignment lengths are multiples of 3:

doc/Tutorial.xhtml

@@ -110,8 +110,8 @@ Now lets use the analysis script to run all the summaries and make a report:
 <code class="prompt">%</code> <strong class="userinput"><code>firefox Results/index.html &amp;</code></strong>
 </pre></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="idm189"></a>5. Multi-gene analyses</h2></div></div></div><p>This is described in more detail in section 4.3 of the <a class="link" href="http://www.bali-phy.org/README.html" target="_top">manual</a>.</p><p>
 Download HIV env and pol alignments with the same sequence names:
-</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>wget http://nucleus.biology.duke.edu/~bredelings/alignment_files/env-common.fasta</code></strong>
-<code class="prompt">%</code> <strong class="userinput"><code>wget http://nucleus.biology.duke.edu/~bredelings/alignment_files/pol-common.fasta</code></strong>
+</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>wget http:</code></strong>//nucleus.biology.duke.edu/~bredelings/alignment_files/env-common.fasta
+<code class="prompt">%</code> <strong class="userinput"><code>wget http:</code></strong>//nucleus.biology.duke.edu/~bredelings/alignment_files/pol-common.fasta
 </pre><p>Try to analyze these genes jointly under a codon model:
 </p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy --test --alphabet Codons env-common.fasta pol-common.fasta --smodel M0</code></strong>
 </pre><p>Take a look at the lengths of these files, to see whether the sequencealignment lengths are multiples of 3:

doc/Tutorial3.tut.xml

@@ -145,11 +145,11 @@ You can allow 5 different rates that are all independently estimated:
 <section><info><title>Codon models</title></info>
 <para>
 We can also conduct codon-based analyses using the Nielsen and Yang (1998) model of diversifying positive selection (dN/dS):
-% bali-phy Globins/bglobin.fasta --test -S yn94+f1x4
-The yn94 model takes a nucleotide exchange model as a parameter. This parameter is optional, and the default is hky85, which you could specify as yn94[,hky85_sym]. You can change this to be more flexible: 
-% bali-phy Globins/bglobin.fasta --test -S yn94[,gtr_sym]+f1x4
+% bali-phy Globins/bglobin.fasta --test -S gy94+f1x4
+The gy94 model takes a nucleotide exchange model as a parameter. This parameter is optional, and the default is hky85, which you could specify as gy94[,hky85_sym]. You can change this to be more flexible: 
+% bali-phy Globins/bglobin.fasta --test -S gy94[,gtr_sym]+f1x4
 You can make the codon frequencies to be generated from a single set of nucleotide frequencies:
-% bali-phy Globins/bglobin.fasta --test -S yn94[,gtr_sym]+mg94
+% bali-phy Globins/bglobin.fasta --test -S gy94[,gtr_sym]+mg94
 The M7 model allows different sites to have different dN/dS values, where the probability of dN/dS values follows a beta distribution:
 % bali-phy Globins/bglobin.fasta --test -S m7
 The M7 model has parameters as well. Here are the defaults:

doc/Tutorial3.xhtml

@@ -7,9 +7,9 @@ Before you start this tutorial, please <a class="link" href="http://www.bali-phy
 </pre><p>Go into the <code class="filename">alignment_files</code> directory:
 </p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>cd alignment_files</code></strong>
 </pre><p>Download the example alignment files:
-</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>wget http://www.bali-phy.org/examples.tgz</code></strong>
+</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>wget http:</code></strong>//www.bali-phy.org/examples.tgz
 </pre><p>Alternatively, you can use <span class="command"><strong>curl</strong></span>
-</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>curl -O http://www.bali-phy.org/examples.tgz</code></strong>
+</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>curl -O http:</code></strong>//www.bali-phy.org/examples.tgz
 </pre><p>Extract the compressed archive:
 </p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>tar -zxf examples.tgz</code></strong>
 </pre><p>Take a look inside the <code class="filename">examples</code> directory:
@@ -85,11 +85,11 @@ You can also allow different sites to evolve at 5 different rates using the gamm
 </p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy 5S-rRNA/25-muscle.fasta --test -S gtr+Rates.free[n=5]</code></strong>
 </pre></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="idm189"></a>6.3. Codon models</h3></div></div></div><p>
 We can also conduct codon-based analyses using the Nielsen and Yang (1998) model of diversifying positive selection (dN/dS):
-</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy Globins/bglobin.fasta --test -S yn94+f1x4</code></strong>
-</pre><p>The yn94 model takes a nucleotide exchange model as a parameter. This parameter is optional, and the default is hky85, which you could specify as yn94[,hky85_sym]. You can change this to be more flexible: 
-</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy Globins/bglobin.fasta --test -S yn94[,gtr_sym]+f1x4</code></strong>
+</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy Globins/bglobin.fasta --test -S gy94+f1x4</code></strong>
+</pre><p>The gy94 model takes a nucleotide exchange model as a parameter. This parameter is optional, and the default is hky85, which you could specify as gy94[,hky85_sym]. You can change this to be more flexible: 
+</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy Globins/bglobin.fasta --test -S gy94[,gtr_sym]+f1x4</code></strong>
 </pre><p>You can make the codon frequencies to be generated from a single set of nucleotide frequencies:
-</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy Globins/bglobin.fasta --test -S yn94[,gtr_sym]+mg94</code></strong>
+</p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy Globins/bglobin.fasta --test -S gy94[,gtr_sym]+mg94</code></strong>
 </pre><p>The M7 model allows different sites to have different dN/dS values, where the probability of dN/dS values follows a beta distribution:
 </p><pre class="screen"><code class="prompt">%</code> <strong class="userinput"><code>bali-phy Globins/bglobin.fasta --test -S m7</code></strong>
 </pre><p>The M7 model has parameters as well. Here are the defaults:

doc/developer.md

@@ -151,7 +151,7 @@ Input:
 
 * The function takes a single `OperationArgs& Args` argument.
 * The `n`th argument is fetched by calling `Args.evaluate(n)`, and is of type `expression_ref` ([src/computation/expression/expression_ref.H](https://github.com/bredelings/BAli-Phy/blob/master/src/computation/expression/expression_ref.H))
-* The `expression_ref` can be converted to `int', `double`, or `log_double_t` using the methods `.as_int()`, `.as_double()` and `.as_log_double()`.
+* The `expression_ref` can be converted to `int`, `double`, or `log_double_t` using the methods `.as_int()`, `.as_double()` and `.as_log_double()`.
 
 Output: