.gitignore

@@ -23,3 +23,6 @@
 /.pydevproject
 *.stash
 tmp.kfg
+*/xcuserdata/*
+.vscode
+.gdb_history

CHANGES.md

 # NCBI External Developer Release:
 
+## SRA Toolkit 2.10.0
+**August 19, 2019**
+
+  **bam-load**: SAM/BAM record sizes are limited only by available memory  
+  **fastq-dump**: fixed help text  
+  **kfg, sra-tools**: use trace.ncbi.nlm.nih.gov to call names service  
+  **kfg, vdb-config**: alternative remote repository URL was added to default configuration   
+  **kfs**: introduced readahead strategy for cloud storage  
+  **klib, vdb**: error report is saved to ncbi_error_report.txt  
+  **kns**: We now use system root CA certs on Unix   
+  **kns**: introduced configurable controls over network timeouts  
+  **prefetch**: fixed crash when run with --output-file option  
+  **prefetch**: updated prefetch help text  
+  **prefetch, srapath**: support for original submission files in cloud storage
+  **prefetch, srapath, vfs**: added possibility to specify data location  
+  **prefetch, vdb**: adjustments for latest name resolution service  
+  **prefetch, vfs**: added support of "run accession as directory"  
+  **prefetch, vfs**: added support of download of reference sequences in "run accession as directory"  
+  **prefetch, vfs**: fixed regression when prefetch does not download vdbcache  
+  **sra-stat**: don't print mismatch warning for old runs having a single SEQUENCE table and CMP_BASE_COUNT=BIO_BASE_COUNT while it should be 0  
+  **sra-stat**: fixed a bug when processing runs having spots with first reads with length 0  
+  **srapath, sratools**: Added support of "run accession as directory"  
+  **sratools**: SRA tools are executed by a driver tool called sratools  
+  **sratools**: installation creates symlinks to invoke some tools via a driver tool (sratools)  
+  **test**: created a standard way to detect ascp availability  
+  **tui, vdb-config**: new look and cloud specific options in 'vdb-config -i'  
+  **vdb**: make greater use of data returned by latest name resolver  
+  **vdb-validate**: referential integrity checker prints out completion message, if it printed out any progress  
+  **vfs**: added possibility to set resolver version from configuration  
+  **vfs**: allow to use SDL as remote service  
+
+
+## SRA Toolkit 2.9.6
+**March 18, 2019**
+
+  **prefetch**, **vfs**: fixed regression that prevented re-download of incomplete files  
+
+
+## SRA Toolkit 2.9.5
+**March 6, 2019**
+
+  **prefetch**: fixed regression that caused download of incomplete files  
+
+
+## SRA Toolkit 2.9.4-1
+**March 4, 2019**
+
+  **sra-tools, vfs**: fixed regression introduced in 2.9.4 release thas caused delay when starting sra tools  
+
+
+## SRA Toolkit 2.9.4
+**January 31, 2019**
+
+  **fasterq-dump**: improved handling of temp files in case of multiple instances  
+  **fasterq-dump**: produces same output as fastq-dump on SRR000001 (empty reads)  
+  **fastq-dump**: updated typo in error report  
+  **sra-tools, vfs**: added support of realign objects 
+
 
 ## SRA Toolkit 2.9.3
-**October 17, 2018**
+**October 11, 2018**
+
   **kns**: added possibility to skip server's certificate validation  
   **kns**: expect to receive HTTP status 200 when sending range-request that includes the whole file  
-  **vdb, vdb-copy**: fixed a bug in accessing pagemap process request for cursors which do not have pagemap thread running
+  **vdb**: fixed a bug in accessing pagemap process request for cursors which do not have pagemap thread running 
 
 
 ## SRA Toolkit 2.9.2-2
 **September 26, 2018**
+
   **read-filter-redact**: Fixed to update HISTORY metadata
 
 
 ## SRA Toolkit 2.9.2
 **July 23, 2018**
+
   **kfg, vfs**: Introduced enhanced handling of download-only NGC files that lack read/decrypt permissions
 
 

README.md

 # The NCBI SRA (Sequence Read Archive)
 
 ### Contact:
-email: sra-tools@ncbi.nlm.nih.gov
-or visit the [SRA Tools web site](http://ncbi.github.io/sra-tools)
+email: sra@ncbi.nlm.nih.gov
 
 ### Download
 Visit our [download page](https://github.com/ncbi/sra-tools/wiki/Downloads) for pre-built binaries.
@@ -15,25 +14,14 @@ The SRA Toolkit and SDK from NCBI is a collection of tools and libraries for
 using data in the INSDC Sequence Read Archives.
 
 ### ANNOUNCEMENT:
+With release 2.10.0 of `sra-tools` we have added cloud-native operation for AWS and GCP environments _(Linux only for this release)_, for use with the public SRA. `prefetch` is capable of retrieving original submission files in addition to ETL data.
+
 With release 2.9.1 of `sra-tools` we have finally made available the tool `fasterq-dump`, a replacement for the much older `fastq-dump` tool. As its name implies, it runs faster, and is better suited for large-scale conversion of SRA objects into FASTQ files that are common on sites with enough disk space for temporary files. `fasterq-dump` is multi-threaded and performs bulk joins in a way that improves performance as compared to `fastq-dump`, which performs joins on a per-record basis _(and is single-threaded)_.
 
 `fastq-dump` is still supported as it handles more corner cases than `fasterq-dump`, but it is likely to be deprecated in the future.
 
 You can get more information about `fasterq-dump` in our Wiki at [https://github.com/ncbi/sra-tools/wiki/HowTo:-fasterq-dump](https://github.com/ncbi/sra-tools/wiki/HowTo:-fasterq-dump).
 
-### IMPORTANT ANNOUNCEMENT
-As was announced in [OMB memorandum M-15-13](https://www.whitehouse.gov/sites/default/files/omb/memoranda/2015/m-15-13.pdf) on June 8, 2015, NCBI and all Federal systems will be transitioning to using HTTPS-only protocols before the end of 2016. This change will affect any software that uses NCBI APIs such as the E-utilities or NCBI software toolkits such as `sra-tools`, `ncbi-vdb` or `ngs`.
-
-The NLM and NCBI may implement the switch to HTTPS-only as early as September 30, 2016.
-
-In particular, software products that depend on `sra-tools`, `ncbi-vdb` or `ngs` may not function as expected after September 30 unless they are properly updated from this site or by the software provider.
-
-If you use software that accesses NCBI SRA data in any way, your software will likely be affected by this change. Please check with your software provider for recent udpates or patches, and be sure to acquire these before September 30.
- 
-If you develop software that relies on `sra-tools`, `ncbi-vdb` or `ngs` in any way, you will likely need to update your code so that it accesses NCBI using HTTPS.
-
-We have released new tools with version 2.8.0 that are HTTPS compatible and `M-15-13` compliant as of October 7, 2016. Please be certain to [update all of your binaries](https://github.com/ncbi/sra-tools/wiki/Downloads) and configuration files.
-
 For additional information on using, configuring, and building the toolkit,
 please visit our [wiki](https://github.com/ncbi/sra-tools/wiki)
 or our web site at [NCBI](http://www.ncbi.nlm.nih.gov/Traces/sra/?view=toolkit_doc)

build/Makefile.env

@@ -365,13 +365,23 @@ LDFLAGS = $(DBG) $(PROF) $(CARCH) $(MIN_DEPLOY_OS_OPT) $(LDPATHS)
 #
 # MallocScribble=1 is for catching allocation problems on Mac
 #
+HOSTNAME = $(shell hostname)
+WHICHASCP = $(shell which ascp 2> /dev/null)
+HAVE_NCBI_ASCP=1
+ifeq (,${WHICHASCP})
+	HAVE_NCBI_ASCP=
+endif
+ifeq (iebdev21,${HOSTNAME})
+	HAVE_NCBI_ASCP=
+endif
+
 ifeq ($(RUNTESTS_OVERRIDE),)
 runtests: all $(TEST_TOOLS)
 	@ export VDB_CONFIG=$(VDB_CONFIG);export LD_LIBRARY_PATH=$(LIBDIR):$$LD_LIBRARY_PATH;export MallocScribble=1;\
 	for i in $(TEST_TOOLS);\
 	do\
 		echo ++++++++++++++++++++++++++++++++++++++++++++++++++++++;\
-		echo Run $(TEST_BINDIR)/$$i;eval $(RUN_REMOTELY) $(TEST_BINDIR)/$$i;r=$$?; \
+		echo Run VDB_CONFIG=$$VDB_CONFIG $(TEST_BINDIR)/$$i;eval $(RUN_REMOTELY) $(TEST_BINDIR)/$$i;r=$$?; \
 		if [ "$$r" != "0" ] ; then exit $$r; fi; \
 	done
 

build/Makefile.install

@@ -35,10 +35,11 @@ include $(TOP)/build/Makefile.$(OS)
 # install
 #
 
-# $TOOLS_TO_INSTALL
+# TOOLS_TO_COPY, TOOLS_TO_SYMLINK 
+# $TOOLS_TO_INSTALL = TOOLS_TO_COPY + TOOLS_TO_SYMLINK
 include $(TOP)/build/Makefile.tools
 ifeq (1,$(HAVE_HDF5))
-        TOOLS_TO_INSTALL += pacbio-load
+        TOOLS_TO_COPY += pacbio-load
 endif
 
 # unset outside defined variables
@@ -68,16 +69,38 @@ installexes:
 	@ echo "Installing executables to $(INST_BINDIR)/..."
 	@ mkdir -p $(INST_BINDIR)
 	@ $(MAKE) -s --no-print-directory -f $(TOP)/build/Makefile.install $(TOOLS_TO_INSTALL)
+	@ # copy the driver 
+	 $(COPY) $(TOP)/tools/driver-tool/sratools.pl $(INST_BINDIR)/sratools$(VERSION_EXEX)
+	 ln -f -s sratools$(VERSION_EXEX) $(INST_BINDIR)/sratools$(MAJVERS_EXEX)
+	 ln -f -s sratools$(MAJVERS_EXEX) $(INST_BINDIR)/sratools$(EXEX)
+	@ #
 	@ printf "Installing magic file..."
 	@ $(COPY) $(TOP)/tools/copycat/magic $(INST_BINDIR) && echo "done"
 
-$(TOOLS_TO_INSTALL):
+$(TOOLS_TO_COPY):
+	@ printf "Installing $@..."
+	@ # report missing tools
+	@ ( test -f $(BINDIR)/$@$(EXEX) || ( echo "not found" && false ) ) && \
+      $(COPY) $(BINDIR)/$@$(EXEX) $(BINDIR)/$@$(VERSION_EXEX) $(BINDIR)/$@$(MAJVERS_EXEX) $(INST_BINDIR) && \
+      echo "done"
+
+$(TOOLS_TO_COPY_OPTIONAL):
 	@ printf "Installing $@..."
 	@ # ignore missing tools
 	@ - ( test -f $(BINDIR)/$@$(EXEX) || ( echo "not found" && false ) ) && \
       $(COPY) $(BINDIR)/$@$(EXEX) $(BINDIR)/$@$(VERSION_EXEX) $(BINDIR)/$@$(MAJVERS_EXEX) $(INST_BINDIR) && \
       echo "done"
 
+$(TOOLS_TO_SYMLINK):
+	@ printf "Installing symlink for $@..."
+	@ # report missing tools
+	@ ( test -f $(BINDIR)/$@$(EXEX) || ( echo "not found" && false ) ) && \
+      $(COPY) $(BINDIR)/$@$(VERSION_EXEX) $(INST_BINDIR)/$@-orig$(VERSION_EXEX) && \
+	  ln -f -s sratools$(VERSION_EXEX) $(INST_BINDIR)/$@$(VERSION_EXEX) && \
+	  ln -f -s $@$(VERSION_EXEX) $(INST_BINDIR)/$@$(MAJVERS_EXEX) && \
+	  ln -f -s $@$(MAJVERS_EXEX) $(INST_BINDIR)/$@$(EXEX) && \
+      echo "done"
+
 copykfg:
 	@ echo "Installing configuration files to $(KONFIG_DIR)"
 	@ $(TOP)/build/install-kfg.sh default.kfg   $(BINDIR)/ncbi          $(KONFIG_DIR) $(TOP)/shared/kfgsums
@@ -108,7 +131,8 @@ endif
 # uninstall
 #
 
-TO_UNINSTALL = $(addsuffix *,$(addprefix $(INST_BINDIR)/,$(TOOLS_TO_INSTALL) magic))
+TO_UNINSTALL = $(addsuffix *,$(addprefix $(INST_BINDIR)/,$(TOOLS_TO_INSTALL) \
+				magic sratools sratools.pl))
 ifneq (true, $(LINUX_ROOT))
     TO_UNINSTALL += $(INST_BINDIR)/ncbi/certs.kfg $(INST_BINDIR)/ncbi/ncbi-vdb.kfg $(INST_BINDIR)/ncbi/default.kfg $(INST_BINDIR)/ncbi/vdb-copy.kfg
 endif

build/Makefile.rpm

@@ -45,7 +45,6 @@ DST =$(PKGDIR)/sra-toolkit-$(VERSION)
 TGZ=$(DST).tar.gz
 
 clean_rpm:
-	echo 'DEBUGGING RPM: clean_rpm'
 	rm -rf $(DST) $(RPM) $(SPEC) $(SRC) $(TGZ)
 
 rpm: $(RPM)

build/Makefile.tools

-TOOLS_TO_INSTALL = \
+TOOLS_TO_COPY = \
 	abi-dump        \
-	abi-load        \
 	align-info      \
 	bam-load        \
 	cache-mgr       \
 	cg-load         \
-	copycat         \
-	fasterq-dump    \
-	fastq-dump      \
-	fastq-load      \
-	helicos-load    \
 	illumina-dump   \
-	illumina-load   \
 	kar             \
 	kdbmeta         \
 	kget            \
 	latf-load       \
-	prefetch        \
 	rcexplain       \
-	remote-fuser    \
-	sam-dump        \
 	sff-dump        \
-	sff-load        \
-	srapath         \
-	sra-pileup      \
 	sra-sort        \
 	sra-stat        \
-	srf-load        \
 	test-sra        \
 	vdb-config      \
 	vdb-copy        \
@@ -37,5 +23,27 @@ TOOLS_TO_INSTALL = \
 	vdb-passwd      \
 	vdb-unlock      \
 	vdb-validate    \
+
+# these tools will be copied to <instdir>/<toolname>-orig; 
+# <toolname> will be created as a symlink to sratools
+TOOLS_TO_SYMLINK = \
+	fasterq-dump    \
+	fastq-dump      \
+	sam-dump        \
+	prefetch        \
+	srapath         \
+	sra-pileup      \
+
+# these tools may be missing from $(BINDIR), not an error
+TOOLS_TO_COPY_OPTIONAL = \
+	abi-load        \
+	copycat         \
+	fastq-load      \
+	helicos-load    \
+	illumina-load   \
+	remote-fuser    \
+	sff-load        \
+	srf-load        \
 	sra-toolkit     \
 
+TOOLS_TO_INSTALL = $(TOOLS_TO_COPY) $(TOOLS_TO_SYMLINK) $(TOOLS_TO_COPY_OPTIONAL)

build/Makefile.vers

@@ -23,4 +23,4 @@
 # ===========================================================================
 
 # SRA-TOOLS and library version
-VERSION = 2.9.3
+VERSION = 2.10.0

configure

@@ -41,5 +41,5 @@ else
         echo >&2 "configure: error: perl not found."; exit 1; }
 
     cd $CURDIR/setup
-    perl  konfigure.perl "$@"
+    perl -w konfigure.perl "$@"
 fi

scripts/test-volume-mgr.pl

@@ -551,7 +551,7 @@ sub prepare {
     my $kfg = "$tmp/k.kfg";
 
     my $node = '/repository/remote/main/CGI/resolver-cgi='
-             . '\"https://www.ncbi.nlm.nih.gov/Traces/names/names.fcgi\"';
+             . '\"https://trace.ncbi.nlm.nih.gov/Traces/names/names.fcgi\"';
     `echo $node                                       >  $kfg`; die if $?;
 
     my $public = "$tmp/ncbi/public";

scripts/volume-mgr.pl

@@ -83,6 +83,7 @@ if ( $OPT{volumes} ) {
 }
 
 my $t = `date +%s.%N`;
+$t = `date +%s` if ($^O eq 'darwin');
 my $TIMESTAMP = strftime ( "%Y%m%d-%H%M%S", localtime $t );
 $TIMESTAMP   .= sprintf ( ".%01d", ( $t - int ( $t ) ) * 10 );
 my $NCBI_SETTINGS = ncbi_settings();

setup/konfigure.perl

@@ -377,7 +377,7 @@ if ($TOOLS =~ /gcc$/) {
 } elsif ($TOOLS eq 'clang') {
     $CPP  = 'clang++' unless ($CPP);
     $CC   = 'clang -c';
-    my $versionMin = '-mmacosx-version-min=10.6';
+    my $versionMin = '-mmacosx-version-min=10.10';
     $CP   = "$CPP -c $versionMin";
     if ($BITS ne '32_64') {
         $ARCH_FL = '-arch i386' if ($BITS == 32);

shared/toolkit.vers

-2.9.3
+2.10.0

shared/toolkit.vers.h

-#define TOOLKIT_VERS 0x02090003
+#define TOOLKIT_VERS 0x020A0000

test/fasterq-dump/Makefile

+# ===========================================================================
+#
+#                            PUBLIC DOMAIN NOTICE
+#               National Center for Biotechnology Information
+#
+#  This software/database is a "United States Government Work" under the
+#  terms of the United States Copyright Act.  It was written as part of
+#  the author's official duties as a United States Government employee and
+#  thus cannot be copyrighted.  This software/database is freely available
+#  to the public for use. The National Library of Medicine and the U.S.
+#  Government have not placed any restriction on its use or reproduction.
+#
+#  Although all reasonable efforts have been taken to ensure the accuracy
+#  and reliability of the software and data, the NLM and the U.S.
+#  Government do not and cannot warrant the performance or results that
+#  may be obtained by using this software or data. The NLM and the U.S.
+#  Government disclaim all warranties, express or implied, including
+#  warranties of performance, merchantability or fitness for any particular
+#  purpose.
+#
+#  Please cite the author in any work or product based on this material.
+#
+# ===========================================================================
+
+default: runtests
+
+TOP ?= $(abspath ../..)
+
+MODULE = test/fasterq-dump
+
+TEST_TOOLS = 
+
+include $(TOP)/build/Makefile.env
+
+.PHONY: $(TEST_TOOLS)
+
+slowtests: append-test
+
+append-test: $(BINDIR)/fasterq-dump
+	@./append-test.sh

test/fasterq-dump/append-test.sh

+#!/bin/bash
+
+SCRATCH_DIR="/panfs/traces01/sra_review/scratch"
+TEMP_DIR="/dev/shm"
+
+ACC1="SRR341578"
+ACC2="SRR341577"
+ACC3="SRR341576"
+ACC4="SRR341575"
+
+TOOL="fasterq-dump"
+
+#create the output directory:
+OUTDIR="${SCRATCH_DIR}/APPEND_TEST"
+if [ -d "$OUTDIR" ]; then
+    rm -rf "$OUTDIR"
+fi
+
+mkdir -p "$OUTDIR"
+
+#create the regular output ( no append-mode ) for later comparison:
+for ACC in $ACC1 $ACC2 $ACC3 $ACC4
+do
+    $TOOL $ACC -O $OUTDIR -t $TEMP_DIR -p
+    #append the output of these for later comparison
+    cat "${OUTDIR}/${ACC}_1.fastq" >> "${OUTDIR}/A_1.fastq"
+    cat "${OUTDIR}/${ACC}_2.fastq" >> "${OUTDIR}/A_2.fastq"
+    rm "${OUTDIR}/${ACC}_1.fastq" "${OUTDIR}/${ACC}_2.fastq"
+done
+
+#run the tool in append-mode
+for ACC in $ACC1 $ACC2 $ACC3 $ACC4
+do
+    $TOOL $ACC -o "${OUTDIR}/B.fastq" -t $TEMP_DIR -p --append
+done
+
+OPT1="--brief"
+OPT2="--report-identical-files"
+diff "$OPT1" "$OPT2" "${OUTDIR}/A_1.fastq" "${OUTDIR}/B_1.fastq"
+diff "$OPT1" "$OPT2" "${OUTDIR}/A_2.fastq" "${OUTDIR}/B_2.fastq"
+
+#get rid of all the data at the end
+if [ -d "$OUTDIR" ]; then
+    rm -rf "$OUTDIR"
+fi
+
+echo "done"

test/fastq-loader/.gdb_history

-r
-bt
-up 5
-p *seq
-p read
-l
-do 
-l
-up
-l 
-p seq
-p read
-quit
-r
-bt
-quit
-b yy_get_next_buffer
-r
-c
-r
-disa 1
-c
-bt
-l
-up
-p number_to_move
-p yyextra->input
-b 1708
-r
-c
-c
-p YY_CURRENT_BUFFER_LVALUE
-p yyextra
-p yyg->yyextra_r
-p * yyg->yyextra_r
-p * yyg->yyextra_r->input
-c
-bt
-disa 2
-r
-up
-p * yyg->yyextra_r->input
-p * yyg->yyextra_r
-p * yyg->yyextra_r->column
-p FASTQ_input
-quit
-r
-bt
-up
- 
-up
-up
-p *pb
-do
-p yyscanner
-p *yyscanner
-do
-p yyg
-p *yyg
-p *yyg->yyin_r
-p yyg->yyin_r
-p yyg->yyextra_r
-p *yyg->yyextra_r
-quit
-quit
-b wb-test-fastq.cpp:279
-p pb
-r
-p pb
-s
-n
-apropos watch
-help watch
-watch -l pb.input
-c
-up
-p yyg->yyextra_r
-p *yyg->yyextra_r
-bt
-r
-disa 2
-l
-up
-up
-up
-bt
-bt
-r
-p pb
-p &pb
-bt
-c
-up
-p (FASTQParseBlock *) 0x7fffffffdf58
-p *(FASTQParseBlock *) 0x7fffffffdf58
-l
-p yyg->yyextra_r
-quit
-b wb-test-fastq.cpp:722
-r
-s
-n
-s
-n
-p *self
-p length
-p start
-p stop
-p start >= length || stop >= length || start >= stop
-p start >= length
-p stop >= length
-p start >= stop
-p start
-p stop
-quit
-b wb-test-fastq-parse.cpp:321
-r
-p pb.record->seq
-b ExpandRead
-r
-p copyTo
-n
-p copyTo
-p *newReadLine
-n
-p pb -> readLength
-p newReadLine -> tokenLength
-n
-p *pb
-b FASTQ_input
-c
-p pb
-p pb.record.seq
-b fastq-reader.c:575
-r
-c
-c
-r
-c
-p ( const char *) ( pb . record -> source . base )
-quit
-b ExpandQuality
-r
-quit
-b ExpandQuality
-r
-quit
-r
-bt
-up
-p p self
-p self
-p result
-p *result
-quit
-b fastq-reader.c:549
-r
-p self->pb.readLength 
-p self->pb.qualityLength
-bt
-c
-bt
-p self->pb.qualityLength
-p self->pb.readLength
-b ExpandQuality
-r
-c
-n
-r
-disa 1
-c
-s
-n
-p ch
-n
-s
-n
-p sb->record->rej
-p *sb->record->rej
-n
-c
-quit
-b wb-test-fastq.cpp:261
-r
-p *this
-p *record
-s
-n
-p reject
-n
-p record
-p *record
-n
-s
-p *self
-n
-n
-n
-s
-n
-p *self
-quit

test/fastq-loader/Makefile

@@ -31,10 +31,13 @@ TOP ?= $(abspath ../..)
 MODULE = test/fastq-loader
 
 TEST_TOOLS = \
+	test-id2name \
 	wb-test-fastq \
 	wb-test-fastq-parse \
 	test-fastq-loader
 
+VDB_CONFIG = $(SRCDIR)
+
 include $(TOP)/build/Makefile.env
 
 $(TEST_TOOLS): makedirs
@@ -66,6 +69,26 @@ $(TEST_BINDIR)/wb-test-fastq-parse: $(FASTQ_PARSE_OBJ)
 parse: wb-test-fastq-parse
 	$(TEST_BINDIR)/wb-test-fastq-parse  2>&1
 
+#-------------------------------------------------------------------------------
+# id2name
+#
+ID2NAME_TEST_SRC = \
+	test-id2name
+
+ID2NAME_TEST_OBJ = \
+	$(addsuffix .$(OBJX),$(ID2NAME_TEST_SRC))
+
+ID2NAME_TEST_LIB = \
+	-skapp \
+	-sktst \
+	-sfastqloader \
+	-sncbi-vdb
+
+$(TEST_BINDIR)/test-id2name: $(ID2NAME_TEST_OBJ)
+	$(LP) --exe -o $@ $^ $(ID2NAME_TEST_LIB)
+
+id: test-id2name
+	$(TEST_BINDIR)/test-id2name  2>&1
 
 #-------------------------------------------------------------------------------
 # white-box test
@@ -111,13 +134,12 @@ tfl:
 # scripted tests
 #
 runtests: wb parse set_schema smalltests
-	rm -v tmp.kfg
 
 set_schema:
 	echo "vdb/schema/paths = \"$(VDB_INCDIR)\"" > tmp.kfg
 
-SMALLRUN = @ export NCBI_SETTINGS=/ ; export VDB_CONFIG=`pwd` ; \
-	$(SRCDIR)/runtestcase.sh $(BINDIR) $(SRCDIR)
+SMALLRUN = @ NCBI_SETTINGS= VDB_CONFIG=$(SRCDIR) $(SRCDIR)/runtestcase.sh $(BINDIR) $(SRCDIR)
+
 smalltests: $(TEST_TOOLS)
 	-rm -rf $(SRCDIR)/actual
 #   Small tests
@@ -136,7 +158,7 @@ smalltests: $(TEST_TOOLS)
 #           2.1 reads only, basespace
 	$(SMALLRUN) 2.1 0 $(SRCDIR)/input/2.1.fastq --quality PHRED_33
 #           2.1.1 reads only, colorspace
-	$(SMALLRUN) 2.1.1 0 $(SRCDIR)/input/2.1.1.fastq --quality PHRED_33
+#CS	$(SMALLRUN) 2.1.1 0 $(SRCDIR)/input/2.1.1.fastq --quality PHRED_33
 #           2.2 qualities only
 	@#-$(SMALLRUN) 2.2 0 $(SRCDIR)/input/2.2.fastq --quality PHRED_33
 #           2.3 reads+qualities, qualities before reads
@@ -187,11 +209,11 @@ smalltests: $(TEST_TOOLS)
 #           4.3.1 --cache-size      rejected if invalid
 	$(SMALLRUN) 4.3.1 3 $(SRCDIR)/input/4.fastq --cache-size BIG --quality PHRED_33
 #           4.4 --max-rec-count     stops at the specified record #
-	$(SMALLRUN) 4.4 0 $(SRCDIR)/input/4.4.fastq --max-rec-count 3 --quality PHRED_33
+#CS	$(SMALLRUN) 4.4 0 $(SRCDIR)/input/4.4.fastq --max-rec-count 3 --quality PHRED_33
 #           4.5 --max-err-count     stops after the specified # of errors
 	$(SMALLRUN) 4.5 3 $(SRCDIR)/input/4.5.fastq --max-err-count 2 --quality PHRED_33
 #           4.5.1 --max-err-pct     fails if % of error records is too high
-	$(SMALLRUN) 4.5.1 3 $(SRCDIR)/input/4.5.1.fastq --quality PHRED_33  --max-err-count 10 --max-err-pct 20
+#CS	$(SMALLRUN) 4.5.1 3 $(SRCDIR)/input/4.5.1.fastq --quality PHRED_33  --max-err-count 10 --max-err-pct 20
 #           4.6 --platform          saves correct value in the PLATFORM column
 	$(SMALLRUN) 4.6 0 $(SRCDIR)/input/4.fastq -p LS454 --quality PHRED_33
 #           4.6.1 --platform        rejects invalid values
@@ -230,14 +252,15 @@ smalltests: $(TEST_TOOLS)
 	$(SMALLRUN) 13.1 0 $(SRCDIR)/input/13.1.fastq --quality PHRED_33
     # SRA-2932: 30 is the preferred value for qualities when reading FASTA
 	$(SMALLRUN) 14.0 0 $(SRCDIR)/input/14.0.fasta $(SRCDIR)/input/14.0.fasta --quality PHRED_33
-	rm -rf $(SRCDIR)/actual
     # VDB-3154: buffer insufficient
 	$(SMALLRUN) 15.0 0 $(SRCDIR)/input/15.0.fastq --quality PHRED_33
     # VDB-3413: multi-line reads and qualities
 	$(SMALLRUN) 16.0 0 $(SRCDIR)/input/16.0.fasta --quality PHRED_33 --platform=PACBIO
 	$(SMALLRUN) 16.1 0 $(SRCDIR)/input/16.1.fastq --quality PHRED_33 --platform=PACBIO
+	#
+	#rm -rf $(SRCDIR)/actual
 
 onetest:
 	rm -rf $(SRCDIR)/actual
-	$(SMALLRUN) 16.1 0 $(SRCDIR)/input/16.1.fastq --quality PHRED_33 --platform=PACBIO
+	$(SMALLRUN) 8.1 3 $(SRCDIR)/input/8.1.fastq --quality PHRED_33
 

test/fastq-loader/bm.sh

+#!/bin/bash
+
+OLD=/panfs/traces01.be-md.ncbi.nlm.nih.gov/trace_software/toolkit/centos64/bin/latf-load
+NEW=/home/boshkina/devel/ncbi-outdir/sra-tools/linux/gcc/x86_64/rel/bin/latf-load
+TOP=/panfs/pan1.be-md.ncbi.nlm.nih.gov/trace_work/backup/qa/cases/load
+OUT=/panfs/pan1.be-md.ncbi.nlm.nih.gov/sra-test/latf-load/qq
+
+function run {
+    echo "$*"
+    rm -rf $OUT
+    echo "old: $OLD --quality PHRED_33 -o $OUT $*"
+    time $OLD --quality PHRED_33 -o $OUT $*
+    du -s $OUT
+    vdb-dump -R1 -C SPOT_COUNT $OUT
+
+    rm -rf $OUT
+    echo "new:"
+    time $NEW --quality PHRED_33 -o $OUT $*
+    du -s $OUT
+    vdb-dump -R1 -C SPOT_COUNT $OUT
+
+    echo
+}
+
+run $TOP/latf/SRR529889/PmaximaFP.fastq
+run $TOP/latf/SRR1692309/Hatch66-3_R1_filtered.fastq $TOP/latf/SRR1692309/Hatch66-3_R2_filtered.fastq
+run $TOP/latf/SRR567550/s_7_1_sequence_100409.fq.gz $TOP/latf/SRR567550/s_7_2_sequence_100409.fq.gz
+run $TOP/latf/XXX034449/S.cerevisiae_Fragmented_Replicate2.fasta
+run $TOP/latf/SRR627950/1920.1.1690.fastq
+run $TOP/genericFastq/SRR8054133/TICK01_R1.fastq.gz $TOP/genericFastq/SRR8054133/TICK01_R2.fastq.gz
+run $TOP/genericFastq/SRR3020730/sim_CFSAN001140_2.fq
+run $TOP/latf/SRR989791/XSBOR_20130726_RS42150_PL100036291-1_C02.fastq $TOP/latf/SRR989791/XSBOR_20130726_RS42150_PL100036291-1_D02.fastq -p PACBIO
+run $TOP/genericFastq/SRR7896971/ARC_s84_1.fq.gz $TOP/genericFastq/SRR7896971/ARC_s84_2.fq.gz
+
+
+

test/fastq-loader/expected/1.1.stdout

@@ -10,9 +10,10 @@ CMP_LINKAGE_GROUP:
     LINKAGE_GROUP: 
       MAX_SPOT_ID: 1
       MIN_SPOT_ID: 1
-             NAME: 1
+             NAME: BILLIEHOLIDAY_1_FC20F3DAAXX:8:2:342:540
          PLATFORM: SRA_PLATFORM_UNDEFINED
-          QUALITY: 5, 13, 27, 22, 22, 19, 22, 23, 23, 23, 24, 24, 24, 23, 22, 22, 21, 22, 22, 21, 25, 24, 24, 22, 24, 22, 19, 18, 20, 19, 22, 22, 19, 22, 22, 24
+          QUALITY: 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30
+         RAW_NAME: BILLIEHOLIDAY_1_FC20F3DAAXX:8:2:342:540
         RD_FILTER: SRA_READ_FILTER_PASS, SRA_READ_FILTER_PASS
              READ: GTCGCTTCTCGGAAGNGTGAAAGACAANAATNTTNN
       READ_FILTER: SRA_READ_FILTER_PASS, SRA_READ_FILTER_PASS
@@ -22,6 +23,7 @@ CMP_LINKAGE_GROUP:
         READ_TYPE: SRA_READ_TYPE_BIOLOGICAL, SRA_READ_TYPE_TECHNICAL
        SIGNAL_LEN: 0
        SPOT_COUNT: 1
+      SPOT_FILTER: 0
        SPOT_GROUP: 
           SPOT_ID: 1
          SPOT_LEN: 36