Skip to content
Commits on Source (11)
......@@ -23,3 +23,6 @@
/.pydevproject
*.stash
tmp.kfg
*/xcuserdata/*
.vscode
.gdb_history
# NCBI External Developer Release:
## SRA Toolkit 2.10.0
**August 19, 2019**
**bam-load**: SAM/BAM record sizes are limited only by available memory
**fastq-dump**: fixed help text
**kfg, sra-tools**: use trace.ncbi.nlm.nih.gov to call names service
**kfg, vdb-config**: alternative remote repository URL was added to default configuration
**kfs**: introduced readahead strategy for cloud storage
**klib, vdb**: error report is saved to ncbi_error_report.txt
**kns**: We now use system root CA certs on Unix
**kns**: introduced configurable controls over network timeouts
**prefetch**: fixed crash when run with --output-file option
**prefetch**: updated prefetch help text
**prefetch, srapath**: support for original submission files in cloud storage
**prefetch, srapath, vfs**: added possibility to specify data location
**prefetch, vdb**: adjustments for latest name resolution service
**prefetch, vfs**: added support of "run accession as directory"
**prefetch, vfs**: added support of download of reference sequences in "run accession as directory"
**prefetch, vfs**: fixed regression when prefetch does not download vdbcache
**sra-stat**: don't print mismatch warning for old runs having a single SEQUENCE table and CMP_BASE_COUNT=BIO_BASE_COUNT while it should be 0
**sra-stat**: fixed a bug when processing runs having spots with first reads with length 0
**srapath, sratools**: Added support of "run accession as directory"
**sratools**: SRA tools are executed by a driver tool called sratools
**sratools**: installation creates symlinks to invoke some tools via a driver tool (sratools)
**test**: created a standard way to detect ascp availability
**tui, vdb-config**: new look and cloud specific options in 'vdb-config -i'
**vdb**: make greater use of data returned by latest name resolver
**vdb-validate**: referential integrity checker prints out completion message, if it printed out any progress
**vfs**: added possibility to set resolver version from configuration
**vfs**: allow to use SDL as remote service
## SRA Toolkit 2.9.6
**March 18, 2019**
**prefetch**, **vfs**: fixed regression that prevented re-download of incomplete files
## SRA Toolkit 2.9.5
**March 6, 2019**
**prefetch**: fixed regression that caused download of incomplete files
## SRA Toolkit 2.9.4-1
**March 4, 2019**
**sra-tools, vfs**: fixed regression introduced in 2.9.4 release thas caused delay when starting sra tools
## SRA Toolkit 2.9.4
**January 31, 2019**
**fasterq-dump**: improved handling of temp files in case of multiple instances
**fasterq-dump**: produces same output as fastq-dump on SRR000001 (empty reads)
**fastq-dump**: updated typo in error report
**sra-tools, vfs**: added support of realign objects
## SRA Toolkit 2.9.3
**October 17, 2018**
**October 11, 2018**
**kns**: added possibility to skip server's certificate validation
**kns**: expect to receive HTTP status 200 when sending range-request that includes the whole file
**vdb, vdb-copy**: fixed a bug in accessing pagemap process request for cursors which do not have pagemap thread running
**vdb**: fixed a bug in accessing pagemap process request for cursors which do not have pagemap thread running
## SRA Toolkit 2.9.2-2
**September 26, 2018**
**read-filter-redact**: Fixed to update HISTORY metadata
## SRA Toolkit 2.9.2
**July 23, 2018**
**kfg, vfs**: Introduced enhanced handling of download-only NGC files that lack read/decrypt permissions
......
# The NCBI SRA (Sequence Read Archive)
### Contact:
email: sra-tools@ncbi.nlm.nih.gov
or visit the [SRA Tools web site](http://ncbi.github.io/sra-tools)
email: sra@ncbi.nlm.nih.gov
### Download
Visit our [download page](https://github.com/ncbi/sra-tools/wiki/Downloads) for pre-built binaries.
......@@ -15,25 +14,14 @@ The SRA Toolkit and SDK from NCBI is a collection of tools and libraries for
using data in the INSDC Sequence Read Archives.
### ANNOUNCEMENT:
With release 2.10.0 of `sra-tools` we have added cloud-native operation for AWS and GCP environments _(Linux only for this release)_, for use with the public SRA. `prefetch` is capable of retrieving original submission files in addition to ETL data.
With release 2.9.1 of `sra-tools` we have finally made available the tool `fasterq-dump`, a replacement for the much older `fastq-dump` tool. As its name implies, it runs faster, and is better suited for large-scale conversion of SRA objects into FASTQ files that are common on sites with enough disk space for temporary files. `fasterq-dump` is multi-threaded and performs bulk joins in a way that improves performance as compared to `fastq-dump`, which performs joins on a per-record basis _(and is single-threaded)_.
`fastq-dump` is still supported as it handles more corner cases than `fasterq-dump`, but it is likely to be deprecated in the future.
You can get more information about `fasterq-dump` in our Wiki at [https://github.com/ncbi/sra-tools/wiki/HowTo:-fasterq-dump](https://github.com/ncbi/sra-tools/wiki/HowTo:-fasterq-dump).
### IMPORTANT ANNOUNCEMENT
As was announced in [OMB memorandum M-15-13](https://www.whitehouse.gov/sites/default/files/omb/memoranda/2015/m-15-13.pdf) on June 8, 2015, NCBI and all Federal systems will be transitioning to using HTTPS-only protocols before the end of 2016. This change will affect any software that uses NCBI APIs such as the E-utilities or NCBI software toolkits such as `sra-tools`, `ncbi-vdb` or `ngs`.
The NLM and NCBI may implement the switch to HTTPS-only as early as September 30, 2016.
In particular, software products that depend on `sra-tools`, `ncbi-vdb` or `ngs` may not function as expected after September 30 unless they are properly updated from this site or by the software provider.
If you use software that accesses NCBI SRA data in any way, your software will likely be affected by this change. Please check with your software provider for recent udpates or patches, and be sure to acquire these before September 30.
If you develop software that relies on `sra-tools`, `ncbi-vdb` or `ngs` in any way, you will likely need to update your code so that it accesses NCBI using HTTPS.
We have released new tools with version 2.8.0 that are HTTPS compatible and `M-15-13` compliant as of October 7, 2016. Please be certain to [update all of your binaries](https://github.com/ncbi/sra-tools/wiki/Downloads) and configuration files.
For additional information on using, configuring, and building the toolkit,
please visit our [wiki](https://github.com/ncbi/sra-tools/wiki)
or our web site at [NCBI](http://www.ncbi.nlm.nih.gov/Traces/sra/?view=toolkit_doc)
......
......@@ -365,13 +365,23 @@ LDFLAGS = $(DBG) $(PROF) $(CARCH) $(MIN_DEPLOY_OS_OPT) $(LDPATHS)
#
# MallocScribble=1 is for catching allocation problems on Mac
#
HOSTNAME = $(shell hostname)
WHICHASCP = $(shell which ascp 2> /dev/null)
HAVE_NCBI_ASCP=1
ifeq (,${WHICHASCP})
HAVE_NCBI_ASCP=
endif
ifeq (iebdev21,${HOSTNAME})
HAVE_NCBI_ASCP=
endif
ifeq ($(RUNTESTS_OVERRIDE),)
runtests: all $(TEST_TOOLS)
@ export VDB_CONFIG=$(VDB_CONFIG);export LD_LIBRARY_PATH=$(LIBDIR):$$LD_LIBRARY_PATH;export MallocScribble=1;\
for i in $(TEST_TOOLS);\
do\
echo ++++++++++++++++++++++++++++++++++++++++++++++++++++++;\
echo Run $(TEST_BINDIR)/$$i;eval $(RUN_REMOTELY) $(TEST_BINDIR)/$$i;r=$$?; \
echo Run VDB_CONFIG=$$VDB_CONFIG $(TEST_BINDIR)/$$i;eval $(RUN_REMOTELY) $(TEST_BINDIR)/$$i;r=$$?; \
if [ "$$r" != "0" ] ; then exit $$r; fi; \
done
......
......@@ -35,10 +35,11 @@ include $(TOP)/build/Makefile.$(OS)
# install
#
# $TOOLS_TO_INSTALL
# TOOLS_TO_COPY, TOOLS_TO_SYMLINK
# $TOOLS_TO_INSTALL = TOOLS_TO_COPY + TOOLS_TO_SYMLINK
include $(TOP)/build/Makefile.tools
ifeq (1,$(HAVE_HDF5))
TOOLS_TO_INSTALL += pacbio-load
TOOLS_TO_COPY += pacbio-load
endif
# unset outside defined variables
......@@ -68,16 +69,38 @@ installexes:
@ echo "Installing executables to $(INST_BINDIR)/..."
@ mkdir -p $(INST_BINDIR)
@ $(MAKE) -s --no-print-directory -f $(TOP)/build/Makefile.install $(TOOLS_TO_INSTALL)
@ # copy the driver
$(COPY) $(TOP)/tools/driver-tool/sratools.pl $(INST_BINDIR)/sratools$(VERSION_EXEX)
ln -f -s sratools$(VERSION_EXEX) $(INST_BINDIR)/sratools$(MAJVERS_EXEX)
ln -f -s sratools$(MAJVERS_EXEX) $(INST_BINDIR)/sratools$(EXEX)
@ #
@ printf "Installing magic file..."
@ $(COPY) $(TOP)/tools/copycat/magic $(INST_BINDIR) && echo "done"
$(TOOLS_TO_INSTALL):
$(TOOLS_TO_COPY):
@ printf "Installing $@..."
@ # report missing tools
@ ( test -f $(BINDIR)/$@$(EXEX) || ( echo "not found" && false ) ) && \
$(COPY) $(BINDIR)/$@$(EXEX) $(BINDIR)/$@$(VERSION_EXEX) $(BINDIR)/$@$(MAJVERS_EXEX) $(INST_BINDIR) && \
echo "done"
$(TOOLS_TO_COPY_OPTIONAL):
@ printf "Installing $@..."
@ # ignore missing tools
@ - ( test -f $(BINDIR)/$@$(EXEX) || ( echo "not found" && false ) ) && \
$(COPY) $(BINDIR)/$@$(EXEX) $(BINDIR)/$@$(VERSION_EXEX) $(BINDIR)/$@$(MAJVERS_EXEX) $(INST_BINDIR) && \
echo "done"
$(TOOLS_TO_SYMLINK):
@ printf "Installing symlink for $@..."
@ # report missing tools
@ ( test -f $(BINDIR)/$@$(EXEX) || ( echo "not found" && false ) ) && \
$(COPY) $(BINDIR)/$@$(VERSION_EXEX) $(INST_BINDIR)/$@-orig$(VERSION_EXEX) && \
ln -f -s sratools$(VERSION_EXEX) $(INST_BINDIR)/$@$(VERSION_EXEX) && \
ln -f -s $@$(VERSION_EXEX) $(INST_BINDIR)/$@$(MAJVERS_EXEX) && \
ln -f -s $@$(MAJVERS_EXEX) $(INST_BINDIR)/$@$(EXEX) && \
echo "done"
copykfg:
@ echo "Installing configuration files to $(KONFIG_DIR)"
@ $(TOP)/build/install-kfg.sh default.kfg $(BINDIR)/ncbi $(KONFIG_DIR) $(TOP)/shared/kfgsums
......@@ -108,7 +131,8 @@ endif
# uninstall
#
TO_UNINSTALL = $(addsuffix *,$(addprefix $(INST_BINDIR)/,$(TOOLS_TO_INSTALL) magic))
TO_UNINSTALL = $(addsuffix *,$(addprefix $(INST_BINDIR)/,$(TOOLS_TO_INSTALL) \
magic sratools sratools.pl))
ifneq (true, $(LINUX_ROOT))
TO_UNINSTALL += $(INST_BINDIR)/ncbi/certs.kfg $(INST_BINDIR)/ncbi/ncbi-vdb.kfg $(INST_BINDIR)/ncbi/default.kfg $(INST_BINDIR)/ncbi/vdb-copy.kfg
endif
......
......@@ -45,7 +45,6 @@ DST =$(PKGDIR)/sra-toolkit-$(VERSION)
TGZ=$(DST).tar.gz
clean_rpm:
echo 'DEBUGGING RPM: clean_rpm'
rm -rf $(DST) $(RPM) $(SPEC) $(SRC) $(TGZ)
rpm: $(RPM)
......
TOOLS_TO_INSTALL = \
TOOLS_TO_COPY = \
abi-dump \
abi-load \
align-info \
bam-load \
cache-mgr \
cg-load \
copycat \
fasterq-dump \
fastq-dump \
fastq-load \
helicos-load \
illumina-dump \
illumina-load \
kar \
kdbmeta \
kget \
latf-load \
prefetch \
rcexplain \
remote-fuser \
sam-dump \
sff-dump \
sff-load \
srapath \
sra-pileup \
sra-sort \
sra-stat \
srf-load \
test-sra \
vdb-config \
vdb-copy \
......@@ -37,5 +23,27 @@ TOOLS_TO_INSTALL = \
vdb-passwd \
vdb-unlock \
vdb-validate \
# these tools will be copied to <instdir>/<toolname>-orig;
# <toolname> will be created as a symlink to sratools
TOOLS_TO_SYMLINK = \
fasterq-dump \
fastq-dump \
sam-dump \
prefetch \
srapath \
sra-pileup \
# these tools may be missing from $(BINDIR), not an error
TOOLS_TO_COPY_OPTIONAL = \
abi-load \
copycat \
fastq-load \
helicos-load \
illumina-load \
remote-fuser \
sff-load \
srf-load \
sra-toolkit \
TOOLS_TO_INSTALL = $(TOOLS_TO_COPY) $(TOOLS_TO_SYMLINK) $(TOOLS_TO_COPY_OPTIONAL)
......@@ -23,4 +23,4 @@
# ===========================================================================
# SRA-TOOLS and library version
VERSION = 2.9.3
VERSION = 2.10.0
......@@ -41,5 +41,5 @@ else
echo >&2 "configure: error: perl not found."; exit 1; }
cd $CURDIR/setup
perl konfigure.perl "$@"
perl -w konfigure.perl "$@"
fi
sra-sdk (2.10.0+dfsg-1) UNRELEASED; urgency=medium
* Team upload.
* New upstream version
* debhelper-compat 12
* Standards-Version: 4.4.1
* TODO: Fails to build from source since it does not find
KClientHttpRequestFormatPostMsg to link against
-- Steffen Moeller <moeller@debian.org> Thu, 28 Nov 2019 21:39:13 +0100
sra-sdk (2.9.4+dfsg-1) unstable; urgency=medium
* Team upload.
* New upstream version.
* Bumped policy to 4.3.0
-- Steffen Moeller <moeller@debian.org> Tue, 26 Feb 2019 15:01:51 +0100
sra-sdk (2.9.3+dfsg-1) unstable; urgency=medium
* Team upload.
......
......@@ -4,7 +4,7 @@ Uploaders: Charles Plessy <plessy@debian.org>,
Andreas Tille <tille@debian.org>
Section: science
Priority: optional
Build-Depends: debhelper (>= 11~),
Build-Depends: debhelper-compat (= 12),
libngs-sdk-dev (>= 2.9.0),
libncbi-vdb-dev (>= 2.9.1),
libvdb-sqlite-dev,
......@@ -13,7 +13,7 @@ Build-Depends: debhelper (>= 11~),
libmagic-dev,
libxml2-dev,
python
Standards-Version: 4.2.1
Standards-Version: 4.4.1
Vcs-Browser: https://salsa.debian.org/med-team/sra-sdk
Vcs-Git: https://salsa.debian.org/med-team/sra-sdk.git
Homepage: https://github.com/ncbi/sra-tools/
......
......@@ -2,8 +2,10 @@ Author: Andreas Tille <tille@debian.org>
LastChanged: Wed, 11 Jan 2017 14:19:27 +0200
Description: Point to correct location of kfg config files in libncbi-vdb2 package
--- a/build/Makefile.env
+++ b/build/Makefile.env
Index: sra-sdk/build/Makefile.env
===================================================================
--- sra-sdk.orig/build/Makefile.env
+++ sra-sdk/build/Makefile.env
@@ -124,8 +124,8 @@ else
NCBIDIR = $(LIBDIR)/ncbi $(BINDIR)/ncbi
endif
......@@ -15,9 +17,11 @@ Description: Point to correct location of kfg config files in libncbi-vdb2 packa
# make reissue command
MAKE_CMD = $(MAKE) BUILD=$(BUILD) TOOLSET=$(TOOLSET) COMP=$(COMP) LNKG=$(LNKG) \
--- a/build/Makefile.install
+++ b/build/Makefile.install
@@ -48,6 +48,7 @@ LINUX_ROOT =
Index: sra-sdk/build/Makefile.install
===================================================================
--- sra-sdk.orig/build/Makefile.install
+++ sra-sdk/build/Makefile.install
@@ -49,6 +49,7 @@ LINUX_ROOT =
#fake root for debugging
#uncomment this line and change the test for root ( see under install: ) to succeed:
#ROOT = ~/root
......
......@@ -6,13 +6,18 @@ Index: sra-sdk/test/vdb-config/Makefile
===================================================================
--- sra-sdk.orig/test/vdb-config/Makefile
+++ sra-sdk/test/vdb-config/Makefile
@@ -31,5 +31,4 @@ runtests: test-vdb-config
@@ -47,8 +47,9 @@ runtests: test-vdb-config
test-vdb-config:
@ printf "Testing exit code vdb-config of vdb-config... "
- @ PATH=$(BINDIR):$(PATH) ; ./test-vdb-config.pl
- @ PATH=$(BINDIR):"$(PATH)" ; ./test-vdb-config.pl
- @ echo OK
+ @ echo "This test is droped since the executable should not be run as root"
+ #@ PATH=$(BINDIR):"$(PATH)" ; ./test-vdb-config.pl
+ #@ echo OK
+ @ echo Test dropped since executable should not run as root
#-------------------------------------------------------------------------------
# test-vdb-config-model
Index: sra-sdk/test/Makefile
===================================================================
--- sra-sdk.orig/test/Makefile
......
......@@ -2,9 +2,11 @@ Author: Andreas Tille <tille@debian.org>
Last-Update: Mon, 06 Jun 2016 11:24:46 +0200
Description: Exclude failing tests
--- a/test/fastq-loader/Makefile
+++ b/test/fastq-loader/Makefile
@@ -128,7 +128,7 @@ smalltests: $(TEST_TOOLS)
Index: sra-sdk/test/fastq-loader/Makefile
===================================================================
--- sra-sdk.orig/test/fastq-loader/Makefile
+++ sra-sdk/test/fastq-loader/Makefile
@@ -150,7 +150,7 @@ smalltests: $(TEST_TOOLS)
# 1.2 phred 64
$(SMALLRUN) 1.2 0 $(SRCDIR)/input/1.2.fastq --quality PHRED_64
# 1.3 phred 33, decimal qualities
......@@ -13,7 +15,7 @@ Description: Exclude failing tests
# 1.4 logodds 64
$(SMALLRUN) 1.4 0 $(SRCDIR)/input/1.4.fastq --quality LOGODDS
#
@@ -155,9 +155,9 @@ smalltests: $(TEST_TOOLS)
@@ -177,9 +177,9 @@ smalltests: $(TEST_TOOLS)
# 2.8.1 handling of N in reads, ALTREAD is a physical column
$(SMALLRUN) 2.8.1 0 $(SRCDIR)/input/2.8.1.fastq --quality PHRED_33
# 2.9 mismatching length of sequence and quality (qualities shorter)
......@@ -25,20 +27,20 @@ Description: Exclude failing tests
#
# Multiple files
# 3.1 reads+qualities together
@@ -185,19 +185,19 @@ smalltests: $(TEST_TOOLS)
@@ -207,19 +207,19 @@ smalltests: $(TEST_TOOLS)
-o actual/4.1.obj --quality PHRED_33 \
| grep "Cache size: 10M" >/dev/null
# 4.3.1 --cache-size rejected if invalid
- $(SMALLRUN) 4.3.1 3 $(SRCDIR)/input/4.fastq --cache-size BIG --quality PHRED_33
+ # $(SMALLRUN) 4.3.1 3 $(SRCDIR)/input/4.fastq --cache-size BIG --quality PHRED_33
# 4.4 --max-rec-count stops at the specified record #
$(SMALLRUN) 4.4 0 $(SRCDIR)/input/4.4.fastq --max-rec-count 3 --quality PHRED_33
#CS $(SMALLRUN) 4.4 0 $(SRCDIR)/input/4.4.fastq --max-rec-count 3 --quality PHRED_33
# 4.5 --max-err-count stops after the specified # of errors
- $(SMALLRUN) 4.5 3 $(SRCDIR)/input/4.5.fastq --max-err-count 2 --quality PHRED_33
+ # $(SMALLRUN) 4.5 3 $(SRCDIR)/input/4.5.fastq --max-err-count 2 --quality PHRED_33
# 4.5.1 --max-err-pct fails if % of error records is too high
- $(SMALLRUN) 4.5.1 3 $(SRCDIR)/input/4.5.1.fastq --quality PHRED_33 --max-err-count 10 --max-err-pct 20
+ # $(SMALLRUN) 4.5.1 3 $(SRCDIR)/input/4.5.1.fastq --quality PHRED_33 --max-err-count 10 --max-err-pct 20
-#CS $(SMALLRUN) 4.5.1 3 $(SRCDIR)/input/4.5.1.fastq --quality PHRED_33 --max-err-count 10 --max-err-pct 20
+#CS # $(SMALLRUN) 4.5.1 3 $(SRCDIR)/input/4.5.1.fastq --quality PHRED_33 --max-err-count 10 --max-err-pct 20
# 4.6 --platform saves correct value in the PLATFORM column
$(SMALLRUN) 4.6 0 $(SRCDIR)/input/4.fastq -p LS454 --quality PHRED_33
# 4.6.1 --platform rejects invalid values
......@@ -50,7 +52,7 @@ Description: Exclude failing tests
# Gzipped input
$(SMALLRUN) 5.0 0 $(SRCDIR)/input/5.0.fastq.gz --quality PHRED_33
# Misparsed quality
@@ -209,7 +209,7 @@ smalltests: $(TEST_TOOLS)
@@ -231,7 +231,7 @@ smalltests: $(TEST_TOOLS)
# secondary read numbers other than 3
$(SMALLRUN) 8.0 0 $(SRCDIR)/input/8.0.fastq --quality PHRED_33
# reject inconsistent secondary read numbers
......@@ -59,9 +61,11 @@ Description: Exclude failing tests
# newest Illumina tag line variation ("_" instead of " /")
$(SMALLRUN) 9.0 0 $(SRCDIR)/input/9.0.fastq --quality PHRED_33
# accepting ascii quality values up to 126
--- a/test/Makefile
+++ b/test/Makefile
@@ -38,7 +38,6 @@ SUBDIRS = \
Index: sra-sdk/test/Makefile
===================================================================
--- sra-sdk.orig/test/Makefile
+++ sra-sdk/test/Makefile
@@ -39,7 +39,6 @@ SUBDIRS = \
prefetch \
dump-test \
vdb-config \
......
......@@ -551,7 +551,7 @@ sub prepare {
my $kfg = "$tmp/k.kfg";
my $node = '/repository/remote/main/CGI/resolver-cgi='
. '\"https://www.ncbi.nlm.nih.gov/Traces/names/names.fcgi\"';
. '\"https://trace.ncbi.nlm.nih.gov/Traces/names/names.fcgi\"';
`echo $node > $kfg`; die if $?;
my $public = "$tmp/ncbi/public";
......
......@@ -83,6 +83,7 @@ if ( $OPT{volumes} ) {
}
my $t = `date +%s.%N`;
$t = `date +%s` if ($^O eq 'darwin');
my $TIMESTAMP = strftime ( "%Y%m%d-%H%M%S", localtime $t );
$TIMESTAMP .= sprintf ( ".%01d", ( $t - int ( $t ) ) * 10 );
my $NCBI_SETTINGS = ncbi_settings();
......
......@@ -377,7 +377,7 @@ if ($TOOLS =~ /gcc$/) {
} elsif ($TOOLS eq 'clang') {
$CPP = 'clang++' unless ($CPP);
$CC = 'clang -c';
my $versionMin = '-mmacosx-version-min=10.6';
my $versionMin = '-mmacosx-version-min=10.10';
$CP = "$CPP -c $versionMin";
if ($BITS ne '32_64') {
$ARCH_FL = '-arch i386' if ($BITS == 32);
......
#define TOOLKIT_VERS 0x02090003
#define TOOLKIT_VERS 0x020A0000