.gitignore

@@ -5,6 +5,7 @@ vgcore.*
 /build/Makefile.config*
 /build/MSVC/2010/ipch
 /build/MSVC/2010/ncbi-vdb.opensdf
+build/ld.linux.exe_cmd.sh
 /centos
 /ilib64
 /lib64
@@ -34,4 +35,5 @@ core
 *.pyc
 .gdb_history
 .vscode
-.gdb_history
+*.cache
+.vs

CHANGES.md

 # NCBI External Developer Release:
 
 
+## NCBI VDB 2.10.2
+**January 15, 2020**
+
+  **build**: install used to fail on OSX when checking md5  
+  **kfg**: Users now provide the ngc access token on the command line  
+  **kfg**: potential memory leak removed  
+  **kfg, vdb-config, vfs**: new names for caching dbGaP files  
+  **kfg, vfs**: added new configuration node for resolving protected data  
+  **klib**: fixed incorrect access to memory in output writers  
+  **kns**: KNS now provides limited support for attaching files to HTTP POST requests  
+  **kns, vfs**: added support of ngc files and JWT-carts in SDL requests  
+  **krypto**: fixed decryption of dbGaP data on the fly  
+  **vfs**: allow to set dbGaP context by ngc file  
+  **vfs**: fixed detection of cache location of public runs on cloud  
+  **vfs**: fixed double-free when processing names service response  
+  **vfs**: fixed names conflict  
+  **vfs**: new naming scheme for caching dbGaP files  
+
+
+## NCBI VDB 2.10.1
+**December 16, 2019**
+
+  **vfs**: fixed generation of unique file name when caching
+
+
+## NCBI VDB 2.10.0
+**August 19, 2019**
+
+  **cloud**: added support for user-pays access to GCP  
+  **cloud**: cloud-related code was moved to new module  
+  **cloud, vfs**: send cloud instance identity just when user allows it  
+  **kfg, sra-tools**: use trace.ncbi.nlm.nih.gov to call names service  
+  **kfg, vdb-config**: alternative remote repository URL was added to default configuration   
+  **kfs**: introduced readahead strategy for cloud storage  
+  **klib, vdb**: error report is saved to ncbi_error_report.txt  
+  **kns**: Don't log mdebtls error messages. Added possibility to enable them.  
+  **kns**: We now use system root CA certs on Unix   
+  **kns**: increased default value of HTTP read to 5 minutes; made it configurable  
+  **kns**: introduced configurable controls over network timeouts  
+  **kns**: special support for handling SDL interaction  
+  **prefetch, vdb**: adjustments for latest name resolution service  
+  **prefetch, vfs**: added support of "run accession as directory"  
+  **prefetch, vfs**: added support of download of reference sequences in "run accession as directory"  
+  **prefetch, vfs**: fixed regression when prefetch does not download vdbcache  
+  **sratools**: VDB can get URLs- for local/remote/cache locations of accession from environment  
+  **tui, vdb-config**: new look and cloud specific options in 'vdb-config -i'  
+  **vdb**: VDBManagerMakeSRASchema() was deprecated  
+  **vdb**: make greater use of data returned by latest name resolver  
+  **vfs**: added possibility to have multiple remote repositories for backward compatibility  
+  **vfs**: added possibility to set resolver version from configuration  
+  **vfs**: allow to use SDL as remote service  
+
+
+## NCBI VDB 2.9.6
+**March 18, 2019**
+
+  **prefetch, **vfs**: fixed regression that prevented re-download of incomplete files  
+
+
+## NCBI VDB 2.9.4-1
+**March 4, 2019**
+
+  **sra-tools, **vfs**: fixed regression introduced in 2.9.4 release causing delay when starting sra tools  
+
+
+## NCBI VDB 2.9.4
+**January 31, 2019**
+
+  **kns**: added detection of zone where cloud instance runs  
+  **kns**: added support of AWS authentication  
+  **kns**: allow to access googleapis.com via proxy  
+  **kns**: don't add "Accept: */*" request header when it already contains an Accept header  
+  **kns**: updated mbedtls to version 2.16  
+  **sra-tools, vfs**: added support of realign objects  
+  **vdb**: A Json parser is now available in klib  
+
+
 ## NCBI VDB 2.9.3
-**October 17, 2018**
-  **kns**: added possibility to skip server's certificate validation
-  **kns**: expect to receive HTTP status 200 when sending range-request that includes the whole file
+**October 11, 2018**
+
+  **kns**: added possibility to skip server's certificate validation  
+  **kns**: expect to receive HTTP status 200 when sending range-request that includes the whole file  
   **vdb**: fixed a bug in accessing pagemap process request for cursors which do not have pagemap thread running
 
 
 ## NCBI VDB 2.9.2-1
 **July 31, 2018**
+
   **vdb**: restored lost exported attribute on several entries in the private API
 
 
 ## NCBI VDB 2.9.2
 **July 23, 2018**
+
   **kfg, vfs**: Introduced enhanced handling of download-only NGC files that lack read/decrypt permissions
 
 

README.md

 # The NCBI SRA ( Sequence Read Archive )
 
 ### Contact:
-email: sra-tools@ncbi.nlm.nih.gov
+email: sra@ncbi.nlm.nih.gov
 
 ### Change Log
 Please check the CHANGES.md file for change history.
@@ -10,18 +10,10 @@ Please check the CHANGES.md file for change history.
 The SRA Toolkit and SDK from NCBI is a collection of tools and libraries for
 using data in the INSDC Sequence Read Archives.
 
-### IMPORTANT ANNOUNCEMENT
-As was announced in [OMB memorandum M-15-13](https://www.whitehouse.gov/sites/default/files/omb/memoranda/2015/m-15-13.pdf) on June 8, 2015, NCBI and all Federal systems will be transitioning to using HTTPS-only protocols before the end of 2016. This change will affect any software that uses NCBI APIs such as the E-utilities or NCBI software toolkits such as `sra-tools`, `ncbi-vdb` or `ngs`.
+### Antivirus Warnings
+Windows Defender and a few other anti-virus tools have recently been [falsely](https://www.virustotal.com/#/file/042af43fc1318b07a4411c1bea2ff7eed8efe07ca954685268ce42f8f818b10e/detection) warning about test-sra.exe.
 
-The NLM and NCBI may implement the switch to HTTPS-only as early as September 30, 2016.
-
-In particular, software products that depend on `sra-tools`, `ncbi-vdb` or `ngs` may not function as expected after September 30 unless they are properly updated from this site or by the software provider.
-
-If you use software that accesses NCBI SRA data in any way, your software will likely be affected by this change. Please check with your software provider for recent udpates or patches, and be sure to acquire these before September 30.
- 
-If you develop software that relies on `sra-tools`, `ncbi-vdb` or `ngs` in any way, you will likely need to update your code so that it accesses NCBI using HTTPS.
-
-We have released new tools with version 2.8.0 that are HTTPS compatible and `M-15-13` compliant as of October 7, 2016. Please be certain to [update all of your binaries](https://github.com/ncbi/sra-tools/wiki/Downloads) and configuration files.
+We suggest verifying [MD5 checksums](https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/current/md5sum.txt) after downloading.
 
 For additional information on using, configuring, and building the toolkit,
 please visit our [wiki](https://github.com/ncbi/sra-tools/wiki)

build/Makefile.clang

@@ -27,7 +27,7 @@
 CC = @ $(TOP)/build/cc.sh $(OS) 'clang -c' \
 	 $(CHECKSUM) --objx $(OBJX) --cflags "$(CFLAGS)" -MD
 CP = @ $(TOP)/build/cc.sh $(OS) 'clang++ -c' \
-	 $(CHECKSUM) --objx $(OBJX) --cflags "$(CPFLAGS)" -MD
+	 $(CHECKSUM) --objx $(OBJX) -stdlib=libc++ --cflags "$(CPFLAGS)" -MD
 
 # C preprocessor
 PP = gcc -E $(CFLAGS)

build/Makefile.env

@@ -82,7 +82,6 @@ endif
 # load OS and COMP specific files
 include $(TOP)/build/Makefile.$(OS)
 include $(TOP)/build/Makefile.$(COMP)
-include $(TOP)/build/Makefile.rules
 
 # external schema modules
 EXT_SCHEMA_MODULES = axf sraxf wgsxf vxf
@@ -98,7 +97,12 @@ ifeq (win,$(OS))
     TEST_BINDIR = $(BINDIR)
 else
     TEST_BINDIR = $(TARGDIR)/test-bin
+    FLIBDIR = $(TARGDIR)/fuzz
+
+$(FLIBDIR):
+	mkdir -p $@
 endif
+
 ILIBDIR = $(TARGDIR)/ilib
 LIBDIR = $(TARGDIR)/lib
 OBJDIR = $(TARGDIR)/obj/$(MODULE)
@@ -127,10 +131,10 @@ JMAKE_CMD = $(MAKE) BUILD=$(BUILD) TOOLSET=$(TOOLSET) COMP=$(COMP) LNKG=$(LNKG)
     TOP=$(TOP) SRCDIR=$(SRCDIR) OUTDIR=$(OUTDIR) -C $(CLASSDIR) -f $(SRCDIR)/Makefile
 
 # directory and link creation
-SUBDIRS = bin test-bin ilib lib
+# compilation rules
+SUBDIRS = bin test-bin ilib lib fuzz
 OUTDIRS = schema
 
-# compilation rules
 MAKE_JOBS = 1
 
 stdcompile: makedirs
@@ -267,6 +271,8 @@ purify:
 purecov:
 	@ echo pcov > $(TOP)/build/BUILD.$(OS)
 
+fuzz: makedirs
+
 .PHONY: out CC GCC ICC debug profile release scm purify purecov local local require-proxy-exec $(RHOSTS) bindir report_config
 
 # includes based upon build
@@ -315,6 +321,9 @@ VPATH = \
 	$(SRCDIRS_OS) \
 	$(SRCDIR)
 
+# include rules now that VPATH is fully set
+include $(TOP)/build/Makefile.rules
+
 INCDIRS = \
 	$(addprefix -I,$(OVERRIDEINC)) \
 	$(addprefix -I,$(SRCDIRS_OS)) \

build/Makefile.gcc

@@ -75,7 +75,8 @@ ifeq (dbg, $(BUILD))
 	PED = -std=gnu11 -pedantic # -fdiagnostics-show-option
 else
 	# TODO: use -fstack-protector-strong for gcc >= 4.9
-	OPT = -O3 -Wno-variadic-macros -fno-strict-aliasing -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector -Wa,--noexecstack -Wall -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-c++11-extensions
+	OPT = -O3 -Wno-variadic-macros -fno-strict-aliasing -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector -Wa,--noexecstack -Wall
+#    -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-c++11-extensions
 endif
 
 ifeq (mac,$(OS))

build/Makefile.rules

@@ -22,7 +22,6 @@
 #
 # ===========================================================================
 
-
 ## build rules
 LOC_INFO = -D__mod_name__=$(MODULE) -D__file_name__=$*
 
@@ -109,3 +108,33 @@ LOC_INFO = -D__mod_name__=$(MODULE) -D__file_name__=$*
 # version include generation
 %.vers.h: %.vers
 	$(TOP)/build/make-vers-inc.sh $^ > $@
+
+# instrumented for fuzzing
+#	clang -c $^ -o $@ $(DEBUG) $(DBG) $(CARCH) $(PROF) $(DEFINES) $(ARCHDEFS) $(MIN_DEPLOY_OS_OPT) $(INCDIRS) $(LOC_INFO) -D__file_ext__=cpp  -fsanitize=fuzzer,address,signed-integer-overflow
+FUZZ_OPT = -fsanitize=fuzzer,address,signed-integer-overflow -fprofile-instr-generate -fcoverage-mapping
+$(OBJDIR)/%.fuzz.$(OBJX): %.c
+	clang -c $< -o $@ $(CFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.fuzz.$(LOBX): %.c
+	clang -c $< -o $@ $(CFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.fuzz.$(OBJX): %.cpp
+	clang++ -c $< -o $@ $(CPFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.fuzz.$(LOBX): %.cpp
+	clang++ -c $< -o $@ $(CPFLAGS) $(FUZZ_OPT)
+
+$(OBJDIR)/%.nopt.fuzz.$(OBJX): %.c
+	clang -c $< -o $@ $(CFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.nopt.fuzz.$(LOBX): %.c
+	clang -c $< -o $@ $(CFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.nopt.fuzz.$(OBJX): %.cpp
+	clang++ -c $< -o $@ $(CPFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.nopt.fuzz.$(LOBX): %.cpp
+	clang++ -c $< -o $@ $(CPFLAGS) $(FUZZ_OPT)
+
+$(OBJDIR)/%.swap.nopt.fuzz.$(OBJX): %.c
+	clang -c $< -o $@ $(CFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.swap.nopt.fuzz.$(LOBX): %.c
+	clang -c $< -o $@ $(CFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.swap.nopt.fuzz.$(OBJX): %.cpp
+	clang++ -c $< -o $@ $(CPFLAGS) $(FUZZ_OPT)
+$(OBJDIR)/%.swap.nopt.fuzz.$(LOBX): %.cpp
+	clang++ -c $< -o $@ $(CPFLAGS) $(FUZZ_OPT)

build/Makefile.targets

@@ -30,6 +30,7 @@
 #   runtests
 #   slowtests
 #   valgrind
+#   fuzz
 #
 # requires $(SUBDIRS) to be defined
 #
@@ -62,7 +63,7 @@ SUBDIRS_STD = $(addsuffix _std,$(SUBDIRS))
 
 std: $(SUBDIRS_STD) $(STD_EXTRA)
 
-$(SUBDIRS_STD): 
+$(SUBDIRS_STD):
 	@ $(MAKE) -C $(subst _std,,$@) std
 
 .PHONY: std $(SUBDIRS_STD)
@@ -115,3 +116,15 @@ $(SUBDIRS_VALGRIND):
 	@ $(MAKE) -C $(subst _valgrind,,$@) valgrind
 
 .PHONY: valgrind $(SUBDIRS_VALGRIND)
+
+#-------------------------------------------------------------------------------
+# fuzz
+#
+SUBDIRS_FUZZ ?= $(addsuffix _fuzz, $(SUBDIRS))
+
+fuzz: $(SUBDIRS_FUZZ)
+
+$(SUBDIRS_FUZZ):
+	@ $(MAKE) -C $(subst _fuzz,,$@) fuzz
+
+.PHONY: fuzz $(SUBDIRS_FUZZ)

build/Makefile.vers

@@ -23,4 +23,4 @@
 # ===========================================================================
 
 # NCBI-VDB and library version
-VERSION = 2.9.3
+VERSION = 2.10.2

build/install-kfg.sh

@@ -39,13 +39,15 @@ TGT_FILE=$3/$1
 mkdir -p ${KONFIG_DIR}
 
 # create a backup if installed file has been modified by user
-if [ -f ${TGT_FILE} ] ; then
+if md5sum --help > /dev/null ; then
+  if [ -f ${TGT_FILE} ] ; then
     md5=$(md5sum ${TGT_FILE} | awk '{print $1;}')
     # echo "$1 md5=$md5"
     if [ "$(grep ${md5} ${MD5SUMS})" == "" ] ; then
         # not a known version of the file; create a backup copy
         mv -v ${TGT_FILE} ${TGT_FILE}.orig
     fi
+  fi
 fi
 
 # copy to the install location

build/ld.linux.exe_cmd.sh

-EXE_CMD="$LD -static-libstdc++ -static-libgcc"

debian/changelog

+ncbi-vdb (2.10.2+dfsg-1) UNRELEASED; urgency=medium
+
+  * New upstream version
+  * debhelper-compat 12 (routine-update)
+  * Standards-Version: 4.4.1 (routine-update)
+  * Adjusted patches.
+
+ -- Steffen Moeller <moeller@debian.org>  Sat, 18 Jan 2020 00:37:25 +0100
+
 ncbi-vdb (2.9.3+dfsg-2) unstable; urgency=medium
 
   * Replace remainin occurrence of vdb_mbedtls_ssl_conf_authmode by

debian/compat

-12

debian/control

@@ -3,7 +3,7 @@ Maintainer: Debian Med Packaging Team <debian-med-packaging@lists.alioth.debian.
 Uploaders: Andreas Tille <tille@debian.org>
 Section: science
 Priority: optional
-Build-Depends: debhelper (>= 12~),
+Build-Depends: debhelper-compat (= 12),
                d-shlibs (>= 0.82),
                libbz2-dev,
                libxml2-dev,
@@ -14,7 +14,7 @@ Build-Depends: debhelper (>= 12~),
                libmbedtls-dev,
                zlib1g-dev,
                default-jdk-headless
-Standards-Version: 4.3.0
+Standards-Version: 4.4.1
 Vcs-Browser: https://salsa.debian.org/med-team/ncbi-vdb
 Vcs-Git: https://salsa.debian.org/med-team/ncbi-vdb.git
 Homepage: https://github.com/ncbi/ncbi-vdb

debian/patches/add_-msse2.patch

@@ -3,8 +3,10 @@ Last-Update: Wed, 31 Aug 2016 07:55:11 +0200
 Bug-Debian: https://bugs.debian.org/836128
 Description: Add -msse2 to libs/search
 
---- a/libs/search/Makefile
-+++ b/libs/search/Makefile
+Index: ncbi-vdb/libs/search/Makefile
+===================================================================
+--- ncbi-vdb.orig/libs/search/Makefile
++++ ncbi-vdb/libs/search/Makefile
 @@ -39,6 +39,8 @@ OBJFILES = \
  
  include $(TOP)/build/Makefile.env

debian/patches/debug_configure.patch

@@ -3,11 +3,13 @@ Last-Update: Tue, 12 Jul 2016 16:15:32 +0200
 Description: Force debugging output (--with-debug in d/rules is ignored
  by the build system)
 
---- a/setup/konfigure.perl
-+++ b/setup/konfigure.perl
-@@ -448,6 +448,7 @@ if ( $PKG{REQ} ) {
- 
- my @dependencies;
+Index: ncbi-vdb/setup/konfigure.perl
+===================================================================
+--- ncbi-vdb.orig/setup/konfigure.perl
++++ ncbi-vdb/setup/konfigure.perl
+@@ -463,6 +463,7 @@ if ( $PKG{OPT} ) {
+     }
+ }
  
 +$OPT{'debug'} = 1;
  my %DEPEND_OPTIONS;

debian/patches/find_hdf5_serial.patch

@@ -4,9 +4,11 @@ Description: use hdf5 serial
  May be there is a more elegant way to use hdf5 rather than patching the source
  but I did not found it.  Any hints are welcome.
 
---- a/setup/konfigure.perl
-+++ b/setup/konfigure.perl
-@@ -1523,8 +1523,8 @@ sub check_compiler {
+Index: ncbi-vdb/setup/konfigure.perl
+===================================================================
+--- ncbi-vdb.orig/setup/konfigure.perl
++++ ncbi-vdb/setup/konfigure.perl
+@@ -1620,8 +1620,8 @@ sub check_compiler {
              $flags = $n;
              $log = '                      int main() {                     }\n'
          } elsif ($n eq 'hdf5') {
@@ -17,8 +19,10 @@ Description: use hdf5 serial
          } elsif ($n eq 'fuse') {
              $flags = '-D_FILE_OFFSET_BITS=64';
              $library = '-lfuse';
---- a/libs/hdf5/Makefile
-+++ b/libs/hdf5/Makefile
+Index: ncbi-vdb/libs/hdf5/Makefile
+===================================================================
+--- ncbi-vdb.orig/libs/hdf5/Makefile
++++ ncbi-vdb/libs/hdf5/Makefile
 @@ -104,7 +104,7 @@ KDF5_OBJ = \
  	$(addsuffix .$(LOBX),$(KDF5_SRC))
  

debian/patches/find_ngs-sdk.patch

@@ -2,9 +2,11 @@ Author: Andreas Tille <tille@debian.org>
 Last-Update: Tue, 12 Jul 2016 16:15:32 +0200
 Description: Enable finding ngs-sdk
 
---- a/setup/konfigure.perl
-+++ b/setup/konfigure.perl
-@@ -1332,7 +1332,9 @@ sub find_in_dir {
+Index: ncbi-vdb/setup/konfigure.perl
+===================================================================
+--- ncbi-vdb.orig/setup/konfigure.perl
++++ ncbi-vdb/setup/konfigure.perl
+@@ -1349,7 +1349,9 @@ sub find_in_dir {
          print "\tlibraries... " unless ($AUTORUN);
          if ($lib) {
              my $builddir = File::Spec->catdir($dir, $OS, $TOOLS, $ARCH, $BUILD);
@@ -15,7 +17,7 @@ Description: Enable finding ngs-sdk
              my $ilibdir = File::Spec->catdir($builddir, 'ilib');
              my $f = File::Spec->catdir($libdir, $lib);
              print "\n\t\tchecking $f\n\t" if ($OPT{'debug'});
-@@ -1413,8 +1415,8 @@ sub find_in_dir {
+@@ -1430,8 +1432,8 @@ sub find_in_dir {
          }
      }
      if ($jar) {
@@ -26,8 +28,10 @@ Description: Enable finding ngs-sdk
          if (-e "$try") {
              println $try unless ($AUTORUN);
              $found_lib = $try;
---- a/setup/package.prl
-+++ b/setup/package.prl
+Index: ncbi-vdb/setup/package.prl
+===================================================================
+--- ncbi-vdb.orig/setup/package.prl
++++ ncbi-vdb/setup/package.prl
 @@ -31,8 +31,8 @@ sub REQ { ( { name    => 'ngs-sdk',
                option  => 'with-ngs-java-prefix',
                origin  => 'I',

debian/patches/fix_install_target.patch

@@ -4,8 +4,10 @@ Description: Install target tries to install to /usr/lib64 which is broken on
  one hand and on the other hand I have not found a proper method to stop this
  by less invasive means
 
---- a/Makefile
-+++ b/Makefile
+Index: ncbi-vdb/Makefile
+===================================================================
+--- ncbi-vdb.orig/Makefile
++++ ncbi-vdb/Makefile
 @@ -65,7 +65,8 @@ $(SUBDIRS_STD):
  install: 
  	@ echo "Checking make status of object libraries..."

debian/patches/i386-uint64_msbit.patch

@@ -6,12 +6,12 @@ Author: Steve Langasek <steve.langasek@ubuntu.com>
 Last-Modified: 2018-08-26
 Bug-Debian: https://bugs.debian.org/907358
 
-Index: ncbi-vdb-2.9.1-1+dfsg/interfaces/cc/gcc/i386/arch-impl.h
+Index: ncbi-vdb/interfaces/cc/gcc/i386/arch-impl.h
 ===================================================================
---- ncbi-vdb-2.9.1-1+dfsg.orig/interfaces/cc/gcc/i386/arch-impl.h
-+++ ncbi-vdb-2.9.1-1+dfsg/interfaces/cc/gcc/i386/arch-impl.h
-@@ -127,6 +127,13 @@ int32_t uint32_msbit ( uint32_t self )
-     return rtn;
+--- ncbi-vdb.orig/interfaces/cc/gcc/i386/arch-impl.h
++++ ncbi-vdb/interfaces/cc/gcc/i386/arch-impl.h
+@@ -75,6 +75,13 @@ int32_t uint32_msbit ( uint32_t self )
+     return -1;
  }
  
 +static __inline__