Skip to content

Python reverse-complement Jeremy Zerfas

CONTRIBUTE SOURCE CODE

Python reverse-complement Jeremy Zerfas

This program uses some techniques from my C #6, 7, & 8 programs. Some of the other techniques from the faster C, C++, Rust, etc... programs would be slower with CPython but maybe might be beneficial if something like PyPy or Cython was being used. This program should have run and CPU times that are close to twice as fast as the Python #6 & 2 programs. Memory usage for these multi-process Python programs is harder to measure (and probably not reported accurately by your current benchmarking program) but this program should have lower memory usage than the Python #6 program and should have lower memory usage than the Python #2 program when running on systems with a single CPU core or when using input with a single sequence. The following is a list of the more notable changes over the top, existing Python #6 program:

General

  • I think this program might be more portable than the Python #6 program. The Python #6 program seems to rely on other processes being able to inherit sequence data from the main process which I think will only work with the "fork" process starting method (CPython uses the "spawn" method on Windows and also with recent versions running on macOS) whereas my program passes some sequence data around in shared Queues and the majority of it is read directly from a shared stdin file stream. I think having processes read most sequence data directly from a shared stdin file stream might work best for performance and memory usage in CPython.

Performance

  • Like the other top programs, this program reads data from stdin in 64 KiB chunks instead of reading it in line by line.
  • Like my C #7 & 8 programs, this program uses a faster alternate code path when sequence line lengths are an optimal length.
  • More efficient concatenation of strings.
  • Like the other top programs, this program can start processing sequences as soon as the first one is fully read instead of having to read the entire input first. This can also reduce memory usage (especially when the input has many sequences) in some circumstances.
  • Does NOT use any type of manager process so it requires one less process which reduces run times, CPU times, and memory a bit.

Memory

  • Uses del in a few spots to free up references/memory.

Attach your source code file

Reverse-Complement.py

Provide an example build command-line

python3 Reverse-Complement.py < revcomp-input100000000.txt > /dev/null

FYI, I think for many recent versions of CPython, using the -OO command line option actually slows down most programs a little instead of speeding them up. I was looking into this option some more and it looks like it could have possibly been of more use over a decade ago but improvements in CPython seem to have largely made it irrelevant with recent versions unless slow assert or debug code is being used in a program (with the exception of the fannkuch-redux #4 program, it doesn't look like any of the top Python programs are using assert or debug code). This StackOverflow comment seems to suggest that older CPython versions had SET_LINENO opcodes that got removed when using the -OO option which did improve performance with those versions. This seems plausible based on CPython 2.3's release notes mentioning that that version removed a SET_LINENO opcode. Just running /usr/bin/time python3 -c 'print("Hello world!")' (and a couple other test programs) with and without the -OO option on two systems I have access to seems to suggest that the '-OO' option adds a seemingly constant overhead of up to ~0.2 seconds of CPU/run time (and also seems to increase memory use a few MB too). ~0.2 seconds isn't much for most of the Python benchmark programs (since they often have run times that are over a minute) but for some of the benchmarks where library functions can be used to do much of the work resulting in short run times, that ~0.2 seconds can result in programs being up to around 10% slower.

I haven't currently been able to test this as thoroughly as I would like (my testing was brief and done on virtual machines) so this is something you may just want to investigate. The upcoming CPython 3.11 release seems to be focusing a bit more on improving performance and I also did test a beta of it and it looks like it could possibly help improve this situation so this could maybe just all be moot.