Optimize update repositories task for memory usage and time of execution
Basically moving the ticket from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=755043 here.
When you have a many repositories, the initial
./manage.py tracker_update_repositories is going to take very long and is going to consume lots of RAM (between 4 and 14Gb apparently). We should refactor the process to take fewer time and less RAM.
A few packages that might be useful to diagnose the heavy memory consumption: python3-memory-profiler, python3-memprof. There's also the tracemalloc standard module: https://docs.python.org/3/library/tracemalloc.html