Looping Through Repetitive Tasks

Automating Repetitive Tasks in Bioinformatics: A Beginner's Guide to Command-Line Loops

Decoding Biology Shorts is a new weekly newsletter sharing tips, tricks, and lessons in bioinformatics. Enjoyed this piece? Show your support by tapping the ❤️ in the header above. Your small gesture goes a long way in helping me understand what resonates with you and in growing this newsletter. Thank you!

---

Humans are prone to making errors during repetitive tasks, especially when those tasks involve rote execution. Consider this scenario: you have a Python program called cool_analysis_script.py that you need to run on 100 different data files, each producing a corresponding output file. If you were to manually execute the script for each file, you’d need to type commands like:

$ python cool_analysis_script.py file_1 > output_1  
$ python cool_analysis_script.py file_2 > output_2  

…and so on, repeating the process 100 times. Not only is this process tedious, but it also significantly increases the likelihood of errors, such as typos in file names or output paths. Even if you execute all the commands perfectly, this approach wastes valuable time.

Instead, you can leverage your computer’s capabilities to automate this repetitive work, making your process faster, more accurate, and more reproducible. Writing a simple loop is an effective solution to this problem. By using a loop, you let the computer handle the repetitive task of varying inputs and outputs, which reduces errors and provides a clear, automated record of your workflow. This reproducibility is critical in bioinformatics, where maintaining a detailed record of analyses ensures that results can be revisited and verified later.

In Bash (a widely-used command-line shell in bioinformatics), loops are straightforward. The general syntax is:

$ for variable in list; do command; done  

Let’s apply this to the earlier example. To automate the process of running cool_analysis_script.py on files numbered 1 through 100 and saving the results, you can use the following one-liner:

$ for i in {1..100}; do python cool_analysis_script.py file_$i > output_$i; done  

This command iterates over the numbers 1 to 100, substituting the value of i into file_$i and output_$i for each iteration. The result is the same as manually typing all 100 commands, but without the risk of human error.

Although this loop can be run directly in the command line, it’s often better practice to store it in a script file for improved organization and maintainability. Here’s how you can do this:

1. First, create a new script file with a text editor, such as nano:

$ nano run_cool_script.sh  

2. Then, inside nano, write the loop:

$ for i in {1..100}; do python cool_analysis_script.py file_$i > output_$i; done  

3. Next, save and exit nano by pressing Ctrl+0, then return, and Ctrl+X.

4. After that, make the script (run_cool_script.sh) executable with the following code:

$ chmod +x run_cool_script.sh  

5. Finally, run the script as follows:

$ ./run_cool_script.sh  

By using this approach, you save time, reduce mistakes, and create a reproducible workflow—a hallmark of good bioinformatics practice.

PS — If you’re new to working with the command line, I recommend checking out my guide, Bash Fundamentals for Bioinformatics, available here: Bash Fundamentals for Bioinformatics. This guide covers foundational concepts that will enhance your efficiency and confidence in bioinformatics scripting.

---