Working with data involves a ton of prerequisites to get up and running with the required set of data, it’s formatting and storage. The first step of a data science process is Data engineering, which plays a crucial role in streamlining every other process of a data science project.
Traditionally, data engineering processes involve three steps: Extract, Transform and Load, which is also known as the ETL process. The ETL process involves a series of actions and manipulations on the data to make it fit for analysis and modeling. Most data science processes require these ETL processes to run almost every day for the purpose of generating daily reports.
Ideally, these processes should be executed automatically in definite time and order. But, it isn’t as easy as it sounds. You might have tried using a time-based scheduler such as Cron by defining the workflows in Crontab. This works fairly well for workflows that are simple. However, when the number of workflows and their dependencies increase, things start getting complicated. It gets difficult to effectively manage as well as monitor these workflows considering they may fail and need to be recovered manually. Apache Airflow is such a tool that can be very helpful for you in that case, whether you are a Data Scientist, Data Engineer, or even a Software Engineer.
What is Apache Airflow?
“Apache Airflow is an open-source workflow management platform. It started at Airbnb in October 2014 as a solution to manage the company’s increasingly complex workflows. ”
Apache Airflow (or simply Airflow) is a highly versatile tool that can be used across multiple domains for managing and scheduling workflows. It allows you to perform as well as automate simple to complex processes that are written in Python and SQL. Airflow provides a method to view and create workflows in the form of Direct Acyclic Graphs (DAGs) with the help of intelligent command-line tools as well as GUIs.
Apache Airflow is a revolutionary open-source tool for people working with data and its pipelines. It is easy to use and deploy considering data scientists have basic knowledge of Python. Airflow provides the flexibility to use Python scripts to create workflows along with various ready to use operators for easy integrations with platforms such as Amazon AWS, Google Cloud Platform, Microsoft Azure, etc. Moreover, it ensures that the tasks are ordered correctly based on dependencies with the help of DAGs, and also continuously tracks the state of tasks being executed.
One of the most crucial features of Airflow is its ability to recover from failure and manage the allocation of scarce resources dynamically. This makes the Airflow a great choice for running any kind of data processing or modeling tasks in a fairly scalable and maintainable way.
In this tutorial, we will understand how to install it, create the pipeline and why data scientists should be using it, in detail.
Understanding the workflow and DAG
The set of processes that take place in regular intervals is termed as the ‘workflow’. It can consist of any task ranging from extracting data to manipulating them.
Direct Acyclic Graphs (DAG) are one of the key components of Airflow. They represent the series of tasks that needs to be run as a part of the workflow. Each task is represented as a single node in the graph along with the path it takes for execution, as shown in the figure.
Image from Source
Airflow also provides you with the ability to specify the order, relationship (if any) in between 2 or more tasks and enables you to add any dependencies regarding required data values for the execution of a task. A DAG file is a Python script and is saved with a .py extension.
Basic Components of Apache Airflow
Now that you have a basic idea about workflows and DAG, we’ve listed below some of the commonly used components of Apache Airflow that make up the architecture of the Apache Airflow pipeline.
Web Server: It is the graphical user interface built as a Flask app. It provides details regarding the status of your tasks and gives the ability to read logs from a remote file store.
Scheduler: This component is primarily responsible for scheduling tasks, i.e., the execution of DAGs. It retrieves and updates the status of the task in the database.
Executer: It is the mechanism that initiates the execution of different tasks one by one.
Metadata Database: It is the centralized database where Airflow stores the status of all the tasks. All read/write operations of a workflow are done from this database..
Now that you understand the basic architecture of the Airflow, let us begin by installing Python and Apache Airflow into our system.
Installing Python and Apache Airflow
Airflow is primarily Python-based but it can be executed for other languages as well. For this tutorial, we will be using Python.
Installing Python is pretty simple and quick. You can download an executable installation file from Python’s official website and run it on your system. Make sure that you download the latest stable build of the Python interpreter for your operating system, i.e. Windows, Linux, Mac or any other: https://www.python.org/downloads.
Once you’ve installed Python successfully, you can proceed towards installing Airflow into your system. Airflow has both a command-line as well as an amazing user interface for monitoring and managing workflows efficiently.
For installing Airflow, you can start by opening up the command prompt/terminal on your operating system. Airflow requires a default location to be installed. So, you will have to specify that first with the following command:
export AIRFLOW_HOME=~/airflow
Now that you’ve specified the location, you can go ahead and run the pip command to install Apache Airflow.
pip install apache-airflow
This will install Apache Airflow into your Python coding environment. And, you are now ready to start creating your first Airflow pipeline.
Create your first Airflow pipeline
The Apache Airflow pipeline is basically an easy and scalable tool for data engineers to create, monitor and schedule one or multiple workflows simultaneously. The pipeline requires a database backend for running the workflows, which is why we will start by initializing the database using the command:
airflow initdb
Upon initializing the database, you can now start the server using the command
airflow webserver -p 8080
This will start an Airflow webserver at port 8080 on your localhost.
Note: You can specify any port as per your preference.
Now, open up another terminal and start the airflow scheduler using the command:
airflow scheduler
Now, if you have successfully started the airflow scheduler, you access the tool from your browser to monitor as well as manage the status of completed and ongoing tasks in your workflow at localhost:8080/admin.
In Conclusion
With all the advanced features and benefits, Apache Airflow is certainly a great tool with limitless potential when it comes to building automated systems. Moreover, the innovative tool has garnered a lot of active users in a short span as it is open-sourced. Have you tried using this tool? If yes, what is your experience using it? Do share with us in the comments below.
Do you want to learn Python, Data Science, and Machine Learning while getting certified? Here are some best selling Datacamp courses that we recommend you enroll in:
- Introduction to Python (Free Course) - 1,000,000+ students already enrolled!
- Introduction to Data Science in Python- 400,000+ students already enrolled!
- Introduction to TensorFlow for Deep Learning with Python - 90,000+ students already enrolled!
- Data Science and Machine Learning Bootcamp with R - 70,000+ students already enrolled!
