How Delta Capita helped a Global Investment Bank integrate Intelligent Automation with Apache Airflow

Our FS clients frequently face challenges with managing complex work flows and data pipelines, leading to high operational costs and lack of trust in data.

For one of our Tier 1 Investment Bank clients, Delta Capita helped solve this problem by integrating Apache Airflow to help them automate and streamline these processes, reducing errors, increasing efficiency, and freeing up resources for other critical tasks. We leveraged Apache Airflow to:

·      Enhance compliance with regulations,

·      Improve auditability and scalability, and

·      Drive digital transformation within the organisation

Airflow can enhance and automate start to end pipelines. A typical E2E process could look something like this:

‍

‍

Using the tool to automate any of the manual steps above will create tangible process efficiencies. Below, we look into the details of how DeltaCapita implemented Airflow to automate several middle and back-office processes.

What did we do?

First, we gained key stakeholder buy-in through demonstrating the power of the tool through proofs of concept (POCs) via automating simple tasks suchas checking the status of on-prem servers.

Next, the DC team began to send automated emails, via Airflow, to report back to team leads on the status of their servers; a task that typicallyrequired team members to log in (even during weekends) and check manually.

From here, stakeholders began to sponsor our team to implement Airflow solutions for increasingly complex tasks.

Here are a few keys to success when launching Apache Airflow, from our experience:

1.    Start with a POC, automating a simple task, before assuming buy-in from less technical stakeholders

2.    Ensure access rights are properly governed; not everybody need have access rights to every part of a user interface (UI)

3.    Familiarise stakeholders with the product through demos and make sure they feel comfortable with its configuration

How does Airflow work?

Airflow uses Directed Acyclic Graphs (DAGs). These graphs consist of one or more tasks which each perform a certain part of a process. Tasks are made using standard, ready to use, operators that come as part of Apache’s package, but you can also create your own. An operator can, for example, be a type of code you use very often, therefore making it easier to repeat or a way of connecting to another platform such as the database. The coding is typically inPython within the Airflow package, enabling you to connect to your UI and schedule to run.

Once finished, an example could look like this:

‍

‍

As the relationships between tasks are customisable, they can be assimple or as complex as needs be. The above graph gives a very quickinterpretable vision of the run status of a DAG.

In this configuration, if a task gets a light green outline it is running; dark green means it succeeded; pink is skipped because of the branching where it should only follow one of the possible paths; and red means it failed. This gives a real-time status update of the process and workflow and allows for immediate intervention to repair a critical process if it’s failed or has been delayed.

‍

Benefits for our Client

‍

How Delta Capita can help you

Our experts can define and implement the most effective solutions with scalability and continuous improvement embedded at its core. This can help you introduce efficiencies and save costs across your business processes. For more information about DC’s extensive Intelligent Automation service offering please reach out directly.

‍