Improving LDAP Group Sync Time using Parallelism

Hello Everybody.


This article talks about how you can utilise a new functionality built to improve the Time it takes to Sync User Groups in Harness when using LDAP as an Auth Mechanism.


Harness supports Single Sign-On (SSO) with LDAP implementations, including Active Directory and OpenLDAP. Integrating Harness with your LDAP directory enables you to log your LDAP users into Harness as part of Harness’ SSO infrastructure. Once you integrate your Harness account with LDAP, you can create a Harness User Group and sync it with your LDAP directory users and groups. Then the users in your LDAP directory can log into Harness using their LDAP emails and passwords.

This is done by configuring the Group Query. The Group Query section is used to search the LDAP directory for user groups that Harness will sync with to create Harness user groups. When you set the Group Query here, you are setting the scope wherein any searches for LDAP groups will be performed.

When you integrate your User Groups in Harness with LDAP a Sync Job is fired which not only Syncs this particular groups but all the other Groups in the Account as well in a serial order. By default, Harness syncs with your server every 15 minutes. If your LDAP server is slow in responding to queries it can cause a ripple effect on other systems. When the number of User Groups in Harness crosses above around 700 or 800 you might observe a few delays in the time it takes to sync all of them and this is due to the sync job taking each group one by one in a serial order. Although this would work it would not be ideal at scale and would result in more time for Group sync than expected.

To improve this behaviour and provide a better solution, Harness has implemented an LDAP Job Iterator which runs the Group sync in Parallel. This Feature is currently behind the Flag PL_LDAP_PARALLEL_GROUP_SYNC, which when enabled on the Account runs the jobs on multiple threads and this further improves the Sync time on the Harness end.

We tested the improvement between the Serial LDAP Group Sync and Parallel Group Sync and could see the below metrics :

  • With Serial Group Sync for around 800 Groups we could observe the time taken was around ~30 mins.

  • With Parallel Group Sync for 800 Groups we could observe the time taken was around ~15 mins.

The above tests were performed with a single delegate which handles the LDAP Sync Tasks, Increasing the delegate pool will improve the sync time further as more threads get added.