Integrating your people systems to improve data flow and save time

Efficient data pipelines running between systems are central to a cohesive people technology ecosystem. However, many organisations run into trouble when copying employee data from their 'source of truth', usually an HRIS or payroll system, to other systems which need the same data. This can mean plenty of manual work, which might include double handling of data, unwieldy spreadsheets and tedious data corrections, making life much more difficult for HR and payroll staff.

This is where system integration comes in. In this article we outline some of the most common integration pathways, with a focus on HRIS and payroll, to showcase how integrations are typically handled in the people technology space. Keep in mind that integration solutions differ widely depending on the systems and data involved, so your exact case may not be reflected here.

1.     Connectors

Many HRIS and payroll systems come with premade connectors which are simple to set up as they have been designed to interface directly with other specific systems. Connectors are typically configured once and can then be left alone, requiring little maintenance and monitoring. They can also be managed entirely through the system interface and are usually easy for admins to set up. Sometimes connectors come bundled with the system and in other cases they must be purchased from a vendor's app marketplace for an additional charge. Typically, connectors are the preferred method for integration if they are available and meet your use case due to their ease of use. We recommend chatting to your vendor about your needs, as connectors may be available which are not advertised, or their development teams may be open to building a new one.

2.     Flat files

The most common integration method is flat file, which usually refers to a CSV, XLSX or TXT file (or files) containing data being exported from one system and imported into another. The files are 'flat' because they have no explicit relationships between their records as there may be in a relational database. In the most common scenario, one system exports a CSV to a server via FTP (file transfer protocol) where it is picked up and 'ingested' by another system on a set schedule. Flat-file integrations can run into issues; for example, if the FTP server goes down, passwords are changed, the time window for import is missed, or data comes through with problematic formatting, then the import process can fail. Some systems account for this by only failing if there is a certain percentage of errors, though others can reject a file entirely due to a single error. The main benefit is that flat file capability is usually available out of the box and is free to use. If flat file is your preferred integration pathway, it's important to ensure your data is exported in a consistent format and at the same time. Flat-file integrations typically occur nightly or up to several times a day; for something more regular, a connector or custom API integration is often more suitable.

3.     Low-code integration platforms

There are several services providing no- or low-code integration services, such as Workato, Tray.io, Zapier or Power Automate. These platforms support many applications and give you the ability to create custom workflows using their platform. Like prebuilt connectors, you are limited to what systems the platform supports, though the added benefit is their typically easy drag and drop design and ability to support multiple triggers and events. For example, when a new employee is created in a supported system, it may trigger an event to create a record in another system, add the employee to a mailing list and then email them a welcome message. Integration platforms charge for their service, though some offer free trials. It’s worth shopping around to see if one may meet your needs.

4.     Custom API

Another type of integration is achieved via API, or application programming interface. An API is like a gateway into a system's data which we can work with using code. Typically, prebuilt connectors and low-code integration platforms use APIs to achieve their connections, so in this scenario we are talking about custom built API integrations for systems which do not have connectors available or businesses with more complex requirements.

A custom API integration requires both systems to have an API available, which may not be the case for legacy systems. It also relies on having the right 'endpoints', or locations of data resources, for the kind of data you wish to bring across. For example, if you want to sync employee data from an HRIS to payroll, then both systems need to have the correct type of employee data endpoint in their APIs. Most vendors have comprehensive API documentation which outlines their API's capabilities and can be used to see if your use case is supported.

Another thing to consider is data transformation. For example, if the date format needs to change when moving data from one system to another, the API integration code (or 'script') can manage this transformation for you. This, as well as the many different variations of API across vendors, means that custom API integrations are scoped on a case-by-case basis and require significant testing before moving into a production environment. The benefit compared to other integration types is in their customisation, with the ability to transform data in complex ways, frequent syncs if required, custom error handling and logging, custom retry policies and so on.

A term you may come across when looking into the API capabilities of your systems is 'webhook'. This is similar to a regular API but works in response to an event within your system. For example, when a new employee is added to your HRIS, you could set up a webhook to react to that event by automatically sending the new employee record out to a certain location. In this way, webhooks 'push' data out of a system through its API. A standard API integration 'requests' the new employee record from the system before it can send it on. Webhooks make it easier to stay on top of frequent system events and can mean near real-time syncs between systems.

Custom API integrations are usually more suitable in cases where frequent updates are needed, for example, hourly syncs. As mentioned, webhooks can help to make connections even more efficient. There is, however, a trade-off in terms of cost; usually more frequent syncs and larger data payloads attract a higher cost due to higher demand on hosting infrastructure. More complex integrations also require longer development time. It's helpful to think about the amount of data moving, its frequency, direction and any transformations needed at the early stages.

5.     Hybrid

Given the different capabilities of cloud software packages, sometimes a hybrid solution is needed. For example, one system may not have an API, while another may not support FTP import. In this case you could export a flat file from the first system to a cloud location, where it is picked up and read by a script which then sends the data into the second system via its API. This could also happen vice versa, with data pulled from one system through its API and written to a flat file, which is then picked up by the second system. There are many use cases involving different system capabilities, data types, volumes and frequencies, which means that a detailed scoping process is needed when designing any custom solution.

This article has outlined the most common ways to integrate people systems, including connectors, flat files, low-code platforms, custom API integrations and hybrid solutions. We have used HRIS and payroll as typical examples, though this may of course apply to other systems like recruitment, health & safety and performance. Each business has its own unique people technology ecosystem. Integration can help to smooth out the journey of data around your business with the benefit of freeing staff from tedious manual work. Talk to Tomorrow's People today about your integration needs, and we can help to get your data pipelines humming.