Connector Framework

The core functionality of the Maestrano tool suite is to enable data-sharing between applications and data-mining from information systems. To bootstrap the integration of new third-party software, Maestrano has developed a standardised Connector Framework.

What is a connector?

A Connector is a microservice which purpose is to synchronise data between an application - either SaaS or on-premises (e.g: SalesForce, Shopify, QuickBooks Online, etc...) - and Connec!™ using their public APIs. Its non-intrusive design makes it easy to implement a new connector and benefits from the Maestrano Connector Framework capabilities. It is designed to manage the data synchronisation workflows so the only piece missing is the data-mapping from the thrid-party system and Connec!™ data model.

The core components of the Maestrano architecture interact with connectors:

• Mno-Hub This is the component that will take care of authenticating the user requests via the IdM API.

• Connec! is in charge of storing the company data. It will receive new data in real time via notifications, or regularly via scheduled synchronisations

Here are the key features of the Connector Framework:

Handling of data synchronisation logic

Connectors are configured to synchronise data from Connec!™ and the external Application on an hourly basis (polling). On top of that, Connec!™ sends notifications in real-time to the connector, so if another application of the user eco-system updates a record, it is pushed straight away to the Application.

Some Applications support webhooks that enable real-time data-sharing both ways, notifying the Connectors of updated record as soon as they occur.

Initial Synchronisation

When a Connector is linked to an application for the first time, the following actions are executed

1. Fetch the data from Connec!™ API
2. Fetch the data from the Application API
3. Match and merge records using unique attributes
4. Push updated record to Connec!™ API
5. Push updated record to Application API

Depending on the amount of data to synchronise, this process can take some time.

Historical data

By default, the connector will only retrieve the data that has been created from the moment you linked your account from the external application. However, at any point, you can decide to sync your historical data (all the data created prior to this moment). An option is provided to the end-user to synchronise pre-existing data between applications.

Subsequent Synchronisations

After the first synchronisation has been executed, only the updated records are synchronised, speeding up the process. The same list of actions described above are executed but only processing data modified after the previous synchronisation time.

Real-time data-sharing with Webhooks

Connec!™ notifies the Connector of records updates in real-time using pre-configured webhooks. It means that when Connec!™ receives an update from another application, it is pushed in real-time to the Application.

On the other side, some Applications support webhooks that will be automatically configured by the Connector when linking the account the first time. Doing so enables real-time data-sharing from the other side when the user enters information within the Application.

API authentication

Applications APIs are usually protected with OAuth1, OAuth2 or API tokens. The Connector needs to be customised to respond to the API security requirements.

The most popular Applications may have open source clients to access their APIs implementing their security strategy. In this case low customisation effort is required to access the API.

For OAuth2 protected APIs, it is highly recommended to follow this example using Omniauth

Some other applications use a Basic Authentication strategy. The password is usually an API token that the user can find in the settings of the external application. The connector should then ask for these credentials on the home page.

All these credentials should be stored as encrypted values in the connector's database, so that the user won't have to authorise the application the next time he accesses it.

Once the authentication with the external API has been setup, the connector can perform calls to this API to fetch the data.

Data mapping

Connec!™ - Data-sharing Engine contains our universal data model to store the data coming from all the external applications available on Maestrano's marketplace. Since every application has a different data model, the idea is to map the data from Connec!™ to the model of the external application, and vice versa.

The following diagram shows how the mapping works:

The connector framework provides mechanisms to easily map the fields between the 2 APIs when implementing a new Connector, so the development team only focuses on mapping the data.

You can find Connec!™'s full API specification here.

Connec!™ validates and sanitises the data received by applications to ensure the data quality. Examples of validation are verifying a Contact has always a name specified or that Accounting Journals lines reconcile correctly. Connec!™ also provides data completion in some cases. (e.g. if you send an invoice with only net prices and tax rate, Connec!™ completes the prices automatically).  

Connector customisation

The interface of the connector is generated automatically by the connector framework. It proposes a basic interface, that allows you to perform all the actions required to communicate between Connec!™ and the external application. You are free to customise it as you wish.

Connectors are designed to be extended with custom functionalities to fit your needs. For example, you may want to automatically generate an Invoice whenever a Sales Order has been accepted by a Customer, this way you can accomplish complex workflows between applications without manual data entries.