Now that you're all setup with Maestrano, it's time to take a look at the external application you are integrating with. Hopefully it has one or several gems for both the authentication process and the API calls. Understanding the API you integrate with is very important, so you need to have a read through their documentation.

You will probably have to request API keys and adds them to the application.yml alongside the Maestrano's ones.

Authentication

You need to implement an authentication process to get a token and be able to do API calls on behalf of the user.

The connector engine is thought to be able to use oauth authentication processes, and an oauth_controller.rb is provided as an example. Please note that it is only an example and you will have to implements most of it, as well as create the needed routes, and use them in the provided view.

For more detailed information, visit: Authentication strategies

If all went well, you should now be able to use the 'Link this company to...' link on the home page. Congrats!

Preparing synchronizations

The aim of the connector is to perform synchronizations between Connec!™ and the external application, meaning fetching data on both ends, process them, and push the result to the relevant parties. The Connec!™ part and the synchronization process itself is handled by the framework, so all you have to do is to implement some methods to bridge the calls to the external application.

External.rb

The first file to look for is models/maestrano/connector/rails/external.rb. It contains three methods that you need to implement:

external_name, which is used for logging purpose only, and should only return the name of the external application, e.g.
```
def self.external_name
  'This awesome CRM'
end
```

get_client, which should return either the external application gem api client, or, in the worst case, a custom API client. e.g
See the section below for a general overview and guidelines
```
def self.get_client(organization)
  MyAppAPIManager.new(organization)
end
```
entities_list that defines the list of all the entities synchronized by the connector
```
def self.entities_list
  %w(item person payment order)
end
```

Entity.rb

The second file is models/maestrano/connector/rails/entity.rb. It contains some methods to retrieve and send data to the external application API, as well as some helpers methods.

The details of each methods are explained in the entity.rb file provided, and you can have a look at the example section.

FAQ

I don't have update timestamps on some of my entities, what should I do?

If creation timestamps are available they can be used as a fallback. In the worst case scenario use Time.now

I don't have any creation timestamps in my app, what should I do?

If update timestamps are available they can be used as a fallback. In the worst case scenario use Time.now

The API does not expose timestamps

If the API does not allow filtering on timestamps or does not expose them, there is no way to filter only updated entities during a synchronization process. In this case, all entities will be re-imported which may impact performances.

Entities shared and naming conventions

When creating your files, some guidelines can be followed in order to keep consistency and avoid confusion.

Files and Classes

When naming your ruby files, the convention is to use the Connec! endpoint name.

I.e. if you are mapping People (Connec!) to Contacts (YourApplication) the ruby file will be named person.rb following Rails conventions.

Your class skeleton would then look like this:

class Entities::Person < Maestrano::Connector::Rails::Entity
  def self.connec_entity_name
    'Person'
  end

  def self.external_entity_name
    'Contact'
  end

  def self.mapper_class
    PersonMapper
  end
  
  # Rest of the code

end

GITHUB Please refer to Maestrano's open source repositories for more examples. Snippets are taken from the BaseCRM repository

If you need to implement complex entities in YourApplication, you can follow a similar process.

I.e. if you are mapping People and Organizations (Connec!) to Contacts (YourApplication) the ruby file for the complex entity would be named person_and_organization.rb.

class Entities::PersonAndOrganization < Maestrano::Connector::Rails::ComplexEntity
  def self.connec_entities_names
    %w(Person Organization)
  end

  def self.external_entities_names
    %w(Contact)
  end

  # Rest of the code
end

You will then have three sub-entities named person.rb , organization.rb and contact.rb respectively. The two mapper files for sub-entities will be named person_mapper.rb and organization_mapper.rb

class Entities::SubEntities::Person < Maestrano::Connector::Rails::SubEntityBase
  
  def self.external?
    false
  end

  def self.entity_name
    'Person'
  end

  # Rest of the code
end

class Entities::SubEntities::Organization < Maestrano::Connector::Rails::SubEntityBase
  
  def self.external?
    false
  end

  def self.entity_name
    'Organization'
  end

  # Rest of the code
end

class Entities::SubEntities::Contact < Maestrano::Connector::Rails::SubEntityBase

  def self.external?
    true
  end

  def self.entity_name
    'Contact'
  end

  def self.mapper_classes
    {
      'Person' => Entities::SubEntities::PersonMapper,
      'Organization' => Entities::SubEntities::OrganizationMapper
    }
  end

  def self.references
    {
      'Person' => Entities::SubEntities::PersonMapper.person_references,
      'Organization' => Entities::SubEntities::OrganizationMapper.organization_references
    }
  end

  def self.object_name_from_external_entity_hash(entity)
    if entity['is_organization']
      entity['name']
    else
      "#{entity['first_name']} #{entity['last_name']}"
    end
  end
end

API Client

If YourApp does not provide a Gem to query the API you would need to create your own.

GUIDELINES

Pagination

Most APIs support pagination options. The following examples assume you will be using RestClient [https://github.com/rest-client/rest-client]

The typical and easiest way to implement a pagination option in your client is possible when the response provided by the API implements '[prev_page]' and '[next_page'] fields:

entities = results_of_your_first_call

  while your_saved_response['next_page'] # or ['some_nested_field']['next_page']
    response = RestClient.get "#{your_saved_response['next_page']}", your_headers
    entities.concat YourDataParser.parse(response.body) # JSON.parse or equivalent if the API returns json
    raise 'No response received while fetching subsequent page' unless response && !response.body.blank?
    break if last_synchronization_date && entities.last['updated_at'] < last_synchronization_date
    your_saved_response = JSON.parse(response)['the_field_containing_next_page'] # The loop will break when '[next_page]' is nil  
 end

entities

If YourApp API does not provide meta fields for pagination or uses a different standard you will have to implement a similar logic:

    # Assuming a first successefull response and data model: 
          {
            "Pagination": {
              "NumberOfItems": 14,
              "PageSize": 200,
              "PageNumber": 1,
              "NumberOfPages": 1
             },
            "Items": [
             {1},
             {2}
            ]
          }
     # Also assuming the API call to get a specific page is: 
       {YourAppUrl}/{entity_name}/{page_number}

    data = JSON.parse(response.body)

    pagination = data['Pagination']
    entities = data['Items']

    if pagination
      while pagination['NumberOfPages'] > pagination['PageNumber']
        next_page = pagination['PageNumber'] + 1
        response = RestClient.get("base_url/#{entity_name}/#{next_page}", query)

        data = JSON.parse(response.body)
        pagination = data['Pagination']
        entities.concat data['Items']
      end
    end

entities

Batched calls

The Connector Framework supports two options that enable batched calls:

__skip and __limit

They can be used to implement specific queries:

batched_call = opts[:__skip] && opts[:__limit]

# You can assign the values based on YourApp API standards (i.e. top/skip)
query_params = {top: opts[:limit], skip: opts[:skip] }

response = RestClient.get "https://base_url/#{entity_name}?#{query_params.to_query}", headers_get

It is possible to set the batch_size for the first synchronization in settings.yml

first_sync_batch_size: 25

Error handling

Two custom classes are available at the moment to handle specific errors:

Maestrano::Connector::Rails::Exceptions::EntityNotFoundError
# and 
Maestrano::Connector::Rails::Exceptions::ExternalAPIError

Use ::EntityNotFoundError if the external API does not provide web-hooks for deletion and entities are not set to INACTIVE (or similar) but deleted entirely
An attempted update on a non existent entity will have to raise this specific error. I.e.:

rescue => e
  if e.class == RestClient::ResourceNotFound
    raise Maestrano::Connector::Rails::Exceptions::EntityNotFoundError.new "The record has been deleted in MyApp"
  else
    Maestrano::Connector::Rails::ConnectorLogger.log('warn', @organization, "Error sending entities", {external_entity_name: external_entity_name, external_id: external_id, message: e.message, backtrace: e.backtrace})
  end

Use ::ExternalAPIError for all the errors returned by the external API.

Token refresh

When implementing the OAuth WebServer flow, you can use this simple implementation to make sure your token is refreshed when needed:

# In your API client

def refresh_token(organization)
  # the logic here would be ideally extracted to another Object/Service
  body = {
    "grant_type" => "refresh_token",
    "refresh_token" => "#{organization.refresh_token}"
  }
  headers = {
    "Content-Type" => "application/json",
    "Authorization" => "Basic #{ENV['your_api_key']}:#{ENV['your_client_secret']}}"
  }
  response = RestClient::Request.execute method: :post, url: "https://your_app_url/oauth2/token",
                                         payload: body.to_json, headers: headers
  parsed_response = JSON.parse(response)
  organization.update(
    oauth_token: parsed_response['access_token'],
    refresh_token: parsed_response['refresh_token']
  )
end

def with_token_refresh
  yield
rescue
  refresh_token(organization)
  yield
end

Now it is sufficient to wrap your API calls in a block:

def get_entities(entity_name, opts = {}, last_synchronization_date = nil)
  with_token_refresh do
    # Your API call and logic
  end
end

Date filtering

In the connector framework last_synchronization_date is the parameter used to implement date filtering:

Your client should provide a way to limit the calls using the date provided and retrieve the entities that have been modified only after the specified date:

def get_entities(entity_name, opts = {}, last_synchronization_date = nil)
  with_token_refresh do
    # Usually, if last_synchronization_date is not nil you should use the API filtering capabilities (if provided) 
    # i.e. add to the query string "update_at=last_synchronization_date"

    # If the API does not support filtering by date, you might fetch the data ordered by updated_at(descending) and limit the number of records.
    # i.e. Break when entities.last['updated_at'] <= last_synchronization_date
  end
end

Maestrano Developers

Integration with the external application