Mapping entities

Now that you're all setup with both Connec!™ and the external application, it's time to decide which entities (contacts, accounts, events, ...) you want to synchronize. For each type of entity your connector will synchronize, you will need to create a class that inherits from Maestrano::Connector::Rails::Entity.

An example of such a class in provided in the models/entities/ folder, and demonstrates which methods you'll need to implements for each entity. The main thing to do is the mapping between the external entity and the Connec!™ entity. For that, we use the hash_mapper gem (https://github.com/ismasan/hash_mapper). Note that in all the instance methods of those classes, you will have access to four instance variables:

@organization # The organization that it currently synchronizing
@connec_client # The Connec! client
@external_client # The external client created using External.get_client(@organization)
@opts # Some options

A mapping in action:

You'll find the Connec!™ API documentation here: http://maestrano.github.io/connec/, and should also refer to the external application API documentation to find out the fields you can map.

This type of entity class enable one-to-one model correspondence. For more complex needs, please refer to the complex entity section below.

Business Rules

When mapping sensible data it is important to maintain consistency, and some applications might have a complex structure (.i.e allowing multi currency, different time-zones etc...). A guide to Business Rules can be found Here

FAQ

How to handle references between entities?

Let's imagine you are mapping people to contacts, and the have a reference to the organization (or company) they belong to.

The first thing you need to do is to map the two reference fields:

class ContactMapper
  extend HashMapper
 
  map from('organization_id'), to('companyId')
end

Once that is done, the second and only thing you need to do is to declare that this field is a reference, so that the framework know it has something specific to do with it. That's done in a method in your entity class:

class Contact < Maestrano::Connector::Rails::Entity
  def self.references
    %w(organization_id)
  end
end

You can define as many references as you need. Note that you should always declare references using the Connec!™ name of the field.

TEST If all go well, you should see in your specs when mapping from external to Connec!™ that the organization_id field is an array of hashes with a provider, a realm and an id.

If the entity you are mapping has nested sub-fields (e.g. an invoice can have an array of lines, a customer can have an array of notes, a warehouse can have an array of items etc...) you would need to declare them as well:

def self.references
  {
    record_references: %w(organization_id lines/item_id lines/tax_code_id lines/account_id)
    id_references: %w(lines/id)
  }
end

id_references does not prevent the entity to be pushed.

What to do when a field is mandatory in my app but not in Connec!™?

There is two cases here:

It does not make sense to push the record without this field (let's say you're integrating a email campaign application, it does not make sense to push a contact that does not have an email address)

In this case, you should not push. Even more, you should not fetch it from Connec!™ (using $filter). You should also make sure that the documentation include this limitation.

It makes sense to push the record, even without this field

Let's push it then. You'll just need to add a default value in your mapper:

class ProductMapper
  extend HashMapper
 
  map from('description'), to('description'), default: 'Default description'
end

TEST You should also add a test in your specs for when there is no description coming from Connec!™

What to do when an unmapped field is mandatory in my app?

Let's say your mapping products, and the field 'category' is mandatory, but Connec!™ is not giving you any 'category' field

Well, unfortunately, in this case, you need to hard code it using a creation only mapper (make sure it inherits from the 'regular' ProductMapper):

 class CreationProductMapper < ProductMapper
  extend HashMapper
 
  after_normalize do |input, output|
    output[:category] = 'Default category'
    output
  end
end

This will allow the framework to map the default category the first time the entity is pushed from Connec!, but use the 'normal' mapper after that.

Define the method in your simple entities:

def self.creation_mapper_class
  CreationEntityMapper
end

And sub entities:

def self.creation_mapper_classes
  {
    'SubEntityOne' => ::CreationEntityOneMapper,
    'SubEntityTwo' => ::CreationEntityTwoMapper
  }
end

How to create an organization in Connec!™ if my app only supports people/customers?

Some applications only provide support for Customers (People). In order to be able to create an organization in Connec!™, it is possible to use the option: `attach_to_organization`

A common way to implement it, is to map the name of the organization (if provided in another field) in the `before_denormalize` hook inside the mapper:

output[:opts] = {attach_to_organization: input['specific_field'] }

(If the company name is not provided or missing entirely, use 'First name + Last name')

This will create an Organization for each customer/person. Please refer to Connec!™ documentation to find more options.

How to map a field that is a string in Connec!™, but an id in my app?

Okay so now you're mapping items, which can be either a product or a service. In Connec!™ we have a field type that can be either SERVICE, MANUFACTURED or PURCHASED, but in your app you have a type_id that is a meaningless integer.

A typical way to approach this is to retrieve the information about the types in YourApp and set it to a TYPE_MAPPER constant:

TYPE_MAPPER = {
  PURCHASED => 1,
  MANUFACTURED => 2, 
  SERVICE => 3
}

And then use the mapper in the after_normalize/denormalize hooks:

after_normalize do |input, output|
  output[:type_id] = TYPE_MAPPER[input['type']
  output
end

How to map entities that contain an array of subentities?

You're trying to map invoices, and you've noticed that invoices can have several lines, and you don't know how to map that.

You could do some loops in your after_normalize and after_denormalize hooks to handle the lines, but there's actually a far better way to do it:

class InvoiceLineMapper
  extend HashMapper
 
  map from('quantity'), to('quantity')
end
 
class InvoiceMapper
  extend HashMapper
 
  map from('lines'), to('invoice_lines'), using: InvoiceLineMapper
end

How to chose object_name_from_..._entity_hash?

In each entity class, you need to define two methods to give a name to a record when they are coming from either Connec!™ or your app. This name should be using a representative name, such as name, first_name last_name, reference, or something alike. Also, because record will do back and forth between Connec!™ and your app, and we will update the name each time, it is good practice to use fields that are mapped together:

class Event < Maestrano::Connector::Rails::Entity
  def self.object_name_from_connec_entity_hash(entity)
    entity['name']
  end
 
  def self.object_name_from_external_entity_hash(entity)
    entity['event_name']
  end
end
 
class EventMapper
  extend HashMapper
 
  map from('name'), to('event_name')
end

Also please note that the object name is used only for display purposes, so it's nothing critical.

How to use Connec!™ filtering capabilities?

Connec! has a built in filter that can be used to query for a specific subset of entities on a given endpoint. Please refer to:

http://maestrano.github.io/connec/#api-|-querying-filtering-get

I.e. In the entity below ('contact.rb'), you want to fetch only the entities that are leads. In order to maintain a consistent workflow (including web-hooks), a method to override is provided (to be used in conjunction with the filtering):

def get_connec_entities(last_synchronization_date)
  @opts.merge!(:$filter => "is_lead eq true")
  super
end  
      
def filter_connec_entities(entities)
  entities.select{|e| e['is_lead']}
end

I need to do something complex in my mapping, where should I do it?

It is now possible to pass options to the before and after hooks in your mapper classes:

def Mapper
  extend HashMapper
 
  # opts contains all the instance variable of your entity, including opts, organization, connec_client, external_client, and any that you have defined yourself
  after_normalize do |input, output, opts|
    lists = opts[:external_client].find_all('contact_list')
    output[:list] = lists.first
    output
  end
end

The Connector Framework implements a before_sync method, that can be used to store useful data.

A common use is an additional API call to retrieve data from an external endpoint: i.e.

def before_sync(last_synchronization_date = nil)
    super
    Maestrano::Connector::Rails::ConnectorLogger.log('info', @organization, "Fetching #{Maestrano::Connector::Rails::External.external_name} Some Useful Data")
    @useful_data = @external_client.find_all('UsefulData')
    @opts[:useful_data] = @useful_data
end

This entity is read only in my app, what do I do?

If an entity is read only, a method can be used to prevent any data from being pushed to that specific endpoint. This will be implemented in each read only entity.rb file:

def can_write_external?
  false
end

Similarly, the following methods can be used to specify other operations:

def can_update_connec?
  true
end

def can_write_connec?
  true
end

def can_write_external?
  true
end

def can_update_external?
  true
end

I would like to display a different name in the Front-end. Is there a method for it?

Yes, in each entity you can override two methods, one for Connec! entities and one for externals:

# For display purposes only
  def public_connec_entity_name
    'Pretty name to display'
  end

# For display purposes only
  def public_external_entity_name
    'Pretty name to display'
  end

Also don't forget that each entity your connector synchronize should be declared in the External class, and, because the synchronizable entities are stored in the local database for each organization, you'll need to create a migration for existing organization if you add an new entity after the release. (And only after the release. In development, just add the new entity to your organization synchronized_entities list with a rails console).

Business Rules FAQ

I would like to not send updates of the prices field after checking that the currency does not match

In each entity model, you can set metadata on each specific entity IdMap and specify which fields you do not want to be updated:

# Inside the model
  def self.currency_check_fields
    %w(sale_price purchase_price)
  end

# Inside the mapper after_normalize hook
  unless keep_price # will be true if you wanted to keep the price 
    output.delete(:the_price_field)
    idmap = input['idmap']
    idmap.update_attributes(metadata: idmap.metadata.merge(ignore_currency_update: true)) if idmap
  end

The specified fields will now be ignored in subsequent updates from your application.

Triggering synchronizations

In production, synchronizations will be automatically triggered every hour. You can force a synchronization from the front end, or by using the following commands:

Performing the synchronization of all the linked organizations can be triggered by executing

Maestrano::Connector::Rails::AllSynchronizationsJob

The synchronization of a specific organization can be performed by executing

Maestrano::Connector::Rails::SynchronizationJob.perform_later(organization, forced: true)

Webhooks

Connec!™ issues webhooks each time an entity is created or updated. This allows real time integration from Connec!™ to the application you're integrating with.

If the application you're integrating with also support webhooks, you can and should use them to allow a full real time integration. The gem provide a job to help you do that: it will perform the necessary checks, mappings and then push the entities to Connec!™. All you have to do is build a controller and config the necessary routes to catch the webhook, and call this job.

For example:

class ExternalWebhooksController < ApplicationController
  def notification
	# The second argument should be a hash: {"external_entity_name1" => [entity1, entity2], "external_entity_name2" => []}
    Maestrano::Connector::Rails::PushToConnecJob.perform_later(current_organization, entities_hash)
  end
end

Complex entities

If you need a more complex mapping, like one-to-many or many-to-many ones, you can use the complex entity workflow.

The most common case is when an entity can be mapped to two different entities, either internally (Connec!) or externally.

For example `people` and `organizations` are both mapped to `contacts` in MyApp, differentiated by the field `is_organization`

Example of use case:

A Person and an Organization in Connec!™ can both be a Contact

def PersonAndOrganization < Maestrano::Connector::Rails::ComplexEntity
  def connec_entities_names
	%w(person organization)
  end
 
  def external_entities_names
    %w(contact)
  end
 
  ...
end

GENERATE To see how it works, you can run

rails g connector:complex_entity

This will generate some example files demonstrating a one-to-many correspondance between Connec!™ person and external contact and lead data models.

The complex entities workflow uses two methods to pre-process data which you have to implements for each complex entity (see person_and_organization.rb). They are called before the mapping step, and you can use them to perform any data model specific operations.

The entities are managed according to the data model provided

# input :  {
  #             connec_entity_names[0]: [unmapped_connec_entitiy1, unmapped_connec_entitiy2],
  #             connec_entity_names[1]: [unmapped_connec_entitiy3, unmapped_connec_entitiy4]
  #          }
  # output : {
  #             connec_entity_names[0]: {
  #               external_entities_names[0]: [unmapped_connec_entitiy1, unmapped_connec_entitiy2]
  #             },
  #             connec_entity_names[1]: {
  #               external_entities_names[0]: [unmapped_connec_entitiy3],
  #               external_entities_names[1]: [unmapped_connec_entitiy4]
  #             }
  #          }
  def connec_model_to_external_model(connec_hash_of_entities)
    contacts = connec_hash_of_entities['Person']
    organizations = ['Organization']

    # Here all the entities are sent to the same endpoint
    { 
      'Person' =>       { 'Contact' => contacts,
                        },
      'Organization' => { 'Contact' => organizations
                        }
    }
  end

  # input :  {
  #             external_entities_names[0]: [unmapped_external_entity1, unmapped_external_entity2],
  #             external_entities_names[1]: [unmapped_external_entity3, unmapped_external_entity4]
  #          }
  # output : {
  #             external_entities_names[0]: {
  #               connec_entity_names[0]: [unmapped_external_entity1],
  #               connec_entity_names[1]: [unmapped_external_entity2]
  #             },
  #             external_entities_names[1]: {
  #               connec_entity_names[0]: [unmapped_external_entity3, unmapped_external_entity4]
  #             }
  #           }
  def external_model_to_connec_model(external_hash_of_entities)
    contacts = external_hash_of_entities['Contact']

    modelled_hash = {'Contact' => { 'Person' => [], 'Organization' => [] }
                    }
    # Here the entities are separated based on the defining condition
    contacts.each do |contact|
    if contact['is_organization'] 
      modelled_hash['Contact']['Organization'] << contact
    else
      modelled_hash['Contact']['Person'] << contact
    end

    modelled_hash
  end

FAQ

The entity in Connec!™ and in my app have the same name, but I can't create two sub_entities with the same name. What do I do?

If two sub-entities have the same name, you would need to specify in which class they can be found. Let's take, for example, an account entity mapped to two sub-entities: Account e Bank Account.

In this case, you would need to create three sub-entities: two named account.rb and one named bank_account.rb, which does not work very well!

One solution is to provide the names as a Hash, specifying the entity name and the file name separately (in this case MyAppAccount will be found in my_app_account.rb):

def self.connec_entities_name
  %w(Account)
end

def self.external_entities_name
  #{entity_name1: 'ClassName1', #entity_name2: 'ClassName2'}
  {Account: 'MyAppAccount', BankAccount: 'BankAccount'}
end

I need to mix data from different endpoints, where should I do it?

You are trying to map entities that have fields split between different endpoints. A typical example is the mapping of Items/Products and Warehouses/Warehouses.

Let's say that the external API gives the quantity of items inside a specific warehouse in the Item model, while Connec!™ provides a field in Warehouses to keep track of the stocks.

In this case you will have to merge data from the two models, and the best place to do it is inside the complex entity product_and_warehouse.rb:

def connec_model_to_external_model(connec_hash_of_entities)
    # Here we are just assigning the entities arrays to local variables
    items = connec_hash_of_entities['Item']
    warehouses = connec_hash_of_entities['Warehouse']

    # So, for each warehouse_stock (list of items) we want to find the specific item
   
    warehouses.each do |warehouse|
      warehouse['warehouse_stocks'].each do |warehouse_stock|
        next unless warehouse_stock['item_id']

        item_id = warehouse_stock['item_id'].find{|id| id['provider'] == 'connec'}['id']
        item = items.find{|item| item['id'].find{|id| id['provider'] == 'connec'}['id'] == item_id}

        # If the item is present, we create a field that will be used in the ProductMapper class to match the fields present in the external API
        # In our example it will be an array of hashes, each containing the fields: WarehousesIDs and Quantity

        if item
          item['warehouse_quantity'] ||= []
          item['warehouse_quantity'] << { 'WarehouseID' => warehouse['id'], 'Quantity' => warehouse_stock['quantity']}
        end
      end unless warehouse['warehouse_stocks'].blank?
    end unless warehouses.blank?

    {'Item' => {'Product' => items}, 'Warehouse' => {'Warehouse' => warehouses}}
  end

def external_model_to_connec_model(external_hash_of_entities)
    products = external_hash_of_entities['Product']
    warehouses = external_hash_of_entities['Warehouse']


    # Here we need to retrieve quantity and warehouse_id from each Product
    # and map it into a custom field that will be used in the mapper to match Connec!'s warehouse_stocks field
    
    products.each do |product|
      next if product['WarehouseQuantity'].blank?

      product['WarehouseQuantity'].each do |warehouse_quantity|
        complete_warehouse_stock(warehouse_quantity, product, warehouses)
      end
    end unless products.blank? || warehouses.blank?

    result_hash = {}
    result_hash['Product'] = {'Item' => products} unless products.blank?
    result_hash['Warehouse'] = {'Warehouse' => warehouses} unless warehouses.blank?

    result_hash
  end

  private

    def complete_warehouse_stock(warehouse_quantity, product, warehouses)
      warehouse = warehouses.find{|w| w['ID'] == warehouse_quantity['WarehouseID']}
      if warehouse

        # This field will be used in WarehouseMapper to map warehouse_stocks in Connec!

        warehouse['Stock'] ||= []
        warehouse['Stock'] << {'id' => product['ID'], 'item_id' => product['ID'], 'quantity' => warehouse_quantity['Quantity'], 'value' => {'total_amount' => product['CostPrice']}}
      end
end

And voilà, the fields are now matching the different data models. It is now sufficient to match them in the after_normalize/denormalize hooks in product_mapper.rb:

# Mapping to Connec!
# In this case we are just summing the quantity value from each warehouse for the specific item

  after_denormalize do |input, output|
    if input['WarehouseQuantity']
      output[:quantity_on_hand] = input['WarehouseQuantity'].map { |q| q['Quantity'].to_i}.sum
    end
   
    output
  end

# Mapping to YourAPP (External)
# All is left to do is to assign the value of the custom field created before to the appropriate field

  after_normalize do |input, output|
    # warehouse_quantity is a field added manually during pre-processing

    output[:WarehouseQuantity] = input['warehouse_quantity'] if input['warehouse_quantity']
    output
  end

Smart Merging

When creating an entity, you can specify which attributes you want to use to merge with an existing record. If a record is matched, it will behave like an update request and return the existing entity with a 200 status code. This can be specified with the option matching_fields.

In your entity_name.rb specify which field has to be used for smart merging. The typical use case is a required field that will uniquely identify the record:

# In this example the field taken into account is 'name' in Organizations

def self.connec_matching_fields
  ['name']
end

Maestrano Developers

Mapping and synchronization