General principle

Performance testing is part of the Software Development Life-Cycle and aims to ensure that API endpoints return successful responses in a reasonable amount of time, even when they have to deal with an unusual amount of data. It also allows to verify that the overall performance does not decrease as more features are developed.

The process of performance testing follows these three steps:

Generation of demo data for all the entities
Starting of the application in the background - Execution of a JMeter script on the tested endpoints
Assessment of performances by comparing the response times with predetermined thresholds

Demo data generator

:generate_demo_data method

To facilitate the execution of performance tests, the generation of demo data has been added to the Reporting Framework. Each entity must define a ":generate_demo_data" method, which will create a number of "demo" records when called.

class Company < ActiveRecord::Base
  include BaseEntity
  # [...]

  # Generates `count` Company objects (default: 1)
  # :nocov:
  def self.generate_demo_data(count = 1)

    # gem ruby-progressbar is used to represent the progress of the data generation when the rake task is called.
    # See: https://github.com/jfelchner/ruby-progressbar/wiki for documentation
    progress_bar = ProgressBar.create(
      title: "#{self}: Generating #{count} records...",
      format: "%t \e[0;32m|%b>>%i| %p%%\e[0m | %a"
    )

    t = Time.zone.now

    # Returns an array of hashes representing the records to add to the DB
    companies_hashes = (1..count).map do

      # Increments the progress bar at each loop, for 33%
      progress_bar.progress += (33 / count.to_f)

      id = SecureRandom.uuid
      {
        id: id,
        channel_id: id,
        name: ::Faker::Company.name,
        currency: 'AUD',
        financial_year_end_month: (1..12).to_a.sample,

        # Don't forget to add the "is_demo" flag!
        is_demo: true
      }
    end

    # Increments the progress bar by 67% in twice the time it took to increment the first 33%
    progress_timer(progress_bar, 2 * (Time.zone.now - t), 67)

    # gem activerecord-import is used to create records in batches (rather than in separate INSERT statements as per the ActiveRecord standard
    # https://github.com/zdennis/activerecord-import
    import(companies_hashes, validate: false)

    # Put the progress at 100%
    progress_bar.finish
  end
  # :nocov:

  # [...]
end

Note there is no need to write unit tests for data generation methods as the generated "demo" data are not meant to be used in production, but only for performance testing purpose.

Rake tasks

Then, two rake tasks can be used to clean and generate these demo data:

# Cleans all the records that have the "is_demo = true" flag from all the tables
rake data:clean

# Cleans the records and generates a new set of demo data
# Finance bolt: will generate 1 Company with 10,000 Invoices and 10,000 Bills (default)
rake data:generate

# Finance bolt: will generate 5 Companies with 500 Invoices and 500 Bills each
rake data:generate[5, 500]

To make sure the :generate_demo_data method of your entity is called by the rake task, the entity must be added to /lib/tasks/data.rake:

# List of entities to generate, from top to bottom of the dependency tree
# Int corresponds to the default number of entities to generate PER PARENT
# Eg: `Account => 50` will generate 50 demo accounts per demo company
def entities
  @entities ||= {
    Company => 1,
    Account => 50,
    Invoice => 10_000,
    Bill => 10_000,
    Journal => 5,
    JournalLine => 2
  }
end

In this method, the entities must be added by order of depency (eg: "Company" should be before "Account" as a Company "has_many" accounts)

End result (finance bolt default)

Performance tests

JMeter tests

Once the demo data is generated, another rake task will be invoked to run JMeter tests on the widgets.

All the performance tests should be added to the /test/performance directory. The file /test/performance/widgets_test.rb includes a JMeter plan to test all the widgets. It will:

Call INDEX /api/v1/organizations?filter[is_demo]=true and take the first "demo" organization's channel_id to use it in the following requests
Call each widget 50 times, passing the channel_id in the request
Verify whether the mean response time was below the threshold for each widget

The gem ruby-jmeter is used: https://github.com/flood-io/ruby-jmeter

When a new widget is implemented, a new test should be added to widgets_test.rb. Eg:

test do
  # [...]
  # Simulates one user calling the widget endpoint 50 times
  threads count: 1, loops: 50 do
    transaction name: 'CashProjection' do
      # Use the helper method `url` to build the full url corresponding to your widget endpoint
      visit(name: 'CashProjection Widget', url: url('cash_projection'))
    end
  end
  # [...]
end.run

# That's it!

Rake tasks

The task rake test:perf can be invoked to generate the demo data and run the JMeter tests:

# Generates the demo data, starts the bolt in the background on port 4001, and runs the JMeter tests
rake test:perf

# Skips the demo data generation
rake test:perf[1]

# Skips the demo data generation and bolt starting (only runs the JMeter tests)
rake test:perf[1,1]

By default, "rake:test" or "rake" will only run the codebase tests (brakeman, bundle_audit, rubocop). "rake test:all" will run Rspec, the codebase tests and the performance tests.

In Codeship

The full performance testing process has been added to a Codeship pipeline for the finance bolt project. This way, the build will break if the performance tests are below threshold.

This pipeline must run on the "development" Rails environment as Warden does not properly authenticate the incoming requests when running on "test".

Here is a basic pipeline configuration to run the performance tests:

export RAILS_ENV=development
bundle exec rails db:schema:load
bundle exec rake test:perf

Browser not supported