Running an IT team in your organization and without automation part of your operations? Learn why you should consider automating your team’s operational tasks as much as you can.

Automating your organization tasks can offer a number of benefits, such as reducing manual work, reducing human errors, increasing task completion speed, and even saving money on operational costs.

Automation in IT is often portrayed as requiring coding, but there are now many tools that require little or no coding knowledge to set up automations, including Okta Workflows and Zapier, which are two of the most popular ones and I personally love.

Furthermore, if you are familiar with coding, you can use free tools like Google AppScript.

A few examples, how I have automate tasks at my Wonolo:

Okta Workflows:

Okta Workflows is one of my favorite no code automation tool. One of the best things about Okta Workflows is the number of workflows and integrations it offers for the most common SaaS tools you use in your organization right out of the box.

Below are some examples of how Okta Workflows are used at my current company:

1. Automating SaaS provisioning for new hires:

Your organization may already have SSO and SCIM solutions in place, however not all SaaS applications support SSO and SCIM provisioning on their beginning billing tiers, so we are unable to make full use of the SSO and SCIM solutions. However, if there is an API solution for provisioning users, you can always automate manual tasks using APIs.

Using Okta Workflows, you can make API calls with different scenarios:

As an example, automatically calling an API endpoint upon Okta account activation to provision an account in the required SaaS service:

Flow Chart: Atlassian User Provisioning with Okta Workflow

As can be seen in the diagram above, the entire process of provisioning the required user account in Atlassian Jira/Confluence is automated. When the new hire account is activated, it triggers the workflow that automatically creates the user account with the necessary permissions.

2. Zoom Cloud Recordings transfer to User Google Drive:

Automation can also help you with your data cleanups, for example, as Zoom Cloud Storage is expensive and limited, we require our users at organization to save Zoom Cloud Recordings to G Drive, but from the user experience, it can be very non-friendly to manually download all Zoom recordings one by one and upload to their G Drives, since Zoom does not currently allow users or admins to bulk download recordings.

In order to solve the above mentioned problem, I developed a workflow that reads users’ emails from a Google Sheet and copies users’ Zoom Cloud recordings to their Google Drive.

Main Workflow (Zoom Videos tranfer to GDrive)

Helper Flow for the Main flow to parse the JSON result and find the URL to download Zoom Cloud Recording

Last Flow to download the recording from the dowload URL and transfer to the GDrive

Zapier:

The Zapier is another amazing tool that can be used to automate without writing code. Zapier also boasts a wide range of integrations, similar to Okta Workflows.

Slack Alerts

a. In a few ways, I utilize Zapier is to set up various Slack alerts that trigger when the RSS feed is updated. Zapier will then send a POST webhook to the custom Slack App, which will provide us with the information we needed in our IT Slack channel. It has helped our IT team to keep up-to-date on service degradations and product updates.

b. Another way I use Zapier to assist our Sales team is by setting up Salesforce Alerts in Slack. Through Zapier and the Salesforce integrations available, I have created alerts for the Sales team in Slack Channel that notify them about new/updated, expansion, etc opportunities with the required details.

Google AppScript:

Last but not the least, One of my favorite free tools for automating tasks and requiring coding skills is Google AppScript, a great tool in particular if your organization uses Google Workspaces.

One of the features of the Google AppScript is that it integrates really well with the Google Workspace applications.

At Wonolo, we use Google Forms for various tasks and processes, so we need to know when the Google Form has been filled out by the user.

In order to configure Slack alerts for when the forms response is submitted by the user and to sort the data automatically in the forms response Google Sheet based on the date and time submitted, I wrote code in Google AppScript that first sorts the data in the Google Sheet and sends a POST API call to the custom Slack App to send the alert in the Slack channel containings the required data

GitHub code link for the above mentioned example.

Conclusion:

To conclude, automations are really crucial for your business operations and team. If you and your team are currently relying on manual tasks to run your daily IT tasks, you should seriously consider setting up automations to bring the automation culture in to your organization.

Data in attr_encrypted and Active Record Encryption:

attr_encrypted uses two columns in the DB to save the data, the encrypted_attribute and encrypted_attribute_iv. Active Record Encryption uses a single column same as the name of the attribute and saves the data as a hash. For example, to encrypt address in the User model, with attr_encrypted, there are two columns encrypted_address and encrypted_address_iv in DB. With active record encryption, there is a single column in DB, address where the data is saved as a hash as below

{"p":"Hashed Value","h":{"iv":"Hashed Value","at":"Hashed Value"}}

As an example, the article will migrate the address attribute from the User model. The migration includes the following steps,

1. Add Active Record Encryption to the Rails 7 app.

2. Add address and address_tmp columns to DB

3. Create migration to fill data into columns

4. Remove the attr_encrypted gem and drop columns

Step 1: Add Active Record Encryption to the Rails 7 app

The new encryption from Rails 7 uses 3 keys, primary_key, deterministic_key, and key_derivation_salt. Generate them using the following command

bin/rails db:encryption:init

Now save them in your application.rb

config.active_record.encryption.primary_key = "Primary Key"
config.active_record.encryption.deterministic_key = "Deterministic Key"
config.active_record.encryption.key_derivation_salt = "Key Derivation Salt"

Step 2: Add address and address_tmp columns to DB

As mentioned in the introduction, attr_encrypted uses different columns in DB and uses methods to return the value of the address attribute. So the next step is to introduce the column with the same name as the attribute in the DB. But before that, ignore the column in the model.

Class User < ApplicationRecord
  self.ignored_columns = %w[address]
end

Now create a migration to add the column in the User Table. The data can’t be moved directly to the address column since it's still used by attr_encrypted, so create two columns in DB. address and address_tmp. address_tmp will be the temporary column to save the encrypted value. The data is first saved in address_tmp and then the address column is updated using the update_all command. The update_all command does not instantiate the User model or trigger Active Record callbacks and validations, which works perfectly in the current scenario.

class AddAddressAndTmpColumnToUser < ActiveRecord::Migration[7.0]
  def change
    add_column :users, :address, :text
    add_column :users, :address_tmp, :text
  end
end

Now add the encrypts command on the User model from Active Record Encryption which does all the magic. The encrypts point to the address_tmp and not address, because the attr_encrypted gem still uses the address method.

encrypts :address_tmp

Step 3: Create migration to fill data into columns

Create a migration and fill in values into the address_tmp column and then to the address column using update_all. The code that goes into the migration,

User.find_in_batches do |users_batch|
  users_batch.map do |user|
    user.address_tmp = user.address
    user.save
  end
end
User.update_all("address=address_tmp")

Also, add a temporary code in the User model which will save the value in the address and address_tmp columns during updates made by clients using your application.

after_commit :write_address, if: -> { previous_changes[:address] }

def write_address
  self.address_tmp = address
  save!
  User.where(id: id).update_all("address=address_tmp")
end

Now the data is migrated to the address and address_tmp columns. Any updates made by clients are saved as well in the new columns.

Step 4: Remove the attr_encrypted gem and drop columns

Remove the attr_encrypted reference from the code before removing the gem completely. The existing code might look something like the one below

attr_encrypted :address, key: "key"

While removing the above code, also point to the new column address with the encrypted data. In the User model add the following code,

encrypts :address

Remove the references that were temporarily created in the User model,

# to be removed from User model
self.ignored_columns = %w[address]
after_commit :write_address, if: -> { previous_changes[:address] }
encrypts :address_tmp

Also if you are using encrypted_address or encrypted_address_iv directly in your code anywhere, make sure to replace them with the new values from the address attribute. Once all the references are removed, it's safe to remove the gem attr_encrypted from the code.

The final step is to remove the following columns encrypted_address, encrypted_address_iv, and address_tmp from User model.

To know more details on Active Record Encryption, here is the official link https://guides.rubyonrails.org/active_record_encryption.html

Keyword Arguments

Sidekiq allows users to define the signature for the perform method that Sidekiq will call through its perform_async, perform_in & perform_at methods, however these methods do not support keyword & named parameters, symbols & some complex objects. This is because the provided parameters are converted to JSON when the job is pushed to Redis to be run later.

This issue can be missed if testing of a new Sidekiq job is done only against the perform method, and not the asynchronous methods. Sidekiq provides methods for testing new jobs here, along with examples on how to use them. In particular, this example:

HardWorker.perform_async(1, 2)
HardWorker.perform_async(2, 3)
assert_equal 2, HardWorker.jobs.size
HardWorker.drain
assert_equal 0, HardWorker.jobs.size

is sufficient to test that HardWorker can be successfully called without issues arising from Sidekiq failing to properly pass the arguments from perform_async (or its equivalents) to the user defined perform method.

Errors from invalid parameters are exacerbated by Sidekiq’s retry mechanisms, which will repeat invalid calls until the retry limit is reached or the affected code is fixed by a new deployment.

Retries

By default, Sidekiq will retry a job that fails 25 times, with increasing delays. This protects jobs from being lost when they fail due to errors that are resolved before the retry limit is hit. Retries come with a number of tradeoffs, especially if the job is not idempotent.

Ideally, each job would be idempotent and transactional, avoiding cases where only part of the job is completed before failing and cases where the retries lead to duplicate outcomes. In practice some tasks, such as network requests to external services require extra care to avoid complications arising from Sidekiq’s default retry strategy.

Network requests are generally not transactional, as they are reliant on external services which might fail part way through their changes, and requests will eventually timeout. In the event of a read timeout where the external service does not respond quickly enough, it is uncertain if the request succeeded on the other end.

By default, Sidekiq will catch exceptions caused by these timeouts, and retry the job, potentially leading to a duplicate call. Polling the status of the external service before making a call each time can prevent this issue, but will be costly. Additionally, catching the error and managing any retries yourself can prevent Sidekiq's default retries. Some APIs offer idempotency guards, such as Stripe’s idempotency keys, which allow Stripe to identify duplicate requests and ignore them. If all else fails, setting the retry count for a job to 0 may seem like an option, but there are other cases where Sidekiq will retry jobs, mainly when Sidekiq is shut down, intentionally or due to a crash.

Deployment Retries

During a deployment, an old Sidekiq process needs to be shutdown so it can be replaced by a new one. As a result, any jobs still active on the old process will stop running when it is terminated, possibly killing them before they are done their work.

Sidekiq outlines its standard (unpaid) solution to this issue here. Sidekiq will push jobs to Redis to be retried later, provided they do not finish between Sidekiq being notified about an incoming shutdown and the actual shutdown. If Sidekiq is shutdown while it is pushing jobs back to Redis, theunexpected shutdown could trigger super_fetch retries after jobs have already been pushed to Redis for deployment retries. This can lead to duplicate jobs, and is another reason to assume that any job can be retried or duplicated and add guards against that in their code.

The Enterprise version of Sidekiq comes with the option to do rolling restarts, where the old process runs until it clears its current jobs while taking no new ones on. New jobs would go to the new process, and this would prevent deployment retries, at the cost of needing to avoid large database changes while rolling restarts are active, and delays in updating code on long running processes.

super_fetch

Paid versions of Sidekiq come with a feature called super_fetch. When a job fails due to issues with Sidekiq itself (such as out of memory errors), Sidekiq might permanently lose the job. This can also occur if the Sidekiq process is killed before restoring all of its jobs to the queue, typically during deployments. These issues can be mitigated by using multiple queues, spreading the risk of a crash, however this involves additional resources and Sidekiq’s documentation recommends against using too many queues.

With super_fetch, when Sidekiq recovers from a crash or unexpected shutdown it will retry the affected jobs. On the current version of Sidekiq Pro, super_fetch will recover a job this way 3 times. On an older version of Sidekiq Pro (before 5.2), it will have no limits on how many times a job is retried.

These retries come with the same tradeoffs as normal ones, with one additional tradeoff. If a specific job is the cause of Sidekiq crashing, then retrying it can cause all the jobs currently being run by Sidekiq to crash and be retried themselves. This can happen in a loop until the retry limit is hit on the problematic job or the job is manually cancelled (this is the only option on versions without the retry limit).

Unlike normal retries, these must be cancelled manually, by redeploying a version of the code that will not crash, or, as Sidekiq recommends, having the job check Redis for a flag indicating that the job has been cancelled before running. In the wiki’s FAQ, Sidekiq provides code that can be added to a job that implements this check, and allows a job to be cancelled from a console on the Sidekiq process using a job’s id.

Sidekiq only performs super_fetch retries if it shuts down unexpectedly but by default these are logged at the same level as normal job start and end logs. As a result, these can be missed, either buried within less important logs or silenced if Sidekiq’s logging level is set high enough. As of Sidekiq Pro 5.2, Sidekiq will fire a callback (details here) on a super_fetch retry, or a job being killed after reaching its super_fetch retry limit. These can be used to log, alert, and track super_fetch retries as needed.

Race Conditions

Sidekiq jobs run concurrently with the rest of the app they are part of and with other instances of the same job. These asynchronous tasks must account for the presence of others in order to avoid race conditions, between jobs doing similar actions, multiple instances of the same job and between the jobs and the rest of the application outside of Sidekiq.

Problematic race conditions often follow Murphy’s Law; if one can happen, it does, especially since Sidekiq’s retries and job scheduling can lead to jobs running at unexpected times. Sidekiq does not guarantee that jobs will be processed sequentially if it is configured to run multiple jobs at the same time. With some concurrency jobs created back to back can be run at the same time, in a non sequential manner. Each job’s code should account for the possibility of duplicate jobs on top of other parts of the application affecting the same database entries and external services.

Job Scheduling

Sidekiq allows jobs to be scheduled at specific times (with a few seconds delay based on how often Sidekiq will check Redis for scheduled jobs & retries, which is outlined here). These scheduled jobs are not guaranteed to be run immediately once polled, but instead enter the queue at the bottom. If there is a backlog of jobs, the scheduled jobs will be delayed until the jobs in front of them are cleared.

This can cause jobs to run at unexpected times, and can cause race conditions for periodic jobs scheduled close enough together.

For example, if a Sidekiq job is scheduled to run every 10 minutes, and the queue is backed up enough to delay them by 10 minutes, then multiple instances of the job can exist in the queue at the same time. When the issue is resolved and the queue clears, these jobs can be run back to back, or concurrently depending on how many processes are running on a Sidekiq queue. If a job does not account for this possibility, this can lead to unexpected or duplicate results.

Locking

Locking is a common solution for most potential race conditions. Only one active thread in an application can hold a properly implemented lock (such as the database row level locking strategies provided by Rails for ActiveRecord objects), removing any potential for race conditions. A common issue with locks is with race conditions occurring between checking the state of an object and updating it, often called a Time of Check vs Time of Use (TOCTOU) race condition.

If a developer is unaware of the need to lock before reading important data, it can lead to code like:

def unsafe_change_state_from_not_started_to_started
  if state == 'not_started'
    self.with_lock do
      update(state: 'started')
    end
  end
end

If another, nearly identical method called unsafe_change_state_from_not_started_to_cancelled exists, a race condition can happen if one method is still holding the lock while the other loads the object. This can lead to the object’s state being unintentionally changed to started after it was changed to cancelled, or vice versa.

Checks made after calling Rails’s lock! or with_lock methods are safe from this issue, as they will reload the locked object as part of seizing the lock. Similarly, Rails' optimistic locking accounts for this by rejecting an update if the database object has been updated elsewhere since it was originally loaded from the database. Rails locks only protect locked objects, leaving any unlocked objects within a lock exposed to race conditions without additional locks or protections. Non-Rails locks that do not reload important objects within the protected code will be similarly exposed.

Conclusion

This is not a full account of every possible issue that can arise while using Sidekiq. Many that are not covered here are detailed in the very helpful Sidekiq wiki. On top of that, more recent issues might only be present in the Sidekiq issues page, and paid versions of Sidekiq come with the option of emailing Sidekiq’s creator for support.

I started at Wonolo in 2019. Back then, there was no rotation, no on-call onboarding, no clear expectations, no playbooks, and no pager system—just alerts on a single slack channel. If you were a backend engineer, it was expected that you fix any alerts and bring the system back into a stable state. Without rotations and well-defined processes, it was much easier for backend engineers to constantly be burdened with potential production issues, risking burnout and fatigue.

Today, we have a much more defined process. We make use of an on-call rotation and onboard all new engineers that contribute code to the system. When an incident does occur, we utilize blameless Post Mortems and update any on-call playbooks to make sure we incorporate anything that happened during the incident.

Rotation: Who Goes On Call

Everyone. More specifically, every Engineer, Engineering Manager, Director and VP at Wonolo is expected to join an on-call rotation three months after their start date. The following chart is taken from our on-call policy in Confluence:

How We Do On-Call Onboarding

The main goal for our on-call onboarding is to set up everyone for success and ensure they are as comfortable as possible before starting that first on-call rotation. We achieve this by having well-written documentation that is sent out in advance of an onboarding session. The documentation includes the following documents:

• On-Call Overview: Policy, rotation, responsibilities, escalation, mitigation, and triage
• On-Call Setup: How to properly set up your mobile device
• On-Call Incident Response: How to handle suspected and confirmed production issues
• On-Call Playbooks: Describes symptoms, impact, action, and helpful links

The onboarding session itself outlines clear expectations for being on call and, more importantly, helps you feel like you are not alone. We try to reassure everyone that it's always okay to ask anyone for help or to escalate issues in the on-call rotation.

Clear Expectations: The Responsibilities of Being On Call

The ultimate goal of being on call is to remediate any issues with the platform deemed a P0.

A P0 means that the system is in an unhealthy state, not functioning, and should be fixed or mitigated down to a lower priority level as soon as possible. We have reduced the main responsibility to three steps:

1. Alert: Acknowledge receipt of pager message.
2. Assess: Determine and communicate impact to stakeholders.
3. Mitigate & Triage: Resolve or reduce to P1, and triage to the appropriate team.

Creating a Useful On-Call Playbook

The On-Call Playbook was created with ease of use in mind. We created a table which has a row for every alert we have created and every symptom we have encountered. Each row contains Symptom, Impact, Action, and Helpful Links. Here is a small sample of the on-call playbook:

Symptom Contains the exact text that maps to an alert.

Impact The best guess of the impact on the system to help speed up the assessment and prioritization of the situation.

Action Suggested action for the on-call person to take.

Helpful Links Contain links to the exact logs, supporting metric charts, how-to documentation, etc.

Conclusion

The on-call process has evolved significantly since the beginning of my time here at Wonolo. As we grow and learn as a team, there will always be more for us to iterate and improve on, and I’m grateful we’ve built processes for our team around doing so. Having a shared responsibility across engineering, creating an onboarding program, defining clear expectations, and creating an easy-to-use on-call playbook are all important pieces of setting your team up for success.

class Foo < ActiveRecord::Base
  before_save :run_callback, if: -> { some_attr_changed? && some_other_attr_changed? }
​
  def run_callback
    # do something
  end
end

I'm mostly concerned with these ones that run during save.

before_validation
after_validation
before_create
after_create
before_update
after_update
before_save
after_save
​
# I include custom active model validations because they operate in much the
# same way and cause similar headaches.
validate

​ Unfortunately, for various reasons, callbacks tend to be overused. Their power can be alluring. They allow you to add code that will run whenever an object is saved to the database without needing to update the save call sites with any new code. There also exists the influence from arguments about Fat Controllers & Thin Models vs. Thin Controllers & Fat Models. While these arguments have died down, echoes remain in the way many developers write Rails code. ​

What also remains are the concrete dependencies that were created when the Thin Controllers & Fat Models methodology was taken to the extreme. Developers were attempting to make controllers completely disappear and codify them into abstractions. Libraries like inherited_resources popped up that completely removed the need to write traditional CRUD controller code. Without any other place for the code to go it would end up being shoehorned into Active Record callbacks. inherited_resources was ultimately abandoned by its original author and deemed a misadventure. However, ActiveAdmin remains a popular library to construct admin panels with and continues to use inherited_resources with a "no controllers" coding style.

At our company these effects lead to a codebase with too many callbacks which has caused us many headaches. As well, a tendency for 'God Objects' to accrue in long lived projects compounded our situation. The combination of these problems left us with a few classes that had over seventy callbacks and custom validations defined on them. This makes reasoning about saving, creating and updating these objects incredibly difficult for a multitude of reasons: Callback code tends to be side effect heavy. Unscrupulously making just in time changes to the model attributes before save or validation. Schlepping data between systems to provide eventual consistency. Triggering updates on another model via the ever-present recalculate! method and of course, last but not least, firing off background jobs to do god knows what. ​

Another aspect of side effects that's important to understand is the order they occur in. A good trick to pull on a co-worker is ask them the order of execution on these callbacks when you create a Foo record without looking at the documentation:

class Foo < ActiveRecord::Base
  before_create :one
  before_save :two
  after_commit :three
  after_commit :four
end

The order will, oddly enough, be: two, one, four, three; see the documentation for further details. ​

Side effects and state changes in the system are the most difficult thing to understand because of their far reaching consequences. Now thread the application of them through a gamut of conditionals that are nigh impossible to understand. Having fun yet? You may as well be staring into the Ark of the Covenant instead of trying to understand the code. "It's not that bad! Just fire up the debugger." you might say. Oh wait, right, since callbacks are declared at the class level you can't actually step through them with a debugger as you would imperative code and the conditionals defined on them often can't be inspected in an easy way at runtime. ​ ​ Unlike Indy, in our case, we could not just close our eyes. We wanted to regain the debuggability and readability of simple imperative Ruby code while maintaining interoperability with the existing code that uses our objects. This meant that if we had a model Foo we didn't want to go to hundreds of places in our code to modify calls to #save, #update, .create, etc. While that might be an admirable goal at some point it wasn't within the scope of this project. We tried a couple different proof of concepts and ultimately landed on something structured like this:

class Foo < ActionRecord::Base
  alias_method :active_model_validate, :valid?
​
  def valid?(...)
  end
​
  alias_method :validate, :valid?
​
  alias_method :active_record_save!, :save!
​
  def save!(...)
  end
​
  def save(...)
  end
end

All entry-point methods that invoke callback style code ultimately execute these methods. By aliasing the methods before overriding them we preserve access to the original implementations. This provides a starting point for our refactor. The theory being: all we need to do is delete the callback declarations from the model and write our callback code inline in these methods. Let's look at an example of what that might look like. First the before code with some nasty callbacks.

class Foo < ActionRecord::Base
  before_validation :before_validation_callback, if: -> { some_attr_changed? }
  after_validation :after_validation_callback, if: -> { some_other_attr_changed? }
  before_create :before_create_callback, if: -> { some_attr_changed? && some_other_attr_changed? }
  after_create :after_create_callback, if: :some_condition?
  before_update :before_update_callback, if: :some_other_condition?
  after_update :after_update_callback
  before_save :before_save_callback, unless: -> { some_attr_changed? }
  before_save :before_save_callback_on_create, on: :create, if: :some_condition?
  before_save :before_save_callback_on_update, on: :update
  after_save :after_save_callback, if: -> { some_other_attr_changed? }
  after_save :after_save_callback_on_create, on: :create
  after_save :after_save_callback_on_update, on: :update, unless: :some_other_condition?
​
  validate :custom_validation
  validate :maybe_custom_validation, if: :some_condition?
  validate :custom_validation_on_create, unless: :some_other_condition?, on: :create
  validate :custom_validation_on_update, if: -> { some_attr_changed? }, on: :update
​
  # referenced methods would be defined below...
end

That's only 16 callbacks and custom validations on a single model. Imagine seventy. Now let's refactor this into our structure above. With some careful reading of the documentation and some experimentation we can discern the expected execution order of these callbacks. Leading us to code that looks like this.

class Foo < ActionRecord::Base
  alias_method :active_model_validate, :valid?
​
  def valid?(...)
    # Before Validation callbacks
    before_validation_callback if some_attr_changed?
​
    # Call the original validations. 
    active_model_validate(...)
​
    # Custom validations
    custom_validation
    maybe_custom_validation if some_condition?
    custom_validation_on_create if !some_other_condition? && creating_record
    custom_validation_on_update if some_attr_changed? && !creating_record
​
    # After Validation callbacks
    after_validation_callback if some_other_attr_changed?
​
    errors.blank?
  end
​
  alias_method :validate, :valid?
​
  alias_method :active_record_save!, :save!
​
  def save(...)
    save!(...)
  rescue ActiveRecord::RecordInvalid, ActiveRecord::RecordNotSaved
    false
  end
​
  def save!(**opts)
    creating_record = new_record?
    saved = false
​
    # All callback code runs inside a single transaction
    Foo.transaction do
      # Throw :abort can be used inside callbacks to skip saving
      catch(:abort) do
        # Don't validate if validate: false was passed in
        if opts[:validate] != false
          raise ActiveRecord::RecordInvalid, self unless valid?
        end
​
        # Before save callbacks
        before_save_callback unless some_attr_changed?
        before_save_callback_on_create if creating_record && some_condition?
        before_save_callback_on_update if creating_record
​
        # Before create/update callbacks
        if creating_record
          before_create_callback if some_attr_changed? && some_other_attr_changed?
        else
          before_update_callback if some_other_condition?
        end
​
        # Save
        saved = active_record_save!(**opts.merge(validate: false))
      end
​
      # Raise error if record wans't saved
      raise ActiveRecord::RecordNotSaved unless saved
​
      # After create/update callbacks
      if creating_record
        after_create_callback if some_condition?
      else
        after_update_callback
      end
​
      # After save callbacks
      after_save_callback if some_other_attr_changed?
      after_save_callback_on_create if creating_record
      after_save_callback_on_update, if !creating_record && !some_other_condition?
    end
  rescue StandardError => e
    # Signal "rollback"
    rolledback!
    raise e
  end
​
  # referenced methods would be defined below...
end

After the refactor you end up with some pretty ugly code. Make no mistake though, just because there were less lines before doesn't mean it was better or clearer. Our refactor has brought us numerous advantages. It's now possible to stick a debug statement at the top of save! and step through all custom validations and callbacks. You can even be debugging another piece of code and step into the save or save! call and step through it. All the weird control flow that was buried in the Active Record documentation is now obvious. Also, now that we are dealing with plain old Ruby code we've opened up more avenues to factor this code out of our model. ​

Which is exactly what we wanted to do considering how large our model was. At Wonolo we have chosen the Command pattern to fill out our service layer so we explored factoring out the callback code into a command. We created two new commands, one for validating and one for saving. These commands can be filled with the code from the valid? and save! methods respectively.

def valid?(*args)
  Commands::Foo::Validate.new(self, *args).call
end
​
def save!(**opts)
  Commands::Foo::Save.new(self, **opts).call
end

With these two new command classes we now have a place to put code. Not only can we move the contents of these methods but with some minor tweaks we can start factoring out the callback method definitions themselves. ​

I think there are clear advantages to ditching callbacks for more traditional style code. As shown here it's possible without needing to change call sites all over the codebase. I think there is still more work to be done to get the code to the place we want it. However, this strategy provides a great starting point to get the ball rolling. From here I can easily see this code being broken up further into create and update commands. Even that may not be the correct stopping point. Depending on the complexity of the model, create and update could be subdivided further into finer grained commands. ​

So far I've also only written about doing this on a single model by hand. To do this to all models across a large code base is going to be a lot of effort. Fortunately, the code transformations to do this are quite rote and a good candidate for automation. So that's exactly what we did to perform this refactor across our entire codebase. However, that's a topic for another day, so stay tuned.

1. Empathy. At the core of every organization are the relationships and bonds between the people on the team. It's important to give direct feedback, but it has to be delivered with empathy and genuine belief in good faith.
2. Respect. Treating others with respect sets the foundation for successful interactions. Respect comes in all forms, from allowing others to voice their opinions even if you disagree, to preparing for and giving your attention in meetings, to understanding how decisions you make will affect cross-team milestones or deadlines.
3. Conflict. There will always be conflict within any team, but the process by which conflict is resolved carries far more weight than the conflict itself. Disagreeing with someone shouldn’t be construed as being difficult, as long as it’s done honestly and with good intent towards prioritizing what’s best for the customer and company.
4. Career advancement. Every engineer on the team has a path to be successful, whether they’re an individual contributor or a manager. The skills necessary to be a good manager are not the same as the skills needed to be a good software engineer. Becoming a manager is often a lateral move, not a promotion.
5. Technical ability. At the end of the day, we are a tech company and it’s our team’s responsibility to drive engineering excellence. And while we value technical expertise in managers, they won’t necessarily be the strongest technical person on a team.
6. Prioritization. It’s important for an engineering team to see the trees through the forest. Our goal is to understand where we’re going and build a path to get there. Managers should be removing obstacles from their team, and should be prioritizing overall team productivity. We should make choices to improve engineering efficiency throughout the organization.
7. Communication. Keeping lines of communication open allows us to do our jobs better and more efficiently. Not only is communication between engineering teams important, but so is cross-team communication. Whether it be Product and Design, RS&O, or Customer Support, our engineers need to build strong relationships so we can build the best product for our customers.
8. Quality. QA is not the first line of defense against bugs. Engineers are ultimately responsible for their code and the bugs they ship. You can have a bug once, you shouldn't have it twice. Writing a test is often the best way to do that.
9. Hiring. One of the highest leverage actions an engineer can do is hiring. Growing our team by adding new engineers can elevate both the level of technical expertise and the culture within the organization. We hire for culture add, not just culture fit.
10. Metrics. We are a metrics driven organization, it allows us to concretely say whether the features we are shipping are moving the needle and getting us the results we want. We don’t ship features for the sake of shipping features.

Thanks for reading, and hope you enjoyed this post! I’m sure these ideas will change as the organization continues to evolve, but hopefully this gives you some insight into how the Wonolo engineering team functions.

• Waynn, VP of Engineering

Welcome to the Wonolo engineering blog! Wonolo is an on-demand staffing platform that helps people find consistent work. We’re a two-sided marketplace that matches workers (or as we call them, Wonoloers) to jobs all over the US.

We’ve been a technology company since day one, from the first versions of our apps to the complex ML models that now power our matching algorithms. We're starting this blog to share insights we’ve gathered through building the software that powers everything at Wonolo, and we hope this will be a tool for other engineers to get an inside look at how things work here, both technically and philosophically.

Some topics we’re hoping to cover in the upcoming weeks and months: Ditching Active Record Callbacks, Migrating to SwiftUI, Building a new code ownership tool, and many more. We’re super excited to get this started, stay tuned for more!

• Waynn