Hello all! I posted this in Slack, but adding here for additional visibility.
I’m looking for some help in determining whether Kafka is the right tool to replace an existing web/SQL-based integration tool that moves data between a .NET/SQL-based application (System A) and a cloud-based/Oracle application (System B). The way the current integration works is as follows:
- System A has an event that occurs at the completion of an activity. C# code is fired on that activity that gathers up specific records based on joins from various tables and serializes that into a JSON payload, which gets sent to the integration tool.
- Integration tool receives JSON payload and executes logic against each item in a specific order. It has a configuration that defines which fields from System A map to what fields in System B, although the field names are somewhat arbitrary because they don’t necessarily need to match the actual names in each system, they are just the fields as they exist in the payloads. The records have parent/child relationships, such that in order to create the parent items in System B, the integration must first check if each child exists in System B. If the child does not exist, it sends a Create request for the child. If it does exist, it sends an Update request. Then the parent records are sent by a payload that includes the parent/child relationships. These four transactions (QueryForExist, Create, Update, Update Relationships) all have different payloads, but they are currently handled by a single code set in System B that runs when each payload is received at the URL. The integration just keeps hitting the target URL with various payloads and records the results in a log for troubleshooting issues with the integration.
I’m trying to understand how this might be designed in Kafka or if this is maybe a bad fit for so robust a platform. I’m sure it could be done in a few different ways, but an example or two of how the producer, consumer and topic(s) should be modeled would be helpful. What would the Producer and Consumer actually do? What would the Messages in each topic look like? The only model that comes to mind for me is a producer that builds the payload as it does now and stores it as a message in a topic (maybe called System A to System B Push), then have the consumer read that payload and process it, but that seems too simplified and not really leveraging much of what Kafka offers. Plus there’s no error-logging/notification aside from whatever we write into System B’s processing.