Migrating to Protobuf

Last updated on January 26, 2021

We are currently in the process of trying to modernize our system and integrate it with all of our various datasources.

Current Technology

Currently, our system is based around a heavily customized Microsoft Dynamics CRM (MSCRM) 2015. It’s been migrated from a MSCRM 2013 installation and before that a bespoke system based around ASP.Net (if I remember correctly). It’s been the desire of several members of the organization to move back to a more bespoke system. Both for flexibility that MSCRM doesn’t give us and to hopefully be able to eke out a little more performance by removing the bloat. Not to mention letting go of all the default entities that aren’t being used.

There are various steps that will need to be taken to get this project off the ground as we’re replacing both the front and back end. The first step would be to write services that can read and write to the existing system. Use it as a shim or stopgap. Once that’s in place, we can rewrite the front end to use the new services. Then we can eventually migrate the database to the new system and everything is good. That’s an incredibly short way to describe a lot of work.

Moving to .NET 5.0 and all that entails also means avoiding Windows Communication Foundation (WCF). The preferred method to create services is gRPC. gRPC services and messages are defined using protobuf. This means writing protobuf messages for every Entity we want to preserve. Then services for retrieving, updating, and otherwise manipulating those Entities.

Missing Types

One thing that MSCRM loves that is missing in protobuf are decimals. Or it might be more accurate to say I love decimals where I need fractions but also accuracy. However, this is an area where protobuf is lacking. We need to make our own Decimal type and reference it in our messages.

The other thing is that the MSCRM API uses EntityReferences for related records. This EntityReference is simply the GUID and a string representing the name of the record. Seeing as how this is a custom type from top to bottom, there is no protobuf equivalent. But, we don’t need to worry about that, because we can just make those items the appropriate messages. Then when we return the object, we can just return the GUID and name. Here’s an example of what I mean.

message Player {
string id = 1;
string name = 2;
Team school = 3;
}

message Team {
string id = 1;
string name = 2;
}

The only thing to be wary of is cyclical imports as I’m trying to keep each table in its own .proto file

Converting SQL Tables

I could manually type everything. But I’m not a masochist. The system tables have most of what I need to spit out rough protobuf messages for a table. They won’t be perfect, but they will be a lot closer to what I need than a blank file. I won’t say it’s pretty, but I will say it works well enough so far. Here’s the github link.

Basically, we start the script by just laying out some default lines that will be present in most, if not all, protos we define. Then we add some commented out lines. These will be copied into the main service proto, but they all fit a certain pattern. I have not thought about if I could make these methods work on multiple message types, but for now, I just want to get to the next part to see this work.

Then we define the major messages, a Request message, a Query message, a Response message, and a message representing the table itself. The Request, Query, and Response message are all fairly boilerplate. They may also not be the final forms. A lot of this is exploratory at the moment. The table message is the meat of the script. It’s basically all of the columns in the table in a huge string. We do some translation of SQL Types into protobuf types. Since I didn’t want to spend time mapping types the current system doesn’t use, I also included an UNKNOWN TYPE in case any of the tables has a type I haven’t included.

Next Steps

First, I”m going to need to cull all of the cruft from the few tables I’ve selected as my base. Then I’m going to need to actually implement the services. And put something behind the Decimal proto so we can appropriately handle it.

toast Written by: