Custom Core Data migration example

This post is a tutorial on custom Core Data migrations. There is no starter project that you can download and work on following the action steps outlined in this post. But there are action steps in this post that you can follow while developing your own app. And there is an example – my own app – of which you will see relevant parts of code and some screenshots. You can also download my app from the App Store to see what it’s all about. Throughout this post I’ll be telling you what to do, referring all the time to the example of my app. So, it’s going to be “Do this, do that. And by the way, here is how I’ve done it in my app”. 

A little bit of history

I’d been wanting to learn custom Core Data migrations for a very long time and here is why. I have an old app (written in Objective-C) which I wrote when I started learning iOS development. Actually, I was learning iOS development by writing that app. With very little previous experience in programming, I’d made some funny decisions about the data structure of the app. I released several versions with some minor changes and design improvements. Then I had some ideas on how to improve the app dramatically by adding new functionality. But in order to do that I needed to change the data structure completely. Since the change would be dramatic, I had to implement custom Core Data migration. The lightweight migration just wouldn’t do.

And because this app is just a part-time hobby for me, I allowed myself to procrastinate for months on this particular update. And one of the reasons for procrastination was a lack of good tutorials on custom Core Data migrations on the internet. There are some examples but they tend to talk about the details without giving you the overall picture of the migration process.

Here are some questions that had been bothering me before I learned custom Core Data migrations.

If I implement NSEntityMigrationPolicy classes do I have to do anything else, or will the migration occur automatically from then on? Answer: Nothing will happen automatically, you start the migration manually.

Do I have to implement all attribute mappings for all entity mappings? Answer: No, I didn’t have to implement any of those.

I use MagicalRecord, how is it going to affect the process of implementing custom migration. Answer: you don’t start using MagicalRecord until after the migration is completed

Core Data Model Versioning and Data Migration Programming Guide is talking about Core Data migrations in general. There is a section about customising the migration but it’s very short and leaves you with a lot of questions unanswered. I had to use information from several different sources on the internet to finally understand custom migrations.

Overall, the process of figuring out how to implement custom Core Data migration felt like collecting a jigsaw puzzle without knowing beforehand what the end picture would look like. So, what I intend to do with this post is to give you a bird’s eye view on the work that is needed to be done in order to implement custom Core Data migration.

I’ll use my app as an example, but this post is not so much about the little details and code snippets as it is about the overall strategy of implementing custom migrations. You can find all the code snippets you need on Stack Overflow. And I don’t talk much about theory either. This is not an in-depth guide to Core Data migrations. Hopefully, you’ve already read some theory, but maybe still need an example of how it’s done from start to finish. You will find such an example here.

As for me personally, when I was learning all this stuff, what I found myself lacking, is an overview or some king of a big picture. I hope this post will give you this big picture.

This post focuses only on custom Core Data migrations. So, before reading further, make sure you have general understanding of Core Data and lightweight migrations.

The original project

First let’s take a look at my app. We’ll see what its data structure was before and after the migration. The original app had a list of questions which you were supposed to answer on a daily basis.

This is what a question list looked like

And these are the entries in the diary

I told you that I’d made some funny decisions about the data structure of the app. Here’s what it looked like.

For the list of questions I had a class called Questions. There was only one object of that class. It had one attribute called questions of type Binary data. An archived array of NSStrings was stored there.

For daily records I had a class called Records (for some reason the name of the class was plural). It had three attributes:

• date – I stored dates as strings that looked like this 2017-12-30. The reason I didn’t use NSDates was that at the time I wasn’t comfortable with timezones. I needed dates to always be the same after the entry was created. What if I used NSDates and a user changed the timezone? Some dates could change as a result. I figured it would be easier to store the date as a string and don’t have that problem. In the new version I fixed this, and you’ll see how later.
• questions – an archived array of strings with question names;
• answers – an archived array of strings with answers to the questions. I made sure in code that questions and answers arrays always had the same number of items. Question of index i had a corresponding answer in answers array, also at index i. If there was a question which had no answer, I stored [NSNull null] in the questions array in place of an answer to that question. I know, very strange decisions, but that was my first experience in app development.

Data structure after the update

The main reason for the update was to have multiple lists of questions. So, instead of one list of questions I now have several lists of question. Each list has a name and a bunch of questions. And each day record in the diary can now contain multiple entries, like this:

The lists of questions are now called templates. You can see them in a list on the Templates screen.

When you edit each template you can see that now there are 3 types of questions: text, number and Yes/No. These are really the types of answers a question can have. To better understand all this, I suggest you download the app and explore “My diary” and “Templates” sections in its menu.

The new data model looks like this:

There are Templates now. Each Template has a name and a list of TemplateQuestions. Each DayRecord has a list of SectionRecords, and each SectionRecord has a list of questions.

When to implement the migration in your development process

Now that you’ve seen ‘before’ and ‘after’ versions of the app let’s do the migration. But first let’s talk a bit about when to do the migration. At first I had this crazy idea of releasing the version of the app that would look and feel just like the old version but would have the new data model. I thought I’d do the migration, see that everything’s OK and then start developing the new UI in the next version of the app (the migration part was kind of mysterious for me at that time, and I just wanted to figure it out first and then continue with the rest of the development process). I started out with this approach but soon realised that it just wasn’t right. First of all, there is no reason to release an update without any new functionality. Second, while developing the new UI there may arise a need for changes in the data model, which it did, by the way.

So, if you have some other stuff to do, like changing the UI and adding new functionality just delete the old store by removing and reinstalling your debug app and pretend for a while that there’s been no previous version of the app. Develop and test the new version of your app. And then, when you are sure that the new data model won’t change, start implementing the migration.

Preparation for migration

The migration development process starts just like it would start for a lightweight migration. You add a model version. Select your data model file and then in Editor select Add Model Version.

You will see something like this

Now select one of these two versions and in File Inspector set new version as current version. 

Now your data model files should look like this (the green dot is on the second version):

And that’s where the similarities between custom and lightweight migrations end. After you created new data model you can forget about data migration for a while and develop your app as I suggested earlier. 

Actually implementing custom Core Data migration

When you are finally ready for data migration, the first thing you need is a mapping model. You create it just like you create any other file in Xcode, by clicking Command+N.

When you create it you specify the source store, destination store and a name. In my case source and destination store names were “Simple Q&A Diary” and “Simple Q&A Diary 2”. I named my mapping model SQADMappingModel1_2.xcmappingmodel. Once you have a mapping model you don’t have to configure anything in it right away. Instead you can leave it alone for a while, go to the AppDelegate and work on the migration code. Later you will return to the mapping model and customise it. You start with creating a mapping model because you will need to refer to it in your data migration code.

The intermediate result you are going to achieve after writing some code in AppDelegate is this: the migration will succeed without crashing the app but the data will be lost in the process (because you haven’t set up the mappings yet).

Should you migrate at all?

Typically you start the migration at startup. The best place for migration code is -application:didFinishLaunchingWithOptions: of AppDelegate.

But you don’t migrate data every time the app starts. You only need to do it once. Also, if the user is relatively new and has never had the older version of the app, the migration is not needed at all. You can find out if the user is new by checking the persistent store file. In the case of my app it is “Simple Q&A Diary.sqlite”. If this file does not exist at startup, it means that the user is new and this is the very first launch of the app. In this case you create a store and start using Core Data as usual. If the file existed at startup you must check if it is compatible with the new data model version. If it isn’t, it means that migration is needed. I made this ugly diagram to help you visualise the logic I just described.

Of course, using MagicalRecord is optional. And here is what this logic looks like in code (this piece of code is located in application:didFinishLaunchingWithOptions: method):

We’ll look into each of these methods. But first I want to tell you that the code I show you is the actual code from my app. I didn’t adapt it in any way for this post. So there may be some lines that are not relevant to the topic we are discussing here. For example, you can see this line

It just creates the initial list of questions, so the users can start adding their entries right away even before they created their own questions. You can see also, that I create initial data not only for the new users but also after the migration in some cases:

The reason is that in the old app the initial data is hard-coded, and is not saved in Core Data until the user changes it. So it is possible that some users will have no initial data after the migration.

Now lets look at the methods. Let’s start with helper methods and then look at the method responsible for the migration itself. Helper methods:

Here you can see that inside these methods I use other helper methods:

Method –[SQADTracker logEventWithName:] is used for sending events into Firebase analytics. 

What migrates where

Before we look into migrateCoreData method there is one thing you need to understand about the migration: the data migrates from one store file into another store file. Prior to figuring out custom Core Data migrations I only had experience with lightweight migrations and I never needed to worry about store files. So, naturally I thought that the data inside my “Simple Q&A Diary.sqlite” file will just be transformed so that it will be compatible with the new data model.

Turns out this is not the case. To migrate data you need another, temporary store file. The data will be migrated from the old store to the temporary store. After the migration completes you will delete the old store and put the temporary store in its place. After everything is done you will end up with the store file which has the same name as the old file, but the data in it will be compatible with the new version of data model. Then you start to use Core Data as usual (for example you can initialise MagicalRecord at this point).

Note that on step 2 on the image you don’t actually create a temporary file yourself. Instead, this file is created on step 3 during the migration. You only need to specify it’s name.

Now let’s look into the migrateCoreData method:

Note how the mapping model is referenced here (with “cdm” extension). You can also see that the temporary store file name is specified (“SQAD_NEW.sqlite”). I could have chosen any other name for this file. After the migration succeeds the store replacement method is called. Here is the implementation of this method:

I leave all the tracking code (-[SQADTrackerlogEventWithName:] method calls) in these code snippets to give you an idea about the things you might want to track in your own app.

Intermediate testing

Now we can test this code by installing the previous version of the app, adding some data into it and then installing the new version on top of the old version. In the logs you should see that migration was successful. But there will be no data in  the store. Well, in the case of my app there will be initial data created after the migration, but the data that was entered in the previous version of the app will be lost.

Configure the mappings

What you need now is mappings. Go back to the mapping model that you created earlier. In my case it’s a file called SQADMappingModel1_2.xcmappingmodel. Inside it you will see a list of mappings. The names of the mappings will be the same as the names of the entities in the new version of your data model. In my case they looked like this:

Now let’s think a little bit about what we are trying to accomplish. You want to turn thisinto this

In the new data model there is a bunch of entities connected to each other through relationships. It’s easy to get confused about what should map to what. This is how you should think about it. If you look at new data model, you will see that entities connected to each other through a relationships can be viewed as hierarchies with parent entities and child entities. What you should be interested in is the topmost parent entities. They are Template and DayRecord.

So, when you configure the mappings in your mapping model you will specify that Questions entity (old data model) maps to Template entity (new data model)

and Records (old data model) maps to DayRecord (new data model) 

Reminder: I called Records entity so by mistake. I should have named it Record (without s). So think of it like this: each Record turns into a DayRecord.

What about the other entities (TemplateQuestion, SectionRecord, Question)? The child entities will be created programmatically, and I’ll show you how later.

In my project I selected Template mapping and set Questions as it’s source. Then I selected DayRecord mapping and set Records as it’s source. Then I created a subclass of NSEntityMigrationPolicy for each of the two mappings and specified the names of those classes in the mapping model. So, for Template I ended up with these settingsAnd these are the settings for DayRecord

Note that when you specify a source for the mapping, the mapping’s name changes. In my case Template turned into QuestionsToTemplate and DayRecord turned into RecordsToDayRecord. In user info you can see that I added a modelVersion key with a value of 2. This will be used later in the code. Also note that of five mappings I used only two.

I guess I could have deleted the other three, but I just kept them in the mapping model because they were harmless. And here is what relevant files look like in Project navigator

Implementing mapping policies in code

If you now test the migration you will see that nothing has changed: the migration succeeds but the data is lost. What you need now, is to implement createDestinationInstancesForSourceInstance: entityMapping: manager: error: method in each of the entity migration policy subclasses. 

This is how I did that in SQADQuestionsToTemplatePolicy.m file:

I don’t use MagicalRecord or NSManagedObject subclasses at this point. Instead I use key value coding. Xcode complained that method addQuestionsObject: was undeclared, so I had to import Template+CoreDataClass.h. You can see that I take all the data I need from the source instance called sInstance. It’s a Questions object from the old data model. Then I create a Template object and populate it with data, create a bunch of TemplateQuestion objects and connect them to the Template object. Note how I check model version before doing anything of it. This is the key-value pair I set in my mapping model previously.

Here is the contents of SQADRecordsToDayRecordPolicy.m file:

I pretty much do the same thing here: I get the data from source instance, create destination instance, and populate it with data . As you can see, I did some funny stuff with sortField. You don’t need to understand it. It’s just something I had to do to sort SectionRecrods according to their date and their order in their DayRecord.

Precautions

What if migration was not successful? In some examples on Stack Overflow people suggested to not delete the old persistent store but to move it to another URL instead. I could have done it, but I thought about it and I couldn’t figure out how it would help me in any way if the migration failed for some users. I’d need to release an update where they could get their data from the backed up persistent store, which would be too much trouble for me.

But I still wanted to minimise the risk for users to lose all their data in case of a migration failure. I thought, if the migration failed there would be two persistent store files: the old one, and the temporary one. On the next launch there will be second attempt to migrate data. I didn’t know if the existing temporary file, possibly with corrupt data in it, would cause second and all the subsequent migration attempts to fail. So, I decided to just delete the temporary file (if it existed) at every launch. This is the method I used for it:

I called it in -application:didFinishLaunchingWithOptions: of AppDelegate before calling all the migration methods I showed you in this post. There is still a chance that after deleting the old store the attempt to move new store into its place will fail. But this never happened according to my analytics. Maybe you can suggest a better strategy for dealing with possible migration failures.

And this is the end. I hope this tutorial was helpful.

[SW]