Migration from v1 to v4

[derwaldgeist]

We're currently migrating our Meteor app from Meteor 2 to Meteor 3.

It's been around a while, so it's still using alanning:roles 1.3.0. I saw that you have removed the migration scripts in v4. This is quite a challenge for us, since we would like to do the update of our live system automatically, using quave:migrations.

Is there any opportunity to get these scripts back in Roles v4, or would we really have to update all our live systems gradually and manually, i.e. first from v1 to v3 and then from v3 to v4?

[github-actions]

Thank you for submitting this issue!

We, the Members of Meteor Community Packages take every issue seriously.
Our goal is to provide long-term lifecycles for packages and keep up
with the newest changes in Meteor and the overall NodeJs/JavaScript ecosystem.

However, we contribute to these packages mostly in our free time.
Therefore, we can't guarantee your issues to be solved within certain time.

If you think this issue is trivial to solve, don't hesitate to submit
a pull request, too! We will accompany you in the process with reviews and hints
on how to get development set up.

Please also consider sponsoring the maintainers of the package.
If you don't know who is currently maintaining this package, just leave a comment
and we'll let you know

[jankapunkt]

@derwaldgeist good point. @StorytellerCZ was there a reason to drop the migration scripts? Do you think we should add them back with async compat? Alternatively, could we provide a documentation on how to run them manually using meteor shell?

[derwaldgeist]

Thanks for the feedback. It would be really nice if they were back in the v4 code. The current situation forces me to first update the live systems to v3 and then to v4. I prepared this two-step approach, but I would love to do this in combination, using just a quave:migration update script. For the migration from v1 to v3, this worked perfectly.

[StorytellerCZ]

I would be only for adding them back in case that we say that the current v4 is only for migration purposes and we archive things afterwards.
There were issues with those scripts due to Meteor 3 migration and there were 2 other assumptions: 1. people will migrate one major version at a time to make sure everything works properly, 2. at the time of v4 release everyone who already wanted has already migrated to v3.
As was mentioned in #352 there are also some other issues with missing migration documentation.

[evolross]

I am in the same boat - migrating from Meteor 2.16 using alanning:roles@1.x to Meteor 3.3 using roles@1.0.1 (v4). I gave up on trying to migrate using the tools in the various packages, couldn't get any of them to work. I just wrote my own migration using roles@1.0.1:

convertLegacyRolesToNewRoles: async function(role, insertRole = false) {
    check(role, String);
    check(insertRoles, Boolean);

    if(insertRole) {
      //  Create role - the new version of the roles package for Meteor 3 requires roles be created first
      await Roles.createRoleAsync(role);
    }

    //  Create query for particular role
    const usersQuery = {
      'roles.__global_roles__': { $in: [role] }
    };

    //  Get all users in role
    const users = await Meteor.users.find(usersQuery, {
      fields: {
        'roles': 1,
      },
      //  Return the newest created users first
      sort: { createdAt: -1 }
    }).fetchAsync();

    //  Iterate over each user
    for (const user of users) {
      const userId = user._id;
      const userRoles = user.roles.__global_roles__ || [];
      //  Iterate over each role
      for (const role of userRoles) {
        //  Add each of user's role using new role package
        await Roles.addUsersToRolesAsync(userId, role);
      }
    }
  }
});

[derwaldgeist]

We eventually did this by migrating our prod system to v3 first - and then to v4.

[evolross]

I couldn't get the built-in migration code to run, and also found the source code hard to understand. I let the above migration code I wrote run overnight last night and at 12+ hours it wasn't even halfway through our 200,000 production users. This length of time isn't ideal for a migration switch-flip to Meteor 3.

Did a slight update this morning to use rawCollection().initializeUnorderedBulkOp() and now it runs in seconds. Just need to make sure the role-assignment collection exists and is cleared since it's a raw operation and won't have the upsert effect of using Roles.addUsersToRolesAsync(userId, role).

convertLegacyRolesToNewRoles: async function(role, insertRole = false) {
    check(role, String);
    check(insertRoles, Boolean);

    if(insertRole) {
      //  Create role - the new version of the roles package for Meteor 3 requires roles be created first
      await Roles.createRoleAsync(role);
    }

    //  Create query for particular role
    const usersQuery = {
      'roles.__global_roles__': { $in: [role] }
    };

    //  NEW: Create a new bulk role assignment insert object
    const bulkRoleAssignmentInsert = Meteor.roleAssignment.rawCollection().initializeUnorderedBulkOp();

    //  Get all users in role
    const users = await Meteor.users.find(usersQuery, {
      fields: {
        'roles': 1,
      },
      //  Return the newest created users first
      sort: { createdAt: -1 }
    }).fetchAsync();

    //  Iterate over each user
    for (const user of users) {
      const userId = user._id;
      const userRoles = user.roles.__global_roles__ || [];
      //  Iterate over each role
      for (const role of userRoles) {
        //  NEW: Create the role schema manually to insert with bulk insert
        const roleAssignment = {
          "_id" : Random.id(),
          "user" : {
            "_id" : userId
          },
          "role" : {
            "_id" : role
          },
          "scope" : null,
          "inheritedRoles" : [
            {
              "_id" : role
            }
          ]
        };

        //  OLD: Add each of user's role using new role package
        //await Roles.addUsersToRolesAsync(userId, role);

        //  NEW: Add the new role assignment to the bulk role assignment inserter
        bulkRoleAssignmentInsert.insert(roleAssignment);
      }
    }
  }

  try {
    await bulkRoleAssignmentInsert.execute();
  } catch(error) {
    console.error(`SERVER ERROR: convertLegacyRolesToNewRoles: Bulk inserting role assignments. ${error.message}`);
  }
});

[derwaldgeist]

Yes, using bulk ops are essential for large collections. 200k users is impressive, congrats!