School Data Sync Overview

School Data Sync

School Data Sync (SDS) is a free service for Education that helps to automate the process of synchronizing user and roster data from Student Information or Management Systems with Microsoft 365. SDS is a tool that helps you manage your educational organization, users, classes, and roles. SDS syncs your data with Microsoft Entra ID and Microsoft 365, so you can use Microsoft Teams, Intune for Education, Exchange Online, SharePoint Online, OneNote Class notebooks, and enable third-party apps with Single Sign On integration. SDS helps make learning more engaging and efficient.

Additionally, SDS synchronizes data to enrich Insights & Analytics.

SDS prerequisites

  • A Microsoft 365 Education tenant

  • Global Administrator permissions

SDS core concepts

Graphic that shows SDS onboarding overview.

School Data Sync models the best practices of the Education Customer Success Team that has helped institutions successfully set up and use School Data Sync to empower Microsoft 365 experiences globally. We believe that applying these best practices, to improve the IT administrator’s experience for onboarding and monitoring, along with better understanding of the health of their data, provides more time for administrators to help educators focus on innovations to improve learning outcomes.

The onboarding experience is broken down into connect data and managing data to enable scenarios to improve learning outcomes.

Screenshot showing SDS connect and manage data overview.

Connect data

Screenshot representation of connect data.

First, you need to connect to your institution’s data. You are defining how we are bringing in or connecting to your Student Information System’s (SMS) / Student Management System’s (SMS) data. Here you define your source system, the format of the incoming data, your user identity matching rules, and your active academic year. By default, your sync activates twice (2x) daily, called a run.

When the run starts, it connects to the defined source and perform basic validation. Basic validation ensures that the connection is correct for OneRoster API as a source, or the filenames and headers are correct for SDS CSV as a source.

Next, the system transforms the data for import in preparation for advanced validation. As part of the data transformation, it's associated with the configured academic year.

  • If the source system/data doesn't provide the users' organization role entry date and exit date, the system won't infer the entry date or exit date based on the presence or nonpresence of the record.

  • If the source system/data doesn't provide the users enrollment entry date and exit date, the system won't infer the entry date or exit data based on the presence or nonpresence of the record.

  • If the source system/data doesn't provide a valid organization/enrollment/ or contact role for a user, the system won't add the user record to the system.

  • Associate data to an academic year and identifying the records' active status.

    • Academic year is used to associate incoming data to help build historical data, year over year, as time progresses.
    • The approach is aligned to an academic year versus calendar year.
      • If the academic year spans across a calendar year, for example starts August 15, 2023 and ends June 15, 2024, the academic year value to be selected is the ending year, 2024.
  • A user's association with a session (academic year) is based on its role that is tied to an organization.

  • A user's association with a class is based on its role tied to an enrollment, which also includes a link to a session.

Additionally, the system will also store an updated copy of your Microsoft Entra ID into the SDS Data Lake. The copy of Microsoft Entra assists with user matching between your SIS/ SMS and Microsoft Entra user object. At this stage, the match link isn't written to Microsoft Entra user object.

In the instance that a user has multiple roles, the following rules are used to determine what staff or student match rules should be used between the user record and the Microsoft Entra user object.

Additionally, the system will also store an updated copy of your Microsoft Entra ID into the Education data lake. The copy of Microsoft Entra assists with user matching between your SIS / SMS and Microsoft Entra user object. At this stage, the match link isn't written to Microsoft Entra user object.

  • If isPrimary is set for all student roles, even if association to a staff role exists, the match is made based on the student role.
  • If isPrimary is set for any staff role, even if association to a student role exists, the match is made based on the staff role.
  • If isPrimary is set for both staff and student role, the match is made based on the staff role.
  • If isPrimary isn't set for any roles, especially with a mix for both staff and student roles, the match is based on the staff role.

Validate, store and data health

Screenshot showing representation of data processing and validation.

Next, your SDS performs advanced validation to determine the health of your data. Validation focuses on identifying errors and/or warnings. Validation follows the concept of bringing in good data and keeping out bad data. Errors mean that a record didn't pass validation based on required data and was removed from further processing. Warnings mean that the value on an optional field on a record didn't pass so the value was removed from the record, but the record was included for further processing. Errors and warnings are used to help you better understand the health of your data.

For the good data that passed validation, the process uses the configured academic year information to determine its association for longitudinal storage. When the data is stored in our internal representation in your SDS Data Lake it identifies when it is first seen by SDS, and for data linked with users' org, role association and class association, it's also identified as active in session for the current academic year. In future runs, for the same inbound flow, source, and the configured academic year, SDS identifies if the record is still seen, and based on the presence or absence of record, and keep the record active or mark it as no longer active for the current Academic Year. This is how SDS determines the historical and longitudinal nature of the data between days and Academic Years.

  • The data reflects when a new record is presented for the first time. FirstSeenDateTime is the time the record was first seen by SDS and the creation data of that row in the table. This does NOT mean it's the creation date of the record in the SIS / SMS as SDS might have run well after the data was added to the external system.

    • SDS sets the first seen date (time) and last modified date (time) to now, and if appropriate, it marks record as "is active in session" to be true.
  • The data reflects when the same record is present in the subsequent run. LastSeenDateTime is the moment SDS last saw the data in a sync.

    • SDS preserves the first seen date (time) value, set the last modified date (time) to current, and leave "is active in session" to be true.
  • The data reflects when the same record isn't present in a subsequent run. This isn't the deletion date of the record from the SIS / SMS, but the date (time) of the last time SDS saw the record during a sync.

    • SDS preserves the first seen date (time) and the last modified date (time) values and if appropriate, mark the record as "IsActiveInSession" to false.
  • The following are General Data Handling rules for data in the EDU Data Lake when data is missing on subsequent run, same source, and same academic year.

    • Orgs: If missing on subsequent run, system does nothing and record persists.
    • Users: If missing on subsequent run, system sets IsActiveInSession to False on the person organization role and enrollment.
    • Roles: If missing on subsequent run, system sets IsActiveInSession to False on the person organization role.
    • Academic sessions: If missing on subsequent run, do nothing and record persists.
    • Classes: If missing on subsequent run, system sets IsActiveInSession to False for section, section session, and enrollment.
    • Enrollments: If missing on subsequent run, system sets IsActiveInSession to False for enrollment.
    • Courses: If missing on subsequent run, system sets IsActiveInSession to "False" for course.
    • Demographics: If missing on subsequent run, system does nothing and record persists.
    • User flags: If missing on subsequent run, system does nothing and record persists.
    • Relationships: If missing on subsequent run, system does nothing and record persists.
  • There will be rolling updates for "inactivated" records. For example, if a user record isn't present, the system preserves the existing first seen date (time) and last modified date (time) values. The system sets "is active in session" to false for the users' org/role and enrollment records.

    • Organization, user and session records persist over time and aren't inactivated.
    • Only the association records to organization, user and session records, as described prior, and the association records IsActiveInSession status are impacted based on General Data Handling rules.
  • Connected data that include academic session data for terms:

    • If an upload occurs for the same source, academic year and connected data, but the linked academic session term and the data that was linked with that existing academic session term isn't present in the data:
      • System won't perform the inactivate process on any linked records and will preserve the existing state.
      • System won't update the [IsActiveInSession] to False to preserve the last state for analytics needs.
    • System utilizes the academic sessions term start date and end date to determine what is the current and active academic sessions term for the active academic year.
  • When the source data stops providing a user’s section enrollment record and then provides the same user's section enrollment within the same academic year (term), the system updates the existing record.

At the end of each run, the data health is determined. If there are no errors and/or warnings, the system determines that no data actions are needed by the administrator. If there are errors and/or warnings, the system notifies that it found some issues with the data and encourage you to investigate the data health. Error and warnings statistics are produced to help provide an initial impact to data health. The administrator, when investigating data health, is able to download a log file that contains information based on the errors and warnings found. The administrator can then begin the data investigation process to correct the data in the source system.

  • Run Status
    • Running: Actively executing
    • Completed: Completed without any errors or warnings
    • Completed with Errors: Completed but errors were found
    • Completed with Warnings: Completed but only warnings were found
    • Error: Run canceled by system
  • Counts
    • Errors: Total number of errors encountered. Error means that a record didn't pass validation and the entire record was removed before processing, after validation. If a record couldn't be processed post validation, the count will also include that record as well.
    • Warnings: Total number of warnings encountered. A warning means that one or more portions of data on the record didn't pass validation. The field that had the data for the record was removed but the rest of the record and validated data remain before processing post validation.

Manage data

Screenshot showing representation of manage data.

Managing data to improve learning outcomes with SDS provides dynamic provisioning and roster updates for virtual classrooms in Microsoft 365, to assist with simplified deployment and adoption of Microsoft 365 Group-enabled apps like Teams, SharePoint, Exchange, and OneNote Class Notebooks. SDS also performs provisioning of education institutions, users and groups for IT to assist with simplified Microsoft 365 identity management, app management, and device management. SDS also provides the option to store SIS/SMS values it syncs within Microsoft Entra ID. Provisioning to Microsoft Entra ID can also make roster information available to third party applications through the Education APIs (Application Programming Interfaces) and Microsoft Graph.

You can set up your Manage data configuration immediately after defining you connect data configuration, during the active first run, or later after the first run has finished.

  • Based on data in the lake, associated to the active academic year, and is active.
  • Uses the sync end date defined during connect data configuration to also know when to stop running changes for managed data.
    • Following academic year transition and defining a new sync end date, the manage data configuration will start up again.

To help support different Microsoft 365 scenarios, while also simplifying the admin experience, admins can define what provision type configurations they would like to enable.

Users provision type allows you to automate management and mapping of your Microsoft 365 users from your SIS users. This will write forward the mapping link, based on the sourcedId value to the user’s object in Microsoft Entra ID.

  • Based on data in the lake, associated to the active academic year, and is identified as active in session.

  • Write the matched user link to the Microsoft Entra user object with default and optional attributes.

    • These default values are written for every mapped user:
      • User External Id: Data found in the sourcedId field for the user.
      • Organization External Id: Data found in the orgSourcedId field for the user’s associated organization and role.
      • Organization Role: Data found in the role field for the user and the associated organization.
    • These optional values are written if selected and there's a value provided in the data:
      • User Grade Level: Data found in the grade field for the user for the associated organization and role.
      • User Number: Data found in the user number field for the user for the associated organization and role.
  • When a user has multiple roles, rules are used to determine the value when writing role to Microsoft Entra user object.

    • If isPrimary is set for all student roles, even if association to a staff role exists, the role attribute value is made based on the student role.
    • If isPrimary is set for any staff role, even if association to a student role exists, the role attribute value is made based on the staff role.
    • If isPrimary is set for both staff and student role, the role attribute value is made based on the staff role.
    • If isPrimary isn't set for any roles, especially with a mix for both staff and student roles, the role attribute value is based on the staff role.
  • If the user is also associated with multiple organizations, an additional rule is used to determine the value when writing the role to the Microsoft Entra user object.

  • There are additional options when managing users with the role of 'Student' that can be enabled.

    • Mark all students as minors: Based on the rules to determine what user role value will be written to the Microsoft Entra user object if the role is Student. This setting identifies the user as a minor so that Microsoft and third-party applications identify them as such. The outcome sets the AgeGroup and ConsentProvidedForMinor user properties. The net result of setting the two properties is the attribute of LegalAgeGroupClassification set to MinorWithParentalConsent.
    • Student contacts associations: Based on the rules to determine what user role value will be written to the Microsoft Entra User object, if the user is a Student and if connected data from the SIS is included to bring in Contact Relationships, also known as Guardians or Parents. If this option is on when processing data, this will include the contact information with the Microsoft Entra user object for educator communication.
  • SDS also provides the ability for creating unmatched users for tenants not using Azure AD Sync, or other user management tools, to manage user creation. You can enable if you would like SDS to create new users.

    • By default, the setting is off. Do not enable this option if users are being created and managed in Microsoft Entra ID through other automated means.
    • The Microsoft Entra UserPrincipalName value, also referred to as username, is constructed from the User identity rule selections, as part of the Connect data configuration linked to the user’s SIS record.
      • This is because, after the user is created, the user identity rules are used to continue linking to the same user object in subsequent runs. This is to prevent creating a new user with one set of rules and then trying to identify them with a different set of rules.
    • For Staff and Student users that weren't matched to existing Microsoft Entra Users and are to be created, you'll define their default password.
      • Password construction follows Microsoft Entra ID Password Protection guidance and assists with scoring calculation for default value to be accepted to successfully create the user.
      • The password configuration can be modified through the edit flow experience for those tenants that wish to rotate default passwords when creating new users at later dates.
    • License selection is optional and doesn't have to be set. If selected this will set the default license when the user is created. If you're planning on using Group Based License assignment and management don't select a value.

Provision type for Class groups is for managing class groups and provides a space for users to connect with each other, communicate, and collaborate across various Microsoft 365 Apps including Teams. ​This creates a Class group for every active class found (where at a minimum, an active user that is linked with a mapped owner role is associated with the class). The admin also has the option to enable the ability to automatically create the Class team based on the corresponding class group.

  • Creates a Microsoft 365 group for each class or section.
  • Write group link to the Microsoft Entra group object with default and optional attributes.
    • These default values are written for every class group managed:
      • Class external ID: Data found in the sourcedId field of the section or class.
      • Class title: Data found in the title field of the section or class.
    • These optional values are written if selected and there's a value provided in the data:
      • Course title: Data found in the title field of the course associated to the section or class.
      • Course code: Data found in the code or course code field of the course associated to the section or class.
      • Course subject: Data found in the subject or subjectCodes field of the course associated to the section or class. As a reminder, values provided from the SIS must match the corresponding List of Values to be accepted with the connected data. If not, the value isn't ingested, therefore no data is written forward to the group object until corrected.
      • Course grade level: Data found in the grade or grades code field of the course associated to the section or class. As a reminder, values provided from the SIS must match the corresponding List of Values to be accepted with the connected data. If not, the value isn't ingested, therefore no data is written forward to the group object until corrected.
      • Course external ID: Data found in the courseSourcedId field of the course associated to the section or class.
      • Academic session external ID: Data found in the sessionSourcedId code field of the academic session associated to the section or class. If academic sessions aren't being included from the SIS/SMS, and this option is selected, the active academic year externalId value is used.
      • Academic session title: Data found in the title field of the academic session associated to the section/class. If academic sessions aren't being included from the SIS/SMS, and this option is selected, the active academic year name value is used.
  • Define owners and members based on class or section enrollment and their enrollment role.
    • A user's association with the Class Group is based on the enrollment role associated with sections or classes.
    • Selecting a role means that for any user with that role for the associated group, the user is included to that section or class.
    • User privileges for a class group are based on the associated enrollment role and if that role is selected in the Owners or Members list.
      • For any roles that aren't selected, they won't be linked or have access to the corresponding group, section, or class--even if there are users that have enrollment role records for the corresponding group.
    • Owners have editing privileges to manage their groups, change group display names, and add or remove group members.
      • The owner will also be seen and listed as a member of the group, which enables them to see the section or class in Microsoft applications.
    • Members that are not Owners, have read-only permissions for their groups. They only gain access to the Class team when an Owner activates the class.
  • There are additional options when managing class groups:
    • Automated Teams creation: Selecting this option creates a Microsoft Group and Class team for each class or section synced. If the option is not selected, only a Microsoft 365 Group is created.
      • If the toggle is on:
        • For Educators, or group owners that are using the created Class Teams from SDS, they have early access before the students and other group members. When the educator is ready, they can select Activate to allow students and other group members access.
      • If the toggle is off:
        • For Educators, or group owners, where SDS isn't creating Class Teams, they can create Class Teams based on the SDS Class groups manually withing teams. When the educator is ready, they can select Activate to allow students and other group members access.
    • Class group display name: Selecting this option will only write the class group display name during initial creation. This option will allow administrators and the group owner to update the group and class team display name and preserve the change from updates in future runs.

Security groups provision type is for managing Security groups and provides a grouping construct for use within various identity management, app management, and device management scenarios in Microsoft 365.​

  • Role Groups: Creates Security Groups by User role groups. Example: One for all Students, another for all Staff.

    • Creates all Students Security Group and manages users with roles in the Students role group.
    • Creates all Staff Security Group and manages users with roles in the Staff role group.
  • Organizations, Organizations + Role Groups: Creates Security Groups by Organization + Role Groups combinations. Example: Contoso School containing Students - Contoso School and Staff - Contoso School.

    • Creates a Security Group for each Organization present with active users.
    • Nests corresponding Role group + Organization Security Groups to the Organization Security Group.
      • Creates a Security Group for each combination of users with roles in the Students role group + Organization present.
      • Creates a Security Group for each combination of users with roles in the Staff role group + Organization present.
  • SDS Security Groups can be used in various administrative functions within Microsoft Entra ID and Microsoft 365. Here are some of the most common uses of the SDS Security Groups:

Administrative units provision type is for managing Administrative units and provides a grouping construct for delegated IT administration and scoped role assignments. Scoped role assignments allow admins to manage subsets of the broader Microsoft 365 directory.

  • Organizations: Creates Administrative Units by Organization and link organization users, organization class groups and organization security groups. The selection provides Organizations with the type ‘school’ to be available in the EDU Graph Schools endpoint. Example: Contoso School containing Contoso School users + class groups + security groups.
    • Creates an Administrative Unit for each Organization present with active users.
    • Links all Students with an active association to the Organization with roles in the Students role group.
    • Links all Staff with an active association to the Organization with roles in the Staff role group.
    • If also provisioning Class groups, links all Classes with an active association to the Organization.
  • Organizations + Role Groups: Creates Administrative Units by Organization + Role Group combination. SDS allows the ability for permitted staff to perform delegated IT administration for students of school administrative unit. Example: Students - Contoso School, Staff - Contoso Schools.
    • Creates Students - Organization Administrative Unit for each Organization present with active users for each combination for users with roles in the Students role group.
    • Creates Staff - Organization Administrative Unit for each Organization present with active users for each combination for users with roles in the Staff role group.

Insights & Analytics

School communities can channel their efforts and achieve a more significant impact when educators know which strategies work for their students. With the timely data provided by Insights & Analytics, educators and education leaders can focus time and energy on taking actions that improve the learning environment, drive student success, and help learners thrive.

Screenshot showing representation of Education Data Lake Export.

  • Education Data Lake Export
    • Allows you to get a copy of the Education Data Lake with Activity data for your own custom analytics. With SIS/SMS and Education Insights data in your Azure Data Lake, you can then combine it with other data sources such LMS, or assessment data.
    • An Azure subscription is required for the tenant. The preferred subscription is used by default.
    • Data that is included in the export:
      • Roster data, including longitudinal nature of the data
      • User activity data, based on user signals
      • Copy of Microsoft Entra User and Group objects, including the mapping between Roster Users to Microsoft Entra User Objects. The link also provides the user association to Group Objects and Activity signal data.
    • Export uses Microsoft Azure Data Share service to copy from your Microsoft subscription boundary by generating a daily snapshot of Education Data Lake data and copy to your Azure subscription and Data Lake.
    • Organizations can then use Microsoft Fabric, Microsoft Synapse, Power BI, Azure Machine Learning, and / or Artificial Intelligence or AI to create your own customized analytics and reports.

Relevant articles

Planning Checklist

User Identity Rules

Data Ingestion with SDS v2.1 CSV

Data Ingestion with OneRoster API

Health and Monitoring

Validation Rules and Descriptions

Manage Microsoft 365 Users

Manage Microsoft 365 Groups and Teams

Enable Sync to Insights for Education Leaders

Enable Education Data Lake Export

Frequently Asked Questions