Sepasoft MES Enterprise Synchronization Protocol

Knowledge Base Article: The Sepasoft MES Enterprise Synchronization Protocol

This article details the technical architecture, features, and use cases of the Sepasoft MES Enterprise Synchronization Protocol. The protocol is a foundational component of the MES enterprise solution, specifically engineered for resilience, self-healing, and scalability in large, multi-site industrial environments. It provides a robust framework for ensuring data consistency and operational continuity across a distributed network of manufacturing gateways.

Core Purpose and Design Goals

The protocol's design is driven by four primary objectives, each addressing critical challenges identified in large-scale, multi-site customer deployments:

Resilience and Self-Healing To strengthen the system's ability to recover from common network issues, such as temporary disconnections or data corruption, without requiring significant manual intervention or repair.
Performance To maintain or improve performance during synchronization, ensuring that the background process of data transfer does not negatively impact active production environments.
Enhanced Monitoring and Troubleshooting To provide new tools and monitoring points that give users and technical support staff clear visibility into the synchronization process, making it easier to diagnose and resolve issues.
Configuration Guidance To embed guidance within the software's configuration interface to help eliminate common errors when adding new gateways to the enterprise network, simplifying setup and ensuring network integrity.

Protocol Terminology

The official name for this system is the "synchronization protocol." This term was chosen as a more specific and accurate descriptor than "network protocol," as it is defined as an "application-level protocol" that governs how MES data is exchanged between gateways, rather than a protocol that defines network connectivity itself.

This architecture enables the system to achieve its core design goals through a carefully structured hierarchy, robust version management, and an intelligent, changelog-based synchronization engine.

Protocol Architecture and Core Concepts

A clear understanding of the protocol's architecture is crucial for effective implementation and troubleshooting. This section covers the network hierarchy, version management policies, and the fundamental data synchronization mechanism that forms the backbone of the enterprise system.

Enterprise Network Hierarchy

An MES Enterprise network is organized in a hierarchical structure with a single Enterprise head at the top and multiple downstream gateways (GWs) connected below it. This structure allows for logical organization of a large manufacturing operation. The protocol supports several types of downstream gateways, enabling flexible configurations:

Enterprise
Site
Area
Line

The system is designed for significant scale, with a requirement to support a minimum of 120 downstream gateways. Known customer configurations have directly informed the protocol's scalability and performance targets, including deployments with 80 site gateways and no area or line gateways, as well as deployments with 200 lines, each containing 4 to 20 cells.

Version Management and Compatibility

In a large, distributed network where it is not feasible to upgrade all gateways simultaneously, the protocol incorporates specific rules for version management to ensure continuous operation during transitional periods.

Enterprise Head Supremacy: The most critical rule is that the enterprise head gateway must always be running the highest version of the software in the entire network. Upgrades should always begin at the top of the hierarchy.
Backwards Compatibility: The protocol is designed to be backwards compatible with the single previous version. This is achieved through the use of "version adapters," which allow newer gateways to communicate effectively with older ones during a rolling upgrade process.
New Feature Impact: When minor version changes (e.g., the third digit in the version number) introduce new features, the protocol handles this gracefully. New properties are typically saved as "custom properties," which prevents synchronization from breaking when communicating with older gateways that do not recognize these new data fields.

The Changelog-Based Synchronization Engine

At the heart of the protocol is a redesigned backend synchronization process, architected to solve the problems of the previous system, which customers found to be "brittle and requiring significant repair when they malfunction." This new architecture moves away from a system based on individual object save triggers to one that sends batches of transactions based on a central changelog. This fundamental shift enables comprehensive verification, the ability to resend missed data, and greater overall accuracy.

Transactional Integrity: Changes are sent in batches, and the entire batch is treated as a single transaction. This means the batch is either successfully synchronized in its entirety or not at all, which prevents partial updates and data inconsistencies.
Per-Gateway Tracking: The receiving system uses a unique marker to track the last successfully synced changelog item for each specific sending gateway. If no marker exists, which is the case for a new or replaced system, the protocol automatically sends all sync-able data to bring the gateway fully up to date.
Data Prioritization: In a recovery scenario (e.g., reconnecting a gateway after an extended outage), the system intelligently prioritizes data transfer. Object configuration data is sent first, followed by historical data. This approach allows production operations at the site to restart as quickly as possible, with historical records backfilled afterward.

This core architecture provides the foundation for the specific synchronization functions and processes that users can configure and manage.

Key Synchronization Functions and Processes

This section provides a detailed look at the specific processes and functions that constitute the synchronization protocol, from routine data transfer and on-demand triggers to specialized mechanisms for configuration management.

On-Demand Synchronization

The protocol includes the triggerValueSync() function, which allows for immediate, manual initiation of the synchronization process. This function overrides the standard timed interval, providing a way to force a data exchange when necessary. It is designed to start the sync process only if one is not already in progress. It takes no arguments and returns a string indicating whether the process was started successfully.

Tag Collector Synchronization

This configuration was refined to address customer reports of unreliable data transfer, which led to a critical business problem: "Reports and Analysis at the enterprise level are inaccurate when sync doesn’t happen." To improve the reliability of historical data collection, the "Send Values at this Rate" parameter for the Tag Collector Sync process has been updated with new, more robust defaults and limits.

Setting	Value
Unit of Measure	Seconds
Default Value	300 seconds
Minimum Value	60 seconds
Maximum Value	3600 seconds

Enterprise Import & Export

The Import/Export feature serves as a critical mechanism for transferring object configurations between systems, such as migrating changes from a development or staging environment to a live production system. To ensure this process is reliable, especially with large and complex configurations, several key requirements have been implemented:

Automatic chunking of large exports to prevent the process from exceeding server memory limits.
Intelligent handling of version differences during import. If the target server has a newer property value, it will not be overwritten by an older value from the import file.
User-facing options to overwrite, abort, or ignore (skip) duplicate objects and properties during the import process, giving administrators fine-grained control over how data is merged.

These functions provide the operational "how" of synchronization, which are complemented by a suite of tools for configuration and monitoring.

Configuration, Monitoring, and Scripting

Robust management tools are essential for maintaining a healthy and performant enterprise network. The protocol includes configurable parameters, dedicated monitoring interfaces, and scripting functions that give administrators and engineers the tools they need to manage and troubleshoot the system.

Gateway Synchronization Settings

Administrators can fine-tune the synchronization behavior for each gateway pair through several key settings.

Parameter	Description	Values / Unit
Synchronization Interval	The fixed rate at which the sync process initiates.	1-60 minutes (Default: 10)
Synchronization Size	The number of changelog rows sent in each transaction.	Integer (Changelog rows)
Maximum Artifact Size	The size limit for artifacts included in an import/export.	Integer (Megabytes, Default: 10 MB)

Monitoring the Enterprise Network

The MES Monitoring component provides critical visibility into the health and status of the entire enterprise network. These tools are essential for proactive maintenance and rapid troubleshooting.

Network Diagram: A hierarchical diagram provides a clear visual representation of all gateways and their parent-child relationships within the enterprise.
Connection Status: The lines connecting gateways on the diagram provide at-a-glance status indicators (e.g., Red-dashed for failure, Gray-solid for success) based on the outcome of the last sync attempt.
Gateway Metrics: Key performance indicators are displayed for each gateway pair, including "Successful Sync Delta" (time since last successful sync), the "Next sync time," and the number of unsynced changelog entries.
Unsynced Item List: For a selected gateway pair, a detailed table can be generated that lists all specific items that are currently out of sync.

Scripting Interface for Sync Status

For programmatic access and automated checks, the protocol exposes the system.mes.getObjectSyncStatus() script function. Its primary use case is to allow engineers to programmatically check the synchronization status of a specific MES object between two gateways. The function takes the object's MESObjectLink and the target GW Equipment Path as parameters. It returns an object detailing any differences found; if the objects are identical, the function returns a null or blank value.

These management tools provide the necessary control and insight to effectively operate the protocol in real-world scenarios.

Practical Use Cases and Scenarios

The true value of the synchronization protocol is demonstrated in its application to common operational scenarios. This section walks through several key use cases that are derived directly from customer requirements and real-world operational needs.

Scenario: Adding a New Gateway to the Enterprise

The process for adding a new downstream server to an existing network incorporates built-in safeguards to prevent configuration errors. Synchronization will not begin until the connection has been explicitly accepted by a user on both the parent and the child gateways. To prevent accidental or unauthorized changes to the network topology, the user interface presents a confirmation dialog that requires the administrator to type PROCEED before the joining process can be finalized.

Scenario: Upgrading an Enterprise Network

The protocol is designed to accommodate the reality of large-scale enterprise upgrades. The recommended procedure is to start the upgrade at the top of the hierarchy (the Enterprise head) and proceed downward through the network. Because large customers cannot update all gateways simultaneously, the protocol is explicitly designed to handle mixed-version environments during this transition period, ensuring that operations can continue without interruption.

Scenario: Disaster Recovery and Disconnection

The protocol's resilience is most evident in disaster recovery situations. If a server is replaced or an extended network disconnection is resolved, the system automatically detects the state of the reconnected gateway. If no valid synchronization marker is found, the system initiates a full data transfer, sending the complete set of required data in manageable batches. To restore operations as quickly as possible, it prioritizes sending object configuration data first, allowing the site to resume production while historical data is backfilled in the background.

Ultimately, the Sepasoft MES Enterprise Synchronization Protocol provides the foundation for a robust, manageable, and scalable enterprise-wide manufacturing execution system.