Sepasoft MES Module Suite
MES Enterprise Challenges and Solutions
Today, when implementing MES for a single server, it is straightforward. This is because all of the configuration and historical data is stored in one database. The single server can provide MES functionality for a single production machine, area, site or even multiple sites.
However, when implementing MES across multiple servers it becomes very complex and leads to increased implementation cost, reliability problems, IT unfriendly, and a poor end user experience. Detailed below are the primary challenges when implementing MES solutions currently available today.
Also learn about sharing Sepasoft MES production data (PDF):
Challenges
Synchronization
When implementing MES for multiple servers, configuration and historical data is stored on multiple systems and in multiple databases. Because of this, keeping data synchronized requires using database replication or a custom approach of passing configuration, scheduling, production status, etc. between systems typically implemented using scripting.
Database replication tends to be reliable, but when using other methods for synchronization, it is common that new data or updates to existing data is missed and causes inaccuracies in the dashboard or reports data. The same can happen for the production configuration data being sent down to a site, area or line.
WAN Performance
True MES implementations involve two-way communication with the OT layer. For example, preventing a lot from being used in the next production step until a SPC sample has been taken and has passed. This requires interlocks with production machinery (OT layer) to eliminate the mistakes by humans.
Depending on the speed of production and the amount of peak production activity, this commonly requires on-premise MES and database servers. Some organizations will attempt to implement an on-premise MES server with a remote database server. Because of the high database activity associated with MES implementations, any latency for a single database call is multiplied by the high activity and becomes a significant factor on overall system performance and user experience.
MES Server Load
Because synchronizing configuration and database data between multiple servers is so complicated, some companies try implementing all of their MES functionality onto one server. Commonly this puts a high load onto the one server. It also requires remote connections to devices in the field, which depending on the technologies being used, prevents tracking and controlling production during WAN outages.
With high speed production environments that utilize traceability and production verification, it may require breaking out production areas or lines on to separate MES servers. However, this leads to major challenges with synchronizing configuration and production data.
Database Load
The performance of database servers have come a long ways, but it is commonly a bottleneck to MES implementations. This is reflected in the overall performance of the MES system.
When database replication is used to synchronize configuration and historical data across production sites, areas, lines it further adds to the load on the database server.
Some queries require locking of rows, blocks, tables, indexes, etc on database resources. This causes contention and will slow down other activity involving the locked resources.
Development, Testing, and Production Support
Enterprise scale systems require having multiple systems in place that resemble the enterprise architecture. Today, propagating changes made in development through to production is cumbersome and time consuming.
It is common that these challenges lead to failed MES implementations, major frustrations and effort to implement, or poor user experiences which contributes to high costs and production inefficiencies. In fact, this is so prevalent that there is a bad taste in the mouths of many individuals leaving little confidence in the MES system. Upper management is sceptical because they have already spend money for previous fail attempts.
Solutions to Challenges
The new MES Enterprise architecture was created to solve these specific challenges and more. The intent is to reduce or even eliminate the major frustrations and effort to implement, and improve the user experience. This will lead to cost savings and increase production efficiency which will make life better for our current and future customers.
Synchronization
The core of MES Enterprise contains MES Object and vertical table synchronization out of the box. Settings will control what data and for what period of time data will be stored at each of the MES servers that are licensed under the MES Enterprise.
The approach being used for synchronization is based on changelogs. The main concept is that the current version, or very close to the current version, of MES objects are used during scheduling and production. This keeps performance high for active production but causes a performance hit when referencing old versions (which we are focusing on minimizing) of MES objects which is done less frequently. This also provides a customer requested feature of having a viewable version history for MES objects that is also accessible through scripting.
This completely eliminates the need to use database replication or some other custom means for synchronization. This results in a much better experience during the implementation, maintenance, and end user experience.
Because a portions of the change log can be exported, configuration changes can be exported from one MES Enterprise network to another. This will support testing on a QA MES Enterprise network and then transferring the updates to a production MES Enterprise network structure. This improves the experience of maintaining pre-production and production systems.
WAN Performance
By having lightweight on-premise MES and DB servers that synchronize with upper tier MES servers in a store and forward fashion, there is no requirement for an active WAN connection. If the WAN connection is down or slow, then production can continue normally and the MES data will be synchronized when the WAN connection can support it.
In essence, MES data is cached on lightweight on-premise servers eliminating the need to request the same data over the WAN over and over again.
This will greatly reduce interruptions in production due to poor network connections resulting in a better user experience.
MES Server Load
The MES Enterprise architecture now supports distributing MES server load across multiple on-premise and/or remote servers. Higher load tasks such as reports or historical analysis can be separated off to servers that are not used for live production.
This supports implementing an architecture that will provide the best possible user experience at the lowest TCO.
Database Load
The MES Enterprise architecture supports distributing storage of MES data across multiple on-premise and remote database servers without the need for vendor specific database replication. This enables greater control of the load of the database server, locks of database resources, etc. on each individual database server.
This also provides a greater level of live MES data backup in the event of failure of any one database server because it is synchronized and exists on more than one server.
Development, Testing, and Production Support
MES Enterprise will have export capability with filtering capability to greatly simplify propagating changes on demand from development through to production. Because all MES configuration data is stored in the MES database, this can be done with one step.
Solutions from Competition
A full analysis of competitors MES solutions has not been performed, but based on information that is easily available and feedback from customers, only one has synchronization across an enterprise architecture named Apriso. All other offerings have run and overload a single server, use database replication, or use complicated reporting tools to analyze data from across the enterprise.
Because all of the major competitors are build on Microsoft Windows, are large complicated systems like SAP, or have code bases that are over 20 years, the flexibility to use modern technologies is limited. They have added new functionality by acquiring companies with proven solutions and are left with different foundational architectures preventing building synchronization into the core of their MES solutions.
Other Items Learned
In addition to receiving positive and enthusiastic feedback when presenting the MES Enterprise architecture to new and potential customers, new benefits that initially were not considered have been learned.
IT Cost
IT departments of manufacturing companies are trying to transition to cloud based servers. It makes sense because having servers in the cloud will allow sharing iron, easier monitoring and support, easier backup and disaster recovery, shared and better environmental control, and etc. all enabling lower TCO.
Currently, MES solutions don’t fit into this structure for the reasons mentioned above detailing server loads and WAN performance. Today, if key MES and supporting database servers are on-premise, then backup servers must be in place and system backups must be maintained, monitored, and tested. This is a significant factor of the TCO when server hardware, labor, environmental control are all considered.
With the new MES Enterprise architecture, a hybrid approach can be used. Smaller servers can be used because data for a reduced time period can be maintained on-premise. The long term data is maintained on servers of upper tiers which can reside in the cloud or in private data centers. This is the best of both worlds with fast lightweight MES systems that require less IT support reside on-premise and heavier MES systems in the cloud or private data centers.
Disaster Recovery
The requirement to have backup servers on-premise for key MES and supporting database servers, can be eliminated. Because the data stored on-premise is for a reduced time period and full data exists at the upper tiers. To recover from a disaster, data on can be restored to a recovery server without using a database backup. Keep in mind the database backup could have been from the previous night before the failure and will not contain the current day’s production. This causes major effort to rebuild the data or else gaps in the data of which neither are good or desirable.
Furthermore, the recovery server(s) can reside in the cloud instead of being on-premise. This may cause a lower performing MES system for the reasons previously described, but may suffice until the on-premise issues are resolved.
Summary
WAN connection have vastly improved in the past 10 years and it is expected to continue to improve moving forward. Currently, WAN connections limit the ability to have MES and database servers in the cloud, but in the future this will probably not be the case. The MES Enterprise solution supports a hybrid architecture today that will easily support a fully cloud architecture in the future.
The experience of implementing, maintaining an MES Enterprise system is a major focus of the MES Enterprise 2.0 product design. It contains functionality that is outside of the normal within this industry and will contribute to MES implementations being successful at a lower cost.
Sepasoft MES Module Suite