To train an AI using deep reinforcement learning (DRL), it is necessary to use an accurate simulator since you cannot train an AI from scratch using a live process. As this article describes, there are multiple approaches to developing simulators. Three of the most common ones are: physics-based, simulation platform-based, or AI-based simulators.

With discreet manufacturing or other discreet processes, using simulations such as those from AnyLogic, makes a lot of sense because even if the total process is very complex it is often constructed of small clearly defined steps. For instance, these tools are great for modeling complex supply chains or distribution centers.

Process manufacturing is quite a different challenge altogether.

Although a process might comprise a limited number of steps — for instance, extrusion and baking for a food snack — each of these steps will have a higher complexity and will be tougher to appropriately model than discreet processes. Because of this process manufacturing specificity, there are four key reasons why an AI (i.e., data-based) simulator is often the best solution to implement reinforcement learning AI agents successfully. And this is true whether you leverage the Microsoft Project Bonsai toolchain or code your DRL agent and its training process.

Top reasons to use an AI simulator in process manufacturing

The top four reasons to use an AI simulator in process manufacturing are

1. Not all process parameters are always necessary for the DRL-trained AI agent to appropriately monitor and control it. Therefore, an AI model may be simpler to define than a physics-based one.
2. A process often involves several complex physical and chemical sub-processes or steps that are hard to model and are often very susceptible to variation based on multiple aspects intrinsic or external aspects.
3. The skills needed to develop physics-based models are usually not present in a manufacturing team. Conversely, AI simulators only require the team to gather real life production data to generate an appropriate AI training data set.
4. Physics or model-based simulators are often not robust enough to adapt to process variations. However, an AI simulator provided with the right training data can easily support a wider range of operating environments.

Let’s dig a bit more into those four reasons.

1. Not all process parameters are always necessary to build a simulation

A process can easily have tens of parameters needed to thoroughly define it. For the sake of illustration, I will use the example of extruders for the remainder of this article, but the concepts apply to many process manufacturing examples.

To exhaustively define a plastic extruder, for instance, the parameters will range from

• Raw material specifications (plastic melting point, viscosity, temperature, shear rate, etc.)
• Feed throat specifications
• Extruder diameter and length
• Screw geometry and rotation speed
• Heating elements added throughout the extruder
• Feed pipe size
• Die properties (shape, size, through-die plastic viscosity and shear rate)
• And more

Simplified plastic extruder schematic

However, to realistically represent an extruder in a simulator used for DRL training, is this level of granularity really needed? Often, it isn’t.

For instance, when working on PepsiCo’s a food extrusion process (granted, corn meal is not the same as plastic), Neal Analytics developed an AI simulation for its Cheetos extrusion and baking process that only needed seven process variables to appropriately train (first) and control (once deployed) the AI agent (aka Project Bonsai “brain”). More details can be found here.

PepsiCo’s Cheetos process variables

2. Physics-based simulations for processes are hard to build

When Neal Analytics built the plastic extruder “in box” demo for Microsoft Project Bonsai, one of the key elements was to develop a realistic simulation for a somewhat generic plastic extruder.

However, even with quite a few simplifications and assumptions, the model is nothing but simple. To build this model, our data scientists made (among others) the following assumptions:

• It is a certain type of plastic: PVC. It’s not a generic “plastic extruder”
• It assumed the plastic is already at temperature and throughout the screw.
• There aren’t any heating elements in the screw
• The die is a simple rod. No fancy shapes, just a rod and the only variables for the simulations and AI agent training are the rod diameter and length.
• And a few more that are too detailed for the purpose of this article but that is documented in the source code available on Microsoft’s GitHub repo: https://go.nealanalytics.com/sim-plastic.py

Still despite those simplifications, and although the equations are simple ones (no integrations, derivatives, or other “fun” algebraic or calculus-based elements), the complete model is quite complex.

The following diagram is a (simplified) representation of the Python simulation code in the demo.

3. Lack of required physics and chemistry skills in house

Unless your team is staffed with the right engineers that happen to be experts of the physics and chemistry behind your processes, it will be very hard to build in-house knowledge to create such models as the simplified PVC extrusion above.

Creating a simulation that is representative of your actual process, and not a demo, can be order of magnitudes more complex. This will often make it virtually impossible to build in-house from a timeline, cost, and skill perspective.

Contrary to physics-based simulations, AI simulations do not require such a detailed scientific knowledge. They only require two things.

• First, your experts will need to leverage their tacit process knowledge to define the relevant parameters that the simulation should comprise, such as the seven in the food extrusion example above.
• Then, using real life production data coming from your existing processes, the AI simulations training data can be gathered and prepared before being used to train this “black box” simulator. This AI will only simulate the relevant variable (i.e., external behavioral and environmental ones needed only) without trying to precisely model the internal details of the process behavior.

4. Physics-based simulators are less robust than AI simulators

As the example above showed, because those physics-based simulators leverage a significant number of internal variables and parameters, they are, by nature, not robust to any changes.

For instance, if the PVC specifications are somewhat different (e.g., shear rate), if the temperature gradient throughout the screw is not as modeled, or if screw wear and tear change its operating parameters, the physics model will undoubtedly fail.

The AI simulator will only have to be retrained adding new data encompassing those changes to adapt and still appropriately represent the process.

Conclusion

There are multiple ways to simulate a process for DRL training purposes. Microsoft Project Bonsai is very flexible and can accept virtually any model, but it does not mean all simulation types work for all projects. If physics-based models have their role, especially with simple and stable processes, AI simulators will often be the best approach for process manufacturing modeling.

Therefore, it’s important, when starting a project, to partner with a company that has the breadth of expertise to help you select and build a simulator using the approach that makes the most sense for your needs.

Have a process manufacturing project in mind? Contact our team to see how we can help you leverage AI simulators in your solution.

Learn more

• Keys to developing an RL-ready simulator
• Optimizing plastic extrusion with AI: Self-service demo
• A beginner’s guide to Autonomous Systems (AS 101) series