Design of Intelligent Assistance Systems
Our conventional production lines in factories are a twentieth century artifact. They were focussed on mass production of identical products. In today’s world, products are customized to the extent of personalization. Hence, the production process for each individual piece may differ.
Monolithic assembly lines are not the best suited strategy in today’s scenario where products are customized to the extent of personalization.
Such production systems can neither be fully automated, due to a huge catalogue and no fixed dropping sequence, nor can an unaided human reliably manage a high number of variants. This has led to the rise of tools and systems that help operators in human-centric workplaces. One of the plausible solutions is to help operators through an assistance system during manual assembly processes.
Such assistance systems guide operators through the assembly process and inform them about special instructions to be kept in mind while performing the work.
In the seminal work, researchers at DFKI have proposed an architecture for such systems. For a system to make informed decisions, it is important that it has as much data as possible about the current state of the process. This can be achieved by adding different sensors to the system. However, their paper neither talks about the system boundary of a central system nor how the next state would be decided in presence of multiple CPS modules.
In this post, I am suggesting a decision making methodology for assistance systems. This key design principle is that CPS modules process their own data and provide information to central system about whether the process is in a valid or an invalid state. Before describing the complete framework, let me briefly introduce the important terms.
Notation
Our process will involve a fixed number of kinds of parts. For example, our process may require two kinds of screws {S₁} and {S₂} and a part {P}. Some of these kinds of parts may be only a transient product created during the assembly process, e.g. {Pₐ} and {Pₑ}, which will be produced while trying to get to the final product {F}. At each point in the assembly process, we will have a finite number of each kind of part on the workplace.
A state is defined as a complete description of the number and kinds of parts on the workspace. For example, a possible state of our process is:
As shown above, the state of the system can be given as a row vector, where each entry corresponds to the number of parts of a certain kind on the workspace. I will refer to them as state vectors for ease of discussion. In the attached Figure 3, the various states are labelled {a, …, g}.
A process is composed of a finite number of states and transitions between them. When a valid event takes place, it triggers a transition from the present state to the next valid state. An event can be an action performed by a human or by a machine. Processes are defined using state diagrams as shown in Figure 3. A state diagram is used to define behaviour of systems.
The central system uses queries to communicate with the CPS modules. A query is a question of the form “Has a successful transition from state A to state B taken place?”. Each module individually responds to the query via flags, described below.
A flag is used to indicate the outcome of the data processed by a CPS module and is the format in which modules communicate the outcome to the central system. So, the central system expects the information from modules in the form of one of the three flags: valid state transition (+1), unknown state transition (0) or invalid state transition (-1).
State Transition Decision Framework
Here, I propose a framework for how the decision of moving from the current state to the next state of a process should be made. Different states of a process are defined using vectors. State diagram resides on central system as well as CPS modules. When a valid event corresponding to a state occurs, central system sends query against next possible states to all the attached modules.
CPS modules receive the trigger and depending upon their state diagram and the changes in their sensors’ readings, they decide the whether the transition to the next state was valid (+1) or invalid (-1). Often, a module will not have the correct sensors to determine whether the transition was successful or not. For example, if two parts which are already on the workspace are being joined together during a transition, a weighing module will not be able to tell the difference between the two states. In that case, the module will respond with a (0) indicating its inability to determine the validity of the transition.
All module communicates this information to the central system which processes it and decides the next state of a process. A simple algorithm which the central system can follow is to take a majority vote among all future transition from its modules and make the transition. This algorithm can be enhanced by adding probabilistic priors to each transition and by weighing the responses of various modules with a different degree of confidence.
State Evolution
As stated above, the size of the state vector is defined by the number of components involved in the system. These components have two states 0 and 1 denoting absence and presence respectively, on the workplace. In the beginning, when the workplace is empty, all components are at state 0. Once a component comes into the system, its state becomes 1. We can also produce a kind of part by combining together multiple parts on the workplace, e.g. {Pₐ} = {P} + {S₁}, where + indicates joining of the parts. So after a state transition which combines {P} and {S₁}, the state of both these parts will become 0 while the state of {Pₐ} will become 1.
Execution
We will illustrate the evolution and the state transition using an example with a weighing module and a hand-tracking module.
A weighing module is a platform that has one or more weighing sensors attached to it. Containers filled with parts are kept on the weighing sensors. Each container has exactly one kind of part.
In the example shown below, a product has to be assembled using a part and two screws. Different states of the process are described using state vectors. There are three participating components, i.e., one part {P} coming from the assembly line and two screws {S₁} and {S₂}.
Looking at the state vectors, it can be seen that in state {g} there are exactly 1 intermediate part {Pₐ} and 1 screw {S₂}.
Process starts with state {a} when the workplace is empty and all state vectors are 0. The process moves from state {a} to state {b} once a part is received on the workplace table. Here, hand-tracking module registers the placement of part on workplace table. However, sensors of weighing module do not report a change in their values since the part is coming from assembly line. Thus, hand-tracking module reports (+1) and weighing module reports (0) corresponding to the query made by the central system for state {b}.
Next possible states are: state {c} and state {d}. Without the loss of any generality, the central system first queries for the validity of transition from state {b} to state {c}. Here, hand-tracking module would not be able to differentiate between state {b} and state {c}, hence the module will respond in (0). Weighing module will respond with (+1),( -1) or (0). The central system then decides on the validity of the transition and informs the modules of the successful transition.
After state {d}, there are two possible states: state {c} and state {f}. Weighing and hand-tracking modules will compare the new state’s vector with the current state’s vector and respond by stating whether the change in their sensor’s readings in between the previous query and the present state and respond with (+1),( -1) or (0).
Since the modules respond in only (0) and (+1), then the processor decides depending on the sum of the responses. Otherwise, as soon as the system encounters (-1), it breaks out of the loop.
The above state diagram can be serialized using graph description language, for example DOT. The statechart for a weighing module can be summarized as below:
a -> b -> c -> e;
b -> d -> f -> g -> e;
d -> c -> e;- {Input} nextStates, currentState, componentChanged
- {getNextState} provides a list of possible valid next states
- All modules validate the state
- {Validate} function for weighing module is below
IF{diff == 0}
STATE return 0
ELSIF{diff == n * component}
STATE return 1
ELSE
STATE return -1
ENDIFweightDiff = weight of component (last state) — current weight of component
- {triggerTransition} function evaluates the responses from various modules
- {setState} function broadcasts the current state to all the modules
Communication
Communication between central system and CPS modules is shown in the figure below. At the onset, central system initializes the system. Modules (weighing module, as an example in the figure below) send ACK to the central system. In the second step, central system broadcasts the present state to all modules. After the central system and modules have the same state, system waits for an event.
As soon as an event occurs corresponding to a state, central system sends the next possible states {e, f} to modules. Weighing module evaluates the event against the possible transitions and sends {e:-1, f:-1}. This means that neither {e} nor {f} is a next valid state for weighing module. This can be due to race condition or any other hardware / software error. Central system iterates the query in some time. Depending upon the transition, weighing module answers in {e:1, f:-1}.
Problems like, if two or more valid states have same response from modules, can be tackled by using shortest path algorithms.
In possible extensions of the work, the central system can be replaced by a protocol which collects flags from all CPS modules and decides the next step, thus moving towards a completely decentralized system.
I have proposed and discussed another approach of designing intelligent, interoperable assistance systems in my master thesis. Read it here.
