Challenge Rules

import os
abs_path = os.path.abspath(os.path.dirname(__file__))

Environment

In this challenge, DJSSP environments (simulators) are developed based on real-world production lines. An environment simulates the producing procedure of a job shop in an iterative manner. In each "step" it takes in the decisions (made by agents), updates the states of the job shop (e.g., assigns works to specified machines, proceeds operation under process, finishes operations and makes machine available again, etc.), and outputs information to the agents.

The unit of time used in all environments is "minute".

Interface

Each environment in this challenge is an instance of the Environment class, implemented with Python. In the rest of this section, we will assume env is an instance of Environment. The interface  Environment includes:

machines is a static member variable (i.e., does not change during simulation) that provides information about machines in the environment. Machines are organized by types (in the form of a dict), and for each type all machine ids (string) are given in a list. Here is an example of machines.

env.machines = {
    'A': ['M01', 'M02'],	# type: list of machine ids, ids are strings
    'B': ['M03' ],
    ...	# other types
}

job_types is a static member variable that provides information about all job types considered in the environment. For each type, the information of all operations is listed in order. For each operation, operation name (string), required machine type (string), processing time (int), pending time requirement (int) are given. Here is an example of job_types.

env.job_types = {
    'a': {  
            [	# operations
                {'op_name': 'AO1',	# operation name
                 'machine_type': 'A',	# required machine type
                 'process_time': 10,	# processing time (minutes)
                 'max_pend_time': 10,	# maximal pending time (minutes)
                }, 
                {'op_name': 'AO2', 
                 ...
                }
                ...	# other operations
            ]	
        }
    ... # other job types
}

jobs is a static member variable that provides information about all jobs that should be processed in the environment. Jobs are organized in types (string), and for each job, the arriving time (int) and the priority (int, $0\sim 5$) can be found. Here is an example of jobs.

env.jobs = {
    'a': {
        'J01': {
            'arrival': 10, 
            'priority': 3,  # 0 ~ 5
        },
        ... # other jobs
    }
    ... # other job types
}

__init__() is the initialization method of the environment class. It takes only one input (besides self):

1. config, the path to the configuration file that indicates the job types, machines, jobs, PTV weights, and breakdown events. Details on configuration files can be found below.

The usage of __init__() is as follows.

 # usage of env.__init__()
 env = Env(config='./config/xxx/train_1.yaml')

reset() is a method that initializes the environment, outputs machine and job status (machine_status and job_status respectively, see below for details), current time (time), as well as the candidate job list (job_list) for the first step. The usage of reset() is as follows.

# usage of env.reset()
machine_status, job_status, time, job_list = env.reset()

• machine_status provides the current status of all machines in the environment. For each machine (key in a dict), machine type (string), current status ('down', 'idle', or 'work'), and remaining process time (int), job under processing (job id, string), and pending job list (set of job ids) will be given. Here is an example of machine_status.

# example of machine_status
machine_status = {
    'M01': {
        'type': 'A',	# machine type
        'status': 'idle',	# 'down', 'idle', 'work'
            # 'down' means the machine is suffering a breakdown,
            # 'idle' means the machine is available for an other job,
            # 'work' means the machine is processing a job
        'remain_time': 5,	# remaining time of processing. 
            # if the job is 'down' or 'idle', remain_time is 0
        'job': 'J01',	# job under processing, None if 'down' or 'idle'
        'job_list': {'J02', 'J05', ... }, # set() of job ids
    },
    ...	# other jobs
}

• job_status provides the current status of all jobs in the environment. For each job (key in a dict), job type (string), priority (int), current status ('work', 'pending', 'to_arrive', or 'done'), remaining arrival time(int), current operation (string), remaining process time (int), remaining pending time (int), and the processing machine (string) are given. Here is an example of job_status.

# example of job_status
job_status = {
    'J01': {
        'type': 'a', # job type
        'priority': 3, # int, 0 ~ 5
        'status': 'work',	# 'work', 'pending', 'to_arrive', or 'done'
            # 'work' means the job is under processing
            # 'pending' means the job is waiting for processing
            # 'to_arrive' means the job has not arrived
            # 'done' means the job is finished
        'arrival': 0,	# remaining time to arrive, 0 if the job has arrived, 
            # > 0 if the job has not arrived
            # None if the remaining time is out of the time window
        'op': 'AO2',	# the current operation name,
            # if not arrived, uses the first operation name
        'remain_process_time': 5,	# remaining process time
            # if 'pending', uses the processing time of the current operation
        'remain_pending_time': 8,	# remaining pending time
            # can be less than 0, if so, means pending time have been violated
        	# inf if the job has not yet arrived
            # if the status is 'work', keeps the last value before processing
        'machine': 'M01',	# processing machine, None if not under processing
    },
    ... #other jobs
}

• time is an int that indicates the current simulation time, in minutes.
• job_list is a dict where each machine id is the key and the value is a list of jobs that can currently be processed by the machine.

# example of job_list
job_list = {
    'M01': ['J01', 'J02'],	# list of job ids
    'M02': ['J01', 'J02'],
    'M03': [],	# empty list if the machine is not idle or no jobs are available
    ...	# other machines
}

step() is a method that takes the decisions (job assignment) as input, simulates for one step further, and outputs the new machine and job status, new candidate-job lists, as well as the intermediate rewards(s) agent(s) receives in the step. It also indicates whether the environment is terminated (i.e., all jobs are done or the time limit is violated). The usage of step() and examples of its input and outputs are as follows.

# usage of step()
machine_status, job_status, time, reward, job_list, done = env.step(job_assignment)

• machine_status, job_status, time, and job_list are the same as the outputs of reset().
• job_assignment is the input  step() that indicates which machine accepts which job at the current step. It is a dict whose keys should be a subset of all machines that accept jobs at this step.

# input: job_assignment
job_assignment = {
    'M01': 'J01',	# assigned job
    'M02': 'J02',
    'M03', None,	# None if assign no jobs
    ...	# other jobs
}

• reward describes the rewards agent gets for this step.

# examples of reward
reward = {
    'makespan': -1,	# reward related to makespan
    'PTV': 0	# reward related to pending time violation
                # for each violated job, for each timestep (min)
                # PTV = job_priority * ptv_weight
}

• done is a bool variable that indicates whether the episode is finished(True) or not (False). An episode terminates if all jobs are done, or a maximum step number is reached.

Configuration File

A configuration file describes a DJSSP problem with specific machines, job types, jobs, machine breakdown events, and the importance of pending time violations. Each configuration file is a .yaml file that comprises five parts:

• job_types: for each job type, information about all the operations are given, including: 1) the name of the operation, 2) the required machine type, 3) the maximal pending time, and 4) the processing time;

job_types:
 a: 	# job type
  - op_name: A01
 	machine_type: A
 	process_time: 10
 	max_pend_time: 10
  - op_name: A02 # other operation
	...
 ... # other type

• machines: for each machine type, ids of the corresponding machines are listed;

machines:
 A:		# machine type
  - M001	# machiens of this type
  - M002
  - M003
 B:
  - M004
  ...	# other machines
 # other types

• jobs: for each type, information about all corresponding jobs are given, including: 1) the arrival time, and 2) the priority of the job;

jobs:
 a:		# job type
  J0001:		# job id
   arrival: 0	# arrival time
   priority: 0	# priority
  ...	# other jobs
 ... # other job types

• breakdown: indicates the machine that will breakdown and the corresponding time;

breakdown:
 - M01:	0 # breakdown machine and breakdown time
 - ... # other machines

• ptv: indicates the weight of pending time violation;

ptv: 10	# the weight of pending time violation

Reward Calculation

For each passing minute, the agent gets a -1 'makespan' reward. And for each minute that a job violates its maximal pending time constraint, the agent gets a penalty equal to job_priority * ptv for this job, where ptv is the pending time violation weight defined in the configuration file.

The rewards related to makespan and pending time violations are given separately. During the evaluation, we calculate the total reward by simply adding them up, but the participants can make use of the split reward signals.

Environment Characteristics

Setup

Phases

In this challenges, three phases are designed to enable participants develop their solutions and check the correctness, and the organizers to evaluate final submissions. To be more specific, the three phases are:

Feedback Phase

In this phase, participants develop their solution and make improvements based on feedback. Solutions will be uploaded to the platform and participants will receive feedback on the performance evaluated by five feedback environments. Each participant is allowed to make ONE submission per day.

Check Phase

In this phase participants will be able to check whether their submission (migrated from the previous phase) can be successfully evaluated by five private environments. Participants will be informed if failure caused by memory or overtime issues occurs, but no results on performances will be revealed. Each participant will be given one chance to correct her submission.

Private Phase

This is the blind test phase with no submissions. In this phase, solutions will be automatically evaluated by private environments, and the participants final scores will be calculated based on the results in this phase. In this phase, solutions will be automatically evaluated by private environments, and the participants final scores will be calculated based on the results in this phase.

Summary

The following table shows the numbers of environments for different phases.

The public environments will be included in the staring kit. The feedback and private environments can not be downloaded by participants.

Notice

The public environments will be included in the staring kit. The feedback and private environments can not be downloaded by participants.

Evaluation Method

In each phase, the solution will be evaluated by all the corresponding environments. Now we will demonstrate how the solution is evaluated by one specific environment.

For each environment, two stages, namely the training stage and the testing stage, are used to evaluate the solution. In each stage, five configurations will be used. All these ten configuration files are generated by a generator with fixed distribution or random process.

The participant should submit their solution in the form of a Trainer (details can be found in the Submission page).

In the training stage, the Environment class, and a list containing the paths to five training configuration files are sent to the Trainer and the latter automatically trains an Agent object (agent).

Then, in the Testing stage, the agent is tested using the Environment class and five unseen testing configuration files. Average long-term return (i.e., the summation of intermediate rewards throughout the episode) throughout these five configurations is used to evaluate the performance of the agent. The higher the better. If randomness is involved in the environment, repeated rollouts will be performed and the average return will be used.

The following figure illustrates the evaluation procedure on one environment.

Scoring

For each phase, each solution will be ranked on each environment by the average long-term return, and the average rank through all environments is used as the solution's rank. For each environment, 10 repeated runs are performed to evaluate the policy, and the total testing time budget for each environment is 1800 seconds (i.e., 1800 seconds for 10 episodes).  The average rank in the Private Phase is the final rank of a participant's solution.

Computing Resource

The submission will be run on a workstation with 6 CPU cores, 56GB RAM, and 1 K80 GPU.

Running Time Constraints

For each environment, Train.train() is allowed to run for a given time budget (calculated in seconds, see Submission page for the detailed interface). If the time budget is exceeded, the solution will be assigned a low score for the environment. Hence, participants should track the running time in Train.train().

Python Environment

The Python environment on the competition platform has installed the following packages (only those that are used frequently in ML and RL are listed).

Package                  Version
------------------------ -------------------
conda                    4.9.2
gym                      0.18.0
numpy                    1.19.5
pandas                   1.2.3
pip                      20.2.4
ray                      2.0.0.dev0
redis                    3.5.3
scipy                    1.6.2
tensorflow               2.4.1
torch                    1.8.1

If the participants would like to install other third-party packages, please refer to the Submission page.

Submission, Starting Kit, and Baselines

The starting kit can be downloaded from Baidu Netdisk (the extraction code is hnjk) or Google Cloud.

Submission

The solution should be a python module solution.py where a Trainer class is implemented.

Trainer

The Trainer should have two main methods, __init__() and train(), as stated below:

__init__(Env, train_conf_list)

The initialization method has two inputs (besides self):

1. Env, an environment class defined in the Environment Page;
2. train_conf_list, a list of paths to five training configuration files

The interface of the __init__() method is as follows.

class Trainer:
    def __init__(self, Env, train_conf_list):
        pass

In most cases, participants need to initialize Environment objects with specific configuration files in the trainer. The initialization method can be found in the Environment Page.

env = Env(config='path_to_the_configuration_file')

train(time)

train() has one input, the time budget time that indicates the time budget for the training, calculated in seconds. train() outputs an Agent object for a given environment. The interface of Trainer.train() and Agent.act() is as follows.

class Trainer:
    def train(self, time):	
        # takes the time budget (sec) as input
        ...
        return agent	# outputs an Agent object

During the evaluation on a given environment, if the running time of train() exceeds the given time budget, the evaluation process will be terminated and the solution will be assigned a low score for the environment.

Agent

Agent is the output of Trainer.train() and it is an object that represents the learned agent(s) or policy for the given DJSSP environment.

An Agent object should have an act() method that takes machine status, job status, current time, and job list as inputs and outputs a job assignment (see Environment Page for more details).

Note that multi-agent algorithms can be employed in this challenge. The participants can integrate multiple agents into an agent object, as long as the interface protocol is followed. The interface of Agent.act() is as follows.

class Agent:
    def act(self, machine_status, job_status, time, job_list):	
        ...
        return job_assignment	# outputs job assignment

Dependencies and Third-party packages

The challenge platform will automatically append the folder where solution.py locates to $PYTHONPATH.

If third-party packages (that are not listed in the Setup page) are used, put a requirements.txt file in the same folder with solution.py.

Starting Kit

To help participants develop their solution codes, we provide the participants a starting-kit that includes:

• the Environment class with three sets of public configuration files.
• two baseline solutions;
• programs that we use in our platform to run participants’ solutions and scoring their results;

Please find the downloading URL at the top of this page.

Please read the README.md carefully to learn how to run local test and upload your code.

Baselines

Reinforcement Learning-based Baseline

A simple multi-agent reinforcement learning (MARL) method is included in the starting kit. Each machine is considered as an agent and a shared policy is learned for all machines. Besides machine status, statistical information on jobs and other machines are designed and used as observation of each agent, in order to improve the generalization ability of the solution. Simple priority rules (rules that used to determine which job to process first) are used as actions. To be more specific, "pending-time-first" (if at least one non-zero priority job is waiting, it processes the job with the shortest remaining pending time, weighted by the priority; otherwise, it processes the job with the shortest processing time, aka. shortest-processing-time) and "waiting" (doing nothing and wait for further jobs) are the two actions that an agent can take. All agents share a global reward signal which is simply the summation of makespan and PTV-related rewards. Parameter sharing PPO (PS-PPO) is selected as the MARL algorithm. The neural network architecture is automatically generated with a self-adaptive input layer but fixed other layers. For each environment, the agents are simply trained with the provided training configuration files, by sampling one out of five files per episode.

We believe the performance and efficiency of the baseline method can be much improved if more sophisticated methods are employed and automated, for instance, better action and state representations, hyper-parameter tuning, reward shaping, better training mechanisms, and model-based approaches, just to name a few.

Rule-based Baseline

We also provide a rule-based baseline method which is based on the pending-time-first priority rule. Different from pending-time-first rule, when all waiting jobs are assigned zero priority, this rule estimates if processing a job with the shortest processing time will cause future pending time violation. If the answer is yes, the rule will wait, and otherwise, it follows the shortest-processing-time rule.

Notice

During the feedback phase, after submitting your code, please see "My Submissions" tab for the running results.

A "Finished" status means the code is evaluated correctly, and the results can be found in the "Results" tab.

If the status is "Failed", a low score will be assigned to each environment and the results can be found in table "Results - Feedback Phase". ( Table "Results - Dataset x" will keep the score of your last successful submission, but the score will not be used for ranking. Hence, only the information in the "Feedback Phase" table counts!)