MIT Researchers Develop an Effective Way to Train more Reliable AI Agents

Fields ranging from robotics to medicine to political science are attempting to train AI systems to make meaningful choices of all kinds. For example, using an AI system to wisely control traffic in a congested city might assist motorists reach their locations quicker, while enhancing security or sustainability.

Unfortunately, teaching an AI system to make great decisions is no simple job.

Reinforcement knowing models, which underlie these AI decision-making systems, still often stop working when faced with even small variations in the jobs they are trained to perform. When it comes to traffic, a design might have a hard time to control a set of intersections with different speed limitations, numbers of lanes, or traffic patterns.

To boost the dependability of reinforcement learning models for complex jobs with irregularity, MIT scientists have presented a more effective algorithm for training them.

The algorithm tactically selects the very best tasks for training an AI agent so it can successfully perform all jobs in a collection of associated tasks. In the case of traffic signal control, each job could be one crossway in a task space that includes all intersections in the city.

By concentrating on a smaller sized variety of crossways that contribute the most to the algorithm’s total efficiency, this technique maximizes efficiency while keeping the training expense low.

The researchers found that their strategy was between five and 50 times more efficient than standard methods on a variety of simulated tasks. This gain in performance helps the algorithm find out a better service in a faster way, eventually improving the performance of the AI agent.

“We were able to see incredible performance enhancements, with an extremely simple algorithm, by thinking outside the box. An algorithm that is not very complex stands a much better opportunity of being embraced by the neighborhood because it is much easier to carry out and easier for others to understand,” states senior author Cathy Wu, the Thomas D. and Virginia W. Cabot Career Development Associate Professor in Civil and Environmental Engineering (CEE) and the Institute for Data, Systems, and Society (IDSS), and a member of the Laboratory for Information and Decision Systems (LIDS).

She is joined on the paper by lead author Jung-Hoon Cho, a CEE graduate student; Vindula Jayawardana, a graduate trainee in the Department of Electrical Engineering and Computer Science (EECS); and Sirui Li, an IDSS college student. The research will be presented at the Conference on Neural Information Processing Systems.

Finding a middle ground

To train an algorithm to manage traffic lights at many crossways in a city, an engineer would normally choose in between 2 primary approaches. She can train one algorithm for each intersection separately, utilizing only that intersection’s information, or train a larger algorithm using information from all intersections and then use it to each one.

But each approach includes its share of disadvantages. Training a different algorithm for each task (such as a provided intersection) is a time-consuming process that needs an enormous quantity of data and computation, while training one algorithm for all jobs often causes subpar performance.

Wu and her partners looked for a sweet area in between these two techniques.

For their approach, they choose a subset of tasks and train one algorithm for each task separately. Importantly, they strategically choose individual jobs which are most likely to enhance the algorithm’s general performance on all jobs.

They utilize a typical trick from the support knowing field called zero-shot transfer learning, in which an already trained design is applied to a brand-new job without being additional trained. With transfer learning, the model frequently performs extremely well on the brand-new neighbor task.

“We understand it would be perfect to train on all the tasks, but we questioned if we might get away with training on a subset of those jobs, apply the result to all the jobs, and still see an efficiency increase,” Wu says.

To recognize which tasks they ought to select to make the most of expected performance, the scientists established an algorithm called Model-Based Transfer Learning (MBTL).

The MBTL algorithm has 2 pieces. For one, it models how well each algorithm would perform if it were trained individually on one job. Then it models just how much each algorithm’s efficiency would deteriorate if it were transferred to each other job, a concept known as generalization performance.

Explicitly modeling generalization efficiency enables MBTL to approximate the worth of training on a new task.

MBTL does this sequentially, picking the job which results in the greatest efficiency gain first, then choosing additional tasks that provide the biggest subsequent minimal enhancements to general performance.

Since MBTL just concentrates on the most promising jobs, it can considerably improve the efficiency of the training procedure.

Reducing training costs

When the scientists tested this method on simulated tasks, including controlling traffic signals, managing real-time speed advisories, and executing numerous timeless control jobs, it was five to 50 times more efficient than other techniques.

This implies they could reach the exact same solution by training on far less data. For instance, with a 50x efficiency increase, the MBTL algorithm could train on just 2 tasks and attain the very same performance as a standard technique which uses data from 100 jobs.

“From the point of view of the 2 primary approaches, that means information from the other 98 jobs was not needed or that training on all 100 tasks is confusing to the algorithm, so the performance winds up worse than ours,” Wu states.

With MBTL, including even a percentage of extra training time could lead to better performance.

In the future, the researchers prepare to create MBTL algorithms that can encompass more complex issues, such as high-dimensional task areas. They are also interested in their technique to real-world problems, particularly in next-generation movement systems.