Knowledge-Based Systems, volume 218, pages 106844

Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations[Formula presented]

Publication typeJournal Article
Publication date2021-04-01
Quartile SCImago
Q1
Quartile WOS
Q1
Impact factor8.8
ISSN09507051
Artificial Intelligence
Software
Management Information Systems
Information Systems and Management
Abstract
Deep reinforcement learning (RL) shows impressive results in complex gaming and robotic environments. These results are commonly achieved at the expense of huge computational costs and require an incredible number of episodes of interactions between the agent and the environment. Hierarchical methods and expert demonstrations are among the most promising approaches to improve the sample efficiency of reinforcement learning methods. In this paper, we propose a combination of methods that allow the agent to use low-quality demonstrations in complex vision-based environments with multiple related goals. Our Forgetful Experience Replay (ForgER) algorithm effectively handles expert data errors and reduces quality losses when adapting the action space and states representation to the agent’s capabilities. The proposed goal-oriented replay buffer structure allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations. Our method has a high degree of versatility and can be integrated into various off-policy methods. The ForgER surpasses the existing state-of-the-art RL methods using expert demonstrations in complex environments. The solution based on our algorithm beats other solutions for the famous MineRL competition and allows the agent to demonstrate the behavior at the expert level.

Citations by journals

1
2
3
Knowledge-Based Systems
Knowledge-Based Systems, 3, 25%
Knowledge-Based Systems
3 publications, 25%
Studies in Computational Intelligence
Studies in Computational Intelligence, 2, 16.67%
Studies in Computational Intelligence
2 publications, 16.67%
IEEE Access
IEEE Access, 1, 8.33%
IEEE Access
1 publication, 8.33%
Lecture Notes in Computer Science
Lecture Notes in Computer Science, 1, 8.33%
Lecture Notes in Computer Science
1 publication, 8.33%
International Journal of Information Technology and Web Engineering
International Journal of Information Technology and Web Engineering, 1, 8.33%
International Journal of Information Technology and Web Engineering
1 publication, 8.33%
Springer Optimization and Its Applications
Springer Optimization and Its Applications, 1, 8.33%
Springer Optimization and Its Applications
1 publication, 8.33%
Information Sciences
Information Sciences, 1, 8.33%
Information Sciences
1 publication, 8.33%
Lecture Notes in Networks and Systems
Lecture Notes in Networks and Systems, 1, 8.33%
Lecture Notes in Networks and Systems
1 publication, 8.33%
1
2
3

Citations by publishers

1
2
3
4
5
Springer Nature
Springer Nature, 5, 41.67%
Springer Nature
5 publications, 41.67%
Elsevier
Elsevier, 4, 33.33%
Elsevier
4 publications, 33.33%
IEEE
IEEE, 1, 8.33%
IEEE
1 publication, 8.33%
IGI Global
IGI Global, 1, 8.33%
IGI Global
1 publication, 8.33%
1
2
3
4
5
  • We do not take into account publications that without a DOI.
  • Statistics recalculated only for publications connected to researchers, organizations and labs registered on the platform.
  • Statistics recalculated weekly.
Metrics
Share
Cite this
GOST |
Cite this
GOST Copy
Skrynnik A. et al. Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations[Formula presented] // Knowledge-Based Systems. 2021. Vol. 218. p. 106844.
GOST all authors (up to 50) Copy
Skrynnik A., Staroverov A., Aitygulov E., Davydov V., Panov A., Aksenov K., Staroverov A. Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations[Formula presented] // Knowledge-Based Systems. 2021. Vol. 218. p. 106844.
RIS |
Cite this
RIS Copy
TY - JOUR
DO - 10.1016/j.knosys.2021.106844
UR - https://doi.org/10.1016%2Fj.knosys.2021.106844
TI - Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations[Formula presented]
T2 - Knowledge-Based Systems
AU - Skrynnik, Alexey
AU - Staroverov, Aleksey
AU - Aitygulov, Ermek
AU - Davydov, Vasilii
AU - Panov, Aleksandr
AU - Aksenov, Kirill
AU - Staroverov, Aleksei
PY - 2021
DA - 2021/04/01 00:00:00
PB - Elsevier
SP - 106844
VL - 218
SN - 0950-7051
ER -
BibTex
Cite this
BibTex Copy
@article{2021_Skrynnik
author = {Alexey Skrynnik and Aleksey Staroverov and Ermek Aitygulov and Vasilii Davydov and Aleksandr Panov and Kirill Aksenov and Aleksei Staroverov},
title = {Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations[Formula presented]},
journal = {Knowledge-Based Systems},
year = {2021},
volume = {218},
publisher = {Elsevier},
month = {apr},
url = {https://doi.org/10.1016%2Fj.knosys.2021.106844},
pages = {106844},
doi = {10.1016/j.knosys.2021.106844}
}
Found error?