Open Access
Open access

Long-Term Exploration in Persistent MDPs

Ugadiarov L., Skrynnik A., Panov A.I.
Тип документаBook Chapter
Дата публикации2021-01-01
Название журналаLecture Notes in Computer Science
ИздательSpringer Nature
ISSN03029743, 16113349
Краткое описание
Exploration is an essential part of reinforcement learning, which restricts the quality of learned policy. Hard-exploration environments are defined by huge state space and sparse rewards. In such conditions, an exhaustive exploration of the environment is often impossible, and the successful training of an agent requires a lot of interaction steps. In this paper, we propose an exploration method called Rollback-Explore (RbExplore), which utilizes the concept of the persistent Markov decision process, in which agents during training can roll back to visited states. We test our algorithm in the hard-exploration Prince of Persia game, without rewards and domain knowledge. At all used levels of the game, our agent outperforms or shows comparable results with state-of-the-art curiosity methods with knowledge-based intrinsic motivation: ICM and RND. An implementation of RbExplore can be found at
Пристатейные ссылки: 18
Hierarchical Reinforcement Learning with Clustering Abstract Machines
Alexey S., Panov A.I.
Q4 Communications in Computer and Information Science 2019 цитирований: 2
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak D., Agrawal P., Efros A.A., Darrell T.
2017 цитирований: 271
Making data structures persistent
Driscoll J.R., Sarnak N., Sleator D.D., Tarjan R.E.
Q2 Journal of Computer and System Sciences 1989 цитирований: 321
1. Ugadiarov L., Skrynnik A., Panov A.I. Long-Term Exploration in Persistent MDPs // Lecture Notes in Computer Science. 2021. С. 108–120.


DO - 10.1007/978-3-030-89817-5_8

UR -

TI - Long-Term Exploration in Persistent MDPs

T2 - Advances in Computational Intelligence

AU - Ugadiarov, Leonid

AU - Skrynnik, Alexey

AU - Panov, Aleksandr I.

PY - 2021

PB - Springer International Publishing

SP - 108-120

SN - 0302-9743

SN - 1611-3349

ER -

BibTex |


doi = {10.1007/978-3-030-89817-5_8},

url = {},

year = 2021,

publisher = {Springer International Publishing},

pages = {108--120},

author = {Leonid Ugadiarov and Alexey Skrynnik and Aleksandr I. Panov},

title = {Long-Term Exploration in Persistent {MDPs}},

booktitle = {Advances in Computational Intelligence}


Ugadiarov, Leonid, Alexey Skrynnik, and Aleksandr I. Panov. “Long-Term Exploration in Persistent MDPs.” Lecture Notes in Computer Science (2021): 108–120. Crossref. Web.