Cognitive Systems Research, volume 65, pages 74-78

Hierarchical Deep Q-Network from imperfect demonstrations in Minecraft

Skrynnik Alexey ¹

Staroverov Aleksei ²

Aitygulov Ermek ²

Aksenov Kirill ³

Davydov Vasilii ⁴

Panov Aleksandr ⁵

Hide authors affiliations Show authors affiliations: 5 affiliations

Artificial Intelligence Research Institute FRC CSC RAS, Russia

Artificial Intelligence Research Institute

Federal Research Center Computer Science and Control of the Russian Academy of Sciences

[Moscow Institute of Physics and Technology, Russia] |

Higher School of Economics, Russia |

⁴

Moscow Aviation Institute, Russia |

⁵

Moscow Institute of Physics and Technology, Russia |

Publication type: Journal Article

Publication date: 2021-01-01

Elsevier

Journal: Cognitive Systems Research

Quartile SCImago

Quartile WOS

Impact factor: 3.9

ISSN: 13890417

DOI: 10.1016/j.cogsys.2020.08.012

Copy DOI

Artificial Intelligence

Software

Experimental and Cognitive Psychology

Cognitive Neuroscience

Abstract

We present Hierarchical Deep Q-Network (HDQfD) that won first place in the MineRL competition. The HDQfD works on imperfect demonstrations and utilizes the hierarchical structure of expert trajectories. We introduce the procedure of extracting an effective sequence of meta-actions and subgoals from the demonstration data. We present a structured task-dependent replay buffer and an adaptive prioritizing technique that allow the HDQfD agent to gradually erase poor-quality expert data from the buffer. In this paper, we present the details of the HDQfD algorithm and give the experimental results in the Minecraft domain.

By date By citations

Citations by journals

	1 2 3
Lecture Notes in Computer Science	Lecture Notes in Computer Science, 3, 30% Lecture Notes in Computer Science 3 publications, 30%
Studies in Computational Intelligence	Studies in Computational Intelligence, 1, 10% Studies in Computational Intelligence 1 publication, 10%
Scientific Programming	Scientific Programming, 1, 10% Scientific Programming 1 publication, 10%
Computational Intelligence and Neuroscience	Computational Intelligence and Neuroscience, 1, 10% Computational Intelligence and Neuroscience 1 publication, 10%
Pattern Recognition and Image Analysis	Pattern Recognition and Image Analysis, 1, 10% Pattern Recognition and Image Analysis 1 publication, 10%
Lecture Notes in Networks and Systems	Lecture Notes in Networks and Systems, 1, 10% Lecture Notes in Networks and Systems 1 publication, 10%
	1 2 3

Citations by publishers

	1 2 3 4 5
Springer Nature	Springer Nature, 5, 50% Springer Nature 5 publications, 50%
Hindawi Limited	Hindawi Limited, 2, 20% Hindawi Limited 2 publications, 20%
Pleiades Publishing	Pleiades Publishing, 1, 10% Pleiades Publishing 1 publication, 10%
	1 2 3 4 5

We do not take into account publications that without a DOI.
Statistics recalculated only for publications connected to researchers, organizations and labs registered on the platform.
Statistics recalculated weekly.

{"yearsCitations":{"type":"bar","data":{"show":true,"labels":[2021,2022,2023],"ids":[0,0,0],"codes":[0,0,0],"imageUrls":["","",""],"datasets":[{"label":"Citations number","data":[2,6,2],"backgroundColor":["#3B82F6","#3B82F6","#3B82F6"],"percentage":["20","60","20"],"barThickness":null}]},"options":{"indexAxis":"x","maintainAspectRatio":true,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":1,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Citations per year","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}},"journals":{"type":"bar","data":{"show":true,"labels":["Lecture Notes in Computer Science","Studies in Computational Intelligence","Scientific Programming","Computational Intelligence and Neuroscience","Pattern Recognition and Image Analysis","Lecture Notes in Networks and Systems"],"ids":[1022,2714,10551,3579,9753,17269],"codes":[0,0,0,0,0,0],"imageUrls":["\/storage\/images\/resized\/voXLqlsvTwv5p3iMQ8Dhs95nqB4AXOG7Taj7G4ra_medium.webp","\/storage\/images\/resized\/voXLqlsvTwv5p3iMQ8Dhs95nqB4AXOG7Taj7G4ra_medium.webp","\/storage\/images\/resized\/hqfzhQAjTGlNSRs6yzFNITgjSMm9Jr2QuotJHIvE_medium.webp","\/storage\/images\/resized\/hqfzhQAjTGlNSRs6yzFNITgjSMm9Jr2QuotJHIvE_medium.webp","\/storage\/images\/resized\/oZgeErrVFhuDksyqFURLvYS1wtVSBWczh001igGo_medium.webp","\/storage\/images\/resized\/voXLqlsvTwv5p3iMQ8Dhs95nqB4AXOG7Taj7G4ra_medium.webp"],"datasets":[{"label":"","data":[3,1,1,1,1,1],"backgroundColor":["#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6"],"percentage":[30,10,10,10,10,10],"barThickness":13}]},"options":{"indexAxis":"y","maintainAspectRatio":false,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":null,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Journals","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}},"publishers":{"type":"bar","data":{"show":true,"labels":["Springer Nature","Hindawi Limited","Pleiades Publishing"],"ids":[8,6921,101],"codes":[0,0,0],"imageUrls":["\/storage\/images\/resized\/voXLqlsvTwv5p3iMQ8Dhs95nqB4AXOG7Taj7G4ra_medium.webp","\/storage\/images\/resized\/hqfzhQAjTGlNSRs6yzFNITgjSMm9Jr2QuotJHIvE_medium.webp","\/storage\/images\/resized\/oZgeErrVFhuDksyqFURLvYS1wtVSBWczh001igGo_medium.webp"],"datasets":[{"label":"","data":[5,2,1],"backgroundColor":["#3B82F6","#3B82F6","#3B82F6"],"percentage":[50,20,10],"barThickness":13}]},"options":{"indexAxis":"y","maintainAspectRatio":false,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":null,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Publishers","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}}}

Metrics

Cite this

GOST |

Cite this

GOST Copy

Skrynnik A. et al. Hierarchical Deep Q-Network from imperfect demonstrations in Minecraft // Cognitive Systems Research. 2021. Vol. 65. pp. 74-78.

GOST all authors (up to 50) Copy

Skrynnik A., Staroverov A., Aitygulov E., Aksenov K., Davydov V., Panov A. Hierarchical Deep Q-Network from imperfect demonstrations in Minecraft // Cognitive Systems Research. 2021. Vol. 65. pp. 74-78.

RIS |

Cite this

RIS Copy

TY - JOUR

DO - 10.1016/j.cogsys.2020.08.012

UR - https://doi.org/10.1016%2Fj.cogsys.2020.08.012

TI - Hierarchical Deep Q-Network from imperfect demonstrations in Minecraft

T2 - Cognitive Systems Research

AU - Skrynnik, Alexey

AU - Staroverov, Aleksei

AU - Aitygulov, Ermek

AU - Aksenov, Kirill

AU - Davydov, Vasilii

AU - Panov, Aleksandr

PY - 2021

DA - 2021/01/01 00:00:00

PB - Elsevier

SP - 74-78

VL - 65

SN - 1389-0417

ER -

BibTex

Cite this

BibTex Copy

@article{2021_Skrynnik,

author = {Alexey Skrynnik and Aleksei Staroverov and Ermek Aitygulov and Kirill Aksenov and Vasilii Davydov and Aleksandr Panov},

title = {Hierarchical Deep Q-Network from imperfect demonstrations in Minecraft},

journal = {Cognitive Systems Research},

year = {2021},

volume = {65},

publisher = {Elsevier},

month = {jan},

url = {https://doi.org/10.1016%2Fj.cogsys.2020.08.012},

pages = {74--78},

doi = {10.1016/j.cogsys.2020.08.012}

}

Found error?

Publisher

Elsevier

Journal

Cognitive Systems Research

Quartile SCImago

Quartile WOS

Impact factor

3.9

ISSN

13890417 (Print)

Labs

MIPT Center for Cognitive Modeling

Profiles