Shape optimization problems involving nonlocal and nonlinear operators

In this research, we investigate a general shape optimization problem in which the state equation is expressed using a nonlocal and nonlinear operator. We prove the existence of a minimum point for a functional F defined on the family of all ’quasi-open’ subsets of a bounded open set $$\Omega $$ in $$\mathbb {R}^n$$ . This is ensured under the condition that F demonstrates decreasing behavior concerning set inclusion and is lower semicontinuous with respect to a suitable topology associated with the fractional p-Laplacian under Dirichlet boundary conditions. Moreover, we study the asymptotic behavior of the solutions when $$s\rightarrow 1$$ and extend this result to the anisotropic case.