RecNN

This is my school project. It focuses on Reinforcement Learning for personalized news recommendation. The main distinction is that it tries to solve online off-policy learning with dynamically generated item embeddings. I want to create a library with SOTA algorithms for reinforcement learning recommendation, providing the level of abstraction you like.

All in all the features can be summed up to:

  • Abstract as you decide: you can import the entire algorithm (say DDPG) and tell it to ddpg.learn(batch), you can import networks and the learning function separately, create a custom loader for your task, or can define everything by yourself.

  • Examples do not contain any of the junk code or workarounds: pure model definition and the algorithm itself in one file. I wrote a couple of articles explaining how it functions.

  • The learning is built around sequential or frame environment that supports ML20M and like. Seq and Frame determine the length type of sequential data, seq is fully sequential dynamic size, while the frame is just a static frame.

  • State Representation module with various methods. For sequential state representation, you can use basic LSTM/RNN/GRU,
    Temporal Convolutional Networks, Echo State Networks and Chaos Free RNNs that are way faster than GRU.

  • Pytorch 1.3 support with Tensorboard visualization.

  • New datasets will be added in the future.

  • SOTA optimizers (RAdam, LookAhead, Ranger) come pre-packaged.

Medium Articles

The repo consists of two parts: the library (./recnn) and the playground (./examples) where I explain how to work with certain things.

  • Pretty much what you need to get started with this library if you know recommenders but don't know much about
    reinforcement learning:

article_1

  • Top-K Off-Policy Correction for a REINFORCE Recommender System:
    article_2

Algorithms that are/will be added:

Algorithm Paper Code
Deep Q Learning (PoC) https://arxiv.org/abs/1312.5602 examples/0. Embeddings/ 1.DQN
Deep Deterministic Policy Gradients https://arxiv.org/abs/1509.02971 examples/1.Vanilla RL/DDPG
Twin Delayed DDPG (TD3) https://arxiv.org/abs/1802.09477 examples/1.Vanilla RL/TD3
Soft Actor-Critic https://arxiv.org/abs/1801.01290 examples/1.Vanilla RL/SAC
Batch Constrained Q-Learning https://arxiv.org/abs/1812.02900 examples/99.To be released/BCQ
REINFORCE Top-K Off-Policy Correction https://arxiv.org/abs/1509.02971 examples/2. REINFORCE TopK


My Trello with useful papers


Repos I used code from:

What is this?

This is my school project. It focuses on Reinforcement Learning for personalized news recommendation. The main distinction is that it tries to solve online off-policy learning with dynamically generated item embeddings. Also, there is no exploration, since we are working with a dataset. In the example section, I use Google's BERT on the ML20M dataset to extract contextual information from the movie description to form the latent vector representations. Later, you can use the same transformation on new, previously unseen items (hence, the embeddings are dynamically generated). If you don't want to bother with embeddings pipeline, I have a DQN embeddings generator as a proof of concept.

portfolio

Getting Started

code

get_started

p.s. Image is clickable. here is direct link:


To learn more about recnn, read the docs: recnn.readthedocs.io

Installing

pip install git+git://github.com/awarebayes/RecNN.git

or

git clone https://github.com/awarebayes/RecNN
pip install ./RecNN

Downloads

Models

Citing

If you find RecNN useful for an academic publication, then please use the following BibTeX to cite it:

@misc{RecNN,
  author = {M Scherbina},
  title = {RecNN: RL Recommendation with PyTorch},
  year = {2018},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/awarebayes/RecNN}},
}

GitHub