A flexible tool for building and sharing deep learning modules

Nest – A flexible tool for building and sharing deep learning modules

Nest is a flexible deep learning module manager, which aims at encouraging code reuse and sharing. It ships with a bunch of useful features, such as CLI based module management, runtime checking, and experimental task runner, etc. You can integrate Nest with PyTorch, Tensorflow, MXNet, or any deep learning framework you like that provides a python interface.

Moreover, a set of Pytorch-backend Nest modules, e.g., network trainer, data loader, optimizer, dataset, visdom logging, are already provided. More modules and framework support will be added later.

• Prerequisites
• Installation
• Basic Usage
  • Create your first Nest module
  • Use your Nest module in Python
  • Debug your Nest modules
  • Install Nest modules from local path
  • Install Nest modules from URL
  • Uninstall Nest modules
  • Version control Nest modules
  • Use Nest to manage your experiments
• Contact
• Issues
• Contribution
• License

Prerequisites

• System (tested on Ubuntu 14.04LTS, Win10, and MacOS High Sierra)
• Python >= 3.5.4
• Git

Installation

# directly install via pip
pip install git+https://github.com/ZhouYanzhao/Nest.git

# manually download and install
git clone https://github.com/ZhouYanzhao/Nest.git
pip install ./Nest

Basic Usage

The official website and documentation are under construction.

Create your first Nest module

1. Create “hello.py” under your current path with the following content:

   str:
   ” “”my first nest module!””” return ‘hello ‘ + name “>

   from nest import register
   
   @register(author='Yanzhao', version='1.0.0')
   def hello_nest(name: str) -> str:
       """My first Nest module!"""
   
       return 'Hello ' + name

   Note that the type of module parameters and return values must be clearly defined. This helps the user to better understand the module, and at runtime Nest automatically checks whether each module receives and outputs as expected, thus helping you to identify potential bugs earlier.

2. Execute the following command in your shell to verify the module:

   
   str # Documentation: # My first Nest module! # author: Yanzhao # module_path: /Users/yanzhao/Workspace/Nest.doc # version: 1.0.0 “>

   $ nest module list -v
   # Output:
   # 
   # 1 Nest module found.
   # [0] main.hello_nest (1.0.0) by "Yanzhao":
   #     hello_nest(
   #         name:str) -> str
   
   # Documentation:
   #     My first Nest module!
   #     author: Yanzhao
   #     module_path: /Users/yanzhao/Workspace/Nest.doc
   #     version: 1.0.0 

   Note that all modules under current path are registered under the “main” namespace.

   With the CLI tool, you can easily manage Nest modules. Execute nest -h for more details.

3. That’s it. You just created a simple Nest module!

Use your Nest module in Python

1. Open an interactive python interpreter under the same path of “hello.py” and run following commands:

   >> from nest import modules
   >>> print(modules.hello_nest) # access the module
   # Output:
   #
   # hello_nest(
   # name:str) -> str
   >>> print(modules[‘*_nes?’]) # wildcard search
   # Output:
   #
   # hello_nest(
   # name:str) -> str
   >>> print(modules[‘r/main.\w+_nest’]) # regex search
   # Output:
   #
   # hello_nest(
   # name:str) -> str
   >>> modules.hello_nest(‘Yanzhao’) # use the module
   # Output:
   #
   # ‘Hello Yanzhao’
   >>> modules.hello_nest(123) # runtime type checking
   # Output:
   #
   # TypeError: The param ” name” of nest module “hello_nest” should be type “str”. got “123”.>
   >> modules.hello_nest(‘Yanzhao’, wrong=True) # Output: # # Unexpected param(s) “wrong” for Nest module: # hello_nest( # name:str) -> str “>

   >>> from nest import modules
   >>> print(modules.hello_nest) # access the module
   # Output:
   # 
   # hello_nest(
   # name:str) -> str
   >>> print(modules['*_nes?']) # wildcard search
   # Output:
   # 
   # hello_nest(
   # name:str) -> str
   >>> print(modules['r/main.\w+_nest']) # regex search
   # Output:
   # 
   # hello_nest(
   # name:str) -> str
   >>> modules.hello_nest('Yanzhao') # use the module
   # Output:
   #
   # 'Hello Yanzhao'
   >>> modules.hello_nest(123) # runtime type checking
   # Output:
   #
   # TypeError: The param "name" of Nest module "hello_nest" should be type of "str". Got "123".
   >>> modules.hello_nest('Yanzhao', wrong=True)
   # Output:
   #
   # Unexpected param(s) "wrong" for Nest module:
   # hello_nest(
   # name:str) -> str

   Note that Nest automatically imports modules and checks them as they are used to make sure everything is as expected.

2. You can also directly import modules like this:

   >> from nest.main.hello import hello_nest
   >>> hello_nest(‘World’)
   # Output:
   #
   # ‘Hello World’
   “>

   >>> from nest.main.hello import hello_nest
   >>> hello_nest('World')
   # Output:
   #
   # 'Hello World'

   The import syntax is from nest.

   .

   import

3. Access to Nest modules through code is flexible and easy.

Debug your Nest modules

1. Open an interactive python interpreter under the same path of “hello.py” and run following commands:

   >> from nest import modules
   >>> modules.hello_nest(‘Yanzhao’)
   # Output:
   #
   # ‘Hello Yanzhao’
   “>

   >>> from nest import modules
   >>> modules.hello_nest('Yanzhao')
   # Output:
   #
   # 'Hello Yanzhao'

2. Keep the interpreter OPEN and use an externel editor to modify the “hello.py”:

   # change Line7 from "return 'Hello ' + name" to
   return 'Nice to meet you, ' + name

3. Back to the interpreter and rerun the same command:

   >> modules.hello_nest(‘Yanzhao’)
   # Output:
   #
   # ‘Nice to meet you, Yanzhao’
   “>

   >>> modules.hello_nest('Yanzhao')
   # Output:
   #
   # 'Nice to meet you, Yanzhao'

   Note that Nest detects source file modifications and automatically reloads the module.

4. You can use this feature to develop and debug your Nest modules efficiently.

Install Nest modules from local path

1. Create a folder my_namespace and move the hello.py into it:

   $ mkdir my_namespace
   $ mv hello.py ./my_namespace/

2. Create a new file more.py under the folder my_namespace with the following content:

   int:
   ” “”sum two numbers.””” return a + b # there is no need to repeatedly declare meta information as modules within the same file automatically reuse previous information. but overriding also supported. @register(version=”2.0.0″ ) def mul(a: float, b: float) ->
   float: “””Multiply two numbers.””” return a * b “>

   from nest import register
   
   @register(author='Yanzhao', version='1.0.0')
   def sum(a: int, b: int) -> int:
       """Sum two numbers."""
   
       return a + b
   
   # There is no need to repeatedly declare meta information
   # as modules within the same file automatically reuse the 
   # previous information. But overriding is also supported.
   @register(version='2.0.0')
   def mul(a: float, b: float) -> float:
       """Multiply two numbers."""
       
       return a * b

   Now we have:

   current path/
   ├── my_namespace/
   │   ├── hello.py
   │   ├── more.py
   

3. Run the following command in the shell:

   
   Search paths. Continue? (Y/n) [Press
   ] “>

   $ nest module install ./my_namespace hello_word
   # Output:
   #
   # Install "./my_namespace/" -> Search paths. Continue? (Y/n) [Press 
         
          ]
         

   This command will add “my_namespace” folder to Nest’s search path, and register all Nest modules in it under the namespace “hello_word“. If the last argument is omitted, the directory name, “my_namespace” in this case, is used as the namespace.

4. Verify the installation via CLI:

   $ nest module list
   # Output:
   #
   # 3 Nest modules found.
   # [0] hello_world.hello_nest (1.0.0)
   # [1] hello_world.mul (2.0.0)
   # [2] hello_world.sum (1.0.0)

   Note that those Nest modules can now be accessed regardless of your working path.

5. Verify the installation via Python interpreter:

   $ ipython # open IPython interpreter

   >> from nest import modules
   >>> print(len(modules))
   # Output:
   #
   # 3
   >>> modules.[Press ] # IPython Auto-completion
   # Output:
   #
   # hello_nest
   # mul
   # sum
   >>> modules.sum(3, 2)
   # Output:
   #
   # 5
   >>> modules.mul(2.5, 4.0)
   # Output:
   #
   # 10.0
   “>

   >>> from nest import modules
   >>> print(len(modules))
   # Output:
   #
   # 3
   >>> modules.[Press <Tab>] # IPython Auto-completion
   # Output:
   #
   # hello_nest
   # mul
   # sum
   >>> modules.sum(3, 2)
   # Output:
   #
   # 5
   >>> modules.mul(2.5, 4.0)
   # Output:
   #
   # 10.0

6. Thanks to the auto-import feature of Nest, you can easily share modules between different local projects.

Install Nest modules from URL

1. You can use the CLI tool to install modules from URL:

   # select one of the following commands to execute
   # 0. install from Github repo via short URL (GitLab, Bitbucket are also supported)
   $ nest module install [email protected]/Nest:pytorch pytorch
   # 1. install from Git repo
   $ nest module install "-b pytorch https://github.com/ZhouYanzhao/Nest.git" pytorch
   # 2. install from zip file URL
   $ nest module install "https://github.com/ZhouYanzhao/Nest/archive/pytorch.zip" pytorch

   The last optional argument is used to specify the namespace, “pytorch” in this case.

2. Verify the installation:

   $ nest module list
   # Output:
   #
   # 26 Nest modules found.
   # [0] hello_world.hello_nest (1.0.0)
   # [1] hello_world.mul (2.0.0)
   # [2] hello_world.sum (1.0.0)
   # [3] pytorch.adadelta_optimizer (0.1.0)
   # [4] pytorch.checkpoint (0.1.0)
   # [5] pytorch.cross_entropy_loss (0.1.0)
   # [6] pytorch.fetch_data (0.1.0)
   # [7] pytorch.finetune (0.1.0)
   # [8] pytorch.image_transform (0.1.0)
   # ...

Uninstall Nest modules

1. You can remove modules from Nest’s search path by executing:

   # given namespace
   $ nest module remove hello_world
   # given path to the namespace
   $ nest module remove ./my_namespace/

2. You can also delete the corresponding files by appending a --delete or -d flag:

   $ nest module remove hello_world --delete

Version control Nest modules

1. When installing modules, Nest adds the namespace to its search path without modifying or moving the original files. So you can use any version control system you like, e.g., Git, to manage modules. For example:

   <div class="highlight highlight-source-shell position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cd
   # update modules
   $ git pull
   # specify version
   $ git checkout v1.0
   “>

   $ cd <path of the namespace>
   # update modules
   $ git pull
   # specify version
   $ git checkout v1.0