Magic Carpet

Magic Carpet is an infrastructure as code and network automation tool that transforms CLI command and REST API data, using the Cisco Genie parsers, the Cisco pyATS Python library, and Python to automatically generate, at scale, better documentation from the output; send #chatbots; #voicebots; even #phonebots!

Getting Started

Requirements (instructions below)

Virtual Environment

We recommend running Magic Carpet in a Python virtual environment while testing or developing. This will help keep your host system clean and allow you to have multiple environments to try new things. If you are not using a virtual environment, start at the download/clone step below.

You will also need Python 3, pip, and venv installed on your host system.

In your project directory, create your virtual environment

python3 -m venv env

Activate (use) your new virtual environment (Linux):

source env/bin/activate

Download or clone the Magic Carpet repository:

git clone https://github.com/automateyournetwork/magic_carpet

Install pyATS, Rich, markmap, and netjsongraph into your environment:

pip install pyats[full]
pip install rich
sudo apt update
sudo apt install npm
sudo npm install markmap-cli -g
npm install netjsongraph.js --save

If you run into any installation issues with pyATS, please see the installation guide here: https://pubhub.devnetcloud.com/media/pyats-getting-started/docs/install/installpyATS.html


You're now ready to fly!

When you are finished with your Magic Carpet session, enter the deactivate command to exit the python virtual environment.

Riding the Magic Carpet

How to update the appropriate testbed/testbed_*.yaml file to reflect your device inventory and configuration:

testbed_routers

Devices:

    4500: <-- Update to your router hostname (MUST MATCH HOSTNAME)

        alias: <-- Update your alias; this can be the hostname or any friendly name you want

        type: <-- This should be "router" for L3 routers with ARP tables and VRFs and such 

        platform: <-- Your Catalyst IOS-XE Platform

        username: <-- Your Cisco device username

        password: <-- Your Enable Secret; It is strongly recommended to follow the "Secret Strings" link to encrypt your secrets

        ip: <-- The management IP address of your router 

    Copy & Paste and make each device unique to scale this up to many routers

testbed_switches

Devices:

    3850: <-- Update to your switch hostname (MUST MATCH HOSTNAME)
    9300: <-- Update to your switch hostname (MUST MATCH HOSTNAME)

        alias: <-- Update your alias; this can be the hostname or any friendly name you want

        type: <-- This should be "switch" for L2 switches without ARP tables or VRFs but features like PoE 

        platform: <-- Your Catalyst IOS-XE Platform

        username: <-- Your Cisco device username

        password: <-- Your Enable Secret; It is strongly recommended to follow the "Secret Strings" link to encrypt your secrets

        ip: <-- The management IP address of your router 

    Copy & Paste and make each device unique to scale this up to many routers    

Take the Magic Carpet to the Cloud!

Magic Carpet now connects to the following Cisco.com APIs:

  • Recommended Software Release (using the real version of software from your devices)

  • Serial Number 2 Infomation (using the real serial numbers, for every part, from your devices)

  • Product Security Incident Response Team (PSIRT) (using the real version of software from your device)

Pro tip:

You can create a testbed file using an Excel spreadsheet (see testbed/SampleTestbedFile.xlsx), this is useful if you are not familiar with YAML and/or have a long list of devices where copying and pasting in Excel would be easier.

  1. Edit the relevant fields in the sample file.
  2. Run the following command:
pyats create testbed file --path SampleTestbedFile.xlsx --output my_testbed.yaml
  1. That's it. You're good to go!

Ensure SSH connectivity at from your host's CLI, and run the pyATS job:

Cisco IOS-XE:

pyats run job IOS_XE_magic_carpet_job.py --testbed-file testbed/testbed_ios_xe.yaml

Cisco DevNet Sandbox for CSR1000v (see more here)

pyats run job DevNet_Sandbox_CSR_magic_carpet_job.py --testbed-file testbed/testbed_DevNet_CSR1000v_Sandbox.yaml

JunOS 17 / 18 / 19:

pyats run job JUNOS_magic_carpet_job.py --testbed-file testbed/testbed_juniper.yaml

F5 BIG-IP:

python3 F5_magic_carpet.py

First - you will get onto the Magic Carpet

Hang_On-1

Next - Genie Magic

Heading_In

Finally - We escape the Cave of Wonders with the network data

We_Made_It

cd Cave_of_Wonders

ls 

To view the pyATS log in a web browser Locally

pyats logs view

To view the pyATS log in a web browser remotely

pyats logs view --host 0.0.0.0 --port 8080 -v

pyATS_Log_Viewer

Explore your Wonders!

Here is an example of just one of the Wonders you will find: the show ip route command!

Here is what a Global Routing Table looks like in JSON:

CaveOfWonders_IP_Route_JSON

The same routing table, but in YAML:

CaveOfWonders_IP_Route_YAML

The JSON and YAML outputs are incredible representations of the routing table and can be used for further pyATS testing or data modeling.

"Business-ready" documentation includes the incredibly powerful and versatile Comma-Separated Values (csv) spreadsheet format.

CaveOfWonders_IP_Route_CSV

Markdown, the format this README file is written in, can also be used to express the data in a lightweight format that renders nicely in modern browsers.

CaveOfWonders_IP_Route_MD

What about a full-blown HTML webpage? Magic Carpet also creates at least one of these per command.

CaveOfWonders_IP_Route_HTML_Raw

Which renders nicely like this in your browser:

CaveOfWonders_IP_Route_HTML_Rendered

Another HTML page, an interactive mindmap, is also created from the Markdown file!

CaveOfWonders_IP_Route_Mind_Map

Another Interactive HTML page, this time a NETJSON NetGraph, is also crteated

CaveOfWonders_IP_Route_netgraph

To launch a Python web server and make the Cave of Wonders available in a browser, where you can view the HTML pages:

Launch a local web server available on the same host:

cd Cave_of_Wonders
pushd;  python3 -m http.server --bind 127.0.0.1 8888; popd;

Launch your web browser and visit

http://127.0.0.1:8888/

Launch a local web server available to remote hosts:

cd Cave_of_Wonders
pushd;  python3 -m http.server --bind 0.0.0.0 8888; popd;

Launch your web browser and visit

http://{{ your server IP }}:8888/

Sharepoint 2016 Plug-in

Every artifact can also be uploaded dynamically to a Sharepoint 2016 Document Library !

The CSV files work with the Office extension and present as version controlled Excel-online files!

Add the code from the SharePoint2016_Plugin.py file. The show version command is used in the example.

Simply add the SHAREPOINT code, updating to reflect the appropriate command:

#Enter your SharePoint site and target library
sharePointUrl = 'https://mydomain.com'
folderUrl = '/DocumentLibrary/Cave_of_Wonders/'

    ##############
    # SHAREPOINT #
    ##############

    #Read filename (relative path) from command line
    sharePointFileName = "Cave_of_Wonders/Cisco/IOS_XE/Show_Version/%s_show_version.csv" % device.alias
    uploadFileName = "%s_show_version.csv" % device.alias     

   # Execute a request to get the FormDigestValue. This will be used to authenticate our upload request
    r = requests.post(sharePointUrl + "/_api/contextinfo",auth=HttpNtlmAuth('DOMAIN\\USERNAME','PASSWORD'), headers=headers)

    # Execute the request. If you run into issues, inspect the contents of uploadResult
    uploadResult = requests.post(requestUrl,auth=HttpNtlmAuth('DOMAIN\\USERNAME','PASSWORD'), headers=headers, data=file.read())

GitHub

https://github.com/automateyournetwork/magic_carpet