Format of vendor and supplemental data

You can provide two types of data to Batfish: 1) configurations of devices in your network, and 2) supplemental data to enhance the model of your network. This page describes the format of these data.

Device configuration formats

Batfish supports the following vendors. Click on the corresponding link to learn how to provide configuration for a specific vendor.

Except for AWS, all vendor configs files must be placed in the configs folder right below the top-level snapshot folder. It is OK to create sub-folders inside configs; Batfish will recursively read all files. It is also OK to mix files from multiple vendors in the same folder.

Note about vendor detection

Batfish can automatically detect the vendor of a configuration file based on certain tell-tale signs in vendor files. For instance, Arista config files tend to contain lines with “! device: … EOS-4.24” or “boot system flash … swi”. Such lines are almost always present in config files pulled from devices but may not be present in auto-generated files.

If you need to explicitly specify the vendor of your configuration file, include the RANCID content type header. So, for Arista devices, you’d include the following line at the top of the file:

!RANCID-CONTENT-TYPE: arista

The supported vendor type strings are arista, bigip (F5), cisco-nx (NX-OS), cisco-xr (IOS-XR), force10 (Dell), foundry, juniper (all JunOS), mrv, and paloalto.

Arista

For each Arista device in the network, create a file in the configs folder. The content of the file should be the equivalent of the output of show running-config command on the device.

AWS

Batfish understands AWS VPC configurations and analyzes it just like physical networks. To use this functionality, place AWS configs in a folder named aws_configs right below the top-level snapshot folder. The subfolders in this folder correspond to individual regions. If there is only one region, then this level of hierarchy may be skipped.

The configuration files for a region should be the JSON output of the following API calls:

  • For EC2: describe_addresses, describe_availability_zones, describe_customer_gateways, describe_internet_gateways, describe_network_acls, describe_network_interfaces, describe_instances, describe_route_tables, describe_security_groups, describe_subnets, describe_vpc_endpoints, describe_vpcs, describe_vpn_connections, describe_vpn_gateways

  • For ES: describe_elasticsearch_domains

  • For RDS: describe_db_instances

This output can be collected using the AWS CLI or the boto3 Python SDK. An example script that packages AWS data into a Batfish snapshot is here.

An example snapshot, which includes both physical and AWS configs, is here. It is OK to have only AWS configs in a snapshot (without any physical device configs).

Cisco

Batfish supports Cisco IOS, IOS-XE, IOS-XR, NX-OS, and ASA platforms. For each such device in the network, create a file in the configs folder. The content of the file should be the equivalent of the output of show running-config command on the device.

For NX-OS, we recommend using show run all where possible. This output has additional detail that helps with modeling accuracy.

Check Point

Check Point data is provided in two parts.

  1. Place the output of show configuration from gateway devices into the configs folder. This data contains information about interfaces and routing but not firewall policies, which must be fetched from the Check Point Manager as described next.

  2. Data from Check Point Manager must be placed in a folder called checkpoint_management right under the top-level snapshot folder (parallel to configs folder). The expected hierarchy under this folder is as follows:

  • checkpoint_management

    • Manager1 [Data from Manager1–you may have multiple managers]

      • DomainA [Data from DomainA–the manager may have multiple]

        • show-gateways-and-servers.json

        • show-groups.json

        • show-hosts.json

        • show-networks.json

        • show-package.json

        • show-service-groups.json

        • show-services-icmp.json

        • show-services-other.json

        • show-services-tcp.json

        • show-services-udp.json

        • Package1 [Data for Package1–the domain may have multiple packages]

          • show-access-rulebase.json

          • show-nat-rulebase.json

          • show-package.json

The json files contain the output of corresponding Check Point Management API calls.

An example script that fetches this data and organizes it as above is here.

Cumulus Linux

Cumulus devices can be configured by editing individual files, such as /etc/network/interfaces, /etc/cumulus/ports.conf, and /etc/frr/frr.conf or invoking the Network Command-Line Utility (NCLU)

Batfish supports both the NCLU format and individual Cumulus configuration files. We recommend using the configuration files because Batfish can extract more data from them than from the NCLU output.

Note

If you are using the BGP Unnumbered feature on Cumulus devices, you will need to supply Layer-1 topology.

F5 BIG-IP

F5 BIG-IP configuration spans multiple individual files, typically the following 3 configuration files:

  • Base

  • LTM

  • Routing [There are 2 formats for this routing configuration. Legacy imish and the newer structured format]

These files are typically in a folder named “config” as:

  • Base: bigip_base.conf

  • LTM: bigip.conf

  • Routing: bigip_routing.conf

For Batfish to correctly process the F5 configuration, the 3 files need to be combined into a single file in a specific order.

As an example, let’s say you have these 3 files:

  • site1-f5-a-base.conf

  • site1-f5-a.conf

  • site1-f5-a-routing.conf

On any unix machine, simply run:

cat site1-f5-a-base.conf site1-f5-a.conf site1-f5-a-routing.conf > site1-f5-a-concat.cfg

Add site1-f5-a-concat.cfg to the configs folder with the rest of the devices.

Note

The routing configuration MUST be the last thing copied into the file, otherwise Batfish will not be able to correctly parse the file.

Fortinet

For each FortiOS firewall, create a file in the configs folder. The content of the file should be the equivalent of the output of show command on the device.

Juniper

Batfish supports all JunOS-based platforms, including EX, MX, PTX, QFX, SRX, and the T-series. For each such device in the network, create a file in the configs folder. The content of the file should be the equivalent of the output of show configuration | display set command on the device.

Batfish will also accept JunOS configuration in the hierarchical format and will internally pre-process files in this format to the “set” format.

Palo Alto Networks

Batfish supports Palo Alto Networks devices with or without Panorama.

From Panorama (preferred)

For devices managed through Panorama, with no configuration done directly on the device, you can pull configurations from Panorama without having to run any commands on the managed-devices themselves. If configuration is also done directly on the managed-device, you should follow the instructions for pulling configuration From individual devices instead.

To pull these configurations through Panorama, use the pan-python module as follows; note: replace <IP_ADDR> and <LOGIN_USERNAME> with the IP address and username for the Panorama device:

# Generate an API key and save in .panrc so subsequent commands don't require logging in again
# This only needs to be done once
panxapi.py -t panorama_tag -h <IP_ADDR> -l <LOGIN_USERNAME> -k >> .panrc
# Pull the XML Panorama config
panxapi.py -t panorama_tag -sxr > panorama_config.xml
# Convert the XML Panorama config into set-format
panconf.py --config pan-panorama_config.xml  --set  > panorama_config.set.txt

This will generate a single configuration (panorama_config.set.txt) representing all firewalls managed by the specified Panorama device, and this config file is what Batfish needs to model the managed firewalls. Place this file in the configs folder.

From individual devices

For each device, concatenate the following show commands into one file and put the file in the configs folder.

set cli config-output-format set
set cli pager off
show config pushed-shared-policy
show config pushed-shared-policy vsys <value> // run for each vsys
show config merged

The first two commands may not be available on all PAN-OS versions; just make sure that the output is NOT in XML format (first command) and that definitions are not truncated (second command).

Supplemental data formats

You can provide additional data to Batfish to enhance the model of your network and to model parts of the network whose configuration is not available. The following types of data is supported.

Modeling hosts

You can model end hosts in the network by adding host files with information about their names, a pointer to their iptables configuration file, and their interfaces. An example host file is:

{
  "hostname" : "host1",
  "iptablesFile" : "iptables/host1.iptables",
  "hostInterfaces" : {
    "eth0" : {
      "name": "eth0",
      "prefix" : "2.128.0.101/24"
    }
  }
}

iptables/host1.iptables is the path relative to the snapshot where this host’s iptables configuration can be found. iptables configuration files should be in the format that is generated by iptables-save.

There should be one such file per host that you want to model and these files should be placed in a folder called hosts right below the top-level snapshot folder.

See this example snapshot with host files.

Layer-1 topology

Batfish can infer Layer-3 interface adjacencies based on IP address configuration on interfaces. For instance, if there are two interfaces in the network with IP assignments 192.168.1.1/24 and 192.128.1.2/24, Batfish will infer that these interfaces are adjacent.

Such inference does not work if the network re-uses IP address space or has link-local addresses. In those situations, you must provide a Layer-1 topology file that has cabling information. Then, Layer-3 adjacencies will be computed by combining the supplied Layer-1 adjacencies with Layer-2 and Layer-3 configuration to get a more accurate model.

The expected Layer-1 topology file is a JSON file that has a list of edge records, where each edge record has node and interface names of the two ends. See this file for an example.

The name of your Layer-1 topology file must be layer1_topology.json and it must be placed in a folder called batfish right below the top-level snapshot folder.

Modeling ISPs

Batfish can model ISPs (and Internet connectivity) for a network based on a file that specifies which ISPs to model and how to configure them. The ISP modeling file has a few different sections that control different aspects of the modeling.

  • BorderInterfaces: Specify the interfaces in the snapshot that connect to the ISP you want to model. Each interface must be a layer-3 interface connected directly to the ISP and establish an eBGP single-hop session to the ISP.

{
  "borderInterfaces": [
    {
      "borderInterface": {
        "hostname": "as2border1",
        "interface": "GigabitEthernet3/0"
      }
    }, 
    ....
  ]
}

  • BgpPeers: Specify the BGP peers defined on border routers in the snapshot. You can also optionally specify how the ISP attaches to the snapshot, by specifying the interface in the snapshot where it establishes physical connectivity. In the absence of this attachment specification, Batfish will assume that the ISP is attached at the interface of the specified peering.

{
  "bgpPeers": [
    {
      "hostname": "as2border1",
      "peerAddress": "10.10.10.10",
      "vrf": "internet",   // Optional; default VRF is assumed if left unspecified
      "ispAttachment": {   // This section is optional. 
          "hostname": "borderSwitch", // Optional; the host of the peering ('as2border1') is assumed if left unspecified
          "interface": "GigabitEthernet3/0",  // mandatory
          "vlanTag": 86 // Specifies the Dot1q encapsulation that the modeled ISP should use for the peering. 
                        // Optional; no tagging is assumed if left unspecified
      }
    }, 
    ....
  ]
}

  • Filter: This section allows you restrict ISPs that are modeled, by specifying the ASNs (onlyRemoteAsns) and IPs (onlyRemoteIps) of the ISPs. Without this section, all ISPs specified by BorderInterfaces and BgpPeers sections.

{
  "filter": {
    "onlyRemoteAsns": [65432],
    "onlyRemoteIps": ["10.10.10.10"]
  }
}

  • IspNodeInfos: This section lets you configure certain aspects of the modeled ISP nodes. One aspect of this configuration is specifying the role of the ISP—whether it is a transit provider (TRANSIT) or a private backbone (PRIVATE_BACKBONE) such as a carrier MPLS or your own wide-area network that connects multiple sites. Transit ISPs connects to the modeled Internet node, do not propagate any communities and block reserved address space traffic. Private backbones do not connect to the Internet, propagate (standard and extended) communities and do not filter any traffic.

{
  "ispNodeInfo": [
    {
      "asn": 65432,
      "name": "ATT", // Specifies the friendly name (not hostname) to use for the ISP
      "role": "PRIVATE_BACKBONE"  // Optional; default is "TRANSIT"
    },
    ...
   ]
}

  • IspPeerings: This section lets you configure eBGP peering between modeled ISP nodes. When not present, none of the ISPs connect to each other. The specification is:

{
   "ispPeerings": [
     {
        "peer1": {
           "asn": 65432,
        },
        "peer2": {
           "asn": 64325,
        }
     },
     ...
   ]
}

All sections that you need must be included in a file called isp_config.json, placed in a folder called batfish right below the top-level snapshot folder. An example snapshot with ISP modeling file is here.

The hostname of the modeled ISP nodes will be of the form isp_<ASN> (e.g., ‘isp_65432’). These nodes can be queried and analyzed just like any other node in the snapshot. Only one node per ASN is generated even if you have multiple peerings to the same ASN.

A node representing the Internet is also created if any of the ISPs is “TRANSIT”. This node announces the default route (0.0.0.0/0) to connected ISPs. Its hostname is ‘internet’ and it too can be queried and analyzed like any other node in the snapshot.

Runtime interface information

Batfish infers interface attributes from static configuration files. All relevant information, however, may not be present in the configuration files, including whether an interface is line down and speed/bandwidth for some vendors. You can enhance Batfish’s inference of such properties by providing runtime data.

The format of this data is

{
  "runtimeData": {
    "router1": {
      "Ethernet1/1": {
         "bandwidth": 100000000000,
         "lineUp": false,
         "speed": 100000000000
      }
    },
    "router2": {
      "Ethernet21/1": {
         "lineUp": true
      }
    }    
  }
}

The name of this must be runtime_data.json and it must be placed in a folder called batfish right below the top-level snapshot folder. The same file should contain information for all devices and interfaces, and only a subset of the properties may be specified for an interface.

The runtime data specification also allows you to analyze the impact of failing certain interfaces in the network—just mark them as line down in the data.

External BGP announcements

To help analyze connectivity to specific external prefixes or analyze the treatment of external routes by your network, you can specify the external routes coming into the network, as follows.

{
  "Announcements": [
      {
          "type" : "ebgp_sent", // whether the attributes of this route are what was sent by the external peer, 
                                // or they represent attributes after processing by import routing policy ('ebgp_received')
          "network" : "4.0.0.0/8",
          "nextHopIp" : "10.14.22.4",
          "srcIp" : "10.14.22.4",
          "dstNode" : "as1border2",
          "dstIp" : "10.14.22.1",
          "srcProtocol" : "AGGREGATE",
          "originType" : "egp",
          "localPreference" : 0,
          "med" : 20,
          "originatorIp" : "0.0.0.0",
          "asPath" : [ [ 1239 ], [7018, 7019]],
          "communities" : [ 262145 ],
          "dstVrf":"default",
          "clusterList":[]
      },
      ...
   ]
}

These announcements must be placed in the file called external_bgp_announcements.json and placed inside the top-level snapshot folder.