{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Analyzing public cloud and hybrid networks\n", "\n", "Public cloud and hybrid networks can be hard to debug and secure. Many of the standard tools (e.g., traceroute) do not work in the cloud setting, even though the types of paths that can emerge are highly complicated, depending on whether the endpoints are in the same region, different regions, across physical and virtual infrastructure, or whether public or private IPs of cloud instances are being used. \n", "\n", "At same time, the fast pace of evolution of these networks, where new subnets and instances can be spun up rapidly by different groups of people, creates a significant security risk. Network engineers need tools than can provide comprehensive guaratnees that services and applications are available and secure as intended at all possible times.\n", "\n", "In this notebook, we show how Batfish can predict and help debug network paths for cloud and hybrid networks and how it can guarantee that the network's availability and security posture is exactly as desired." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Import packages\n", "%run startup.py\n", "bf = Session(host=\"localhost\")\n", "\n", "\n", "def show_first_trace(trace_answer_frame):\n", " \"\"\"\n", " Prints the first trace in the answer frame.\n", " \n", " In the presence of multipath routing, Batfish outputs all traces \n", " from the source to destination. This function picks the first one.\n", " \"\"\"\n", " if len(trace_answer_frame) == 0:\n", " print(\"No flows found\")\n", " else:\n", " show(\"Flow: {}\".format(trace_answer_frame.iloc[0]['Flow']))\n", " show(trace_answer_frame.iloc[0]['Traces'][0])\n", " \n", "def is_reachable(start_location, end_location, headers=None):\n", " \"\"\"\n", " Checks if the start_location can reach the end_location using specified packet headers.\n", " \n", " All possible headers are considered if headers is None. \n", " \"\"\"\n", " ans = bf.q.reachability(pathConstraints=PathConstraints(startLocation=start_location, \n", " endLocation=end_location),\n", " headers=headers).answer()\n", " return len(ans.frame()) > 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Initializing the Network and Snapshot\n", "\n", "SNAPSHOT_PATH below can be updated to point to a custom snapshot directory. See [instructions](https://github.com/batfish/batfish/wiki/Packaging-snapshots-for-analysis) for how to package data for analysis. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'snapshot'" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Initialize a network and snapshot\n", "NETWORK_NAME = \"hybrid-cloud\"\n", "SNAPSHOT_NAME = \"snapshot\"\n", "\n", "SNAPSHOT_PATH = \"networks/hybrid-cloud\"\n", "\n", "bf.set_network(NETWORK_NAME)\n", "bf.init_snapshot(SNAPSHOT_PATH, name=SNAPSHOT_NAME, overwrite=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The network snapshot that we just initialized is illustrated below. It has a datacenter network with the standard leaf-spine design on the left. Though not strictly necessary, we have included a host `srv-101` in this network to enable end-to-end analysis. The exit gateway of the datacenter connects to an Internet service provider (ASN 65200) that we call `isp_dc`.\n", "\n", "The AWS network is shown on the right. It is spread across two regions, `us-east-2` and `us-west-2`. Each region has two VPCs, one of which is meant to host Internet-facing services and the other is meant to host only private services. Subnets in the public-facing VPCs use an Internet gateway to send and receive traffic outside of AWS. The two VPCs in a region peer via a transit gateway. Each VPC has two subnets, and we have some instances running as well. \n", "\n", "The physical network connects to the AWS network using IPSec tunnels, shown in pink, between `exitgw` and the two transit gateways. BGP sessions run atop these tunnels to make endpoints aware of prefixes on the other side.\n", "\n", "You can view configuration files that we used [here](https://github.com/batfish/pybatfish/tree/master/jupyter_notebooks/networks/hybrid-cloud). The AWS portion of the configuration is in the aws_configs subfolder. It has JSON files obtained via AWS APIs. An example script that packages AWS data into a Batfish snapshot is [here](https://github.com/ratulm/bf-aws-snapshot).\n", "\n", "![hybrid-cloud-network](https://raw.githubusercontent.com/batfish/pybatfish/master/jupyter_notebooks/networks/hybrid-cloud/hybrid-cloud.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analyzing network paths\n", "\n", "Batfish can help analyze cloud and hybrid networks by showing how exactly traffic flows (or not) in the network, which can help debug and fix configuration errors. Batfish can also help ensure that the network is configured exactly as desired, with respect to reachability and security policies. We illustrate these types of analysis below.\n", "\n", "First, lets define a couple of maps to help with the analysis." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "#Instances in AWS in each region and VPC type (public, private)\n", "hosts = {}\n", "hosts[\"east2_private\"] = \"i-04cd3db5124a05ee6\"\n", "hosts[\"east2_public\"] = \"i-01602d9efaed4409a\"\n", "hosts[\"west2_private\"] = \"i-0a5d64b8b58c6dd09\"\n", "hosts[\"west2_public\"] = \"i-02cae6eaa9edeed70\"\n", "\n", "#Public IPs of instances in AWS\n", "public_ips = {}\n", "public_ips[\"east2_public\"] = \"13.59.144.125\" # of i-01602d9efaed4409a\n", "public_ips[\"west2_public\"] = \"54.191.42.182\" # of i-02cae6eaa9edeed70" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Paths across VPCs within an AWS region\n", "\n", "To see how traffic flows between two instances in the same region but across different VPCs, say, from `hosts[\"east2_private\"]` to `hosts[\"east2_public\"]`, we can run a traceroute query across them as follows. \n", "\n", "In the query below, we use the name of the instance as the destination for the traceroute. This makes Batfish pick the instance's private (i.e., non-Elastic) IP (`10.20.1.207`). It does not pick the public IP because that those IPs do not reside on instances but are used by the Internet gateway to NAT instance's traffic in and out of AWS (see [documentation](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html)). If an instance has multiple private IPs, Batfish will pick one at random. To make Batfish use a specific IP, supply that IP as the argument to the `dstIps` parameter." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Flow: start=i-04cd3db5124a05ee6 [10.30.1.166:49152->10.20.1.207:22 TCP (SYN)]'" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "ACCEPTED
1. node: i-04cd3db5124a05ee6
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: eni-05452497daf80ccb3 with resolved next-hop IP: 10.30.1.1, Routes: [static (Network: 0.0.0.0/0, Next Hop: ip 10.30.1.1)])
  PERMITTED(~EGRESS_ACL~eni-05452497daf80ccb3 (EGRESS_FILTER))
  SETUP_SESSION(Incoming Interfaces: [eni-05452497daf80ccb3], Action: Accept, Match Criteria: [ipProtocol=TCP, srcIp=10.20.1.207, dstIp=10.30.1.166, srcPort=22, dstPort=49152])
  TRANSMITTED(eni-05452497daf80ccb3)
2. node: subnet-0cb5f4c094bee5214
  RECEIVED(to-instances)
  FORWARDED(Forwarded out interface: vpc-0276455718806058a-vrf-tgw-attach-021f89744fac566dd with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.0.0/16, Next Hop: interface vpc-0276455718806058a-vrf-tgw-attach-021f89744fac566dd ip 169.254.0.1)])
  PERMITTED(acl-0b3d0f6b0978f09f8_egress (EGRESS_FILTER))
  TRANSMITTED(vpc-0276455718806058a-vrf-tgw-attach-021f89744fac566dd)
3. node: vpc-0276455718806058a
  RECEIVED(subnet-0cb5f4c094bee5214-vrf-tgw-attach-021f89744fac566dd)
  FORWARDED(Forwarded out interface: tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 0.0.0.0/0, Next Hop: interface tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03 ip 169.254.0.1)])
  TRANSMITTED(tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03)
4. node: tgw-06b348adabd13452d
  RECEIVED(vpc-0276455718806058a-tgw-rtb-00e37bc5142347b03)
  FORWARDED(Forwarded out interface: vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.0.0/16, Next Hop: interface vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03 ip 169.254.0.1)])
  TRANSMITTED(vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03)
5. node: vpc-0574d08f8d05917e4
  RECEIVED(tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03)
  FORWARDED(Forwarded out interface: subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.1.0/24, Next Hop: interface subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5 ip 169.254.0.1)])
  TRANSMITTED(subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5)
6. node: subnet-06a692ed4ef84368d
  RECEIVED(vpc-0574d08f8d05917e4-vrf-tgw-attach-0648110513acd6de5)
  PERMITTED(acl-09c0bb4e71ae5f9e4_ingress (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: to-instances, Routes: [connected (Network: 10.20.1.0/24, Next Hop: interface to-instances)])
  TRANSMITTED(to-instances)
7. node: i-01602d9efaed4409a
  RECEIVED(eni-01997085076a9b98a)
  PERMITTED(~INGRESS_ACL~eni-01997085076a9b98a (INGRESS_FILTER))
  SETUP_SESSION(Originating VRF: default, Action: ForwardOutInterface(Next Hop: subnet-06a692ed4ef84368d, Next Hop Interface: to-instances, Outgoing Interface: eni-01997085076a9b98a), Match Criteria: [ipProtocol=TCP, srcIp=10.20.1.207, dstIp=10.30.1.166, srcPort=22, dstPort=49152])
  ACCEPTED(eni-01997085076a9b98a)" ], "text/plain": [ "Trace(disposition='ACCEPTED', hops=[Hop(node='i-04cd3db5124a05ee6', steps=[Step(detail=OriginateStepDetail(originatingVrf='default'), action='ORIGINATED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopIp(ip='10.30.1.1', type='ip'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='eni-05452497daf80ccb3', resolvedNextHopIp='10.30.1.1', type='ForwardedOutInterface'), arpIp='10.30.1.1', outputInterface='eni-05452497daf80ccb3'), action='FORWARDED'), Step(detail=FilterStepDetail(filter='~EGRESS_ACL~eni-05452497daf80ccb3', filterType='EGRESS_FILTER', inputInterface='', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-04cd3db5124a05ee6', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.30.1.166', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=SetupSessionStepDetail(sessionScope=IncomingSessionScope(incomingInterfaces=['eni-05452497daf80ccb3']), sessionAction=Accept(), matchCriteria=SessionMatchExpr(ipProtocol='TCP', srcIp='10.20.1.207', dstIp='10.30.1.166', srcPort=22, dstPort=49152), transformation=[]), action='SETUP_SESSION'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='eni-05452497daf80ccb3', transformedFlow=None), action='TRANSMITTED')]), Hop(node='subnet-0cb5f4c094bee5214', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='to-instances', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.0.0/16', nextHop=NextHopInterface(interface='vpc-0276455718806058a-vrf-tgw-attach-021f89744fac566dd', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='vpc-0276455718806058a-vrf-tgw-attach-021f89744fac566dd', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='vpc-0276455718806058a-vrf-tgw-attach-021f89744fac566dd'), action='FORWARDED'), Step(detail=FilterStepDetail(filter='acl-0b3d0f6b0978f09f8_egress', filterType='EGRESS_FILTER', inputInterface='to-instances', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-04cd3db5124a05ee6', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.30.1.166', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='vpc-0276455718806058a-vrf-tgw-attach-021f89744fac566dd', transformedFlow=None), action='TRANSMITTED')]), Hop(node='vpc-0276455718806058a', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='subnet-0cb5f4c094bee5214-vrf-tgw-attach-021f89744fac566dd', inputVrf='vrf-tgw-attach-021f89744fac566dd'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopInterface(interface='tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03', transformedFlow=None), action='TRANSMITTED')]), Hop(node='tgw-06b348adabd13452d', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpc-0276455718806058a-tgw-rtb-00e37bc5142347b03', inputVrf='vrf-tgw-rtb-00e37bc5142347b03'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.0.0/16', nextHop=NextHopInterface(interface='vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03', transformedFlow=None), action='TRANSMITTED')]), Hop(node='vpc-0574d08f8d05917e4', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03', inputVrf='vrf-tgw-attach-0648110513acd6de5'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.1.0/24', nextHop=NextHopInterface(interface='subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5', transformedFlow=None), action='TRANSMITTED')]), Hop(node='subnet-06a692ed4ef84368d', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpc-0574d08f8d05917e4-vrf-tgw-attach-0648110513acd6de5', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='acl-09c0bb4e71ae5f9e4_ingress', filterType='INGRESS_FILTER', inputInterface='vpc-0574d08f8d05917e4-vrf-tgw-attach-0648110513acd6de5', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-04cd3db5124a05ee6', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.30.1.166', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='connected', network='10.20.1.0/24', nextHop=NextHopInterface(interface='to-instances', ip=None, type='interface'), nextHopIp=None, admin=0, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='to-instances', resolvedNextHopIp=None, type='ForwardedOutInterface'), arpIp=None, outputInterface='to-instances'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='to-instances', transformedFlow=None), action='TRANSMITTED')]), Hop(node='i-01602d9efaed4409a', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='eni-01997085076a9b98a', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='~INGRESS_ACL~eni-01997085076a9b98a', filterType='INGRESS_FILTER', inputInterface='eni-01997085076a9b98a', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-04cd3db5124a05ee6', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.30.1.166', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=SetupSessionStepDetail(sessionScope=OriginatingSessionScope(originatingVrf='default'), sessionAction=ForwardOutInterface(nextHopHostname='subnet-06a692ed4ef84368d', nextHopInterface='to-instances', outgoingInterface='eni-01997085076a9b98a'), matchCriteria=SessionMatchExpr(ipProtocol='TCP', srcIp='10.20.1.207', dstIp='10.30.1.166', srcPort=22, dstPort=49152), transformation=[]), action='SETUP_SESSION'), Step(detail=InboundStepDetail(interface='eni-01997085076a9b98a'), action='ACCEPTED')])])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# traceroute between instances in the same region, using SSH\n", "ans = bf.q.traceroute(startLocation=hosts[\"east2_private\"], \n", " headers=HeaderConstraints(dstIps=hosts[\"east2_public\"], \n", " applications=\"ssh\")).answer()\n", "show_first_trace(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The trace above shows how traffic goes from `host[\"east2_private\"]` to `host[\"east2_public\"]` -- via the source subnet and VPC, then to the transit gateway, and finally to the destination VPC and subnet. Along the way, it also shows where the flow encounters security groups (at both instances) and network ACLs (at subnets). In this instance, all security groups and network ACLs permit this particular flow.\n", "\n", "This type of insight into traffic paths, which helps understand and debug network configuration, is difficult to obtain otherwise. Traceroutes on the live AWS network do not yield any information if the flow does not make it through, and do not show why or where a packet is dropped." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Paths across AWS regions\n", "\n", "The traceroute query below shows paths across instances in two different regions." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Flow: start=i-01602d9efaed4409a [10.20.1.207:49152->10.40.2.80:22 TCP (SYN)]'" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "DENIED_OUT
1. node: i-01602d9efaed4409a
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: eni-01997085076a9b98a with resolved next-hop IP: 10.20.1.1, Routes: [static (Network: 0.0.0.0/0, Next Hop: ip 10.20.1.1)])
  PERMITTED(~EGRESS_ACL~eni-01997085076a9b98a (EGRESS_FILTER))
  SETUP_SESSION(Incoming Interfaces: [eni-01997085076a9b98a], Action: Accept, Match Criteria: [ipProtocol=TCP, srcIp=10.40.2.80, dstIp=10.20.1.207, srcPort=22, dstPort=49152])
  TRANSMITTED(eni-01997085076a9b98a)
2. node: subnet-06a692ed4ef84368d
  RECEIVED(to-instances)
  FORWARDED(Forwarded out interface: vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 0.0.0.0/0, Next Hop: interface vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7 ip 169.254.0.1)])
  PERMITTED(acl-09c0bb4e71ae5f9e4_egress (EGRESS_FILTER))
  TRANSMITTED(vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7)
3. node: vpc-0574d08f8d05917e4
  RECEIVED(subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7)
  FORWARDED(Forwarded out interface: igw-02fd68f94367a67c7 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 0.0.0.0/0, Next Hop: interface igw-02fd68f94367a67c7 ip 169.254.0.1)])
  TRANSMITTED(igw-02fd68f94367a67c7)
4. node: igw-02fd68f94367a67c7
  RECEIVED(vpc-0574d08f8d05917e4)
  PERMITTED(~DENY~UNASSOCIATED~PRIVATE~IPs~ (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: backbone with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 0.0.0.0/0, Next Hop: interface backbone ip 169.254.0.1)])
  TRANSFORMED(SOURCE_NAT srcIp: 10.20.1.207 -> 13.59.144.125)
  TRANSMITTED(backbone)
5. node: isp_16509
  RECEIVED(To-igw-02fd68f94367a67c7-backbone)
  FORWARDED(Forwarded out interface: To-Internet with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 0.0.0.0/0, Next Hop: interface To-Internet ip 169.254.0.1)])
  DENIED(Block outgoing traffic using reserved addresses (EGRESS_FILTER))" ], "text/plain": [ "Trace(disposition='DENIED_OUT', hops=[Hop(node='i-01602d9efaed4409a', steps=[Step(detail=OriginateStepDetail(originatingVrf='default'), action='ORIGINATED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopIp(ip='10.20.1.1', type='ip'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='eni-01997085076a9b98a', resolvedNextHopIp='10.20.1.1', type='ForwardedOutInterface'), arpIp='10.20.1.1', outputInterface='eni-01997085076a9b98a'), action='FORWARDED'), Step(detail=FilterStepDetail(filter='~EGRESS_ACL~eni-01997085076a9b98a', filterType='EGRESS_FILTER', inputInterface='', flow=Flow(dscp=0, dstIp='10.40.2.80', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.20.1.207', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=SetupSessionStepDetail(sessionScope=IncomingSessionScope(incomingInterfaces=['eni-01997085076a9b98a']), sessionAction=Accept(), matchCriteria=SessionMatchExpr(ipProtocol='TCP', srcIp='10.40.2.80', dstIp='10.20.1.207', srcPort=22, dstPort=49152), transformation=[]), action='SETUP_SESSION'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='eni-01997085076a9b98a', transformedFlow=None), action='TRANSMITTED')]), Hop(node='subnet-06a692ed4ef84368d', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='to-instances', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopInterface(interface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7'), action='FORWARDED'), Step(detail=FilterStepDetail(filter='acl-09c0bb4e71ae5f9e4_egress', filterType='EGRESS_FILTER', inputInterface='to-instances', flow=Flow(dscp=0, dstIp='10.40.2.80', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.20.1.207', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', transformedFlow=None), action='TRANSMITTED')]), Hop(node='vpc-0574d08f8d05917e4', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7', inputVrf='vrf-igw-02fd68f94367a67c7'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopInterface(interface='igw-02fd68f94367a67c7', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='igw-02fd68f94367a67c7', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='igw-02fd68f94367a67c7'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='igw-02fd68f94367a67c7', transformedFlow=None), action='TRANSMITTED')]), Hop(node='igw-02fd68f94367a67c7', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpc-0574d08f8d05917e4', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='~DENY~UNASSOCIATED~PRIVATE~IPs~', filterType='INGRESS_FILTER', inputInterface='vpc-0574d08f8d05917e4', flow=Flow(dscp=0, dstIp='10.40.2.80', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.20.1.207', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='0.0.0.0/0', nextHop=NextHopInterface(interface='backbone', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='backbone', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='backbone'), action='FORWARDED'), Step(detail=TransformationStepDetail(transformationType='SOURCE_NAT', flowDiffs=[FlowDiff(fieldName='srcIp', oldValue='10.20.1.207', newValue='13.59.144.125')]), action='TRANSFORMED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='backbone', transformedFlow={'dscp': 0, 'dstIp': '10.40.2.80', 'dstPort': 22, 'ecn': 0, 'fragmentOffset': 0, 'ingressNode': 'i-01602d9efaed4409a', 'ingressVrf': 'default', 'ipProtocol': 'TCP', 'packetLength': 512, 'srcIp': '13.59.144.125', 'srcPort': 49152, 'state': 'NEW', 'tag': 'tag', 'tcpFlagsAck': 0, 'tcpFlagsCwr': 0, 'tcpFlagsEce': 0, 'tcpFlagsFin': 0, 'tcpFlagsPsh': 0, 'tcpFlagsRst': 0, 'tcpFlagsSyn': 1, 'tcpFlagsUrg': 0}), action='TRANSMITTED')]), Hop(node='isp_16509', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='To-igw-02fd68f94367a67c7-backbone', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='0.0.0.0/0', nextHop=NextHopInterface(interface='To-Internet', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='To-Internet', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='To-Internet'), action='FORWARDED'), Step(detail=FilterStepDetail(filter='Block outgoing traffic using reserved addresses', filterType='EGRESS_FILTER', inputInterface='To-igw-02fd68f94367a67c7-backbone', flow=Flow(dscp=0, dstIp='10.40.2.80', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='13.59.144.125', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='DENIED')])])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# traceroute between instances across region using the destination's private IP\n", "ans = bf.q.traceroute(startLocation=hosts[\"east2_public\"], \n", " headers=HeaderConstraints(dstIps=hosts[\"west2_public\"],\n", " applications=\"ssh\")).answer()\n", "show_first_trace(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that such traffic does not reach the destination but instead is dropped by the AWS backbone (ASN 16509). This happens because, in our network, there is no (transit gateway or VPC) peering between VPCs in different regions. So, the source subnet is unaware of the address space of the destination subnet, which makes it use the default route that points to the Internet gateway (`igw-02fd68f94367a67c7`). The Internet gateway forwards the packet to `aws-backbone`, after NAT'ing its source IP. The packet is eventually dropped as it is using a private address as destination. Recall that using the instance name as destination amounts to using its private IP.\n", "\n", "The behavior is different if we use the public IP instead, as shown below." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Flow: start=i-01602d9efaed4409a [10.20.1.207:49152->54.191.42.182:22 TCP (SYN)]'" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "ACCEPTED
1. node: i-01602d9efaed4409a
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: eni-01997085076a9b98a with resolved next-hop IP: 10.20.1.1, Routes: [static (Network: 0.0.0.0/0, Next Hop: ip 10.20.1.1)])
  PERMITTED(~EGRESS_ACL~eni-01997085076a9b98a (EGRESS_FILTER))
  SETUP_SESSION(Incoming Interfaces: [eni-01997085076a9b98a], Action: Accept, Match Criteria: [ipProtocol=TCP, srcIp=54.191.42.182, dstIp=10.20.1.207, srcPort=22, dstPort=49152])
  TRANSMITTED(eni-01997085076a9b98a)
2. node: subnet-06a692ed4ef84368d
  RECEIVED(to-instances)
  FORWARDED(Forwarded out interface: vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 0.0.0.0/0, Next Hop: interface vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7 ip 169.254.0.1)])
  PERMITTED(acl-09c0bb4e71ae5f9e4_egress (EGRESS_FILTER))
  TRANSMITTED(vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7)
3. node: vpc-0574d08f8d05917e4
  RECEIVED(subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7)
  FORWARDED(Forwarded out interface: igw-02fd68f94367a67c7 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 0.0.0.0/0, Next Hop: interface igw-02fd68f94367a67c7 ip 169.254.0.1)])
  TRANSMITTED(igw-02fd68f94367a67c7)
4. node: igw-02fd68f94367a67c7
  RECEIVED(vpc-0574d08f8d05917e4)
  PERMITTED(~DENY~UNASSOCIATED~PRIVATE~IPs~ (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: backbone with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 0.0.0.0/0, Next Hop: interface backbone ip 169.254.0.1)])
  TRANSFORMED(SOURCE_NAT srcIp: 10.20.1.207 -> 13.59.144.125)
  TRANSMITTED(backbone)
5. node: isp_16509
  RECEIVED(To-igw-02fd68f94367a67c7-backbone)
  FORWARDED(Forwarded out interface: To-igw-0a8309f3192e7cea3-backbone with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 54.191.42.182/32, Next Hop: interface To-igw-0a8309f3192e7cea3-backbone ip 169.254.0.1)])
  TRANSMITTED(To-igw-0a8309f3192e7cea3-backbone)
6. node: igw-0a8309f3192e7cea3
  RECEIVED(backbone)
  TRANSFORMED(DEST_NAT dstIp: 54.191.42.182 -> 10.40.2.80)
  FORWARDED(Forwarded out interface: vpc-00b65e98077106059 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.40.0.0/16, Next Hop: interface vpc-00b65e98077106059 ip 169.254.0.1)])
  TRANSMITTED(vpc-00b65e98077106059)
7. node: vpc-00b65e98077106059
  RECEIVED(igw-0a8309f3192e7cea3)
  FORWARDED(Forwarded out interface: subnet-06005943afe32f714-vrf-igw-0a8309f3192e7cea3 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.40.2.0/24, Next Hop: interface subnet-06005943afe32f714-vrf-igw-0a8309f3192e7cea3 ip 169.254.0.1)])
  TRANSMITTED(subnet-06005943afe32f714-vrf-igw-0a8309f3192e7cea3)
8. node: subnet-06005943afe32f714
  RECEIVED(vpc-00b65e98077106059-vrf-igw-0a8309f3192e7cea3)
  PERMITTED(acl-087574e8620270842_ingress (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: to-instances, Routes: [connected (Network: 10.40.2.0/24, Next Hop: interface to-instances)])
  TRANSMITTED(to-instances)
9. node: i-02cae6eaa9edeed70
  RECEIVED(eni-087e18628dadd9b48)
  PERMITTED(~INGRESS_ACL~eni-087e18628dadd9b48 (INGRESS_FILTER))
  SETUP_SESSION(Originating VRF: default, Action: ForwardOutInterface(Next Hop: subnet-06005943afe32f714, Next Hop Interface: to-instances, Outgoing Interface: eni-087e18628dadd9b48), Match Criteria: [ipProtocol=TCP, srcIp=10.40.2.80, dstIp=13.59.144.125, srcPort=22, dstPort=49152])
  ACCEPTED(eni-087e18628dadd9b48)" ], "text/plain": [ "Trace(disposition='ACCEPTED', hops=[Hop(node='i-01602d9efaed4409a', steps=[Step(detail=OriginateStepDetail(originatingVrf='default'), action='ORIGINATED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopIp(ip='10.20.1.1', type='ip'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='eni-01997085076a9b98a', resolvedNextHopIp='10.20.1.1', type='ForwardedOutInterface'), arpIp='10.20.1.1', outputInterface='eni-01997085076a9b98a'), action='FORWARDED'), Step(detail=FilterStepDetail(filter='~EGRESS_ACL~eni-01997085076a9b98a', filterType='EGRESS_FILTER', inputInterface='', flow=Flow(dscp=0, dstIp='54.191.42.182', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.20.1.207', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=SetupSessionStepDetail(sessionScope=IncomingSessionScope(incomingInterfaces=['eni-01997085076a9b98a']), sessionAction=Accept(), matchCriteria=SessionMatchExpr(ipProtocol='TCP', srcIp='54.191.42.182', dstIp='10.20.1.207', srcPort=22, dstPort=49152), transformation=[]), action='SETUP_SESSION'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='eni-01997085076a9b98a', transformedFlow=None), action='TRANSMITTED')]), Hop(node='subnet-06a692ed4ef84368d', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='to-instances', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopInterface(interface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7'), action='FORWARDED'), Step(detail=FilterStepDetail(filter='acl-09c0bb4e71ae5f9e4_egress', filterType='EGRESS_FILTER', inputInterface='to-instances', flow=Flow(dscp=0, dstIp='54.191.42.182', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.20.1.207', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', transformedFlow=None), action='TRANSMITTED')]), Hop(node='vpc-0574d08f8d05917e4', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7', inputVrf='vrf-igw-02fd68f94367a67c7'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopInterface(interface='igw-02fd68f94367a67c7', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='igw-02fd68f94367a67c7', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='igw-02fd68f94367a67c7'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='igw-02fd68f94367a67c7', transformedFlow=None), action='TRANSMITTED')]), Hop(node='igw-02fd68f94367a67c7', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpc-0574d08f8d05917e4', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='~DENY~UNASSOCIATED~PRIVATE~IPs~', filterType='INGRESS_FILTER', inputInterface='vpc-0574d08f8d05917e4', flow=Flow(dscp=0, dstIp='54.191.42.182', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='10.20.1.207', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='0.0.0.0/0', nextHop=NextHopInterface(interface='backbone', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='backbone', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='backbone'), action='FORWARDED'), Step(detail=TransformationStepDetail(transformationType='SOURCE_NAT', flowDiffs=[FlowDiff(fieldName='srcIp', oldValue='10.20.1.207', newValue='13.59.144.125')]), action='TRANSFORMED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='backbone', transformedFlow={'dscp': 0, 'dstIp': '54.191.42.182', 'dstPort': 22, 'ecn': 0, 'fragmentOffset': 0, 'ingressNode': 'i-01602d9efaed4409a', 'ingressVrf': 'default', 'ipProtocol': 'TCP', 'packetLength': 512, 'srcIp': '13.59.144.125', 'srcPort': 49152, 'state': 'NEW', 'tag': 'tag', 'tcpFlagsAck': 0, 'tcpFlagsCwr': 0, 'tcpFlagsEce': 0, 'tcpFlagsFin': 0, 'tcpFlagsPsh': 0, 'tcpFlagsRst': 0, 'tcpFlagsSyn': 1, 'tcpFlagsUrg': 0}), action='TRANSMITTED')]), Hop(node='isp_16509', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='To-igw-02fd68f94367a67c7-backbone', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='54.191.42.182/32', nextHop=NextHopInterface(interface='To-igw-0a8309f3192e7cea3-backbone', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='To-igw-0a8309f3192e7cea3-backbone', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='To-igw-0a8309f3192e7cea3-backbone'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='To-igw-0a8309f3192e7cea3-backbone', transformedFlow=None), action='TRANSMITTED')]), Hop(node='igw-0a8309f3192e7cea3', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='backbone', inputVrf='default'), action='RECEIVED'), Step(detail=TransformationStepDetail(transformationType='DEST_NAT', flowDiffs=[FlowDiff(fieldName='dstIp', oldValue='54.191.42.182', newValue='10.40.2.80')]), action='TRANSFORMED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.40.0.0/16', nextHop=NextHopInterface(interface='vpc-00b65e98077106059', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='vpc-00b65e98077106059', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='vpc-00b65e98077106059'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='vpc-00b65e98077106059', transformedFlow={'dscp': 0, 'dstIp': '10.40.2.80', 'dstPort': 22, 'ecn': 0, 'fragmentOffset': 0, 'ingressNode': 'i-01602d9efaed4409a', 'ingressVrf': 'default', 'ipProtocol': 'TCP', 'packetLength': 512, 'srcIp': '13.59.144.125', 'srcPort': 49152, 'state': 'NEW', 'tag': 'tag', 'tcpFlagsAck': 0, 'tcpFlagsCwr': 0, 'tcpFlagsEce': 0, 'tcpFlagsFin': 0, 'tcpFlagsPsh': 0, 'tcpFlagsRst': 0, 'tcpFlagsSyn': 1, 'tcpFlagsUrg': 0}), action='TRANSMITTED')]), Hop(node='vpc-00b65e98077106059', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='igw-0a8309f3192e7cea3', inputVrf='vrf-igw-0a8309f3192e7cea3'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.40.2.0/24', nextHop=NextHopInterface(interface='subnet-06005943afe32f714-vrf-igw-0a8309f3192e7cea3', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='subnet-06005943afe32f714-vrf-igw-0a8309f3192e7cea3', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='subnet-06005943afe32f714-vrf-igw-0a8309f3192e7cea3'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='subnet-06005943afe32f714-vrf-igw-0a8309f3192e7cea3', transformedFlow=None), action='TRANSMITTED')]), Hop(node='subnet-06005943afe32f714', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpc-00b65e98077106059-vrf-igw-0a8309f3192e7cea3', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='acl-087574e8620270842_ingress', filterType='INGRESS_FILTER', inputInterface='vpc-00b65e98077106059-vrf-igw-0a8309f3192e7cea3', flow=Flow(dscp=0, dstIp='10.40.2.80', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='13.59.144.125', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='connected', network='10.40.2.0/24', nextHop=NextHopInterface(interface='to-instances', ip=None, type='interface'), nextHopIp=None, admin=0, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='to-instances', resolvedNextHopIp=None, type='ForwardedOutInterface'), arpIp=None, outputInterface='to-instances'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='to-instances', transformedFlow=None), action='TRANSMITTED')]), Hop(node='i-02cae6eaa9edeed70', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='eni-087e18628dadd9b48', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='~INGRESS_ACL~eni-087e18628dadd9b48', filterType='INGRESS_FILTER', inputInterface='eni-087e18628dadd9b48', flow=Flow(dscp=0, dstIp='10.40.2.80', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='i-01602d9efaed4409a', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='13.59.144.125', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=SetupSessionStepDetail(sessionScope=OriginatingSessionScope(originatingVrf='default'), sessionAction=ForwardOutInterface(nextHopHostname='subnet-06005943afe32f714', nextHopInterface='to-instances', outgoingInterface='eni-087e18628dadd9b48'), matchCriteria=SessionMatchExpr(ipProtocol='TCP', srcIp='10.40.2.80', dstIp='13.59.144.125', srcPort=22, dstPort=49152), transformation=[]), action='SETUP_SESSION'), Step(detail=InboundStepDetail(interface='eni-087e18628dadd9b48'), action='ACCEPTED')])])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# traceroute betwee instances across region using the destination's public IP\n", "ans = bf.q.traceroute(startLocation=hosts[\"east2_public\"], \n", " headers=HeaderConstraints(dstIps=public_ips[\"west2_public\"],\n", " applications=\"ssh\")).answer()\n", "show_first_trace(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This traceroute starts out like the previous one, up until the AWS backbone (`isp_16509`) -- from source subnet to the Internet gateway which forwards it to the backbone, after source NAT'ing the packet. The backbone carries it to the internet gateway in the destination region (`igw-0a8309f3192e7cea3`), and this gateway NATs the packet's destination from the public IP to the instance's private IP. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Connectivity between DC and AWS\n", "\n", "A common mode to connect to AWS is using VPNs and BGP, that is, establish IPSec tunnels between exit gateways on the physical side and AWS gateways and run BGP on top of these tunnels to exchange prefixes. Incompatibility in either IPSec or BGP settings on the two sides means that connectivity between the DC and AWS will not work. \n", "\n", "Batfish can determine if the two sides are compatibly configured with respect to IPSec and BGP settings and if those sessions will come up. \n", "\n", "The query below lists the status of all IPSec sessions between the `exitgw` and AWS transit gateways (specified using the regular expression `^tgw-` that matches those node names). This filtering lets us ignore any other IPSec sessions that may exist in our network and focus on DC-AWS connectivity." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 NodeNode_InterfaceNode_IPRemote_NodeRemote_Node_InterfaceRemote_Node_IPTunnel_InterfacesStatus
0exitgwexitgw[GigabitEthernet3]147.75.69.27tgw-06b348adabd13452dtgw-06b348adabd13452d[external-vpn-01c45673532d3e33e-1]3.19.24.131Tunnel1 -> vpn-vpn-01c45673532d3e33e-1IPSEC_SESSION_ESTABLISHED
1exitgwexitgw[GigabitEthernet3]147.75.69.27tgw-06b348adabd13452dtgw-06b348adabd13452d[external-vpn-01c45673532d3e33e-2]52.14.53.162Tunnel2 -> vpn-vpn-01c45673532d3e33e-2IPSEC_SESSION_ESTABLISHED
2exitgwexitgw[GigabitEthernet3]147.75.69.27tgw-0888a76c8a371246dtgw-0888a76c8a371246d[external-vpn-0dc7abdb974ff8a69-1]34.209.88.227Tunnel3 -> vpn-vpn-0dc7abdb974ff8a69-1IPSEC_SESSION_ESTABLISHED
3exitgwexitgw[GigabitEthernet3]147.75.69.27tgw-0888a76c8a371246dtgw-0888a76c8a371246d[external-vpn-0dc7abdb974ff8a69-2]44.227.244.7Tunnel4 -> vpn-vpn-0dc7abdb974ff8a69-2IPSEC_SESSION_ESTABLISHED
\n" ], "text/plain": [ " Node Node_Interface Node_IP Remote_Node \\\n", "0 exitgw exitgw[GigabitEthernet3] 147.75.69.27 tgw-06b348adabd13452d \n", "1 exitgw exitgw[GigabitEthernet3] 147.75.69.27 tgw-06b348adabd13452d \n", "2 exitgw exitgw[GigabitEthernet3] 147.75.69.27 tgw-0888a76c8a371246d \n", "3 exitgw exitgw[GigabitEthernet3] 147.75.69.27 tgw-0888a76c8a371246d \n", "\n", " Remote_Node_Interface Remote_Node_IP \\\n", "0 tgw-06b348adabd13452d[external-vpn-01c45673532d3e33e-1] 3.19.24.131 \n", "1 tgw-06b348adabd13452d[external-vpn-01c45673532d3e33e-2] 52.14.53.162 \n", "2 tgw-0888a76c8a371246d[external-vpn-0dc7abdb974ff8a69-1] 34.209.88.227 \n", "3 tgw-0888a76c8a371246d[external-vpn-0dc7abdb974ff8a69-2] 44.227.244.7 \n", "\n", " Tunnel_Interfaces Status \n", "0 Tunnel1 -> vpn-vpn-01c45673532d3e33e-1 IPSEC_SESSION_ESTABLISHED \n", "1 Tunnel2 -> vpn-vpn-01c45673532d3e33e-2 IPSEC_SESSION_ESTABLISHED \n", "2 Tunnel3 -> vpn-vpn-0dc7abdb974ff8a69-1 IPSEC_SESSION_ESTABLISHED \n", "3 Tunnel4 -> vpn-vpn-0dc7abdb974ff8a69-2 IPSEC_SESSION_ESTABLISHED " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# show the status of all IPSec tunnels between exitgw and AWS transit gateways\n", "ans = bf.q.ipsecSessionStatus(nodes=\"exitgw\", remoteNodes=\"/^tgw-/\").answer()\n", "show(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the output above, we see all expected tunnels. Each transit gateways has two established sessions to `exitgw`. The default AWS behavior is to have two IPSec tunnels between gateways and physical nodes.\n", "\n", "Now that we know IPSec tunnels are working, we can check BGP sessions. The query below lists the status of all BGP sessions where one end is an AWS transit gateway." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 NodeVRFLocal_ASLocal_InterfaceLocal_IPRemote_ASRemote_NodeRemote_InterfaceRemote_IPAddress_FamiliesSession_TypeEstablished_Status
0exitgwdefault65100None169.254.252.7864512tgw-0888a76c8a371246dNone169.254.252.77IPV4_UNICASTEBGP_SINGLEHOPESTABLISHED
1exitgwdefault65100None169.254.25.16264512tgw-06b348adabd13452dNone169.254.25.161IPV4_UNICASTEBGP_SINGLEHOPESTABLISHED
2exitgwdefault65100None169.254.172.264512tgw-06b348adabd13452dNone169.254.172.1IPV4_UNICASTEBGP_SINGLEHOPESTABLISHED
3exitgwdefault65100None169.254.215.8264512tgw-0888a76c8a371246dNone169.254.215.81IPV4_UNICASTEBGP_SINGLEHOPESTABLISHED
\n" ], "text/plain": [ " Node VRF Local_AS Local_Interface Local_IP Remote_AS \\\n", "0 exitgw default 65100 None 169.254.252.78 64512 \n", "1 exitgw default 65100 None 169.254.25.162 64512 \n", "2 exitgw default 65100 None 169.254.172.2 64512 \n", "3 exitgw default 65100 None 169.254.215.82 64512 \n", "\n", " Remote_Node Remote_Interface Remote_IP Address_Families \\\n", "0 tgw-0888a76c8a371246d None 169.254.252.77 ['IPV4_UNICAST'] \n", "1 tgw-06b348adabd13452d None 169.254.25.161 ['IPV4_UNICAST'] \n", "2 tgw-06b348adabd13452d None 169.254.172.1 ['IPV4_UNICAST'] \n", "3 tgw-0888a76c8a371246d None 169.254.215.81 ['IPV4_UNICAST'] \n", "\n", " Session_Type Established_Status \n", "0 EBGP_SINGLEHOP ESTABLISHED \n", "1 EBGP_SINGLEHOP ESTABLISHED \n", "2 EBGP_SINGLEHOP ESTABLISHED \n", "3 EBGP_SINGLEHOP ESTABLISHED " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# show the status of all BGP sessions between exitgw and AWS transit gateways\n", "ans = bf.q.bgpSessionStatus(nodes=\"exitgw\", remoteNodes=\"/^tgw-/\").answer()\n", "show(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output above shows that all BGP sessions are established as expected. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Paths from the DC to AWS\n", "\n", "Finally, lets look at paths from the datacenter to AWS. The query below does that using the private IP of the public instance in us-east-2 region." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Flow: start=srv-101 [203.0.113.12:49152->10.20.1.207:22 TCP (SYN)]'" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "ACCEPTED
1. node: srv-101
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: eth0 with resolved next-hop IP: 203.0.113.2, Routes: [static (Network: 0.0.0.0/0, Next Hop: interface eth0 ip 203.0.113.2)])
  TRANSMITTED(eth0)
2. node: leaf1
  RECEIVED(Ethernet10)
  FORWARDED(Forwarded out interface: Ethernet1 with resolved next-hop IP: 10.10.11.1, Routes: [bgp (Network: 10.20.0.0/16, Next Hop: ip 10.10.11.1)])
  TRANSMITTED(Ethernet1)
3. node: spine1
  RECEIVED(Ethernet1)
  FORWARDED(Forwarded out interface: Ethernet10 with resolved next-hop IP: 10.10.100.2, Routes: [bgp (Network: 10.20.0.0/16, Next Hop: ip 10.10.100.2)])
  TRANSMITTED(Ethernet10)
4. node: exitgw
  RECEIVED(GigabitEthernet1)
  FORWARDED(Forwarded out interface: Tunnel1 with resolved next-hop IP: 169.254.25.161, Routes: [bgp (Network: 10.20.0.0/16, Next Hop: ip 169.254.25.161)])
  TRANSMITTED(Tunnel1)
5. node: tgw-06b348adabd13452d
  RECEIVED(vpn-vpn-01c45673532d3e33e-1)
  FORWARDED(Forwarded out interface: vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.0.0/16, Next Hop: interface vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03 ip 169.254.0.1)])
  TRANSMITTED(vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03)
6. node: vpc-0574d08f8d05917e4
  RECEIVED(tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03)
  FORWARDED(Forwarded out interface: subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.1.0/24, Next Hop: interface subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5 ip 169.254.0.1)])
  TRANSMITTED(subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5)
7. node: subnet-06a692ed4ef84368d
  RECEIVED(vpc-0574d08f8d05917e4-vrf-tgw-attach-0648110513acd6de5)
  PERMITTED(acl-09c0bb4e71ae5f9e4_ingress (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: to-instances, Routes: [connected (Network: 10.20.1.0/24, Next Hop: interface to-instances)])
  TRANSMITTED(to-instances)
8. node: i-01602d9efaed4409a
  RECEIVED(eni-01997085076a9b98a)
  PERMITTED(~INGRESS_ACL~eni-01997085076a9b98a (INGRESS_FILTER))
  SETUP_SESSION(Originating VRF: default, Action: ForwardOutInterface(Next Hop: subnet-06a692ed4ef84368d, Next Hop Interface: to-instances, Outgoing Interface: eni-01997085076a9b98a), Match Criteria: [ipProtocol=TCP, srcIp=10.20.1.207, dstIp=203.0.113.12, srcPort=22, dstPort=49152])
  ACCEPTED(eni-01997085076a9b98a)" ], "text/plain": [ "Trace(disposition='ACCEPTED', hops=[Hop(node='srv-101', steps=[Step(detail=OriginateStepDetail(originatingVrf='default'), action='ORIGINATED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopInterface(interface='eth0', ip='203.0.113.2', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='eth0', resolvedNextHopIp='203.0.113.2', type='ForwardedOutInterface'), arpIp='203.0.113.2', outputInterface='eth0'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='eth0', transformedFlow=None), action='TRANSMITTED')]), Hop(node='leaf1', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='Ethernet10', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='10.20.0.0/16', nextHop=NextHopIp(ip='10.10.11.1', type='ip'), nextHopIp=None, admin=200, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='Ethernet1', resolvedNextHopIp='10.10.11.1', type='ForwardedOutInterface'), arpIp='10.10.11.1', outputInterface='Ethernet1'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='Ethernet1', transformedFlow=None), action='TRANSMITTED')]), Hop(node='spine1', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='Ethernet1', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='10.20.0.0/16', nextHop=NextHopIp(ip='10.10.100.2', type='ip'), nextHopIp=None, admin=200, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='Ethernet10', resolvedNextHopIp='10.10.100.2', type='ForwardedOutInterface'), arpIp='10.10.100.2', outputInterface='Ethernet10'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='Ethernet10', transformedFlow=None), action='TRANSMITTED')]), Hop(node='exitgw', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='GigabitEthernet1', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='10.20.0.0/16', nextHop=NextHopIp(ip='169.254.25.161', type='ip'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='Tunnel1', resolvedNextHopIp='169.254.25.161', type='ForwardedOutInterface'), arpIp='169.254.25.161', outputInterface='Tunnel1'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='Tunnel1', transformedFlow=None), action='TRANSMITTED')]), Hop(node='tgw-06b348adabd13452d', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpn-vpn-01c45673532d3e33e-1', inputVrf='vrf-tgw-rtb-00e37bc5142347b03'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.0.0/16', nextHop=NextHopInterface(interface='vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='vpc-0574d08f8d05917e4-tgw-rtb-00e37bc5142347b03', transformedFlow=None), action='TRANSMITTED')]), Hop(node='vpc-0574d08f8d05917e4', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='tgw-06b348adabd13452d-tgw-rtb-00e37bc5142347b03', inputVrf='vrf-tgw-attach-0648110513acd6de5'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.1.0/24', nextHop=NextHopInterface(interface='subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='subnet-06a692ed4ef84368d-vrf-tgw-attach-0648110513acd6de5', transformedFlow=None), action='TRANSMITTED')]), Hop(node='subnet-06a692ed4ef84368d', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpc-0574d08f8d05917e4-vrf-tgw-attach-0648110513acd6de5', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='acl-09c0bb4e71ae5f9e4_ingress', filterType='INGRESS_FILTER', inputInterface='vpc-0574d08f8d05917e4-vrf-tgw-attach-0648110513acd6de5', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='srv-101', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='203.0.113.12', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='connected', network='10.20.1.0/24', nextHop=NextHopInterface(interface='to-instances', ip=None, type='interface'), nextHopIp=None, admin=0, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='to-instances', resolvedNextHopIp=None, type='ForwardedOutInterface'), arpIp=None, outputInterface='to-instances'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='to-instances', transformedFlow=None), action='TRANSMITTED')]), Hop(node='i-01602d9efaed4409a', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='eni-01997085076a9b98a', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='~INGRESS_ACL~eni-01997085076a9b98a', filterType='INGRESS_FILTER', inputInterface='eni-01997085076a9b98a', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='srv-101', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='203.0.113.12', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=SetupSessionStepDetail(sessionScope=OriginatingSessionScope(originatingVrf='default'), sessionAction=ForwardOutInterface(nextHopHostname='subnet-06a692ed4ef84368d', nextHopInterface='to-instances', outgoingInterface='eni-01997085076a9b98a'), matchCriteria=SessionMatchExpr(ipProtocol='TCP', srcIp='10.20.1.207', dstIp='203.0.113.12', srcPort=22, dstPort=49152), transformation=[]), action='SETUP_SESSION'), Step(detail=InboundStepDetail(interface='eni-01997085076a9b98a'), action='ACCEPTED')])])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# traceroute from DC host to an instances using private IP\n", "ans = bf.q.traceroute(startLocation=\"srv-101\", \n", " headers=HeaderConstraints(dstIps=hosts[\"east2_public\"],\n", " applications=\"ssh\")).answer()\n", "show_first_trace(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that this traffic travels on the IPSec links between the datacenter's `exitgw` and the transit gateway in the destination region (`tgw-06b348adabd13452d`), and then makes it to the destination instance after making it successfully past the network ACL on the subnet node and the security group on the instance.\n", "\n", "A different path emerges if we use the public IP of the same instance, as shown below." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Flow: start=srv-101 [203.0.113.12:49152->13.59.144.125:22 TCP (SYN)]'" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "ACCEPTED
1. node: srv-101
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: eth0 with resolved next-hop IP: 203.0.113.2, Routes: [static (Network: 0.0.0.0/0, Next Hop: interface eth0 ip 203.0.113.2)])
  TRANSMITTED(eth0)
2. node: leaf1
  RECEIVED(Ethernet10)
  FORWARDED(Forwarded out interface: Ethernet1 with resolved next-hop IP: 10.10.11.1, Routes: [bgp (Network: 0.0.0.0/0, Next Hop: ip 10.10.11.1)])
  TRANSMITTED(Ethernet1)
3. node: spine1
  RECEIVED(Ethernet1)
  FORWARDED(Forwarded out interface: Ethernet10 with resolved next-hop IP: 10.10.100.2, Routes: [bgp (Network: 0.0.0.0/0, Next Hop: ip 10.10.100.2)])
  TRANSMITTED(Ethernet10)
4. node: exitgw
  RECEIVED(GigabitEthernet1)
  FORWARDED(Forwarded out interface: GigabitEthernet3 with resolved next-hop IP: 147.75.69.26, Routes: [static (Network: 0.0.0.0/0, Next Hop: ip 147.75.69.26)])
  TRANSMITTED(GigabitEthernet3)
5. node: isp_65200
  RECEIVED(To-exitgw-GigabitEthernet3)
  FORWARDED(Forwarded out interface: To-Internet with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 0.0.0.0/0, Next Hop: interface To-Internet ip 169.254.0.1)])
  PERMITTED(Block outgoing traffic using reserved addresses (EGRESS_FILTER))
  TRANSMITTED(To-Internet)
6. node: internet
  RECEIVED(To-isp_65200)
  FORWARDED(Forwarded out interface: To-isp_16509 with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 13.59.144.125/32, Next Hop: interface To-isp_16509 ip 169.254.0.1)])
  TRANSMITTED(To-isp_16509)
7. node: isp_16509
  RECEIVED(To-Internet)
  PERMITTED(Block incoming traffic using reserved addresses (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: To-igw-02fd68f94367a67c7-backbone with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 13.59.144.125/32, Next Hop: interface To-igw-02fd68f94367a67c7-backbone ip 169.254.0.1)])
  TRANSMITTED(To-igw-02fd68f94367a67c7-backbone)
8. node: igw-02fd68f94367a67c7
  RECEIVED(backbone)
  TRANSFORMED(DEST_NAT dstIp: 13.59.144.125 -> 10.20.1.207)
  FORWARDED(Forwarded out interface: vpc-0574d08f8d05917e4 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.0.0/16, Next Hop: interface vpc-0574d08f8d05917e4 ip 169.254.0.1)])
  TRANSMITTED(vpc-0574d08f8d05917e4)
9. node: vpc-0574d08f8d05917e4
  RECEIVED(igw-02fd68f94367a67c7)
  FORWARDED(Forwarded out interface: subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.1.0/24, Next Hop: interface subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7 ip 169.254.0.1)])
  TRANSMITTED(subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7)
10. node: subnet-06a692ed4ef84368d
  RECEIVED(vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7)
  PERMITTED(acl-09c0bb4e71ae5f9e4_ingress (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: to-instances, Routes: [connected (Network: 10.20.1.0/24, Next Hop: interface to-instances)])
  TRANSMITTED(to-instances)
11. node: i-01602d9efaed4409a
  RECEIVED(eni-01997085076a9b98a)
  PERMITTED(~INGRESS_ACL~eni-01997085076a9b98a (INGRESS_FILTER))
  SETUP_SESSION(Originating VRF: default, Action: ForwardOutInterface(Next Hop: subnet-06a692ed4ef84368d, Next Hop Interface: to-instances, Outgoing Interface: eni-01997085076a9b98a), Match Criteria: [ipProtocol=TCP, srcIp=10.20.1.207, dstIp=203.0.113.12, srcPort=22, dstPort=49152])
  ACCEPTED(eni-01997085076a9b98a)" ], "text/plain": [ "Trace(disposition='ACCEPTED', hops=[Hop(node='srv-101', steps=[Step(detail=OriginateStepDetail(originatingVrf='default'), action='ORIGINATED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopInterface(interface='eth0', ip='203.0.113.2', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='eth0', resolvedNextHopIp='203.0.113.2', type='ForwardedOutInterface'), arpIp='203.0.113.2', outputInterface='eth0'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='eth0', transformedFlow=None), action='TRANSMITTED')]), Hop(node='leaf1', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='Ethernet10', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='0.0.0.0/0', nextHop=NextHopIp(ip='10.10.11.1', type='ip'), nextHopIp=None, admin=200, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='Ethernet1', resolvedNextHopIp='10.10.11.1', type='ForwardedOutInterface'), arpIp='10.10.11.1', outputInterface='Ethernet1'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='Ethernet1', transformedFlow=None), action='TRANSMITTED')]), Hop(node='spine1', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='Ethernet1', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='0.0.0.0/0', nextHop=NextHopIp(ip='10.10.100.2', type='ip'), nextHopIp=None, admin=200, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='Ethernet10', resolvedNextHopIp='10.10.100.2', type='ForwardedOutInterface'), arpIp='10.10.100.2', outputInterface='Ethernet10'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='Ethernet10', transformedFlow=None), action='TRANSMITTED')]), Hop(node='exitgw', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='GigabitEthernet1', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='0.0.0.0/0', nextHop=NextHopIp(ip='147.75.69.26', type='ip'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='GigabitEthernet3', resolvedNextHopIp='147.75.69.26', type='ForwardedOutInterface'), arpIp='147.75.69.26', outputInterface='GigabitEthernet3'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='GigabitEthernet3', transformedFlow=None), action='TRANSMITTED')]), Hop(node='isp_65200', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='To-exitgw-GigabitEthernet3', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='0.0.0.0/0', nextHop=NextHopInterface(interface='To-Internet', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='To-Internet', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='To-Internet'), action='FORWARDED'), Step(detail=FilterStepDetail(filter='Block outgoing traffic using reserved addresses', filterType='EGRESS_FILTER', inputInterface='To-exitgw-GigabitEthernet3', flow=Flow(dscp=0, dstIp='13.59.144.125', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='srv-101', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='203.0.113.12', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='To-Internet', transformedFlow=None), action='TRANSMITTED')]), Hop(node='internet', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='To-isp_65200', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='13.59.144.125/32', nextHop=NextHopInterface(interface='To-isp_16509', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='To-isp_16509', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='To-isp_16509'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='To-isp_16509', transformedFlow=None), action='TRANSMITTED')]), Hop(node='isp_16509', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='To-Internet', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='Block incoming traffic using reserved addresses', filterType='INGRESS_FILTER', inputInterface='To-Internet', flow=Flow(dscp=0, dstIp='13.59.144.125', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='srv-101', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='203.0.113.12', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='13.59.144.125/32', nextHop=NextHopInterface(interface='To-igw-02fd68f94367a67c7-backbone', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='To-igw-02fd68f94367a67c7-backbone', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='To-igw-02fd68f94367a67c7-backbone'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='To-igw-02fd68f94367a67c7-backbone', transformedFlow=None), action='TRANSMITTED')]), Hop(node='igw-02fd68f94367a67c7', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='backbone', inputVrf='default'), action='RECEIVED'), Step(detail=TransformationStepDetail(transformationType='DEST_NAT', flowDiffs=[FlowDiff(fieldName='dstIp', oldValue='13.59.144.125', newValue='10.20.1.207')]), action='TRANSFORMED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.0.0/16', nextHop=NextHopInterface(interface='vpc-0574d08f8d05917e4', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='vpc-0574d08f8d05917e4', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='vpc-0574d08f8d05917e4'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='vpc-0574d08f8d05917e4', transformedFlow={'dscp': 0, 'dstIp': '10.20.1.207', 'dstPort': 22, 'ecn': 0, 'fragmentOffset': 0, 'ingressNode': 'srv-101', 'ingressVrf': 'default', 'ipProtocol': 'TCP', 'packetLength': 512, 'srcIp': '203.0.113.12', 'srcPort': 49152, 'state': 'NEW', 'tag': 'tag', 'tcpFlagsAck': 0, 'tcpFlagsCwr': 0, 'tcpFlagsEce': 0, 'tcpFlagsFin': 0, 'tcpFlagsPsh': 0, 'tcpFlagsRst': 0, 'tcpFlagsSyn': 1, 'tcpFlagsUrg': 0}), action='TRANSMITTED')]), Hop(node='vpc-0574d08f8d05917e4', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='igw-02fd68f94367a67c7', inputVrf='vrf-igw-02fd68f94367a67c7'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.1.0/24', nextHop=NextHopInterface(interface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7', transformedFlow=None), action='TRANSMITTED')]), Hop(node='subnet-06a692ed4ef84368d', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='acl-09c0bb4e71ae5f9e4_ingress', filterType='INGRESS_FILTER', inputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='srv-101', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='203.0.113.12', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='connected', network='10.20.1.0/24', nextHop=NextHopInterface(interface='to-instances', ip=None, type='interface'), nextHopIp=None, admin=0, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='to-instances', resolvedNextHopIp=None, type='ForwardedOutInterface'), arpIp=None, outputInterface='to-instances'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='to-instances', transformedFlow=None), action='TRANSMITTED')]), Hop(node='i-01602d9efaed4409a', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='eni-01997085076a9b98a', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='~INGRESS_ACL~eni-01997085076a9b98a', filterType='INGRESS_FILTER', inputInterface='eni-01997085076a9b98a', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=22, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface=None, ingressNode='srv-101', ingressVrf='default', ipProtocol='TCP', packetLength=512, srcIp='203.0.113.12', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=SetupSessionStepDetail(sessionScope=OriginatingSessionScope(originatingVrf='default'), sessionAction=ForwardOutInterface(nextHopHostname='subnet-06a692ed4ef84368d', nextHopInterface='to-instances', outgoingInterface='eni-01997085076a9b98a'), matchCriteria=SessionMatchExpr(ipProtocol='TCP', srcIp='10.20.1.207', dstIp='203.0.113.12', srcPort=22, dstPort=49152), transformation=[]), action='SETUP_SESSION'), Step(detail=InboundStepDetail(interface='eni-01997085076a9b98a'), action='ACCEPTED')])])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# traceroute from DC host to an instances using public IP\n", "ans = bf.q.traceroute(startLocation=\"srv-101\", \n", " headers=HeaderConstraints(dstIps=public_ips[\"east2_public\"],\n", " applications=\"ssh\")).answer()\n", "show_first_trace(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now see that the traffic traverses the Internet via `isp_65200` and the Internet gateway (`igw-02fd68f94367a67c7`), which NATs the destination address of the packet from the public to the private IP." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluating the network's availability and security\n", "\n", "In addition to helping you understand and debug network paths, Batfish can also help ensure that the network is correctly configured with respect to its availability and security policies.\n", "\n", "As examples, the queries below evaluate which instances are or are not reachable from the Internet." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Instances reachable from the Internet: ['east2_public', 'west2_public']\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Instances NOT reachable from the Internet: ['east2_private', 'west2_private']\n" ] } ], "source": [ "# compute which instances are open to the Internet\n", "reachable_from_internet = [key for (key, value) in hosts.items() if is_reachable(\"internet\", value)]\n", "print(\"\\nInstances reachable from the Internet: {}\".format(sorted(reachable_from_internet)))\n", "\n", "# compute which instances are NOT open to the Internet\n", "unreachable_from_internet = [key for (key, value) in hosts.items() if not is_reachable(\"internet\", value)]\n", "print(\"\\nInstances NOT reachable from the Internet: {}\".format(sorted(unreachable_from_internet)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that Batfish correctly computes that the two instances in the public subnets are accessible from the Internet, and the other two are not. \n", "\n", "We can compare the answers produced by Batfish to what is expected based on network policy. This comparison can ensure that all instances that are expected to host public-facing services are indeed reachable from the Internet, and all instances that are expecpted to host private services are indeed not accessible from the Internet.\n", "\n", "We can similarly compute which instances are reachable from hosts in the datacenter, using the query like the following." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Instances reachable from the DC: ['east2_private', 'east2_public', 'west2_private', 'west2_public']\n" ] } ], "source": [ "# compute which instances are reachable from data center\n", "reachable_from_dc = [key for (key,value) in hosts.items() if is_reachable(\"srv-101\", value)]\n", "print(\"\\nInstances reachable from the DC: {}\".format(sorted(reachable_from_dc)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that all four instances are accessible from the datacenter host.\n", "\n", "Batfish allows a finer-grained evaluation of security policy as well. In our network, our intent is that the public instances should only allow SSH traffic. Let us see if this invariant actually holds." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Instances reachable from the Internet with non-SSH traffic: ['east2_public']\n" ] } ], "source": [ "tcp_non_ssh = HeaderConstraints(ipProtocols=\"tcp\", dstPorts=\"!22\")\n", "reachable_from_internet_non_ssh = [key for (key, value) in hosts.items() \n", " if is_reachable(\"internet\", value, tcp_non_ssh)]\n", "print(\"\\nInstances reachable from the Internet with non-SSH traffic: {}\".format(\n", " sorted(reachable_from_internet_non_ssh)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that, against our policy, the public-facing instance allows non-SSH traffic. To see examples of such traffic, we can run the following query. " ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Flow: start=internet interface=out [8.8.8.8:49152->13.59.144.125:3306 TCP (SYN)]'" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "ACCEPTED
1. node: internet
  RECEIVED(out)
  FORWARDED(Forwarded out interface: To-isp_16509 with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 13.59.144.125/32, Next Hop: interface To-isp_16509 ip 169.254.0.1)])
  TRANSMITTED(To-isp_16509)
2. node: isp_16509
  RECEIVED(To-Internet)
  PERMITTED(Block incoming traffic using reserved addresses (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: To-igw-02fd68f94367a67c7-backbone with resolved next-hop IP: 169.254.0.1, Routes: [bgp (Network: 13.59.144.125/32, Next Hop: interface To-igw-02fd68f94367a67c7-backbone ip 169.254.0.1)])
  TRANSMITTED(To-igw-02fd68f94367a67c7-backbone)
3. node: igw-02fd68f94367a67c7
  RECEIVED(backbone)
  TRANSFORMED(DEST_NAT dstIp: 13.59.144.125 -> 10.20.1.207)
  FORWARDED(Forwarded out interface: vpc-0574d08f8d05917e4 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.0.0/16, Next Hop: interface vpc-0574d08f8d05917e4 ip 169.254.0.1)])
  TRANSMITTED(vpc-0574d08f8d05917e4)
4. node: vpc-0574d08f8d05917e4
  RECEIVED(igw-02fd68f94367a67c7)
  FORWARDED(Forwarded out interface: subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7 with resolved next-hop IP: 169.254.0.1, Routes: [static (Network: 10.20.1.0/24, Next Hop: interface subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7 ip 169.254.0.1)])
  TRANSMITTED(subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7)
5. node: subnet-06a692ed4ef84368d
  RECEIVED(vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7)
  PERMITTED(acl-09c0bb4e71ae5f9e4_ingress (INGRESS_FILTER))
  FORWARDED(Forwarded out interface: to-instances, Routes: [connected (Network: 10.20.1.0/24, Next Hop: interface to-instances)])
  TRANSMITTED(to-instances)
6. node: i-01602d9efaed4409a
  RECEIVED(eni-01997085076a9b98a)
  PERMITTED(~INGRESS_ACL~eni-01997085076a9b98a (INGRESS_FILTER))
  SETUP_SESSION(Originating VRF: default, Action: ForwardOutInterface(Next Hop: subnet-06a692ed4ef84368d, Next Hop Interface: to-instances, Outgoing Interface: eni-01997085076a9b98a), Match Criteria: [ipProtocol=TCP, srcIp=10.20.1.207, dstIp=8.8.8.8, srcPort=3306, dstPort=49152])
  ACCEPTED(eni-01997085076a9b98a)" ], "text/plain": [ "Trace(disposition='ACCEPTED', hops=[Hop(node='internet', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='out', inputVrf='default'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='13.59.144.125/32', nextHop=NextHopInterface(interface='To-isp_16509', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='To-isp_16509', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='To-isp_16509'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='To-isp_16509', transformedFlow=None), action='TRANSMITTED')]), Hop(node='isp_16509', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='To-Internet', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='Block incoming traffic using reserved addresses', filterType='INGRESS_FILTER', inputInterface='To-Internet', flow=Flow(dscp=0, dstIp='13.59.144.125', dstPort=3306, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface='out', ingressNode='internet', ingressVrf=None, ipProtocol='TCP', packetLength=512, srcIp='8.8.8.8', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='bgp', network='13.59.144.125/32', nextHop=NextHopInterface(interface='To-igw-02fd68f94367a67c7-backbone', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=20, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='To-igw-02fd68f94367a67c7-backbone', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='To-igw-02fd68f94367a67c7-backbone'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='To-igw-02fd68f94367a67c7-backbone', transformedFlow=None), action='TRANSMITTED')]), Hop(node='igw-02fd68f94367a67c7', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='backbone', inputVrf='default'), action='RECEIVED'), Step(detail=TransformationStepDetail(transformationType='DEST_NAT', flowDiffs=[FlowDiff(fieldName='dstIp', oldValue='13.59.144.125', newValue='10.20.1.207')]), action='TRANSFORMED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.0.0/16', nextHop=NextHopInterface(interface='vpc-0574d08f8d05917e4', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='vpc-0574d08f8d05917e4', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='vpc-0574d08f8d05917e4'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='vpc-0574d08f8d05917e4', transformedFlow={'dscp': 0, 'dstIp': '10.20.1.207', 'dstPort': 3306, 'ecn': 0, 'fragmentOffset': 0, 'ingressInterface': 'out', 'ingressNode': 'internet', 'ipProtocol': 'TCP', 'packetLength': 512, 'srcIp': '8.8.8.8', 'srcPort': 49152, 'state': 'NEW', 'tag': 'tag', 'tcpFlagsAck': 0, 'tcpFlagsCwr': 0, 'tcpFlagsEce': 0, 'tcpFlagsFin': 0, 'tcpFlagsPsh': 0, 'tcpFlagsRst': 0, 'tcpFlagsSyn': 1, 'tcpFlagsUrg': 0}), action='TRANSMITTED')]), Hop(node='vpc-0574d08f8d05917e4', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='igw-02fd68f94367a67c7', inputVrf='vrf-igw-02fd68f94367a67c7'), action='RECEIVED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='static', network='10.20.1.0/24', nextHop=NextHopInterface(interface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7', ip='169.254.0.1', type='interface'), nextHopIp=None, admin=1, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7', resolvedNextHopIp='169.254.0.1', type='ForwardedOutInterface'), arpIp='169.254.0.1', outputInterface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='subnet-06a692ed4ef84368d-vrf-igw-02fd68f94367a67c7', transformedFlow=None), action='TRANSMITTED')]), Hop(node='subnet-06a692ed4ef84368d', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='acl-09c0bb4e71ae5f9e4_ingress', filterType='INGRESS_FILTER', inputInterface='vpc-0574d08f8d05917e4-vrf-igw-02fd68f94367a67c7', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=3306, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface='out', ingressNode='internet', ingressVrf=None, ipProtocol='TCP', packetLength=512, srcIp='8.8.8.8', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=RoutingStepDetail(routes=[RouteInfo(protocol='connected', network='10.20.1.0/24', nextHop=NextHopInterface(interface='to-instances', ip=None, type='interface'), nextHopIp=None, admin=0, metric=0)], forwardingDetail=ForwardedOutInterface(outputInterface='to-instances', resolvedNextHopIp=None, type='ForwardedOutInterface'), arpIp=None, outputInterface='to-instances'), action='FORWARDED'), Step(detail=ExitOutputIfaceStepDetail(outputInterface='to-instances', transformedFlow=None), action='TRANSMITTED')]), Hop(node='i-01602d9efaed4409a', steps=[Step(detail=EnterInputIfaceStepDetail(inputInterface='eni-01997085076a9b98a', inputVrf='default'), action='RECEIVED'), Step(detail=FilterStepDetail(filter='~INGRESS_ACL~eni-01997085076a9b98a', filterType='INGRESS_FILTER', inputInterface='eni-01997085076a9b98a', flow=Flow(dscp=0, dstIp='10.20.1.207', dstPort=3306, ecn=0, fragmentOffset=0, icmpCode=None, icmpVar=None, ingressInterface='out', ingressNode='internet', ingressVrf=None, ipProtocol='TCP', packetLength=512, srcIp='8.8.8.8', srcPort=49152, tcpFlagsAck=0, tcpFlagsCwr=0, tcpFlagsEce=0, tcpFlagsFin=0, tcpFlagsPsh=0, tcpFlagsRst=0, tcpFlagsSyn=1, tcpFlagsUrg=0)), action='PERMITTED'), Step(detail=SetupSessionStepDetail(sessionScope=OriginatingSessionScope(originatingVrf='default'), sessionAction=ForwardOutInterface(nextHopHostname='subnet-06a692ed4ef84368d', nextHopInterface='to-instances', outgoingInterface='eni-01997085076a9b98a'), matchCriteria=SessionMatchExpr(ipProtocol='TCP', srcIp='10.20.1.207', dstIp='8.8.8.8', srcPort=3306, dstPort=49152), transformation=[]), action='SETUP_SESSION'), Step(detail=InboundStepDetail(interface='eni-01997085076a9b98a'), action='ACCEPTED')])])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ans = bf.q.reachability(pathConstraints=PathConstraints(startLocation=\"internet\", \n", " endLocation=hosts[\"east2_public\"]),\n", " headers=tcp_non_ssh).answer()\n", "show_first_trace(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We thus see that our misconfigured public instance allows TCP traffic to port 3306 (MySQL).\n", "\n", "In this and earlier reachability queries, we are not specifying anything about the flow to Batfish. It automatically figures out that the flow from the Internet that can reach `hosts[\"east2_public\"]` must have `13.59.144.125` as its destination address, which after NAT'ing becomes the private IP of the instance. Such exhaustive analysis over all possible header spaces is unique to Batfish, which makes it an ideal tool for comprehensive availability and security analysis.\n", "\n", "Batfish can also diagnose *why* certain traffic makes it past security groups and network ACLs. For example, we can run the testFilters question as below to reveal why the flow above made it past the security group on `hosts[\"east2_public\"]`." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 NodeFilter_NameFlowActionLine_ContentTrace
0i-01602d9efaed4409a~INGRESS_ACL~eni-01997085076a9b98aStart Location: i-01602d9efaed4409a
Src IP: 8.8.8.8
Src Port: 49152
Dst IP: 10.20.1.207
Dst Port: 3306
IP Protocol: TCP (SYN)		PERMITSecurity Group launch-wizard-1
  • Matched security group launch-wizard-1
    • Matched rule with description Connectivity test
      • Matched protocol TCP
      • Matched destination port 3306
      • Matched source address CIDR IP 0.0.0.0/0
\n" ], "text/plain": [ " Node Filter_Name \\\n", "0 i-01602d9efaed4409a ~INGRESS_ACL~eni-01997085076a9b98a \n", "\n", " Flow \\\n", "0 start=i-01602d9efaed4409a [8.8.8.8:49152->10.20.1.207:3306 TCP (SYN)] \n", "\n", " Action Line_Content \\\n", "0 PERMIT Security Group launch-wizard-1 \n", "\n", " Trace \n", "0 - Matched security group launch-wizard-1\n", " - Matched rule with description Connectivity test\n", " - Matched protocol TCP\n", " - Matched destination port 3306\n", " - Matched source address CIDR IP 0.0.0.0/0 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "flow=ans.frame().iloc[0]['Flow'] # the rogue flow uncovered by Batfish above\n", "ans = bf.q.testFilters(nodes=hosts[\"east2_public\"], \n", " filters=\"~INGRESS_ACL~eni-01997085076a9b98a\",\n", " headers=HeaderConstraints(srcIps=flow.srcIp, \n", " dstIps=\"10.20.1.207\", # destination IP after the NAT at Step 3 above\n", " srcPorts=flow.srcPort, \n", " dstPorts=flow.dstPort, \n", " ipProtocols=flow.ipProtocol)).answer()\n", "show(ans.frame())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The \"Trace\" column shows that the flow was permitted because the security group \"launch-wizard-1\" has a matching rule called \"Connectivity test.\" (Perhaps someone added this rule to test connectivity but forgot to remove it.) \n", "\n", "Such introspection capability is indispensable for complex security groups and network ACLs. See [this notebook](https://github.com/batfish/pybatfish/tree/master/jupyter_notebooks/Analyzing%20ACLs%20and%20Firewall%20Rules.ipynb) for a more detailed illustration of these capabilities of Batfish." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "Batfish allows you to analyze, debug, and secure your cloud and hybrid networks. It can shed light on different types of traffic paths between different types of endpoints (e.g., intra-region, cross-region, across hybrid links), and it can reveal the detailed availability and security posture of the network. \n", "\n", "Want to learn more? Come find us on [Slack](https://join.slack.com/t/batfish-org/shared_invite/enQtMzA0Nzg2OTAzNzQ1LTcyYzY3M2Q0NWUyYTRhYjdlM2IzYzRhZGU1NWFlNGU2MzlhNDY3OTJmMDIyMjQzYmRlNjhkMTRjNWIwNTUwNTQ) or [GitHub](https://github.com/batfish/batfish) " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.17" } }, "nbformat": 4, "nbformat_minor": 2 }