What is Netflow? Why should I care? How does it work and what do you not realize about Netflow? We'll go over the fundamentals of Netflow as well as an introduction to network traffic monitoring.   What is the function of Netflow? What are Netflow Surface Scrapers? What is the difference between SNMP & Netflow? What is the difference between Netflow, IPFIX, sFlow, ixFlow, jflow and Netstream? What do Netflow Predictive AI Analytics systems do differently? Can Netflow big-data benefit from machine learning? What are the most common ways that Netflow is used by IT pros? How does CySight use Netflow data to eliminate risks?   We provide answers to these questions and demonstrate how CySight's Deep Netflow analytics with its EDR (End Point Threat Detection) and XDR (Extended Detection and Response) that make use of bigdata to build smart networking patterns that result in autonomous Predictive AI Baselining and the ability to mitigate that may benefit your company.   What is Netflow?   Netflow was initially created by Cisco Systems and is still used today. Netflow is a common protocol utilized to get information on IP traffic flows passing via a network device such as across a router, switch, or host.   As soon as a device is Netflow-enabled, metadata is generated and sent from the device to a flow collector, which stores and analyzes the recorded data for network analysis. With Netflow data, a network administrator may look at the throughput, loss, and congestion at a single interface level. As well as supporting DDoS detection and BGP peering, Netflow data may be used for various network-level monitoring use cases which will be discussed later.   Cisco created Netflow, which measures the amount and kind of traffic passing through a network device. It is the most widely used standard for flow data analytic, to monitor and record every traffic entering or leaving an interface. IPFIX, is a similar Netflow variant based on Netflow Version 9 implementation. The Internet Engineering Task Force (IETF), and network equipment makers have widely adopted the protocol.   Cisco's Netflow network protocol system gathers IP network traffic as it enters or exits an interface as it comes in. If you are using CySight then afterwards, the Netflow data is also processed to produce a visual representation of cyber threats within the deep network traffic volume and flow retained. Read more about this later or download a 30-day free trial and experience CySight's ability to reduce your heavy lifting in finding Cyber and Network issues.   IT professionals utilize the Netflow protocol in conjunction with a solution that can collect and retain network traffic to find out where data is coming from, where it is going, how much data is in it, and where it is going. Understanding the collection and retention capabilities of the solutions it imperative to ensure you have tools that work for you rather than against your interests.   There is a lot of traffic flowing through a network, so it's important to have a clear picture of where it is coming from, going to, and how much traffic is created at any one moment. Usage monitoring, anomaly detection, and other network management duties can all benefit from the recorded data, but it is subject to the level of flow data that can actually be retained by a netflow collector.   A flow must be defined first to properly configure Netflow and its helpful to know why you need it as although it is utilized for network monitoring different requirements can be challenging for a typical netflow analyzer.   The term "connections" refers to the pathways or routes that computers use to communicate with one another when they need to do so. If the TCP protocol is being used, these communication channels should be referred to as connections and any communication route that has the appearance of a connection, or is otherwise functionally similar to a connection, is referred to as a flow.   The source and destination IP addresses exchange information to describe a flow in more technical terms: the 5-tuple, a collection of five data points. If there are any ports involved, the source and destination addresses, and the protocol. Each communication channel has a unique flow identifier, and any packets that have the same 5-tuple fields are considered to be part of that flow.   To gather and analyze data, the Netflow protocol is built into network devices to collect and export flow measurements. The timestamp of the first and last packets in a flow (and hence the flow's duration), the total amount of bytes and packets transferred, number of conversations, latency, drops, time to live and a summary of the TCP flags used are only some of the information that Netflow records.   Using CySight, your knowledge of how the network is utilized will grow as you gather and analyze this flow data. You may use CySight's deep flow analysis built on big-data and autonomous predictive AI baselining, to detect which foreign IPs or nations are attacking your organization or exfiltrating your data and mitigate this and at the same time still do the NOC basic and fix network difficulties, discover video steaming and other undesirable traffic at a depth and speed.   What are the benefits of using Netflow?   Netflow should be used because it provides you network awareness, to put it plainly. One of the most often used network visibility technologies in IT service management is network traffic analysis.   It's incomplete when Netflow is collected by surface level scrapers but its also a lot better than a lot of the other options out there. Deep packet inspection is another option but its expensive to deploy and maintain and places a burden on the network and its prime benefit is limited to focus on discovering packet abnormalities, which we cover in some detail in the section on Netflow's and Netflow surface level scraper analyzer's limitations.   Using flow analysis has the apparent advantage of saving you time because you already have the flow tapping technology already built into many routers, switches, firewalls, packet brokers and virtual machine. To use Netflow all you have to do is enable it and point the output to a netflow collector and pretty soon you are going to get some information about the device and data flowing through it.   Installing flow analysis is simple and low-cost. It is quite rare to require additional hardware and it takes a few minutes to configure a few nodes on the network, and there is minimal risk of downtime. Adding flow analysis to your network in a flash will bring quick visibility into traffic patterns.   Hopefully with the information we have provided in this whitepaper you will become more adept in being able to distinguish the value provided by the Netflow surface level scrapers and CySight's extreme visibility, predictive AI baselining and end point detection and response.   Several applications benefit from netflow statistics.   Businesses and users may use Netflow to visualize traffic patterns throughout the whole network using flow-based analytic techniques to identify bottlenecks and for bandwidth analysis. Bandwidth analysis is the most common legacy use case which has largely fallen away because in most countries bandwidth has become cheap.   Network operations (NetOps) and security operations (SecOps) teams may monitor when and how often users hit and undesirable endpoint, manage Distributed Denial of Service (DDoS) or visit a network application using this overall view of traffic flow.   Looking for security or policy breaches or watch and forecast network growth using Netflow data and discover individual servers' baselines that provide both security and trending benefits. It has become possible for example with CySight's Netflow detection for teams enables even an individual user's consumption of network and application resources to be profiled.   Using Netflow, for example, you may anticipate future demand by increasing the number of ports, routing devices, or segmenting your business or network or protecting business assets from prying eyes or monitoring when important files leave the building to the basic monitoring of high-bandwidth connections.   Observing network traffic yields information on both the underlying data transfers and the communication's subject when detached from packet content. The data may also be used as a forensic tool to learn from security incidents by understanding and replaying their history.   Netflow's progress over time   As a software approach for summarizing network flow statistics for packets transmitted through Cisco equipment, Netflow was first released in 1995 as part of Cisco routers. Due to the fact that it was designed for local area networks (LANs), it was later superseded by another method known as express forwarding.   Cisco, on the other hand, saw the value in having access to network flow data and began implementing Netflow in their network gear. Other companies have subsequently replicated Netflow, adopting slightly different names for their flavor of flow analysis to avoid trademark concerns, and it has since become the de facto industry standard. KeySight for example has Ixia IxFlow, Juniper has J-Flow, Huawei has NetStream; sFlow is also used by a variety of vendors. As we'll see in the section sFlow versus Netflow, sFlow is significantly distinct from the rest of the pack. IPFIX, because of its resemblance to Netflow v9, is frequently referred to as Netflow v10. As flow-based monitoring protocols grew so widespread, even the IETF issued IPFIX in 2008 as an official industry standardization of Netflow.   The evolution of Netflow has been fascinating as well. Once a static protocol that gathered the same statistics for all flows, it has evolved into something that can be extended. Today vendors can add extensions to bring their proprietary deep data analytics or other collated information to flow entries in a flexible netflow version and the user can pick which statistics to enable.   Because it's still the most frequently used standard, we'll use Netflow terminology throughout our whitepaper, but the points we make apply to all other flow protocols as well.   Information gleaned through the use of IPFIX.   Various counter metrics such as packets and octets are traditionally aggregated along with the five tuple Source IP, Destination IP, Source Port, Destination Port and Protocol. Other metrics and ratios (such as host name, average round trip time, and average server response time), latency, and other metrics and ratios, can give unique visibility in addition to Netflow statistics.   With regard to Netflow formats, most people are using Netflow v5. The independent IPFIX format has evolved among a range of proprietary formats such as jflow, sflow, ixflow or Netstream to satisfy the demand for extracting a broader data set.   Although it is customary to refer to all kinds of flow records and datagrams as "Netflow," however active production networks often employ three significant variants which are similar in terms of methodology and structure, IPFIX is quite similar to Netflow v9, an IETF standard flow record format. sFlow, an alternative flow protocol and data record standard created and supported by InMon Corp and IPFIX often referred to as "Netflow v10" since IPFIX plays a crucial role in consolidating all Netflow variations and equivalents throughout time as the standards process improves the IPFIX specifications.   Unlike Netflow, sflow does not take packet samples or timestamp traffic flows. There is more data to analyze but it is therefore a very accurate and reliable protocol but highlights the inefficiencies of Netflow surface scraping solutions which are served better by using statistical sampling methods to document flows. As already stated, Netflow has become the generic term often used to refer collectively to all of the flow record variants; Netflow, sFlow, IPFIX, ixFlow and even J-Flow; however, this is changing as extended Netflow and IPFIX support becomes more common. ixFlow and similar flow exports from Packet Brokers are extending the concept of flow by adding Applications Intelligence metadata into the flow solution thereby enabling the extended analysis of protocols such as DNS, Radius, SSL, Email, HTTP but also now extending into the applications arena to monitor login and file transfers via social networking or video streams and other public domain information. The balance between privacy and security is already challenged by DPI and those solutions that allow the user to have control over what is sent or what is collected will go a long way towards balancing cyber security intelligence requirements for scrutiny and individual privacy.   Netflow v5, v9, and IPFIX are the most frequent sources of flow data. In addition to Cisco Netflow v5 and v9, two of the most widely used network protocol systems, CySight can gather network traffic measurements from any flow protocol and offers the broadest support for this.   Netflow v5 protocol was basically a straightforward export data with fields defined in a specified position similar to a comma delimited file format (CSV) The template-based v9 protocol, on the other hand, gives you additional format options as the template or header of the flow data is sent first and the data for the specified format then follows. IPFIX template structure has now become a standard for exporting IP network flow data to a collector device and formatting and transferring that data.   Using CySight, you'll be able to keep tabs on all fields provided in the flow data and all measurement metrics and much more. As additional fields from various suppliers become available, CySight easily provides support for all of them, expanding to value views much beyond the typical Netflow.   In general, Netflow monitoring systems are made up of three major parts: Exporter (Netflow-enabled device), Collector (Flow Collector), and Monitor (NetFlow-enabled device). Server-based flow collector software collects that flow data from Netflow-enabled devices, stores it, and performs preprocessing on it before sending it to an analysis application.   There are multiple reasons why Netflow analyzer solutions fail to provide appropriate levels of flow data and why not all netflow collectors are created equal. In a sense, its because some inaccurate assumptions and expectations about the benefits and capabilities of Netflow have been filtered across the networking and security engineering community due to the original netflow analyzer and netflow collector technologies were only ever able to surface scrape from the rich data that Netflow provides.   Unlike other solutions, CySight make full use of Netflow data employing machine learning that extends its abilities from building big-data lakes that are retained in patent designed small footprints , It then qualifies the information using additional machine learning filtering and AI diagnostics to allow new deep insights and correlations to enhance defense in depth from this most valuable Deep Netflow resource recognized by the SANS Institute as one of the most important and most cost-effective means to analyze and protect your network and network connected assets.   You can read more about these in the CySight whitepapers https://cysight.ai   Network management protocol (SNMP) in comparison to other protocols like Netflow, IPFIX, and sFlow and ixFlow   Before Netflow, network managers and engineers analyzed and monitored network traffic using the Simple Network Management Protocol (SNMP). In terms of network monitoring and capacity planning, SNMP worked well, but it didn't provide you a lot of information about how much bandwidth you were using.   Network monitoring has long relied on the Simple Network Management Protocol (SNMP), which is now in its third major iteration, SNMPv3. SNMP is similar to Netflow in that it contains agents (which are like flow exporters) and supervisors (analogous to a flow collector).   There are some parallels between Netflow and SNMP, but where they diverge is where the real interest lies. While both may be used to keep tabs on a network's throughput, only the data passed via the Netflow protocol has the potential show you exactly what and where the traffic is going.   Data gathering, on the other hand, differs slightly. Data from the flow exporter to the flow collector can only be sent using Netflow in one method. An active data transfer occurs between the flow exporter and the flow collector. There are two ways to get data from an agent to a manager in contrast with SNMP. SNMP can use traps, a manager push notice, or SNMP polls, a management pull request to the agent. Polling SNMP devices for performance data is the most common method used by network management systems.   Additionally, Netflow only monitors traffic flowing via a network device, as opposed to SNMP, which monitors all network traffic passing through a network device. In addition to CPU and memory usage, SNMP allows network managers to gather data on additional device performance indicators such as fan speed and temperature.   When Netflow appeared, bandwidth was expensive, and it therefore made sense to build new kinds of network performance solutions to find the bandwidth hogs. Today the world is awash with Netflow tools that still promise the extended value that they cannot provide but because they predicate it with Netflow that as a protocol coupled with the right kind of Netflow collector analyzer could provide more visibility than they had it must be right. No?   The reality is that engineers depend on marketing to make decisions and don’t realize they are being cheated out of real visibility through inaccurate algorithms based on surface level analytics. So, while you are looking at the inaccurate bandwidth alerts those systems are giving you, hackers are stealing your intellectual property, holding you to ransom and real network issues are occurring causing you inefficiencies and costing you money. Netflow analyzers abound that "talk the talk", but no number of fancy graphics can make what they don’t collect suddenly appear! In fact, SNMP with RMON compared to Netflow surface scrapers abilities is not much better than the average Netflow analyzer.   Existing network management and network security point solutions are facing a major challenge due to the increasing complexity of the IT infrastructure. Over 95% of network and cyber visibility technologies retain only 2% to 5% of all data acquired, resulting in significantly inaccurate analytics and risk!   This is because those vendors' products are designed to capture the surface level of network communication flow records orientated to a single value proposition and are not constructed to retain the critical flow records present in a typical medium to large enterprise, campus, or ISP.   Spend a little bit of time checking under the hood and kicking the tires of the CySight solution you will be pleasantly surprised to find a scalable stable solution that not only easily scales to consume and retain substantially more flow data than other solutions but also provides machine learning and automation that will reduce your time in finding the real underlying causes of what's impacting your network and mapping risks and real threats without breaking the bank.   Every minute counts when resolving IT incidents and Security Risks and assessing the impact to the business. CySight's smart network predictive AI baselining solution continues to generate actionable insight by delivering the right monitoring information to the right teams at the right time.   The SNMP protocol has been around for a long time and is still quite effective when it comes to monitoring bandwidth. Only Netflow and DPI can tell you what your network is being used for and by whom, but SNMP is the most appropriate choice for real-time monitoring. Netflow is suitable for detecting anomalies in complicated networks with heavy traffic but don’t bet your money on most netflow collectors or netflow analyzers ability to show you the information you need.   To give real, real-time visibility into network traffic our flow-based network management software CySight, with in built Deep Flow retention and integrated with Netflow, sFlow, jFlow, IPFIX, AWS, Azure and GCP will give you the information you need and help you mitigate issues before they become an issue. Oh, and we throw in bandwidth performance analysis with full root cause analytics, so you really know what other solutions don’t know what they don’t know. In addition to diagnosing and troubleshooting network slowness and abnormalities, CySight's thorough reports assist you to estimate your future security and bandwidth requirements.   Is it safe to say that SNMP is superior to Netflow in this case?   No, it's not. When it comes to network administration, Netflow and SNMP may operate together since they are complementary in some respects. Netflow fills up the gaps left by SNMP in terms of providing network administrators with visibility into the what and where of traffic. SNMP (Simple Network Management Protocol) is a well-known and straightforward mechanism for managing IT infrastructure resources. Network devices' availability and status are reported using this program (CPU and RAM utilization, how much bandwidth network device consume etc.). The manager receives the requested data from an agent operating on a managed device through SNMP.   Passive, agentless flow data technology such as Netflow, IPFIX, sflow and ixflow are often used for network monitoring but due to scaling are limited to only a small variety of operational and security-related applications.   It is important to appreciate that the scaling issue in Netflow is endemic with Deep Packet Inspection (DPI) solutions that have a tendency to suffer from the same scaling issues and with probably more severity as they have the added tasks of decryption, data tokenizing and storing all that packet data.   Network monitoring with flow produces information on both the underlying data transfers and what's being communicated at the same time. Information on who connects with whom, when, for how long, and how often is provided but the content when based on basic Netflow, the communication is not stored. Only metadata about the information transfer is retained thereby mitigating the legal issues that come with DPI solutions. As pointed out earlier, now that Packet Brokers have entered the Netflow fray there is a new opportunity that provides a DPI experience with Netflow metadata that extends well beyond the traditional IP addresses, data volumes, time, ports, protocols, and other TCP/IP communication characteristics. Cojoining these solutions requires at the outset a scalable Deep Netflow collection and analytics engine that has been unavailable in the market until CySight.   As a test case, think of a circumstance when network traffic suddenly spikes due to an abnormality – in one case, SNMP informs an administrator about the increased number of packets and volume of data being sent across network interfaces. In a second case a Netflow Performance Analyzer just gets the top of the bandwidth ( because it is a surface level scraper) and in the third a Deep Packet Inspector looks at traffic waiting for signatures within the traffic and providing simplistic bandwidth analytics.   Could you identify the root cause of this strange occurrence? What's causing all this extra traffic? Why am I seeing an increase in TCP Acknowledgements when there has not been an increase in Syn (sends). How come there are multiple IP sources sending some small payload to different sensitive parts of my infrastructure? Is there any protocol or service that's being used? Can I detect instantly in real terms which location and systems are being impacted? What's the on effect? What other systems are involved? Are there any End points of ill repute associated with the traffic? Is this a ransomware or a DDoS or perhaps a slowDoS? In fact, could it be identified with a combined effort of all the other tools I already think I've got visibility from?   Traditional SNMP infrastructure monitoring and Netflow surface level scrapers and DPI systems simply cannot address these queries because there are many common network blind spots that are not examined by other solutions as they focus on top of network traffic or focus on micro specks and therefore "can't see the wood for the trees". As a result, Netflow coupled with CySight is the right solution to put to use in this scenario.   Using CySight to keep an eye on Netflow, IPFIX, ixFlow and sFlow   CySight gathers Netflow, IPFIX, sflow, ixflow and other variant data from your flow-enabled infrastructure components as well as processing data from your Amazon and other cloud environments and saves it for future analysis and reporting using its own proprietary network deep flow collection and retention and with predictive AI baselining coupled with End point threat intelligence (EDR) and machine learning, Network and Security operations teams can be alerted (XDR) to and learn who connects with whom and when, for how long, and how frequently.   Determining the origin and the nature and assessing the impact are some of the major visibility issues that are encountered with maintaining service levels, understanding network slowdowns and outages, and detecting, DDoS, Ransomware and other cyber‑attacks and risky traffic.   Most importantly, you need a solid solution that will autonomously analyze network connected assets look for variations in normal network behavior that might indicate a security breach or a system malfunction. Using this information and CySight's forensic tools and smarts will learn from security events by understanding and reliving their history.   Perform Intelligent Baselining, Threat Intelligence, Machine Learning, and A.I. Cyber Forensic Diagnostics analyzing the deepest flow retention of flow data globally and uncover previously unknown threats.   Dropless granular contextual analytics allows you to gain extreme visibility into Network and cloud visibility and security with vendor agnostic and cloud agnostic collection with broadest metadata support.   Identify Ransomware, DDoS, ToR, Bitcoin and other outliers with real-time attack maps.   Unprecedented Network and Cloud Visibility eliminates blindspots and improves Defense in Depth using powerful visualization, forensics and alerting.   Monitor your network to see who is coming in and going out, and who is doing what with your data and sensitive systems. Check who or what IoT devices is communicating with across WiFi points and alert on unwanted MAC Addresses (MAC addresses are primarily assigned by device manufacturers and are therefore often referred to as the burned-in address, or as an Ethernet hardware address, hardware address, or physical address.)   QoS parameters must be validated and confirmed to be reserved for applications that require priority.   Asynchronous Network Numbers and peers must be analyzed to improve business practices and gain costs and inter business relationships.   Naturally, monitor for network all kinds of traffic surges may all be identified, quantified, and remedied using these troubleshooting techniques as slowdowns that today are least likely to be caused by the bandwidth wasters. Bandwidth hogs cause you a different kind of loss that has a value to keep track of.   Understanding performance aspects is of course, still important but CySight correctly rectifies and focuses your vision so you can accurately plan for future needs by understanding what communication resources (not necessarily, but also including bandwidth) each network connected asset such as your servers or instances requires.   All this and so much more. Read more about this later or download a 30-day free trial and experience CySight's ability to reduce your heavy lifting in finding and mitigating Cyber and Network issues.