Rafi Sabel, Author at CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response

Hunt SUNBURST and Trojans with Turbocharged Netflow.

Rafi Sabel — Mon, 10 Mar 2025 12:06:31 +0000

US: December 13 of 2020 was an eye-opener worldwide as Solarwinds software Orion, was hacked using a trojanized update known as SUNBURST backdoor vulnerability. The damage reached thousands of customers, many of which are world leaders in their markets like Intel, Microsoft, Lockheed, Visa, and several USA governmental agencies. The extent of the damage has not been fully quantified as still more is being learned, nevertheless, the fallout includes real-world harm.

The recent news of the SolarWinds Orion hack is very unfortunate. The hack has left governments and customers who used the SolarWinds Orion tools especially vulnerable and the fallout will still take many months to be recognized. This is a prime example of an issue where a flow metadata tool has the inability to retain sufficient records, causing ineffective intelligence, and that the inability to reveal hidden issues and threats is now clearly impacting organizations’ and government networks and connected assets.

Given what we already know and that more is still being learned, it makes good sense to investigate an alternative solution.

What Is the SUNBURST Trojan Attack?

SUNBURST, as named by FireEye, is a kind of malware that acts as a trojan horse designed to look like a safe and trustworthy update for Solarwinds customers. To accomplish such infiltration to seemingly well-protected organizations, the hackers had to first infiltrate the Solarwinds infrastructure. Once Solarwinds was successfully hacked, the bad actors could now rely on the trust between Solarwinds and the targeted organizations to carry out the attack. The malware, which looked like a routine update, was in fact creating a back door, compromising the Solarwinds Orion software and any customer who updates their system.

How was SUNBURST detected?

Initially, SUNBURST malware was completely undetected for some time. The attackers started to install a remote access tool malware into the Solarwinds Orion software all the way back in March of 2020, essentially trojaning them. On December 8, 2020, FireEye discovered their own red team tools have been stolen and started to investigate while reporting the event to the NSA. The NSA, also a Solarwinds software user, who is responsible for the USA cybersecurity defense, was unaware of the hack at the time. A few days later, as soon as the information became more public, different cybersecurity firms began to work on reverse engineering and analyzing the hack.

IT’S WHAT WE DON’T SEE THAT POSES THE BIGGEST THREATS AND INVISIBLE DANGERS!

You may be surprised to learn that most well-known tools lack the REAL Visibility that could have prevented attacks on a network and its local and cloud-connected assets. There are some serious shortcomings in the base designs of other flow solutions that result in their inability to scale in retention. This is why smart analysts are realizing that Threat Intelligence and Flow Analytics today is all about having access to long term granular intelligence.

From a forensics perspective, you would appreciate that you can only analyze the data you retain, and with large and growing network and cloud data flows most tools (regardless of their marketing claims) actually cannot scale in retention and choose to drop records in lieu of what they believe is salient data.

Imputed outcome data leads to misleading results and missing data causes high risk and loss!

A simple way to think about this is if you could imagine trying to collect water from a blasting fire hose into a drinking cup. You just simply cannot collect very much!

Many engineers build scripts to try to attain the missing visibility and do a lot of heavy lifting and then finally come to the realization that no matter how much lifting you do that if the data ain’t there you can’t analyze it.

We found that over 95% of network and cyber visibility tools retain as little as 2% to 5% of all information collected resulting in completely missed analytics, severely misleading analytics, and risk!

How does CySight hunt SUNBURST and other Malware?

It’s often necessary to try and look back with new knowledge that we become aware of to analyze.

For a recently discovered Ransomware or Trojan, such as SUNBURST, it is helpful to see if it’s been active in the past and when it started. Another example is being able to analyze all the related traffic and qualify how long a specific user or process has been exfiltrating an organization’s Intellectual Property and quantify the risk.

SUNBURST enabled the criminals to install a Remote Access Trojan (RAT). RATs, like most Malware, are introduced as part of legitimate-looking files. Once enabled they allow the hacker to view a screen or a terminal session enabling them to look for sensitive data like customer’s credit cards, intellectual property or sensitive company or government secrets.

Even though many antivirus products can identify many RAT signatures, the software and protocols used to view remotely and to exfiltrate files continues to evade many malware detection systems. We must therefore turn to traffic analytics and machine learning to identify traffic behaviors and data movements that are out of the ordinary.

Anonymity by Obscurity

In order to evade detection, hackers try to hide in plain sight and use protocols that are not usually blocked like DNS, HTTP, and Port 443 to exfiltrate your data.

Many methods are used to exfiltrate your data. An often-used method is to use p2p technologies to break files into small pieces and slowly send the data unnoticed by other monitoring systems. Due to CySight’s small footprint Dropless Collection you can easily identify sharding and our anomaly detection will identify the outlier traffic and quickly bring it to your attention. When used in conjunction with a packet broker partner such as Keysight, Gigamon, nProbe or other supported packet metadata exporter, CySight provides the extreme application intelligence to help you with complete visibility to control the breach.

Identifying exposure

In todays connected world every incident has a communications component

You need to keep in mind that all Malware needs to “call home” and today this is going to be through onion routed connections, encrypted VPNs, or via zombies that have been seeded as botnets making it difficult if not impossible to identify the hacking teams involved which may be personally, commercially or politically motivated bad actors.

Multi-focal threat hunting

Threat hunting for SUNBURST or other Malware requires multi-focal analysis at a granular level that simply cannot be attained by sampling methods. It does little good to be alerted to a possible threat without having the detail to understand context and impact. The Hacker who has control of your system will likely install multiple back-doors on various interrelated systems so they can return when you are off guard.

CySight Turbocharges Flow and Cloud analytics for SecOps and NetOps

As with all CySight analytics and detection, you don’t have to do any heavy lifting. We do it all for you!

There is no need to create or maintain special groups with Sunburst or other Malware IP addresses or domains. Every CySight instance is built to keep itself aware of new threats that are automatically downloaded in a secure pipe from our Threat Intelligence qualification engine that collects, collates and categorizes threats from around the globe or from partner threat feeds.

CySight Identifies your systems conversing with Bad Actors and allows you to back track through historical data to see how long it’s been going on.

Using Big Data threat feeds collated from multiple sources, thousands of IPs of bad reputation are correlated in real-time with your traffic against threat data that is freshly derived from many enterprises and sources to provide effective visibility of threats and attackers.

Cyber feedback
Global honeypots
Threat feeds
Crowd sources
Active crawlers
External 3rd Party

So how exactly do you go about turbocharging your Flow and Cloud metadata?

CySight software is capable of the highest level of granularity, scalability, and flexibility available in the network and cloud flow metadata market. Lack of granular visibility is one of, if not the main flaw in such products today as they retain as little as 2% to 5% of all information collected, due to inefficient design, severely impacting visibility and risk as a result of missing and misleading analytics, costing organizations greatly.

CySight’s Intelligent Visibility, Dropless Collection, automation, and machine intelligence reduce the heavy lifting in alerting, auditing, and discovering your network making performance analytics, anomaly detection, threat intelligence, forensics, compliance, zero trust and IP accounting and mitigation a breeze!

The post Hunt SUNBURST and Trojans with Turbocharged Netflow. appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

End Point Threat Detection Using NetFlow Analytics

Rafi Sabel — Fri, 21 Feb 2025 08:47:11 +0000

So, with that we’re going to get started. Again, we appreciate everyone taking the time today to listen to what we have to say and learn about our product, and learn about some of the new features. If you’re on here and you’re an existing customer, that you’ll learn a little bit about one of our new features. So, today we’re going to be talking a lot about security, that’s really the focus of this presentation. NetFlow in general, and CySight in particular can do a lot of things with the data that we have, and one of those things is really focused on being able to identify security threats to your network.

This is obviously very important, right? I mean you literally cannot go a day anymore without hearing of some company, some organization out there that’s been attacked or that has been infiltrated. I was reading about a hospital system recently that was held up by a Ransomware company, and actually had to pay money to unlock their files and this is not a home user, this is not a person who opened up the wrong email and their desktop got under attack or held for ransom. This is a legitimate hospital organization that had that happened to them and so, it really underscores the pervasiveness of these kinds of attacks.

Crawlers, botnets, Ransomware, they’re finding new ways to cause denial of service attacks and other kinds of attacks that can put your business or organization at an extremely high risk and, your network could be used to download or host illicit materials, leak intellectual property. That’s another thing that we’ve seen, this sort of cybercrime. Intellectual property cybercrime where it’s not that they’re just trying to bring down your site or bring down your network, but they’re actually trying to take intellectual property out and again, either hold it for ransom or just sell it or whatever it may be. So, this is certainly an important topic.

There are a number of major challenges for security teams to try and figure out what’s going on and how to lock down that network. The sophistication of the cybercrime organizations out there is just growing and growing. They’re always seemingly one step ahead of the for-profit companies that are trying to block them; the anti-virus companies, firewall companies and so forth. The growing complexity of the infrastructure is making it more difficult, there’s not a single point of entry and exit anymore. You’ve got BYOD, you’ve got lots of wireless, you’ve got VPNs, cloud-based services, you’ve got all kinds of things that people are using today. So it’s not just a lock it down at the firewall and we’re good, it’s really all over the place, and you need to be able to look at the traffic to understand what’s going on.

Of course, it’s very difficult or can be very difficult to retain and analyze that network transaction data across a big organization. Again, you have lots of lots of systems, lots of points of entry and exit, and it can be a challenge to really be able to collect all of that data and be able to use it. Because of that, because of the highly complicated and complex nature of networks, we’ve got this graphic here that talks about the really scary things that are out there. About do you know where things are happening? Do you…? You have certain aspects that you know and that you maybe know that you don’t know, but the really scary stuff is when you don’t know what you don’t know, right? It’s happening or could be happening and you have no idea, and you don’t even know that you should be looking at that, or could be looking at that data to try and understand what’s going on.

But in fact, products like ours and technologies like ours, allow you to, or allow a system to be watching for those unknown unknowns all the time. So, it’s not something that you wake up in the morning and say, “I’m going to go, look at this.” It’s actually happening in the background and looking for you. That machine learning capability is really what makes the new level of systems like ours trying… you know being able to catch up with the sophistication of the attack profiles out there.

When there is an attack or when there is a detection of something, then Incident Response Teams always have to look at that communications component, right? So, they’re going to look at hardware, they’re going to look at software, but they also have to look at the communications. They have to look at historical behavior, they have to look to see if there’s been data breaches, they have to look to see if there’s been internal threats.

There is a certain percentage, depending on who you talk to, 30%, 35%, 40% of data breaches happen from the inside out. So, these are internal employees who have access to something that they shouldn’t, and they email that out or they otherwise try to get that data out of the network. Of course, there’s the external threats from bad actors, those malicious types that are probing, probing, probing trying to find holes to get in and do whatever, the nefarious things that they’re trying to do.

So, being able to have some insight into the nature of how those systems, all of your systems communicate with each other and how they have communicated is critical. It’s really about being able to go from the blind area into a much more aware and certain area, right? So, do you really have… and thinking about, do you really have visibility in terms of what’s going on inside your network, because if you don’t, that can certainly hurt you.

The way we look at it, there’s the very basic things that virtually everybody has. Everybody has a firewall, most people have virus protection on their desktops. That sort of blocking and tackling, very basic prevention at the edge of a network is only a piece, right? It is not the most effective place anymore. You have to have it, we certainly wouldn’t tell you not to have it, but if you really want to move to a defense in depth, then it’s more than just trying to put up a blocking of things coming in. It’s being able to look at the live traffic and see what’s happening and identify if there are threats going on that got through. If something gets through the defenses that you have, how can you then further identify that it has happened and what’s going on? If you just think, “Well, I’ve got this firewall and I got my rules setup and I’m good, nothing can ever touch me,” and don’t look any further, then you’re really setting yourself up for a failure.

So, the way we approach the problem as a piece of this overall security landscape, is through the use of NetFlow information. So, NetFlow’s been around for a long time, it’s a quite a mature technology. But the great thing about it is, it’s continually even further maturing as we go on. What used to be sort of a traffic accounting product only, that was based on data coming from core routers and switches, has now been extended out to other systems in the network. Things like wireless LAN controllers, cloud servers, firewalls themselves. You can get the data from taps and probes that collect passively information about data traffic, and then turn that into a NetFlow export that can be sent to us that we can read.

Virtually every vendor… certainly every major vendor out there supports Flow in some way … Cisco of course is NetFlow and we use the term NetFlow to generically mean all of the various Flow types out there. Jflow from Juniper, anything that’s IPFIX compatible as the standard, and some of the other kind of specialized versions of Flow, if you will. But all of them have the common theme that they’re going to look at that traffic and they’re going to be able to send that metadata to a collector like ours and then we can use that information intelligently to help both give you and allow you to report on and look deeply into the data, but also, and what we’re going to be talking about today, is really using that intelligence that’s built into the product to be able to identify threats, look at anomalies. Not just show you who your top talkers were, but actually say, “Hey, look. We’ve identified people that are communicating to known bad actors out there,” or, “We’ve seen an unusual bit of behavior in traffic between here and there, and this is something that really needs to be investigated.”

Talking about more of the specifics about how we do that. There’s two major pieces we’re going to be focusing on today. The first one is Anomaly Detection. Anomaly Detection for us means that we can baseline your network and the traffic on your network across a number of different dimensions. There’s actually quite a few metrics that we’re watching, some of the ones you could see below like flows, and packets, and bytes, and bits per second, packet size, it can be flags, it can be counts it can be all kinds of different metrics, and we can baseline each of them over time, across all of your interfaces or potentially even other aspects. So, it could be a specific conversation or a specific application, but at its most basic level through all of your interfaces to understand what is normal and what is normal activity for that time of day, that day of the week from those devices or whatever it may be.

Then of course, once we know what is normal, we can detect any activity that deviates from that normal baseline, right? This gives you a really great way of watching traffic 24/7 for things that you wouldn’t potentially pick up if you were just you know kind of eyeballing it if you will, or waiting certainly for someone to contact you and say there’s a problem. So, the statistical power of an application to be doing this behind the scenes and running all the time, and noticing things that you wouldn’t notice in the middle of the night, is incredibly useful for this sort of thing and then when we do detect an anomaly, we move into phase two as we call it, into diagnostics? So, diagnostics says, “Okay, there’s been some anomaly that has been detected, let’s look at this. Let’s figure out what’s going on here. We then kick off this diagnostic approach, which qualifies the cause and impact for each offending behavior breach. We’re looking it for KPIs that are specific to things like DOS attacks or scanners or sweepers or peer-to-peer activity. We roll all of that information up into a single ticket so to speak, for you on a screen that you can very easily look at and understand exactly what’s going on. When did it happen? Where did it happen? What was involved? What baseline was breached? What does that mean? What could that possibly be?

You can also do of course advance things like intelligent whitelisting. You can send the information out of our system up to another system that you may have, like an ITSM or trouble ticket system, via SNMP and via email and so forth. So, really this again this is the intelligent piece of the product with machine learning as its background. So it’s doing this whether you’re watching it or not. It’s looking for those baseline breaches and then when we see them, it’s really coordinating all of the information about what happened into a single easy-to-use place, which you can then drill down into using all of our standard features to try and identify other things that are happening or where do you need to go next.

Anomaly Detection or NBAD as you may hear us talk about it, has been in the product for a number of years now. So, that’s not something new, it’s continually being improved, and it’s a wonderful piece of the product, and it’s been there for a while.

The new thing that we have introduced and are introducing is what we call our Endpoint Threat Detection. So this is another module added onto the product that adds additional security capabilities while still utilizing all of the things that you typically utilize. So we’re still taking the data from NetFlow information but now we are applying to that information other outside data sources that we have, basically using some big data threat feeds collated from multiple sources that you can match up to or coordinate with the information about your traffic.

So, I’ve got information about my traffic, I’ve had that. Now, I’ve got information about what is bad in the world and in real time, where known bad actors, known bad IP addresses, Ransomware, malware, DDoS attacks, Tor and so forth are coming from and then looking at the two of them and saying, “Are any of my people talking to those things?” At the very most basic level that’s what we’re looking for, right? So, it’s things global in terms of getting all of these feeds and using pattern matching, and Anomaly Detection and so forth, and then it’s acting very local against the traffic that you have in your network.

This capability of having network connection logging or NetFlow, just as everybody in the industry agrees, is one of the best places that you can get this data. It’s almost impossible to get the kind of granular level of information from any other source. Especially if you are held to any sort of standard in terms of retention or policies around not being able to look directly into the data. If you’ve got compliance requirements that say, “Hey, I can’t store my customers’ data.” That is fine with NetFlow because NetFlow is not looking inside the packets; it’s looking at the metadata. Who’s talking to whom, and when are they doing it, and how much talking are they doing and so forth. But it’s not actually reading an e-mail or anything inside of that. So, you’re not going to run into a foul of any of those regulatory problems, but you’re still able to get a huge amount of benefit from a network investigation using that data.

It’s important that even without content, NetFlow provides an excellent means of guiding that investigation because there’s still so much data there. As it’s called in our world, metadata – Data about the data! There’s still so much information there. But what’s great also is that, you don’t have to retain content… unlike let’s say a probe or other type of system that is collecting every bit and byte. You run into problems there too, they’re expensive, and you run into storage requirements trying to store historically every conversation including the data, over a long period of time is just incredibly expensive and incredibly unwieldy to do. The amount of storage you have to have to be able to do that, and the difficulty in quickly and effectively retrieving that information and searching for things, just becomes next to impossible. But when you can still get the same benefit of what you need to look at from a security standpoint without those complications of price and just the logistics of handling it all, you end up with having a really valuable product and that’s what NetFlow can give to you.

So, with our Endpoint threat Detection, I’ve got a few screens here that can really dive down into what it looks like and how it works. Again, we’ve got these big data feeds of threat information out there in the world, collected from various sources, and honeypots and so forth and we’re continuously then monitoring for communications with those IPs of poor reputation. So, you’ve got your communication that we can see because of NetFlow, and you’ve got these known bad actors out there that we know about. We can match up those two pieces of information and when we do it, we’re not just saying it happened, but we’re giving you much more detail about it happening. So, if we kind of zoom in here a little bit, threat data can be seen in summary or in detail. We’ve got a categorization of what’s happening and different threat types. So, I can see this is a peer-to-peer kind of thing, is this known malware, is it Tor, is it an FTP or an SSH attacker? What kind of thing is happening from or on these known bad IP address?

So, from a high of macro level you can see what the threat categories are and what the threat types are and then of course, you can drill down using the standard CySight tools to investigate them and provide complete visibility into that threat. So, now I’ve seen it, I have traffic that’s been identified as a threat. I can use our drill down, right-click, or however you want to do it capability. In this case we’re showing a right-click on threat detection and saying show me the affected IP addresses. I want to know, let’s drill down and see in this case on Ransomware, command and control Ransomware what the infected IP addresses are and then you’re going to get into the individual affected IPs, the threat IP where it’s coming from and, how much traffic was done?

These are Ransomware-type attacks, and I can see this is happening in my network at this period of time and I can even then of course change the view to be a time view. When did this start? Has this been a long-lived thing that’s been going on over a period of time where it’s been sucking information out of my organization, or did this pop off and go away? And if it did, when did that happen? All of that kind of deep level investigation is something that you can get using all of the normal tools that we have. You can get this deep dive investigation of traffic for regular traffic. Not just malicious traffic, but just using our tool for what I’ll call normal traffic accounting. Who is talking to who and when, is all available to you and more now with the threat detection features.

So, we’re watching for those threats, we’ve identified them and then using all of the common things that you’re used to using if you’re already a customer of ours, being able to identify or drill down into that data and provide those reports when you want to see it.

Here’s another example: let’s look at threat-port usage over the last few hours. So, it’s may be a couple hour time frame and I can see specifically which ports, which protocols have been detected as potential threats. What kind of threats, of course again how much traffic did they use? How long has this gone on for, and so forth. So, you can in fact in this case, know that increasing Tor usage. That we’ve highlighted in yellow and green … but you can also notice it’s been this continual botnet chatter, this red line. It’s just been going on and on forever, and that’s obviously something that needs to be absolutely looked into. It might be very difficult to find this in any other way, it’s just ongoing background chatter that’s been happening. It may not spike to anything that’s incredibly large that would set off a threshold alert, or maybe not even set off an anomaly alert. But, we’ve identified this is being definitely an issue because it’s communicating to something that we know is bad out there.

Of course you have all of the common reporting type tools. So, you can automate those threats, I want a threat report every hour emailed to me, or every day, or whatever makes sense or a roll up report every month to provide to management to say, okay, over the last 30 days, here are all the threats that were identified as happening in our network, and then here’s what’s been remediated, here’s what we’ve blocked, here’s what we’ve stopped, here’s what we’ve fixed, here’s what we’ve cleaned up kind of thing and all of those reports that look good and can be scheduled in a great for both live use and for management, are part of and parcel of the product that we’ve been delivering for over a decade now.

As well as those deep dive threats forensics. So the high level reports are good for some people but the deep dive of course reports are important for other people and that’s something that we can give you because we store an archive all of this flow information, it’s not just the top 100, or the top 500, it’s the top 5,000 or 10,000 or every single Flow using our compliance version. The compliance version store has the ability to store all of those flows all the time for you to pull up and review may not have been yesterday, it may have been last week or last month or six months ago or whenever. You can still drill in, you can still see every individual flow in terms of IPs, source and destination and ports and protocols interfaces and all of that kind of information. It gives you that super granular capability that you’re just not going to find anywhere else.

We also try to give you different viewpoints; we’re very big on flexibility in terms of giving you an easy-to-understand way of looking at the traffic. Some people like to view numbers and other people like to view pictures, and there’s lots of ways that we can show that data to you. The visualization capability is outstanding within our product and one of the ways that that can be really useful. We’ve got this example here of a Tor correlation attack. So, it’s de-anonymizing Tor is a difficult but super important issue within the world of identifying Tor, and so for us, when we see that there has been Tor traffic we can build this visualization and we can see all the different places that that Tor traffic has hopped to within your network or in and out of your network and that really gives you a way to get in and say, “Okay, I need to look here, I need to stop at here, I need to stop at there.” From a service provider perspective, this can be a really, really useful example of what we can do in the power of our product.

So with the last few minutes here, I know we’re getting close to the time frame, but we do want to talk about the many options you have in terms of our scalable architecture. Whether you are small or mid-size organization, or very, very large organization, we have a way of delivering our product to you. It could be in a single standalone environment with a single database and single software installation, it could be as you grow and maybe you have various components of traffic that are disseminated globally, and you need local collection, we can do that. So, we can offer split off collectors or helper collectors that communicate up to a single master database or we can even do multi-site server, multi-database hierarchical architecture for really, really massively scaled organizations. So, no matter who you are, if you’re listening to this, if you’re just small organization with one site and a few devices, or a massively global corporation with thousands of devices and data traversing it in many different areas, we can fit your organization and we can architect a solution that is right for you.

We’ve got a number of exciting features one of the great things about us is that, we never stop developing and we never stop investigating what the best things are to add to the product. We’ve got some really cool enhancements coming on, all things that people have asked about or have inquired about, or we’ve decided to build on our own and we love talking to our customers.

Our best source of future development is request from our customers. So, anything that you can think of I can’t guarantee that that our team will do it, but I can certainly guarantee you that we’ll listen to you and we’ll think about it and we’ll do our absolute best to solve whatever issue you may have and because of our commitment to our customers and our willingness to listen to them, we really have built up a wonderful group of customers. You can see a few of their logos on the screen here again, everything from traditional organizations enterprises to service providers, educational institutions, Telco’s, whatever it may be, we can handle it and we’d love if you’re not already a customer of ours, but you’re listening to this webinar, certainly we’d love to have your logo on this list in the future and we feel like once you get to working with us and really get used to our product, you’re going to be super thrilled about how we do things. What we offer to you and the support we provide to you.

So, with that I think we’re at the end of the presentation, almost exactly right on time here, about 30 minutes. So, I want to thank everyone for taking the time to join today, as always it does not look like we have… I’m just looking. Does not look like we have any questions right now, so, if you do have any now would be the time to type them in. But if not, we just want to thank you for joining us today. This presentation has been recorded and will be available to any of the folks who registered, and it’ll eventually make it up into the website. So, please check it out. Also please check out our website for other information about future webinars or other documentation that we have, there’s a lot of good resources up there and we invite you to take a look at those and certainly if you have any questions to reach out to us either to the sales team or the support or engineering team depending on what you’re interested in.

So, with that, I’ll end the session and I look forward to speaking with all of you at some point in the future.

Thanks.

The post End Point Threat Detection Using NetFlow Analytics appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

5 Ways Flow Based Network Monitoring Solutions Need to Scale

Rafi Sabel — Thu, 19 Sep 2024 10:10:22 +0000

Partial Truth Only Results in Assumptions

A common gripe for Network Engineers is that their current network monitoring solution doesn’t provide the depth of information needed to quickly ascertain the true cause of a network issue. Imagine reading a book that is missing 4 out of every 6 words, understanding the context will be hopeless and the book has near to no value. Many already have over-complicated their monitoring systems and methodologies by continuously extending their capabilities with a plethora of add-ons or relying on disparate systems that often don’t interface very well with each other. There is also an often-mistaken belief that the network monitoring solutions that they have invested in will suddenly give them the depth they need to have the required visibility to manage complex networks.

A best-value approach to NDR, NTA and general network monitoring is to use a flow-based analytics methodology such as NetFlow, sFlow or IPFIX.

The Misconception & What Really Matters

In this market, it’s common for the industry to express a flow software’s scaling capability in flows-per-second. Using Flows-per-second as a guide to scalability is misleading as it is often used to hide a flow collector’s inability to archive flow data by overstating its collection capability and enables them to present a larger number considering they use seconds instead of minutes. It’s important to look not only at flows-per-second but to understand the picture created once all the elements are used together. Much like a painting of a detailed landscape, the finer the brush and the more colors used will ultimately provide the complete and truly detailed picture of what was being looked at when drawing the landscape.

Granularity is the prime factor to start focusing on, specifically referring to granularity retained per minute (flow retention rate). Naturally, speed impediment is a significant and critical factor to be aware of as well. The speed and flexibility of alerting, reporting, forensic depth, and diagnostics all play a strategic role but will be hampered when confronted with scalability limitations. Observing the behavior when impacted by high-flow-variance or sudden-bursts and considering the number of devices and interfaces can enable you to appreciate the absolute significance of scalability in producing actionable insights and analytics. Not to mention the ability to retain short-term and historical collections, which provide vital trackback information, would be nonexistent. To provide the necessary visibility to accomplish the ever-growing number of tasks analysts and engineers must deal with daily along with resolving issues to completion, NDR, NTA and general Network Monitoring System (NMS) must have the ability to scale in all its levels of consumption and retention.

How Should Monitoring Solutions Scale?

Flow-Based Network Detection and Response (NDR) / Network Traffic Analysis (NTA) software needs to scale in its collection of data in five ways:

Ingestion Capability – Also referred to as Collection, means the number of flows that can be consumed by a single collector. This is a feat that most monitoring solutions are able to accomplish, unfortunately, it is also the one they pride themselves on. It is an important ability but is only the first step of several crucial capabilities that will determine the quality of insights and intelligence of a monitoring system. Ingestion is only the ability to take in data, it does not mean “retention”, and therefore could do very little on its own.

Digestion Capability – Also referred to as Retention, means the number of flow records that can be retained by a single collector. The most overlooked and difficult step in the network monitoring world. Digestion / Flow retention rates are particularly critical to quantify as they dictate the level of granularity that allows a flow-based NMS to deliver the visibility required to achieve quality Predictive AI Baselining, Anomaly Detection, Network Forensics, Root Cause Analysis, Billing Substantiation, Peering Analysis, and Data Retention compliance. Without retaining data, you cannot inspect it beyond the surface level, losing the value of network or cloud visibility.

Multitasking Processes– Pertains to the multitasking strength of a solution and its ability to scale and spread a load of collection processes across multiple CPUs on a single server. This seems like an obvious approach to the collection but many systems have taken a linear serial approach to handle and ingest multiple streams of flow data that don’t allow their technologies to scale when new flow generating devices, interfaces, or endpoints are added forcing you to deploy multiple instances of a solution which becomes ineffective and expensive.

Clustered Collection – Refers to the ability of a flow-based solution to run a single data warehouse that takes its input from a cluster of collectors as a single unit as a means to load balance. In a large environment, you mostly have very large equipment that sends massive amounts of data to collectors. In order to handle all that data, you must distribute the load amongst a number of collectors in a cluster to multiple machines that make sense of it instead of a single machine that will be overloaded. This ability enables organizations to scale up in data use instead of dropping it as they attempt to collect it.

Hierarchical Correlation – The purpose of Hierarchical correlation is to take information from multiple databases and aggregate them into a single Super SIEM. With the need to consume and retain huge amounts of data, comes the need to manage and oversee that data in an intelligent way. Hierarchical correlation is designed to enable parallel analytics across distributed data warehouses to aggregate their results. In the field of network monitoring, getting overwhelmed with data to the point where you cannot find what you need is a as useful as being given all the books in the world and asked a single question that is answered in only one.

Network traffic visibility is considerably improved by reducing network blindspots and providing qualified sources and reasons of communications that impair business continuity.The capacity to capture flow at a finer level allows for new Predictive AI Baselining and Machine Learning application analysis and risk mitigation.

There are so many critical abilities that a network monitoring solution must enable its user, all are affected by whether or not the solution can scale.

Visibility is a range and not binary, you do not have or don’t have visibility, its whether you have enough to achieve your goals and keep your organization productive and safe.

The post 5 Ways Flow Based Network Monitoring Solutions Need to Scale appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

How NetFlow Solves for Mandatory Data Retention Compliance

Rafi Sabel — Thu, 08 Aug 2024 11:24:12 +0000

Compliance in IT is not new and laws regulating how organizations should manage their customer data exist such as: HIPPA, PCI, SCADA and Network transaction logging has begun to be required of business. Insurance companies are gearing up to qualify businesses by the information they retain to protect their services and customer information. Government and industry regulations and enforcement are becoming increasingly stringent.

Most recently many countries have begun to implement Mandatory Data Retention laws for telecom service providers.

Governments require a mandatory data retention scheme because more and more crime is moving online from the physical world and ISP‘s are keeping less data and retaining it for a shorter time. This negatively impacts the investigative capabilities of law enforcement and security agencies that need timely information to help save lives by early spotting lone-wolf terrorists or protect vulnerable members of society from abuse by sexual deviants, ransomware or other crimes online.

Although there is no doubt as to the value of mandatory data retention schemes they are not without justifiable privacy, human rights and expense concerns to implement.

It takes a lot of cash, time and skills that many ISP’s and companies simply cannot afford. Internet and managed service providers and large organizations must take proper precautions to remain in compliance. Heavy fines, license and certification issues and other penalties can result from non-compliance with mandatory data retention requirements.

According to the Australian Attorney-General’s Department, Australian telecommunications companies must keep a limited set of metadata for two years. Metadata is information about a communication (the who, when, where and how)—not the content or substance of a communication (the what).

A commentator from the Sydney morning herald qualified“…Security, intelligence and law enforcement access to metadata which overrides personal privacy is now in contention worldwide…” and speculated that with the introduction of Australian metadata laws that “…this country’s entire communications industry will be turned into a surveillance and monitoring arm of at least 21 agencies of executive government. …”.

In Australia many smaller ISP’s are fearful that failing to comply will send them out of business. Internet Australia’s Laurie Patton said, “It’s such a complicated and fundamentally flawed piece of legislation that there are hundreds of ISPs out there that are still struggling to understand what they’ve got to do”.

As for the anticipated costs, a survey sent to ISPs by telecommunications industry lobby group Communications Alliance found that “There is a huge variance in estimates for the cost to business of implementing data retention – 58 per cent of ISPs say it will cost between $10,000 and $250,000; 24 per cent estimate it will cost over $250,000; 12 per cent think it will cost over $1,000,000; some estimates go as high as $10 million.”

An important cost to consider in compliance is the ease of data reporting when requested by government or corporate compliance teams to produce information for a specific ipv4 or ipv6 address. If the data is stored in a data-warehouse that is difficult to filter this may cause the service provider to incur penalties or be seen to be non-complying. Flexible filtering and automated reporting is therefore critical to produce the forensics required for the compliance in a timely and cost effective manner.

Although there are different laws governing different countries the main requirement of mandatory data retention laws at ISP’s is to maintain sufficient information at a granular level in order to assist governments in finding bad actors such as terrorists, corporate espionage, ransom-ware and pedophiles. In some countries this means that telcos are required to keep data of the IP addresses users connect to, for up to 10 weeks and in others just the totals of subscriber usage for each IP used for up to 2 years.

Although information remains local to each country and governed by relevant privacy laws, the benefits to law enforcement in the future will eventually provide the ability to have the visibility to track relayed data such as communications used by Tor Browsers, Onion routers and Freenet beyond their relay and exit nodes.

There is no doubt in my mind that with heightened states of security and increasing online crime there is a global need for governments to intervene with online surveillance to protect children from exploitation, reduce terrorism and to build defensible infrastructures whilst at the same time implementing data retention systems that have the inbuilt smarts to enable a balance between compliance and privacy rather than just a blanket catch all. There is already an available solution for the Internet communications component based on Netflow that assists ISP’s to quickly comply at a low cost whilst properly allowing data retention rules to be implemented to limit intruding on an individual’s privacy.

NetFlow solutions are cheap to deploy and are not required to be deployed at every interface such as a packet analyzer and can use the existing router, switch or firewall investment to provide continuous network monitoring across the enterprise, providing the service provider or organization with powerful tools for data retention compliance.

NetFlow technology if sufficiently scalable, granular and flexible can deliver on the visibility, accountability and measurability required for data retention because it can include features that:

Supply a real-time look at network and host-based activities down to the individual user and device;
Increase user accountability for introducing security risks that impact the entire network;
Track, measure and prioritize network risks to reduce Mean Time to Know (MTTK) and Mean Time to Repair or Resolve (MTTR);
Deliver the data IT staff needs to engage in in-depth forensic analysis related to security events and official requests;
Seamlessly extend network and security monitoring to virtual environments;
Assist IT departments in maintaining network up-time and performance, including mission critical applications and software necessary to business process integrity;
Assess and enhance the efficacy of traditional security controls already in place, including firewalls and intrusion detection systems;
Capture and archive flows for complete data retention compliance

Compared to other analysis solutions, NetFlow can fill in the gaps where other technologies cannot deliver. A well-architected NetFlow solution can provide a comprehensive landscape of tools to help business and service providers to achieve and maintain data retention compliance for a wide range of government and industry regulations.

The post How NetFlow Solves for Mandatory Data Retention Compliance appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

Scalable NetFlow – 3 Key Questions to Ask Your NetFlow Vendor

Rafi Sabel — Tue, 30 Jul 2024 07:07:14 +0000

NetFlow scalability, data retention and AI accuracy

Why Flows-per-Second Is a Flawed Measure of NetFlow Collector Capability

A high flows-per-second rating tells you how quickly a collector can receive flow records. It does not tell you how many records are retained, how much context survives, how quickly the information can be analyzed, or whether AI is reasoning from a complete and accurate view of the network.

The answer in one sentence

Collection rate measures input. Retention determines visibility, forensic depth and the quality of the conclusions produced by both human analysts and AI.

Incomplete retention produces incomplete AI

AI-assisted detection, anomaly analysis and investigations depend on the evidence presented to the model. When flow records are sampled, aggregated or discarded, the AI is not analyzing the network as it actually behaved. It is analyzing a reduced and potentially distorted version of events.

This can cause missed threats, false positives, false negatives, incorrect root-cause conclusions and recommendations based on incomplete context.

AI cannot reason accurately about network activity that the collector failed to retain.

What FPS measures

How quickly exported flow records can enter the collector.

What FPS omits

Retention, enrichment, historical context, query performance and AI data quality.

What really matters

What is retained, at what granularity, for how long and how effectively it can be analyzed.

The headline number measures only the entrance

A vendor may claim that its collector accepts hundreds of thousands or millions of flows per second. That figure can be technically correct while still creating a misleading impression of the product's real capability.

Receiving a flow record is only the first stage. A useful NetFlow platform must also process, enrich, retain and analyze the record.

Stage 1 Receive Accept flow records exported by routers, switches, firewalls, cloud platforms and other sources.

Stage 2 Process Parse, normalize, classify, enrich and correlate each record.

Stage 3 Retain Preserve sufficient detail, context and time granularity in memory or storage.

Stage 4 Analyze Support dashboards, alerts, reports, investigations, forensics and AI reasoning.

A flows-per-second benchmark generally measures only the first stage. It may reveal little about the performance or limitations of the remaining three.

Collection rate is not information retention

A collector can receive a large volume of flow records while retaining only a small fraction of them. The discarded records may be sampled, aggregated, summarized or permanently removed before they reach long-term storage.

This distinction matters because a platform can only report on, investigate or apply AI to the information that remains available.

High ingestion rate

≠

High retention rate

≠

Accurate intelligence

Once a flow is discarded, it cannot be searched, reconstructed, correlated or supplied to an AI model during a later investigation.

A simple comparison

Collector A

Receives 2 million flows per second
Retains only 500 selected records per minute
Prioritizes the largest bandwidth consumers
Discards most low-volume conversations
Provides limited historical forensic context
Supplies AI with a reduced version of events

Collector B

Receives 500,000 flows per second
Retains every relevant flow record
Preserves communication context and timing
Supports detailed short-term and historical analysis
Maintains searchable forensic evidence
Supplies AI with broader and more accurate context

Collector A has the stronger marketing number. Collector B may deliver far greater operational, security and AI value.

Why flows-per-second became the wrong yardstick

Technology performance has often been reduced to one simple number. That approach works only while the number continues to represent the overall capability of the system.

Computer performance was once judged mainly by CPU clock speed. Multi-core processors changed that relationship. A processor could run at a lower clock speed while performing more work through parallel processing.

Flows-per-second has become the NetFlow equivalent of CPU clock speed.

It remains useful as one metric, but it does not describe the capability of the entire flow analytics platform.

What a modern flow collector must preserve

NetFlow was developed to provide network visibility without requiring packet capture systems on every network segment.

A basic flow record commonly includes:

Source IP address
Destination IP address
Source port
Destination port
Protocol
Type of Service
Ingress interface
Egress interface

Modern exporters may also include:

Application identifiers
Application categories
URLs and hostnames
MAC addresses
TCP flags
Latency measurements
MPLS information
Cloud metadata
Network performance measurements
Security and identity context

These fields improve analytical value, but they also increase record size and processing requirements. A benchmark based on a minimal flow template may bear little resemblance to performance with a real production template.

Why detailed retention matters to AI

AI systems identify patterns by comparing activity across devices, users, applications, destinations, time periods and prior behavior. Their effectiveness depends on data continuity and context.

When records are selectively removed, the AI may lose:

The sequence in which activity occurred
The relationship between small and large flows
Repeated low-volume communication patterns
Evidence of gradual behavioral change
The baseline needed to identify anomalies
The communications that preceded or followed an incident
The distinction between unusual and genuinely malicious activity

Better AI does not come only from stronger models. It requires sufficiently complete, granular and historically consistent telemetry.

An advanced model working from incomplete telemetry may still produce a confident answer. Confidence does not make the answer correct.

The risk of Top-N retention

Many traditional NetFlow products were designed primarily for bandwidth reporting. To reduce processing and storage requirements, they may retain only selected records using methods such as:

Top Bytes
Top Conversations
Top Sources or Destinations
First N records
Sampling
Time-based aggregation
Progressive historical roll-ups

These methods can be useful for identifying major bandwidth consumers. They are much less effective when the activity of interest is distributed across many small flows.

Modern security and operational risks often include:

DDoS activity
Botnet communications
Command and Control traffic
Malware beaconing
DNS tunneling
Credential attacks
Lateral movement
Peer-to-peer activity
Insider activity
Low-and-slow data exfiltration
Automated scanning
MFA bypass-related activity

Individually, these flows may never appear in a Top 10 or Top 100 report. Collectively, they may describe the complete attack chain.

A platform that retains only the largest flows may provide an accurate view of bandwidth consumption while remaining blind to the activity that presents the greatest operational or security risk.

Granularity determines what can be investigated

Retention is not merely a question of whether some data is stored. The level of detail and time granularity must also be considered.

A product might retain:

Minute-level data for one day
Hourly summaries after two days
Six-hour aggregates after one week
Daily summaries after one month

This may be adequate for long-term trending, but it can be insufficient for reconstructing a security event or explaining why a device, user or application behaved abnormally at a specific time.

AI-assisted historical analysis faces the same limitation. If detailed temporal relationships have been removed, the model cannot accurately reconstruct the original sequence of events.

Real scalability must be measured across the entire system

A NetFlow platform is scalable only if it continues to collect, retain and analyze useful information as the environment becomes more demanding.

A meaningful assessment should include:

Sustained collection rate
Burst handling capacity
Flow retention rate
Percentage of records retained
Information granularity
Retention logic
Number of devices and interfaces
IPv4 and IPv6 cardinality
Flow template complexity
Enrichment processing requirements
Alerting responsiveness
Historical query performance
Concurrent reporting performance
Forensic depth
AI analysis performance
Storage efficiency
Performance as the database grows
Performance under attack conditions

A short benchmark against an empty database does not demonstrate how the platform will behave after months or years of continuous collection.

Three questions to ask your NetFlow vendor

How many flows can be collected per second?

Is the quoted figure a burst rate or a sustained rate?
For how long was the rate maintained?
What hardware, memory and storage architecture were used?
Was the test performed against an empty or mature database?
How does performance change during high flow variance?
What happens during a DDoS attack or sudden traffic burst?
How does performance change as devices and interfaces are added?
How do IPv4 and IPv6 cardinality affect performance?
How do richer flow templates affect the quoted rate?
Does enabling enrichment, analytics or AI reduce the stated capacity?

How many flow records are actually retained?

How many individual records are written to storage each minute?
What percentage of received records is retained?
Can the retained record count be independently verified?
Does the same retention rate apply to recent and historical data?
What happens to records that are not retained?
Are records aggregated, sampled, summarized or permanently discarded?
What selection logic determines which records survive?
What information is available to AI before and after retention reduction?

What information granularity remains available over time?

Is every individual flow retained?
Does minute-level data later become hourly or daily data?
Are detailed records replaced by summaries?
Can administrators control the retention granularity?
How long is full-resolution information retained?
Can historical records still be searched quickly?
Can the original communication sequence be reconstructed?
Does the AI analyze original records or only summarized data?
How does reduced granularity affect anomaly detection and investigation accuracy?

The practical test

Do not ask only how much data enters the system

Ask what remains available when an analyst or AI system must investigate an event that occurred yesterday, last month or six months ago.

Determine whether the platform retained the individual flows, contextual fields, relationships and time granularity required to explain what actually happened.

Then test whether reports, searches, alerts and AI-assisted investigations remain accurate and responsive after the system has accumulated a realistic volume of historical data.

Frequently asked questions

What does flows-per-second measure?

Flows-per-second measures how quickly a collector can receive exported flow records. It does not measure how many records are retained, how much context survives or how quickly historical data can be analyzed.

Why is flow retention important for AI?

AI requires sufficiently complete and consistent telemetry to identify patterns, compare behavior and reconstruct events. Missing or aggregated flow records can distort the context and reduce the accuracy of AI-assisted detection and investigation.

Can a collector advertise high FPS while storing very little?

Yes. A product may accept a high volume of incoming records while storing only selected flows, Top-N results, samples or time-based summaries.

What matters more than a headline FPS figure?

Sustained ingestion, retention percentage, information granularity, historical depth, query speed, forensic capability, AI data quality and performance as the database grows are all important.

Conclusion

Flows-per-second remains a useful indication of ingestion capacity, but it should never be treated as a complete measure of NetFlow collector capability.

The real measure is the platform's ability to preserve useful information, maintain sufficient granularity, support fast analysis and continue performing as traffic, devices, users and historical data increase.

This is particularly important for AI-driven security observability. The accuracy of AI-assisted detection and investigation depends not only on the quality of the model, but also on the completeness and fidelity of the evidence supplied to it.

Rate of collection does not equal information retention. AI can only reason accurately from the evidence that has been preserved.

The post Scalable NetFlow – 3 Key Questions to Ask Your NetFlow Vendor appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

Big Data – A Global Approach To Local Threat Detection

Rafi Sabel — Sat, 27 Jul 2024 09:42:51 +0000

From helping prevent loss of life in the event of a natural disaster, to aiding marketing teams in designing more targeted strategies to reach new customers, big data seems to be the chief talking point amongst a broad and diverse circle of professionals.

For Security Engineers, big data analytcs is proving to be an effective defense against evolving network intrusions thanks to the delivery of near real-time insights based on high volumes of diverse network data. This is largely thanks to technological advances that have resulted in the capacity to transmit, capture, store and analyze swathes of data through high-powered and relatively low-cost computing systems.

In this blog, we’ll take a look at how big data is bringing deeper visibility to security teams as environments increase in complexity and our reliance on pervading network systems intensifies.

Big data analysis is providing answers to the data deluge dilemma

Large environments generate gigabytes of raw user, application and device metrics by the minute, leaving security teams stranded in a deluge of data. Placing them further on the back foot is the need to sift through this data, which involves considerable resources that at best only provide a retrospective view on security breaches.

Big data offers a solution to the issue of “too much data too fast” through the rapid analysis of swathes of disparate metrics through advanced and evolving analytical platforms. The result is actionable security intelligence, based on comprehensive datasets, presented in an easy-to-consume format that not only provides historic views of network events, but enables security teams to better anticipate threats as they evolve.

In addition, big data’s ability to facilitate more accurate predictions on future events is a strong motivating factor for the adoption of the discipline within the context of information security.

Leveraging big data to build the secure networks of tomorrow

As new technologies arrive on the scene, they introduce businesses to new opportunities – and vulnerabilities. However, the application of Predictive AI Baselining analytics to network security in the context of the evolving network is helping to build the secure, stable and predictable networks of tomorrow. Detecting modern, more advanced threats requires big data capabilities from incumbent intrusion prevention and detection (IDS\IPS) solutions to distinguish normal traffic from potential threats.

By contextualizing diverse sets of data, Security Engineers can more effectively detect stealthily designed threats that traditional monitoring methodologies often fail to pick up. For example, Advanced Persistent Threats (APT) are notorious for their ability to go undetected by masking themselves as day-to-day network traffic. These low visibility attacks can occur over long periods of time and on separate devices, making them difficult to detect since no discernible patterns arise from their activities through the lens of traditional monitoring systems.

Big data Predictive AI Baselining analytics lifts the veil on threats that operate under the radar of traditional signature and log-based security solutions by contextualizing traffic and giving NOCs a deeper understanding of the data that traverses the wire.

Gartner states that, “Big data Predictive AI Baselining analytics enables enterprises to combine and correlate external and internal information to see a bigger picture of threats against their enterprises.” It also eliminates the siloed approach to security monitoring by converging network traffic and organizing it in a central data repository for analysis; resulting in much needed granularity for effective intrusion detection, prevention and security forensics.

In addition, Predictive AI Baselining analytics eliminates barriers to internal collaborations between Network, Security and Performance Engineers by further contextualizing network data that traditionally acted as separate pieces of a very large puzzle.

So is big data Predictive AI Baselining analytics the future of network monitoring?

In a way, NOC teams have been using big data long before the discipline went mainstream. Large networks have always produced high volumes of data at high speeds – only now, that influx has intensified exponentially.

Thankfully, with the rapid evolution of computing power at relatively low cost, the possibilities of what our data can tell us about our networks are becoming more apparent.

The timing couldn’t have been more appropriate since traditional perimeter-based IDS\IPS no longer meet the demands of modern networks that span vast geographical areas with multiple entry points.

In the age of cloud, mobility, ubiquitous Internet and the ever-expanding enterprise environment, big data capabilities will and should become an intrinsic part of virtually every security apparatus.

The post Big Data – A Global Approach To Local Threat Detection appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

NetFlow for Usage-Based Billing and Peering Analysis

Rafi Sabel — Thu, 25 Jul 2024 12:09:57 +0000

Usage–based billing refers to the methods of calculating and passing back the costs of running a network to the consumers of data that occur through the network. Both Internet Service Providers (ISP) and Corporations have a need for Usage-based billing with different billing models.

NetFlow is the ideal technology for usage-based billing because it allows for the capture of all transactional information pertaining to the usage and some smart NetFlow technologies already exist to assist in the counting, allocation, and substantiation of data usage.

Advances in telecommunication technology have enabled ISPs to offer more convenient, streamlined billing options to customers based on bandwidth usage.

One billing model used most commonly by ISPs in the USA is known as the 95^th percentile. The ISP filters the samples and disregards the highest 5% in order to establish the bill amount. This is an advantage to data consumers who have bursts of traffic because they’re not financially penalized for exceeding a traffic threshold for brief periods of time. The solution measures traffic employing a five-minute granularity standard typically over the course of a month.

The disadvantage of the 95^th percentile model is that its not sustainable business model as data continues to become a utility like electricity.

A second approach is a utility-based metered billing model that involves retaining a tally of all bytes consumed by a customer with some knowledge of data path to allow for premium or free traffic plans.

Metered Internet usage is used in countries like Australia and most recently Canada who have nationally moved away from a 95^th percentile model. This approach is also very popular in corporations whose business units share common network infrastructure and who are unwilling to accept “per user” cost, but rather a real consumption-based cost.

Benefits of usage-based billing are:

Improved transparency about the cost of services;
Costs feedback to the originator;
Raised cost sensitivity;
Good basis for active cost management;
The basis for Internal and external benchmarking;
Clear substantiation to increase bandwidth costs;
Shared infrastructure costs can also be based on consumption;
Network performance improvements.

For corporations, usage-based billing enables the IT department to become a shared service and viewed as a profit center rather than a cost center. It can become viewed as something that’s a benefit and a catalyst for business growth rather than a necessary but expensive line item in the budget.

For ISPs in the USA, there is no doubt that utility-based costs per byte model will continually be contentious as video and TV over Internet usage increases. In other regions, new business models that include packaging of video over “free zones” services have become popular meaning that the cost of premium content provision has fallen onto the content provider making utility billing viable in the USA.

NetFlow tools can include methods for building billing reports and offer a variety of usage-based billing model calculations.

Some NetFlow tools even include an API to allow the chart-of-accounts to be retained and driven from traditional accounting systems using the NetFlow system to focus on the tallying. Grouping algorithms should be flexible within the solution to allow for grouping of all different variables such as interfaces, applications, Quality of Service (QoS), MAC Addresses, MPLS, and IP groups. For ISPs and large corporations Asynchronous Network Numbers (ASN) also allow for analysis of data-paths allowing sensible negotiations with Peering partners and Content partners.

Look out for more discussion on peering in an upcoming blog…

The post NetFlow for Usage-Based Billing and Peering Analysis appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

How to counter-punch botnets, viruses, ToR & more with Netflow [Pt 1]

Rafi Sabel — Thu, 25 Jul 2024 07:30:39 +0000

You can’t secure what you can’t see and you don’t know what you don’t know.

Many network and security professionals assume that they can simply analyze data captured using their standard security devices like firewalls and intrusion detection systems, however they quickly discover their limitations as these devices are not designed for and cannot record and report on every transaction due to lack of granularity, scalability and historic data retention. Network devices like routers, switches, Wi-Fi or VMware servers also typically lack any sophisticated anti-virus software.

Presenting information in a manner that quickly enables security teams to act with simple views with deep contextual data supporting the summaries is the mark of a well constructed traffic analyzer ensuring teams are not bogged down by the detail unless required and even then allowing elegant means to extract forensics with simple but powerful visuals to enable quick contextual grasp and impact of a security event.

Using NetFlow Correlation to Detect intrusions

Host Reputation is one of the best detection methods that can be used against Advanced Persistent Threats. There are many data sources to choose from and some are more comprehensive than others.

Today these blacklists are mostly IPv4 and Domain orientated designed to be used primarily by firewalls, network intrusion systems and antivirus software.

They can also be used in NetFlow systems very successfully as long as the selected flow technology can scale to support the thousands of known compromised end-points, the ability to frequently update the threat data and the ability to record the full detail of every compromised flow and subsequent conversations that communicate with the compromised systems to discover other related breaches that may have occurred or are occurring.

According to Mike Schiffman at Cisco,

“If a given IP address is known to be that of a spammer or a part of a botnet army it can be flagged in one of the ill repute databases … Since these databases are all keyed on IP address, NetFlow data can be correlated against them and subsequent malicious traffic patterns can be observed, blocked, or flagged for further action. This is NetFlow Correlation.“

The kind of data can we expect to find in the reputation databases are IP addresses that have known to be acting in some malicious or negative manner such as being seen by multiple global honeypots. Some have been identified to be part of a well-known botnet such as Palevo or Zeus whilst other IP’s are known to have been distributing Malware or Trojans. Many kinds of lists can be useful to correlate such as known ToR end points or Relays that have become particularly risky of late being a common means to introduce RansomWare and should certainly not be seen conversing to any host within a corporate, government or other sensitive environment.

Using a tool like CySight’s advanced End-Point Threat Detection allows NetFlow data to be correlated against hundreds of thousands of IP addresses of questionable reputation including ToR exits and relays in real-time with comprehensive historical forensics that can be deployed in a massively parallel architecture.

As a trusted source of deep network insights built on big data analysis capabilities, Netflow provides NOCs with an end-to-end security and performance monitoring and management solution. For more information on Netflow as a performance and security solution for large-scale environments, download our free Guide to Understanding Netflow.

Cutting-edge and innovative technologies like CySight delivers the deep end-to-end network visibility and security context required assisting in speedily impeding harmful attacks.

The post How to counter-punch botnets, viruses, ToR & more with Netflow [Pt 1] appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

How to Improve Cyber Security with Advanced Netflow Network Forensics

Rafi Sabel — Sun, 07 Jul 2024 07:30:06 +0000

Most organizations today deploy network security tools that are built to perform limited prevention – traditionally “blocking and tackling” at the edge of a network using a firewall or by installing security software on every system.

This is only one third of a security solution, and has become the least effective measure.

The growing complexity of the IT infrastructure is the major challenge faced by existing network security tools. The major forces impacting current network security tools are the rising level of sophistication of cybercrimes, growing compliance and regulatory mandates, expanding virtualization of servers and the constant need for visibility compounded by ever-increasing data volumes. Larger networks involve enormous amounts of data, into which the incident teams must have a high degree of visibility for analysis and reporting purposes.

An organization’s network and security teams are faced with increasing complexities, including network convergence, increased data and flow volumes, intensifying security threats, government compliance issues, rising costs and network performance demands.

With network visibility and traceability also top priorities, companies must look to security network forensics to gain insight and uncover issues. The speed with which an organization can identify, diagnose, analyze, and respond to an incident will limit the damage and lower the cost of recovery.

Analysts are better positioned to mitigate risk to the network and its data through security focused network forensics applied at the granular level. Only with sufficient granularity and historic visibility and tools that are able to machine learn from the network Big Data can the risk of an anomaly be properly diagnosed and mitigated.

Doing so helps staff identify breaches that occur in real-time, as well as Insider threats and data leaks that take place over a prolonged period. Insider threats are one of the most difficult to detect and are missed by most security tools.

Many network and security professionals assume that they can simply analyze data captured using their standard security devices like firewalls and intrusion detection systems, however they quickly discover limitations as these devices are not designed for and cannot record and report on every transaction due to lack of deep visibility, scalability and historic data retention making old fashioned network forensic reporting expensive and impractical.

NetFlow is an analytics software technology that enables IT departments to accurately audit network data and host-level activity. It enhances network security and performance making it easy to identify suspicious user behaviors to protect your entire infrastructure.

A well-designed NetFlow forensic tool should include powerful features that can allow for:

Micro-level data recording to assist in identification of real-time breaches and data leaks;
Event notifications and alerts for network administrators when irregular traffic movements are detected;
Tools that highlight trends and baselines, so IT staff can provision services accordingly;
Tools that learn normal behavior, so Network Security staff can quickly detect and mitigate threats;
Capture highly granular traffic over time to enable deep visibility across the entire network infrastructure;
24-7 automation, flexible reporting processes to deliver usable business intelligence and security forensics specifically for those analytics that can take a long time to produce.

Forensic analysts require both high-level and detailed visibility through aggregating, division and drilldown algorithms such as:

Deviation / Outlier analysis
Bi-directional analysis
Cross section analysis
Top X/Y analysis
Dissemination analysis
Custom Group analysis
Baselining analysis
Percentile analysis
QoS analysis
Packet Size analysis
Count analysis
Latency and RTT analysis

Further when integrated with a visual analytics process it will enable additional insights to the forensic professional when analyzing subsets of the flow data surrounding an event.

In some ways it needs to act as a log analyzer, security information and event management (SIEM) and a network behavior anomaly and threat detector all rolled into one.

The ultimate goal is to deploy a multi-faceted flow-analytics solution that can compliment your business by providing extreme visibility and eliminating network blindspots, both in your physical infrastructure and in the cloud, automatically detecting and diagnosing your entire network for anomalous traffic and improving your mean time to detect and repair.

The post How to Improve Cyber Security with Advanced Netflow Network Forensics appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.

5 Perks of Network Performance Management

Rafi Sabel — Thu, 04 Jul 2024 12:59:03 +0000

Network performance management is something that virtually every business needs, but not something that every business is actively doing, or even aware of. And why should they?

While understanding the technical side of things is best left to the IT department, understanding the benefits of a properly managed network is something that will help get the business managers on board, especially when good performance management solutions might be a cost that hadn’t been considered. So what are the benefits?

1. Avoiding downtime – Downtime across an entire network is going to be rare, but downtime in small areas of the network are possible if it gets overloaded. Downtime of any kind is just not something that business can tolerate, for a few reasons:

it leaves that area of the network unmonitored, which is a serious security issue
shared files won’t be accessible, nor will they be updating as users save the files. This will lead to multiple versions of the same file, and quite a few headaches when the network is accessible again
downtime that affects customers is even worse, and can result in lost revenue or negative customer experiences

2. Network speed – This is one of the most important and easily quantified aspects of managing netflow. It will affect every user on the network constantly, and anything that slows down users means either more work hours or delays. Obviously, neither of these is a good problem to have. Whether it’s uploading a file, sending a file to a coworker, or sending a file to a client; speed is of paramount importance.

3. Scalability – Almost every business wants to grow, and nowhere is that more true than the tech sector. As the business grows, the network will have to grow with it to support more employees and clients. By managing the performance of the network, it is very easy to see when or where it is being stretched too thin or overwhelmed. As performance degrades, it’s very easy to set thresholds that show when the network need upgraded or enlarged.

4. Security – Arguably the most important aspect of network management, even though it might not be thought of as a performance aspect. An unsecured network is worse than a useless network, and data breaches can ruin a company. So how does this play into performance management?

By monitoring netflow performance, it’s easy to see where the most resources are being used. Many security attacks drain resources, so if there are resource spikes in unusual areas it can point to a security flaw. With proper software, these issues can be not only monitored, but also recorded and corrected.

5. Usability – Unfortunately, not all employees have a working knowledge of how networks operate. In fact, as many in IT support will attest, most employees aren’t tech savvy. However, most employees will need to use the network as part of their daily work. This conflict is why usability is so important. The easiest way to minimize training costs with any network management program is to ensure that it is as user-friendly as possible.

The fanciest, most impressive network performance management system isn’t worth anything if no one knows how to use and optimize it properly. Even if the IT department has no issues with it, the reports and general information should be as easy to decipher as possible.

Is your network as optimized as it could be? Are you able to monitor the network’s performance and flow, or do network forensics to determine where issues are? Don’t try to tackle all of this on your own, contact us and let us help you support your business with the best network monitoring for your specific needs.

The post 5 Perks of Network Performance Management appeared first on CySight - Integrated AI-Driven Cyber Network and EndPoint Detection and Response.