Articles

NetFlow and IPFIX: flow records, exporters and what they can see

How flow telemetry turns packets into records, and what a record can and cannot show.

Reading: 5 minNetworking

Article cover: NetFlow and IPFIX: flow records, exporters and what they can see

A flow record is a summary of a conversation, not the conversation itself. NetFlow is Cisco’s flow-export format, documented for its version 9 in RFC 3954; IPFIX, the IP Flow Information Export protocol, is the IETF standard for the same job — RFC 7011, STD 77, which obsoletes RFC 5101. A device watches packets at an observation point, groups them into flows that share common properties, keeps a record per flow, and exports those records to a collector. The record carries the flow’s key fields, its packet and byte counts, timestamps and interfaces — never the payload. And it exists only where the platform supports and is configured to export it: flow telemetry is a capability some devices have, not a given.

The mental model: observe, record, export, collect

Three functions run on the device and one runs on another host. A metering process watches the packets at an observation point and builds flow records; an exporting process sends them as messages; a collecting process on the collector decodes and stores them. The records travel with a small control layer — templates — that tells the collector how to read them.

packets ─► Observation Point ─► Metering Process ─► Exporting Process (Exporter)
             (where it is seen)   (builds Flow Records)        │
                                                                 │  templates + data records
                                                                 ▼  SCTP · TCP · UDP
                                                          Collecting Process (Collector)
                                                                 │
                                                                 ▼
                                             key, counters, times, interfaces — no payload

Terms

  • Flow — a set of packets passing an observation point during a time interval, whose members share a set of common properties (RFC 5470).
  • Flow key — the packet fields that define which packets belong to the flow; add a field and the flow becomes finer.
  • Observation point / observation domain — where packets are observed on the device, and the group of such points that shares one identity in the export.
  • Metering process — classifies packets into flows and maintains the flow records, including when a flow ends.
  • Exporting process / exporter — sends the messages to one or more collectors.
  • Collecting process / collector — receives, decodes and stores the records.
  • Template — a record that defines the structure and interpretation of the fields in a data record (RFC 7011); the exporter sends a template first, then data records that refer to it.
  • Information element — one attribute in a record, defined in the IANA “IPFIX Information Elements” registry, with enterprise-specific elements allowed (RFC 7012).
  • Sampling — measuring only a subset of the packets, so the flow record is an estimate.

How it works, step by step

  1. Packets are classified into flows. A metering process observes the packets at an observation point, timestamps them and sorts them into flows by the flow key (RFC 5470). Every packet that satisfies the key’s properties belongs to the flow.
  2. Counters accumulate. The record holds the flow’s fields and the per-flow counters — the bytes and packets seen — so a long conversation is represented by one record rather than one per packet (RFC 3954).
  3. The flow ends. Expiration happens when no packet of the flow is seen for a timeout, when the device runs out of memory for records, or periodically for long-running flows (RFC 5470). A zero timeout reports a flow as a sequence of single-packet flows.
  4. The exporter sends a template, then the data. A template record fixes the structure and interpretation of the fields; the data records carry values against it. The collector cannot decode a data record without the template that defines it (RFC 7011).
  5. Transport decides what survives. IPFIX specifies PR-SCTP as its mandatory transport, with TCP and UDP optional, whereas NetFlow version 9 first ran over UDP (RFC 5153). Control information should travel over a reliable transport; UDP alone is unreliable (RFC 5470).

What a flow record can and cannot see

The record is a summary, and its content is exactly what its template carried. It shows a key, counters and treatment at one observation point — not payload, and not the whole path.

Availability bounds it. Export happens only on a device and at an observation point where the capability is enabled; where it is not, there is no record at all. Because a record describes one observation point, it tells you what that point saw, not what every device along the path did.

Sampling and aggregation make it an estimate. A device under load may sample, filter or aggregate before export (RFC 5153); sampling at a rate such as one packet in a hundred turns the counters into an approximation, not a census. Choosing all packets is the special case of a 1:1 rate (RFC 5470).

Middleboxes change what it means. A device that is the head, midpoint or end of a tunnel, serves VPN or NAT traffic, or drops packets by policy reports the flow as it saw it: NAT rewrites the addresses a flow key would otherwise use, and a firewall’s drops may need their own counters to be visible (RFC 5470; RFC 5153).

The common error: reading a record as the packet

The common mistake is to treat a flow record as evidence of what happened on the wire. It is not the packet and not the payload; it is the fields the template carried, counted at one point. Two confusions grow from the same root. The first is transport: an exporter over UDP offers no reliable transmission of templates, so a collector can hold data records it cannot decode; even over reliable SCTP, congestion can make the exporter discard records, and the sequence numbers carry that loss (RFC 7011). The second is time: a device without an absolute clock exports relative times, and without the system-init time the collector cannot turn them into absolute times accurately (RFC 5153). Read the record for what the architecture promises — a summary at a point, from the fields that were exported.

Level and prerequisites

L3 — integration and technical control. It assumes the L2 vocabulary of addresses, ports and sockets, plus the layer’s logging sheets that fix what syslog and severity are; here the subject is a second, structured telemetry stream and the boundary of what it can tell you. Configuring a specific platform’s exporter belongs to a project, and sits behind the availability question this sheet states.

Where to go next

References

  • RFC 3954 — Cisco Systems NetFlow Services Export Version 9 (Oct 2004, Informational); a flow defined as a unidirectional sequence of packets with common properties, and the fields a flow record carries, §1. The document specifies Cisco’s own export format and is published at Informational status, not as a Standards-Track IETF protocol.
  • RFC 5470 — Architecture for IP Flow Information Export (Mar 2009, Informational); the definitions of flow and flow key, §3; metering process, flow expiration and export, §5.1; observation point, §5.2; sampling functions, §5.3.1; observation domain, §5.4; exporting and collecting processes, §5.5–5.6; transport recommendations, §8.1; special situations (tunnels, VPN, NAT, firewalls), §9.
  • RFC 7011 — Specification of the IP Flow Information Export (IPFIX) Protocol (Sep 2013, Standards Track, STD 77); template record and data record definitions, §3.2; template management across transport sessions, §8; transport-specific reliability (SCTP, TCP, UDP), §10.
  • RFC 7012 — Information Model for IP Flow Information Export (IPFIX) (Sep 2013, Standards Track); the IANA “IPFIX Information Elements” registry and the properties of an information element, §2.1.
  • RFC 5471 — Guidelines for IP Flow Information Export (IPFIX) Testing (Mar 2009, Informational) — consulted for the protocols’ test vocabulary; no body claim rests on it.
  • RFC 5153 — IPFIX Implementation Guidelines (Apr 2008, Informational); transport choices and message-loss mitigation by sampling, filtering or aggregation, §6; middlebox behaviour, §7; devices without an absolute clock, §4.7; differences between IPFIX and NetFlow version 9, §10.1.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.