The APNIC 52 conference took the virtual stage last week from September 12-15, 2021. The conference was chock full of interesting tutorials, talks, and opportunities to learn and network, but we were most looking forward to the IPv6 Deployment session scheduled for September 13th.
The session kicked off with welcoming remarks from Kenny Huang, Executive Director of the Taiwan Network Information Center and moderator. Kenny introduced us to Hannah Durarack, Grants Management Lead at APNIC Foundation, who informed everyone of IPv6 deployment grants that will become available next month.
IPv6 Fragmentation Revisited
Next, Geoff Huston, Chief Scientist at APNIC, dived into the issue of IP packet fragmentation and the differences between IPv4 and IPv6 in handling fragments. We explored the overall situation of packet drop rates between 2013 and now to answer the question: Are we getting any better with IPv6 Extension Headers and Fragmentation? And how serious is the problem?
When you fragment a packet, it often doesn’t reach its destination. Normally a packet has headers and payloads, but if a packet is a trailing fragment, there is no transport header. Letting the packet through without a check is a security risk, so that is not an option. IPv6 changed the fragmentation header from always being in the IPv4 header, to optional in IPv6. So most of the time in IPv6, a packet will not have a fragmentation header. This means that the transport header in IPv6 is either 40 bytes away from the start of the packet, or 48 or longer if there are extension headers. Sometimes high-speed equipment doesn’t like extension headers, causing them to do a software search where the packet can often get dropped.
We know extension headers are a problem in IPv6, but just how bad is it?
In 2017, APNIC ran a month-long experiment using the APNIC IPv6 measurement platform to test the drop rate of IPv6 packets flowing from server to client. The fragmentation loss was 21%.
Fast forward to this year, and APNIC wanted to see if the numbers had improved. This time, they kept the mechanism the same but set up a middlebox that performed fragmentation on demand. APNIC also randomly varied the initial fragmented packet size between 1,200 and 1,416 bytes to see if drops vary by size. This revised test found that for the world’s IPv6 networks, the drop rate had improved to 12%.
An unexpected finding was that in North and South America, the larger the initial fragmented packet, the smaller the drop rate. In Asia on the other hand, the larger the initial fragmented packet, the higher the drop rate. Geoff speculated that the difference could be due to service providers and their equipment and configurations, local security policies and triggering of “slow path” processing in network equipment.
Geoff also discussed how APNIC has been working on “atomic” fragments. APNIC found that adding 8 bytes before the IP header and transport header can let servers know that it is not a fragment and it is the entire packet. These “atomic” fragments brought the drop rate down to 2% on average worldwide. But the “atomic” fragments drop rate still varies by region and provider. While the Americas and Asia experienced drop rates of 0.5% and 1% respectively, Europe’s drop rate was 8%.
Geoff wrapped up his presentation by noting that IPv6 is in fact improving its handling of fragmented packets. He also pointed out that recent deployments appear to show more strength in the handling of packets than older IPv6 infrastructures. And oddly enough, if you make the initial fragment smaller (12,40 or less), followed by a larger one, it is less likely to get dropped.
Akamai IPv4/IPv6 Experience
Next, Alex Leung, Senior Enterprise Architect at Akamai, discussed the world’s shift to working remotely and being online for longer, and how the shift has led to increased IPv6 adoption in APNIC nations.
Alex began his presentation by explaining that he sometimes works on remote beaches. These interesting work locations lead him to compare the number of IPv6 addresses to grains of sand on a beach. He stated that the number of IPv6 addresses is so abundant that each grain of sand on the beach could be assigned its own IP address!
Alex explained that in 2020, Malaysia saw elevated internet usage during lockdown, but even when the lockdown was relaxed, internet usage remained elevated. This is likely due to people accepting a “new normal” in their lifestyles.
There are three main business drivers for OTT customers to adopt IPv6: throughput, scale and latency. For example, they may want users to enjoy higher bitrate video and segments to be downloaded at a fast rate. They could want the infrastructure to be able to scale to service tens of millions of content viewers and for those viewers to have an instant video experience. IPv6 is needed to fulfill these expectations.
The factor determining latency is Round-Trip Time or RTT. If RTT is 100 milliseconds (ms), it will take a total of 400 ms after the three-way handshake of TCP, TLS, and HTTP. However, in reality, it could take much longer. As more people work remotely during the prolonged pandemic, without the help of a content delivery network, there will be choking points causing longer RTT.
This is where IPv6 comes in. Alex explained that throughput and RTT in IPv6 is often faster than IPv4, and the reasons are many. First, we often see a smaller routing table with IPv6 and more efficient routing. There is also no checksum in IPv6. He stated that we don’t usually see as much fragmentation in IPv6 at IPv4 because fragmentation can be handled by source devices, although Alex mentioned that he was going to review Geoff’s presentation on fragmentation to learn more. Lastly, the IPv6 world got rid of Network Address Translation (NAT) which means there are fewer middleboxes to cause latency increase.
Alex shared that his team has measured recent RTT improvement from IPv4 to IPv6 using US mobile carriers T-Mobile, Verizon and AT&T/Sprint. They found that at T-Mobile IPv6 connections are 49-64% faster than IPv4. T-Mobile has invested the most in IPv6 infrastructure in comparison to the other companies and has seen the largest RTT improvement.
IPv6 Deployment and Challenges in Online Games in Japan
Noboru Matsumoto, Director of the Japan Internet Providers Association (JAIPA), was last on the docket and led an interesting discussion about game and entertainment network connectivity issues. Japan has been facing connectivity issues with online gaming for years, so a workgroup was established to address the problem. The workgroup has been working together since 2019 and members include CE router vendors and carrier-grade device vendors, network operators, and game companies such as Nintendo, Konami Digital Entertainment and Sony Interactive Entertainment.
Noboru described how the IPv6 deployment status for home access lines has progressed to just over 80%, but the deployment of game content is not progressing at any significant rate, leading to unsatisfactory experiences for gamers. Gamers often encounter disconnects which can trigger the games to recognize them as malicious players due to the frequency. Once a gamer is flagged as being malicious, it can be difficult to get a clear solution from the ISP, game company, or router vendor support.
The disconnection issues Noboru laid out have lead to increased costs for customer support and troubleshooting and software modification for CE vendors for game companies. So why hasn’t game content progressed if it’s better for gamers and game companies?
IPv6 deployment is difficult for online game companies because it involves an expensive investment. Engineer resources for verification of various environment networks and complicated online behavior (asset download, authentication, player matching system, relay server, P2P fallback system) are major costs. Not to mention the need to keep both IPv4 and IPv6.
So why is game content performing so poorly over IPv6? The workgroup discussed issues spanning from NAT traversal in online games, NAT session timers unmatching between gamers and CE routers and IPv4 address sharing. While the problem is a combination of all of these issues, the workgroup found that the main cause is increased IPv4 address sharing.
Once we learned the main cause, Noboru detailed what we can do to make improvements. In the short term, standardized common terms for customer support and information sharing across industries will help. But in the long term, progress must be made toward IPv6 deployment. Noboru closed out his presentation by highlighting the importance of the customer experience and that improvement is imperative.
Overall, APNIC 52 delivered a great session on IPv6 deployment. IPv6 has been steadily gaining ground as nations around the world have been making the transition, so it was fascinating to learn about the presenters’ experiences as part of the shift to IPv6.
In his closing remarks, Geoff Huston said, “We still have so much to learn about v6. It’s just one of those learning experiences that never stops.” Cheers to continuous learning and to APNIC for providing a space to do so.
If you missed the IPv6 Deployment session and would like to watch a recorded version, you can find it on APNIC’s YouTube.
Still hanging in at #APNIC52, though Cinder (my cat) has officially decided I’m crazy. Great content so far. Just waiting on the v6 discussions 3:15am EST 🙂 pic.twitter.com/ks1ALlI6mb
— Kat Hunter (@katsanimehats) September 14, 2021
Playing the usual game of “just how many screens do you need to participate in #APNIC52 and do work at the same time.” Who else is running more than 5 screens? pic.twitter.com/jZ8yg3Sr7z
— Lim May-Ann (@eilonwy) September 14, 2021
