Repository navigation
how to guarantee packets sending to specific process #418
Description
Activity
- changed the title
[-]how to guarantee packets[/-][+]how to guarantee packets send to specific process[/+]on Aug 5, 2019 - changed the title
[-]how to guarantee packets send to specific process[/-][+]how to guarantee packets sending to specific process[/+]on Aug 5, 2019 it seems that f-stack will run ff_rss_check when pick up a port to connect to remote server . But I don't see when this function will be called . Is it called after ff_connect() ?
More info : my nic is virtio driver.
will ff_rss_check run correctly under this situation ? I really don't know there would be another reason that tcp connection failsActually , I found there is an another ff_api called ff_regist_packet_dispatcher. What I thought is :
if process 1 on core 1, process 2 on core 2 , I just bind the src_port before ff_connect ，maybe the port range is 10000-20000 in core 1 , port range is 20000-30000 in core 2 . And then write the dispatch callback:if dstport in 10000-20000: sendto core1 if dstport in 20000-30000: sendto core2
to realize a function like flow director.
do you think it's feasible ？ If so , how to use this function . And the queue id seems a little difficult to get . And how to specify the parameter of ff_regist_packet_dispatcher ? Do you have any examples ? do i need to put ff_regist_packet_dispatcher function before ff_run() to init ?Another solution is to check the result of ff_rss_check . But it seems to change the code is a little difficult .
Could you give me some advice ? Thanks so much.
See #231 (comment).
Thanks for your reply.
yes，I have seen this issue before. However, I still don’t know how to use this function in C.
Could you give me some example to use it ? Like how to use it in /example/main_epoll.c .Thanks very much!
like below:
int pipeline_dispatch_cb (void *data, uint16_t *len, uint16_t queue_id, uint16_t nb_queues) { struct ipv4_hdr *iph; int iph_len; uint32_t hash; iph = (struct ipv4_hdr *)(data + ETHER_HDR_LEN); iph_len = (iph->version_ihl & 0x0f) << 2; if (iph->next_proto_id != IPPROTO_IP) { return queue_id; } iph = (struct ipv4_hdr *)((char *)iph + iph_len); iph_len = (iph->version_ihl & 0x0f) << 2; if (iph->next_proto_id == IPPROTO_TCP) { struct tcp_hdr *tcph = (struct tcp_hdr *)((char *)iph + iph_len); hash = get_hash(iph->src_addr, iph->dst_addr, tcph->src_port, tcph->dst_port); } else if (iph->next_proto_id == IPPROTO_UDP) { struct udp_hdr *udph = (struct udp_hdr *)((char *)iph + iph_len); hash = get_hash(iph->src_addr, iph->dst_addr, udph->src_port, udph->dst_port); } else { return queue_id; } return hash % nb_queues; } int main(int argc, char * argv[]) { ff_init(argc, argv); /* regist a packet dispath function */ ff_regist_packet_dispatcher(pipeline_dispatch_cb); . . . }@jfb8856606 Thank you so much . It works for me. Another question is : is this function called after dpdk receives packets and put them on the ring buffer related to different process ?
Yes, it will put the muf another ring buffer to different process while return value != queue_id.
@jfb8856606
Hi
I tried your code with a little modify:int port_dispatcher(void *data, uint16_t *len, uint16_t queue_id, uint16_t nb_queues) { struct ipv4_hdr *iph; int iph_len; iph = (struct ipv4_hdr *)((char*)data + ETHER_HDR_LEN); iph_len = (iph->version_ihl & 0x0f) << 2; if (iph->next_proto_id != IPPROTO_IP) { fprintf(stderr, "not ip \n"); return queue_id; } iph = (struct ipv4_hdr *)((char *)iph + iph_len); iph_len = (iph->version_ihl & 0x0f) << 2; if (iph->next_proto_id == IPPROTO_TCP) { struct tcp_hdr *tcph = (struct tcp_hdr *)((char *)iph + iph_len); uint64_t port = tcph->src_port; fprintf(stderr, "port %lu, nb_queue %u\n", port, nb_queues); //hash = get_hash(iph->src_addr, iph->dst_addr, tcph->src_port, tcph->dst_port); return (port - 8000) % nb_queues; } fprintf(stderr, "not tcp \n"); return queue_id; }
However, it keeps printing 'not ip'. Can you please help me ?
In my demo, it's process ip-ip tunnel packets.
You need to modify it according to your network environment, such as comment out this code
if (iph->next_proto_id != IPPROTO_IP) { fprintf(stderr, "not ip \n"); return queue_id; } iph = (struct ipv4_hdr *)((char *)iph + iph_len); iph_len = (iph->version_ihl & 0x0f) << 2;In my demo, it's process ip-ip tunnel packets.
You need to modify it according to your network environment, such as comment out this code
if (iph->next_proto_id != IPPROTO_IP) { fprintf(stderr, "not ip \n"); return queue_id; } iph = (struct ipv4_hdr *)((char *)iph + iph_len); iph_len = (iph->version_ihl & 0x0f) << 2;Yes I use a standard tcp-ip protocol.
The client is using ordinary linux network stack with tcp connection.
I think your code can directly be used in my case. But it doesn't work.In my demo, it's process ip-ip tunnel packets.
You need to modify it according to your network environment, such as comment out this code
if (iph->next_proto_id != IPPROTO_IP) { fprintf(stderr, "not ip \n"); return queue_id; } iph = (struct ipv4_hdr *)((char *)iph + iph_len); iph_len = (iph->version_ihl & 0x0f) << 2;Oh I got what you mean. Thanks a lot :D
This issue has been fully resolved in the discussion above. To summarize:
Root cause: In a multi-process F-Stack client setup, RSS hashes incoming packets (e.g., SYN+ACK) based on the 5-tuple independently of which process sent the outgoing SYN. The reply may land on a different process than the one that initiated the connection, causing those TCP handshakes to silently fail.
Solution: Use
ff_regist_packet_dispatcher()to register a custom packet dispatch callback. Assign each process a distinct source port range, then route incoming packets to the correct process based on the destination port:int pipeline_dispatch_cb(void *data, uint16_t *len, uint16_t queue_id, uint16_t nb_queues) { struct rte_ether_hdr *eth = (struct rte_ether_hdr *)data; struct rte_ipv4_hdr *iph = (struct rte_ipv4_hdr *)(eth + 1); int iph_len = (iph->version_ihl & 0x0f) << 2; if (iph->next_proto_id == IPPROTO_TCP) { struct rte_tcp_hdr *tcph = (struct rte_tcp_hdr *)((char *)iph + iph_len); /* route by destination port (= the source port we used when connecting) */ return ntohs(tcph->dst_port) % nb_queues; } return queue_id; } int main(int argc, char *argv[]) { ff_init(argc, argv); ff_regist_packet_dispatcher(pipeline_dispatch_cb); /* ... */ ff_run(loop, NULL); }
Call
ff_regist_packet_dispatcher()afterff_init()and beforeff_run().Note on VLAN: if
vlan_strip=1is enabled, use the newerff_regist_packet_dispatcher_context()API (added in PR #1029) which provides VLAN context to the callback.Note on the sample code in comments: the example posted above parses an IP-in-IP tunnel header. For standard TCP/IP environments, skip the inner IP header parsing and work directly on the outer IP + TCP headers as shown above.
Closing as resolved.
hi , f-stack team
here I have ported an application to f-stack. It is :
1 single process with one thread ( thread is to collect data and no ff_api in it )
2 it’s a client app which send tcp connection to a nginx server.
And if I run this app on one cpu core , there is no problem. the command is below:
and the
lcore_maskin fileconfig.iniis set to 1 :[dpdk] # Hexadecimal bitmask of cores to run on. lcore_mask=1However, when I want to run my app with multi process on multi cores , there’s some error.
I want run first myapp on core 0 and second myapp on core 1.
The command is :
And the file
config.ini:[dpdk] # Hexadecimal bitmask of cores to run on. lcore_mask=3And both of these two apps run and
usis up to 100% usingtopcommand.But I found that some tcp connections were time out. And there were about half of these tcp connections failed because of timeout . Half of tcp connections were successful. Then I captured packets on the server . I found that after server sends syn+ack packets to client , Myapp just can receive about half syn+ack packets and then send ack to server to establish the tcp connection . But the other syn+ack packets sent from server could not be received by client . So myapp would always send first syn packet to server and the connection couldn’t be established .
AFAK, if a server app ported on F-stack , the RSS will hash the five tuple from client to process. And then these packets from same tcp flow would be always sent to same process . But if a client app ported on F-stack , client would send packets out to server to establish tcp connections. However , RSS maybe not hash the packets from server to the same process. For example , process 1 on core 1 ,process 2 on core 2 ,both on f-stack as clients :
I don’t know if my understanding is right and I guess this is the reason. So how to guarantee the packet from server would be sent to a specific process . I think maybe RSS couldn’t do this mission. Is there another way ? How the nginx proxy realized ?
I would appreciate it if you could offer any help.