{"id":482,"date":"2019-08-06T17:12:13","date_gmt":"2019-08-06T17:12:13","guid":{"rendered":"https:\/\/www.spotonoracle.com\/?p=482"},"modified":"2019-08-07T05:59:38","modified_gmt":"2019-08-07T05:59:38","slug":"tracing-data-guard-traffic","status":"publish","type":"post","link":"https:\/\/www.spotonoracle.com\/?p=482","title":{"rendered":"Tracing Data Guard traffic"},"content":{"rendered":"<p>Sometimes it is helpful to trace Data Guard background services so we can look at matching NSSn and RFS traces. For deep dives we also want to run TCP captures on both ends of a Data Guard configuration, and possibly on network components in the middle.<br \/>\nTo minimize processing overhead on devices and noise in the capture files we want the packet filter to be as specific as possible. Just the source and target IP is not good enough, we also need a port number. Ideally, we apply the following filters to &#8220;tcpdump&#8221; in order to caputre the entire NSS <-> RFS traffic (but nothing more):<\/p>\n<pre class=\"brush: plain; collapse: false; title: ; wrap-lines: false; notranslate\" title=\"\">\r\ntcpdump 'host &lt;primary IP&gt; and port &lt;NSSn port&gt;'\r\n<\/pre>\n<p><em>NSS<strong>n<\/strong> is the process shipping the redo data from primary to standby. <strong>n<\/strong> is the number matching the LOG_ARCHIVE_DEST_<strong>n<\/strong> parameter for your redo transport configuration<\/em><\/p>\n<p>Getting the IP of the primary host is easy, even if there is a separate network for Data Guard.<br \/>\nBut how do we get the PORT of the NSSn process?<\/p>\n<p><strong>Method A<\/strong><br \/>\nIdentify the NSSn process on the primary:<\/p>\n<pre class=\"brush: sql; collapse: false; highlight: [4]; title: ; wrap-lines: false; notranslate\" title=\"\">\r\nselect ses.sid, ses.serial#, ses.machine, ses.port, prc.pname, prc.spid, prc.stid, prc.tracefile\r\nfrom gv$session ses\r\n  join gv$process prc on (prc.inst_id = ses.inst_id and prc.addr = ses.paddr)\r\nwhere prc.pname like 'NSS%'\r\n;\r\n<\/pre>\n<p><a href=\"https:\/\/www.spotonoracle.com\/?attachment_id=483\" rel=\"attachment wp-att-483\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-NSSx.png\" alt=\"\" width=\"936\" height=\"46\" class=\"alignnone size-full wp-image-483\" srcset=\"https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-NSSx.png 936w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-NSSx-300x15.png 300w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-NSSx-768x38.png 768w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-NSSx-624x31.png 624w\" sizes=\"auto, (max-width: 936px) 100vw, 936px\" \/><\/a><br \/>\nThe trick is to use the SPID as it is reflected in &#8220;v$session.process&#8221; on the standby database instance. In this case we query for process 4582 on the standby database:<\/p>\n<pre class=\"brush: sql; collapse: false; highlight: [4]; title: ; wrap-lines: false; notranslate\" title=\"\">\r\nselect ses.sid, ses.serial#, ses.machine, ses.port, prc.pname, prc.spid, prc.stid, prc.tracefile\r\nfrom gv$session ses\r\n  join gv$process prc on (prc.inst_id = ses.inst_id and prc.addr = ses.paddr)\r\nwhere ses.process = '4582'\r\n;\r\n<\/pre>\n<p><a href=\"https:\/\/www.spotonoracle.com\/?attachment_id=487\" rel=\"attachment wp-att-487\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/standby-RFS.png\" alt=\"\" width=\"928\" height=\"42\" class=\"alignnone size-full wp-image-487\" srcset=\"https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/standby-RFS.png 928w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/standby-RFS-300x14.png 300w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/standby-RFS-768x35.png 768w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/standby-RFS-624x28.png 624w\" sizes=\"auto, (max-width: 928px) 100vw, 928px\" \/><\/a><\/p>\n<p>In our example the port is 18274 resulting in this &#8220;tcpdump&#8221; command:<\/p>\n<pre class=\"brush: plain; collapse: false; title: ; wrap-lines: false; notranslate\" title=\"\">\r\ntcpdump 'host 192.168.55.71 and port 18274'\r\n<\/pre>\n<p><strong>Method B<\/strong><br \/>\nUse the same query to get the SPID of the NSSn process on the primary as in the first method.<br \/>\nWith the SPID we can find the TCP connection details using the &#8220;ss&#8221; Linux utility:<\/p>\n<pre class=\"brush: plain; collapse: false; title: ; wrap-lines: false; notranslate\" title=\"\">\r\nss -o -p -n -i | grep 4582\r\n<\/pre>\n<p><a href=\"https:\/\/www.spotonoracle.com\/?attachment_id=489\" rel=\"attachment wp-att-489\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-ss.png\" alt=\"\" width=\"1600\" height=\"34\" class=\"alignnone size-full wp-image-489\" srcset=\"https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-ss.png 1600w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-ss-300x6.png 300w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-ss-768x16.png 768w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-ss-1024x22.png 1024w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/primary-ss-624x13.png 624w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><\/a><\/p>\n<p>Voil\u00e0.<\/p>\n<p><strong>SQL trace<\/strong><br \/>\nObviously, with the respective SID\/SERIAL# combinations you can enable the SQL trace on NSSn (primary) and RFS (standby):<\/p>\n<p>Primary<\/p>\n<pre class=\"brush: sql; collapse: false; title: ; wrap-lines: false; notranslate\" title=\"\">\r\nexec dbms_monitor.session_trace_enable(session_id=&gt; 83, serial_num =&gt; 57385, waits =&gt; true, binds =&gt; false)\r\n<\/pre>\n<p>Standby<\/p>\n<pre class=\"brush: sql; collapse: false; title: ; wrap-lines: false; notranslate\" title=\"\">\r\nexec dbms_monitor.session_trace_enable(session_id=&gt; 191, serial_num =&gt; 37747, waits =&gt; true, binds =&gt; false)\r\n<\/pre>\n<p><strong>Observation about the RFS process<\/strong><br \/>\nThe RFS trace file name shows as &#8220;DEV1_ora_4187.trc&#8221; in &#8220;v$process&#8221;. In fact, RFS is not a background process (v$session.type = &#8216;USER&#8217;).<br \/>\nBut here&#8217;s an oddity, when activating SQL trace on RFS the trace file does not materialize on the filesystem with that name. It actually materializes with &#8220;DEV1_rfs_4187.trc&#8221;.<br \/>\n<a href=\"https:\/\/www.spotonoracle.com\/?attachment_id=490\" rel=\"attachment wp-att-490\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/RFS-trace-file-name.png\" alt=\"\" width=\"546\" height=\"79\" class=\"alignnone size-full wp-image-490\" srcset=\"https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/RFS-trace-file-name.png 546w, https:\/\/www.spotonoracle.com\/wp-content\/uploads\/2019\/08\/RFS-trace-file-name-300x43.png 300w\" sizes=\"auto, (max-width: 546px) 100vw, 546px\" \/><\/a><\/p>\n<p><strong>RAC and Data Guard<\/strong><br \/>\nIn a MMA environment you&#8217;d probably want to capture traffic from all primary instances (on the host with the apply instance):<\/p>\n<pre class=\"brush: plain; collapse: false; title: ; wrap-lines: false; notranslate\" title=\"\">\r\ntcpdump '(host &lt;primary IP A&gt; and port &lt;NSSn port A&gt;) or (host &lt;primary IP B&gt; and port &lt;NSSn port B&gt;)'\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes it is helpful to trace Data Guard background services so we can look at matching NSSn and RFS traces. For deep dives we also want to run TCP captures on both ends of a Data Guard configuration, and possibly on network components in the middle. To minimize processing overhead on devices and noise in [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-482","post","type-post","status-publish","format-standard","hentry","category-internals"],"_links":{"self":[{"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=\/wp\/v2\/posts\/482","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=482"}],"version-history":[{"count":27,"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=\/wp\/v2\/posts\/482\/revisions"}],"predecessor-version":[{"id":513,"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=\/wp\/v2\/posts\/482\/revisions\/513"}],"wp:attachment":[{"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=482"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=482"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.spotonoracle.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=482"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}